Documentation
¶
Overview ¶
Package tensor provides an N-dimensional computation tensor with NCHW memory layout, float32-only data, and factory functions for creating, indexing, slicing, reshaping, and broadcasting tensors.
Index ¶
- Constants
- Variables
- func AttachQ8_0(t *Tensor, q *Q8_0Tensor)
- func BroadcastCompatible(a, b *Tensor) bool
- func BroadcastShape(a, b *Tensor) ([]int, error)
- type Allocator
- type Pool
- type Q8_0Tensor
- type Tensor
- func FromONNXTensor(data []float32, shape []int64) (*Tensor, error)
- func FromSlice(data []float32, shape ...int) (*Tensor, error)
- func Full(value float32, shape ...int) *Tensor
- func New(shape ...int) *Tensor
- func NewFromPool(pool *Pool, shape ...int) *Tensor
- func NewPlaceholder(shape ...int) *Tensor
- func Ones(shape ...int) *Tensor
- func (t *Tensor) At(indices ...int) (float32, error)
- func (t *Tensor) Clone() *Tensor
- func (t *Tensor) Data() []float32
- func (t *Tensor) MustAt(indices ...int) float32
- func (t *Tensor) MustSetAt(value float32, indices ...int)
- func (t *Tensor) Ndim() int
- func (t *Tensor) Release(pool *Pool)
- func (t *Tensor) Reshape(shape ...int) (*Tensor, error)
- func (t *Tensor) SetAt(value float32, indices ...int) error
- func (t *Tensor) Shape() []int
- func (t *Tensor) ShapeRef() []int
- func (t *Tensor) Size() int
- func (t *Tensor) Slice(dim, start, end int) (*Tensor, error)
- func (t *Tensor) Strides() []int
- func (t *Tensor) StridesRef() []int
Constants ¶
const Q8_0BlockBytes = 4 + Q8_0BlockSize
Q8_0BlockBytes is the on-disk / in-memory byte width of a Q8_0 block: 4 bytes for the fp32 scale plus 32 bytes for the int8 weights.
const Q8_0BlockSize = 32
Q8_0BlockSize is the number of fp32 weights packed into a single Q8_0 block.
Variables ¶
var ( // ErrShapeMismatch indicates a shape incompatibility such as a reshape // size change, FromSlice length mismatch, or incompatible broadcast. ErrShapeMismatch = errors.New("tensor: shape mismatch") // ErrInvalidDimension indicates negative or zero dimensions, an // invalid slice dimension, or an out-of-range index. ErrInvalidDimension = errors.New("tensor: invalid dimension") // ErrUseAfterRelease indicates an attempt to use a tensor whose // underlying buffer has already been returned to the pool. ErrUseAfterRelease = errors.New("tensor: use after release") // ErrIndexOutOfBounds indicates an At/SetAt call whose index arity // or value did not match the tensor's shape. Hot paths use raw // Data() access and never see this error; callers that index // dynamically (tests, debug tools) must check for it. ErrIndexOutOfBounds = errors.New("tensor: index out of bounds") )
Sentinel errors returned by tensor operations.
Functions ¶
func AttachQ8_0 ¶
func AttachQ8_0(t *Tensor, q *Q8_0Tensor)
AttachQ8_0 records q as the Q8_0 sidecar for the given fp32 tensor. The tensor's float32 data is typically left as a placeholder (its shape is what carries the logical metadata for downstream operators); the int8 weights live in q.Blocks. Callers MUST keep the *Tensor pointer stable for the lifetime of the sidecar, which is the natural case for ONNX initializers loaded once at engine start.
Calling AttachQ8_0 with a nil q removes any existing sidecar.
func BroadcastCompatible ¶
BroadcastCompatible reports whether the shapes of a and b are compatible under ONNX/NumPy broadcasting rules. Shapes are right-aligned, and for each pair of dimensions they must be equal or one of them must be 1.
func BroadcastShape ¶
BroadcastShape computes the output shape that results from broadcasting tensors a and b together. It returns ErrShapeMismatch if the shapes are not broadcast-compatible.
Types ¶
type Allocator ¶
Allocator is a function that creates a tensor with the given shape. It abstracts over tensor.New (heap) and NewFromPool (pooled) so that operators can be allocation-strategy agnostic.
type Pool ¶
type Pool struct {
// contains filtered or unexported fields
}
Pool is a memory pool for float32 slices used as tensor backing buffers. Slices are bucketed by power-of-2 capacity so that similarly sized allocations reuse the same buffers, eliminating per-Run heap churn.
Internally the buckets are stored as a fixed-size array indexed by log2(capacity) (computed via bits.Len) so that Get and Put avoid the hash and bounds-check overhead of a Go map. The mutex protects the bucket stacks so the pool is safe for concurrent use — required for tile-level parallelism where multiple goroutines run inferences against the same pool.
func (*Pool) BucketCount ¶
BucketCount returns the number of buffers currently available in the bucket whose capacity equals bucketSize. Intended for test observability only. Thread-safe.
func (*Pool) Get ¶
Get returns a zero-cleared []float32 of the requested length. The underlying capacity is rounded up to the next power of 2 so that similarly sized requests share a bucket. If a previously returned buffer is available it is reused; otherwise a fresh slice is allocated.
A size <= 0 returns nil. Thread-safe.
type Q8_0Tensor ¶
type Q8_0Tensor struct {
// Shape is the logical (unquantized) tensor shape.
Shape []int
// Blocks holds ceil(numel/32) contiguous Q8_0 blocks. Each block is
// 36 bytes: a little-endian fp32 scale followed by 32 int8 weights.
Blocks []byte
}
Q8_0Tensor is a weight-only int8 quantized carrier mirroring the llama.cpp Q8_0 block layout: each block holds 32 int8 weights paired with one fp32 scale factor. A weight is reconstructed as dequant(q_i) = float32(q_i) * scale.
The block layout in Blocks is byte-packed and contiguous:
block_i := [4 bytes fp32 scale (little-endian)] [32 bytes int8 weights]
i.e. 36 bytes per 32 weights, ≈ 9 bits/weight versus 32 bits for fp32.
Shape carries the *logical* tensor shape (e.g. [outC, inC, kH, kW] for a Conv weight). The flat element index for shape walking is the same as for an fp32 tensor; the in-memory layout differs in that the bytes are partitioned into Q8_0 blocks.
NumElements is computed lazily from Shape; if Shape's product is not a multiple of 32, the final block is zero-padded on the int8 side. Callers MUST never read beyond the logical element count.
Q8_0Tensor is a separate carrier (rather than a flag on *Tensor) so the hot fp32 paths in ops/*.go remain monomorphic on the float32 backing slice and the type switch happens once per Conv at dispatch time.
func LookupQ8_0 ¶
func LookupQ8_0(t *Tensor) *Q8_0Tensor
LookupQ8_0 returns the Q8_0 sidecar for t, or nil if t is a regular fp32 tensor. The returned *Q8_0Tensor MUST NOT be mutated by callers; the same pointer is shared across all Conv dispatches.
func NewQ8_0Tensor ¶
func NewQ8_0Tensor(shape []int, blocks []byte) *Q8_0Tensor
NewQ8_0Tensor constructs a Q8_0Tensor with the given logical shape and packed block bytes. The caller is responsible for ensuring len(blocks) equals ceil(product(shape)/32) * Q8_0BlockBytes; this constructor performs no validation so it can be used on the hot model-load path.
func (*Q8_0Tensor) NumBlocks ¶
func (q *Q8_0Tensor) NumBlocks() int
NumBlocks returns the number of Q8_0 blocks backing this tensor, i.e. ceil(NumElements/32).
func (*Q8_0Tensor) NumElements ¶
func (q *Q8_0Tensor) NumElements() int
NumElements returns the logical element count of the tensor (the product of Shape). Padding inside the final block is NOT included.
type Tensor ¶
type Tensor struct {
// contains filtered or unexported fields
}
Tensor is an N-dimensional array of float32 values stored in row-major (C-contiguous, NCHW) memory layout.
func FromONNXTensor ¶
FromONNXTensor creates a tensor from raw float32 data and an int64 shape, matching the types returned by onnx.Tensor's FloatData() and Shape fields. It converts the []int64 shape to []int, validates that no dimension is negative or zero, and delegates to FromSlice. The tensor package never imports internal/onnx/onnx/; the caller is responsible for extracting the raw values before calling this function.
func FromSlice ¶
FromSlice wraps an existing float32 slice as a tensor without copying. It returns ErrShapeMismatch if len(data) does not equal the product of the shape dimensions.
func Full ¶
Full creates a tensor filled with the given value. It panics if the shape is empty or contains negative dimensions.
func New ¶
New creates a zero-initialized tensor with the given shape. It panics if the shape is empty or contains negative dimensions.
func NewFromPool ¶
NewFromPool creates a zero-initialized tensor with the given shape, drawing its backing buffer from the provided Pool instead of the heap. The returned tensor is otherwise identical to one created by New.
func NewPlaceholder ¶
NewPlaceholder creates a tensor with the given shape but NO backing data. It is used for GPU-resident tensors where only shape metadata is needed and the actual data lives in GPU buffers. Calling Data() returns nil.
func (*Tensor) At ¶
At reads one element by N-dimensional index. It computes the flat offset via the dot product of indices and strides. It returns ErrIndexOutOfBounds if the number of indices does not match the number of dimensions or any index is out of range.
Operator hot paths bypass At entirely and read Data() directly; this API exists for callers that index dynamically (tests, debug tools, programmatic graph inspection) where the error path is preferable to a runtime panic.
func (*Tensor) Clone ¶
Clone returns a deep copy of the tensor with independent data, shape, and strides slices.
func (*Tensor) Data ¶
Data returns the underlying float32 slice directly (shared, not copied) for zero-copy performance in operators.
func (*Tensor) MustAt ¶
MustAt is the panic-on-error wrapper around At, intended for tests and debug tools where an indexing error is a programmer mistake. Do not call MustAt from production code: use At and handle the error.
func (*Tensor) MustSetAt ¶
MustSetAt is the panic-on-error wrapper around SetAt, intended for tests and debug tools. Production code must use SetAt and handle the returned error.
func (*Tensor) Release ¶
Release returns the tensor's backing buffer to the pool and nils the data slice, preventing further use. It is a no-op if pool is nil or the tensor's data has already been released.
func (*Tensor) Reshape ¶
Reshape returns a new tensor sharing the same underlying data with the given shape. It returns ErrShapeMismatch if the product of the new shape differs from the current total size.
func (*Tensor) SetAt ¶
SetAt writes one element by N-dimensional index. It returns ErrIndexOutOfBounds if the number of indices does not match the number of dimensions or any index is out of range.
func (*Tensor) ShapeRef ¶
ShapeRef returns the internal shape slice directly without copying. The caller must not modify the returned slice. Use this on hot paths where the defensive copy from Shape() is unnecessary overhead.
func (*Tensor) Slice ¶
Slice extracts a contiguous sub-tensor along one dimension, returning a new tensor with copied data. It returns ErrInvalidDimension if dim is out of range or start/end are invalid (negative, start >= end, or end > shape[dim]).
func (*Tensor) StridesRef ¶
StridesRef returns the internal strides slice directly without copying. The caller must not modify the returned slice. Use this on hot paths where the defensive copy from Strides() is unnecessary overhead.
Source Files