cache

package
v0.40.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 20, 2026 License: MIT Imports: 6 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Attention

type Attention interface {
	Cache

	// Update appends (k, v) and returns an opaque nn.KVHistory for
	// this layer's SDPA.
	Update(b *batch.Batch, keys, values *mlx.Array) *nn.KVHistory

	// View returns the current attention history without writing.
	View(b *batch.Batch) *nn.KVHistory
}

Attention is the contract for caches that back attention layers (KVCache, RotatingKVCache).

type Cache

type Cache interface {
	// State returns the cache-owned state roots that should be kept/evaluated.
	State() []*mlx.Array
	Free()
	Offset() int

	// Snapshot retains state from fromOffset to the current offset without
	// changing the active cache. Storage may be copied, shared immutably, or
	// copied lazily before the active cache overwrites it.
	Snapshot(fromOffset int) Snapshot

	// PrepareSnapshots schedules the cache to capture a snapshot as its
	// storage offset reaches each listed offset during subsequent writes.
	// Offsets are storage offsets, must not already be passed (current
	// offset <= offset), and must be sorted ascending and unique. Scheduled
	// offsets persist across multiple writes until TakeSnapshots is called;
	// captures accumulate. The previous schedule must be drained first.
	//
	// Unlike Snapshot, capture happens at the moment a write crosses the
	// scheduled offset, so it can record states interior to a batched write
	// without the caller breaking the write into pieces.
	PrepareSnapshots(offsets []int)

	// TakeSnapshots returns the snapshots captured since PrepareSnapshots,
	// one per scheduled offset in the caller's order, and clears the
	// schedule. An entry may be nil for an empty range or an offset that
	// writes never reached, for example after cancellation.
	TakeSnapshots() []Snapshot

	// Restore brings the cache to target. If snapshot is nil, rewinds
	// using the cache's own live state. Returns false if the requested state
	// is unavailable or target is negative.
	Restore(snapshot Snapshot, target int) bool

	// Merge combines two sequential snapshots [a,b) and [b,c) into [a,c).
	// Takes ownership of both inputs.
	Merge(parent, child Snapshot) Snapshot

	// Split divides a snapshot [a,c) at offset b into [a,b) and [b,c).
	// Takes ownership of the input. Cache types that cannot split
	// (e.g. recurrent) return (nil, snapshot).
	Split(snapshot Snapshot, at int) (parent, child Snapshot)
}

Cache is common state management shared by every cache kind. Writers live on the specific caches

type HiddenCache added in v0.40.0

type HiddenCache struct {
	// contains filtered or unexported fields
}

HiddenCache retains backbone outputs for heads that read earlier tokens. It follows the same prefix snapshots as the attention and recurrent caches. Arrays are immutable: appending or restoring builds a new view or concatenation, so a snapshot never observes a later request's writes.

func NewHiddenCache added in v0.40.0

func NewHiddenCache() *HiddenCache

func (*HiddenCache) Append added in v0.40.0

func (c *HiddenCache) Append(hidden *mlx.Array)

func (*HiddenCache) Free added in v0.40.0

func (c *HiddenCache) Free()

func (*HiddenCache) Merge added in v0.40.0

func (c *HiddenCache) Merge(parent, child Snapshot) Snapshot

func (*HiddenCache) Offset added in v0.40.0

func (c *HiddenCache) Offset() int

func (*HiddenCache) PrepareSnapshots added in v0.40.0

func (c *HiddenCache) PrepareSnapshots(offsets []int)

func (*HiddenCache) Restore added in v0.40.0

func (c *HiddenCache) Restore(snapshot Snapshot, target int) bool

func (*HiddenCache) Snapshot added in v0.40.0

func (c *HiddenCache) Snapshot(start int) Snapshot

func (*HiddenCache) Split added in v0.40.0

func (c *HiddenCache) Split(snapshot Snapshot, at int) (Snapshot, Snapshot)

func (*HiddenCache) State added in v0.40.0

func (c *HiddenCache) State() []*mlx.Array

func (*HiddenCache) TakeSnapshots added in v0.40.0

func (c *HiddenCache) TakeSnapshots() []Snapshot

type KVCache

type KVCache struct {
	// contains filtered or unexported fields
}

func NewKVCache

func NewKVCache() *KVCache

func (*KVCache) Free

func (c *KVCache) Free()

func (*KVCache) Merge

func (c *KVCache) Merge(parent, child Snapshot) Snapshot

func (*KVCache) Offset

func (c *KVCache) Offset() int

func (*KVCache) PrepareSnapshots

func (c *KVCache) PrepareSnapshots(offsets []int)

func (*KVCache) Restore

func (c *KVCache) Restore(snapshot Snapshot, target int) bool

func (*KVCache) Snapshot

func (c *KVCache) Snapshot(fromOffset int) Snapshot

func (*KVCache) Split

func (c *KVCache) Split(snapshot Snapshot, at int) (Snapshot, Snapshot)

func (*KVCache) State

func (c *KVCache) State() []*mlx.Array

func (*KVCache) TakeSnapshots

func (c *KVCache) TakeSnapshots() []Snapshot

func (*KVCache) Update

func (c *KVCache) Update(_ *batch.Batch, keys, values *mlx.Array) *nn.KVHistory

Assumes B = 1; heterogeneous batches are not supported.

func (*KVCache) View

func (c *KVCache) View(_ *batch.Batch) *nn.KVHistory

View returns the current cache contents as attention history without writing.

type RecurrentCache

type RecurrentCache struct {
	// contains filtered or unexported fields
}

RecurrentCache stores state for linear-recurrent layers.

Conv state takes its dtype from the first Get call (the activation dtype). Delta state is always float32: the gated-delta recurrent accumulator runs for the full sequence length and needs the extra precision regardless of activation dtype.

Conv state shape: [B, convTail, convDim] Delta state shape: [B, numVHeads, headVDim, headKDim]

func NewRecurrentCache

func NewRecurrentCache(convTail, convDim, numVHeads, headVDim, headKDim int32) *RecurrentCache

func (*RecurrentCache) CommitCompaction added in v0.40.4

func (c *RecurrentCache) CommitCompaction(compact *mlx.Array)

CommitCompaction replaces the live state only after the copy has evaluated.

func (*RecurrentCache) Free

func (c *RecurrentCache) Free()

func (*RecurrentCache) Get

func (c *RecurrentCache) Get(b *batch.Batch, dtype mlx.DType) *nn.RecurrentHistory

Get returns the current conv/delta state for the SSM layer's read phase. On first call it lazy-initializes zero-filled state tensors sized from b.InputIDs and dtyped from the caller's activation dtype. On subsequent calls it returns the existing state; batch size and dtype must match the first call, since recurrent state is cumulative and cannot be reshaped without losing history.

func (*RecurrentCache) Merge

func (c *RecurrentCache) Merge(parent, child Snapshot) Snapshot

func (*RecurrentCache) Offset

func (c *RecurrentCache) Offset() int

func (*RecurrentCache) PrepareCompaction added in v0.40.4

func (c *RecurrentCache) PrepareCompaction() *mlx.Array

PrepareCompaction returns nil unless the live delta state may share a multi-state backing allocation. The caller must evaluate a returned copy while the source remains in scope, then commit it. A normal Put replaces the restored state with the kernel's final state and needs no copy.

func (*RecurrentCache) PrepareSnapshots

func (c *RecurrentCache) PrepareSnapshots(offsets []int)

PrepareSnapshots schedules snapshot capture. Recurrent state is cumulative; an interior offset within a forward has no state unless the recurrent kernel is run in segments cut at that offset (see SnapshotSplits + Put). The current offset is a boundary now (the pre-forward state) and is captured immediately. Interior offsets are captured when Put receives the matching per-boundary state; the end offset is captured by Put's final state.

func (*RecurrentCache) Put

func (c *RecurrentCache) Put(b *batch.Batch, convStates, deltaStates []*mlx.Array)

Put stores the conv/delta states produced by the SSM layer's write phase. convStates/deltaStates are the per-boundary recurrent states, one per boundary ending with the forward-end state. The boundaries align with this forward's snapshot splits plus the end: the leading entries are captured as snapshots at the scheduled interior offsets, and the final entry becomes the committed live state, advancing the cache offset by the forward's real token count.

In the common (unsegmented) case both slices have length 1 — just the forward-end state.

Assumes B = 1; heterogeneous batches are not supported.

func (*RecurrentCache) Restore

func (c *RecurrentCache) Restore(snapshot Snapshot, target int) bool

func (*RecurrentCache) Snapshot

func (c *RecurrentCache) Snapshot(fromOffset int) Snapshot

func (*RecurrentCache) SnapshotSplits

func (c *RecurrentCache) SnapshotSplits(forwardLen int) []int

SnapshotSplits returns the scheduled offsets strictly interior to the upcoming forward [offset, offset+forwardLen), expressed relative to the forward start — the points at which the caller must segment the recurrent kernel so each interior state can be captured. Empty when nothing is scheduled or no interior offsets fall in range.

func (*RecurrentCache) Split

func (c *RecurrentCache) Split(snapshot Snapshot, at int) (Snapshot, Snapshot)

func (*RecurrentCache) State

func (c *RecurrentCache) State() []*mlx.Array

func (*RecurrentCache) TakeSnapshots

func (c *RecurrentCache) TakeSnapshots() []Snapshot

type RotatingKVCache

type RotatingKVCache struct {
	// contains filtered or unexported fields
}

RotatingKVCache implements sliding window attention with bounded memory.

func NewRotatingKVCache

func NewRotatingKVCache(maxSize int) *RotatingKVCache

func (*RotatingKVCache) Free

func (c *RotatingKVCache) Free()

func (*RotatingKVCache) Merge

func (c *RotatingKVCache) Merge(parent, child Snapshot) Snapshot

func (*RotatingKVCache) Offset

func (c *RotatingKVCache) Offset() int

func (*RotatingKVCache) PrepareSnapshots

func (c *RotatingKVCache) PrepareSnapshots(offsets []int)

func (*RotatingKVCache) Restore

func (c *RotatingKVCache) Restore(snapshot Snapshot, target int) bool

func (*RotatingKVCache) Snapshot

func (c *RotatingKVCache) Snapshot(fromOffset int) Snapshot

func (*RotatingKVCache) Split

func (c *RotatingKVCache) Split(snapshot Snapshot, at int) (Snapshot, Snapshot)

func (*RotatingKVCache) State

func (c *RotatingKVCache) State() []*mlx.Array

func (*RotatingKVCache) TakeSnapshots

func (c *RotatingKVCache) TakeSnapshots() []Snapshot

func (*RotatingKVCache) Update

func (c *RotatingKVCache) Update(b *batch.Batch, keys, values *mlx.Array) *nn.KVHistory

Assumes B = 1; heterogeneous batches are not supported.

func (*RotatingKVCache) View

func (c *RotatingKVCache) View(b *batch.Batch) *nn.KVHistory

View returns the current cache contents as a read-only KV history, used by an assistant model that shares this cache. It sets L=1 so rotatingApplier treats the buffer as ring-ordered (its stored layout); L=1 is a layout selector, not a query length. A post-concat oversize buffer (K > maxSize) is already in logical order, so View trims to the trailing maxSize tokens and resets ringIdx to 0, collapsing the applier's gather to identity.

type Snapshot

type Snapshot interface {
	// Size returns the retained VRAM bytes charged to the snapshot. Immutable
	// views may conservatively charge shared backing storage more than once. A lazy
	// snapshot that still indexes a live cache buffer returns 0 — it owns
	// no extra memory yet. Once materialized (the cache copies the range
	// out before overwriting its slots), Size returns the owned bytes.
	Size() int

	// SetMaterializeHook installs a callback fired once when a lazy
	// snapshot materializes (allocates its owned arrays). delta is the
	// newly-allocated byte count. Pass nil to detach. Snapshots that are
	// never lazy may treat this as a no-op.
	SetMaterializeHook(func(delta int))

	// Close releases the snapshot's ownership of retained arrays.
	Close()
}

Snapshot is paged-out cache state that can be restored later.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL