Documentation
¶
Overview ¶
Package residency contains modeld's backend-neutral KV residency policy.
It deliberately owns only logical decisions: which token ranges should remain hot under a derived budget, and which ranges may be moved cold. Backend adapters execute those decisions only when their engine exposes the necessary KV controls.
Index ¶
- func ClassForSegment(kind string, stable bool, explicit string) contextasm.CacheClass
- func ParseCacheClass(tag string) (contextasm.CacheClass, bool)
- type Block
- type BlockFlags
- type Capabilities
- type Controller
- type EvictionBudget
- type Executor
- type ManifestOptions
- type MissingTokenRangesError
- type Plan
- type PlanInput
- type Range
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ClassForSegment ¶
func ClassForSegment(kind string, stable bool, explicit string) contextasm.CacheClass
ClassForSegment returns the explicit manifest cache class when valid, else a conservative default derived from segment kind and stable/volatile placement.
func ParseCacheClass ¶
func ParseCacheClass(tag string) (contextasm.CacheClass, bool)
ParseCacheClass parses the stable manifest cache-class tag.
Types ¶
type Block ¶
type Block struct {
Range Range
Kind string
Stable bool
CacheClass contextasm.CacheClass
TokenHash string
LastUsed int64
Flags BlockFlags
Segment int
SplitOrdinal int
}
Block is the planner's unit of residency. Ranges must not overlap.
func BlocksFromManifest ¶
func BlocksFromManifest(m contextasm.ContextManifest, opts ManifestOptions) ([]Block, error)
BlocksFromManifest converts backend-tokenized manifest segments into logical residency blocks. Missing CacheClass tags are normalized from kind/stability.
type BlockFlags ¶
type BlockFlags uint16
BlockFlags are policy hints applied above CacheClass. Sinks and recent-window blocks are protected because sparse/streaming attention requires them hot.
const ( FlagPinned BlockFlags = 1 << iota FlagSink FlagRecent FlagRetrieved )
func (BlockFlags) Has ¶
func (f BlockFlags) Has(want BlockFlags) bool
type Capabilities ¶
type Capabilities struct {
RemoveTail bool
RemoveMiddle bool
PositionShift bool
SparseAttention bool
SlidingWindowAttentionTokens int
ColdStore bool
RecomputeRange bool
}
Capabilities describes what a backend adapter can actually execute.
type Controller ¶
type Controller interface {
Capabilities() Capabilities
}
Controller is the optional engine-facing seam. It is intentionally not part of runtime/transport.Session.
type EvictionBudget ¶
type EvictionBudget struct {
SinkTokens int // always-hot leading tokens (attention sinks)
RecentTokens int // always-hot trailing window
MaxTokens int // hot budget; eviction keeps physical KV within this
}
EvictionBudget is the sink/recent/max split a backend uses to bound its hot KV while letting generation continue past the physical window. Both adapters derive it the same way so llama (imperative slide) and OpenVINO (declarative CacheEvictionConfig) enforce one policy.
func DeriveEvictionBudget ¶
func DeriveEvictionBudget(windowTokens, blockSize int) EvictionBudget
DeriveEvictionBudget splits a served window into attention sinks, a recent window, and the total hot budget. It is eviction-algorithm policy (à la StreamingLLM/H2O), not hardware sizing: ~1/16 of the window as sinks, ~1/4 as the recent window, Max = the window. blockSize aligns sizes for block-based caches (OpenVINO); pass <=1 for token-granular backends (llama). Windows too small to split keep everything hot (Valid() is false → no eviction).
func (EvictionBudget) Valid ¶
func (b EvictionBudget) Valid() bool
Valid reports whether the split can drive an eviction config: non-zero sizes with an evictable middle (Max > Sink + Recent).
type Executor ¶
type Executor interface {
Controller
EvictRange(ctx context.Context, r Range) error
AdmitRange(ctx context.Context, r Range) error
}
Executor is implemented only by adapters that can mutate physical KV ranges.
type ManifestOptions ¶
type ManifestOptions struct {
// ResidentTokens limits the manifest to the token range currently resident.
// This lets callers run the planner after EnsurePrefix before volatile
// segments have token ranges.
ResidentTokens int
// BlockSize splits large manifest segments into uniform logical blocks.
// A non-positive value keeps each segment as one block.
BlockSize int
// LastUsed is copied onto every generated block. Callers that track richer
// access recency can rewrite LastUsed before planning.
LastUsed int64
// RequireComplete reports missing ranges for every non-empty segment. Leave
// false for prefix-only planning after EnsurePrefix, where volatile suffix
// ranges have not been tokenized yet.
RequireComplete bool
}
ManifestOptions controls block construction from a transport manifest.
type MissingTokenRangesError ¶
type MissingTokenRangesError struct {
Segments []string
}
MissingTokenRangesError reports non-empty manifest segments that are within the resident region but have not yet been backend-tokenized.
func (*MissingTokenRangesError) Error ¶
func (e *MissingTokenRangesError) Error() string
type Plan ¶
type Plan struct {
BudgetTokens int
TotalTokens int
HotTokens int
ProtectedTokens int
OverBudget bool
KeepHot []Block
EvictCold []Block
Diagnostics []string
}
Plan is the planner output: KeepHot plus EvictCold partitions the input blocks. HotTokens can exceed BudgetTokens only when protected blocks alone do.
func PlanHotSet ¶
PlanHotSet produces the hot/cold partition for a token budget.
type PlanInput ¶
type PlanInput struct {
Blocks []Block
BudgetTokens int
// SinkTokens and RecentTokens mark blocks overlapping those token spans as
// protected. The spans are coarse by design; block splitting controls
// precision.
SinkTokens int
RecentTokens int
}
PlanInput is the pure policy input. BudgetTokens must be derived by capacity planning, not chosen here.
Source Files