llama

type AdapterSpec ¶ added in v0.32.8

type AdapterSpec = transport.AdapterSpec

AdapterSpec is one LoRA adapter applied to a session. Its digest and scale are part of the session/manifest cache identity: base+A and base+B must not share warm KV (see docs/development/blueprints/modeld/lora-adapters.md).

type Config ¶

type Config = transport.Config

Config is the explicit runtime configuration for a local session — every knob is a tested setting, not a magic default.

type ContextManifest ¶

type ContextManifest = contextasm.ContextManifest

type ContextOverflowError ¶

type ContextOverflowError struct {
	Stage            string
	ResidentTokens   int
	AdditionalTokens int
	NumCtx           int
}

ContextOverflowError carries token counts for an overflow at a specific primitive boundary.

func (e *ContextOverflowError) Error() string

func (e *ContextOverflowError) Is(target error) bool

func (e *ContextOverflowError) OverflowDetail() transport.ContextOverflowDetail

type ContextReport ¶

type ContextReport = transport.ContextReport

ContextReport explains the session's resident context (explain-context). It aliases the transport wire type for the same anti-drift reason as the inputs above.

type DecodeConfig ¶

type DecodeConfig struct {
	MaxTokens        int
	Temperature      *float64
	TopP             *float64
	TopK             int
	Seed             *int
	ParserProtocols  []string
	ReasoningFormat  string
	StructuredOutput transport.StructuredOutputConfig
}

DecodeConfig controls a single decode pass.

type EmbedFunc ¶

type EmbedFunc func(ctx context.Context, modelPath string, cfg Config, input string) ([]float64, error)

EmbedFunc computes a single embedding via the native backend. The llama.cpp adapter registers one from its init when built with the 'llamanode' tag.

type ManifestMismatchError ¶

type ManifestMismatchError = contextasm.ManifestMismatchError

type ManifestSegment ¶

type ManifestSegment = contextasm.ManifestSegment

type PrefixInput ¶

type PrefixInput = transport.PrefixInput

PrefixInput is the stable prefix text plus the manifest that makes reuse valid (tokenizer, template, runtime config, BOS policy, and model identity are part of the cache key — byte equality alone is not enough).

type PrefixStatus ¶

type PrefixStatus = transport.PrefixStatus

PrefixStatus reports what EnsurePrefix reused versus (re)computed — the live-reuse signal: ReusedTokens > 0 means a warm hit.

type Session ¶

type Session interface {
	// EnsurePrefix makes the resident KV equal `prefix`, reusing the longest
	// already-resident matching token prefix and prefilling only the divergent
	// tail (this also drops any previous suffix and generated tokens).
	EnsurePrefix(ctx context.Context, prefix PrefixInput) (PrefixStatus, error)

	// PrefillSuffix prefills the volatile suffix (diff / test output / user turn)
	// after the stable prefix, leaving the stable KV untouched.
	PrefillSuffix(ctx context.Context, suffix SuffixInput) (SuffixStatus, error)

	// Decode streams generated text from the current resident state.
	Decode(ctx context.Context, cfg DecodeConfig) (<-chan StreamChunk, error)

	// ExplainContext reports the resident context for observability.
	ExplainContext() ContextReport

	// Snapshot captures backend state for durability, branching, and benchmark
	// reproducibility.
	Snapshot(ctx context.Context) (SessionSnapshot, error)

	// Restore replaces resident state from a compatible snapshot.
	Restore(ctx context.Context, snap SessionSnapshot) error

	// Close releases the session's resources.
	Close() error
}

Session is a persistent, workspace-scoped inference session.

The hot coding loop is: keep the stable prefix's KV hot, prefill only the changed suffix, decode. EnsurePrefix does token-level longest-common-prefix reuse, so an unchanged stable workspace context stays warm across turns and only the divergent tail is recomputed.

type SessionFactory ¶

type SessionFactory func(modelPath string, cfg Config) (Session, error)

SessionFactory creates a backend session for a model with explicit config.

type SessionSnapshot ¶ added in v0.32.4

type SessionSnapshot = transport.SessionSnapshot

type StreamChunk ¶

type StreamChunk struct {
	Text      string
	Thinking  string
	ToolCalls []ToolCall
	Error     error
}

StreamChunk is a decoded text delta, parsed model output, or a terminal error.

type SuffixInput ¶

type SuffixInput = transport.SuffixInput

SuffixInput is the volatile text appended after the stable prefix, carrying the same manifest so a suffix cannot be prefilled against resident KV from a different profile/template/runtime.

type SuffixStatus ¶

type SuffixStatus = transport.SuffixStatus

SuffixStatus reports the volatile suffix added after the stable prefix.

type TokenizeFunc ¶

type TokenizeFunc = contextasm.TokenizeFunc

type ToolCall ¶ added in v0.32.4

type ToolCall = transport.ToolCall

type UnsupportedFeatureError ¶

type UnsupportedFeatureError struct {
	Feature string
}

UnsupportedFeatureError describes a deliberately unsupported surface.

func (e *UnsupportedFeatureError) Error() string

func (e *UnsupportedFeatureError) Is(target error) bool