llama

package
v0.35.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 12, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package llama defines the modeld-side llama backend contract: persistent inference sessions keep a stable prefix's KV hot and re-prefill only the changed suffix.

This package defines the backend-neutral session contract. Backend adapters implement it; product code talks to Session, not llama.cpp internals. Snapshot/restore is part of the same contract for durability and branching. The main generation path is EnsurePrefix -> PrefillSuffix -> Decode on a live session.

Index

Constants

View Source 
const DefaultMinHotContextTokens = 4096

DefaultMinHotContextTokens is the usable-context floor modeld guarantees for an auto (unpinned) session. A chat model handed only a few hundred KV tokens silently degrades — it cannot even hold a system prompt, so instruction following collapses — so modeld prefers to shed GPU layers, trading some speed for a usable window, and refuses only when even a minimal offload cannot reach the floor. Operators override it with modeld.json memory.min_hot_context_tokens or CONTENOX_MODELD_MIN_HOT_CONTEXT.

Variables

View Source 
var (
	// ErrSessionUnavailable means no native llama backend was compiled into
	// this binary.
	ErrSessionUnavailable = errors.New("llama: session backend unavailable")
	// ErrSessionClosed means the caller used a closed persistent session.
	//
	// ErrSessionClosed, ErrContextOverflow, and ErrUnsupportedFeature alias the
	// transport sentinels (as manifest.go already aliases contextasm.ErrManifestMismatch)
	// so the daemon's session errors survive the modeld wire boundary: the gRPC
	// error map (runtime/transport/grpc/errors.go) keys on the transport.Err*
	// values, and an error that is not Is-compatible with them is downgraded to
	// codes.Internal. The typed *ContextOverflowError / *UnsupportedFeatureError
	// still wrap these, so errors.As keeps returning their structured fields.
	ErrSessionClosed = transport.ErrSessionClosed
	// ErrContextOverflow means a prefix, suffix, or decode would exceed NumCtx.
	ErrContextOverflow = transport.ErrContextOverflow
	// ErrUnsupportedFeature marks explicit product-surface gaps such as tools.
	ErrUnsupportedFeature = transport.ErrUnsupportedFeature
	// ErrSessionFatal means the backend marked the session unusable and callers
	// must evict it instead of trying to reuse resident KV. It aliases the
	// transport sentinel so the existing decode/restore emissions classify over
	// the modeld wire boundary instead of degrading to codes.Internal.
	ErrSessionFatal = transport.ErrSessionFatal
)
View Source 
var ErrManifestMismatch = contextasm.ErrManifestMismatch

ErrManifestMismatch is returned when a prefix/suffix cannot be safely paired with resident KV under the current manifest.

Functions

func BuildCommit added in v0.32.6 

func BuildCommit() string

BuildCommit returns the pinned llama.cpp source commit this backend was built against, as injected at link time. It is empty for a plain `go build` with no -ldflags. Cheap and side-effect free, so `modeld version` can report it without loading native libraries.

func ContextLength added in v0.34.0 

func ContextLength(ggufPath string) (int, error)

ContextLength returns a llama GGUF model's trained context ceiling by reading only file metadata (no tensor data, no device query) — the cheap header-only part of what Describe's capacity planner already computes. Callers (e.g. modelstore.Admin.ListModels) should treat an error as "unknown" and skip enrichment for that one model, not fail the scan.

func EmbedAvailable 

func EmbedAvailable() bool

EmbedAvailable reports whether an embedding backend is compiled into this build.

func NewContextOverflowError 

func NewContextOverflowError(stage string, resident, additional, numCtx int) error

func NewManifestMismatchError 

func NewManifestMismatchError(reason string) error

NewManifestMismatchError builds a manifest-mismatch error with a reason.

func NewUnsupportedFeatureError 

func NewUnsupportedFeatureError(feature string) error

func RuntimeInfo added in v0.32.4 

func RuntimeInfo() transport.ModelInfo

RuntimeInfo reports the linked llama.cpp runtime identity and device inventory. In non-direct builds this returns an empty record.

func SessionAvailable 

func SessionAvailable() bool

SessionAvailable reports whether a session backend is compiled into this build.

func SetEmbedFunc 

func SetEmbedFunc(f EmbedFunc)

SetEmbedFunc registers the native embedding backend.

func SetSessionFactory 

func SetSessionFactory(f SessionFactory)

SetSessionFactory registers the backend that creates sessions. The llama.cpp adapter (./llamasession) calls this from its init when built with the 'llamanode' tag, so the provider never imports the CGo package directly (no import cycle, default build stays CGo-free).

Types

type AdapterSpec added in v0.32.8 

type AdapterSpec struct {
	Name   string
	Path   string
	Digest string
	Scale  float32
}

AdapterSpec identifies one LoRA adapter to apply to a session: a GGUF adapter file (Path) applied at Scale, plus Name/Digest carried for cache identity and diagnostics. Applying an adapter does not modify the base model weights, but it changes model behavior, so adapter identity must be part of every session and manifest cache key (see docs/development/blueprints/modeld/lora-adapters.md). It mirrors the transport-level adapter handle without importing the wire shape here.

type ColdKVBlock added in v0.33.0 

type ColdKVBlock = transport.ColdKVBlock

type Config 

type Config = transport.Config

type ContextManifest 

type ContextManifest = contextasm.ContextManifest

The llama backend keys warm KV reuse on the backend-neutral context manifest owned by the runtime (runtime/contextasm, surfaced to the runtime as transport.ContextManifest). These aliases let the llama.cpp session adapter and its tests refer to those types through this package without importing contextasm directly. The manifest is assembled by the runtime and crosses the transport; modeld only fills the backend-resolved token data during prefill.

type ContextOverflowError 

type ContextOverflowError = transport.ContextOverflowError

type ContextReport 

type ContextReport = transport.ContextReport

type DecodeConfig 

type DecodeConfig = transport.DecodeConfig

type EmbedFunc 

type EmbedFunc func(ctx context.Context, modelPath string, cfg Config, input string) ([]float64, error)

EmbedFunc computes a single embedding via the native backend. The llama.cpp adapter registers one from its init when built with the 'llamanode' tag.

type ManifestMismatchError 

type ManifestMismatchError = contextasm.ManifestMismatchError

The llama backend keys warm KV reuse on the backend-neutral context manifest owned by the runtime (runtime/contextasm, surfaced to the runtime as transport.ContextManifest). These aliases let the llama.cpp session adapter and its tests refer to those types through this package without importing contextasm directly. The manifest is assembled by the runtime and crosses the transport; modeld only fills the backend-resolved token data during prefill.

type ManifestSegment 

type ManifestSegment = contextasm.ManifestSegment

The llama backend keys warm KV reuse on the backend-neutral context manifest owned by the runtime (runtime/contextasm, surfaced to the runtime as transport.ContextManifest). These aliases let the llama.cpp session adapter and its tests refer to those types through this package without importing contextasm directly. The manifest is assembled by the runtime and crosses the transport; modeld only fills the backend-resolved token data during prefill.

type PrefixInput 

type PrefixInput = transport.PrefixInput

type PrefixStatus 

type PrefixStatus = transport.PrefixStatus

type Service added in v0.32.2 

type Service struct {
	// contains filtered or unexported fields
}

Service implements the runtime/transport.Service boundary. It acts as the opener for native llama.cpp backend sessions.

func NewService added in v0.32.4 

func NewService(opts ...ServiceOption) *Service

func (*Service) Describe added in v0.32.2 

func (s *Service) Describe(_ context.Context, req transport.OpenSessionRequest) (transport.ModelInfo, error)

Describe reports the model's trained context window read from the GGUF header (no tensor load). The runtime consumes this as the model's capacity; it never reads the GGUF itself.

func (*Service) Embed added in v0.32.4 

func (s *Service) Embed(ctx context.Context, req transport.EmbedRequest) (transport.EmbedResult, error)

Embed runs a one-shot native llama.cpp embedding for req.Text through the embedding backend registered by the CGo session package (see llamasession.embed). Like OpenVINO's Embed it is separate from OpenSession: embedding models do not use the chat session's prefix/suffix/Decode lifecycle. In a build without the native backend (no 'llamanode' tag) the embed func is unregistered and this reports ErrUnsupportedFeature.

func (*Service) OpenSession added in v0.32.2 

func (s *Service) OpenSession(ctx context.Context, req transport.OpenSessionRequest) (transport.Session, error)

OpenSession binds a session to the requested model. It rejects a model typed for a different backend (ErrBackendMismatch) before loading, so a GGUF request sent to an openvino-mode daemon — or vice versa — fails at the boundary, not deep in the engine. The model is loaded from req.Path (resolved by the runtime); identity/caching uses req.Digest.

type ServiceOption added in v0.32.4 

type ServiceOption func(*Service)

func WithCapacityPolicy added in v0.32.4 

func WithCapacityPolicy(p capacity.Policy) ServiceOption

func WithHostMemorySource added in v0.32.6 

func WithHostMemorySource(src capacity.MemorySource) ServiceOption

func WithMemorySource added in v0.32.4 

func WithMemorySource(src capacity.MemorySource) ServiceOption

type Session 

type Session = transport.Session

type SessionFactory 

type SessionFactory func(modelPath string, cfg Config, adapters []AdapterSpec) (Session, error)

SessionFactory creates a backend session for a model with explicit config and any LoRA adapters to apply to the session. Empty adapters = the base model.

type SessionSnapshot added in v0.32.4 

type SessionSnapshot = transport.SessionSnapshot

type StreamChunk 

type StreamChunk = transport.StreamChunk

type SuffixInput 

type SuffixInput = transport.SuffixInput

type SuffixStatus 

type SuffixStatus = transport.SuffixStatus

type TokenizeFunc 

type TokenizeFunc = contextasm.TokenizeFunc

The llama backend keys warm KV reuse on the backend-neutral context manifest owned by the runtime (runtime/contextasm, surfaced to the runtime as transport.ContextManifest). These aliases let the llama.cpp session adapter and its tests refer to those types through this package without importing contextasm directly. The manifest is assembled by the runtime and crosses the transport; modeld only fills the backend-resolved token data during prefill.

type ToolCall added in v0.32.4 

type ToolCall = transport.ToolCall

type UnsupportedFeatureError 

type UnsupportedFeatureError struct {
	Feature string
}

UnsupportedFeatureError describes a deliberately unsupported surface.

func (*UnsupportedFeatureError) Error 

func (e *UnsupportedFeatureError) Error() string

func (*UnsupportedFeatureError) Is 

func (e *UnsupportedFeatureError) Is(target error) bool

Source Files

View all Source files

Directories

Path Synopsis
Package llamacppshim owns the direct llama.cpp C API boundary for modeld.
 Package llamacppshim owns the direct llama.cpp C API boundary for modeld.
Capability mapping kept in an untagged, cgo-free file so the backend-parity contract can pin it verbatim in plain CI: capability drift must fail there, not only in tagged native builds.
 Capability mapping kept in an untagged, cgo-free file so the backend-parity contract can pin it verbatim in plain CI: capability drift must fail there, not only in tagged native builds.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.