llama

package
v0.32.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 24, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package llama defines the modeld-side llama backend contract: persistent inference sessions keep a stable prefix's KV hot and re-prefill only the changed suffix.

This package defines the backend-neutral session contract. Backend adapters implement it; product code talks to Session, not llama.cpp internals. Snapshot/restore is part of the same contract for durability and branching. The main generation path is EnsurePrefix -> PrefillSuffix -> Decode on a live session.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrSessionUnavailable means no native llama backend was compiled into
	// this binary.
	ErrSessionUnavailable = errors.New("llama: session backend unavailable")
	// ErrSessionClosed means the caller used a closed persistent session.
	//
	// ErrSessionClosed, ErrContextOverflow, and ErrUnsupportedFeature alias the
	// transport sentinels (as manifest.go already aliases contextasm.ErrManifestMismatch)
	// so the daemon's session errors survive the modeld wire boundary: the gRPC
	// error map (runtime/transport/grpc/errors.go) keys on the transport.Err*
	// values, and an error that is not Is-compatible with them is downgraded to
	// codes.Internal. The typed *ContextOverflowError / *UnsupportedFeatureError
	// still wrap these, so errors.As keeps returning their structured fields.
	ErrSessionClosed = transport.ErrSessionClosed
	// ErrContextOverflow means a prefix, suffix, or decode would exceed NumCtx.
	ErrContextOverflow = transport.ErrContextOverflow
	// ErrUnsupportedFeature marks explicit product-surface gaps such as tools.
	ErrUnsupportedFeature = transport.ErrUnsupportedFeature
	// ErrSessionFatal means the backend marked the session unusable and callers
	// must evict it instead of trying to reuse resident KV. It aliases the
	// transport sentinel so the existing decode/restore emissions classify over
	// the modeld wire boundary instead of degrading to codes.Internal.
	ErrSessionFatal = transport.ErrSessionFatal
)
View Source 
var ErrManifestMismatch = contextasm.ErrManifestMismatch

ErrManifestMismatch is returned when a prefix/suffix cannot be safely paired with resident KV under the current manifest.

Functions

func BuildCommit added in v0.32.6 

func BuildCommit() string

BuildCommit returns the pinned llama.cpp source commit this backend was built against, as injected at link time. It is empty for a plain `go build` with no -ldflags. Cheap and side-effect free, so `modeld version` can report it without loading native libraries.

func EmbedAvailable 

func EmbedAvailable() bool

EmbedAvailable reports whether an embedding backend is compiled into this build.

func NewContextOverflowError 

func NewContextOverflowError(stage string, resident, additional, numCtx int) error

func NewManifestMismatchError 

func NewManifestMismatchError(reason string) error

NewManifestMismatchError builds a manifest-mismatch error with a reason.

func NewUnsupportedFeatureError 

func NewUnsupportedFeatureError(feature string) error

func RuntimeInfo added in v0.32.4 

func RuntimeInfo() transport.ModelInfo

RuntimeInfo reports the linked llama.cpp runtime identity and device inventory. In non-direct builds this returns an empty record.

func SessionAvailable 

func SessionAvailable() bool

SessionAvailable reports whether a session backend is compiled into this build.

func SetEmbedFunc 

func SetEmbedFunc(f EmbedFunc)

SetEmbedFunc registers the native embedding backend.

func SetSessionFactory 

func SetSessionFactory(f SessionFactory)

SetSessionFactory registers the backend that creates sessions. The llama.cpp adapter (./llamasession) calls this from its init when built with the 'llamanode' tag, so the provider never imports the CGo package directly (no import cycle, default build stays CGo-free).

Types

type Config 

type Config = transport.Config

type ContextManifest 

type ContextManifest = contextasm.ContextManifest

The llama backend keys warm KV reuse on the backend-neutral context manifest owned by the runtime (runtime/contextasm, surfaced to the runtime as transport.ContextManifest). These aliases let the llama.cpp session adapter and its tests refer to those types through this package without importing contextasm directly. The manifest is assembled by the runtime and crosses the transport; modeld only fills the backend-resolved token data during prefill.

type ContextOverflowError 

type ContextOverflowError struct {
	Stage            string
	ResidentTokens   int
	AdditionalTokens int
	NumCtx           int
}

ContextOverflowError carries token counts for an overflow at a specific primitive boundary.

func (*ContextOverflowError) Error 

func (e *ContextOverflowError) Error() string

func (*ContextOverflowError) Is 

func (e *ContextOverflowError) Is(target error) bool

type ContextReport 

type ContextReport = transport.ContextReport

type DecodeConfig 

type DecodeConfig = transport.DecodeConfig

type EmbedFunc 

type EmbedFunc func(ctx context.Context, modelPath string, cfg Config, input string) ([]float64, error)

EmbedFunc computes a single embedding via the native backend. The llama.cpp adapter registers one from its init when built with the 'llamanode' tag.

type ManifestMismatchError 

type ManifestMismatchError = contextasm.ManifestMismatchError

The llama backend keys warm KV reuse on the backend-neutral context manifest owned by the runtime (runtime/contextasm, surfaced to the runtime as transport.ContextManifest). These aliases let the llama.cpp session adapter and its tests refer to those types through this package without importing contextasm directly. The manifest is assembled by the runtime and crosses the transport; modeld only fills the backend-resolved token data during prefill.

type ManifestSegment 

type ManifestSegment = contextasm.ManifestSegment

The llama backend keys warm KV reuse on the backend-neutral context manifest owned by the runtime (runtime/contextasm, surfaced to the runtime as transport.ContextManifest). These aliases let the llama.cpp session adapter and its tests refer to those types through this package without importing contextasm directly. The manifest is assembled by the runtime and crosses the transport; modeld only fills the backend-resolved token data during prefill.

type PrefixInput 

type PrefixInput = transport.PrefixInput

type PrefixStatus 

type PrefixStatus = transport.PrefixStatus

type Service added in v0.32.2 

type Service struct {
	// contains filtered or unexported fields
}

Service implements the runtime/transport.Service boundary. It acts as the opener for native llama.cpp backend sessions.

func NewService added in v0.32.4 

func NewService(opts ...ServiceOption) *Service

func (*Service) Describe added in v0.32.2 

func (s *Service) Describe(_ context.Context, req transport.OpenSessionRequest) (transport.ModelInfo, error)

Describe reports the model's trained context window read from the GGUF header (no tensor load). The runtime consumes this as the model's capacity; it never reads the GGUF itself.

func (*Service) Embed added in v0.32.4 

func (s *Service) Embed(_ context.Context, req transport.EmbedRequest) (transport.EmbedResult, error)

Embed is not served through the modeld transport for llama yet. The runtime's llama provider still owns its native one-shot embedding path separately.

func (*Service) OpenSession added in v0.32.2 

func (s *Service) OpenSession(ctx context.Context, req transport.OpenSessionRequest) (transport.Session, error)

OpenSession binds a session to the requested model. It rejects a model typed for a different backend (ErrBackendMismatch) before loading, so a GGUF request sent to an openvino-mode daemon — or vice versa — fails at the boundary, not deep in the engine. The model is loaded from req.Path (resolved by the runtime); identity/caching uses req.Digest.

type ServiceOption added in v0.32.4 

type ServiceOption func(*Service)

func WithCapacityPolicy added in v0.32.4 

func WithCapacityPolicy(p capacity.Policy) ServiceOption

func WithHostMemorySource added in v0.32.6 

func WithHostMemorySource(src capacity.MemorySource) ServiceOption

func WithMemorySource added in v0.32.4 

func WithMemorySource(src capacity.MemorySource) ServiceOption

type Session 

type Session = transport.Session

type SessionFactory 

type SessionFactory func(modelPath string, cfg Config) (Session, error)

SessionFactory creates a backend session for a model with explicit config.

type SessionSnapshot added in v0.32.4 

type SessionSnapshot = transport.SessionSnapshot

type StreamChunk 

type StreamChunk = transport.StreamChunk

type SuffixInput 

type SuffixInput = transport.SuffixInput

type SuffixStatus 

type SuffixStatus = transport.SuffixStatus

type TokenizeFunc 

type TokenizeFunc = contextasm.TokenizeFunc

The llama backend keys warm KV reuse on the backend-neutral context manifest owned by the runtime (runtime/contextasm, surfaced to the runtime as transport.ContextManifest). These aliases let the llama.cpp session adapter and its tests refer to those types through this package without importing contextasm directly. The manifest is assembled by the runtime and crosses the transport; modeld only fills the backend-resolved token data during prefill.

type ToolCall added in v0.32.4 

type ToolCall = transport.ToolCall

type UnsupportedFeatureError 

type UnsupportedFeatureError struct {
	Feature string
}

UnsupportedFeatureError describes a deliberately unsupported surface.

func (*UnsupportedFeatureError) Error 

func (e *UnsupportedFeatureError) Error() string

func (*UnsupportedFeatureError) Is 

func (e *UnsupportedFeatureError) Is(target error) bool

Source Files

View all Source files

Directories

Path Synopsis
Package llamacppshim owns the direct llama.cpp C API boundary for modeld.
 Package llamacppshim owns the direct llama.cpp C API boundary for modeld.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.