Documentation
¶
Overview ¶
Package sessionkit holds the small backend-neutral helpers shared by the modeld transport.Session adapters (llama.cpp in modeld/llama/llamasession and OpenVINO in modeld/openvino). Both adapters drive the same EnsurePrefix/PrefillSuffix/Decode contract over genuinely different engines, so their KV mechanics stay separate — but the surrounding plumbing (cancel-safe stream sends, longest-common-prefix reuse, the chat-template role vocabulary) is identical and lives here once instead of drifting as per-adapter copies.
Index ¶
- func ChatRole(kind string) string
- func CommonPrefixLen(a, b []int) int
- func ResidencyReport(plan residency.Plan, errMsg string, caps residency.Capabilities) *transport.ResidencyReport
- func Send[T any](ctx context.Context, ch chan<- T, v T) bool
- func StructuredToolCallChunk(text string) (transport.StreamChunk, error)
- func TransportToolCalls(in []ParsedToolCall) ([]transport.ToolCall, error)
- func TrySend[T any](ch chan<- T, v T)
- type ParsedToolCall
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ChatRole ¶
ChatRole maps a manifest segment kind to a chat-template role, or "" for a control segment the model's own template renders itself (BOS, the assistant generation cue). Centralizing the role vocabulary keeps the two adapters from recognizing different role sets.
func CommonPrefixLen ¶
CommonPrefixLen returns the length of the longest shared prefix of a and b. The adapters use it to find how many already-resident tokens a new prefix can reuse before the divergent tail must be (re)prefilled.
func ResidencyReport ¶
func ResidencyReport(plan residency.Plan, errMsg string, caps residency.Capabilities) *transport.ResidencyReport
ResidencyReport maps a residency.Plan (plus any planning error and the backend's capabilities) onto the backend-neutral transport.ResidencyReport for explain-context observability, so the two adapters surface residency the same way. It returns nil when there is no plan to report (empty session).
func Send ¶
Send delivers v on ch, or reports false if ctx is canceled before a slot is free. Decode loops use the bool to stop streaming once the consumer is gone.
func StructuredToolCallChunk ¶ added in v0.33.0
func StructuredToolCallChunk(text string) (transport.StreamChunk, error)
StructuredToolCallChunk parses the complete text of a structured tool-call generation into a StreamChunk carrying transport tool calls. It accepts the two shapes constrained decoding produces: a JSON envelope ({"content":…,"tool_calls":[…]} or a bare call object) and Qwen-style <tool_call>…</tool_call> tag blocks.
func TransportToolCalls ¶ added in v0.33.0
func TransportToolCalls(in []ParsedToolCall) ([]transport.ToolCall, error)
TransportToolCalls normalizes permissive tool-call shapes onto transport.ToolCall, defaulting IDs and the "function" type.
Types ¶
type ParsedToolCall ¶ added in v0.33.0
type ParsedToolCall struct {
ID string `json:"id"`
Type string `json:"type"`
Name string `json:"name"`
ToolName string `json:"tool_name"`
Arguments json.RawMessage `json:"arguments"`
Parameters json.RawMessage `json:"parameters"`
Function struct {
Name string `json:"name"`
Arguments json.RawMessage `json:"arguments"`
Parameters json.RawMessage `json:"parameters"`
} `json:"function"`
}
ParsedToolCall is the permissive wire shape of one model-emitted tool call. Models and native parsers disagree on field names (name/tool_name/function, arguments/parameters), so it accepts all spellings; TransportToolCalls normalizes them onto the backend-neutral transport.ToolCall. Shared by both backend adapters so structured tool-call output parses identically regardless of which engine constrained the generation.
Source Files