modelcontext

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 29, 2026 License: Apache-2.0, MIT Imports: 13 Imported by: 0

README

Model context boundary

modelcontext.Registry is the only application-facing boundary for values that are shortened before an LLM call and restored afterwards. The whole codec lives in this one package:

File Role
registry.go Registry facade — the only type request lifecycles use
handle_table.go handleTable[M], the generic bidirectional primitive behind every handle space
sources.go source handles (chunk://N/doc://N/kb://N/url://N) and the tool-argument codec
source_text.go handle syntax, weknora:// durable URIs, and free-text / streaming handle rewriting
citations.go protocol prompts, <ref/> → <kb/>/<web/> expansion, citation stream expander
model_output.go compact source-centric renderings of tool results
resources.go durable-resource handles (res://NNNN)
stream.go streamHold suffix-hold primitive and every streaming decoder
tool_policy.go the single policy layer: per-tool key contracts + sourceKeySpaces dispatch
mcp.go MCP bridge routing handles (msN/mtN), definition projection and envelope codec
mcp_sources.go bounded citation sidecar for HTTP(S) links in successful MCP results
handles.go exported HandleTable for invocation-local spaces (iN, ref-N, c000)
leaks.go LeakedIdentifiers report of raw UUIDs that survive EncodeMessages, logged at every model-call site

Identity rules

  • Durable application identities: UUIDs, Wiki slugs, URLs, and resource://... references.
  • Request-local model handles: chunk://N, doc://N, kb://N, url://N, msN, mtN, and res://NNNN. The legacy bare cN/dN/bN/wN form is still accepted, but only as the whole value of a declared ID field.
  • Durable text forms of source handles: weknora://kb/<id>, weknora://doc/<id>, weknora://chunk/<id>; a web page becomes its URL.
  • Tool-private request handles: iN for Wiki issues.
  • Ingest-call handles: cNNN and ref-N, allocated with HandleTable.
  • Temporary handles are decoded before tool execution, persistence, or UI delivery. They are never accepted as durable authorization or routing data.

Request lifecycle

  1. Create one Registry for the complete request or Agent execution.
  2. Register structured source identities as they enter the context. For Agent tools, use EncodeTools to project MCP routing enums and source summaries.
  3. Call EncodeMessages immediately before the model call.
  4. Decode tool calls through DecodeToolCalls before parsing or execution.
  5. Decode complete responses with DecodeResponse, or streaming text with one StreamDecoder per response channel.

Unknown handle-shaped values in declared tool fields are rejected before tool execution. The same per-tool policy is applied when historical assistant tool calls and tool results are replayed, so handles neither disappear nor drift to new numbers across Agent rounds.

The registry owns codec ordering. Resource references are encoded before source IDs so a Wiki slug such as summary/<knowledge-id> cannot be corrupted into summary/doc://1.

Source addressability is separate from citation eligibility. Use RegisterContextChunk for bound-KB directory entries. Replayed citations, legacy tool history, and tool arguments also register navigation handles only. ModelToolResultForTool and explicitly supplied retrieval results authorize citations for the current execution; rereading a historical source promotes the same handle without renumbering it. Both complete and streaming decoders drop references that have not been backed by current evidence.

Observability

Every model-call site runs LeakedIdentifiers on the encoded messages and logs a warning naming the role or tool whose text still carries a raw UUID. A registered leak means the codec missed a rewrite site; an unregistered leak means a producer emitted an identifier that never entered the registry. Text-level compaction (CompactKnownText) stays in place as defense in depth until those producers are fixed and the report stays quiet.

Langfuse generation observations contain the exact encoded payload sent to and returned by the model. Agent tool spans contain both model_arguments (the model-emitted handles) and resolved_arguments (the durable values actually executed), plus argument_resolution and any unresolved handles. Sensitive tools report only argument keys and resolution counts.

Langfuse is an observer of this boundary; it must not implement another handle mapping.

Built-in tool policies

Canonical ID-bearing arguments for knowledge, graph, data, web, and Wiki tools are decoded centrally through a tool-name plus JSON-field allowlist. The sourceKeySpaces table is the single source of truth for which keys are ID-bearing and which handle space they register into; per-tool sourceIDKeys contracts gate which of those keys each tool may use. database_query.sql has an explicit policy: handles anywhere in the SQL text decode to bare durable IDs so the model can filter the real knowledges/chunks tables. data_analysis.sql carries no handles: the selected document is always exposed as the fixed table dataset, so only knowledge_id is decoded. Wiki issue IDs use the same lifecycle via an iN handle space.

Handles in free text

Models copy handles into prose they write: automation instructions, tasks delegated to other agents, Markdown links, reasoning and final answers. Because the scheme-shaped syntax cannot collide with ordinary words, the registry rewrites handles in every string, not just declared fields:

  • Tool arguments (built-in and MCP): declared ID fields receive bare durable IDs, SQL fields receive bare IDs, and every other string receives the durable text form (weknora://kb/<id>, weknora://doc/<id>, weknora://chunk/<id>, or the page URL). An unknown handle anywhere in the arguments fails closed.
  • Model output (complete and streaming, content and reasoning): <ref/> citations expand as before; bare kb://N/doc://N become weknora:// URIs, url://N becomes the URL, and chunk://N becomes a <kb/> citation when it is current evidence, otherwise its document URI. Unknown handles are dropped.
  • Input: weknora:// URIs in any message, reasoning or replayed tool call are registered as navigation-only handles and shown to the model as handles, so a saved instruction round-trips across requests and agents.

Handles and URIs are never authorization: every tool still authorizes the durable ID it receives.

Dynamic MCP tools do not get per-field ID contracts. Their schemas and identity semantics are controlled by the MCP server, so the application does not guess that an arbitrary id, knowledge_id, or url field is durable; values that are not scheme-shaped handles pass through untouched. Handles the model copied into MCP arguments get the free-text rewrite above, and durable resource handles still use the normal res://NNNN codec. A future MCP ID mapping must be an explicit server/tool annotation and should plug into this registry rather than create a parallel mapper.

Successful MCP execution results additionally receive an external_source_candidates sidecar with up to 50 distinct observed HTTP(S) links mapped to url://N. The external payload is not rewritten, discovery schemas are not scanned, and article IDs are never used to guess URLs. A candidate link does not establish that its page was read: the model must select the link whose associated result supports the claim. When no handle is supplied, the citation protocol permits an exact supplied source URL as a Markdown link, never a borrowed knowledge-base citation.

The application-owned MCP bridge is an explicit exception for routing fields: discover_mcp_tools.server_id uses msN, and call_mcp_tool.tool_ref uses mtN. Tool definition enums, service summaries, discovery results and replayed calls share the same request registry. Only directory envelope fields are registered; input_schema and external execution results remain opaque, and call_mcp_tool.arguments only gets the free-text handle rewrite. Arguments are restored before authorization, validation, execution, persistence and UI events. Unknown routing handles fail closed.

For model-emitted call_mcp_tool calls, one extra JSON-string encoding of the arguments envelope is accepted only when it contains a complete JSON object (including {}). Normalization runs before resource-handle decoding; original model arguments are retained for tracing. Nested business strings are unchanged, and malformed JSON, arrays, null, and further encoding layers remain invalid. The canonical object still passes the existing routing, schema and approval checks.

Wiki routing

WikiRouteResolver is intentionally separate. It stores server-side slug-to-knowledge-base provenance gathered during Wiki search/read operations, and can only return knowledge bases already present in the Agent's authorized Wiki scope. It is routing state, not a model handle table. Reads scan every legal Wiki KB so cached provenance cannot hide a duplicate slug; mutations require one unique owner. New pages use link provenance, a single Wiki scope, or a single source-document owner and reject ambiguous routing instead of asking the model for a durable KB ID.

Document/tag-constrained Wiki and graph requests are filtered by the same server-owned SearchTargets provenance. Uncited/global Wiki pages and graph results outside that subset fail closed; only an explicit whole-KB target authorizes whole-KB content.

Documentation

Overview

citations.go owns the public-citation surface of the model context: the system protocol prompt, expansion of private <ref/> handles into canonical <kb/> / <web/> tags, re-compaction of canonical tags replayed from history, and the stream expander that keeps partial tags off the wire.

Package modelcontext owns every request-local handle exposed to a language model. Application code should use Registry instead of coordinating source references and durable-resource codecs independently.

Its boundaries are deliberate:

  • UUIDs, wiki slugs, URLs, and resource:// handles are durable identities.
  • kb://N, doc://N, chunk://N, url://N, iN, res://NNNN and ref-N values are temporary model handles; weknora:// URIs are their durable text form.
  • temporary handles are never persisted or accepted outside their registry.
  • every model response is decoded before tools, storage, or UI consume it.

resources.go is the durable-resource half of the model-context registry: it assigns request-local res://NNNN handles to stored resource references and restores them before application code consumes model output.

sources.go is the source-reference half of the model-context registry: request-local chunk://N, doc://N, kb://N and url://N handles for chunks, documents, knowledge bases and web pages, plus the tool-argument codec that maps them back to durable identifiers. Request lifecycles use Registry so source and resource handles cannot be encoded or decoded out of order.

stream.go owns streaming-safe handle decoding: the shared suffix-hold primitive, the per-space decoders built on it, and the composed public StreamDecoder that applies every stage in the only safe order.

Index

Constants

View Source 
const (
	ArgumentResolutionUnchanged         = "unchanged"
	ArgumentResolutionResolved          = "resolved"
	ArgumentResolutionPartiallyResolved = "partially_resolved"
	ArgumentResolutionUnresolved        = "unresolved"
)

Variables

This section is empty.

Functions

func HasToolPolicy 

func HasToolPolicy(toolName string) bool

HasToolPolicy reports whether a tool has an explicit model-handle policy. Built-in tools must declare one (even an empty policy) so that adding a tool is a deliberate decision about what the model may see, rather than a silent fall-through to fully opaque output.

func SummarizeLeaks 

func SummarizeLeaks(leaks []IdentifierLeak) string

SummarizeLeaks renders leaks as one log line per message field so the producing tool or role can be identified without dumping the context.

Types

type ChunkReference 

type ChunkReference struct {
	ChunkID         string
	KnowledgeID     string
	KnowledgeBaseID string
	DocumentTitle   string
	ChunkIndex      int
	ChunkType       string
}

type HandleStreamDecoder 

type HandleStreamDecoder struct {
	// contains filtered or unexported fields
}

HandleStreamDecoder restores HandleTable values without leaking a handle split across provider chunks.

func NewHandleStreamDecoder 

func NewHandleStreamDecoder(table *HandleTable) *HandleStreamDecoder

func (*HandleStreamDecoder) Feed 

func (d *HandleStreamDecoder) Feed(chunk string) string

func (*HandleStreamDecoder) Flush 

func (d *HandleStreamDecoder) Flush() string

type HandleTable 

type HandleTable struct {
	// contains filtered or unexported fields
}

HandleTable is a typed, invocation-local bidirectional mapping used when a prompt needs compact handles for durable values (wiki issue iN handles, ingest ref-N slug handles, c000 citation-batch handles). Handles must never be persisted; Resolve converts model output back to the durable value first. It is the exported word-boundary-aware wrapper over handleTable.

func NewHandleTable 

func NewHandleTable(prefix string, width, start int) *HandleTable

NewHandleTable creates a handle space such as c000 (prefix=c, width=3, start=0) or ref-1 (prefix=ref-, width=0, start=1).

func (*HandleTable) DecodeKnownText 

func (t *HandleTable) DecodeKnownText(value string) string

DecodeKnownText restores registered handles in complete text. Unlike the resource codec, replacement is word-boundary-aware so short handles such as i1 cannot fire inside ordinary prose tokens.

func (*HandleTable) Empty 

func (t *HandleTable) Empty() bool

func (*HandleTable) EncodeKnownText 

func (t *HandleTable) EncodeKnownText(value string) string

EncodeKnownText replaces already-registered durable values with their model handles. Longer values are processed first to avoid substring shadowing.

func (*HandleTable) Handle 

func (t *HandleTable) Handle(value string) (string, bool)

Handle returns an already registered handle without creating one.

func (*HandleTable) Len 

func (t *HandleTable) Len() int

func (*HandleTable) Register 

func (t *HandleTable) Register(value string) string

Register returns the stable handle assigned to value in this table.

func (*HandleTable) Resolve 

func (t *HandleTable) Resolve(handle string) (string, bool)

Resolve converts a known model handle back to its durable value.

type IdentifierLeak 

type IdentifierLeak struct {
	MessageIndex int
	Role         string
	// ToolName is the tool that produced a tool message or the tool an
	// assistant call targets; empty for plain content.
	ToolName string
	// Field names the message part: content, reasoning, multi_content, or
	// tool_call_arguments.
	Field string
	// Registered holds leaked values the registry knows a handle for: the
	// codec missed a rewrite site. Unregistered holds values that never
	// entered the registry: the producer emitted a raw identifier.
	Registered   []string
	Unregistered []string
}

IdentifierLeak is one message field that still carries UUID-shaped values after EncodeMessages.

type Registry 

type Registry struct {
	// contains filtered or unexported fields
}

Registry is the single request-scoped boundary between durable application identities and temporary model handles.

Durable resource references are encoded before source identifiers. This ordering is intentionally private: summary/<knowledge-id> wiki slugs must be protected as one resource-like handle before the embedded document ID can be compacted to doc://N. Callers therefore cannot accidentally reverse the codecs.

func NewRegistry 

func NewRegistry(citationsEnabled bool) *Registry

NewRegistry creates a registry for one model request/Agent execution.

func (*Registry) ChunkHandle 

func (r *Registry) ChunkHandle(id string) string

func (*Registry) CompactKnownText 

func (r *Registry) CompactKnownText(text string) string

CompactKnownText replaces only previously registered durable source IDs.

func (*Registry) DecodeOutputText 

func (r *Registry) DecodeOutputText(text string) string

DecodeOutputText applies the public citation policy to complete text. It is primarily used by non-streaming cleanup/fallback paths.

func (*Registry) DecodeResponse 

func (r *Registry) DecodeResponse(response *types.ChatResponse)

DecodeResponse restores resources, expands citations, and decodes tool arguments in a non-streaming response.

func (*Registry) DecodeToolCalls 

func (r *Registry) DecodeToolCalls(toolCalls []types.LLMToolCall)

DecodeToolCalls restores all temporary handles in tool-call arguments.

func (*Registry) EncodeKnownIssues added in v0.9.0 

func (r *Registry) EncodeKnownIssues(text string) string

EncodeKnownIssues replaces registered Wiki issue IDs with their handles. CompactKnownText leaves them alone because tools opt into issue encoding per policy; runtime context the engine renders opts in explicitly.

func (*Registry) EncodeMessages 

func (r *Registry) EncodeMessages(messages []chat.Message) []chat.Message

EncodeMessages returns a model-facing copy with every temporary handle encoded in the only safe order.

func (*Registry) EncodeTools 

func (r *Registry) EncodeTools(tools []chat.Tool) []chat.Tool

EncodeTools returns a model-facing copy. The registry/executor retains the original UUID enum and validates only decoded durable identities.

func (*Registry) ImageSourcesNote 

func (r *Registry) ImageSourcesNote(chunkIDs []string, offset int) string

ImageSourcesNote binds images to handles already present in the final model context. offset counts user images preceding the retrieved ones in this message. Unknown sources stay explicitly unavailable, never fabricated.

func (*Registry) LeakedIdentifiers 

func (r *Registry) LeakedIdentifiers(messages []chat.Message) []IdentifierLeak

LeakedIdentifiers scans an encoded message list for UUID-shaped values. User-authored messages are skipped: a user may legitimately paste an ID. Everything else the application composed, so any UUID there is a leak.

func (*Registry) ModelToolResult 

func (r *Registry) ModelToolResult(result *types.ToolResult) string

ModelToolResult renders a tool result using registered model handles.

func (*Registry) ModelToolResultForTool 

func (r *Registry) ModelToolResultForTool(toolName string, result *types.ToolResult) string

ModelToolResultForTool renders a result and applies any explicit private-ID policy owned by that built-in tool family.

func (*Registry) OrphanResourceHandles 

func (r *Registry) OrphanResourceHandles(decoded string) []string

OrphanResourceHandles reports model-generated resource handles with no backing durable reference.

func (*Registry) ProtocolPrompt 

func (r *Registry) ProtocolPrompt() string

ProtocolPrompt returns the system-owned model handle and citation protocol.

func (*Registry) RegisterChunk 

func (r *Registry) RegisterChunk(ref ChunkReference) string

func (*Registry) RegisterContextChunk 

func (r *Registry) RegisterContextChunk(ref ChunkReference) string

RegisterContextChunk makes a directory entry addressable by tools without allowing it to substantiate an answer before retrieval.

func (*Registry) RegisterDocument 

func (r *Registry) RegisterDocument(id string) string

func (*Registry) RegisterIssue added in v0.9.0 

func (r *Registry) RegisterIssue(id string) string

RegisterIssue allocates the iN handle the Wiki issue tools accept for id.

func (*Registry) RegisterKnowledgeBase 

func (r *Registry) RegisterKnowledgeBase(id string) string

func (*Registry) RegisterSearchResults 

func (r *Registry) RegisterSearchResults(results []*types.SearchResult)

func (*Registry) RegisterWeb 

func (r *Registry) RegisterWeb(rawURL, title string) string

func (*Registry) StreamDecoder 

func (r *Registry) StreamDecoder() *StreamDecoder

StreamDecoder creates one ordered decoder for a response text channel.

func (*Registry) ToolImageSourcesNote 

func (r *Registry) ToolImageSourcesNote(result *types.ToolResult) string

ToolImageSourcesNote is rendered beside the actual images after tool result rendering has registered its source handles. Stored results have no images and must not claim to attach them on replay.

type StreamDecoder 

type StreamDecoder struct {
	// contains filtered or unexported fields
}

StreamDecoder composes resource restoration, source-citation expansion, source-handle expansion, issue-handle decoding and orphan filtering so callers cannot split handle processing or apply stages in the wrong order.

func (*StreamDecoder) Feed 

func (d *StreamDecoder) Feed(chunk string) string

func (*StreamDecoder) Flush 

func (d *StreamDecoder) Flush() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.