exp

type Agents ¶

type Agents struct {
	// Agents lists the sub-agents available for delegation: by name
	// (aix.AgentRef{Name: ...}) or as a captured instance (agentValue.Ref()).
	// At least one is required.
	Agents []aix.AgentRef `json:"agents,omitempty" jsonschema_description:"Sub-agents available for delegation. At least one is required."`
	// ToolPrefix is the prefix for generated delegation tool names. A nil value
	// defaults to "delegate_to" (tools become delegate_to_<agent>); a pointer to
	// the empty string uses bare agent names. An explicitly set, non-empty
	// prefix also namespaces the shared tools: the [Agents.Async]
	// background-task tools (e.g. research_check_background_tasks) and the
	// continue tool (registered whenever any configured sub-agent may be
	// server-managed). Two instances in one generate call that both register
	// shared tools therefore need distinct, explicitly set prefixes: left at
	// the default they emit the same bare names, and the generate call is
	// rejected for duplicate tools. New rejects colliding names within one
	// instance; it cannot see a sibling.
	ToolPrefix *string `` /* 296-byte string literal not displayed */
	// MaxDelegations caps the number of sub-agent delegations per generate call,
	// preventing runaway delegation loops. 0 means unlimited.
	MaxDelegations int `` /* 191-byte string literal not displayed */
	// HistoryLength is the number of recent user/model messages forwarded to a
	// sub-agent as context. 0 means only the task description is sent. History is
	// forwarded only to client-managed sub-agents (those without a session
	// store); server-managed sub-agents receive only the task.
	HistoryLength int `` /* 231-byte string literal not displayed */
	// ArtifactStrategy controls how sub-agent artifacts are surfaced. Defaults to
	// ArtifactStrategyInline.
	ArtifactStrategy ArtifactStrategy `` /* 264-byte string literal not displayed */
	// Async enables background delegation. Delegation tools gain a "background"
	// input flag that starts the sub-agent through its detach support and
	// returns a task ID immediately, and three shared tools are added, one per
	// control the orchestrator has over a launched task:
	// check_background_tasks (non-blocking status and results),
	// wait_for_background_tasks (blocks until the listed tasks settle), and
	// abort_background_tasks (stops tasks whose results are no longer needed).
	// Background delegation requires server-managed sub-agents whose stores
	// implement [aix.SnapshotSubscriber].
	//
	// The tools that act on a task accept its ID only while the result that
	// launched or continued it is still in the request's history, or this
	// generate call launched it. History trimmed or compacted past that result
	// takes the task out of the orchestrator's reach.
	Async bool `` /* 320-byte string literal not displayed */
	// MaxWaitSeconds bounds how long one wait_for_background_tasks call
	// blocks, whatever timeoutSeconds the model asks for, including 0 ("until
	// every task settles"). At the bound the tool returns the current
	// statuses with TimedOut set, as it does at the model's own timeout, so a
	// sub-agent that keeps running cannot hold the orchestrator's turn open
	// past it. 0 means no bound; a negative value is rejected.
	MaxWaitSeconds int `` /* 325-byte string literal not displayed */

}

Agents is a middleware that enables sub-agent delegation.

For every configured agent it injects a dedicated delegation tool (e.g. delegate_to_researcher) whose description is the agent's configured description or, in the system prompt, the description auto-discovered from the registry. A <sub-agents> block listing the available agents is appended to the system prompt.

When the model calls a delegation tool the middleware resolves the target agent from the registry (via the github.com/firebase/genkit/go/genkit.Genkit instance carried on the context), optionally forwards recent conversation history, runs the sub-agent with the task, and returns its response as the tool result.

Artifact handling follows Agents.ArtifactStrategy: ArtifactStrategyInline (default) returns artifact content in the tool result and merges artifacts into the parent session; ArtifactStrategySession merges into the session only and returns names. Merged artifacts are namespaced by an invocation ID (<agent>_<snapshotId prefix>/<name> for a run with a snapshot behind it, <agent>_<n>/<name> otherwise) and tagged with the source agent.

If a sub-agent interrupts (e.g. for human input) it is reported back to the orchestrator as a normal tool response, not propagated as an interrupt: there is no stateful sub-agent runtime to resume into, so interactive sub-agent interaction is a future feature.

With Agents.Async set, delegation tools additionally accept a "background" flag. A background delegation starts the sub-agent through its detach support (aix.AgentInput.Detach): the sub-agent's runtime persists a pending snapshot, hands back a task ID immediately, and keeps working in the background, so the orchestrator can continue calling tools and collect the result later through the added check_background_tasks and wait_for_background_tasks tools, or drop it with abort_background_tasks. The pending snapshot (heartbeated while the worker lives, finalized in place with the cumulative state when the work settles) is the durable record, and task IDs are self-contained ("<agent>:<snapshotId>"), so a re-instantiated orchestrator can pick results up using nothing but the IDs recorded in its conversation history. Background delegation requires server-managed sub-agents whose stores implement aix.SnapshotSubscriber (e.g. the localstore stores); launches on other agents are rejected by the sub-agent runtime and reported as tool text.

The background-task tools and the continue tool accept only the task IDs this conversation launched: ones this generate call minted, or ones a delegation or continue tool's result in the conversation history carries. Text that reaches the orchestrator model (a sub-agent's answer, a retrieved document) therefore cannot steer them at another conversation's task. A history compacted past its launch results loses those tasks' handles. The sub-agent's own companion actions (getSnapshot, waitForSnapshot, abort) stay unscoped, so in multi-tenant deployments treat snapshot IDs as capability-like secrets where those are exposed.

The middleware resolves agents through genkit.FromContext, which is seeded by genkit.Generate and by agents defined via the genkit/exp constructors (genkitx.DefineAgent and friends). It is therefore typically attached to an orchestrator agent (or a genkit.Generate call).

Usage:

orchestrator := genkitx.DefineAgent(g, "orchestrator",
    aix.InlinePrompt{
        ai.WithModelName("googleai/gemini-flash-latest"),
        ai.WithSystem("You are a helpful project assistant."),
        ai.WithUse(
            &middlewarex.Agents{
                Agents: []aix.AgentRef{
                    {Name: "researcher"}, // by name
                    coderAgent.Ref(),     // by instance (carries its description)
                },
                MaxDelegations:   5,
                HistoryLength:    4,
                ArtifactStrategy: middlewarex.ArtifactStrategySession,
            },
            &middlewarex.Artifacts{},
        ),
    },
)

func (a Agents) Name() string

func (a Agents) New(ctx context.Context) (*ai.Hooks, error)

type ArtifactStrategy ¶

type ArtifactStrategy string

ArtifactStrategy controls how a sub-agent's artifacts are surfaced back to the orchestrator by the Agents middleware.

const (
	// ArtifactStrategyInline includes artifact content in the delegation tool
	// result so the orchestrator model can see it, and also merges artifacts
	// into the parent session. This is the default.
	ArtifactStrategyInline ArtifactStrategy = "inline"
	// ArtifactStrategySession merges artifacts into the parent session only; the
	// tool result names the artifacts but omits their content. Pair it with the
	// [Artifacts] middleware so the model can read/write session artifacts.
	ArtifactStrategySession ArtifactStrategy = "session"
)

type Artifacts ¶

type Artifacts struct {
	// Readonly, when true, provides only the read_artifact tool; the model
	// cannot create or update artifacts. Defaults to false.
	Readonly bool `` /* 154-byte string literal not displayed */
}

Artifacts is a middleware that gives the model tools to interact with session artifacts and injects a listing of available artifacts into the system prompt.

It provides:

• read_artifact: reads an artifact by name from the session and returns its text content.
• write_artifact (unless Readonly): creates or updates a session artifact. Artifacts are deduplicated by name, so writing to an existing name replaces it.

On every generate turn an <artifacts> block listing the names and sizes of the session's artifacts is injected into (or refreshed within) the system message, so the model knows what is available without spending context on the full content.

This is useful standalone (e.g. a workspace-builder agent that creates files as artifacts) or combined with Agents using ArtifactStrategySession, where sub-agent artifacts are merged into the parent session and the model reaches them through these tools.

Artifacts live on the active agent session, so this middleware only has an effect when generation runs inside an agent invocation (see github.com/firebase/genkit/go/genkit/exp.DefineAgent). With no active session the tools report that gracefully and the listing is empty.

Usage:

builder := genkitx.DefineAgent(g, "builder",
    aix.InlinePrompt{
        ai.WithModelName("googleai/gemini-flash-latest"),
        ai.WithSystem("You are a code generator. Use write_artifact to create files."),
        ai.WithUse(&middlewarex.Artifacts{}),
    },
)

func (a Artifacts) Name() string

func (a Artifacts) New(ctx context.Context) (*ai.Hooks, error)

type Budget ¶ added in v1.14.0

type Budget struct {
	// Limit caps each of its non-zero fields. At least one must be set.
	Limit ai.GenerationUsage `json:"limit" jsonschema_description:"Usage the run or session may spend. Each non-zero field is a cap, custom key by key."`
	// Scope is what the limit counts against. Defaults to [BudgetScopeRun].
	Scope BudgetScope `` /* 170-byte string literal not displayed */
	// Interrupt pauses a run-scoped budget for approval instead of stopping
	// the run.
	Interrupt bool `` /* 142-byte string literal not displayed */
}

Budget caps the usage a generate run, or an agent session, may spend.

Limit caps each of its non-zero fields, ai.GenerationUsage.Custom key by key: InputTokens and OutputTokens, but also ThoughtsTokens, OutputImages, or a provider-reported cost such as Custom["cost"]. Usage counts as models report it, under the convention ai.GenerationUsage documents, so OutputTokens leaves out ThoughtsTokens: cap both to cap what a model generates. A model that reports no usage spends nothing, and a background model (such as a video model) is not counted.

The limit is checked before each model call of the tool loop. Once a capped field reaches its limit, the run stops with ai.ErrBudgetExceeded, which reports it as aborted rather than failed and keeps its partial response. A turn already under way finishes, so a run can overshoot by one turn, and with a Retry middleware in the chain, one turn can be several billed calls. Because the check sits outside every WrapModel hook, a retry or fallback middleware never sees the stop, whatever the order in ai.WithUse.

A budget stops only the run it wraps. A background delegation the run started (see Agents.Async) counts while the run lasts, and keeps running, and spending, after the run stops.

Budgets in one chain count the same calls and claim the same holds, so stacking them enforces all their limits together.

With Interrupt set, a run-scoped budget pauses instead: the tool calls that would lead to the next turn are held, for the caller to decide whether to continue. The hold happens only when the run is over its limit by the time its tool calls start: a run that goes over while its tools run (a nested generate, a subagent) has nothing left to hold, and the check before the next model call stops it. BudgetInterrupted claims each held call, BudgetExceeded is what it carries, and a restart with a BudgetDecision decides it. A response to a hold, which stands in for the tool's output, refuses it:

for _, part := range resp.Interrupts() {
	if call, ok := middlewarex.BudgetInterrupted(part); ok {
		parts = append(parts, call.Restart(middlewarex.BudgetDecision{Approved: true}))
	}
}

A session-scoped budget cannot pause, since a later turn would start over its limit with nothing to pause on; New rejects the combination.

A run-scoped budget counts one run across its resumes. The run's final message records what it spent, and a resume of that conversation continues from there, so answering another middleware's interrupt, such as a ToolApproval hold, does not renew the limit; only approving the budget's own hold does. A conversation the caller sends back without that record, which a client-managed history can do, starts the count over.

func (Budget) JSONSchemaExtend(s *jsonschema.Schema)

func (b Budget) Name() string

func (b Budget) New(ctx context.Context) (*ai.Hooks, error)

type BudgetDecision ¶ added in v1.14.0

type BudgetDecision struct {
	// Approved decides the run: true continues it with the full limit
	// again, and false, the zero value, stops it with [ai.ErrBudgetExceeded]
	// before it runs the held calls.
	Approved bool `json:"budgetApproved"`
	// Reason, when Approved is false, is added to the error the run stops
	// with.
	Reason string `json:"reason,omitempty"`
}

BudgetDecision is the answer to a Budget hold, sent with ai.InterruptedCall.Restart on the call BudgetInterrupted claims. The holds of one run share one decision: the run continues only if no restart of them refuses.

type BudgetExceeded ¶ added in v1.14.0

type BudgetExceeded struct {
	// Message says which limit the run reached.
	Message string `json:"message"`
	// Spent and Limit are what the run spent and its limit, for each capped
	// field of [Budget.Limit] by its JSON name ("totalTokens",
	// "custom.cost").
	Spent map[string]float64 `json:"spent"`
	Limit map[string]float64 `json:"limit"`
}

BudgetExceeded is the data a Budget hold carries; read it with ai.InterruptAs on the part BudgetInterrupted claims.

type BudgetScope ¶ added in v1.14.0

type BudgetScope string

BudgetScope is what a Budget counts against.

const (
	// BudgetScopeRun counts every model call made under one Generate call:
	// the calls its TotalUsage counts (the tool loop, hooks, nested
	// generates), plus those of any subagent it runs. For a prompt-backed
	// agent, that is one turn.
	BudgetScopeRun BudgetScope = "run"
	// BudgetScopeSession counts the agent session's own usage across all its
	// turns and invocations, as the session state's Usage records it when the
	// run starts, plus every call the run makes, subagents included. It needs a
	// server-managed agent (one with a session store): client-managed state
	// comes back from the caller, who could drop its usage. Outside such an
	// agent, New fails.
	BudgetScopeSession BudgetScope = "session"
)

type CompressionDedupe ¶ added in v1.14.0

type CompressionDedupe struct {
	// MatchBy selects how duplicates are identified. Defaults to
	// [CompressionDedupeNameAndInput].
	MatchBy CompressionDedupeMatch `` /* 303-byte string literal not displayed */
	// KeepRecent is how many of the newest responses in each duplicate group
	// stay intact. Defaults to 1.
	KeepRecent int `` /* 136-byte string literal not displayed */
	// Notice replaces the output of each older duplicate. A default notice
	// is used when empty.
	Notice string `` /* 140-byte string literal not displayed */
}

CompressionDedupe configures how ContextCompression replaces older duplicate tool responses with a short notice.

type CompressionDedupeMatch ¶ added in v1.14.0

type CompressionDedupeMatch string

CompressionDedupeMatch selects how CompressionDedupe identifies duplicate tool responses.

const (
	// CompressionDedupeNameAndInput matches responses from calls to the same
	// tool with the same input. The input is read from the tool request the
	// response answers.
	CompressionDedupeNameAndInput CompressionDedupeMatch = "name-and-input"
	// CompressionDedupeNameOnly matches every response from the same tool,
	// whatever its input, so only the newest survives. Use it only for tools
	// that return the latest overall state.
	CompressionDedupeNameOnly CompressionDedupeMatch = "name-only"
)

type CompressionSummarizer ¶ added in v1.14.0

type CompressionSummarizer struct {
	// Model writes the summaries, typically a smaller and faster model than
	// the one being compressed. Its config is used as is, so set the
	// summary's output limit there. Required.
	Model ai.ModelRef `` /* 164-byte string literal not displayed */
	// PreserveRecent is how many of the newest non-system messages stay out
	// of the summary. Defaults to ContextCompression.PreserveRecent when that
	// is set, and to 6 otherwise.
	PreserveRecent int `` /* 194-byte string literal not displayed */
	// Prompt replaces the default summarization prompt. Each "{conversation}"
	// in it is replaced with a text rendering of the messages to summarize;
	// without one, the rendering is appended.
	Prompt string `` /* 207-byte string literal not displayed */
}

CompressionSummarizer configures how ContextCompression folds older messages into a summary.

type CompressionToolTruncation ¶ added in v1.14.0

type CompressionToolTruncation struct {
	// MaxChars is the most characters an older tool response keeps, output and
	// content together. Text content is folded into the output when a
	// response is truncated. Required.
	MaxChars int `` /* 184-byte string literal not displayed */
	// PreserveRecent is how many of the newest tool messages stay intact.
	// Defaults to 2. Set it negative to truncate every tool message.
	PreserveRecent int `` /* 159-byte string literal not displayed */
}

CompressionToolTruncation configures how ContextCompression truncates older tool responses.

type ContextCompression ¶ added in v1.14.0

type ContextCompression struct {
	// MaxInputTokens triggers compression when the context exceeds this many
	// tokens. Zero disables the token trigger.
	MaxInputTokens int `` /* 247-byte string literal not displayed */
	// MaxMessages triggers compression when the conversation holds more than
	// this many messages, and caps the messages kept. Zero disables it.
	// Without Summarize, a tight cap drops the turns that record a tool
	// loop's progress, and the model may repeat calls it already made. A cap
	// that the system and protected messages fill still keeps the newest
	// turn.
	MaxMessages int `` /* 236-byte string literal not displayed */
	// PreserveRecent is how many of the newest non-system messages a
	// compaction keeps when it drops messages, and the default window for
	// Summarize. Defaults to 4. Compaction shrinks it when the context is far
	// over budget: halved beyond 1.5 times MaxInputTokens, and down to 2
	// beyond twice.
	PreserveRecent int `` /* 238-byte string literal not displayed */
	// MaxToolResponseChars caps every tool response, output and content
	// together, on every iteration. Defaults to 400000. Set it negative to
	// disable the cap.
	MaxToolResponseChars int `` /* 221-byte string literal not displayed */
	// DedupeToolResponses replaces older duplicate tool responses with a
	// notice when compression runs. Nil disables it.
	DedupeToolResponses *CompressionDedupe `` /* 148-byte string literal not displayed */
	// TruncateToolResponses truncates older tool responses when compression
	// runs. Nil disables it.
	TruncateToolResponses *CompressionToolTruncation `` /* 149-byte string literal not displayed */
	// Summarize folds older messages into a summary written by a model when
	// compression runs. Nil disables it.
	Summarize *CompressionSummarizer `` /* 128-byte string literal not displayed */
	// SkipSummarizationThreshold skips Summarize when deduplication and
	// truncation saved at least this fraction of the context, between 0 and
	// 1, and brought it under MaxInputTokens. Zero always summarizes.
	SkipSummarizationThreshold float64 `` /* 235-byte string literal not displayed */
	// TruncationNotice replaces the default notice inserted when messages
	// are dropped.
	TruncationNotice string `json:"truncationNotice,omitempty" jsonschema_description:"Custom notice inserted when messages are dropped."`
	// NoTruncationNotice drops messages without telling the model.
	NoTruncationNotice bool `` /* 161-byte string literal not displayed */
	// ReplaceHistory replaces the history the caller gets back with the
	// compressed messages, instead of keeping the original messages and
	// recording the compression in their metadata.
	ReplaceHistory bool `` /* 223-byte string literal not displayed */
}

ContextCompression is a middleware that keeps a conversation inside a context budget through long chats and agentic tool loops. It ports the contextCompression middleware of the JS runtime and records its work in the same message metadata, so either runtime resolves a history the other one compressed.

Before each tool-loop iteration, compression runs when a trigger fires: the context exceeds MaxInputTokens, measured by the input tokens the model reported for the previous call (estimated from the messages before there is one), or the conversation holds more than MaxMessages messages. The strategies run cheapest first:

1. DedupeToolResponses replaces older duplicate tool responses with a short notice.
2. TruncateToolResponses truncates older tool responses, leaving the newest tool messages intact.
3. Summarize has a model fold older messages into a summary, unless the first two strategies already saved SkipSummarizationThreshold of the context and brought it under budget.
4. Older messages are dropped when nothing was summarized and the conversation still exceeds its message budget. A notice tells the model that context was removed.

MaxToolResponseChars caps every tool response on every iteration, whether or not a trigger fires. Leading system messages are always kept, and the kept messages never start with a tool response separated from its request.

Two kinds of message are never summarized or dropped: the scaffolding an agent's prompt renders on every turn, and the message that carries injected output-format instructions. When a compaction covers one, the message stays in the model's view, after the system messages and before the summary. Both count toward MaxMessages and the recent windows. The JS runtime does not protect them.

History ¶

The history the caller gets back (ai.ModelResponse.History) keeps every original message, and the model receives a compressed view of it. The compression is recorded in metadata under the "contextCompression" key:

• Every model message gets {"inputTokens": N}, the input token count the model reported for the call that produced it. Later calls read it to decide whether to compress.
• The last message that a compaction covers, its boundary, gets {"summary": "...", "stats": {...}}, plus "anchorUser" when the view keeps the user message that started the current tool loop and "truncationNotice" when a custom notice was inserted. The view is the leading system messages, the summary, then every message after the newest boundary. A summary is empty when messages were dropped without one.
• A truncated, capped, or deduplicated tool response part keeps its original output and gets {"truncated", "capped", or "deduplicated": true, "maxChars": N, "rawOutput": true}. The view applies the recorded limit.

The metadata is the whole state, so compression continues across Generate calls, sessions, and process restarts when the caller carries the history forward. ResolveCompressedHistory derives the model's view from a history. Set ReplaceHistory to have the compressed messages replace the history instead.

The final response carries the stats of the newest compression under Custom["contextCompression"]. The middleware never modifies the messages it receives, so one history is safe to share between concurrent calls.

Ordering ¶

List ContextCompression after middleware that adds context (Skills, Filesystem, Agents, Artifacts), so the compression accounts for what they add, and before Retry and Fallback, so retried and fallback calls receive the compressed view:

resp, err := genkit.Generate(ctx, g,
    ai.WithPrompt("Research this topic thoroughly."),
    ai.WithTools(searchTool),
    ai.WithUse(
        &middlewarex.ContextCompression{
            MaxInputTokens:        80_000,
            DedupeToolResponses:   &middlewarex.CompressionDedupe{},
            TruncateToolResponses: &middlewarex.CompressionToolTruncation{MaxChars: 2000},
            Summarize: &middlewarex.CompressionSummarizer{
                Model: googlegenai.ModelRef("googleai/gemini-flash-lite-latest", nil),
            },
        },
        &middleware.Retry{},
    ),
)

func (c ContextCompression) Name() string

func (c ContextCompression) New(ctx context.Context) (*ai.Hooks, error)

type Middleware ¶

type Middleware struct{}

Middleware provides the experimental middleware (Agents, Artifacts, Budget, ContextCompression) as a Genkit plugin. Register it with genkit.WithPlugins during genkit.Init to make them resolvable by name (e.g. for the Dev UI). Using them directly via ai.WithUse does not require the plugin.

func (p *Middleware) Init(ctx context.Context) []api.Action

func (p *Middleware) Middlewares(ctx context.Context) ([]*ai.MiddlewareDesc, error)

func (p *Middleware) Name() string