Documentation
¶
Overview ¶
Package middleware provides reusable middleware for Genkit model generation, including retry with exponential backoff and model fallback.
Index ¶
Constants ¶
const ( // SkillToolName loads a skill's instructions by name. The name matches the // JS and Python runtimes so prompts and evaluations port between them. SkillToolName = "use_skill" // SkillResourceToolName reads a file bundled inside a skill directory. It // is registered only when [Skills.AllowResourceAccess] is set. SkillResourceToolName = "read_skill_file" )
Tool names registered by Skills. When Skills.ToolNamePrefix is set, the registered name is the prefix followed by the constant.
Use them to name the tools in ToolApproval.AllowedTools: that field is matched by exact string, so a skills tool missing from the list is held for approval rather than run.
const SkillActivationMetadataKey = "skillActivation"
SkillActivationMetadataKey is the metadata key stamped on every message part that carries a skill's instructions, whether the model loaded them with SkillToolName or Skills.Preload injected them. Its value is the skill name as a string.
Skills reads the key back out of the conversation to recognize a skill that is already loaded. Context-management middleware can use it to find skill instructions in a transcript and exempt them from summarization.
Variables ¶
This section is empty.
Functions ¶
func ToolApprovalInterrupted ¶ added in v1.14.0
func ToolApprovalInterrupted(part *ai.Part) (*ai.InterruptedCall[any, any, ToolCallDecision], bool)
ToolApprovalInterrupted claims part for ToolApproval: it reports whether part is a call the middleware held and, when it is, returns the call, to be answered with a ToolCallDecision. Every ToolApproval in a chain claims the same holds. See ai.MiddlewareInterrupted.
Types ¶
type Fallback ¶
type Fallback struct {
// Models is the ordered list of fallback models to try.
// These are tried in order after the primary model fails. Each ref's
// Config is used verbatim for that model -- the original request's
// Config is not inherited. Use [ai.NewModelRef] to attach config.
Models []ai.ModelRef `` /* 221-byte string literal not displayed */
// Statuses is the set of status codes that trigger a fallback for
// classified errors; unclassified errors propagate immediately and never
// trigger one. Defaults to [defaultFallbackStatuses].
Statuses []status.Name `` /* 605-byte string literal not displayed */
}
Fallback is a middleware that tries alternative models when the primary model fails with a retryable error status.
It only hooks the Model stage -- when a model API call fails with a matching status, the request is forwarded to the next model in the list.
Models are specified as ai.ModelRef values (created via ai.NewModelRef) and resolved via the genkit.Genkit instance at call time.
A model that fails with a status that does not clear up between calls (NOT_FOUND, PERMISSION_DENIED, UNAUTHENTICATED, INVALID_ARGUMENT or UNIMPLEMENTED) is skipped for the rest of the generate call, so each turn of a tool loop starts at the next model instead of repeating the failure. The next generate call tries every model again.
Provider SDKs can retry on their own before an error reaches this middleware (anthropic-sdk-go retries twice by default). Those retries multiply with Retry and delay the fallback; disable one layer, for example with the plugin's SDK options, when combining them.
Usage:
resp, err := genkit.Generate(ctx, g,
ai.WithModel(primary),
ai.WithPrompt("hello"),
ai.WithUse(&middleware.Fallback{Models: []ai.ModelRef{
googlegenai.ModelRef("googleai/gemini-flash-latest", ...),
googlegenai.ModelRef("vertexai/gemini-flash-latest", ...),
}}),
)
type Filesystem ¶
type Filesystem struct {
// RootDir is the directory that all operations are confined to.
RootDir string `json:"rootDirectory,omitempty" jsonschema_description:"Directory that all file operations are confined to. Required."`
// AllowWriteAccess adds write_file and edit_file.
AllowWriteAccess bool `json:"allowWriteAccess,omitempty" jsonschema_description:"Adds the write_file and edit_file tools. Defaults to false."`
// ToolNamePrefix is prepended to each tool name. Use distinct prefixes
// when attaching multiple Filesystem middlewares to one call so their
// tool names don't collide.
ToolNamePrefix string `` /* 203-byte string literal not displayed */
}
Filesystem is a middleware that grants the LLM scoped file access under a single root directory. It registers list_files and read_file, plus write_file and edit_file when AllowWriteAccess is true.
Path safety is enforced by os.Root (Go 1.25+), which rejects any path that resolves outside the root, including via "..", absolute paths, or symbolic links.
A failed file operation does not fail the generation: its error answers the call, as with SoftToolErrors, so the model can correct itself.
Usage:
resp, err := genkit.Generate(ctx, g,
ai.WithModel(m),
ai.WithPrompt("summarise docs/ and save the summary to out.md"),
ai.WithUse(&middleware.Filesystem{
RootDir: "./workspace",
AllowWriteAccess: true,
}),
)
type Middleware ¶
type Middleware struct{}
Middleware provides the built-in middleware (Retry, Fallback, ToolApproval) as a Genkit plugin. Register it with genkit.WithPlugins during genkit.Init.
func (*Middleware) Middlewares ¶
func (p *Middleware) Middlewares(ctx context.Context) ([]*ai.MiddlewareDesc, error)
func (*Middleware) Name ¶
func (p *Middleware) Name() string
type Retry ¶
type Retry struct {
// MaxRetries is the maximum number of retry attempts. Defaults to 3.
MaxRetries int `json:"maxRetries,omitempty" jsonschema_description:"Maximum number of retry attempts. Defaults to 3."`
// Statuses is the set of status codes that trigger a retry for classified
// errors; unclassified errors are always retried regardless of this list.
// Defaults to [defaultRetryStatuses].
Statuses []status.Name `` /* 575-byte string literal not displayed */
// InitialDelayMs is the delay before the first retry, in milliseconds. Defaults to 1000.
InitialDelayMs int `json:"initialDelayMs,omitempty" jsonschema_description:"Delay before the first retry, in milliseconds. Defaults to 1000."`
// MaxDelayMs is the upper bound on retry delay, in milliseconds. Defaults to 60000.
MaxDelayMs int `json:"maxDelayMs,omitempty" jsonschema_description:"Upper bound on the retry delay, in milliseconds. Defaults to 60000."`
// BackoffFactor is the multiplier applied to the delay after each retry. Defaults to 2.
BackoffFactor float64 `json:"backoffFactor,omitempty" jsonschema_description:"Multiplier applied to the delay after each retry. Defaults to 2."`
// NoJitter disables random jitter on the delay. Jitter helps prevent
// thundering-herd problems when many clients retry simultaneously.
NoJitter bool `` /* 194-byte string literal not displayed */
}
Retry is a middleware that retries failed model calls with exponential backoff.
It only hooks the Model stage — individual model API calls are retried, not the entire generate loop.
A classified error is retried when its status is in Statuses, which defaults to UNAVAILABLE, DEADLINE_EXCEEDED, RESOURCE_EXHAUSTED, ABORTED, and INTERNAL. An unclassified error (no status.Error or sentinel in its chain) is always retried, regardless of Statuses. A cancelled context reports CANCELLED and is not retried.
Usage:
resp, err := ai.Generate(ctx, r,
ai.WithModel(m),
ai.WithPrompt("hello"),
ai.WithUse(&middleware.Retry{MaxRetries: 3}),
)
type Skills ¶
type Skills struct {
// SkillPaths lists directories that are scanned for skills. Each direct
// subdirectory containing a SKILL.md file is exposed as a skill; scanning
// is one level deep. A symbolic link to a skill directory is followed, as
// skill installers commonly create them, but a SKILL.md that is itself a
// symbolic link is not. Relative paths resolve against the process working
// directory, and a skill's directory is shown to the model the way its
// entry here names it, so a relative entry keeps host paths out of the
// conversation.
//
// When two paths hold a skill of the same name the later path wins, and
// the shadowing is logged.
//
// Defaults to []string{".agents/skills", "skills"}.
SkillPaths []string `` /* 310-byte string literal not displayed */
// Preload names skills whose instructions are injected before the first
// model turn instead of waiting for the model to call use_skill. Use it
// when the application, rather than the model, decides that a skill
// applies.
//
// Preloaded skills are left out of the catalog's list of loadable skills.
// A preload injects the SKILL.md bytes the scan read, so a skill that was
// discovered is always delivered, with the same content on every turn. A
// name matching no discovered skill, including one whose SKILL.md could
// not be read, is logged and ignored: skills are rescanned on every
// request, so a temporarily unreadable directory must not fail the
// request.
Preload []string `` /* 220-byte string literal not displayed */
// AllowResourceAccess registers read_skill_file, which reads files bundled
// inside a skill directory (references/, scripts/, assets/, and anything
// else the skill ships), and appends a listing of those files to each
// activation. Paths resolve against that one skill's directory and are
// confined to it: by [os.Root] on disk, and by path validation in SkillFS.
//
// Defaults to false: enabling it grants the model file access it does not
// otherwise have, and adds a tool name that [ToolApproval.AllowedTools]
// must list.
AllowResourceAccess bool `` /* 216-byte string literal not displayed */
// ToolNamePrefix is prepended to each registered tool name. Use distinct
// prefixes when attaching more than one Skills middleware to a call, or to
// avoid colliding with a caller-supplied tool of the same name, since a
// collision fails the whole request. A prefix breaks tool-name parity with
// the other Genkit runtimes; leave it empty unless you need it.
ToolNamePrefix string `` /* 198-byte string literal not displayed */
// SkillFS, when set, is the file system that SkillPaths are read from, in
// place of the operating system's. Use it to ship skills inside the binary
// with an [embed.FS]:
//
// //go:embed skills
// var skillsFS embed.FS
//
// ai.WithUse(&middleware.Skills{SkillFS: skillsFS})
//
// SkillPaths are then slash-separated paths inside SkillFS, and the
// defaults apply unchanged. A go:embed pattern that names a directory
// leaves out the files in it whose names begin with "." or "_", so name
// ".agents/skills" in the pattern itself, and add the "all:" prefix if a
// skill bundles such a file.
//
// A skill in SkillFS has no path that another tool can open, so its
// activation does not tell the model where relative paths resolve. Its
// bundled files are reachable only through read_skill_file.
//
// Containment comes from path validation, which refuses ".." and absolute
// paths, and from SkillFS itself. An [os.DirFS] follows symbolic links out
// of its tree, so for a directory on disk, set SkillPaths and leave SkillFS
// unset.
//
// SkillFS is not part of the JSON configuration. A call that the Dev UI
// dispatches by name reads skills from disk.
SkillFS fs.FS `json:"-"`
}
Skills is a middleware that makes a local library of "skills" available to the model, following the Agent Skills specification (https://agentskills.io). A skill is a directory containing a SKILL.md file whose contents become specialized instructions the model can load on demand.
Skills implements the specification's three tiers of progressive disclosure:
- Catalog. A system prompt lists each available skill's name and description, and names the tool that loads one.
- Instructions. A use_skill tool returns a skill's full SKILL.md, wrapped in <skill_content> together with the skill's directory. A skill already loaded in the conversation is not sent twice.
- Resources. When AllowResourceAccess is set, each activation lists the files the skill bundles, and a read_skill_file tool reads them on demand, confined to that one skill directory.
SKILL.md should start with a YAML frontmatter block carrying name and description. Following the specification's guidance for clients, a skill that violates the format is loaded anyway and reported through the logger, so a library authored for another agent still works here.
Security: skill paths resolve against the process working directory by default, or inside SkillFS when it is set, and a discovered SKILL.md becomes instruction text the model follows. Treat a skills tree the way you treat source code, and prefer an absolute path over the default in a server process. Genkit applies no trust gate of its own.
Usage:
resp, err := genkit.Generate(ctx, g,
ai.WithModel(m),
ai.WithPrompt("use the python skill to compute ..."),
ai.WithUse(&middleware.Skills{SkillPaths: []string{"skills"}}),
)
func (Skills) New ¶
New scans the configured skill paths and returns the ai.Hooks that inject the skills catalog and expose the skill tools. Scanning happens once per ai.Generate call, so WrapGenerate and the tools agree on one skill set and an edited SKILL.md takes effect on the next call.
A skill tool with nothing to load is not registered, as the specification requires. Unreadable paths, malformed frontmatter, and oversized files are logged and skipped rather than reported as errors.
type SoftToolErrors ¶ added in v1.14.0
type SoftToolErrors struct {
// Tools limits the middleware to these tool names. Empty covers every
// tool, and every tool name the model asks for that does not exist.
Tools []string `` /* 161-byte string literal not displayed */
}
SoftToolErrors is a middleware that returns tool errors to the model instead of failing the generation. The model receives the error as the tool's response, {"error": "<message>"}, and can correct its input and try again.
It covers the errors a tool returns, input validation included, and calls to tool names that do not exist. These still stop the generation: errors that a WrapTool hook raises in place of the tool's own, interrupts, and cancellation. Other WrapTool hooks, such as a retry, see the tool's error before it becomes a response, wherever SoftToolErrors sits in the chain.
A tool author can return a specific error to the model without this middleware, with tool.Fail from the ai/tool package.
Usage:
resp, err := genkit.Generate(ctx, g,
ai.WithPrompt("what is the weather in Paris?"),
ai.WithTools(weatherTool),
ai.WithUse(&middleware.SoftToolErrors{}),
)
func (SoftToolErrors) Name ¶ added in v1.14.0
func (t SoftToolErrors) Name() string
Name implements ai.Middleware.
func (SoftToolErrors) New ¶ added in v1.14.0
New implements ai.Middleware, hooking each tool-loop iteration.
type ToolApproval ¶
type ToolApproval struct {
// AllowedTools is the list of tool names pre-approved to run without
// interruption. Tools not in this list trigger an interrupt, or go to
// the Judge when one is set. An empty list covers no tools.
AllowedTools []string `` /* 214-byte string literal not displayed */
// Judge is the model that decides the calls AllowedTools does not
// approve. The zero value interrupts those calls.
Judge ai.ModelRef `` /* 237-byte string literal not displayed */
// JudgePolicy is extra rules for the Judge, in natural language, added
// to its default instructions.
JudgePolicy string `` /* 136-byte string literal not displayed */
}
ToolApproval is a middleware that interrupts tool execution unless the tool is in [AllowedTools], the call has been explicitly approved on resume, or the [Judge] model allows it.
Claim a hold with ToolApprovalInterrupted and answer it with a restart carrying a ToolCallDecision. Every restart of a hold decides it: Approved true runs the call, and any other restart, a bare one included, answers it with an error the model sees, as a judge's "deny" does, so the generation continues without it:
for _, part := range resp.Interrupts() {
if call, ok := middleware.ToolApprovalInterrupted(part); ok {
parts = append(parts, call.Restart(middleware.ToolCallDecision{Approved: true}))
}
}
Answer a hold with a restart: a response from ai.InterruptedCall.Respond does not refuse the call but stands in for the tool's output, which must match the tool's output schema.
The hold is the middleware's own interrupt, so a tool's ai.ResumableToolAction.Interrupted declines it and the approval is read here, never by the tool: once approved, the tool runs as a fresh call and may interrupt with a question of its own, which the caller answers through the tool as usual. That restart passes the gate, since the call records that the gate let it through, unless it replaces the call's input, which the gate then holds for approval again. A hold that does not record which stage raised it, such as one stored before stages were recorded, is not claimed by ToolApprovalInterrupted; answer it with part.ToToolRestart and a ToolCallDecision.
Usage:
resp, err := ai.Generate(ctx, r,
ai.WithModel(m),
ai.WithPrompt("do something"),
ai.WithTools(toolA, toolB),
ai.WithUse(&middleware.ToolApproval{AllowedTools: []string{"toolA"}}),
)
// toolA runs; toolB triggers an interrupt.
// Resume with ai.WithResume(call.Restart(middleware.ToolCallDecision{Approved: true})).
Judge ¶
With [Judge] set, a model decides each call that the allowlist does not approve, in place of an unconditional interrupt. A restart that answers a hold is the caller's decision, which the judge does not revisit. The judge answers with one of three verdicts:
- "allow" runs the tool.
- "deny" answers the call with an error that the model sees, as tool.Fail from the ai/tool package does, and the generation continues.
- "ask" interrupts, as a call without a judge does.
The judge sees the text the user wrote and the pending call (the tool's name, description, and input) as one JSON document. It does not see model text, tool results, retrieved documents, or file contents the Filesystem middleware adds, since injected instructions usually arrive through those. A judge that fails, or answers with anything other than a verdict, interrupts the call.
An interrupt the judge caused carries a "judge" key in its metadata, "ask" or "failed", so the caller can tell a judge that wants the user to confirm from one that could not decide.
The judge is asked through the enum output format, so any model that can answer with one of a list of values works, including decision models such as TypeSafe's jev. A smaller, faster model than the one being judged keeps the added latency low; each judged call costs one model call.
resp, err := genkit.Generate(ctx, g,
ai.WithPrompt("clean up the build directory"),
ai.WithTools(listFiles, deleteFile),
ai.WithUse(&middleware.ToolApproval{
AllowedTools: []string{"listFiles"},
Judge: googlegenai.ModelRef("googleai/gemini-3.5-flash-lite", nil),
JudgePolicy: "Never allow deleting files outside the build directory.",
}),
)
func (ToolApproval) New ¶
New implements ai.Middleware, hooking tool execution, and with a Judge, each tool-loop iteration.
type ToolCallDecision ¶ added in v1.14.0
type ToolCallDecision struct {
// Approved decides the held call: true runs it, and false, the zero
// value, answers it with an error the model sees instead, so the
// generation continues without it.
Approved bool `json:"toolApproved"`
// Reason, when Approved is false, is added to the error the model
// sees.
Reason string `json:"reason,omitempty"`
}
ToolCallDecision is the answer to a ToolApproval hold, sent with ai.InterruptedCall.Restart on the call ToolApprovalInterrupted claims.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package exp provides experimental middleware: Agents for sub-agent delegation and Artifacts for session artifact access, which build on the agent APIs in github.com/firebase/genkit/go/ai/exp, and ContextCompression, which keeps long conversations inside a context budget.
|
Package exp provides experimental middleware: Agents for sub-agent delegation and Artifacts for session artifact access, which build on the agent APIs in github.com/firebase/genkit/go/ai/exp, and ContextCompression, which keeps long conversations inside a context budget. |