Documentation
¶
Overview ¶
Package execution owns attempt state, retry and fallback budgets, and the response-byte commitment boundary for inference execution.
Index ¶
- Variables
- func CanFallback(providerFailure *failure.Failure) bool
- func RecordCredential(ctx context.Context, evidence CredentialEvidence)
- func RecordCredentialAccepted(ctx context.Context)
- func WithAttemptAction(err error, action AttemptAction) error
- type Attempt
- type AttemptAction
- type AttemptEvidence
- type AttemptOutcome
- type Availability
- type ChatAttempt
- type ChatResult
- type Clock
- type Config
- type CredentialEvidence
- type CredentialOwner
- type EmbeddingAttempt
- type EmbeddingResult
- type Error
- type Executor
- func (e *Executor) ExecuteChat(ctx context.Context, plan *routing.Plan, attempt ChatAttempt) (*ChatResult, error)
- func (e *Executor) ExecuteEmbedding(ctx context.Context, plan *routing.Plan, attempt EmbeddingAttempt) (*EmbeddingResult, error)
- func (e *Executor) StartChatStream(ctx context.Context, plan *routing.Plan, attempt StreamAttempt) (ManagedStream, error)
- type ManagedStream
- type OutcomePublisher
- type OverheadTimer
- type Result
- type State
- type Stream
- type StreamAttempt
- type Transition
Constants ¶
This section is empty.
Variables ¶
var ( // ErrPlanRequired reports a missing route plan. ErrPlanRequired = errors.New("route plan is required") // ErrAttemptRequired reports a missing provider attempt function. ErrAttemptRequired = errors.New("provider attempt function is required") // ErrAttemptBudget reports exhaustion of the total logical-attempt budget. ErrAttemptBudget = errors.New("logical attempt budget exhausted") // ErrElapsedBudget reports exhaustion of the total elapsed-time budget. ErrElapsedBudget = errors.New("execution elapsed-time budget exhausted") // ErrAllAttemptsFailed reports that no planned route completed. ErrAllAttemptsFailed = errors.New("all planned attempts failed") )
Functions ¶
func CanFallback ¶
CanFallback reports whether policy can move to the next planned route.
func RecordCredential ¶ added in v1.0.3
func RecordCredential(ctx context.Context, evidence CredentialEvidence)
RecordCredential binds secret-free selection evidence to the current provider attempt. Calls outside an executor-owned attempt are ignored.
func RecordCredentialAccepted ¶ added in v1.0.3
RecordCredentialAccepted records that the provider accepted the selected material far enough to return a provider response or stream.
func WithAttemptAction ¶ added in v1.0.2
func WithAttemptAction(err error, action AttemptAction) error
WithAttemptAction annotates a stream read failure with execution policy. The wrapped error remains available through errors.Is and errors.As.
Types ¶
type Attempt ¶ added in v1.1.0
type Attempt[Response any] func(context.Context, routing.Attempt) (*Response, *failure.Failure, AttemptAction)
Attempt makes one non-streaming provider invocation for any canonical response type. Chat and embeddings each have a named alias below, because they are the two the gateway served before the media operations arrived.
type AttemptAction ¶ added in v1.0.2
type AttemptAction uint8
AttemptAction tells the executor how an attempt failure affects the current route. The executor still applies the one total attempt budget.
const ( // AttemptActionDefault applies normal retry and route-fallback policy. AttemptActionDefault AttemptAction = iota // AttemptActionContinueRoute consumes another attempt on the same route // without changing provider-health state or applying retry backoff. AttemptActionContinueRoute // AttemptActionFallbackRoute moves directly to the next planned route // without changing provider-health state or applying retry policy. AttemptActionFallbackRoute // AttemptActionStop ends execution without changing provider-health state. AttemptActionStop )
type AttemptEvidence ¶
type AttemptEvidence struct {
Number int
Route routing.Route
Retry int
State State
StartedAt time.Time
FinishedAt time.Time
Duration time.Duration
Failure *failure.Failure
// Credential names which credential plane paid for this attempt. It is
// evidence about the attempt in the same sense the route is, and a
// fallback can move between planes, so it is recorded per attempt rather
// than once per request. A skipped attempt carries none.
Credential CredentialEvidence
Transitions []Transition
}
AttemptEvidence records one provider invocation or availability skip.
type AttemptOutcome ¶ added in v1.0.3
type AttemptOutcome struct {
Route routing.Route
Credential CredentialEvidence
Failure *failure.Failure
}
AttemptOutcome is the safe state-transition evidence from one provider invocation.
type Availability ¶
type Availability interface {
Acquire(routing.Route) bool
Release(routing.Route)
RecordSuccess(routing.Route, time.Duration)
RecordFailure(routing.Route, *failure.Failure, time.Duration)
}
Availability owns attempt admission and offering outcome transitions.
type ChatAttempt ¶
type ChatAttempt func(context.Context, routing.Attempt) (*inference.ChatResponse, *failure.Failure, AttemptAction)
ChatAttempt makes one non-streaming provider invocation.
type ChatResult ¶
type ChatResult struct {
Response inference.ChatResponse
Route routing.Route
Attempts []AttemptEvidence
StartedAt time.Time
FinishedAt time.Time
}
ChatResult is one canonical completed result with execution evidence.
type Config ¶
type Config struct {
MaxAttempts int
MaxRetriesPerRoute int
MaxElapsed time.Duration
RetryBackoff time.Duration
BackoffMultiplier float64
MaxBackoff time.Duration
}
Config defines the one total execution budget.
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig returns bounded production defaults. A retry and a fallback consume the same MaxAttempts budget.
type CredentialEvidence ¶ added in v1.0.3
type CredentialEvidence struct {
Owner CredentialOwner
// Source is the caller's name for the plane the credential came from.
// Execution never interprets it: the owner is what decides whether an
// attempt may move the shared availability state, and the source is
// carried only so an operator can later see which of the owner's planes
// answered. Splitting the two keeps the credential vocabulary with the
// package that owns it instead of mirroring it here.
Source string
MaterialVersion string
Accepted bool
}
CredentialEvidence identifies one selected credential version without exposing its values or source reference.
type CredentialOwner ¶ added in v1.0.3
type CredentialOwner string
CredentialOwner identifies the request credential plane used for one provider attempt. It contains no account identity or credential material.
const ( // CredentialOwnerOperator identifies deployment-owned inference material. CredentialOwnerOperator CredentialOwner = "operator" // CredentialOwnerAccount identifies request-scoped account BYOK material. CredentialOwnerAccount CredentialOwner = "account" )
type EmbeddingAttempt ¶ added in v1.0.2
type EmbeddingAttempt func(context.Context, routing.Attempt) (*inference.EmbeddingResponse, *failure.Failure, AttemptAction)
EmbeddingAttempt makes one non-streaming provider invocation.
type EmbeddingResult ¶ added in v1.0.2
type EmbeddingResult struct {
Response inference.EmbeddingResponse
Route routing.Route
Attempts []AttemptEvidence
StartedAt time.Time
FinishedAt time.Time
}
EmbeddingResult is one canonical completed embedding result with execution evidence.
type Error ¶
type Error struct {
Reason error
Failure *failure.Failure
Attempts []AttemptEvidence
}
Error reports terminal execution evidence and preserves the normalized failure.
type Executor ¶
type Executor struct {
// contains filtered or unexported fields
}
Executor applies one total attempt and elapsed-time budget to one route plan.
func New ¶
func New( config Config, clock Clock, availability Availability, outcomes OutcomePublisher, ) (*Executor, error)
New creates an executor with explicit total-budget policy.
func (*Executor) ExecuteChat ¶
func (e *Executor) ExecuteChat( ctx context.Context, plan *routing.Plan, attempt ChatAttempt, ) (*ChatResult, error)
ExecuteChat executes one immutable plan for a non-streaming request.
func (*Executor) ExecuteEmbedding ¶ added in v1.0.2
func (e *Executor) ExecuteEmbedding( ctx context.Context, plan *routing.Plan, attempt EmbeddingAttempt, ) (*EmbeddingResult, error)
ExecuteEmbedding executes one immutable plan for an embedding request.
func (*Executor) StartChatStream ¶
func (e *Executor) StartChatStream( ctx context.Context, plan *routing.Plan, attempt StreamAttempt, ) (ManagedStream, error)
StartChatStream starts an execution-owned stream. It can retry or fall back only before it returns the first canonical event to its caller.
Route timing here is route-specific. The elapsed budget bounds route selection alone: it ends a stream that never delivers a first byte, and it releases as soon as one arrives. A stream that a caller reads is a stream the gateway must not cut in half, so the committed stream carries a cancelable lifetime and no deadline.
type ManagedStream ¶
type ManagedStream interface {
Stream
Attempts() []AttemptEvidence
Committed() bool
ModelUsed() string
}
ManagedStream exposes execution evidence without changing the protocol stream contract.
type OutcomePublisher ¶ added in v1.0.3
type OutcomePublisher interface {
PublishOutcome(AttemptOutcome)
}
OutcomePublisher receives completed provider invocation outcomes. It must not block on external I/O.
type OverheadTimer ¶ added in v1.1.0
type OverheadTimer struct {
// contains filtered or unexported fields
}
OverheadTimer measures the latency the gateway itself adds to one request: wall time elapsed minus time spent waiting on the upstream provider. Attempt callbacks mark upstream intervals; the HTTP layer reads the difference when it writes the response.
func OverheadTimerFrom ¶ added in v1.1.0
func OverheadTimerFrom(ctx context.Context) *OverheadTimer
OverheadTimerFrom returns the request's timer, or nil when the request did not start one. Every timer method accepts a nil receiver, so call sites need no guard.
func WithOverheadTimer ¶ added in v1.1.0
func WithOverheadTimer(ctx context.Context) (context.Context, *OverheadTimer)
WithOverheadTimer starts a timer for one request and stores it in the context so attempt callbacks and response writers share it.
func (*OverheadTimer) OverheadMS ¶ added in v1.1.0
func (t *OverheadTimer) OverheadMS() int64
OverheadMS reports the gateway-added latency measured so far in whole milliseconds. It never reports below zero.
func (*OverheadTimer) TrackUpstream ¶ added in v1.1.0
func (t *OverheadTimer) TrackUpstream() func()
TrackUpstream marks the start of one upstream wait and returns the function that ends it.
type Result ¶ added in v1.1.0
type Result[Response any] struct { Response Response Route routing.Route Attempts []AttemptEvidence StartedAt time.Time FinishedAt time.Time }
Result is one canonical completed result with execution evidence.
func Execute ¶ added in v1.1.0
func Execute[Response any]( ctx context.Context, executor *Executor, plan *routing.Plan, attempt Attempt[Response], clone func(Response) Response, ) (*Result[Response], error)
Execute applies one total attempt and elapsed-time budget to any canonical response type. A Go method carries no type parameter, so the one execution path the gateway owns is a function rather than a method. Every operation reaches it, which is what keeps one retry budget, one availability rule, and one evidence record over all of them.
clone is required. The executor hands the caller a value it also records, so a response holding a slice or a pointer would otherwise be shared with the evidence and with a replay.
type State ¶
type State string
State is one state in the logical-attempt state machine.
const ( // StateQueued identifies an attempt that has not started. StateQueued State = "queued" // StateRunning identifies an active provider attempt. StateRunning State = "running" // StateSucceeded identifies a completed provider attempt. StateSucceeded State = "succeeded" // StateFailed identifies a provider attempt that returned a failure. StateFailed State = "failed" // StateSkipped identifies a route that availability policy rejected. StateSkipped State = "skipped" // StateCanceled identifies an attempt stopped by context cancellation. StateCanceled State = "canceled" )
type Stream ¶
type Stream interface {
Read() (*inference.StreamEvent, error)
Close() error
}
Stream is a provider-neutral inference event stream.
type StreamAttempt ¶
StreamAttempt starts one provider stream. Read failures should be normalized as *failure.Failure values. Wrap a pre-commit read error with WithAttemptAction when it must continue the same route.