execution

package
v1.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 11, 2026 License: AGPL-3.0 Imports: 12 Imported by: 0

Documentation

Overview

Package execution owns attempt state, retry and fallback budgets, and the response-byte commitment boundary for inference execution.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrPlanRequired reports a missing route plan.
	ErrPlanRequired = errors.New("route plan is required")
	// ErrAttemptRequired reports a missing provider attempt function.
	ErrAttemptRequired = errors.New("provider attempt function is required")
	// ErrAttemptBudget reports exhaustion of the total logical-attempt budget.
	ErrAttemptBudget = errors.New("logical attempt budget exhausted")
	// ErrElapsedBudget reports exhaustion of the total elapsed-time budget.
	ErrElapsedBudget = errors.New("execution elapsed-time budget exhausted")
	// ErrAllAttemptsFailed reports that no planned route completed.
	ErrAllAttemptsFailed = errors.New("all planned attempts failed")
)

Functions

func CanFallback

func CanFallback(providerFailure *failure.Failure) bool

CanFallback reports whether policy can move to the next planned route.

func RecordCredential added in v1.0.3

func RecordCredential(ctx context.Context, evidence CredentialEvidence)

RecordCredential binds secret-free selection evidence to the current provider attempt. Calls outside an executor-owned attempt are ignored.

func RecordCredentialAccepted added in v1.0.3

func RecordCredentialAccepted(ctx context.Context)

RecordCredentialAccepted records that the provider accepted the selected material far enough to return a provider response or stream.

func WithAttemptAction added in v1.0.2

func WithAttemptAction(err error, action AttemptAction) error

WithAttemptAction annotates a stream read failure with execution policy. The wrapped error remains available through errors.Is and errors.As.

Types

type Attempt added in v1.1.0

type Attempt[Response any] func(context.Context, routing.Attempt) (*Response, *failure.Failure, AttemptAction)

Attempt makes one non-streaming provider invocation for any canonical response type. Chat and embeddings each have a named alias below, because they are the two the gateway served before the media operations arrived.

type AttemptAction added in v1.0.2

type AttemptAction uint8

AttemptAction tells the executor how an attempt failure affects the current route. The executor still applies the one total attempt budget.

const (
	// AttemptActionDefault applies normal retry and route-fallback policy.
	AttemptActionDefault AttemptAction = iota
	// AttemptActionContinueRoute consumes another attempt on the same route
	// without changing provider-health state or applying retry backoff.
	AttemptActionContinueRoute
	// AttemptActionFallbackRoute moves directly to the next planned route
	// without changing provider-health state or applying retry policy.
	AttemptActionFallbackRoute
	// AttemptActionStop ends execution without changing provider-health state.
	AttemptActionStop
)

type AttemptEvidence

type AttemptEvidence struct {
	Number     int
	Route      routing.Route
	Retry      int
	State      State
	StartedAt  time.Time
	FinishedAt time.Time
	Duration   time.Duration
	Failure    *failure.Failure
	// Credential names which credential plane paid for this attempt. It is
	// evidence about the attempt in the same sense the route is, and a
	// fallback can move between planes, so it is recorded per attempt rather
	// than once per request. A skipped attempt carries none.
	Credential  CredentialEvidence
	Transitions []Transition
}

AttemptEvidence records one provider invocation or availability skip.

type AttemptOutcome added in v1.0.3

type AttemptOutcome struct {
	Route      routing.Route
	Credential CredentialEvidence
	Failure    *failure.Failure
}

AttemptOutcome is the safe state-transition evidence from one provider invocation.

type Availability

type Availability interface {
	Acquire(routing.Route) bool
	Release(routing.Route)
	RecordSuccess(routing.Route, time.Duration)
	RecordFailure(routing.Route, *failure.Failure, time.Duration)
}

Availability owns attempt admission and offering outcome transitions.

type ChatAttempt

ChatAttempt makes one non-streaming provider invocation.

type ChatResult

type ChatResult struct {
	Response   inference.ChatResponse
	Route      routing.Route
	Attempts   []AttemptEvidence
	StartedAt  time.Time
	FinishedAt time.Time
}

ChatResult is one canonical completed result with execution evidence.

type Clock

type Clock interface {
	Now() time.Time
	Sleep(context.Context, time.Duration) error
}

Clock supplies deterministic attempt time and waits.

type Config

type Config struct {
	MaxAttempts        int
	MaxRetriesPerRoute int
	MaxElapsed         time.Duration
	RetryBackoff       time.Duration
	BackoffMultiplier  float64
	MaxBackoff         time.Duration
}

Config defines the one total execution budget.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns bounded production defaults. A retry and a fallback consume the same MaxAttempts budget.

type CredentialEvidence added in v1.0.3

type CredentialEvidence struct {
	Owner CredentialOwner
	// Source is the caller's name for the plane the credential came from.
	// Execution never interprets it: the owner is what decides whether an
	// attempt may move the shared availability state, and the source is
	// carried only so an operator can later see which of the owner's planes
	// answered. Splitting the two keeps the credential vocabulary with the
	// package that owns it instead of mirroring it here.
	Source          string
	MaterialVersion string
	Accepted        bool
}

CredentialEvidence identifies one selected credential version without exposing its values or source reference.

type CredentialOwner added in v1.0.3

type CredentialOwner string

CredentialOwner identifies the request credential plane used for one provider attempt. It contains no account identity or credential material.

const (
	// CredentialOwnerOperator identifies deployment-owned inference material.
	CredentialOwnerOperator CredentialOwner = "operator"
	// CredentialOwnerAccount identifies request-scoped account BYOK material.
	CredentialOwnerAccount CredentialOwner = "account"
)

type EmbeddingAttempt added in v1.0.2

EmbeddingAttempt makes one non-streaming provider invocation.

type EmbeddingResult added in v1.0.2

type EmbeddingResult struct {
	Response   inference.EmbeddingResponse
	Route      routing.Route
	Attempts   []AttemptEvidence
	StartedAt  time.Time
	FinishedAt time.Time
}

EmbeddingResult is one canonical completed embedding result with execution evidence.

type Error

type Error struct {
	Reason   error
	Failure  *failure.Failure
	Attempts []AttemptEvidence
}

Error reports terminal execution evidence and preserves the normalized failure.

func (*Error) Error

func (e *Error) Error() string

func (*Error) Is

func (e *Error) Is(target error) bool

Is preserves the terminal budget reason for errors.Is.

func (*Error) Unwrap

func (e *Error) Unwrap() error

Unwrap preserves the canonical failure for errors.As.

type Executor

type Executor struct {
	// contains filtered or unexported fields
}

Executor applies one total attempt and elapsed-time budget to one route plan.

func New

func New(
	config Config,
	clock Clock,
	availability Availability,
	outcomes OutcomePublisher,
) (*Executor, error)

New creates an executor with explicit total-budget policy.

func (*Executor) ExecuteChat

func (e *Executor) ExecuteChat(
	ctx context.Context,
	plan *routing.Plan,
	attempt ChatAttempt,
) (*ChatResult, error)

ExecuteChat executes one immutable plan for a non-streaming request.

func (*Executor) ExecuteEmbedding added in v1.0.2

func (e *Executor) ExecuteEmbedding(
	ctx context.Context,
	plan *routing.Plan,
	attempt EmbeddingAttempt,
) (*EmbeddingResult, error)

ExecuteEmbedding executes one immutable plan for an embedding request.

func (*Executor) StartChatStream

func (e *Executor) StartChatStream(
	ctx context.Context,
	plan *routing.Plan,
	attempt StreamAttempt,
) (ManagedStream, error)

StartChatStream starts an execution-owned stream. It can retry or fall back only before it returns the first canonical event to its caller.

Route timing here is route-specific. The elapsed budget bounds route selection alone: it ends a stream that never delivers a first byte, and it releases as soon as one arrives. A stream that a caller reads is a stream the gateway must not cut in half, so the committed stream carries a cancelable lifetime and no deadline.

type ManagedStream

type ManagedStream interface {
	Stream
	Attempts() []AttemptEvidence
	Committed() bool
	ModelUsed() string
}

ManagedStream exposes execution evidence without changing the protocol stream contract.

type OutcomePublisher added in v1.0.3

type OutcomePublisher interface {
	PublishOutcome(AttemptOutcome)
}

OutcomePublisher receives completed provider invocation outcomes. It must not block on external I/O.

type OverheadTimer added in v1.1.0

type OverheadTimer struct {
	// contains filtered or unexported fields
}

OverheadTimer measures the latency the gateway itself adds to one request: wall time elapsed minus time spent waiting on the upstream provider. Attempt callbacks mark upstream intervals; the HTTP layer reads the difference when it writes the response.

func OverheadTimerFrom added in v1.1.0

func OverheadTimerFrom(ctx context.Context) *OverheadTimer

OverheadTimerFrom returns the request's timer, or nil when the request did not start one. Every timer method accepts a nil receiver, so call sites need no guard.

func WithOverheadTimer added in v1.1.0

func WithOverheadTimer(ctx context.Context) (context.Context, *OverheadTimer)

WithOverheadTimer starts a timer for one request and stores it in the context so attempt callbacks and response writers share it.

func (*OverheadTimer) OverheadMS added in v1.1.0

func (t *OverheadTimer) OverheadMS() int64

OverheadMS reports the gateway-added latency measured so far in whole milliseconds. It never reports below zero.

func (*OverheadTimer) TrackUpstream added in v1.1.0

func (t *OverheadTimer) TrackUpstream() func()

TrackUpstream marks the start of one upstream wait and returns the function that ends it.

type Result added in v1.1.0

type Result[Response any] struct {
	Response   Response
	Route      routing.Route
	Attempts   []AttemptEvidence
	StartedAt  time.Time
	FinishedAt time.Time
}

Result is one canonical completed result with execution evidence.

func Execute added in v1.1.0

func Execute[Response any](
	ctx context.Context,
	executor *Executor,
	plan *routing.Plan,
	attempt Attempt[Response],
	clone func(Response) Response,
) (*Result[Response], error)

Execute applies one total attempt and elapsed-time budget to any canonical response type. A Go method carries no type parameter, so the one execution path the gateway owns is a function rather than a method. Every operation reaches it, which is what keeps one retry budget, one availability rule, and one evidence record over all of them.

clone is required. The executor hands the caller a value it also records, so a response holding a slice or a pointer would otherwise be shared with the evidence and with a replay.

type State

type State string

State is one state in the logical-attempt state machine.

const (
	// StateQueued identifies an attempt that has not started.
	StateQueued State = "queued"
	// StateRunning identifies an active provider attempt.
	StateRunning State = "running"
	// StateSucceeded identifies a completed provider attempt.
	StateSucceeded State = "succeeded"
	// StateFailed identifies a provider attempt that returned a failure.
	StateFailed State = "failed"
	// StateSkipped identifies a route that availability policy rejected.
	StateSkipped State = "skipped"
	// StateCanceled identifies an attempt stopped by context cancellation.
	StateCanceled State = "canceled"
)

type Stream

type Stream interface {
	Read() (*inference.StreamEvent, error)
	Close() error
}

Stream is a provider-neutral inference event stream.

type StreamAttempt

StreamAttempt starts one provider stream. Read failures should be normalized as *failure.Failure values. Wrap a pre-commit read error with WithAttemptAction when it must continue the same route.

type Transition

type Transition struct {
	From State
	To   State
	At   time.Time
}

Transition is one timestamped attempt state transition.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL