execution

package
v1.9.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 8, 2026 License: MIT Imports: 11 Imported by: 4

Documentation

Overview

Package execution provides the primitives failover is built from: a HealthTracker that breaks circuits and cools down instances, and executors that try candidates in order and never fail over once output has reached the caller.

Routing stays product code. A product resolves its candidates, orders them, applies its policy gates and decides which stream frames carry output; the executors walk the list they are given.

Health

A HealthTracker keeps one state per key, the instance a candidate runs on, behind a mutex and with an injectable clock. What an operation's outcome says about health is an Observation; the default, Observe, reads the error's classification through core.ClassifyError. A circuit failure extends the key's streak of consecutive failures, and once the streak reaches the policy's FailureThreshold the circuit opens for OpenDuration, or for what Backoff returns. When that time has passed the circuit is half-open: it admits requests and, because the streak is kept, the next failure reopens it at once. A success ends the streak and closes the circuit. A RetryAfter makes the key unavailable until then, as the upstream asked, whatever succeeds meanwhile.

Execution

Execute tries candidates in order. It skips a candidate its Health reports unavailable, stops at a terminal disposition, and moves on after a failover or retryable one, repeating the candidate first only when Retry asks. It records the outcome of each candidate tried and returns the result with a trace of every candidate it reached. When the caller's context ends it stops with the context's error and records nothing.

ExecuteStream does the same for streams. A candidate serves once one of its frames carries output, content or reasoning, as the product's predicate decides. Until then its frames are held back and a failure moves on to the next candidate; from then on nothing fails over, and a failure reaches the caller as the stream's *AfterOutputError.

Existing breakers

The gateway's per-provider breaker is a HealthPolicy with the provider's circuit threshold and cooldown, no FailureWindow and IgnoreRetryAfter, read by an Observe that fails circuit failures, succeeds every other invocation error and ignores errors that never reached the upstream. Facet Studio's cooldown is a FailureThreshold of one with a 24-hour FailureWindow, IgnoreRetryAfter, and a Backoff of 1, 5 and 25 minutes and then an hour, or of 5, 10 and 20 hours and then a day for billing failures. The tests named after each product pin these mappings.

Index

Examples

Constants

View Source
const (
	DefaultFailureThreshold = 5
	DefaultOpenDuration     = 30 * time.Second
)

DefaultFailureThreshold and DefaultOpenDuration are the policy of every key of a HealthTracker without a Policy. They are router.CircuitBreaker's defaults.

Variables

View Source
var ErrNoCandidates = errors.New("execution: no candidates")

ErrNoCandidates is the error of an execution given no candidates.

Functions

func ExponentialBackoff added in v0.14.0

func ExponentialBackoff(initial time.Duration, multiplier float64, maxDelay time.Duration) func(failed int, err error) time.Duration

ExponentialBackoff returns a Retry.Delay that waits initial after the first failed try and multiplier times longer after each later one, at most maxDelay, as the gateway's resilience wrapper waits. When the upstream asked callers to wait longer, through Retry-After, it waits that long instead.

func RepeatableRequest added in v0.14.0

func RepeatableRequest(request core.Request) bool

RepeatableRequest reports whether sending request again is as safe as sending it once, as the gateway's resilience wrapper decides:

  • Chat Completions, Messages and speech repeat.
  • Responses repeats unless the request is stateful: it stores the response, continues a previous response or a conversation, uses a stored prompt, or carries a tool or an input item that acts on upstream state, such as a hosted tool, a vector store or an item reference. A body that is no JSON object is not repeated.
  • Embeddings, images, video and transcription never repeat. The gateway repeats none of them, since a repeat may pay for a second result.

func Resilient added in v0.14.0

func Resilient(provider core.Provider, key string, policy Policy) core.Provider

Resilient returns provider guarded by policy, under key: each operation is refused while the circuit is open, repeated while its failure permits a repeat, and its outcome recorded.

Invoke and CountTokens repeat the operation. Stream repeats only opening the stream: once a stream is open it is the caller's, and its later failures are neither repeated nor recorded. A caller that gives up gets ctx's error and nothing is recorded. Otherwise the last try's error is returned as the provider returned it.

ListModels and NativeSurfaces pass through, as in the gateway: a catalog read neither repeats nor counts against the circuit, so an open circuit never keeps a catalog from being rediscovered. The wrapper unwraps to provider, so core.PreservesWire reads provider's declarations, and it counts tokens only when provider can.

A provider that keys an upstream invocation on its context, as Zen does, needs that identity in ctx before the call, so repeats reuse it.

func TransientClassification added in v0.14.0

func TransientClassification(err error) core.ProviderErrorClassification

TransientClassification reads err as the gateway's resilience wrapper does, for a Policy that repeats and breaks circuits as the gateway did. It is opt-in: core's own reading, which the executors and the default Observe use, stays as it is.

It is err's own classification, with the gateway's reading of an upstream status. Of the statuses that refuse a request, only 408, 429, 500, 502, 503 and 504 are transient, so only they repeat and count against the circuit; any other ends the streak. Core's own reading repeats every 5xx, and counts neither 408 nor 429.

Every failure that repeats counts against the circuit, a transport failure included. A failure without a status, an answer the upstream sent that could not be used, which carries a 2xx, and an operation its caller gave up on keep their own reading. Failover and RetryAfter stay err's own.

Types

type Admission added in v1.9.0

type Admission interface {
	// Admit reports whether key may be tried now, taking its share of any
	// limit when it may, and when it may not, until when; the zero time
	// when it cannot tell.
	Admit(key string) (admitted bool, until time.Time)
}

Admission decides whether a key may be tried now, before an Executor tries it: a per-instance limit on requests a minute, for one.

type AfterOutputError

type AfterOutputError struct {
	// Err is the stream's own error, for errors.Is and errors.As.
	Err error
}

AfterOutputError is a stream failure after output reached the caller.

Its classification is empty, so it is terminal and neutral for health whatever Err says: no candidate can take back output the caller has seen, so neither an execution nor a product's own retry may repeat the request. A product whose streaming pushes output instead of returning a stream returns one from Execute's run, to stop there.

func (*AfterOutputError) Error

func (e *AfterOutputError) Error() string

Error is Err's message.

func (*AfterOutputError) ProviderErrorClassification

func (e *AfterOutputError) ProviderErrorClassification() core.ProviderErrorClassification

ProviderErrorClassification is empty, shadowing Err's: terminal, and not a circuit failure.

func (*AfterOutputError) Unwrap

func (e *AfterOutputError) Unwrap() error

Unwrap returns Err.

type Attempt

type Attempt[C any] struct {
	// Candidate is the candidate the entry describes.
	Candidate C
	// Key is the candidate's health key, when the execution tracks health
	// or admits candidates.
	Key string
	// Unavailable reports a candidate skipped without a try because Health
	// reported it unavailable until Until, and Unadmitted one the
	// Admission refused until Until.
	Unavailable bool
	Unadmitted  bool
	Until       time.Time
	// Tries counts the tries, repeats included.
	Tries int
	// Err is the last try's error: nil when the candidate served or was
	// skipped.
	Err error
	// Classification is Err's routing metadata, Class names its kind, and
	// Disposition is what it permitted. A terminal disposition ended the
	// execution.
	Classification core.ProviderErrorClassification
	Class          core.ProviderErrorClass
	Disposition    core.Disposition
	// Duration spans the tries and the waits between them.
	Duration time.Duration
}

Attempt is one candidate's entry in an execution's trace.

type CircuitOpenError added in v0.14.0

type CircuitOpenError struct {
	// Key is the instance's health key.
	Key string
	// Until is when the instance becomes available, and RetryAfter is how
	// long after the error that is. Both are zero when Health gave no time.
	Until      time.Time
	RetryAfter time.Duration
}

CircuitOpenError is Resilient's answer while Health reports an instance unavailable: its circuit is open, or a Retry-After cooldown runs. Nothing was sent. It is a 503, as in the gateway, which permits failover but no repeat, since a repeat would only meet the open circuit again.

It says nothing about the instance, so record it in no Health: Observe would read its status as an answer.

func (*CircuitOpenError) Error added in v0.14.0

func (e *CircuitOpenError) Error() string

Error names the instance and how long it stays unavailable.

func (*CircuitOpenError) ProviderErrorClassification added in v0.14.0

func (e *CircuitOpenError) ProviderErrorClassification() core.ProviderErrorClassification

ProviderErrorClassification is a 503 that permits failover and carries RetryAfter.

type Effect

type Effect int

Effect is what one outcome does to a key's health.

const (
	// EffectNeutral changes nothing: the operation never reached the
	// instance, or its failure says nothing about the instance.
	EffectNeutral Effect = iota
	// EffectSuccess shows the instance answers. It ends the streak and
	// closes the circuit. A RetryAfter cooldown still runs its course: an
	// operation admitted before the upstream asked callers to wait proves
	// nothing about the wait.
	EffectSuccess
	// EffectFailure is instability. It extends the streak toward opening the
	// circuit.
	EffectFailure
)

type Executor

type Executor[C any] struct {
	// Health skips the candidates it reports unavailable and records the
	// outcome of every candidate tried. Nil tracks nothing.
	Health Health
	// Admission, when set, decides whether a candidate Health admits may
	// be tried now, as a client-side rate limiter keyed by instance does.
	// A candidate it refuses is skipped without a try, as one Health
	// reports unavailable is.
	Admission Admission
	// Key returns a candidate's health key: the instance it runs on. It is
	// required with Health or Admission.
	Key func(C) string
	// CredentialFailover moves on to the next candidate when one's
	// credential is rejected, a failure of class auth or forbidden, rather
	// than ending the execution, for candidates that hold different
	// credentials. The rejection counts against no instance: Health
	// records an answer, as it records any other refusal.
	CredentialFailover bool
	// Retry repeats a candidate that fails with a retryable disposition
	// before moving on. The zero value never repeats one.
	Retry Retry
	// Now returns the current time, for durations in the trace and the
	// RetryAfter of an UnavailableError, so it should agree with Health's
	// clock. Nil uses time.Now.
	Now func() time.Time
	// Sleep waits d before a repeat, or returns ctx's error once ctx ends.
	// Nil waits on a timer.
	Sleep func(ctx context.Context, d time.Duration) error
}

Executor tries candidates in order. The zero value tries each candidate once and tracks no health.

type Failure

type Failure struct {
	// Key is the key that failed.
	Key string
	// Class is the failure's Observation.Class.
	Class string
	// Streak is the length of the streak, this failure included, and
	// ClassFailures the number of its failures of Class.
	Streak        int
	ClassFailures int
}

Failure is a failure at or past the threshold, as a Backoff sees it.

type Health

type Health interface {
	// Available reports whether key may be tried now and, when it may not,
	// until when.
	Available(key string) (available bool, until time.Time)
	// Record reports the outcome of one operation on key: nil is a success.
	Record(key string, err error)
}

Health is what an Executor consults before trying a candidate and tells afterwards. HealthTracker implements it; a product may adapt its own.

type HealthOptions

type HealthOptions struct {
	// Policy returns the policy of one key, so keys can differ and follow
	// settings as they change. Nil gives every key DefaultFailureThreshold
	// and DefaultOpenDuration.
	Policy func(key string) HealthPolicy
	// Observe decides what an operation's error says about health. Nil uses
	// Observe.
	Observe func(error) Observation
	// Now returns the current time. Nil uses time.Now.
	Now func() time.Time
}

HealthOptions configures a HealthTracker. Every field is optional.

type HealthPolicy

type HealthPolicy struct {
	// FailureThreshold is the streak of consecutive circuit failures that
	// opens the circuit. Zero or less disables circuit breaking: failures
	// are not counted and the circuit never opens.
	FailureThreshold int
	// OpenDuration is how long an open circuit refuses requests.
	OpenDuration time.Duration
	// Backoff, when set, replaces OpenDuration for each failure at or past
	// the threshold, so the open time can grow with the streak or depend on
	// the failure's class. It runs under the tracker's lock and must be a
	// pure function of its argument.
	Backoff func(Failure) time.Duration
	// FailureWindow forgets a streak whose last failure is older than the
	// window, so the next failure starts a new streak. Zero never forgets.
	FailureWindow time.Duration
	// IgnoreRetryAfter keeps RetryAfter from starting cooldowns, for a
	// product whose failover predates them.
	IgnoreRetryAfter bool
}

HealthPolicy is how failures make one key unavailable.

When an open circuit's time has passed the circuit is half-open: it admits requests, and because the streak is kept the next failure reopens it at once. A success closes it.

type HealthState

type HealthState struct {
	// Streak counts consecutive circuit failures, and ClassFailures counts
	// them by Observation.Class.
	Streak        int
	ClassFailures map[string]int
	// LastFailure is when the streak last grew.
	LastFailure time.Time
	// OpenUntil is when the circuit stops refusing requests. Once it has
	// passed with the streak still at the threshold, the circuit is
	// half-open.
	OpenUntil time.Time
	// CooldownUntil is when the latest RetryAfter cooldown ends.
	CooldownUntil time.Time
}

HealthState is what a HealthTracker holds for one key.

type HealthTracker

type HealthTracker struct {
	// contains filtered or unexported fields
}

HealthTracker tracks the health of keys, typically provider instances. It is safe for concurrent use.

func NewHealthTracker

func NewHealthTracker(options HealthOptions) *HealthTracker

NewHealthTracker returns a tracker that has recorded nothing.

func (*HealthTracker) Available

func (t *HealthTracker) Available(key string) (bool, time.Time)

Available reports whether key may be tried now and, when it may not, until when: the later of the open circuit's end and the cooldown's. It reads the key's current policy, so a key whose policy stopped breaking circuits or honouring RetryAfter is no longer held back by them.

func (*HealthTracker) Record

func (t *HealthTracker) Record(key string, err error)

Record updates key's health with the outcome of one operation: nil is a success. Do not record an operation whose caller gave up: it says nothing about the instance, and Execute never records one.

func (*HealthTracker) Reset

func (t *HealthTracker) Reset(key string)

Reset forgets key, as if nothing had been recorded for it.

func (*HealthTracker) State

func (t *HealthTracker) State(key string) HealthState

State returns what the tracker holds for key: the zero HealthState when it holds nothing.

type Observation

type Observation struct {
	// Effect is what the outcome does to the instance's key.
	Effect Effect
	// Class names the kind of a failure, for a Backoff that treats kinds
	// differently, such as billing failures. The empty class is a kind too.
	Class string
	// RetryAfter is how long the upstream asked callers to wait. A positive
	// value starts a cooldown, which only a later RetryAfter can extend,
	// unless the policy ignores RetryAfter.
	RetryAfter time.Duration
}

Observation is what one outcome says about the instance that produced it.

func Observe

func Observe(err error) Observation

Observe is the default Observation of an operation's error, read from its classification:

  • nil is a success.
  • A circuit failure is a failure.
  • Another failure with an upstream status is a success: the instance answered, so a definitive rejection such as 400 or 401 ends a streak of instability instead of extending it.
  • Anything else is neutral, including a configuration error, a surface the provider lacks and a canceled request.

RetryAfter is the classification's, and Class the error's class, as an Attempt names it, so a Backoff can hold back a key that failed for billing longer than one that was rate limited.

type Policy added in v0.14.0

type Policy struct {
	// Retry repeats a try whose error Classify reads as retryable, up to
	// Retry.Attempts tries of one operation, waiting Retry.Delay between
	// them. ExponentialBackoff is the gateway's Delay.
	Retry Retry
	// Health is the instance's circuit, under the key Resilient is given.
	// While it reports the key unavailable, every operation is refused
	// with a *CircuitOpenError and nothing is sent. It records the outcome
	// of each operation's last try, so an operation whose repeats all fail
	// counts once. Nil guards nothing.
	Health Health
	// Classify reads a try's error: whether it may repeat, whether it
	// counts against the circuit, and how long the upstream asked callers
	// to wait. Retry.Delay and Health read the error as Classify does. Nil
	// uses core.ClassifyError.
	Classify func(error) core.ProviderErrorClassification
	// Repeatable reports whether a request may be sent more than once; one
	// it refuses is tried once, whatever Retry says. Nil uses
	// RepeatableRequest.
	Repeatable func(core.Request) bool
	// Now returns the current time, for a CircuitOpenError's RetryAfter,
	// so it should agree with Health's clock. Nil uses time.Now.
	Now func() time.Time
	// Sleep waits d before a repeat, or returns ctx's error once ctx ends.
	// Nil waits on a timer.
	Sleep func(ctx context.Context, d time.Duration) error
}

Policy is how Resilient serves one instance: how often it tries an operation, and the circuit that guards the instance. It holds what the gateway's per-provider retry and circuit-breaker policy holds. The zero value tries every operation once and guards nothing.

type Result

type Result[C, R any] struct {
	// Value is what the serving candidate returned, and Candidate is that
	// candidate. Both are zero when no candidate served.
	Value     R
	Candidate C
	// Attempts is the trace: an entry for each candidate reached, in order.
	Attempts []Attempt[C]
}

Result is an execution's outcome.

func Execute

func Execute[C, R any](ctx context.Context, executor Executor[C], candidates []C, run func(context.Context, C) (R, error)) (Result[C, R], error)

Execute tries candidates in order until one serves, calling run for each.

A candidate Health reports unavailable, or the Admission refuses, is skipped. When run fails, the error's disposition decides what follows: a terminal one ends the execution with that error, unless it rejects a credential under CredentialFailover, while failover and retryable ones move on to the next candidate, after repeating a retryable one as Retry allows. Health records the outcome of each candidate's last try. Once ctx ends, Execute stops with ctx's error and records nothing more. A value run returns with an error is discarded, so run releases whatever it opened when it fails.

The error is nil when a candidate served. Otherwise it is ctx's error, the error of the last candidate tried, an *UnavailableError when every candidate was skipped, or ErrNoCandidates. The trace is complete either way.

Example
package main

import (
	"context"
	"fmt"

	core "github.com/xibodev/llmgw-core"
	"github.com/xibodev/llmgw-core/execution"
)

func main() {
	health := execution.NewHealthTracker(execution.HealthOptions{})
	executor := execution.Executor[string]{
		Health: health,
		Key:    func(instance string) string { return instance },
	}
	invoke := func(_ context.Context, instance string) (string, error) {
		if instance == "primary" {
			return "", &core.ProviderError{Message: "primary is overloaded", Classification: core.ProviderErrorClassification{
				StatusCode: 503, Retryable: true, FailoverEligible: true, CircuitFailure: true,
			}}
		}
		return "hello from " + instance, nil
	}

	result, err := execution.Execute(context.Background(), executor, []string{"primary", "secondary"}, invoke)
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(result.Value)
	for _, attempt := range result.Attempts {
		if attempt.Err != nil {
			fmt.Println(attempt.Candidate, attempt.Disposition, attempt.Classification.StatusCode)
		}
	}
	fmt.Println("primary's streak:", health.State("primary").Streak)
}
Output:
hello from secondary
primary retryable 503
primary's streak: 1

func ExecuteStream

func ExecuteStream[C any](ctx context.Context, executor Executor[C], candidates []C, open func(context.Context, C) (core.StreamIter, error), output func(frame []byte) bool) (Result[C, core.StreamIter], error)

ExecuteStream is Execute for streams: it opens candidates in order, and a candidate serves once its stream carries output.

output reports whether a frame carries output the caller would see, content or reasoning; keepalives and role-only frames carry none. Nil counts every frame. Frames before the first that carries output are held back, and a failure while they are held is the candidate's failure: its stream is closed and execution goes on as for Execute. The frame that carries output commits: the returned stream replays the held frames, then yields the rest, and nothing fails over any more. A later failure reaches the caller as the stream's error, an *AfterOutputError, and is not recorded. A stream that ends before any output commits too, so an empty answer is delivered rather than retried elsewhere.

Health records a success when a stream commits. The returned stream reports the losses of a stream that translates, and the caller closes it.

Example
package main

import (
	"bytes"
	"context"
	"fmt"
	"io"
	"strings"

	core "github.com/xibodev/llmgw-core"
	"github.com/xibodev/llmgw-core/execution"
)

// sseFrames yields fixed SSE frames, then fails with err or ends.
type sseFrames struct {
	frames []string
	err    error
}

func (s *sseFrames) Next() ([]byte, error) {
	if len(s.frames) == 0 {
		if s.err != nil {
			return nil, s.err
		}
		return nil, io.EOF
	}
	frame := s.frames[0]
	s.frames = s.frames[1:]
	return []byte(frame), nil
}

func (s *sseFrames) Close() error { return nil }

func main() {
	dropped := &core.ProviderError{Message: "connection reset", Classification: core.ProviderErrorClassification{
		Retryable: true, FailoverEligible: true, CircuitFailure: true,
	}}
	open := func(_ context.Context, instance string) (core.StreamIter, error) {
		if instance == "primary" {
			// A role frame carries no output, so the caller never sees it.
			return &sseFrames{frames: []string{"data: {\"role\":\"assistant\"}\n\n"}, err: dropped}, nil
		}
		return &sseFrames{frames: []string{
			"data: {\"role\":\"assistant\"}\n\n",
			"data: {\"content\":\"hello\"}\n\n",
			"data: [DONE]\n\n",
		}}, nil
	}
	carriesOutput := func(frame []byte) bool { return bytes.Contains(frame, []byte(`"content"`)) }

	result, err := execution.ExecuteStream(context.Background(), execution.Executor[string]{}, []string{"primary", "secondary"}, open, carriesOutput)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer result.Value.Close()
	fmt.Println("served by", result.Candidate)
	for {
		frame, err := result.Value.Next()
		if err != nil {
			// io.EOF, or an *AfterOutputError once output reached the caller.
			break
		}
		fmt.Println(strings.TrimSpace(string(frame)))
	}
}
Output:
served by secondary
data: {"role":"assistant"}
data: {"content":"hello"}
data: [DONE]

type Retry

type Retry struct {
	// Attempts is how many times one candidate may be tried, the first try
	// included. Below two, a candidate is tried once.
	Attempts int
	// Delay returns how long to wait before trying again, given how many
	// tries have failed and the last one's error. A product that honours
	// Retry-After waits at least core.ClassifyError(err).RetryAfter. Nil
	// tries again at once.
	Delay func(failed int, err error) time.Duration
}

Retry repeats a candidate whose try fails with core.DispositionRetryable. A failover disposition is never repeated: it permits another candidate only.

type UnavailableError

type UnavailableError struct {
	// Until is the earliest time a candidate becomes available, and
	// RetryAfter is how long after the error that is. Both are zero when
	// Health gave no time.
	Until      time.Time
	RetryAfter time.Duration
}

UnavailableError reports that Health or the Admission held back every candidate, so none was tried.

func (*UnavailableError) Error

func (e *UnavailableError) Error() string

Error says that no candidate was tried; the trace says why.

func (*UnavailableError) ProviderErrorClassification

func (e *UnavailableError) ProviderErrorClassification() core.ProviderErrorClassification

ProviderErrorClassification permits failover, so an enclosing execution can move on, and carries RetryAfter.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL