execution

package
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 25, 2026 License: MIT Imports: 8 Imported by: 4

Documentation

Overview

Package execution provides the primitives failover is built from: a HealthTracker that breaks circuits and cools down instances, and executors that try candidates in order and never fail over once output has reached the caller.

Routing stays product code. A product resolves its candidates, orders them, applies its policy gates and decides which stream frames carry output; the executors walk the list they are given.

Health

A HealthTracker keeps one state per key, the instance a candidate runs on, behind a mutex and with an injectable clock. What an operation's outcome says about health is an Observation; the default, Observe, reads the error's classification through core.ClassifyError. A circuit failure extends the key's streak of consecutive failures, and once the streak reaches the policy's FailureThreshold the circuit opens for OpenDuration, or for what Backoff returns. When that time has passed the circuit is half-open: it admits requests and, because the streak is kept, the next failure reopens it at once. A success ends the streak and closes the circuit. A RetryAfter makes the key unavailable until then, as the upstream asked, whatever succeeds meanwhile.

Execution

Execute tries candidates in order. It skips a candidate its Health reports unavailable, stops at a terminal disposition, and moves on after a failover or retryable one, repeating the candidate first only when Retry asks. It records the outcome of each candidate tried and returns the result with a trace of every candidate it reached. When the caller's context ends it stops with the context's error and records nothing.

ExecuteStream does the same for streams. A candidate serves once one of its frames carries output, content or reasoning, as the product's predicate decides. Until then its frames are held back and a failure moves on to the next candidate; from then on nothing fails over, and a failure reaches the caller as the stream's *AfterOutputError.

Existing breakers

The gateway's per-provider breaker is a HealthPolicy with the provider's circuit threshold and cooldown, no FailureWindow and IgnoreRetryAfter, read by an Observe that fails circuit failures, succeeds every other invocation error and ignores errors that never reached the upstream. Facet Studio's cooldown is a FailureThreshold of one with a 24-hour FailureWindow, IgnoreRetryAfter, and a Backoff of 1, 5 and 25 minutes and then an hour, or of 5, 10 and 20 hours and then a day for billing failures. The tests named after each product pin these mappings.

Index

Examples

Constants

View Source
const (
	DefaultFailureThreshold = 5
	DefaultOpenDuration     = 30 * time.Second
)

DefaultFailureThreshold and DefaultOpenDuration are the policy of every key of a HealthTracker without a Policy. They are router.CircuitBreaker's defaults.

Variables

View Source
var ErrNoCandidates = errors.New("execution: no candidates")

ErrNoCandidates is the error of an execution given no candidates.

Functions

This section is empty.

Types

type AfterOutputError

type AfterOutputError struct {
	// Err is the stream's own error, for errors.Is and errors.As.
	Err error
}

AfterOutputError is a stream failure after output reached the caller.

Its classification is empty, so it is terminal and neutral for health whatever Err says: no candidate can take back output the caller has seen, so neither an execution nor a product's own retry may repeat the request. A product whose streaming pushes output instead of returning a stream returns one from Execute's run, to stop there.

func (*AfterOutputError) Error

func (e *AfterOutputError) Error() string

Error is Err's message.

func (*AfterOutputError) ProviderErrorClassification

func (e *AfterOutputError) ProviderErrorClassification() core.ProviderErrorClassification

ProviderErrorClassification is empty, shadowing Err's: terminal, and not a circuit failure.

func (*AfterOutputError) Unwrap

func (e *AfterOutputError) Unwrap() error

Unwrap returns Err.

type Attempt

type Attempt[C any] struct {
	// Candidate is the candidate the entry describes.
	Candidate C
	// Key is the candidate's health key, when the execution tracks health.
	Key string
	// Unavailable reports a candidate skipped without a try because Health
	// reported it unavailable until Until.
	Unavailable bool
	Until       time.Time
	// Tries counts the tries, repeats included.
	Tries int
	// Err is the last try's error: nil when the candidate served or was
	// skipped.
	Err error
	// Classification is Err's routing metadata, Class names its kind, and
	// Disposition is what it permitted. A terminal disposition ended the
	// execution.
	Classification core.ProviderErrorClassification
	Class          core.ProviderErrorClass
	Disposition    core.Disposition
	// Duration spans the tries and the waits between them.
	Duration time.Duration
}

Attempt is one candidate's entry in an execution's trace.

type Effect

type Effect int

Effect is what one outcome does to a key's health.

const (
	// EffectNeutral changes nothing: the operation never reached the
	// instance, or its failure says nothing about the instance.
	EffectNeutral Effect = iota
	// EffectSuccess shows the instance answers. It ends the streak and
	// closes the circuit. A RetryAfter cooldown still runs its course: an
	// operation admitted before the upstream asked callers to wait proves
	// nothing about the wait.
	EffectSuccess
	// EffectFailure is instability. It extends the streak toward opening the
	// circuit.
	EffectFailure
)

type Executor

type Executor[C any] struct {
	// Health skips the candidates it reports unavailable and records the
	// outcome of every candidate tried. Nil tracks nothing.
	Health Health
	// Key returns a candidate's health key: the instance it runs on. It is
	// required with Health.
	Key func(C) string
	// Retry repeats a candidate that fails with a retryable disposition
	// before moving on. The zero value never repeats one.
	Retry Retry
	// Now returns the current time, for durations in the trace and the
	// RetryAfter of an UnavailableError, so it should agree with Health's
	// clock. Nil uses time.Now.
	Now func() time.Time
	// Sleep waits d before a repeat, or returns ctx's error once ctx ends.
	// Nil waits on a timer.
	Sleep func(ctx context.Context, d time.Duration) error
}

Executor tries candidates in order. The zero value tries each candidate once and tracks no health.

type Failure

type Failure struct {
	// Key is the key that failed.
	Key string
	// Class is the failure's Observation.Class.
	Class string
	// Streak is the length of the streak, this failure included, and
	// ClassFailures the number of its failures of Class.
	Streak        int
	ClassFailures int
}

Failure is a failure at or past the threshold, as a Backoff sees it.

type Health

type Health interface {
	// Available reports whether key may be tried now and, when it may not,
	// until when.
	Available(key string) (available bool, until time.Time)
	// Record reports the outcome of one operation on key: nil is a success.
	Record(key string, err error)
}

Health is what an Executor consults before trying a candidate and tells afterwards. HealthTracker implements it; a product may adapt its own.

type HealthOptions

type HealthOptions struct {
	// Policy returns the policy of one key, so keys can differ and follow
	// settings as they change. Nil gives every key DefaultFailureThreshold
	// and DefaultOpenDuration.
	Policy func(key string) HealthPolicy
	// Observe decides what an operation's error says about health. Nil uses
	// Observe.
	Observe func(error) Observation
	// Now returns the current time. Nil uses time.Now.
	Now func() time.Time
}

HealthOptions configures a HealthTracker. Every field is optional.

type HealthPolicy

type HealthPolicy struct {
	// FailureThreshold is the streak of consecutive circuit failures that
	// opens the circuit. Zero or less disables circuit breaking: failures
	// are not counted and the circuit never opens.
	FailureThreshold int
	// OpenDuration is how long an open circuit refuses requests.
	OpenDuration time.Duration
	// Backoff, when set, replaces OpenDuration for each failure at or past
	// the threshold, so the open time can grow with the streak or depend on
	// the failure's class. It runs under the tracker's lock and must be a
	// pure function of its argument.
	Backoff func(Failure) time.Duration
	// FailureWindow forgets a streak whose last failure is older than the
	// window, so the next failure starts a new streak. Zero never forgets.
	FailureWindow time.Duration
	// IgnoreRetryAfter keeps RetryAfter from starting cooldowns, for a
	// product whose failover predates them.
	IgnoreRetryAfter bool
}

HealthPolicy is how failures make one key unavailable.

When an open circuit's time has passed the circuit is half-open: it admits requests, and because the streak is kept the next failure reopens it at once. A success closes it.

type HealthState

type HealthState struct {
	// Streak counts consecutive circuit failures, and ClassFailures counts
	// them by Observation.Class.
	Streak        int
	ClassFailures map[string]int
	// LastFailure is when the streak last grew.
	LastFailure time.Time
	// OpenUntil is when the circuit stops refusing requests. Once it has
	// passed with the streak still at the threshold, the circuit is
	// half-open.
	OpenUntil time.Time
	// CooldownUntil is when the latest RetryAfter cooldown ends.
	CooldownUntil time.Time
}

HealthState is what a HealthTracker holds for one key.

type HealthTracker

type HealthTracker struct {
	// contains filtered or unexported fields
}

HealthTracker tracks the health of keys, typically provider instances. It is safe for concurrent use.

func NewHealthTracker

func NewHealthTracker(options HealthOptions) *HealthTracker

NewHealthTracker returns a tracker that has recorded nothing.

func (*HealthTracker) Available

func (t *HealthTracker) Available(key string) (bool, time.Time)

Available reports whether key may be tried now and, when it may not, until when: the later of the open circuit's end and the cooldown's. It reads the key's current policy, so a key whose policy stopped breaking circuits or honouring RetryAfter is no longer held back by them.

func (*HealthTracker) Record

func (t *HealthTracker) Record(key string, err error)

Record updates key's health with the outcome of one operation: nil is a success. Do not record an operation whose caller gave up: it says nothing about the instance, and Execute never records one.

func (*HealthTracker) Reset

func (t *HealthTracker) Reset(key string)

Reset forgets key, as if nothing had been recorded for it.

func (*HealthTracker) State

func (t *HealthTracker) State(key string) HealthState

State returns what the tracker holds for key: the zero HealthState when it holds nothing.

type Observation

type Observation struct {
	// Effect is what the outcome does to the instance's key.
	Effect Effect
	// Class names the kind of a failure, for a Backoff that treats kinds
	// differently, such as billing failures. The empty class is a kind too.
	Class string
	// RetryAfter is how long the upstream asked callers to wait. A positive
	// value starts a cooldown, which only a later RetryAfter can extend,
	// unless the policy ignores RetryAfter.
	RetryAfter time.Duration
}

Observation is what one outcome says about the instance that produced it.

func Observe

func Observe(err error) Observation

Observe is the default Observation of an operation's error, read from its classification:

  • nil is a success.
  • A circuit failure is a failure.
  • Another failure with an upstream status is a success: the instance answered, so a definitive rejection such as 400 or 401 ends a streak of instability instead of extending it.
  • Anything else is neutral, including a configuration error, a surface the provider lacks and a canceled request.

RetryAfter is the classification's.

type Result

type Result[C, R any] struct {
	// Value is what the serving candidate returned, and Candidate is that
	// candidate. Both are zero when no candidate served.
	Value     R
	Candidate C
	// Attempts is the trace: an entry for each candidate reached, in order.
	Attempts []Attempt[C]
}

Result is an execution's outcome.

func Execute

func Execute[C, R any](ctx context.Context, executor Executor[C], candidates []C, run func(context.Context, C) (R, error)) (Result[C, R], error)

Execute tries candidates in order until one serves, calling run for each.

A candidate Health reports unavailable is skipped. When run fails, the error's disposition decides what follows: a terminal one ends the execution with that error, while failover and retryable ones move on to the next candidate, after repeating a retryable one as Retry allows. Health records the outcome of each candidate's last try. Once ctx ends, Execute stops with ctx's error and records nothing more. A value run returns with an error is discarded, so run releases whatever it opened when it fails.

The error is nil when a candidate served. Otherwise it is ctx's error, the error of the last candidate tried, an *UnavailableError when every candidate was skipped, or ErrNoCandidates. The trace is complete either way.

Example
package main

import (
	"context"
	"fmt"

	core "github.com/xibodev/llmgw-core"
	"github.com/xibodev/llmgw-core/execution"
)

func main() {
	health := execution.NewHealthTracker(execution.HealthOptions{})
	executor := execution.Executor[string]{
		Health: health,
		Key:    func(instance string) string { return instance },
	}
	invoke := func(_ context.Context, instance string) (string, error) {
		if instance == "primary" {
			return "", &core.ProviderError{Message: "primary is overloaded", Classification: core.ProviderErrorClassification{
				StatusCode: 503, Retryable: true, FailoverEligible: true, CircuitFailure: true,
			}}
		}
		return "hello from " + instance, nil
	}

	result, err := execution.Execute(context.Background(), executor, []string{"primary", "secondary"}, invoke)
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(result.Value)
	for _, attempt := range result.Attempts {
		if attempt.Err != nil {
			fmt.Println(attempt.Candidate, attempt.Disposition, attempt.Classification.StatusCode)
		}
	}
	fmt.Println("primary's streak:", health.State("primary").Streak)
}
Output:
hello from secondary
primary retryable 503
primary's streak: 1

func ExecuteStream

func ExecuteStream[C any](ctx context.Context, executor Executor[C], candidates []C, open func(context.Context, C) (core.StreamIter, error), output func(frame []byte) bool) (Result[C, core.StreamIter], error)

ExecuteStream is Execute for streams: it opens candidates in order, and a candidate serves once its stream carries output.

output reports whether a frame carries output the caller would see, content or reasoning; keepalives and role-only frames carry none. Nil counts every frame. Frames before the first that carries output are held back, and a failure while they are held is the candidate's failure: its stream is closed and execution goes on as for Execute. The frame that carries output commits: the returned stream replays the held frames, then yields the rest, and nothing fails over any more. A later failure reaches the caller as the stream's error, an *AfterOutputError, and is not recorded. A stream that ends before any output commits too, so an empty answer is delivered rather than retried elsewhere.

Health records a success when a stream commits. The returned stream reports the losses of a stream that translates, and the caller closes it.

Example
package main

import (
	"bytes"
	"context"
	"fmt"
	"io"
	"strings"

	core "github.com/xibodev/llmgw-core"
	"github.com/xibodev/llmgw-core/execution"
)

// sseFrames yields fixed SSE frames, then fails with err or ends.
type sseFrames struct {
	frames []string
	err    error
}

func (s *sseFrames) Next() ([]byte, error) {
	if len(s.frames) == 0 {
		if s.err != nil {
			return nil, s.err
		}
		return nil, io.EOF
	}
	frame := s.frames[0]
	s.frames = s.frames[1:]
	return []byte(frame), nil
}

func (s *sseFrames) Close() error { return nil }

func main() {
	dropped := &core.ProviderError{Message: "connection reset", Classification: core.ProviderErrorClassification{
		Retryable: true, FailoverEligible: true, CircuitFailure: true,
	}}
	open := func(_ context.Context, instance string) (core.StreamIter, error) {
		if instance == "primary" {
			// A role frame carries no output, so the caller never sees it.
			return &sseFrames{frames: []string{"data: {\"role\":\"assistant\"}\n\n"}, err: dropped}, nil
		}
		return &sseFrames{frames: []string{
			"data: {\"role\":\"assistant\"}\n\n",
			"data: {\"content\":\"hello\"}\n\n",
			"data: [DONE]\n\n",
		}}, nil
	}
	carriesOutput := func(frame []byte) bool { return bytes.Contains(frame, []byte(`"content"`)) }

	result, err := execution.ExecuteStream(context.Background(), execution.Executor[string]{}, []string{"primary", "secondary"}, open, carriesOutput)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer result.Value.Close()
	fmt.Println("served by", result.Candidate)
	for {
		frame, err := result.Value.Next()
		if err != nil {
			// io.EOF, or an *AfterOutputError once output reached the caller.
			break
		}
		fmt.Println(strings.TrimSpace(string(frame)))
	}
}
Output:
served by secondary
data: {"role":"assistant"}
data: {"content":"hello"}
data: [DONE]

type Retry

type Retry struct {
	// Attempts is how many times one candidate may be tried, the first try
	// included. Below two, a candidate is tried once.
	Attempts int
	// Delay returns how long to wait before trying again, given how many
	// tries have failed and the last one's error. A product that honours
	// Retry-After waits at least core.ClassifyError(err).RetryAfter. Nil
	// tries again at once.
	Delay func(failed int, err error) time.Duration
}

Retry repeats a candidate whose try fails with core.DispositionRetryable. A failover disposition is never repeated: it permits another candidate only.

type UnavailableError

type UnavailableError struct {
	// Until is the earliest time a candidate becomes available, and
	// RetryAfter is how long after the error that is. Both are zero when
	// Health gave no time.
	Until      time.Time
	RetryAfter time.Duration
}

UnavailableError reports that Health found every candidate unavailable, so none was tried.

func (*UnavailableError) Error

func (e *UnavailableError) Error() string

Error says that no candidate was tried; the trace says why.

func (*UnavailableError) ProviderErrorClassification

func (e *UnavailableError) ProviderErrorClassification() core.ProviderErrorClassification

ProviderErrorClassification permits failover, so an enclosing execution can move on, and carries RetryAfter.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL