errs

Error Processing (platform/errs)

The errs package provides a framework for classifying errors by origin and retryability. It wraps Go's standard errors package — all framework types support errors.Is/errors.As and participate in the standard error chain.

Design

Errors are classified along two axes:

Non-retryable by default. A plain fmt.Errorf(...) is treated as a non-retryable infra error. Retryability must be explicitly opted into by wrapping with NewRetryableError. This prevents accidental infinite retry loops from unclassified errors.

Only infra errors can be retryable. User errors are never retryable — if a user action caused the failure, retrying the same operation will produce the same result. If an error is retryable, it is by definition an infrastructure issue.

Infra by default. Any error that is not explicitly wrapped with NewUserError is an infra error. There is no NewInfraError constructor — infra is the default classification.

Two Routes to a Classification

A returned error reaches IsUserError / IsRetryable / IsDependencyError carrying one of the framework types (*userError / *infraError). It gets there one of two ways:

1. Explicit wrap by the controller — the controller knows the meaning of the failure and wraps the cause with NewUserError, NewRetryableError, NewDependencyError, or NewRetryableDependencyError before returning.
2. Automatic wrap by the classifier-based ErrorProcessor — the controller returns a raw driver/library/sentinel error, and a per-backend Classifier recognises it later in the pipeline (typically inside the consumer, after ErrorProcessor.Process runs) and adds the appropriate framework wrap.

Both routes feed the same downstream helpers; the chain that reaches IsRetryable looks identical regardless of who wrapped it.

ErrorProcessor, Classifier, and the Processing Pass

Classifier inspects a single error node and returns a Verdict:

type Classifier interface {
    Classify(err error) Verdict
}

Verdicts: Unknown (this node carries no signal), User, Infra, InfraRetryable, InfraDependency, InfraDependencyRetryable.

An ErrorProcessor runs the per-chain pass that turns a raw chain into a wrapped one. It is called exactly once per chain — typically by the consumer immediately after the controller returns. After that point, callers use only the IsXxx helpers, which are pure type checks.

Two implementations ship in this package:

• NewClassifierProcessor(classifiers...) — the standard pass for primary pipeline consumers. Walks the chain twice:

  1. Pass 1 — framework-wrap check. Looks for an existing *userError / *infraError on the error's single-cause spine. If found, the chain is already interpretable and the processor returns err unchanged. No classifier is invoked.
  2. Pass 2 — classifier walk. From outermost to innermost node, each registered classifier is asked for a verdict. The first non-Unknown verdict wins and err is wrapped with the matching framework constructor.

  If no classifier recognises anything, err is returned unchanged — and behaves as non-retryable infra at the helper layer.

• AlwaysRetryableProcessor — unconditionally wraps every non-nil error with NewRetryableError, overriding any inner framework wrap. Use it for narrowly-scoped consumers — typically DLQ reconciliation — that must redeliver on any failure because there is no further dead-letter destination. Side-effect: an inner *infraError(dependency=true) is masked by the outer retryable=true wrap, since errors.As matches the outermost *infraError first. This is acceptable for the intended DLQ use case where only IsRetryable drives transport behaviour; do not pair this processor with a primary pipeline consumer or genuine user errors will retry forever instead of reaching their DLQ.

Grouped errors

Group(errs...) reports several failures that happened together as one error. It drops nils and returns nil when every member is nil, so a step that fans work out to independent handlers can accumulate failures in a loop and return the result directly:

var failures []error
for _, h := range handlers {
    if err := h.Handle(ctx, event); err != nil {
        failures = append(failures, fmt.Errorf("%s: %w", h.Name(), err))
    }
}
return errs.Group(failures...)

Pass 2 descends into a group's members as well as into ordinary single-cause wraps; Pass 1 walks only the single-cause spine. Without that descent, errors.Unwrap returns nil for a group, so a walk built on it alone sees the group node and nothing beneath it, and every member goes unclassified.

Grouping is opt-in — only Group is weighed. Unwrap() []error is also what errors.Join and fmt.Errorf with several %w produce, and there the extra causes are incidental: a cleanup failure hung off the real one, or context that happens to be an error. Weighing those would let an unrelated sibling decide retryability for the whole chain. Group is the one spelling that means "these failures are independent, rank them against each other", so the walk keys on that type and every other multi-cause error stays opaque — classified by its outermost recognisable node like any single-cause chain. A caller that wants its members ranked says so by returning Group, as submitqueue/extension/validator/composite does for its children.

The two shapes combine differently, because they mean different things:

• Down a wrap chain, the outermost verdict wins. A wrapper saw the error it wrapped and classified anyway, so it speaks with more knowledge than its cause.
• Across the members of a group, nothing shadows anything. The members are independent failures reported together, and their order is the order the caller ran them in, not a precedence. They combine by rank, so the result cannot depend on which member failed first.

The rank puts retryable above non-retryable, because the two mistakes cost differently: a wrong "retryable" spends a bounded retry budget and then dead-letters anyway, while a wrong "non-retryable" throws away a failure that would have cleared on its own. Within a retryability tier, the verdict that implicates this service outranks the one pointing elsewhere, so a partly-local failure is not reported as a pure dependency or user problem — that ordering only moves attribution, since every non-retryable verdict produces the same transport outcome. See verdictRank for the table.

A framework wrap classifies the subtree beneath it and no further. Above a group it covers the whole group and Pass 1 returns the error verbatim, so no member is consulted. Inside a member it is one member's account of one failure, with no standing to classify the failures beside it — so it contributes its own verdict to the rank like any other member. That is what keeps a sibling's transient failure from being discarded by a member that happened to arrive pre-classified, and it also removes an ordering artifact: two wrapped members of differing retryability used to resolve by whichever one errors.As reached first.

The losing member keeps its wrap in the chain, so IsUserError and IsRetryable can both report true for the same grouped error — one from a member, one from the outer wrap. Only the outer wrap drives the retry decision, the same way it does under AlwaysRetryableProcessor; IsUserError carries that precedence as a contract note, since a caller checking it before IsRetryable would drop the transient member.

One operational consequence worth knowing before relying on any of this: retrying a group re-runs everything. The retry redelivers to every child, including the ones that succeeded, so children must be idempotent, and a child that fails persistently with a retryable-looking error (a decommissioned service returning connection-refused, say) will spend the whole retry budget on every message. Drop such a child rather than absorbing it.

Choosing a processor

• Primary pipeline consumer → NewClassifierProcessor(...). Controllers' explicit NewUserError / NewDependencyError wraps must survive so user errors don't get retried, and unclassified backend errors must be inspected by the registered classifiers.
• DLQ reconciliation consumer → AlwaysRetryableProcessor. The DLQ is the last stop; any unprocessable message must come back for another attempt rather than silently drop. The DLQ subscription itself runs with a very high Retry.MaxAttempts and with its own DLQ disabled, so "always retryable + bounded-but-effectively-infinite attempts" is the convergence guarantee.

Adding a Backend-Specific Classifier

Backend classifiers live alongside the extension they classify, under platform/errs/<backend>/. The canonical examples are platform/errs/mysql (MySQL driver errors), platform/errs/http (rejected status codes and transport failures from clients built on platform/http), platform/errs/git (structured Git process failures), platform/errs/yarpc (YARPC status codes), and platform/errs/generic (transport-agnostic concerns such as context.Canceled).

A classifier:

• Inspects exactly one node — the err argument passed in. Do not call errors.Is / errors.As from inside Classify; the framework owns the chain walk. Calling it yourself can shadow a deeper-but-different verdict and breaks the controller-override rules described below.
• Returns Unknown for anything it does not recognise, so the surrounding walker can continue.
• Is stateless. The convention is to expose a package-level singleton value rather than a constructor:

// platform/errs/foo/foo.go
package foo

import "github.com/uber/submitqueue/platform/errs"

var Classifier errs.Classifier = classifier{}

type classifier struct{}

func (classifier) Classify(err error) errs.Verdict {
    // Type-assert / sentinel-compare on err directly, never errors.As / errors.Is.
    if fe, ok := err.(*FooError); ok {
        return classifyFooCode(fe.Code)
    }
    return errs.Unknown
}

Servers wire each classifier into the consumer's ErrorProcessor. Order matters only when two classifiers might both match a node — earlier classifiers win:

import (
    "github.com/uber/submitqueue/platform/errs"
    genericerrs "github.com/uber/submitqueue/platform/errs/generic"
    giterrs     "github.com/uber/submitqueue/platform/errs/git"
    httperrs    "github.com/uber/submitqueue/platform/errs/http"
    mysqlerrs   "github.com/uber/submitqueue/platform/errs/mysql"
    yarpcerrs   "github.com/uber/submitqueue/platform/errs/yarpc"
)

c := consumer.New(logger, scope, registry,
    errs.NewClassifierProcessor(
        genericerrs.Classifier,
        giterrs.Classifier,
        httperrs.Classifier,
        yarpcerrs.Classifier,
        mysqlerrs.Classifier,
    ),
)

Classifiers are not installed globally. A host that wants YARPC statuses classified adds yarpcerrs.Classifier to the ErrorProcessor at the boundary that consumes those errors, as above. This wiring covers outbound YARPC failures returned into that processor; inbound RPC handlers do not pass through it automatically and need their own transport middleware or mapper if they require the same classification.

httperrs precedes mysqlerrs for a reason worth knowing before reordering the list: the MySQL classifier treats any net.Error as retryable infra, and the *url.Error an HTTP client returns satisfies net.Error. Whichever runs first claims that node, so with the order reversed an HTTP transport failure is classified as a MySQL one — retryable either way, but no longer attributed to the dependency it came from. This is the cross-extension ambiguity NewClassifierProcessor documents as deferred; registration order is the workaround.

The YARPC classifier reads the typed status code rather than matching its rendered message. Cancellation is retryable caller-side infrastructure; transient or ambiguous server codes (Unknown, DeadlineExceeded, ResourceExhausted, Aborted, Internal, and Unavailable) are retryable dependency failures; request verdicts and permanent server failures are non-retryable dependency failures. A deadline may expire after a mutating RPC succeeded, so this classification relies on the repository-wide requirement that queue-driven operations are idempotent.

The Git classifier reads gitexec.CommandError, which preserves the Git subcommand and the underlying os/exec error through contextual wrapping. Git has no typed status to read — a connection reset and a deleted branch both leave fetch at a non-zero exit — so the classifier pairs the subcommand with the diagnostic git printed: a transport fragment counts only against a command that talks to the remote, and a lock fragment counts against any command that writes to the checkout. Only a recognised pair is retryable; every other Git failure, including a diagnostic the package has never seen, is a permanent infrastructure failure attributed to the remote or to this service. The direction is deliberate — an unlisted transient failure costs one lost retry, while a permanent failure defaulting to retryable would replay a deterministic error through the whole retry budget before dead-lettering anyway — and it is what makes the fragment lists safe to extend as Git's wording drifts between versions. Cancellation is not the Git classifier's to report: os/exec kills a context-cancelled child and reports only signal: killed, so gitexec.CommandFailure puts context.Canceled back in the chain and the generic classifier recognises it there.

Tests follow the same shape: assert per-node behaviour against Classifier.Classify(node) directly, and assert end-to-end behaviour by running errs.NewClassifierProcessor(Classifier).Process(err) and checking the helpers (IsRetryable, IsUserError, …) on the result. See platform/errs/mysql/mysql_test.go, platform/errs/git/git_test.go, platform/errs/yarpc/yarpc_test.go, and platform/errs/generic/generic_test.go.

Overriding Classification from a Controller

Because pass 1 short-circuits on the first framework wrap it finds, an explicit wrap by the controller always wins over any classifier. Use this when the controller has context the classifier cannot — typically when the same low-level error means different things in different call sites.

result, err := c.storage.Get(ctx, id)
if errors.Is(err, storage.ErrNotFound) {
    // This caller treats "not found" as a user error: the user asked for an
    // unknown resource. The mysql classifier never gets a vote because the
    // framework wrap short-circuits pass 1.
    return errs.NewUserError(fmt.Errorf("request %s: %w", id, err))
}
if err != nil {
    // Hand the raw error to the consumer's ErrorProcessor — the mysql
    // classifier will recognise deadlocks, lock-wait timeouts, etc. and wrap
    // them as retryable infra.
    return fmt.Errorf("get %s: %w", id, err)
}

Two practical rules fall out of the short-circuit semantics:

• Wrap with a framework constructor as soon as the controller knows the right verdict. Any wrap added later in the chain still wins, but wrapping early keeps the intent close to the decision.
• A wrap blocks all classifiers beneath it, including for nodes deeper than the wrap. If you want a classifier to still get a look at the cause, do not wrap above it. (In practice this is rare: controllers wrap because they have the final answer.) It does not block the sibling members of a group, which are classified and ranked independently.

When not to classify in a controller

The controller-override path is for the rare case where the controller has certain knowledge a classifier cannot derive from the error value alone — typically a sentinel (storage.ErrNotFound) that means "the user asked for something missing" in this specific call site. The default and overwhelmingly common case is the opposite: the controller returns the raw error (return fmt.Errorf("...: %w", err)) and lets the consumer's ErrorProcessor classify it.

In particular, do not reach for NewRetryableError just because replaying the message would be convenient. A failed queue publish, a failed enqueue, a "the hand-off that keeps this alive" step — these are not a license to mark the error retryable. Whether such a failure is transient is exactly what a classifier exists to decide: a transport-level classifier wraps genuine connection/timeout blips as retryable, while a malformed-request or permission failure stays non-retryable and dead-letters instead of replaying forever. Blanket NewRetryableError on a publish path defeats that and turns every permanent failure into an infinite retry loop.

Extensions Return Go Errors

Extension interfaces (ChangeProvider, Storage, Publisher) return error values and may define domain-specific sentinels. Most sentinels remain unclassified because their meaning depends on the call site; for example, storage.ErrNotFound might be a user error in one controller and an infrastructure error in another. A sentinel whose classification is intrinsic in every context may carry that classification at its declaration. storage.ErrVersionMismatch, for example, is always a retryable infrastructure error because it reports a lost optimistic-concurrency race.

Controllers should return intrinsically classified sentinels without adding another framework wrapper. The declaration remains reusable across implementations while every caller observes the same classification.

Error Chain Compatibility

Framework types preserve the full error chain. Extensions can wrap their own custom errors, and both framework-level and cause-level matching work through errors.Is/errors.As:

// Extension defines a domain error
var ErrNotFound = errors.New("record not found")

// Extension implementation wraps it
return fmt.Errorf("request id=%s: %w", id, ErrNotFound)

// Controller classifies and wraps again
return errs.NewUserError(fmt.Errorf("lookup failed: %w", extensionErr))

// All of these work on the resulting error:
errs.IsUserError(err)             // true — framework classification
errs.IsRetryable(err)             // false — user errors are never retryable
errors.Is(err, ErrNotFound)       // true — cause is in the chain

Helpers

All three are type-only checks. They do not invoke classifiers — pair them with a preceding ErrorProcessor.Process call when the controller's error may not carry an explicit wrap.