idempotency

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package idempotency runs a keyed request's local transactional effect at most once under its Idempotency-Key: while the key is unexpired, a retry of a committed effect gets the stored receipt, confirmed against the current state, instead of running the effect again. It holds the replay every keyed operation shares, the gate that refuses a key where none is accepted, and the port of the receipt store.

Role

platform layer: replay of local transactional effects under an idempotency key. Retry safety belongs to no module. The router runs Gate ahead of authentication and admission, and installs Strict on every generated server; an operation whose API description references the shared IdempotencyKey parameter is wrapped, and its module supplies a Binding (decision 0043, "Replay for keyed operations"). The replay turns the key, the credential, the operation and the resource into a ReplayRequest and runs the generated handler as the effect through ReplayStore. The port and its values are declared here, apart from any store, so that the platform imports no module; the platform's store adapters (platform/store/sql and platform/store/inmem) implement ReplayStore over the api_replays table, and the metadata store's Replays returns theirs. The package imports the kernel, platform/tx, platform/httpkit and the generated operation table (platform/gen/operations).

Inputs

  • The generated operation table: which operations declare the IdempotencyKey parameter, for Gate and Strict.
  • From the router: the request's Idempotency-Key values, its principal and its exact body (httpkit.RawBody, kept for requests Keyed reports), and the generated strict server's request object.
  • From the module that serves a keyed operation: its Binding (the canonical resource, the effect parameters the fingerprint adds, the reload that confirms a committed effect and builds its answer, and the error mapping), and from its handler the resource ID, through Report.
  • ReplayStore and the metadata store's tx.Runner, which Replay holds.

Outputs

  • Gate: the middleware that answers 400 to a key on any operation that does not declare one, or to more than one key.
  • Strict: the generated servers' strict middleware, which runs a keyed request under Replay and passes every other request through.
  • Replay and Effect: Do runs an effect and its reload once under a request, within a budget, after pruning a bounded batch of expired receipts. Retention, Budget and PruneLimit are its constants.
  • Binding, Report, Keyed, ParseKey and Header.
  • ReplayStore: Do, which runs the effect or replays its receipt, and Prune, which removes expired receipts in bounded batches.
  • ReplayRequest, with Validate, which checks the digests and the retention interval.
  • ReplayReceipt: the resource ID, the expiry and whether the call replayed a committed effect.
  • ErrReplayMismatch, which Strict answers with 422.

Invariants

  • Which operations accept a key is the operation table's to say: Gate and Strict read it, and the router's binding test fails while an operation that declares the parameter has no binding, or a binding names one that does not. A keyed operation without a binding answers 500 rather than run unprotected.
  • A request without a key runs exactly as it would unwrapped.
  • Scope, key and fingerprint are SHA-256 digests, never credentials, raw keys or request bodies; Validate refuses anything else with an error that wraps kernel.ErrInvalid.
  • The claim, the handler's mutations, the reload and the receipt commit together, in the caller's metadata transaction when there is one, and any error rolls them all back, so a crash leaves no pending state to recover. A handler that reports no resource commits nothing either.
  • A receipt carries no response body, so no secret enters the replay store; a replay's answer is the reload's, built from current state.
  • An unexpired key reused with another fingerprint is ErrReplayMismatch; an expired key may run a new effect.
  • Now and Expires come from the caller's clock, so replicas need synchronized clocks.
  • It is not an outbox for external calls: an effect outside the metadata transaction needs a recovery design of its own.

References

  • Decision 0043: docs/research/decisions/0043-api-reliability.md ("Optional protected build append": the scope, the digests, the receipt and its expiry, and what a replay rechecks; "Replay for keyed operations": the shared replay and its bindings; "A key where none is accepted": the gate; "Retained validation": the tests that keep it).
  • How to add a keyed operation: docs/research/api-reliability/retrying.md.
  • Decision 0044: docs/research/decisions/0044-modular-monolith-migration.md ("Package placement and names": platform/idempotency is one of the new packages the target layout does not name).
  • Target layout: docs/research/architecture-options/5-modular-monolith.md, lines 40 and 41 (the platform imports no module and no gateway) and line 727 (platform/events' idempotency.go, consumer-side event dedupe, a different concern from this package).
  • Plan: docs/implementation/plan.md#p9d-modular-monolith-migration, epic PLT (https://github.com/weaveplatform/artifactweave/issues/938).
  • https://github.com/weaveplatform/artifactweave/issues/890
  • https://github.com/weaveplatform/artifactweave/issues/989
  • https://github.com/weaveplatform/artifactweave/issues/1406

Status

Implemented with decision 0043 (issue 890, pull request 905) and released in v0.2.0, the P9c release: the replay port in the metadata store, with its SQL and in-memory stores, and protected build appends. Moved here in P9d: the port and its values are declared here. IDEM-01 (issue 1406) added the shared replay (Replay, Strict and Binding) and the table-driven Gate, and moved build append onto them.

Testing

  • run_test.go: Do refuses invalid requests and missing functions, stops on a prune failure, answers a deadline within its budget, and keeps the deadline behind an error a store maps.
  • keyed_test.go: ParseKey's accepted and refused keys; Strict over an in-memory store runs a handler once and replays it, passes unkeyed and unwrapped operations through, checks the resource before the key, refuses a missing, anonymous, registry or otherwise unstable credential, answers an unbound operation 500 and a store-less replay 501, fingerprints effect parameters, rolls back a failed handler, reload or unreported resource, and answers 503 at its budget.
  • gate_test.go: over the generated table the gate answers as main's hand-written gate did for every method, path and key combination tried; its wording names build append only while that is the only keyed operation; the refusal uses the path's error envelope; a bad table fails.
  • The platform's replay contract (the Replays, ReplayRollback, ReplayInProgressCancellation and Keyed* subtests of platform/store/storetest: concurrent identical requests run the effect once and replay it after, a changed fingerprint is ErrReplayMismatch, another scope and an expired key run new effects, bounded pruning, the requests Validate refuses, failed and nested transactions, a refused reload, and a duplicate that stops waiting when its context or the budget ends) runs on the platform's in-memory and SQLite stores and on the metadata store's in-memory, SQLite and PostgreSQL stores.
  • The metadata store's SQLite and PostgreSQL stores' TestAccReplayProcessCrash and TestAccKeyedReplayProcessCrash kill a process inside the receipt transaction, and the PostgreSQL store's TestAccReplayPruneDoesNotDeleteRenewal keeps a renewed receipt that a concurrent prune had selected.
  • The router's keyed_test.go checks the bindings against the table and the gate through the router; build append's binding is tested in the artifacts module's HTTP adapter and in test/server/artifacts.

Index

Constants

View Source
const (
	// Retention is how long a receipt holds its key: 24 hours after the
	// first successful request's admission (decision 0043).
	Retention = 24 * time.Hour
	// Budget bounds a keyed request, its wait for a concurrent original
	// included, when Replay.Budget is zero (decision 0043).
	Budget = 30 * time.Second
	// PruneLimit is how many expired receipts each keyed request removes
	// before it runs, so that cleanup keeps pace with keyed traffic without
	// a sweeper (decision 0043).
	PruneLimit = 100
)
View Source
const Header = "Idempotency-Key"

Header is the request header that carries an idempotency key.

Variables

View Source
var ErrReplayMismatch = errors.New("store: idempotency key has different intent")

ErrReplayMismatch means an unexpired key was reused with different intent.

Functions

func Gate

func Gate(table []operations.Operation) (func(http.Handler) http.Handler, error)

Gate returns middleware that answers 400 to a request carrying Idempotency-Key unless it carries exactly one and table's operation for its method and path declares the shared IdempotencyKey parameter. The router runs it ahead of authentication and admission, so the 400 uses the path's error envelope, is charged to no allowance and carries no rate header, and no handler that would ignore the key can run with it. A hand- written alias, a native protocol route and an unknown path match no row, and a 501 stub declares no parameter, so each answers 400 too. It fails as operations.NewIndex does.

func Keyed

func Keyed(r *http.Request) bool

Keyed reports whether r carries an Idempotency-Key, so that the router keeps the exact body bytes the fingerprint digests.

func ParseKey

func ParseKey(raw string) (string, error)

ParseKey checks a raw Idempotency-Key value, a quoted string of 1 to 128 ASCII letters, digits, ".", "_", ":" or "-" (decision 0043, a bounded subset of the draft's structured string), and returns the SHA-256 hex digest of the key inside the quotes, never the key itself.

func Report

func Report(ctx context.Context, id int64)

Report records id, the positive, stable ID of the resource that a keyed handler's effect created or changed, for the receipt. A handler under the replay calls it on every success; one that answers without it has its work rolled back and gets no receipt, which is how a typed error response that is not a Go error stays unreplayed. Outside the replay it does nothing.

func Strict

func Strict[H ~func(context.Context, http.ResponseWriter, *http.Request, any) (any, error)](
	p Replay,
	service string,
	table []operations.Operation,
	bindings map[string]Binding,
) func(H, string) H

Strict returns the generated strict servers' middleware for service's operations in table. It wraps each operation that declares the shared IdempotencyKey parameter, and passes every other one through untouched. A wrapped operation called without a key also runs as it does unwrapped. With a key it runs under the replay with its binding from bindings, keyed by operation ID; one without a binding answers 500 rather than run unprotected. H is the generated package's StrictHandlerFunc, whose middleware receives the Go name of the operation ID.

Types

type Binding

type Binding struct {
	// Name names the operation's effect in the answer to a request that
	// runs out of budget, "Protected <Name> exceeded its wait budget";
	// empty means "request".
	Name string
	// Resource returns the canonical resource the request addresses, as the
	// operation derives it from its path and query: aliases resolved and
	// timestamps at the store's microsecond precision. The scope digests it
	// as JSON. It runs before the key is read, so it makes the checks the
	// handler makes first and returns the handler's error for a request
	// they refuse. Nil scopes the key to the credential and the operation
	// alone.
	Resource func(ctx context.Context, request any) (any, error)
	// Fingerprint returns the request parameters that change what the
	// effect does without selecting its target, such as a dry-run flag,
	// which the fingerprint digests as JSON after the body. Nil means none.
	// A parameter that only checks consistency belongs in neither.
	Fingerprint func(ctx context.Context, request any) (any, error)
	// Reload confirms a committed effect and answers for it: it locks and
	// loads the resource the receipt names, rechecks the current
	// credential and permissions, and returns the operation's success
	// response built from the committed state. It runs after the handler
	// and on every replay, in the same transaction; only a replay's answer
	// is its response.
	Reload func(ctx context.Context, request any, receipt ReplayReceipt) (any, error)
	// Error maps an error from the handler, Reload or the replay's store to
	// the operation's answer. Nil passes an *apierror.Error through and
	// answers anything else 500.
	Error func(ctx context.Context, request any, err error) error
}

Binding is what one keyed operation gives the replay beyond its handler, which the replay runs as the effect. The request each function receives is the generated strict server's request object for the operation.

type Effect

type Effect struct {
	// Run makes the effect through the context it is given and returns the
	// positive, stable ID of the resource it created or changed. It runs
	// only for a key that has no unexpired receipt.
	Run func(ctx context.Context) (int64, error)
	// Reload locks and loads the resource the receipt names and rechecks
	// the current credential and permissions. It runs after Run and on
	// every replay, in the same transaction, so a receipt commits only once
	// its resource has been confirmed, and a replay never outlives the
	// access that made it.
	Reload func(ctx context.Context, receipt ReplayReceipt) error
}

Effect is the work of one keyed request.

type Replay

type Replay struct {
	// Store keeps the receipts.
	Store ReplayStore
	// Tx is the metadata store's unit of work, which every store call the
	// effect makes with its context joins.
	Tx tx.Runner
	// Now supplies the time a keyed request is admitted at; nil means
	// time.Now.
	Now func() time.Time
	// Budget bounds each request; zero means the constant Budget. Tests
	// shorten it.
	Budget time.Duration
}

Replay runs keyed requests once each (decision 0043, "Replay for keyed operations"). Its zero value is not usable: Store and Tx must be the receipts and the unit of work of one metadata store, so that a claim, the effect and the receipt share a transaction.

func (Replay) Do

Do prunes at most PruneLimit expired receipts, then runs e under r in one transaction: the claim, e.Run for a new or expired key, e.Reload, and the receipt commit together or roll back together. The whole of it, a wait for a concurrent original included, is bounded by ctx and the budget; when either ends first the error wraps context.DeadlineExceeded, which leaves the caller unable to tell whether a concurrent original committed. A request Validate refuses and a nil Run or Reload are kernel.ErrInvalid; an unexpired key with another fingerprint is ErrReplayMismatch. Called inside a transaction of the same store, Do joins it, and the outer commit decides for both.

type ReplayReceipt

type ReplayReceipt struct {
	ResourceID int64
	Expires    time.Time
	Replayed   bool
}

ReplayReceipt identifies the committed resource. It deliberately has no response-body field: secrets must not enter a generic replay cache.

type ReplayRequest

type ReplayRequest struct {
	Scope, Key, Fingerprint string
	Now, Expires            time.Time
}

ReplayRequest identifies one protected effect. Scope, Key and Fingerprint are lowercase SHA-256 hex digests, never credentials, raw keys or request bodies. Scope includes the authenticated credential and canonical resource. Now and Expires use the caller's clock; replicas must have synchronized clocks.

func (ReplayRequest) Validate

func (r ReplayRequest) Validate() error

Validate checks the fixed storage bounds and a positive retention interval. A request out of bounds gives an error that wraps kernel.ErrInvalid.

type ReplayStore

type ReplayStore interface {
	// Do serializes concurrent calls for (Scope, Key), checks the fingerprint,
	// and runs effect only for a new/expired key. effect must use its supplied
	// context for all mutations and return a positive stable resource ID.
	// The receipt and mutations commit together; any error rolls both back.
	// A nested call joins the metadata store's outer transaction. Replays do
	// not call effect. Waiting obeys context cancellation; there is no
	// committed pending state to recover after a process crash. Callers must
	// reauthorize the returned resource, including on replay, before their
	// outer transaction commits. A request that Validate refuses, a nil
	// effect and a resource ID that is not positive are kernel.ErrInvalid; an
	// unexpired key with another fingerprint is ErrReplayMismatch.
	Do(
		ctx context.Context,
		request ReplayRequest,
		effect func(context.Context) (int64, error),
	) (ReplayReceipt, error)
	// Prune removes at most limit expired receipts (1..kernel.MaxLimit; any
	// other limit is kernel.ErrInvalid). Retention ends at expiry, whether or
	// not the physical row has been pruned.
	Prune(ctx context.Context, now time.Time, limit int) (int64, error)
}

ReplayStore deduplicates local transactional effects. It is not an outbox for external calls. An expired key may perform a new effect.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL