webhooks

package
v13.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 30, 2026 License: AGPL-3.0 Imports: 37 Imported by: 0

Documentation

Overview

Package webhooks delivers outbound webhooks: signed, retried, ordered, and replayable.

Its mirror is webhooks/inbound, which receives them: verify the provider's signature, publish, ack.

Everything here is a guarantee rather than an opinion. What an event means, when it fires, and what its payload contains are the application's; that a subscriber can authenticate the payload, that a delivery is not lost when the process dies, that a dead subscriber cannot starve a healthy one, and that resource.updated cannot overtake resource.created are this package's.

The two halves

Dispatcher is the write side. Dispatch takes the caller's transaction executor, resolves who is subscribed, and writes one dispatch row per subscriber inside that transaction:

err := client.WithTransaction(ctx, func(q database.Tx) error {
	if err := updateOrder(ctx, q, order); err != nil {
		return err
	}

	return dispatcher.Dispatch(ctx, q, &webhooks.Delivery{
		Scope:       tenancy.Of(order.AccountID),
		EventType:   OrderUpdated,
		OrderingKey: order.ID,
		Payload:     body,
	})
})

The deliveries commit with the state change that caused them, or not at all. There is no way to dispatch outside a transaction by accident: holding a SQLQueryExecutor from WithTransaction means you are already in one. This is the same seam outbox.Enqueue uses, for the same reason.

Worker is the delivery side. It claims due dispatches, signs and sends them, records every attempt, and schedules retries. It runs in its own process or goroutine and is started by Run, stopped by Close.

Why dispatches are rows and not queue messages

The unit of retry is one endpoint's copy of one event, and no broker can express that. A delivery that fanned out to five subscribers is not "failed" or "delivered" — four may have accepted it on the first attempt while the fifth is on its sixth retry. Retrying at the message level redelivers to the four that already succeeded; tracking state at the message level cannot represent the fifth's attempt count at all.

So per-endpoint state lives in a row: attempts, next_attempt, last_error, and a terminal dead flag, per (delivery, endpoint). Retry is a scheduled timestamp rather than a nack, which means it survives a worker restart, and "give up" is a state a row is in rather than a message that stopped being redelivered.

Signing

X-Platform-Signature: v1,t=1753900000,s=<hex(HMAC-SHA256(secret, "v1." + t + "." + body))>

The scheme is cryptography/requestsigning, and this package is one of its callers rather than its owner. Secret is requestsigning.Keyring, deliveries are signed with requestsigning.Sign, and a subscriber verifies them with requestsigning.Verify — the same functions guarding first-party service-to-service calls, over the same wire format. Read that package for why both the version and the timestamp are inside the signed material, and for what the Current/Previous pair buys.

What is worth repeating here is the receiving end. Verification is where these schemes are actually got wrong: subscribers compare with ==, skip the timestamp check, or verify a re-serialized body rather than the received bytes. Point subscribers at requestsigning.Verify rather than at the paragraph above — a scheme described in prose is a scheme reimplemented, and reimplemented verification is where the timing leak goes.

body, err := io.ReadAll(req.Body)   // the exact bytes, before any decoding
if err != nil {
	return err
}

err = requestsigning.Verify(secret, body, req.Header.Get(requestsigning.SignatureHeader))

A subscriber that is itself a platform service can skip even that and install requestsigning/http's middleware on the callback route.

Tenancy

An Endpoint belongs to somebody and a Delivery is somebody's event, and both say so with a tenancy.Scope. Fan-out is bounded by it: Dispatch resolves subscribers within the delivery's scope, so an endpoint registered by one account never receives another account's copy of the same event type.

err := dispatcher.Register(ctx, &webhooks.Endpoint{
	Scope:         tenancy.Of(accountID),
	URL:           "https://subscriber.example/hooks",
	Secret:        webhooks.Secret{Current: key},
	Subscriptions: webhooks.SubscribeTo(OrderUpdated),
})

An application whose events are global says tenancy.Global() in both places and gets what this package did before the dimension existed — Global is a scope like any other, matching only itself, and it is stored as the empty identifier that the scope columns default to.

There is no unscoped read. Every Store method that reaches an endpoint, a subscription, or a delivery takes a scope or carries one on the value it is given, the zero tenancy.Scope is not a scope, and a query that lost one fails at the driver rather than widening. The exceptions are the worker's own machinery — Claim, Backlog, and Reap span every scope, because one worker drains one queue for the whole deployment — and they say so.

What a scope is not is permission. Passing tenancy.Of(accountID) says these rows are that account's; whether the caller may act for that account is authorization's question, asked before this one.

Endpoints and subscriptions

An endpoint is a subscriber, and a subscription is one of the event types it wants. Subscriptions are rows of their own — identified, timestamped, and archivable one at a time — rather than a list of strings on the endpoint.

That is not bookkeeping. An application's own API is asked to retire one subscription: a user unticks order.created and leaves the other four alone. Against a flat list the only available answer is to rewrite the whole set, which cannot say when the subscription ended, has no identifier for the request that asked to name, and silently reverts whatever a concurrent edit of the same endpoint did. Against rows it is Unsubscribe on one ID:

sub, err := dispatcher.Subscribe(ctx, scope, endpointID, OrderShipped)
// ...
err = dispatcher.Unsubscribe(ctx, scope, sub.ID)

Register and SaveEndpoint still take the whole set, because registration names event types rather than subscription IDs — there are none yet — and SubscribeTo builds it. What a save does to the stored rows is reconcile rather than replace: one the endpoint already has keeps its identity and its creation time, one it no longer names is archived rather than deleted, and Endpoint.Subscriptions comes back filled with what is live.

Endpoint.EventTypes derives the flat list where one is wanted — a catalog check, a subscription UI's checkboxes. It is derived and not stored, so there is no second copy of the set to keep in step with the rows.

An endpoint also carries a Name, a CreatedBy, and the convention triple's timestamps. CreatedBy is a tenancy.Scope rather than a user ID string: "whose is this" already has a type here, and the principal who registered an endpoint is a finer-grained answer to it rather than a different question. Unlike Scope it is optional — an application that does not attribute endpoints to a person leaves it unset — and it is written once, with the row, because an endpoint does not change hands.

Ordering

Deliveries sharing an OrderingKey reach a given endpoint in dispatch order. The guarantee lives in the claim predicate: a keyed dispatch is claimable only when no earlier undelivered dispatch shares its key *and its endpoint*, so at most one is ever in flight per (endpoint, key) across the whole fleet.

The endpoint is in that tuple deliberately. Keyed on the ordering key alone, one subscriber timing out on resource-42.updated would hold back every other subscriber's copy of the same event — a dead endpoint stalling healthy ones, which is the failure per-endpoint circuit breaking exists to prevent, reintroduced in the claim predicate.

Deliveries with no ordering key are unordered and claim freely.

Delivery semantics

At-least-once, and it cannot be otherwise: the subscriber and the database have no shared commit, so a crash between a 200 response and the row update redelivers on restart. Subscribers must tolerate duplicates, and DeliveryIDHeader is the key to deduplicate on — it is stable across every attempt and every replay of one delivery.

Failures back off exponentially with full jitter via retrycfg.DelayFor, persisted as a timestamp so the schedule survives a restart. Past Backoff.MaxAttempts the dispatch is marked dead: skipped by every future claim, counted, and left for an operator to replay. Without that terminal state one permanently broken subscriber blocks its ordering key forever.

Some failures skip the budget entirely. A 4xx other than 408 or 429 means the subscriber understood and refused, and a URL that no longer passes CheckEndpointURL will not start passing; both are marked retry.Unretryable and go straight to dead rather than spending twenty-five attempts proving it.

Circuit breaking is the other direction. A short-circuited delivery is a failure but is explicitly *not* charged an attempt — an endpoint down for an hour would otherwise exhaust the budget of every delivery queued behind it, and they would all need replaying by hand once it recovered.

SSRF

An endpoint URL is attacker-supplied and this package makes authenticated requests to it, which is textbook server-side request forgery: point it at 169.254.169.254 and the worker fetches cloud credentials on the attacker's behalf. CheckEndpointURL enforces https and refuses any host resolving into loopback, link-local, private, or non-global space — at registration, where the rejection can be reported to whoever submitted it, and again at delivery, because DNS is mutable.

Redirects are refused rather than followed. Following one would deliver a signed payload to a host that was never registered and never checked, turning an open redirect on any subscriber's domain into a way to point the worker anywhere.

Persistence

Store is the seam. This package ships a SQL implementation (NewSQLStore) and the DDL it needs (webhooks/migrations), so adopting webhooks does not mean writing one — but an application with its own schema conventions can implement the interface instead of forking the package.

A deployment whose tables predate subscriptions being rows migrates with migrations.UpgradeSQL, which adds the columns and backfills the existing flat subscription set. Read that package's Upgrading section first: unlike the create-only DDL, it is one-shot.

The five tables are rendered from one prefix rather than five names. They reference each other by foreign key and the queries join across them, so a consumer who could name them independently could also name them inconsistently, and nothing would catch it until the first dispatch.

Where the SQL comes from

Every statement the SQL store executes is in a canonical .sql that sqlc checks against this package's own schema, on all three dialects, and is executed through the querier sqlc-gen-unison emits from it. A column renamed in a migration is a failed `make generate` with no database running, where it used to be a scan error on whichever dialect a container test happened to reach first.

The schema's facts — the five table names, each table's columns in projection order, the two subsets a write may assign — are spelled once, in webhooks/internal/queries. `make generate` renders most of the corpus from them through database/querygen; eleven statements are written out there in full, because they are the shapes querygen deliberately does not render — a self-join, a three-table projection, a bounded delete, an incrementing counter, an aggregate, a scope reached through another table, and the two creation instants that are the emitting transaction's rather than the row's. That package's comment says which is which and why, and its tests pin the committed .sql byte for byte against the renderer, so "the SQL sqlc checks is the SQL the store runs" is a fact a test states rather than a property of a pipeline taken on trust.

One consequence reaches a caller. The store holds no clock: the convention columns are stamped by the statements themselves, from the database server, and the instants that are a caller's to choose — when a delivery was enqueued, how long a lease runs, how far back a reap goes — are arguments as they always were. A row's created_at and the filter window compared against it now come from one clock rather than from however many application instances are writing.

The catalog

Subscribable event types are supplied at construction via WithCatalog, not stored. What an event means is an application opinion and this package has none.

Both Register and Dispatch reject a type outside the catalog. That matters because an event type is a string underneath and string literals are typo-prone: a subscription to "reciped.created" accepted silently produces an endpoint that never fires, and diagnosing it means noticing an absence.

Declare the event types as EventType constants and key the catalog by them:

const (
    OrderCreated webhooks.EventType = "order.created"
    OrderUpdated webhooks.EventType = "order.updated"
)

webhooks.WithCatalog(webhooks.Catalog{
    OrderCreated: {Description: "an order was created"},
    OrderUpdated: {Description: "an order was updated"},
})

EventType is a defined type rather than a string so that this form is available, and it is a defined type rather than an alias because an alias would be indistinguishable from string to a type checker, which is the whole point of having one.

The point is the catalog's second copy. The catalog must list every event type the application publishes — a missing entry fails the dispatch gate, and where Dispatch runs inside the write transaction that is a failed write rather than a missing webhook — so an application of any size ends up deriving it rather than maintaining it beside the constants that are its source of truth.

Deriving it means answering "which of these constants are event types", and there are two ways to answer. By name, matching a suffix: a convention nothing enforces, where a constant spelled differently is silently not an event type and the miss surfaces as a failed dispatch rather than a failed build. Or by declared type, which the compiler already knows, cannot be spelled wrong, and does not care which package or which directory the constant was declared in.

This package ships no generator. Which constants exist, where they live, and where the descriptions come from are the application's business, and a scan over them is a few dozen lines once the type makes the question answerable. What webhooks owes that scan is a declaration form it can rely on, which is this one.

Nothing here constrains the string itself. Dots, colons, and underscores are all fine, and the catalog remains the authority on which values exist.

Watching it

The two that matter most are webhooks_backlog_depth and webhooks_backlog_age_seconds, sampled on the reap tick. Every other instrument is a rate or a latency, and none of them separates "delivering steadily" from "delivering steadily while falling further behind" — only the age does.

The rest: webhooks_deliveries_dispatched against webhooks_deliveries_sent (the gap is the rollback rate), webhooks_deliveries_failed, webhooks_deliveries_short_circuited, and webhooks_deliveries_dead — alert on any increase in the last, since a dead dispatch is an event a subscriber will never see — plus webhooks_claim_errors, webhooks_dispatches_reaped, and the webhooks_delivery_latency_ms, webhooks_cycle_latency_ms, and webhooks_claimed_batch_size distributions.

Per-delivery measurements carry an endpoint attribute, because one worker serves every subscriber and a single broken one is invisible in the total. That attribute's cardinality grows with the endpoints table; an operator with enough subscribers to care should drop it in their collector rather than lose the distinction at the source.

Spans cover Dispatch, each claim, and each delivery. A cycle that claims nothing is not traced: a root span every poll interval is noise.

Where this package stops

Endpoint management is a store and no handlers: the routes a subscriber uses to register a URL, rotate its secret or retire it are an application's, over an application's types and its own authorization. Receiving somebody else's callbacks is the other case entirely, and webhooks/inbound does ship an http.Handler for it — the shape of a Stripe or GitHub payload is Stripe's or GitHub's, and no application has a say in it. The module README's "Stores and Transports" section is where that distinction is drawn for the module as a whole.

Example (VerifyingADelivery)

What a subscriber does on receipt. The scheme lives in cryptography/requestsigning, which this package signs through; a subscriber verifies through the same package, so the two halves cannot drift.

package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"strings"
	"time"

	"github.com/primandproper/platform-go/v13/cryptography/requestsigning"
	"github.com/primandproper/platform-go/v13/webhooks"
)

func main() {
	secret := webhooks.Secret{Current: []byte("the shared signing key")}

	subscriber := httptest.NewServer(http.HandlerFunc(func(res http.ResponseWriter, req *http.Request) {
		// The exact bytes received, read before any decoding. Decoding and
		// re-encoding changes key order and whitespace, and the signature covers
		// bytes rather than meaning.
		body, err := io.ReadAll(req.Body)
		if err != nil {
			res.WriteHeader(http.StatusBadRequest)

			return
		}

		if err = requestsigning.Verify(secret, body, req.Header.Get(requestsigning.SignatureHeader)); err != nil {
			res.WriteHeader(http.StatusUnauthorized)

			return
		}

		// The delivery ID is stable across every retry and replay of one
		// delivery, so it is the key to deduplicate on.
		_ = req.Header.Get(webhooks.DeliveryIDHeader)

		res.WriteHeader(http.StatusNoContent)
	}))
	defer subscriber.Close()

	payload := []byte(`{"id":"order-7"}`)

	signature, err := requestsigning.Sign(secret, payload, time.Now())
	if err != nil {
		panic(err)
	}

	req, err := http.NewRequestWithContext(context.Background(),
		http.MethodPost, subscriber.URL, strings.NewReader(string(payload)))
	if err != nil {
		panic(err)
	}

	req.Header.Set(requestsigning.SignatureHeader, signature)

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer func() { _ = res.Body.Close() }()

	fmt.Println(res.StatusCode)

	// A tampered body no longer verifies.
	fmt.Println(requestsigning.Verify(secret, []byte(`{"id":"order-8"}`), signature))

}
Output:
204
invalid request signature

Index

Examples

Constants

View Source
const (
	// DefaultBatchSize is how many dispatches one cycle claims.
	DefaultBatchSize = 100
	// DefaultConcurrency is how many deliveries run at once within a batch.
	DefaultConcurrency = 16
	// DefaultPollInterval is how often the worker looks for work.
	DefaultPollInterval = time.Second
	// DefaultLeaseDuration is how long a claim is held before another worker
	// may reclaim the dispatch. It must comfortably exceed RequestTimeout, or
	// two workers will deliver the same payload concurrently.
	DefaultLeaseDuration = 60 * time.Second
	// DefaultRequestTimeout bounds one delivery request.
	DefaultRequestTimeout = 10 * time.Second
	// DefaultCircuitOpenRetryDelay is how long a dispatch waits after being
	// short-circuited.
	DefaultCircuitOpenRetryDelay = 30 * time.Second
	// DefaultRetention is how long delivered dispatches and their attempts are
	// kept before reaping.
	DefaultRetention = 7 * 24 * time.Hour
	// DefaultReapInterval is how often the reaper runs.
	DefaultReapInterval = 5 * time.Minute
	// DefaultReapBatchSize caps one reap, so a large backlog is removed over
	// several passes instead of one long-running DELETE.
	DefaultReapBatchSize = 1000
	// DefaultUserAgent identifies deliveries to subscribers.
	DefaultUserAgent = "platform-go-webhooks/1"
)
View Source
const (
	// EventTypeHeader carries the delivery's event type, so a subscriber can
	// route without parsing the body.
	EventTypeHeader = "X-Platform-Event"
	// DeliveryIDHeader carries the delivery ID, which is stable across retries
	// of the same delivery and is therefore the subscriber's deduplication key.
	DeliveryIDHeader = "X-Platform-Delivery"
	// AttemptHeader carries which attempt this is, 1-indexed, so a subscriber
	// can tell a first delivery from a redelivery.
	AttemptHeader = "X-Platform-Attempt"
)

The headers this package sets that are specific to a webhook delivery. The signature and timestamp headers are not among them: those belong to the signing scheme rather than to webhooks, and live in requestsigning.SignatureHeader and requestsigning.TimestampHeader.

View Source
const DefaultContentType = "application/json"

DefaultContentType is the Content-Type deliveries carry when an Endpoint does not set one.

View Source
const DefaultTablePrefix = ""

DefaultTablePrefix is the namespace the webhooks tables carry when none is configured, which is none — rendering webhooks_endpoints and its four siblings.

The webhooks_ segment is the schema's, not the caller's: a table always says which package created it. Setting a namespace of "ddb" renders ddb_webhooks_endpoints, for a database shared between applications. A namespace must not end in '_'; database/ddl supplies the separator.

View Source
const EndpointAttributeKey = endpointIDKey

EndpointAttributeKey is the attribute name this package labels per-endpoint measurements with.

It is exported so that instruments configured alongside a Worker — a circuit breaker's own counters, most obviously — can tag themselves identically. An endpoint that keeps tripping its breaker and an endpoint whose deliveries keep failing are the same endpoint, and a dashboard can only join those two series if both agree on the label.

Variables

View Source
var (
	// ErrInvalidEndpointURL indicates a URL that is unparseable, not absolute,
	// or not https.
	ErrInvalidEndpointURL = platformerrors.New("invalid webhook endpoint URL")

	// ErrDisallowedEndpointHost indicates a URL whose host resolves somewhere a
	// webhook must not reach — loopback, link-local, or private address space.
	// See CheckEndpointURL for why this is enforced at registration.
	ErrDisallowedEndpointHost = platformerrors.New("webhook endpoint host is not publicly routable")

	// ErrReservedHeader indicates an Endpoint whose static headers would
	// overwrite one this package sets.
	ErrReservedHeader = platformerrors.New("webhook endpoint sets a reserved header")

	// ErrNoEvents indicates an endpoint subscribing to nothing, which is never
	// what the registrant meant.
	//
	// It is raised for an endpoint whose every subscription is archived as well as
	// for one that named none: both describe a subscriber that will never receive
	// anything, and registering one is a mistake rather than a way to park an
	// endpoint. Disabled is how an endpoint is parked.
	ErrNoEvents = platformerrors.New("webhook endpoint subscribes to no events")

	// ErrUnknownSubscription indicates a subscription ID that names nothing in
	// the scope that asked — including one that exists under another scope's
	// endpoint, which is the same answer as far as this scope is concerned.
	ErrUnknownSubscription = platformerrors.New("unknown webhook subscription")

	// ErrEmptyEventType indicates a subscription carrying no event type. It is
	// distinct from ErrUnknownEventType: nothing was named at all, so no catalog
	// lookup could have failed.
	ErrEmptyEventType = platformerrors.Wrap(platformerrors.ErrEmptyInputParameter, "webhook subscription names no event type")
)
View Source
var (
	// ErrUnknownEventType indicates an event type absent from the Catalog. It is
	// returned both when registering an endpoint that subscribes to it and when
	// dispatching it, so a typo cannot reach the wire from either direction.
	ErrUnknownEventType = platformerrors.New("unknown webhook event type")

	// ErrNoSigningSecret indicates an endpoint with no current signing secret.
	// Unsigned delivery is not an option this package offers: a subscriber that
	// cannot authenticate a payload cannot safely act on it.
	//
	// It is requestsigning's own sentinel rather than one of this package's, so
	// that an endpoint rejected at registration and a delivery that failed to
	// sign report the same error — they are the same condition, found at two
	// different moments.
	ErrNoSigningSecret = requestsigning.ErrNoSigningKey

	// ErrNoScope indicates an endpoint or a delivery that does not say whose it
	// is, or a store read that was not told whose rows it wanted. Every read and
	// write carrying consumer data takes a scope, and the zero tenancy.Scope is
	// not one — see the Tenancy section of this package's documentation.
	//
	// It is tenancy's own sentinel rather than one of this package's, so that an
	// endpoint rejected at registration, a delivery rejected at dispatch, and a
	// query refused at the driver all report the same condition, found at three
	// different moments.
	ErrNoScope = tenancy.ErrNoScope

	// ErrEndpointOutOfScope indicates a save naming an endpoint ID that is
	// already registered in a different scope. An endpoint does not change hands,
	// so this is a collision rather than a move — and accepting it would rewrite
	// another subscriber's URL and signing secret.
	//
	// It is distinct from "not found", which is what a read in the wrong scope
	// gets: a read has no business learning that the ID exists elsewhere, whereas
	// a write must not be told its endpoint was saved when it was not.
	ErrEndpointOutOfScope = platformerrors.New("webhook endpoint ID is registered in another scope")

	// ErrEndpointDisabled indicates a Replay targeting an endpoint that is
	// disabled. Dispatch skips disabled endpoints silently — that is what
	// disabling means — but an operator naming one explicitly is told why
	// nothing happened.
	ErrEndpointDisabled = platformerrors.New("webhook endpoint is disabled")

	// ErrDeliveryNotFound indicates a Replay naming a delivery/endpoint pair
	// that was never dispatched, or has since been reaped.
	ErrDeliveryNotFound = platformerrors.New("webhook delivery not found")

	// ErrNilStore indicates a nil Store. It wraps errors.ErrNilInputParameter,
	// so a caller may check either.
	ErrNilStore = platformerrors.Wrap(platformerrors.ErrNilInputParameter, "nil webhook store")

	// ErrNilExecutor indicates Dispatch was called without a query executor. It
	// wraps errors.ErrNilInputParameter, so a caller may check either.
	ErrNilExecutor = platformerrors.Wrap(platformerrors.ErrNilInputParameter, "nil query executor")

	// ErrNilDelivery indicates Dispatch was called with no Delivery.
	ErrNilDelivery = platformerrors.Wrap(platformerrors.ErrNilInputParameter, "nil webhook delivery")

	// ErrNilEndpoint indicates a nil Endpoint was passed for registration.
	ErrNilEndpoint = platformerrors.Wrap(platformerrors.ErrNilInputParameter, "nil webhook endpoint")
)
View Source
var ErrCircuitOpen = platformerrors.Wrap(circuitbreaking.ErrCircuitBroken, "webhook endpoint circuit is open")

ErrCircuitOpen indicates a delivery skipped because the endpoint's circuit breaker is open. It is a failure for retry purposes but is deliberately not counted against the attempt budget — see deliver.

View Source
var ErrLeaseTooShort = platformerrors.New("webhooks lease duration must exceed the request timeout")

ErrLeaseTooShort indicates a lease that does not outlast a request, which would let two workers deliver the same payload concurrently.

View Source
var ErrNilDatabaseClient = platformerrors.Wrap(platformerrors.ErrNilInputParameter, "nil database client")

ErrNilDatabaseClient indicates a nil database.Client. It wraps errors.ErrNilInputParameter, so a caller may check either.

View Source
var ErrNonSuccessStatus = platformerrors.New("webhook endpoint returned a non-success status")

ErrNonSuccessStatus indicates a subscriber that answered with something other than 2xx.

Functions

func CheckEndpointURL

func CheckEndpointURL(ctx context.Context, rawURL string) error

CheckEndpointURL reports whether u is acceptable as a delivery target.

This is SSRF prevention, and it is worth being explicit about what it does and does not buy. A webhook endpoint is a URL supplied by a user that the server will then make authenticated requests to, which is the textbook shape of a server-side request forgery: point it at 169.254.169.254 and the delivery worker fetches cloud instance credentials on the attacker's behalf, or point it at an internal admin service and the worker reaches something the attacker cannot.

So: https only, and no host that resolves into loopback, link-local, private, or otherwise non-global address space.

The check runs at registration, where a rejection can be reported to whoever submitted the URL, and again at delivery, because registration alone is not sound: DNS is mutable, and a name that resolved publicly when it was registered can resolve to 127.0.0.1 by the time the worker dials it.

What this does not close is DNS rebinding. Resolution and connection are separate steps, and an attacker controlling the authoritative server can return a public address to this lookup and a private one to the dial moments later. Closing that needs the checked IP pinned into the dial itself — a custom DialContext that resolves once and refuses anything else — which this package does not do, because it would mean owning the transport rather than accepting the caller's. Deployments where that gap matters should supply a pinning transport via WithHTTPClient; this function raises the cost without claiming to eliminate it.

Types

type Attempt

type Attempt struct {
	// CreatedAt is when the request was issued, which is the instant the log
	// line was written.
	CreatedAt time.Time `json:"createdAt"`
	// ID identifies the attempt.
	ID string `json:"id"`
	// DeliveryID is the delivery this attempted.
	DeliveryID string `json:"deliveryID"`
	// EndpointID is the endpoint it was sent to.
	EndpointID string `json:"endpointID"`
	// Error is the transport or status error, rendered. Empty on success. It is
	// a string because it is stored and read by a human, not re-wrapped.
	Error string `json:"error,omitempty"`
	// Duration is how long the request took.
	Duration time.Duration `json:"duration"`
	// StatusCode is the response status, or 0 if no response was received.
	StatusCode int `json:"statusCode"`
	// AttemptCount is which attempt this was, 1-indexed.
	AttemptCount int `json:"attemptCount"`
}

Attempt is one recorded HTTP attempt against one endpoint. Attempts are append-only and are the delivery log: what was tried, when, and what came back.

func (*Attempt) Succeeded

func (a *Attempt) Succeeded() bool

Succeeded reports whether the attempt was accepted by the subscriber.

type Catalog

type Catalog map[EventType]EventDefinition

Catalog is the set of event types an application publishes, keyed by event type. It is supplied at construction rather than stored, because what an event means is an application opinion and the library has none.

Subscribing to an event outside the catalog is rejected at registration, and dispatching one is rejected at Dispatch. Both matter: an event type is a string underneath, string literals are typo-prone, and a subscription to "reciped.created" that is accepted silently produces an endpoint that never fires and no signal explaining why. Declaring the event types as EventType constants and keying the catalog by those constants moves that check to compile time for everything except the catalog's own literals.

func (Catalog) EventTypes

func (c Catalog) EventTypes() []EventType

EventTypes returns the catalog's event types, sorted, for rendering a subscription UI or an API response.

func (Catalog) Known

func (c Catalog) Known(eventType EventType) bool

Known reports whether eventType is in the catalog.

type CircuitBreakerFactory

type CircuitBreakerFactory func(endpointID string) (circuitbreaking.CircuitBreaker, error)

CircuitBreakerFactory builds the circuit breaker guarding one endpoint. It is called at most once per endpoint per worker, lazily, and the result is retained.

It is a factory rather than a fixed map because endpoints are registered at runtime — the set is not known when the worker is constructed, so partitioned.KeyedCircuitBreaker's operator-chosen key list does not fit. Cardinality is bounded by the endpoints table, which is bounded by however many subscribers an operator has accepted.

type ClaimedDispatch

type ClaimedDispatch struct {
	// Endpoint is the subscriber, resolved at claim time rather than at
	// dispatch time — so a secret rotated between the event and its delivery
	// signs with the current key, and an endpoint disabled in between is not
	// delivered to at all.
	Endpoint *Endpoint `json:"endpoint"`
	// Payload is the delivery body, verbatim as dispatched.
	Payload []byte `json:"payload"`
	// Scope is the delivery's scope, read back from the delivery row.
	//
	// The worker does not filter on it — it delivers whatever it claimed, to the
	// endpoint the dispatch names, and the fan-out that produced the row already
	// resolved subscribers within this scope. It is here so that a delivery
	// failure, a dead dispatch, and a slow subscriber are attributable to a
	// tenant in the worker's logs and spans, which is otherwise the one place in
	// the pipeline where whose event it was has been forgotten.
	Scope tenancy.Scope `json:"scope"`
	// EventType is the delivery's event type.
	EventType EventType `json:"eventType"`

	Dispatch
}

ClaimedDispatch is a Dispatch the worker has leased, joined with everything needed to actually issue the request. It is assembled by the Store so the worker makes one round trip per batch rather than one per dispatch.

type Delivery

type Delivery struct {
	// Scope is whose event this is, and it bounds the fan-out: Dispatch resolves
	// subscribers within it, so an endpoint registered by one account never
	// receives another account's copy of the same event type.
	//
	// Required. An unset scope is refused rather than read as "every
	// subscriber", because the reading that makes a missing filter convenient is
	// the one that leaks a tenant's payload to every other tenant. An
	// application whose events are global says tenancy.Global().
	Scope tenancy.Scope `json:"scope"`
	// ID identifies the delivery, and is what Replay names. Generated when empty.
	ID string `json:"id"`
	// EventType is the catalog event type. Must be in the Catalog.
	EventType EventType `json:"eventType"`
	// OrderingKey groups deliveries that must arrive in order — typically the
	// subject resource's ID. Deliveries sharing a key reach a given endpoint in
	// the order they were dispatched; deliveries with different keys, or with
	// none, are unordered relative to each other.
	//
	// Ordering is per endpoint as well as per key. A subscriber that is timing
	// out delays only its own queue for that key, never another subscriber's.
	OrderingKey string `json:"orderingKey,omitempty"`
	// Payload is the event body, delivered to every subscriber byte for byte as
	// supplied. It is json.RawMessage rather than any so that what is signed and
	// what is sent are the same bytes — re-marshaling between signing and
	// sending is exactly how a signature comes to cover something other than the
	// request body.
	Payload json.RawMessage `json:"payload"`
}

Delivery is one event to fan out. It is the application's unit; the per-endpoint unit it expands into is a dispatch, which callers do not construct.

type Dispatch

type Dispatch struct {
	// NextAttempt is when this dispatch next becomes claimable.
	NextAttempt time.Time `json:"nextAttempt"`
	// ID identifies the dispatch.
	ID string `json:"id"`
	// DeliveryID is the delivery being sent.
	DeliveryID string `json:"deliveryID"`
	// EndpointID is the subscriber it is being sent to.
	EndpointID string `json:"endpointID"`
	// OrderingKey is denormalized from the delivery so the claim predicate can
	// enforce ordering without joining. See the claim in
	// webhooks/internal/queries, which is where the guarantee is written down.
	OrderingKey string `json:"orderingKey,omitempty"`
	// LastError is the most recent failure, rendered.
	LastError string `json:"lastError,omitempty"`
	// Attempts is how many attempts have been made.
	Attempts int `json:"attempts"`
	// Dead marks a dispatch that exhausted its attempts. It is skipped by every
	// future claim and is what an operator replays.
	Dead bool `json:"dead"`
}

Dispatch is one endpoint's copy of one delivery: the unit the worker actually retries, backs off, and gives up on.

It exists as a distinct row from the Delivery because per-endpoint state is the whole point. A delivery that fanned out to five subscribers is not "failed" or "delivered" — four may have accepted it on the first attempt while the fifth is on its sixth retry, and a single status on the delivery cannot express that. Retrying at the delivery level would also redeliver to the four that already accepted it.

type Dispatcher

type Dispatcher interface {
	// Dispatch fans an event out to every endpoint in the delivery's scope that is
	// subscribed to it, writing through the caller's transaction so the deliveries
	// commit with the state change that caused them.
	Dispatch(ctx context.Context, q database.Tx, delivery *Delivery) error
	// Replay re-drives a specific past delivery to a specific one of the scope's
	// endpoints, for operator recovery.
	Replay(ctx context.Context, scope tenancy.Scope, deliveryID, endpointID string) error
	// Register validates and stores an endpoint, under the scope the endpoint
	// carries. Validation is not optional and not separable: an unvalidated
	// endpoint is an SSRF target.
	Register(ctx context.Context, endpoint *Endpoint) error
	// Subscribe adds one event type to one of the scope's endpoints, gating on the
	// catalog, and returns the subscription.
	Subscribe(ctx context.Context, scope tenancy.Scope, endpointID string, eventType EventType) (*Subscription, error)
	// Unsubscribe retires one of the scope's subscriptions by ID, leaving the
	// endpoint and its other subscriptions alone.
	Unsubscribe(ctx context.Context, scope tenancy.Scope, subscriptionID string) error
}

Dispatcher is the write side: it turns an application event into per-endpoint work, and re-drives that work when an operator asks.

type DispatcherOption

type DispatcherOption func(*StoreDispatcher)

DispatcherOption configures a Dispatcher.

func WithCatalog

func WithCatalog(catalog Catalog) DispatcherOption

WithCatalog supplies the set of event types the application publishes.

Without it every event type is unknown and both Register and Dispatch reject everything, which is deliberate: a catalog-free dispatcher would accept subscriptions to typo'd event types that then never fire, and diagnosing that means noticing an absence.

func WithDispatcherClock

func WithDispatcherClock(c clock.Clock) DispatcherOption

WithDispatcherClock swaps the clock stamping deliveries.

func WithDispatcherLogger

func WithDispatcherLogger(logger logging.Logger) DispatcherOption

WithDispatcherLogger attaches a logger.

func WithDispatcherMetricsProvider

func WithDispatcherMetricsProvider(metricsProvider metrics.Provider) DispatcherOption

WithDispatcherMetricsProvider attaches a metrics provider. Pair it with the Worker's: dispatch rate against delivery rate is what says whether the worker is keeping up, and neither number answers that alone.

func WithDispatcherTracerProvider

func WithDispatcherTracerProvider(tracerProvider tracing.Provider) DispatcherOption

WithDispatcherTracerProvider attaches a tracer provider, so a Dispatch shows up as a child of the span that owns the transaction.

func WithDispatcherURLChecker

func WithDispatcherURLChecker(checker URLChecker) DispatcherOption

WithDispatcherURLChecker replaces the URL policy Register enforces.

Pair it with the Worker's WithWorkerURLChecker: an endpoint accepted at registration and refused at delivery sits in the backlog until it dies, so the two halves must agree. See URLChecker for what replacing it costs.

type Endpoint

type Endpoint struct {
	// CreatedAt is when the endpoint was registered, stamped by the Store.
	CreatedAt time.Time `json:"createdAt"`
	// LastUpdatedAt is when the endpoint was last re-registered.
	LastUpdatedAt *time.Time `json:"lastUpdatedAt"`
	// ArchivedAt is when the endpoint was retired by ArchiveEndpoint. The row is
	// kept rather than deleted: the attempts log outlives the endpoint, because
	// "what did we send them" is asked most often after someone has been removed.
	//
	// Re-registering an archived endpoint clears it — SaveEndpoint is an upsert,
	// and a caller who registers an ID that exists means to have it live.
	ArchivedAt *time.Time `json:"archivedAt"`
	// Headers are static headers added to every request to this endpoint, for
	// subscribers that need a routing token or a tenant hint. The signature,
	// timestamp, content type, and event headers this package sets are not
	// overridable from here — see reservedHeaders.
	Headers map[string]string `json:"headers,omitempty"`
	// Scope is whose endpoint this is. Required, and rejected at registration
	// when unset: an endpoint that belongs to nobody in particular is one an
	// application with tenants registered by accident, and it would receive
	// deliveries the account it was meant for never sees.
	//
	// An application whose events are global says tenancy.Global() here, which
	// is a scope like any other and matches only deliveries dispatched in it.
	Scope tenancy.Scope `json:"scope"`
	// CreatedBy names the principal that registered the endpoint — the user who
	// clicked the button, where an application tracks that.
	//
	// It is a tenancy.Scope rather than a bare user ID on purpose. "Whose is
	// this" already has a type in this module, and the answer here is finer
	// grained than Scope rather than a different kind of answer: an application
	// that lists "the endpoints I created" filters on the same value type it
	// filters tenants by, and a second string field would be a parallel owner
	// dimension that nothing else in the module understands.
	//
	// Unlike Scope it is optional. An endpoint belongs to its Scope, which is
	// what bounds fan-out and every read; who registered it is provenance, and an
	// application with no per-user attribution leaves this unset. It is written
	// once, at registration, and is not part of what a re-registration updates —
	// an endpoint does not change hands, and neither does its provenance.
	CreatedBy tenancy.Scope `json:"createdBy"`
	// ID identifies the endpoint. Generated at registration when empty.
	ID string `json:"id"`
	// Name is the label an endpoint-management UI shows. Optional, and free-form:
	// this package never reads it, and two endpoints in one scope may share one.
	Name string `json:"name"`
	// URL is the absolute https:// URL deliveries are POSTed to.
	URL string `json:"url"`
	// ContentType is the request's Content-Type. Defaults to application/json.
	ContentType string `json:"contentType"`
	// Secret carries the signing keys. Never serialized: an endpoint travels
	// through API responses and logs, and its secret must not.
	Secret Secret `json:"-"`
	// Subscriptions are the endpoint's live subscriptions — one identified,
	// individually archivable row per catalog event type it wants.
	//
	// On the way in it is the set to register: SubscribeTo builds one from event
	// types, and SaveEndpoint reconciles the stored rows against it, so an entry
	// that is not there any more is archived rather than deleted. On the way out
	// the Store fills it with the live rows, IDs and timestamps included.
	//
	// It is rows rather than a []EventType because a subscription is something an
	// application's own API lets a user retire one of, and a flat list can only be
	// rewritten: "unsubscribe from order.created" becomes "here is the list
	// again, minus one", which loses when it happened, and there is no identifier
	// to name in the request that asked for it. EventTypes derives the flat form
	// where one is wanted, so nothing has to keep two copies agreeing.
	Subscriptions []Subscription `json:"subscriptions"`
	// Disabled stops delivery without deleting the endpoint or its history,
	// which is what an operator wants when a subscriber is misbehaving.
	Disabled bool `json:"disabled"`
}

Endpoint is one subscriber: whose it is, where deliveries go, what they are signed with, and which events reach it.

func (*Endpoint) Archived

func (e *Endpoint) Archived() bool

Archived reports whether the endpoint has been retired.

func (*Endpoint) EnsureDefaults

func (e *Endpoint) EnsureDefaults()

EnsureDefaults fills an Endpoint's optional fields.

func (*Endpoint) EventTypes

func (e *Endpoint) EventTypes() []EventType

EventTypes returns the event types the endpoint's live subscriptions name, in the order the subscriptions are held.

It is derived rather than stored. The flat list is still the useful shape for a Catalog check or a subscription UI's checkboxes, and keeping a second copy of it on the struct would be a copy that can disagree with the rows.

func (*Endpoint) Validate

func (e *Endpoint) Validate(ctx context.Context, catalog Catalog, checkURL URLChecker) error

Validate checks an Endpoint against the catalog it is being registered into and the URL policy it will be delivered under.

The catalog is an argument because an endpoint is only meaningful relative to the set of events an application publishes: a subscription to an event that does not exist is a silent no-op forever. checkURL is an argument so that registration and delivery cannot apply different policies — an endpoint accepted here and refused by the worker would sit in the backlog until it died. A nil checkURL means CheckEndpointURL.

The scope is checked here, with the other invariants, rather than being left to the store: an endpoint that says nothing about whose it is is one an application with tenants registered by accident, and the account it was meant for would never see a delivery. Say tenancy.Global() to mean it.

type EventDefinition

type EventDefinition struct {
	// Description is human-facing prose explaining when the event fires.
	Description string `json:"description"`
}

EventDefinition describes one subscribable event type. It is deliberately thin: the library needs to know an event type exists in order to reject a subscription to one that does not, and needs nothing else about it.

Description is what an endpoint-management UI shows beside the checkbox, and the reason this is a struct rather than a set — a bare set would push every consumer into maintaining that text somewhere else, out of step with the events themselves.

type EventType

type EventType string

EventType names one kind of event an application publishes. It is the string that travels in EventTypeHeader, the key a Catalog is keyed by, and what an Endpoint subscribes to.

It is a defined type rather than a string so that an application's event types are declarable in a form both a reader and a type checker recognize:

const OrderCreated webhooks.EventType = "order.created"

The type is what makes the set of them discoverable. A catalog has to list every event type an application publishes — a missing entry fails the dispatch gate — and keeping that list by hand beside the constants that are its source of truth is what makes it drift. Derived instead, the question "which constants are event types" has to be answerable, and answering it by matching on the constant's *name* means a convention nothing enforces: a declaration that spells the name differently is silently not an event type, and the miss surfaces as a failed dispatch rather than as a failed build. Declared type is a fact the compiler already holds and no one can spell wrong.

It is deliberately not an alias. An alias is indistinguishable from string to a type checker, which would leave the set exactly as undiscoverable as it was.

Nothing here constrains the format. Dots, colons, and underscores are all fine; the Catalog is the authority on which ones exist, and this package has no opinion beyond that.

func (EventType) String

func (e EventType) String() string

String returns the event type as a plain string.

It exists for the observability seams that take an any and switch on its type: a defined string type is neither string nor fmt.Stringer to that switch and falls through to a reflective default, which records the same text by a slower path that nothing would flag if it stopped matching. Spelling the conversion at those call sites keeps what is recorded a decision rather than a fallback.

type SQLStore

type SQLStore struct {
	// contains filtered or unexported fields
}

SQLStore is the SQL-backed Store, against the schema webhooks/migrations renders. It is exported, and returned by NewSQLStore, so a caller who has chosen SQL storage can depend on that choice rather than on the Store seam every backing shares.

func NewSQLStore

func NewSQLStore(client database.Client, opts ...SQLStoreOption) (*SQLStore, error)

NewSQLStore builds a Store over the given database.

The dialect comes from the client, so the two cannot disagree. The prefix must still match the one the migrations were rendered with — nothing here can check that, and a mismatch surfaces as a missing table on the first query rather than at construction.

Observability is optional and defaults to nothing: an unconfigured store logs to a noop logger and traces to a noop provider.

It takes no clock. Every timestamp this store writes is the database server's — the convention columns are stamped by the statements themselves, and the instants a caller cares about (when a delivery was enqueued, when a dispatch was leased, how far back a reap goes) are arguments rather than clock reads. That is database/querygen's rule rather than a preference taken here: a row's created_at and the filter window compared against it have to come from one clock, or two application instances a second apart write rows a window excludes at random.

func (*SQLStore) AddSubscription

func (s *SQLStore) AddSubscription(ctx context.Context, scope tenancy.Scope, endpointID string, eventType EventType) (*Subscription, error)

AddSubscription subscribes one of the scope's endpoints to eventType.

It is an upsert and a read rather than an insert, because a subscription is identified by the (endpoint, event type) pair: subscribing to something the endpoint already subscribes to is not an error, and re-subscribing to something it archived revives that row rather than minting a second one for the same pair. Both cases return the row that is now live, which is why the write is followed by a read — a revived row keeps the ID it already had.

All three statements run in one transaction, so the row read back is the row written and not one a concurrent archive has since retired.

func (*SQLStore) ArchiveEndpoint

func (s *SQLStore) ArchiveEndpoint(ctx context.Context, scope tenancy.Scope, endpointID string) error

ArchiveEndpoint retires one of the scope's endpoints. An endpoint in another scope is not touched.

func (*SQLStore) ArchiveSubscription

func (s *SQLStore) ArchiveSubscription(ctx context.Context, scope tenancy.Scope, subscriptionID string) error

ArchiveSubscription retires one of the scope's subscriptions. A subscription under another scope's endpoint is not touched.

Like ArchiveEndpoint it does not report whether it matched a row. An archive that names nothing and an archive of something already archived are both "this subscription is not live", which is the state the caller asked for; the distinction between them is a read, and GetSubscription is that read.

func (*SQLStore) Backlog

func (s *SQLStore) Backlog(ctx context.Context) (depth int64, oldest time.Time, err error)

Backlog reads how many dispatches are waiting and how old the oldest is.

An empty queue is no row rather than a row of zeroes — see webhooks/internal/queries on why the statement is grouped — so a driver's empty result is the answer here rather than a failure: nothing is waiting, and there is no oldest row to date.

func (*SQLStore) Claim

func (s *SQLStore) Claim(ctx context.Context, now time.Time, limit int, leaseUntil time.Time) ([]ClaimedDispatch, error)

Claim selects a batch, leases it, and reads it back — all in one transaction, so two workers cannot lease the same rows.

func (*SQLStore) EndpointsForEvent

func (s *SQLStore) EndpointsForEvent(ctx context.Context, q database.SQLQueryExecutor, scope tenancy.Scope, eventType EventType) ([]*Endpoint, error)

EndpointsForEvent resolves the fan-out set within one scope, using the caller's executor so it sees the same snapshot as the transaction that is dispatching.

func (*SQLStore) Enqueue

func (s *SQLStore) Enqueue(ctx context.Context, q database.Tx, delivery *Delivery, endpointIDs []string, now time.Time) error

Enqueue writes the delivery and its dispatches through the caller's transaction, so they commit with whatever else that transaction did.

Every dispatch in one call carries the instant the caller named, as both its creation time and its first next_attempt: a new dispatch is eligible immediately, and the claim's per-key ordering predicate compares (created_at, id) across the rows one fan-out produced.

func (*SQLStore) GetEndpoint

func (s *SQLStore) GetEndpoint(ctx context.Context, scope tenancy.Scope, endpointID string) (*Endpoint, error)

GetEndpoint reads one of the scope's endpoints and its subscriptions. An endpoint registered in another scope reads as absent.

func (*SQLStore) GetSubscription

func (s *SQLStore) GetSubscription(ctx context.Context, scope tenancy.Scope, subscriptionID string) (*Subscription, error)

GetSubscription reads one of the scope's subscriptions, archived ones included. A subscription under another scope's endpoint reads as absent.

func (*SQLStore) ListAttempts

func (s *SQLStore) ListAttempts(ctx context.Context, scope tenancy.Scope, deliveryID string, filter *filtering.QueryFilter) (*filtering.QueryFilteredResult[Attempt], error)

ListAttempts pages one of the scope's deliveries' logs, in the direction the filter names. A delivery in another scope reads as one with no attempts.

func (*SQLStore) ListEndpoints

ListEndpoints pages one scope's registry, in the direction the filter names.

The direction is a choice between two generated statements rather than an argument either of them binds — see sortedRows — so what this method does with filter.SortBy is pick the one whose ORDER BY and cursor comparison agree with it.

func (*SQLStore) ListSubscriptions

func (s *SQLStore) ListSubscriptions(ctx context.Context, scope tenancy.Scope, endpointID string, filter *filtering.QueryFilter) (*filtering.QueryFilteredResult[Subscription], error)

ListSubscriptions pages the live subscriptions of one of the scope's endpoints, in the direction the filter names.

func (*SQLStore) MarkDelivered

func (s *SQLStore) MarkDelivered(ctx context.Context, dispatchID string, at time.Time) error

MarkDelivered retires an accepted dispatch.

func (*SQLStore) Reap

func (s *SQLStore) Reap(ctx context.Context, before time.Time, limit int) (int64, error)

Reap deletes delivered dispatches past the retention window, then the log rows and deliveries left without one.

The three DELETEs run in one transaction so a crash between them cannot leave a delivery whose dispatches are gone but whose payload lingers forever — nothing would ever revisit it.

func (*SQLStore) RecordAttempt

func (s *SQLStore) RecordAttempt(ctx context.Context, attempt *Attempt) error

RecordAttempt appends to the delivery log.

func (*SQLStore) RecordFailure

func (s *SQLStore) RecordFailure(ctx context.Context, dispatchID string, attempts int, nextAttempt time.Time, lastErr string, dead bool) error

RecordFailure schedules the retry, or marks the dispatch dead.

func (*SQLStore) Requeue

func (s *SQLStore) Requeue(ctx context.Context, deliveryID, endpointID string, at time.Time) error

Requeue re-drives one delivery to one endpoint.

func (*SQLStore) SaveEndpoint

func (s *SQLStore) SaveEndpoint(ctx context.Context, endpoint *Endpoint) error

SaveEndpoint upserts the endpoint and reconciles its subscription set, both in one transaction — a half-registered endpoint would either receive events it no longer subscribes to or silently receive none.

The scope comes off the endpoint rather than being passed beside it, so the row and the predicate cannot disagree.

type SQLStoreOption

type SQLStoreOption func(*SQLStore)

SQLStoreOption configures a SQL Store.

func WithStoreLogger

func WithStoreLogger(logger logging.Logger) SQLStoreOption

WithStoreLogger attaches a logger.

func WithStoreTracerProvider

func WithStoreTracerProvider(tracerProvider tracing.Provider) SQLStoreOption

WithStoreTracerProvider attaches a tracer provider.

Worth setting. The store's spans are where a slow dispatch turns out to be a slow claim, which is otherwise a gap inside the worker's own span.

func WithTablePrefix

func WithTablePrefix(prefix string) SQLStoreOption

WithTablePrefix overrides DefaultTablePrefix. It must be a plain SQL identifier fragment: it is interpolated into the query text, not bound as a parameter, and it must match the prefix the migrations were rendered with.

type Secret

type Secret = requestsigning.Keyring

Secret carries an endpoint's HMAC signing keys.

It is a pair rather than a single value so that rotation is not an outage. Every delivery is signed under Current and, while Previous is set, again under Previous; both signatures travel in the same header. A subscriber therefore accepts deliveries throughout the window in which it is switching keys, and the operator clears Previous once every subscriber has moved.

A single per-account secret — which is what this package exists partly to replace — makes that impossible: rolling it breaks every subscriber for the account at the same instant, so in practice it never gets rolled.

It is an alias rather than a type of its own, so that an endpoint's keys and the keys any other signed call in the platform uses are the same thing. The scheme lives in requestsigning; webhooks is one of its callers, not its owner.

type Store

type Store interface {
	// SaveEndpoint creates or replaces an endpoint, under the scope the endpoint
	// carries, and reconciles its subscriptions against the set it names.
	//
	// Reconciles rather than replaces: a subscription the endpoint already has is
	// kept, with its identity and its creation time; one it names for the first
	// time is created; one it no longer names is archived rather than deleted, so
	// that a subscription an endpoint has ended is still something the delivery
	// log can be read against. It fills the endpoint's Subscriptions with the rows
	// that are live afterwards, IDs included.
	SaveEndpoint(ctx context.Context, endpoint *Endpoint) error
	// GetEndpoint reads one of scope's endpoints, secrets included. It returns an
	// error wrapping database/sql.ErrNoRows when the endpoint does not exist —
	// including when it exists in another scope, which is the same answer as far
	// as this scope is concerned.
	GetEndpoint(ctx context.Context, scope tenancy.Scope, endpointID string) (*Endpoint, error)
	// ListEndpoints pages through the endpoints registered in scope.
	ListEndpoints(ctx context.Context, scope tenancy.Scope, filter *filtering.QueryFilter) (*filtering.QueryFilteredResult[Endpoint], error)
	// ArchiveEndpoint retires one of scope's endpoints. Its delivery history is
	// retained: the attempts log outlives the endpoint, because "what did we send
	// them" is asked most often after someone has been removed.
	//
	// Its subscriptions are left as they are. An archived endpoint is excluded
	// from fan-out by its own archived_at, so archiving them too would buy
	// nothing and would lose which event types it was subscribed to if it is ever
	// re-registered.
	ArchiveEndpoint(ctx context.Context, scope tenancy.Scope, endpointID string) error

	// AddSubscription subscribes one of scope's endpoints to eventType and
	// returns the resulting row.
	//
	// It is idempotent on the (endpoint, event type) pair: subscribing to
	// something the endpoint already subscribes to returns the existing row, and
	// re-subscribing to something it archived revives that row rather than
	// minting a second one for the same pair. It returns an error wrapping
	// database/sql.ErrNoRows when the endpoint does not exist in scope.
	//
	// The catalog is not checked here — a Store has none. StoreDispatcher.Subscribe
	// is the entry point that gates on it, and it is the one an application should
	// call, for the reason Register exists rather than SaveEndpoint being public
	// API: an accepted subscription to an event type nothing publishes is an
	// endpoint that never fires and no signal explaining why.
	AddSubscription(ctx context.Context, scope tenancy.Scope, endpointID string, eventType EventType) (*Subscription, error)
	// GetSubscription reads one of scope's subscriptions, archived ones included —
	// "when did they stop receiving this" is a question about an archived row. It
	// returns an error wrapping database/sql.ErrNoRows when the subscription does
	// not exist under one of scope's endpoints.
	GetSubscription(ctx context.Context, scope tenancy.Scope, subscriptionID string) (*Subscription, error)
	// ListSubscriptions pages the live subscriptions of one of scope's endpoints.
	ListSubscriptions(ctx context.Context, scope tenancy.Scope, endpointID string, filter *filtering.QueryFilter) (*filtering.QueryFilteredResult[Subscription], error)
	// ArchiveSubscription retires one of scope's subscriptions, so the endpoint
	// stops receiving that event type without its other subscriptions, its
	// delivery history, or its identity being touched.
	//
	// This is the method a flat event list cannot offer. Against one, "stop
	// sending me order.created" can only be expressed as a rewrite of the whole
	// set, which loses when it happened and races any concurrent edit of the same
	// endpoint.
	ArchiveSubscription(ctx context.Context, scope tenancy.Scope, subscriptionID string) error

	// EndpointsForEvent returns the enabled, unarchived endpoints in scope that
	// are subscribed to eventType, using the caller's executor.
	//
	// The scope is a parameter and not an option: this is the query whose missing
	// filter delivers one account's event to every other account's subscribers.
	EndpointsForEvent(ctx context.Context, q database.SQLQueryExecutor, scope tenancy.Scope, eventType EventType) ([]*Endpoint, error)
	// Enqueue writes a delivery and one dispatch per endpoint, in the caller's
	// transaction, so both commit with whatever else that transaction did.
	// The delivery's scope is stored with it.
	Enqueue(ctx context.Context, q database.Tx, delivery *Delivery, endpointIDs []string, now time.Time) error

	// Claim leases the next batch of due dispatches, incrementing their attempt
	// counts, and returns them ready to send.
	//
	// It spans every scope, deliberately: a worker delivers the whole
	// deployment's backlog, and a per-scope claim would need a list of scopes
	// nothing maintains. What it returns is scoped — each ClaimedDispatch says
	// which scope it came from.
	Claim(ctx context.Context, now time.Time, limit int, leaseUntil time.Time) ([]ClaimedDispatch, error)
	// MarkDelivered retires a dispatch that was accepted.
	MarkDelivered(ctx context.Context, dispatchID string, at time.Time) error
	// RecordFailure releases the lease, schedules the retry, and sets dead once
	// the dispatch has exhausted its attempts.
	//
	// attempts is persisted as given rather than left as Claim incremented it,
	// so the caller can decline to charge an attempt for a failure the
	// subscriber never saw — an open circuit being the case that matters.
	RecordFailure(ctx context.Context, dispatchID string, attempts int, nextAttempt time.Time, lastErr string, dead bool) error
	// RecordAttempt appends to the delivery log.
	RecordAttempt(ctx context.Context, attempt *Attempt) error
	// ListAttempts pages through the attempts recorded for one of scope's
	// deliveries. A delivery in another scope reads as one with no attempts,
	// which is what it is from here.
	ListAttempts(ctx context.Context, scope tenancy.Scope, deliveryID string, filter *filtering.QueryFilter) (*filtering.QueryFilteredResult[Attempt], error)

	// Requeue makes a delivery/endpoint pair claimable again, clearing its dead
	// flag and attempt count. It returns an error wrapping ErrDeliveryNotFound
	// if the pair was never dispatched or has been reaped.
	//
	// It takes no scope because the pair is already one: a dispatch exists only
	// where a delivery fanned out to an endpoint, and Dispatch resolves those
	// within one scope. StoreDispatcher.Replay is the scoped entry point, and it
	// establishes the scope by reading the endpoint in it first.
	Requeue(ctx context.Context, deliveryID, endpointID string, at time.Time) error

	// Backlog reports how many dispatches are waiting and when the oldest was
	// created, for the worker's health gauges. Like Claim, it spans every scope.
	Backlog(ctx context.Context) (depth int64, oldest time.Time, err error)
	// Reap deletes delivered dispatches, their deliveries, and their attempts
	// once they age past the retention window, up to limit rows.
	Reap(ctx context.Context, before time.Time, limit int) (int64, error)
}

Store is the persistence seam.

This package ships a SQL implementation (NewSQLStore) together with the DDL it needs (webhooks/migrations), so adopting webhooks does not mean writing this. The interface exists because delivery mechanics and persistence are genuinely separable, and an application with its own schema conventions — or one storing endpoints somewhere that is not a SQL database — should not have to fork the package to keep them.

Methods taking a database.SQLQueryExecutor run inside the caller's transaction and must use it rather than a handle of their own; that is what makes Dispatch atomic with the state change that caused it. The rest own their own statements.

Every method reaching an endpoint, a subscription, or a delivery takes a tenancy.Scope, or reads one off the value it was handed, and none of them offers an unscoped variant — an implementation must filter on it rather than treat it as a hint. A subscription carries no scope of its own; its owner is its endpoint's, and the scope is reached through that, the same way the delivery log reaches it through the delivery.

The exceptions are the worker's own machinery below Enqueue: Claim, Backlog, and Reap deliberately span every scope, because one worker drains one queue for the whole deployment, and MarkDelivered, RecordFailure, RecordAttempt, and Requeue address a dispatch the worker or an operator is already holding. Those are the component servicing itself, not a consumer read.

type StoreDispatcher

type StoreDispatcher struct {
	// contains filtered or unexported fields
}

StoreDispatcher is the Dispatcher backed by a Store. It is exported, and returned by NewDispatcher, so a caller can depend on the dispatcher it built rather than on the Dispatcher seam.

func NewDispatcher

func NewDispatcher(store Store, opts ...DispatcherOption) (*StoreDispatcher, error)

NewDispatcher builds a Dispatcher over the given Store.

func (*StoreDispatcher) Dispatch

func (d *StoreDispatcher) Dispatch(ctx context.Context, q database.Tx, delivery *Delivery) error

Dispatch fans a delivery out to its subscribers, inside the caller's transaction.

Taking the executor rather than opening its own is the entire transactional guarantee, and it is the same seam outbox.Enqueue uses:

err := client.WithTransaction(ctx, func(q database.Tx) error {
	if err := updateOrder(ctx, q, order); err != nil {
		return err
	}

	return dispatcher.Dispatch(ctx, q, &webhooks.Delivery{
		EventType:   OrderUpdated,
		OrderingKey: order.ID,
		Payload:     body,
	})
})

The deliveries live or die with the state change that caused them. There is no way to dispatch outside a transaction by accident: holding a SQLQueryExecutor from WithTransaction means you are already in one.

An event nobody subscribes to is not an error and writes nothing. That is the common case for most event types most of the time, and making it an error would have every publisher branch on it.

The fan-out is bounded by the delivery's Scope: subscribers are resolved within it, so an endpoint registered by one account never receives another account's copy of the same event type. A delivery with no scope is refused rather than fanned out to everybody — see Delivery.Scope. An application whose events are global says tenancy.Global() and gets what it had before the dimension existed.

func (*StoreDispatcher) Register

func (d *StoreDispatcher) Register(ctx context.Context, endpoint *Endpoint) error

Register validates an endpoint against the catalog and the SSRF rules, then stores it.

Validation happens here rather than being left to the caller because the consequence of skipping it is not a bad row — it is a server that will make authenticated requests to whatever URL was submitted. There is no variant of this that stores without checking.

func (*StoreDispatcher) Replay

func (d *StoreDispatcher) Replay(ctx context.Context, scope tenancy.Scope, deliveryID, endpointID string) error

Replay makes one past delivery to one endpoint claimable again.

It is the operator's recovery tool, and it is scoped to a pair rather than to a delivery because that is what recovery actually looks like: one subscriber was down, the others were fine, and re-driving the whole delivery would send duplicates to everyone who already accepted it.

The attempt count is reset, so a dead dispatch gets a full budget rather than dying again on its next attempt.

The scope is what makes this a replay of one's own delivery rather than of anybody's. It is established on the endpoint, which is read within it first: an endpoint in another scope reads as absent, and the requeue that follows names a (delivery, endpoint) pair, which exists only where a fan-out in that scope put it.

func (*StoreDispatcher) Subscribe

func (d *StoreDispatcher) Subscribe(ctx context.Context, scope tenancy.Scope, endpointID string, eventType EventType) (*Subscription, error)

Subscribe adds one event type to an existing endpoint, without rewriting the rest of its subscriptions.

It goes through the dispatcher rather than straight to the Store for the reason Register does: the catalog gate lives here. A subscription to an event type nothing publishes is accepted silently by any storage layer and produces an endpoint that never fires, with no signal explaining why — which is the failure the catalog exists to turn into an error at the moment somebody can still fix the typo.

It is idempotent on the (endpoint, event type) pair. Subscribing to something the endpoint already receives returns the existing subscription, and re-subscribing to one that was archived revives it, keeping its ID.

func (*StoreDispatcher) Unsubscribe

func (d *StoreDispatcher) Unsubscribe(ctx context.Context, scope tenancy.Scope, subscriptionID string) error

Unsubscribe retires one subscription, named by its own ID.

This is the operation a flat event list has no form for. Against one, "stop sending me order.created" is a rewrite of the whole set — which loses when the subscription ended, has nothing for the request that asked to name, and silently reverts any concurrent edit of the same endpoint.

There is no catalog gate here, and there should not be: an event type can be removed from an application's catalog, and the subscriptions to it are exactly what somebody then needs to be able to retire.

type Subscription

type Subscription struct {
	// CreatedAt is when the subscription was made, stamped by the Store.
	CreatedAt time.Time `json:"createdAt"`
	// LastUpdatedAt is when a write last named this subscription: the save or the
	// AddSubscription that would have created it had it not already existed.
	// Nil until one has, so a subscription made once and left alone has none.
	LastUpdatedAt *time.Time `json:"lastUpdatedAt"`
	// ArchivedAt is when the subscription was retired. Archived subscriptions
	// are excluded from fan-out and from the endpoint's Subscriptions, and are
	// kept so that "when did they stop receiving this" has an answer.
	ArchivedAt *time.Time `json:"archivedAt"`
	// ID identifies the subscription, and is what ArchiveSubscription names.
	// Generated by the Store when empty.
	ID string `json:"id"`
	// EndpointID is the endpoint that subscribed.
	EndpointID string `json:"endpointID"`
	// EventType is the catalog event type subscribed to.
	EventType EventType `json:"eventType"`
}

Subscription is one endpoint's interest in one event type, as a row of its own: identified, timestamped, and archivable without touching the rest.

It is a row and not an entry in a list because retiring one subscription is something an application's API is asked to do. Against a flat list the only available answer is to rewrite the whole set, which cannot say when a subscription ended, has no identifier for the request to name, and races any concurrent edit of the same endpoint. Against rows it is an archive of one ID.

It carries no tenancy.Scope. Its owner is the endpoint's — a subscription is a fact about an endpoint, and a second copy of the scope on every row here is a copy that can disagree with the first. Every store read of a subscription takes a scope and reaches it through the endpoint, the same way the delivery log reaches it through the delivery.

func SubscribeTo

func SubscribeTo(events ...EventType) []Subscription

SubscribeTo builds the subscription set for an endpoint being registered.

Registration names event types, not subscription IDs — there are none yet — so this is the form the write side takes:

dispatcher.Register(ctx, &webhooks.Endpoint{
	Scope:         tenancy.Of(accountID),
	URL:           "https://subscriber.example/hooks",
	Secret:        webhooks.Secret{Current: key},
	Subscriptions: webhooks.SubscribeTo(OrderCreated, OrderUpdated),
})

Duplicates are not filtered here. SaveEndpoint reconciles by event type, so a repeated type is one row either way, and dropping them silently would hide a caller building the list from something that had a bug in it.

func (*Subscription) Archived

func (s *Subscription) Archived() bool

Archived reports whether the subscription has been retired.

type URLChecker

type URLChecker func(ctx context.Context, rawURL string) error

URLChecker vets a delivery target. CheckEndpointURL is the implementation this package uses unless a caller replaces it.

It is replaceable because a minority of deployments deliver webhooks to internal services on purpose — a sidecar, another service on the same private network — and for them CheckEndpointURL's refusal is not a safety property but a wall. Replacing it means owning the SSRF question yourself: the replacement is the only thing standing between a user-supplied URL and an authenticated request from inside your network, so it should be an allowlist of hosts you operate, not a function that returns nil.

type Worker

type Worker struct {
	// contains filtered or unexported fields
}

Worker delivers claimed dispatches. It owns a goroutine started by Run and stopped by Close.

func NewWorker

func NewWorker(ctx context.Context, cfg *WorkerConfig, store Store, opts ...WorkerOption) (*Worker, error)

NewWorker builds a Worker. It does not start it; call Run.

ctx is used to validate the config and is not retained — Run takes its own.

func (*Worker) Close

func (w *Worker) Close(ctx context.Context) error

Close stops the worker and waits for the in-flight cycle to finish. Safe to call more than once.

func (*Worker) Run

func (w *Worker) Run()

Run is the worker loop. Like outbox.Relay.Run it takes no context: tied to a server context it would stop delivering while requests were still committing dispatch rows. The owner calls Close after the server has shut down.

Run returns only after Close.

type WorkerConfig

type WorkerConfig struct {
	// UserAgent identifies deliveries to subscribers.
	UserAgent string `env:"USER_AGENT" json:"userAgent,omitempty" yaml:"userAgent,omitempty"`
	// Backoff drives the retry schedule for failed deliveries. MaxAttempts is
	// the threshold past which a dispatch is marked dead.
	Backoff retrycfg.Config `envPrefix:"BACKOFF_" json:"backoff,omitzero" yaml:"backoff,omitempty"`
	// PollInterval is how often the worker looks for work.
	PollInterval time.Duration `env:"POLL_INTERVAL" json:"pollInterval,omitempty" yaml:"pollInterval,omitempty"`
	// LeaseDuration is how long a claim is held before it can be reclaimed.
	LeaseDuration time.Duration `env:"LEASE_DURATION" json:"leaseDuration,omitempty" yaml:"leaseDuration,omitempty"`
	// RequestTimeout bounds one delivery request.
	RequestTimeout time.Duration `env:"REQUEST_TIMEOUT" json:"requestTimeout,omitempty" yaml:"requestTimeout,omitempty"`
	// CircuitOpenRetryDelay is how long a short-circuited dispatch waits before
	// becoming claimable again. It is a flat delay rather than a backoff step,
	// because backing off exponentially against an open circuit means the first
	// delivery after recovery can wait far longer than the outage did.
	CircuitOpenRetryDelay time.Duration `env:"CIRCUIT_OPEN_RETRY_DELAY" json:"circuitOpenRetryDelay,omitempty" yaml:"circuitOpenRetryDelay,omitempty"`
	// Retention is how long delivered dispatches are kept before reaping.
	Retention time.Duration `env:"RETENTION" json:"retention,omitempty" yaml:"retention,omitempty"`
	// ReapInterval is how often the reaper runs.
	ReapInterval time.Duration `env:"REAP_INTERVAL" json:"reapInterval,omitempty" yaml:"reapInterval,omitempty"`
	// BatchSize is how many dispatches one cycle claims.
	BatchSize int `env:"BATCH_SIZE" json:"batchSize,omitempty" yaml:"batchSize,omitempty"`
	// Concurrency is how many deliveries run at once within a batch.
	Concurrency int `env:"CONCURRENCY" json:"concurrency,omitempty" yaml:"concurrency,omitempty"`
	// ReapBatchSize caps how many rows one reap deletes.
	ReapBatchSize int `env:"REAP_BATCH_SIZE" json:"reapBatchSize,omitempty" yaml:"reapBatchSize,omitempty"`
}

WorkerConfig configures a Worker.

func (*WorkerConfig) EnsureDefaults

func (cfg *WorkerConfig) EnsureDefaults()

EnsureDefaults fills unset knobs with the package defaults.

func (*WorkerConfig) ValidateWithContext

func (cfg *WorkerConfig) ValidateWithContext(ctx context.Context) error

ValidateWithContext validates a WorkerConfig.

type WorkerOption

type WorkerOption func(*Worker)

WorkerOption configures a Worker.

func WithCircuitBreakerFactory

func WithCircuitBreakerFactory(factory CircuitBreakerFactory) WorkerOption

WithCircuitBreakerFactory supplies the per-endpoint circuit breakers.

Without it every endpoint gets a noop breaker and a permanently dead subscriber is retried at full rate forever, competing with healthy endpoints for the same worker pool.

func WithHTTPClient

func WithHTTPClient(client *http.Client) WorkerOption

WithHTTPClient supplies the client every delivery goes through.

One client for the whole worker is the point. A client built per delivery — which is what this package was extracted to replace — reuses no connections, so every delivery pays a fresh TCP handshake and a fresh TLS handshake to a subscriber it just talked to.

The supplied client's redirect policy is overridden: following a redirect would deliver a signed payload to a host the operator never registered and never had checked, which turns an open redirect on a subscriber's domain into an SSRF. Its transport is left alone.

func WithWorkerClock

func WithWorkerClock(c clock.Clock) WorkerOption

WithWorkerClock swaps the clock driving the poll loop, leases, and backoff.

func WithWorkerLogger

func WithWorkerLogger(logger logging.Logger) WorkerOption

WithWorkerLogger attaches a logger. The worker reports every delivery failure and every dead dispatch through it; without one, a subscriber that has stopped accepting deliveries is visible only in metrics.

func WithWorkerMetricsProvider

func WithWorkerMetricsProvider(metricsProvider metrics.Provider) WorkerOption

WithWorkerMetricsProvider attaches a metrics provider.

func WithWorkerTracerProvider

func WithWorkerTracerProvider(tracerProvider tracing.Provider) WorkerOption

WithWorkerTracerProvider attaches a tracer provider. Cycles that claim nothing are not traced — a root span every poll interval is noise.

func WithWorkerURLChecker

func WithWorkerURLChecker(checker URLChecker) WorkerOption

WithWorkerURLChecker replaces the URL policy re-checked at delivery.

Pair it with the Dispatcher's WithDispatcherURLChecker: an endpoint accepted at registration and refused here sits in the backlog until it dies, so the two halves must agree. See URLChecker for what replacing it costs.

Directories

Path Synopsis
Package webhookscfg assembles the webhook machinery from environment configuration: the Store both halves share, the Dispatcher applications write through, and the Worker that delivers.
Package webhookscfg assembles the webhook machinery from environment configuration: the Store both halves share, the Dispatcher applications write through, and the Worker that delivers.
Package inbound receives webhooks: it verifies the provider's signature over the bytes as they arrived, publishes the delivery to a message queue, and acks.
Package inbound receives webhooks: it verifies the provider's signature over the bytes as they arrived, publishes the delivery to a message queue, and acks.
config
Package inboundcfg assembles an inbound webhook receiver from environment configuration: the Verifier for the provider's signing scheme, and the Receiver that mounts on a router and publishes what it verifies.
Package inboundcfg assembles an inbound webhook receiver from environment configuration: the Verifier for the provider's signing scheme, and the Receiver that mounts on a router and publishes what it verifies.
internal
queries
Package queries is the webhooks schema described as data — the canonical table names and each table's columns in the order every read projects them — together with the thirty statements the store executes, rendered from it.
Package queries is the webhooks schema described as data — the canonical table names and each table's columns in the order every read projects them — together with the thirty statements the store executes, rendered from it.
queriesgen command
Command queriesgen writes the canonical sqlc input for the webhooks schema, one file per dialect, from webhooks/internal/queries.
Command queriesgen writes the canonical sqlc input for the webhooks schema, one file per dialect, from webhooks/internal/queries.
Package migrations supplies the webhook tables' DDL, rendered for a dialect and table prefix.
Package migrations supplies the webhook tables' DDL, rendered for a dialect and table prefix.
Package webhooksmock provides moq-generated mock implementations of interfaces in the webhooks package.
Package webhooksmock provides moq-generated mock implementations of interfaces in the webhooks package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL