outboundsend

package
v1.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 5, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package outboundsend is Layer 3 of the outbound pipeline (docs/design/async-message-pipeline.md): the River execution stage that submits an accepted message to the upstream provider (SES) and records the terminal outcome. It mirrors internal/webhookdelivery — a River Worker on the shared `outbound` queue, with River owning claim / retry / rescue.

Delivery is at-least-once: River re-drives a crashed job, so the provider may receive a duplicate if the SMTP submit is accepted but the worker crashes before marking the message sent. That residual is narrowed by the X-E2A-Message-ID wire header + SNS reconciliation (async-send-contract §3.1): the SNS consumer records provider-accept evidence on the row, the re-driven claim then settles the message as sent instead of re-submitting, and the terminal-failure guard (here and in the terminal reconciler, via the store's guarded MarkFailed) never declares a provider-accepted row failed. A final attempt that fails ambiguously defers its terminal write to the reconciler's provider-evidence grace window rather than firing an immediate — possibly false — email.failed.

Every provider call passes through the sending-protection Gate (internal/sendingpolicy). The worker order is fixed: Reserve the durable attempt; on a hold, snooze without provider I/O; on a rate deferral, DeferAttempt; on a final suppression match, CancelAttempt; ConsumeAttempt is the last serialized decision; the authorized submitter redeems the token immediately before the socket opens and settles the provider's answer. A later execution after a confirmed attempt returns to Reserve, which allocates the next ordinal — the worker never chooses one.

One SMTP attempt per job attempt — River owns the multi-attempt envelope via NextRetry, so Work() stays short (the deliverer does a single submit, not an internal retry loop). See the design's "claim + rescue, not a lease" note.

Index

Constants

View Source
const MaxSendAttempts = 6

MaxSendAttempts caps app/permanent-error retries (bounded 4xx/unknown tail).

View Source
const PolicyBudgetHoldHorizon = 7 * 24 * time.Hour

PolicyBudgetHoldHorizon bounds a sending-budget hold: a message may wait through several UTC days for capacity, but not forever. Seven days is the policy's budget_hold_max_days default; the worker holds it as a constant because the deadline is derived, never stored, and every execution must derive the same one.

View Source
const SendRetryHorizon = 72 * time.Hour

SendRetryHorizon bounds the outage-tolerant tail: past this age a message in a rate/ramp/provider or tenant-setup hold is declared terminally failed. 72h matches the industry MTA retry horizon (and the webhook deliverer's envelope) — long enough to ride out a multi-hour regional SES incident, not forever.

Variables

View Source
var ErrSendingPaused = errors.New("outboundsend: account sending is paused")

ErrSendingPaused is returned by the enqueue entry points when the owning account is paused: the acceptance surface must reject the request rather than queue mail that can never leave.

Functions

func SubmissionDedupeKey added in v1.2.1

func SubmissionDedupeKey(jobID int64, attempt int, reason messagelifecycle.ReasonCode) string

SubmissionDedupeKey is the stable message-local identity for one observed River submission attempt and reason.

Types

type DailyQuotaDeferredError added in v1.8.0

type DailyQuotaDeferredError struct {
	RetryAt time.Time
}

DailyQuotaDeferredError is returned by Store.ClaimSend when the owning account's per-day send cap is exhausted at fire time. The store has already released the send claim; the worker snoozes the job until RetryAt (the next UTC midnight, when the daily window resets) instead of failing the message.

func (*DailyQuotaDeferredError) Error added in v1.8.0

func (e *DailyQuotaDeferredError) Error() string

type DeliverOutcome

type DeliverOutcome struct {
	ProviderMessageID string
	SentAs            string
	Err               error
	// Permanent marks a non-retryable failure (validation / permanent 5xx): the
	// worker fails the message terminally instead of retrying.
	Permanent bool
	// Outage marks a provider-connection failure (relay unreachable/misconfigured):
	// the worker snoozes without burning an attempt (design §8), up to the retry
	// horizon. Mutually exclusive with Permanent in practice.
	Outage bool
	// AcceptanceUnknown marks a failure AFTER the whole body was handed to the
	// provider (the 250 never came): the provider may hold the message. Never
	// permanent; the next attempt is a new ordinal, and provider feedback
	// carrying the attempt header is the only authoritative answer.
	AcceptanceUnknown bool
	// SettlementErr reports that the provider ACCEPTED the message but the
	// local settlement did not commit. The send happened; the caller must not
	// resubmit.
	SettlementErr error
}

DeliverOutcome is the result of one authorized provider submission.

type Deliverer

type Deliverer interface {
	Deliver(ctx context.Context, j *SendJob, auth sendingpolicy.ProviderAuthorization) DeliverOutcome
}

Deliverer performs a SINGLE authorized SMTP submit — River owns re-attempts. The token is the authorization for exactly this call; the production implementation (the outbound.ProviderSubmitter) redeems it immediately before the socket opens and refuses to dial without it.

type HoldClass added in v1.9.0

type HoldClass string

HoldClass is the durable finite-hold classification persisted on the message the first time it waits for something with a clock.

const (
	// HoldRateRampOrProvider: per-agent rate, custom-domain ramp, or provider
	// outage. 72-hour deadline; expiry reason submission.local_retries_exhausted.
	HoldRateRampOrProvider HoldClass = "rate_ramp_or_provider"
	// HoldTenantSetup: the account's SES tenant is not ready. 72-hour deadline;
	// expiry reason submission.sending_setup_expired. Transitions exactly once
	// to HoldRateRampOrProvider when readiness lands before the setup deadline.
	HoldTenantSetup HoldClass = "tenant_setup"
	// HoldPolicyBudget: a sending-budget pool is exhausted. Seven-day deadline
	// from the existing anchor; every finite class promotes to it and nothing
	// moves it afterwards. Expiry reason submission.policy_budget_expired.
	HoldPolicyBudget HoldClass = "policy_budget"
)

func HoldClassFor added in v1.9.0

func HoldClassFor(reason string) HoldClass

HoldClassFor maps a gate hold reason to its finite-hold class; "" means the hold has no clock (an account pause).

type Jobs

type Jobs struct {
	// contains filtered or unexported fields
}

Jobs is the outbound-send integration on the shared River client: a jobs.Registrar (contributes SendWorker + the terminal reconciler) plus the transactional enqueue entry point the accept-tx calls. The shared client is injected via SetEnqueuer after jobs.New builds it (two-phase wiring, same as webhookdelivery / senderidentity).

func NewJobs

func NewJobs(store Store, deliverer Deliverer, pool *pgxpool.Pool) *Jobs

NewJobs builds the integration with its dependencies (no client yet). pool backs the periodic terminal-state reconciler's scan and the legacy-argument resolver's transaction.

func (*Jobs) Deliverer added in v1.9.0

func (j *Jobs) Deliverer() Deliverer

Deliverer exposes the wired provider deliverer, for the same test.

func (*Jobs) EnqueueScheduledSendTx added in v1.5.0

func (j *Jobs) EnqueueScheduledSendTx(ctx context.Context, tx pgx.Tx, messageID string, at time.Time) (int64, error)

EnqueueScheduledSendTx is EnqueueSendTx for a scheduled send: it enqueues the same outbound_send job in the caller's transaction, but with river.InsertOpts.ScheduledAt=at so River holds the job in state `scheduled` and does not promote it to a worker until `at`. Everything downstream (claim, suppression re-check, retry envelope, terminal reconciler) is byte-identical to an immediate send — scheduling changes only WHEN the job first runs. A zero `at` behaves exactly like EnqueueSendTx (immediate).

func (*Jobs) EnqueueSendTx

func (j *Jobs) EnqueueSendTx(ctx context.Context, tx pgx.Tx, messageID string) (int64, error)

EnqueueSendTx enqueues a send job WITHIN the caller's transaction — the outbox pattern: the accept-tx's messages-row insert and this job commit together, so an `accepted` message can never exist without a send job (or vice versa). The accept-tx stamps the returned river_job id on messages.send_job_id so the reconciler can find stranded rows (`accepted` with no job). Mirrors webhookdelivery.EnqueueDeliveryTx.

func (*Jobs) Gate added in v1.9.0

func (j *Jobs) Gate() sendingpolicy.Gate

Gate exposes the wired sending-protection gate, for the composition root's wiring test. nil when none is wired.

func (*Jobs) ReconcilePending

func (j *Jobs) ReconcilePending(ctx context.Context, pool *pgxpool.Pool) (int, error)

ReconcilePending enqueues an outbound_send job for every accepted message that has no send job yet (send_job_id IS NULL). Run ONCE at startup as the cutover.

Because the accept-tx is a single transaction (message insert + job enqueue + send_job_id stamp all commit together), a committed `accepted` row in steady state ALWAYS has send_job_id set — so the send_job_id IS NULL set is normally empty. This exists to enqueue (a) any pre-async `accepted` rows at the moment the mode is first flipped on, and (b) rows from a future accept-tx variant that doesn't stamp atomically. Idempotent: the per-row FOR UPDATE + send_job_id IS NULL guard means a re-run (or concurrent replica) never double-enqueues. Mirrors webhookdelivery.ReconcilePending. Returns the count enqueued.

func (*Jobs) RegisterJobs

func (j *Jobs) RegisterJobs(w *river.Workers) []*river.PeriodicJob

RegisterJobs adds the SendWorker and terminal-state safety net to the shared client's bundle. Implements jobs.Registrar.

func (*Jobs) RegisteredSendWorker added in v1.9.0

func (j *Jobs) RegisteredSendWorker() *SendWorker

RegisteredSendWorker returns the send worker the last RegisterJobs call registered with River, or nil before any registration.

func (*Jobs) ResolveLegacyOperation added in v1.9.0

func (j *Jobs) ResolveLegacyOperation(ctx context.Context, messageID string) (sendingpolicy.AcceptanceDecision, sendingpolicy.OperationRef, error)

ResolveLegacyOperation is the compatibility resolver for a job enqueued by a pre-floor slot with no operation reference. It runs the same PrepareExternalTx an accept transaction runs — idempotent on the durable operation row — in its own committed transaction, so an old job and a new one authorize identically. There is deliberately no other way to obtain an operation from a bare message id.

func (*Jobs) SendWorker added in v1.9.0

func (j *Jobs) SendWorker() *SendWorker

SendWorker builds the fully armed send worker RegisterJobs registers: the gate, the legacy resolver, the rate gate, and metrics. It is the one place those are wired, and the composition root's test inspects its result.

func (*Jobs) SetEnqueuer

func (j *Jobs) SetEnqueuer(e jobs.Enqueuer)

SetEnqueuer injects the shared client so EnqueueSendTx can insert jobs.

func (*Jobs) TerminalReconcileWorker added in v1.9.0

func (j *Jobs) TerminalReconcileWorker() *TerminalReconcileWorker

TerminalReconcileWorker builds the reconciler RegisterJobs registers.

func (*Jobs) WithGate added in v1.9.0

func (j *Jobs) WithGate(g sendingpolicy.Gate) *Jobs

WithGate injects the sending-protection gate. Every enqueue then prepares a durable operation in the accept transaction, and every worker execution authorizes through it. Chainable; nil keeps the gateless default (unit tests only — see NewSendWorker).

func (*Jobs) WithMetrics added in v1.2.2

func (j *Jobs) WithMetrics(m Metrics) *Jobs

WithMetrics injects the outbound SLI recorder, threaded to both workers at RegisterJobs. Chainable so the cmd wiring stays one expression; nil keeps the no-op default.

func (*Jobs) WithRateGate added in v1.5.0

func (j *Jobs) WithRateGate(g RateGate) *Jobs

WithRateGate injects the fire-time per-agent rate gate (internal/sendrate), threaded to the SendWorker at RegisterJobs. Chainable; nil keeps the allow-all default.

type Metrics added in v1.2.2

type Metrics interface {
	// OutboundQueueWait is the enqueue→worker-pickup latency of one send
	// attempt (River attempted_at − scheduled_at — due→pickup, never
	// cumulative message age).
	OutboundQueueWait(seconds float64)
	// OutboundTerminal records one terminal outcome for an outbound message.
	// outcome ∈ {sent, failed_suppressed, failed_provider,
	// failed_local_retries, failed_cancelled}.
	OutboundTerminal(outcome string)
	// OutboundTerminalLatency records eligibility→terminal latency for one
	// outbound message (the terminal write's occurred_at − submissionAnchor).
	// Observed at most once per message, co-located with OutboundTerminal so
	// the two share their exactly-once contract; a terminal whose occurred_at
	// precedes its anchor records the count with no latency sample.
	OutboundTerminalLatency(seconds float64)
	// OutboundAttempt records one submission attempt to the upstream relay.
	// outcome ∈ {success, temporary_failure, permanent_failure}.
	OutboundAttempt(outcome string, seconds float64)
	// OutboundRateDeferred records one submission deferred by the per-agent
	// fire-time rate gate — a snooze, not an attempt, and never terminal.
	OutboundRateDeferred()
}

Metrics is the narrow slice of telemetry.Metrics the outbound send pipeline emits (the janitor.Metrics pattern): injectable so tests assert emission with a fake, satisfied by every telemetry backend. Label values are normalized by the backend — never pass message ids or addresses.

type OperationResolver added in v1.9.0

type OperationResolver func(ctx context.Context, messageID string) (sendingpolicy.AcceptanceDecision, sendingpolicy.OperationRef, error)

OperationResolver recovers the durable operation for a job that carries no reference — a legacy argument shape from a pre-floor slot. It runs the same Prepare path an accept transaction runs, idempotently, so an old job and a new one authorize identically.

type OutboundSendArgs

type OutboundSendArgs struct {
	MessageID    string                      `json:"message_id"`
	OperationRef *sendingpolicy.OperationRef `json:"operation_ref,omitempty"`
}

OutboundSendArgs drives one outbound send. Args carry the message id and the durable operation reference the accept transaction prepared; the worker re-reads the messages row (the source of truth) each attempt. A job enqueued before the reference existed (a pre-floor slot) carries none and is resolved at fire time through the same Prepare path.

func (OutboundSendArgs) Kind

func (OutboundSendArgs) Kind() string

type RateDecision added in v1.5.0

type RateDecision = sendrate.Decision

RateDecision is the fire-time rate gate's answer for one submission slot: Allowed=false carries RetryAt, the earliest the agent's window frees capacity. Aliased to the storage type so a *sendrate.Store satisfies RateGate directly — no adapter.

type RateGate added in v1.5.0

type RateGate interface {
	Reserve(ctx context.Context, agentID string) (RateDecision, error)
	Window() time.Duration
}

RateGate reserves one slot in the per-agent fire-time submission budget (internal/sendrate) — the durable counterpart to the acceptance-time in-memory send limit, enforced immediately before provider submission so scheduled-send bursts and multi-replica deployments cannot exceed it. It stays separate from the sending-protection gate because it controls provider throughput, not reputation admission. A nil gate allows everything.

type SendJob

type SendJob struct {
	MessageID string
	// UserID is the owning account — the tenant scope for the pre-provider
	// suppression guard (suppressions are per-account).
	UserID       string
	AgentID      string // exact sending agent for agent-scoped consent checks
	Domain       string // exact registered sender domain
	MessageType  string // send|reply|test; platform tests are ramp-exempt
	Status       string // messages.delivery_status
	EnvelopeFrom string
	Recipients   []string
	RawMessage   []byte // composed MIME
	SentAs       string // From identity decided at accept ("own_address"|"relay")
	// AcceptedAt is messages.created_at.
	AcceptedAt time.Time
	// ScheduledAt is messages.scheduled_at for a scheduled send (zero for an
	// immediate one).
	ScheduledAt time.Time
	// ReviewedAt is messages.reviewed_at — when a HITL hold was resolved into the
	// send pipeline, zero for a message that was never held.
	ReviewedAt time.Time
	// ProviderAccepted is set when authoritatively correlated provider-accept
	// evidence has been recorded for this message: the provider already has it,
	// so the worker settles the row as sent instead of re-submitting a duplicate.
	ProviderAccepted   bool
	ProviderAcceptedAt *time.Time
	// ProviderMessageID is the evidence-repaired provider id accompanying
	// ProviderAccepted (” when no evidence).
	ProviderMessageID string
	// LocalHoldClass / LocalHoldAnchor are the durable finite-hold pair a
	// previous execution persisted (empty/zero when never held). The deadline
	// is derived from them on every execution and never stored.
	LocalHoldClass  HoldClass
	LocalHoldAnchor time.Time
	// LastResumedAt is the owning account's last pause→active transition; a
	// first finite hold anchors no earlier than it, so a pause that preceded
	// the hold does not consume its horizon. Zero when unknown.
	LastResumedAt time.Time
	// TenantReadyAt is when the account's SES tenant became ready (zero until
	// it is). Drives the one-way tenant_setup → rate_ramp_or_provider move.
	TenantReadyAt time.Time
}

SendJob is the send payload the worker loads from the messages row (Store.ClaimSend).

type SendWorker

type SendWorker struct {
	river.WorkerDefaults[OutboundSendArgs]
	// contains filtered or unexported fields
}

SendWorker submits an accepted message and records the terminal outcome. Mirrors webhookdelivery.DeliverWorker.

func NewSendWorker

func NewSendWorker(store Store, deliverer Deliverer) *SendWorker

NewSendWorker builds a worker with no sending-protection gate. Without a gate every provider call is made with an empty authorization, which the production submitter refuses before it dials; the composition root always installs one via WithGate, and its wiring test proves it.

func (*SendWorker) Gate added in v1.9.0

func (w *SendWorker) Gate() sendingpolicy.Gate

Gate exposes the wired sending-protection gate (nil when none), for the composition root's wiring test.

func (*SendWorker) HasOperationResolver added in v1.9.0

func (w *SendWorker) HasOperationResolver() bool

HasOperationResolver reports whether a legacy-argument resolver is wired. Without one every job from a pre-floor slot fails closed, so the wiring test insists on it.

func (*SendWorker) NextRetry

func (w *SendWorker) NextRetry(job *river.Job[OutboundSendArgs]) time.Time

NextRetry overrides River's default backoff with the decided send envelope.

func (*SendWorker) WithClock added in v1.9.0

func (w *SendWorker) WithClock(now func() time.Time) *SendWorker

WithClock overrides the worker's clock for deadline tests. Chainable.

func (*SendWorker) WithGate added in v1.9.0

func (w *SendWorker) WithGate(g sendingpolicy.Gate) *SendWorker

WithGate injects the sending-protection gate every provider call must pass. Chainable; nil keeps the gateless default described on NewSendWorker.

func (*SendWorker) WithMetrics added in v1.2.2

func (w *SendWorker) WithMetrics(m Metrics) *SendWorker

WithMetrics injects the SLI recorder. Chainable; nil keeps the no-op default so metrics stay optional wiring.

func (*SendWorker) WithOperationResolver added in v1.9.0

func (w *SendWorker) WithOperationResolver(r OperationResolver) *SendWorker

WithOperationResolver injects the legacy-argument resolver. Chainable; nil leaves a legacy job failing closed.

func (*SendWorker) WithRateGate added in v1.5.0

func (w *SendWorker) WithRateGate(g RateGate) *SendWorker

WithRateGate injects the fire-time per-agent rate gate (internal/sendrate). Chainable; nil keeps the allow-all default (no gate wired).

func (*SendWorker) Work

Work intentionally has no Timeout() override — a single SES submit comfortably fits River's 60s default JobTimeout. (Contrast the maintenance/sweep workers, which override it because they can run for minutes.)

type Store

type Store interface {
	// ClaimSend returns nil when the message is gone, trashed, terminal, or owned
	// by a different River job. It returns *DailyQuotaDeferredError (claim
	// released) when the account's daily send cap is exhausted at fire time.
	// (agent-delete cascade / TTL) — the worker treats a nil job as a no-op.
	ClaimSend(ctx context.Context, messageID string, jobID int64) (*SendJob, error)
	// ReleaseSend clears a side-effect-free attempt before River backoff.
	ReleaseSend(ctx context.Context, messageID string, jobID int64) error
	// RecordHold persists the message's finite-hold class and anchor. Terminal
	// writes clear the pair.
	RecordHold(ctx context.Context, messageID string, class HoldClass, anchor time.Time) error
	// MarkSent records the provider outcome monotonically from a pre-terminal
	// state, including when trash won after ClaimSend.
	MarkSent(ctx context.Context, messageID string, jobID int64, attempt int, occurredAt time.Time, providerMessageID, sentAs string) error
	// MarkFailed is the GUARDED terminal write (async-send-contract §3.1): if
	// provider-accept evidence has reached the row it settles the message as
	// sent (+ email.sent) instead; otherwise it sets delivery_status='failed'
	// with the given failure provenance + detail and emits email.failed — all
	// in one transaction. Callers therefore invoke it to "finalize a terminal
	// state", not to unconditionally fail.
	// The returned status reports what the guarded write actually did:
	// StatusFailed, StatusSent (evidence settle), or "" (no-op). The returned
	// time is the occurred_at the write actually used, and the returned
	// provider id is the evidence's provider message id on an evidence
	// settle (” otherwise), so the attempt that dialed can be settled with
	// it.
	MarkFailed(ctx context.Context, messageID string, jobID int64, attempt int, occurredAt time.Time, detail string, source delivery.FailureSource, reason messagelifecycle.ReasonCode, blockedRecipients []string) (delivery.Status, time.Time, string, error)
	PreserveTerminalFailure(ctx context.Context, messageID string, jobID int64, attempt int, occurredAt time.Time, detail string, source delivery.FailureSource, reason messagelifecycle.ReasonCode, blockedRecipients []string) error
	// DeferTerminalFailure records a final attempt's diagnostic + releases the
	// I/O claim WITHOUT declaring failed: the terminal reconciler declares the
	// outcome after the provider-evidence grace window (or settles the row as
	// sent when evidence arrives first).
	DeferTerminalFailure(ctx context.Context, messageID string, jobID int64, attempt int, occurredAt time.Time, detail string) error
	// RecordTemporaryFailure atomically records the retryable observation and
	// releases the send claim for River's next attempt.
	RecordTemporaryFailure(ctx context.Context, messageID string, jobID int64, attempt int, occurredAt time.Time, detail string) error
	// SuppressedRecipients returns the effective account-wide + exact-agent
	// subset — the last-line guard before provider I/O.
	SuppressedRecipients(ctx context.Context, userID, agentID string, recipients []string) ([]string, error)
}

Store is the messages-store surface the worker needs. Implemented over internal/identity in the binary. ClaimSend atomically checks that the message and agent are live and persists delivery_status='sending' for the stamped River job before provider I/O begins.

type TerminalReconcileArgs

type TerminalReconcileArgs struct{}

TerminalReconcileArgs drives the periodic safety net for outbound messages whose stamped send job reached a terminal state before recording delivery.

func (TerminalReconcileArgs) Kind

type TerminalReconcileWorker

type TerminalReconcileWorker struct {
	river.WorkerDefaults[TerminalReconcileArgs]
	// contains filtered or unexported fields
}

TerminalReconcileWorker settles accepted/sending outbound messages after their stamped River job is terminal or has already been pruned. SendWorker is still the primary owner; the compare-and-set store transitions make races safe. The store's guarded MarkFailed is the single terminal write: a row with provider-accept evidence is settled as sent (+ email.sent), a row without evidence — once past the providerEvidenceGrace window — is declared failed with provenance 'local' (correctable, §3.1) + exactly one email.failed.

func NewTerminalReconcileWorker

func NewTerminalReconcileWorker(pool *pgxpool.Pool, store Store) *TerminalReconcileWorker

NewTerminalReconcileWorker builds the periodic safety-net worker.

func (*TerminalReconcileWorker) WithGate added in v1.9.0

WithGate injects the sending-protection gate so an evidence-settled row can also settle its provider attempt (ramp progress, provider-id binding). Reconciliation is settlement-only: it never resubmits and never reserves.

func (*TerminalReconcileWorker) WithMetrics added in v1.2.2

WithMetrics injects the SLI recorder. Chainable; nil keeps the no-op default so metrics stay optional wiring.

func (*TerminalReconcileWorker) Work

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL