aws

package
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 24, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Overview

Package aws is the AWS-native Lambda/SQS/DynamoDB runtime for anyhook: it implements the go/core ports (core.Transport, core.StateStore, core.Scheduler, core.HTTPClient) against aws-sdk-go-v2 directly, and wires Lambda handler factories for the ingest, delivery, and retry-sweeper entrypoints.

It mirrors packages/aws/src in the TypeScript implementation, which remains the source of truth for behavior:

  • config.go <-> config.ts (Env/config, single-table key helpers)
  • transport_sqs.go <-> transport-sqs.ts (SQS Transport + per-record dispatch)
  • state_dynamo.go <-> state-dynamo.ts (single-table StateStore, G7 tenant scoping)
  • scheduler.go <-> scheduler.ts (900s SQS-delay vs. due_at row split, D2)
  • sweeper.go <-> sweeper.ts (lease-claim-before-send due_at sweep)
  • handlers.go <-> handlers.ts (Lambda entrypoints)
  • adapter.go <-> adapter.ts (port wiring)
  • http_fetch.go <-> http-fetch.ts (HTTPClient over net/http)

Deliberate Go-side difference from TS: no cross-module anyq dependency

The TypeScript transport-sqs.ts composes @anyq/sqs's SQS producer for send() (P1 in the TS design: transport drivers are named/owned by anyq, never anyhook). This Go package does NOT take an equivalent dependency on anyq's Go module (github.com/sns45/anyq/go/sqs). Instead transport_sqs.go calls aws-sdk-go-v2's SQS client directly. This is a considered choice, not a shortcut:

  • anyq's Go SQS producer/consumer types are built for anyq's own multi-broker Producer/Consumer contract (core.BaseProducer, structured logging, health checks, FIFO group/dedup handling). Pulling that whole surface in to get one SendMessage call is a heavier dependency edge than aws-sdk-go-v2 itself, and none of the extra surface is exercised here.
  • go/core's ports (Transport, StateStore, Scheduler, HTTPClient) are already plain Go interfaces satisfied structurally (no base-type wiring needed), so package aws only needs a `DynamoDBAPI`/`SQSAPI` slice of the real SDK clients -- trivial to fake in tests without anyq's own test doubles.
  • Behaviorally nothing changes: SendMessage today, durable at-least-once enqueue (G5), no in-process consumer loop (Lambda's SQS event-source mapping is already the push consumer, exactly as in TS). This is purely an implementation-dependency choice, not a wire-format or semantics choice -- the Go SQS queue and the TS SQS queue are independently deployed per runtime and never need to interoperate at the message-bus layer.

Fakes over LocalStack

Every port implementation here depends only on DynamoDBAPI/SQSAPI (this package's own minimal client interfaces, see config.go), never on *dynamodb.Client/*sqs.Client concretely. Tests inject in-memory fakes that evaluate the small ConditionExpression/UpdateExpression vocabulary this adapter actually emits (mirroring the TS test suite's fake-dynamo.ts) -- no LocalStack or other live AWS dependency is required to exercise the conditional-write, lease-claim, or partial-batch-failure paths.

Index

Constants

View Source
const DefaultDueAtIndexName = "due_at-index"

DefaultDueAtIndexName names the GSI indexing due_at (partition key gsi1pk, sort key gsi1sk) that RunSweeper queries. Mirrors TS DEFAULT_DUE_AT_INDEX_NAME.

View Source
const MaxPayloadBytes int64 = 200 * 1024

MaxPayloadBytes is the max event payload for the AWS runtime. The SQS message body is the JSON encoding of a Message -- the payload PLUS the envelope (ids, tenant, timestamps) -- and SQS caps a body at 256KB. Cap the payload below that (~200KB) so a near-limit payload plus envelope still fits and enqueues. Mirrors TS AWS_MAX_PAYLOAD_BYTES.

View Source
const SchedGSIPK = "SCHED"

SchedGSIPK is the constant GSI partition key value every SCHED# row is written under. A single logical partition is an accepted v0.1 scale tradeoff (see sweeper.go); revisit with a bucketed key if volume demands it. Mirrors TS SCHED_GSI_PK.

View Source
const SchedTTLGraceSeconds int64 = 7 * 24 * 60 * 60

SchedTTLGraceSeconds is how long a swept SCHED# row survives before DynamoDB TTL reclaims it. GC only -- never the scheduling trigger (D2). Mirrors TS SCHED_TTL_GRACE_SECONDS.

View Source
const SqsMaxDelayMs int64 = 900_000

SqsMaxDelayMs is SQS's hard cap on native DelaySeconds. Retries due sooner than this use SQS delay directly (D2). Mirrors TS SQS_MAX_DELAY_MS.

View Source
const SweeperLeaseSeconds int64 = 300

SweeperLeaseSeconds is the lease duration for a sweeper claim. A claim stamps claimed_at; a row whose claim is older than this is re-claimable. This makes the claim crash/failure-safe: if SendMessage fails after a successful claim, the lease expires and a later sweep re-enqueues the retry rather than losing it (G5). Must exceed the sweep interval (60s) so an in-flight claim isn't re-swept early. Mirrors TS SWEEPER_LEASE_SECONDS.

Variables

This section is empty.

Functions

func AttemptCounterSK

func AttemptCounterSK(messageID string) string

AttemptCounterSK is the sort key for a message's atomic attempt-sequence counter.

func AttemptSK

func AttemptSK(messageID string, seq int64) string

AttemptSK is the sort key for one delivery attempt (zero-padded seq preserves insertion order on a begins_with query).

func AttemptSKPrefix

func AttemptSKPrefix(messageID string) string

AttemptSKPrefix is the sort-key prefix for querying every attempt of a message.

func CircuitSK

func CircuitSK(endpointID string) string

CircuitSK is the sort key for an endpoint's circuit-breaker record.

func DlqCounterSK

func DlqCounterSK() string

DlqCounterSK is the sort key for the tenant's atomic DLQ-sequence counter.

func DlqSK

func DlqSK(seq int64) string

DlqSK is the sort key for one dead-lettered message (zero-padded global-per-tenant seq).

func EndpointSK

func EndpointSK(endpointID string) string

EndpointSK is the sort key for an endpoint row.

func IdemSK

func IdemSK(idemKey string) string

IdemSK is the sort key for an idempotency record.

func MessageIndexSK

func MessageIndexSK(messageID string) string

MessageIndexSK is the sort key for a message's write-only endpoint index row.

func MessageSK

func MessageSK(messageID string) string

MessageSK is the sort key for a message row.

func NewDeliverHandler

func NewDeliverHandler(env Env, opts AdapterOptions) func(ctx context.Context, event events.SQSEvent) (events.SQSEventResponse, error)

NewDeliverHandler builds the SQS event-source Lambda handler: one invocation per batch. Each record drives one engine delivery attempt via SqsTransport.DispatchRecords; a failing record is reported through BatchItemFailures so only it is redelivered (requires FunctionResponseTypes: ReportBatchItemFailures configured on the event source mapping). Mirrors TS createDeliverHandler.

func NewFetchHTTPClient

func NewFetchHTTPClient(opts FetchHTTPClientOptions) core.HTTPClient

NewFetchHTTPClient builds a core.HTTPClient backed by net/http. Redirects are NOT followed (http.ErrUseLastResponse): a 3xx on a webhook target is treated as a delivery failure (§8), reported with its real status code (no TS-style opaqueredirect normalization needed in Go -- the underlying transport hands back the actual 3xx response as-is). Deadline-exceeded errors map to StatusTimeout; every other transport error maps to StatusNetwork. Mirrors TS createFetchHttpClient (http-fetch.ts).

func NewIngestHandler

NewIngestHandler builds the API Gateway (HTTP API, payload format 2.0) ingest Lambda handler: parses the JSON request body and calls Engine.Send.

The TS ingest handler (handlers.ts) mounts core's framework-agnostic createPortalRouter, which also serves endpoint CRUD and the delivery-log query surface; that portal router has not been ported to go/core (only the engine's own Send/ProcessMessage surface has), so this handler covers just the send path -- the one every producer integration needs -- rather than the full portal. See the task report for this scope note.

func NewSweeperHandler

func NewSweeperHandler(env Env) func(ctx context.Context) error

NewSweeperHandler builds the scheduled (EventBridge, fixed 60s rate per D2/ADR-0001) Lambda handler: sweeps due SCHED# rows and enqueues them to SQS. Mirrors TS createSweeperHandler.

func RateSK

func RateSK(endpointID string) string

RateSK is the sort key for an endpoint's rate-limit token bucket.

func SchedSK

func SchedSK(messageID string) string

SchedSK is the sort key for a scheduled-in-the-future retry row.

func TenantPK

func TenantPK(tenant string) string

TenantPK is the partition key every item for tenant lives under (G7: tenant scoping by construction).

Types

type Adapter

type Adapter struct {
	Transport *SqsTransport
	State     *DynamoStateStore
	Scheduler *DynamoScheduler
	HTTP      core.HTTPClient
	Env       Env
}

Adapter bundles the four AWS-backed engine ports plus the resolved Env they share. Mirrors TS AwsAdapter (adapter.ts).

func BuildEngine

func BuildEngine(env Env, opts AdapterOptions) (*core.Engine, *Adapter)

BuildEngine builds a ready-to-use core.Engine wired to the AWS adapter (used by every Lambda handler factory in handlers.go). Mirrors TS buildEngine.

func NewAdapter

func NewAdapter(env Env, opts AdapterOptions) *Adapter

NewAdapter builds the four engine ports for the AWS runtime from an already-resolved Env (see ResolveEnv). Mirrors TS createAdapter.

type AdapterOptions

type AdapterOptions struct {
	// URLPolicy is applied by the HTTP client's pre-dispatch re-check.
	// nil -> core.DefaultURLPolicy(core.URLPolicyOptions{}). Mirrors TS
	// CreateAdapterOptions.urlPolicy.
	URLPolicy core.URLPolicy
	// Telemetry is an optional observability sink (§11) threaded into the
	// engine; nil -> core.NoopTelemetry. The current TS adapter.ts snapshot
	// does not yet forward a telemetry option through buildEngine -- this
	// Go port adds the wiring per the same zero-behavioral-drift default
	// (NoopTelemetry) established by the core engine backport.
	Telemetry core.Telemetry
}

AdapterOptions configures NewAdapter/BuildEngine.

type CircuitWriteConflictError

type CircuitWriteConflictError struct {
	Tenant     string
	EndpointID string
}

CircuitWriteConflictError is returned by DynamoStateStore.PutCircuit when a concurrent writer already advanced the circuit's version (optimistic concurrency, D2). Mirrors TS CircuitWriteConflictError.

func (*CircuitWriteConflictError) Error

func (e *CircuitWriteConflictError) Error() string

Error implements the error interface.

type Config

type Config struct {
	TableName string
	QueueURL  string
	// DueAtIndexName names the GSI indexing due_at (partition key gsi1pk,
	// sort key gsi1sk). "" -> DefaultDueAtIndexName.
	DueAtIndexName string
	Region         string
	// Endpoint overrides both clients' endpoint (e.g. LocalStack
	// http://localhost:4566). Ignored for a client that was injected directly.
	Endpoint string
	// DDB is a pre-built (or faked) DynamoDB client. nil -> built from
	// Region/Endpoint via the default AWS credential chain.
	DDB DynamoDBAPI
	// SQS is a pre-built (or faked) SQS client. nil -> built from
	// Region/Endpoint via the default AWS credential chain.
	SQS SQSAPI
}

Config is deploy-time configuration for the AWS adapter. TableName/QueueURL name the single DynamoDB table and SQS queue this adapter owns. DDB/SQS can be pre-built (or faked) and injected directly -- required for tests and useful for a custom endpoint (e.g. LocalStack) or credential chain.

func ConfigFromEnv

func ConfigFromEnv(getenv func(string) string) (Config, error)

ConfigFromEnv reads Config from the conventional Lambda environment variables (used by the deployed handlers' entrypoint). getenv is typically os.Getenv; injected so tests don't need real process env vars.

type DynamoDBAPI

type DynamoDBAPI interface {
	GetItem(ctx context.Context, params *dynamodb.GetItemInput, optFns ...func(*dynamodb.Options)) (*dynamodb.GetItemOutput, error)
	PutItem(ctx context.Context, params *dynamodb.PutItemInput, optFns ...func(*dynamodb.Options)) (*dynamodb.PutItemOutput, error)
	UpdateItem(ctx context.Context, params *dynamodb.UpdateItemInput, optFns ...func(*dynamodb.Options)) (*dynamodb.UpdateItemOutput, error)
	DeleteItem(ctx context.Context, params *dynamodb.DeleteItemInput, optFns ...func(*dynamodb.Options)) (*dynamodb.DeleteItemOutput, error)
	Query(ctx context.Context, params *dynamodb.QueryInput, optFns ...func(*dynamodb.Options)) (*dynamodb.QueryOutput, error)
}

DynamoDBAPI is the subset of *dynamodb.Client this package calls, expressed as an interface so tests can inject an in-memory fake instead of a live DynamoDB table (see doc.go "Fakes over LocalStack"). The real *dynamodb.Client satisfies this interface as-is.

type DynamoScheduler

type DynamoScheduler struct {
	// contains filtered or unexported fields
}

DynamoScheduler is core.Scheduler for AWS (D2 -- "the 900-second split, no delay-chaining"): a retry due within SQS's 900s DelaySeconds cap is sent immediately with that native delay; anything further out is persisted as a SCHED# row carrying a due_at (epoch seconds) attribute, indexed by the due_at GSI, for RunSweeper to discover on its fixed interval. Per P2, the DynamoDB item IS the schedule; this type never chains multiple delayed sends to cover a longer delay. Mirrors TS DynamoScheduler (scheduler.ts).

func NewDynamoScheduler

func NewDynamoScheduler(env Env, clock core.Clock) *DynamoScheduler

NewDynamoScheduler builds a DynamoScheduler over env. clock may be nil (falls back to the wall clock).

func (*DynamoScheduler) ScheduleRetry

func (s *DynamoScheduler) ScheduleRetry(ctx context.Context, m core.Message, at int64) error

ScheduleRetry implements core.Scheduler.

type DynamoStateStore

type DynamoStateStore struct {
	// contains filtered or unexported fields
}

DynamoStateStore is core.StateStore on a single DynamoDB table (PK = TENANT#<tenant>, item-type-prefixed SK -- see config.go for the full key design). Every method takes/derives tenant first and every key is built from it, so no query can address another tenant's rows (G7). Idempotency uses a conditional PutItem (attribute_not_exists(SK)) so a racing duplicate RecordEvent is detected atomically.

Circuit writes carry an internal _version attribute checked via ConditionExpression. NOTE: this guards only PutCircuit's OWN read->write window, not the engine's full GetCircuit -> OnFailure -> PutCircuit cycle (GetCircuit intentionally strips _version, so the engine can't round-trip it). Under heavy same-endpoint concurrency a consecutive-failure increment can therefore still be lost -- bounded impact: the breaker may trip slightly late, never data corruption or a miscount toward a false trip. A detected conflict returns *CircuitWriteConflictError, which the SQS consumer (via a returned error from ProcessMessage) turns into a batchItemFailures redelivery (at-least-once, self-healing). Mirrors TS DynamoStateStore (state-dynamo.ts).

func NewDynamoStateStore

func NewDynamoStateStore(env Env) *DynamoStateStore

NewDynamoStateStore builds a DynamoStateStore over env.

func (*DynamoStateStore) AddToDlq

func (s *DynamoStateStore) AddToDlq(ctx context.Context, m core.Message, reason core.DlqReason) error

AddToDlq implements core.StateStore.

func (*DynamoStateStore) AppendAttempt

func (s *DynamoStateStore) AppendAttempt(ctx context.Context, a core.Attempt) error

AppendAttempt implements core.StateStore.

func (*DynamoStateStore) CreateEndpoint

CreateEndpoint implements core.StateStore.

func (*DynamoStateStore) DeleteEndpoint

func (s *DynamoStateStore) DeleteEndpoint(ctx context.Context, tenant, endpointID string) error

DeleteEndpoint implements core.StateStore.

func (*DynamoStateStore) FinalizeEvent

func (s *DynamoStateStore) FinalizeEvent(ctx context.Context, tenant, idemKey, eventID string, messageCount int) error

FinalizeEvent implements core.StateStore.

func (*DynamoStateStore) GetCircuit

func (s *DynamoStateStore) GetCircuit(ctx context.Context, tenant, endpointID string) (core.CircuitRecord, error)

GetCircuit implements core.StateStore.

func (*DynamoStateStore) GetEndpoint

func (s *DynamoStateStore) GetEndpoint(ctx context.Context, tenant, endpointID string) (*core.Endpoint, error)

GetEndpoint implements core.StateStore.

func (*DynamoStateStore) GetMessage

func (s *DynamoStateStore) GetMessage(ctx context.Context, tenant, messageID string) (*core.Message, error)

GetMessage implements core.StateStore.

func (*DynamoStateStore) GetRateBucket

func (s *DynamoStateStore) GetRateBucket(ctx context.Context, tenant, endpointID string) (*core.RateBucket, error)

GetRateBucket implements core.StateStore.

func (*DynamoStateStore) ListDeliveries

func (s *DynamoStateStore) ListDeliveries(ctx context.Context, q core.DeliveryQuery) ([]core.DeliveryRow, error)

ListDeliveries implements core.StateStore.

func (*DynamoStateStore) ListDlq

func (s *DynamoStateStore) ListDlq(ctx context.Context, tenant, endpointID string) ([]core.DlqEntry, error)

ListDlq implements core.StateStore.

func (*DynamoStateStore) ListEndpoints

func (s *DynamoStateStore) ListEndpoints(ctx context.Context, tenant string) ([]core.Endpoint, error)

ListEndpoints implements core.StateStore.

func (*DynamoStateStore) MatchEndpoints

func (s *DynamoStateStore) MatchEndpoints(ctx context.Context, tenant, eventType string) ([]core.Endpoint, error)

MatchEndpoints implements core.StateStore.

func (*DynamoStateStore) PutCircuit

func (s *DynamoStateStore) PutCircuit(ctx context.Context, tenant, endpointID string, rec core.CircuitRecord) error

PutCircuit implements core.StateStore.

func (*DynamoStateStore) PutMessage

func (s *DynamoStateStore) PutMessage(ctx context.Context, m core.Message) error

PutMessage implements core.StateStore.

func (*DynamoStateStore) PutRateBucket

func (s *DynamoStateStore) PutRateBucket(ctx context.Context, tenant, endpointID string, bucket core.RateBucket) error

PutRateBucket implements core.StateStore. An approximate limiter: a plain put is sufficient (no OCC needed -- a lost update at most lets one extra delivery through, which throttling tolerates).

func (*DynamoStateStore) RecordEvent

func (s *DynamoStateStore) RecordEvent(ctx context.Context, tenant, eventID, idemKey string) (core.EventRecordResult, error)

RecordEvent implements core.StateStore.

func (*DynamoStateStore) RotateSecret

func (s *DynamoStateStore) RotateSecret(ctx context.Context, tenant, endpointID string) (string, error)

RotateSecret implements core.StateStore. Keeps the previous primary secret valid during a dual-secret rotation window.

func (*DynamoStateStore) UpdateEndpoint

func (s *DynamoStateStore) UpdateEndpoint(ctx context.Context, tenant, endpointID string, patch core.UpdateEndpointPatch) (core.Endpoint, error)

UpdateEndpoint implements core.StateStore.

type Env

type Env struct {
	TableName      string
	QueueURL       string
	DueAtIndexName string
	DDB            DynamoDBAPI
	SQS            SQSAPI
}

Env is the resolved runtime environment: config plus the concrete client interfaces every port implementation in this package shares.

func ResolveEnv

func ResolveEnv(ctx context.Context, cfg Config) (Env, error)

ResolveEnv builds the resolved Env from Config, constructing default SDK clients only for those not already injected.

type FetchHTTPClientOptions

type FetchHTTPClientOptions struct {
	// URLPolicy is re-run immediately before dispatch (belt-and-suspenders
	// SSRF; DNS-rebinding note: net/http does not expose the resolved IP
	// before connecting, so this is the static hostname policy re-applied).
	// nil disables the pre-dispatch check (the engine's own DeliverOnce check
	// still applies).
	URLPolicy core.URLPolicy
	// SnippetLimit caps the response-body snippet retained (bytes). 0 -> 2048.
	SnippetLimit int
}

FetchHTTPClientOptions configures NewFetchHTTPClient.

type SQSAPI

type SQSAPI interface {
	SendMessage(ctx context.Context, params *sqs.SendMessageInput, optFns ...func(*sqs.Options)) (*sqs.SendMessageOutput, error)
}

SQSAPI is the subset of *sqs.Client this package calls. The real *sqs.Client satisfies this interface as-is.

type SqsTransport

type SqsTransport struct {
	// contains filtered or unexported fields
}

SqsTransport is core.Transport backed directly by AWS SQS (see doc.go for why this package calls aws-sdk-go-v2's SQS client directly rather than composing anyq's Go SQS producer/consumer).

There is no consumer LOOP here: on AWS, delivery runs on Lambda's native SQS event-source mapping, which is already a push consumer. DispatchRecords is the seam NewDeliverHandler calls once per invocation, mapping each record 1:1 to one Engine.ProcessMessage call. A record whose handler returns an error is reported back via BatchItemFailures (Lambda's ReportBatchItemFailures partial-batch-failure mechanism) so ONLY that record is redelivered -- one record's failure must never fail the whole invocation, which under Lambda's default SQS semantics would redeliver every record in the batch and defeat per-message isolation (G3). Mirrors TS SqsTransport (transport-sqs.ts).

func NewSqsTransport

func NewSqsTransport(env Env) *SqsTransport

NewSqsTransport builds an SqsTransport over env.

func (*SqsTransport) DispatchRecords

func (t *SqsTransport) DispatchRecords(ctx context.Context, event events.SQSEvent, handler func(ctx context.Context, m core.Message) error) (events.SQSEventResponse, error)

DispatchRecords maps one SQS event batch (as delivered to a Lambda by the event-source mapping) to per-record engine deliveries, returning BatchItemFailures for any record whose handler returned an error or whose body failed to parse. handler overrides the Subscribe-registered handler when non-nil (used by tests and by handlers that want an explicit handler without going through Subscribe first).

func (*SqsTransport) Send

func (t *SqsTransport) Send(ctx context.Context, m core.Message) error

Send implements core.Transport: durably enqueues m onto the configured queue.

func (*SqsTransport) Subscribe

func (t *SqsTransport) Subscribe(_ context.Context, handler func(ctx context.Context, m core.Message) error) error

Subscribe implements core.Transport: registers the engine's ProcessMessage handler, invoked per-record by DispatchRecords.

type SweeperResult

type SweeperResult struct {
	// Due is the count of rows found due (due_at <= now) and not currently
	// lease-held, across all pages this run scanned.
	Due int
	// Claimed is the count of rows this run successfully claimed (won the
	// lease compare-and-set).
	Claimed int
	// Sent is the count of rows sent to SQS and cleaned up.
	Sent int
}

SweeperResult summarizes one RunSweeper pass. Mirrors TS SweeperResult.

func RunSweeper

func RunSweeper(ctx context.Context, env Env, now int64) (SweeperResult, error)

RunSweeper discovers every SCHED# row whose due_at <= now via the due_at GSI, lease-claims each with a conditional UpdateItem (stamp claimed_at) BEFORE sending it to SQS, then deletes the row.

The lease is what makes two concurrent sweeper invocations safe AND crash/failure-safe:

  • Only one invocation can win the conditional write for a given row, so the common path enqueues a message at most once (the loser's ConditionalCheckFailedException is caught and skipped).
  • Crucially, the claim is a LEASE, not a permanent flag: if SendMessage fails after a successful claim (SQS throttling/outage), the row is neither sent nor deleted, but its claimed_at lease expires after SweeperLeaseSeconds, so a later sweep re-claims and re-enqueues it. A due retry is therefore never lost (G5) -- at worst delayed by one lease interval, or (rarely, under a claim race that overlaps a lease boundary) delivered twice, which the stable webhook-id dedupes.

The attribute_exists(SK) guard prevents an UpdateItem from resurrecting a row another sweep already sent+deleted (no orphan rows). DynamoDB's ttl is garbage-collection only; this sweeper -- never TTL expiry -- is the scheduling trigger. Mirrors TS runSweeper (sweeper.ts) exactly, including the send-then-delete-under-lease ordering.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL