storage

package
v0.3.0-20260727164855-... Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 27, 2026 License: Apache-2.0 Imports: 5 Imported by: 0

README

Storage

Pluggable persistence interfaces for SubmitQueue entities (requests, batches, dependents, logs, etc.). Implementations live under extension/storage/<impl>/.

Optimistic locking contract

Entities that support concurrent mutation carry an int32 Version field. Updates are conditional on the version: the write only succeeds if the persisted version matches the caller's expected version. On mismatch, the implementation returns storage.ErrVersionMismatch, which is declared as a retryable infrastructure error so callers can return it without reclassifying it.

Version arithmetic is owned by the controller, not the store. Update methods take both oldVersion (the where-clause guard) and newVersion (the value to write):

UpdateState(ctx, id, oldVersion, newVersion int32, newState entity.RequestState) error

The store performs a pure conditional write — it does not compute oldVersion + 1 internally. This keeps the in-memory entity and the persisted row in sync without the storage layer mutating values the caller didn't supply.

Caller pattern
newVersion := entity.Version + 1
if err := store.UpdateState(ctx, entity.ID, entity.Version, newVersion, newState); err != nil {
    return err // entity.Version unchanged on failure — safe to retry
}
entity.Version = newVersion // only after the write succeeded

The post-success assignment matters whenever the entity is read again later in the same flow. Pre-incrementing in memory before the call is a bug pattern: if the call fails and the caller swallows the error, the in-memory version is now ahead of the database and subsequent updates will fail with ErrVersionMismatch for non-obvious reasons.

Read-after-write consistency

A Get immediately following a successful write (Create/Update) — by the same caller, or a causally-dependent one such as a queue consumer processing a message published after the write committed — must return that write. This is a requirement on every storage implementation, not a condition callers negotiate around: a MySQL primary (including after a promotion) satisfies it, and any other backend (KV, document, etc.) must too.

Controllers must not treat ErrNotFound as "not visible yet, retry." The store interface is intentionally general enough to run over any backend, so a controller has no way to know whether a missing row will appear shortly or does not exist at all — retrying on that assumption just reintroduces, in business logic, the consistency gap the storage contract exists to close. If a Get misses a row that a causally-prior write should already have produced, that is a storage implementation defect: let the error surface as a normal (non-retryable, per platform/errs's default) failure rather than absorbing it with a retryable wrapper.

Key-value contract

Store interfaces are designed for the storage technology space, not for SQL (see the Extensions section of the repo CLAUDE.md): every method must be satisfiable by a plain key-value backend (DynamoDB, Bigtable, an in-memory map) as cheaply as by MySQL. Concretely, a store exposes only get/put/conditional-update by primary key. No lookups by other attributes, no listings filtered server-side, no joins.

The smell test is the index. If implementing a proposed store method in MySQL requires adding a secondary index (KEY idx_*) to the schema, the method is a query-by-attribute in disguise and the contract has left the key-value space — a KV backend would need a global secondary index or a hand-maintained index table to fake it. Treat a new KEY line in a schema diff as a design review flag, not a tuning detail.

Reach for the derived-key pattern instead. When callers need "all X belonging to Y", encode the relationship in the primary key rather than querying for it: derive the key deterministically from the composite identity the caller already holds — for example {parentID}/{hash(child identity)}. Every caller that wants the children can recompute the keys and issue per-key reads; creation under a deterministic key is naturally idempotent (a redelivery finds the existing row); and "at most one row per identity" holds by construction instead of by query discipline.

Domain state is often already the index. Before adding any lookup, check whether an entity the caller already loads enumerates the children — an aggregate that references its parts by ID (e.g. a tree whose paths record their build identities) is the batch→children index, persisted and versioned as domain state. Duplicating that relationship as a database index adds a second source of truth for something the domain already owns.

When neither applies, the reverse lookup is real — give it its own mapping store. In the KV space there is no third mechanism: the only way to look up by an attribute is to make that attribute a primary key somewhere. So promote the relationship to a first-class mapping entity — keyed by the lookup attribute, written by the same flow that creates the source entity with idempotent puts, and rebuildable as a projection if it drifts. ChangeRecord is the in-repo example: it exists so "which requests claimed this change URI" is a by-key read on (queue, URI). Unlike a KEY idx_*, the relationship is visible in the contract and portable to any backend.

Decision path

Take the first branch that applies:

  1. Derive — the caller already holds the composite identity → encode it in the primary key. No new state.
  2. Enumerate — an entity already on the caller's path references the children by ID → that aggregate is the index. Escalate to 3 if the list would grow unbounded or take appends from many concurrent writers (a version-contention hotspot under optimistic locking).
  3. Map — a pipeline controller needs the lookup at runtime → a dedicated mapping store keyed by the attribute.

The bar for 3 is a hot-path need: one mapping per access path, never per attribute, and never for ops/debug queries — run those against SQL replicas directly. But don't contort 1–2 to dodge a legitimate 3; a primary key that hashes half the entity's fields, or an aggregate bloated into listing everything, is the same duplication hidden in a worse place.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrAlreadyExists = errors.New("record already exists")

ErrAlreadyExists is returned by storage implementations when attempting to create a record with an ID that already exists.

View Source
var ErrNotFound = errors.New("record not found")

ErrNotFound is returned by storage implementations when the requested record is not found in the database.

View Source
var ErrVersionMismatch = errs.NewRetryableError(errors.New("version mismatch"))

ErrVersionMismatch is returned by storage implementations when the expected entity version does not match the current version of the object. This is used to implement an optimistic locking mechanism, allowing multiple clients to update the same entity concurrently and either retry or implement idempotent operations. It is intrinsically a retryable infrastructure error.

Functions

func IsNotFound

func IsNotFound(err error) bool

IsNotFound returns true if any error in the error chain is a ErrNotFound.

func WrapNotFound

func WrapNotFound(err error) error

WrapNotFound wraps ErrNotFound with the original error from the storage implementation.

Types

type BatchDependentStore

type BatchDependentStore interface {
	// Get retrieves the batch dependent by batch ID.
	// If the batch contains no dependents, the returned BatchDependent will have an empty Dependents list.
	// Returns ErrNotFound if the batch itself is not found, which should never happen in steady-state system and
	// therefore does not need a special handling.
	Get(ctx context.Context, batchID string) (entity.BatchDependent, error)

	// Create creates a new batch dependent.
	// Returns ErrAlreadyExists if the entry already exists for the given batch ID.
	Create(ctx context.Context, batchDependent entity.BatchDependent) error

	// UpdateDependents updates the dependents of a batch dependent and the version to newVersion
	// if the current persisted version matches oldVersion. If versions do not match, returns ErrVersionMismatch.
	// Version arithmetic is owned by the caller; the store performs a pure conditional write.
	UpdateDependents(ctx context.Context, batchID string, oldVersion, newVersion int32, dependents []string) error
}

BatchDependentStore is an interface that defines methods for managing batch dependent information in the database.

A BatchDependent is a reverse index ("batches that depend on me") paired one-to-one with a Batch. The batch-creation flow always calls Create here before creating the Batch itself, so every active Batch is guaranteed to have a corresponding BatchDependent row. Lookups via Get are only performed for batch IDs returned from the active-batch set, meaning a missing row indicates data corruption or out-of-band manipulation rather than a normal "not found" outcome. ErrNotFound is therefore part of the contract for completeness but is not expected to be returned in steady-state operation.

type BatchStore

type BatchStore interface {
	// Get retrieves a batch by ID. Returns ErrNotFound if the batch is not found.
	Get(ctx context.Context, id string) (entity.Batch, error)

	// Create creates a new batch. The batch must have a unique ID already assigned.
	// Returns ErrAlreadyExists if a batch with the same ID already exists.
	Create(ctx context.Context, batch entity.Batch) error

	// UpdateState updates the state of a batch to newState and the version to newVersion
	// if the current persisted version matches oldVersion. If versions do not match, returns ErrVersionMismatch.
	// Version arithmetic is owned by the caller; the store performs a pure conditional write.
	UpdateState(ctx context.Context, id string, oldVersion, newVersion int32, newState entity.BatchState) error

	// GetByQueueAndStates retrieves all batches that belong to the given queue and are in the given states.
	GetByQueueAndStates(ctx context.Context, queue string, states []entity.BatchState) ([]entity.Batch, error)
}

BatchStore is an interface that defines methods for managing batches in the database.

type BuildStore

type BuildStore interface {
	// Get retrieves a build by ID. Returns ErrNotFound if the build is not found.
	Get(ctx context.Context, id string) (entity.Build, error)

	// Create creates a new build. The build must have a unique ID already assigned.
	// Returns ErrAlreadyExists if a build with the same ID already exists.
	Create(ctx context.Context, build entity.Build) error

	// UpdateStatus updates the status of a build.
	UpdateStatus(ctx context.Context, id string, newStatus entity.BuildStatus) error
}

BuildStore is an interface that defines methods for managing builds in the database.

type ChangeStore

type ChangeStore interface {
	// Create persists a single ChangeRecord (identity + Details) in one write. A
	// primary-key conflict on (Queue, URI, RequestID) is silently ignored, which makes
	// the call idempotent under queue redeliveries of the same request (first write
	// wins). Records belonging to different requests do not conflict on the PK —
	// cross-request overlap is detected by GetByURI, not by Create.
	Create(ctx context.Context, record entity.ChangeRecord) error

	// GetByURI returns every ChangeRecord for the given (queue, uri). Multiple
	// requests can have claimed the same URI over time, so the slice may have
	// any number of entries; an empty slice means no claim has ever been
	// recorded for this URI in this queue.
	//
	// The store does not filter by request_id or by the owning request's
	// state — callers that want to skip self filter by RequestID, and callers
	// that want only live owners consult RequestStore for liveness.
	GetByURI(ctx context.Context, queue string, uri string) ([]entity.ChangeRecord, error)
}

ChangeStore manages per-URI claim records for in-flight land requests. Each row records that a specific URI was claimed by a specific request, scoped to a queue. The (Queue, URI, RequestID) triple is the immutable identity of a record, and a record's Details are captured at claim time and never updated — records are immutable.

The interface is intentionally per-record / per-URI so that any backend (SQL, DynamoDB, Bigtable, …) can implement it without needing batch-atomicity or multi-key query support. Callers loop when they have multiple URIs to claim or check; the typical request has a small number of URIs (a single PR or a short stack), so the loop overhead is negligible.

type RequestLogStore

type RequestLogStore interface {
	// Insert appends a new request log record. Timestamps should be generated by the caller and not modified by the implementation.
	Insert(ctx context.Context, log entity.RequestLog) error

	// List retrieves all request log records for a given request ID, ordered by timestamp ascending.
	// Returns ErrNotFound if no records exist for the given request ID.
	List(ctx context.Context, requestID string) ([]entity.RequestLog, error)
}

RequestLogStore is an interface that defines methods for managing request log records in an append-only database. Request logs are used to reconcile request statuses with eventual consistency into a separate database from RequestStore.

type RequestQueueSummaryCursor

type RequestQueueSummaryCursor struct {
	// ReceivedAtMs is the receipt timestamp of the last item from the previous page.
	ReceivedAtMs int64
	// RequestID is the request ID of the last item from the previous page.
	RequestID string
}

RequestQueueSummaryCursor is the exclusive keyset boundary for a descending queue-summary query.

type RequestQueueSummaryQuery

type RequestQueueSummaryQuery struct {
	// Queue is the exact queue partition to scan.
	Queue string
	// ReceivedAtOrAfterMs is the inclusive lower receipt-time bound.
	ReceivedAtOrAfterMs int64
	// ReceivedBeforeMs is the exclusive upper receipt-time bound.
	ReceivedBeforeMs int64
	// Cursor is an exclusive continuation boundary when HasCursor is true.
	Cursor RequestQueueSummaryCursor
	// HasCursor selects whether Cursor participates in the query.
	HasCursor bool
	// Limit is the maximum number of rows returned and must be positive.
	Limit int
}

RequestQueueSummaryQuery specifies one bounded queue-summary page query.

type RequestQueueSummaryStore

type RequestQueueSummaryStore interface {
	// Create inserts summary and returns ErrAlreadyExists when its full primary key already exists.
	Create(ctx context.Context, summary entity.RequestQueueSummary) error

	// Get returns the row identified by its full primary key, or ErrNotFound when absent.
	Get(ctx context.Context, queue string, receivedAtMs int64, requestID string) (entity.RequestQueueSummary, error)

	// Update conditionally replaces mutable fields when the persisted projection version equals oldVersion.
	// The store writes newVersion exactly as supplied and returns ErrVersionMismatch when the guard does not match.
	Update(ctx context.Context, summary entity.RequestQueueSummary, oldVersion, newVersion int32) error

	// List returns at most query.Limit rows ordered by received_at_ms descending, then request_id descending.
	List(ctx context.Context, query RequestQueueSummaryQuery) ([]entity.RequestQueueSummary, error)
}

RequestQueueSummaryStore persists the queue-ordered request projection.

type RequestStore

type RequestStore interface {
	// Get retrieves a land request by ID. Returns ErrNotFound if the request is not found.
	Get(ctx context.Context, id string) (entity.Request, error)

	// Create creates a new land request. The request must have a unique ID already assigned.
	// Returns ErrAlreadyExists if a request with the same ID already exists.
	Create(ctx context.Context, request entity.Request) error

	// UpdateState updates the state of a land request to newState and the version to newVersion
	// if the current persisted version matches oldVersion. If versions do not match, returns ErrVersionMismatch.
	// Version arithmetic is owned by the caller; the store performs a pure conditional write.
	UpdateState(ctx context.Context, id string, oldVersion, newVersion int32, newState entity.RequestState) error
}

RequestStore is an interface that defines methods for managing land requests in the database.

type RequestSummaryStore

type RequestSummaryStore interface {
	// Create inserts summary and returns ErrAlreadyExists when RequestID already exists.
	// The caller owns retry identity and decides whether an existing row is an identical retry or a conflict.
	Create(ctx context.Context, summary entity.RequestSummary) error

	// Get returns the summary for requestID, or ErrNotFound when absent.
	Get(ctx context.Context, requestID string) (entity.RequestSummary, error)

	// Update conditionally replaces the mutable status fields when the persisted projection version equals oldVersion.
	// The store writes newVersion exactly as supplied and returns ErrVersionMismatch when the guard does not match.
	Update(ctx context.Context, summary entity.RequestSummary, oldVersion, newVersion int32) error
}

RequestSummaryStore persists the authoritative request-ID materialized view.

type RequestURIStore

type RequestURIStore interface {
	// Create inserts mapping and returns ErrAlreadyExists when its full primary key already exists.
	Create(ctx context.Context, mapping entity.RequestURI) error

	// ListByURI returns at most limit mappings ordered by received_at_ms descending, then request_id descending.
	ListByURI(ctx context.Context, changeURI string, limit int) ([]entity.RequestURI, error)
}

RequestURIStore persists the immutable change-URI reverse mapping.

type Storage

type Storage interface {
	// GetRequestStore returns the RequestStore instance.
	GetRequestStore() RequestStore

	// GetChangeStore returns the ChangeStore instance.
	GetChangeStore() ChangeStore

	// GetBatchStore returns the BatchStore instance.
	GetBatchStore() BatchStore

	// GetBatchDependentStore returns the BatchDependentStore instance.
	GetBatchDependentStore() BatchDependentStore

	// GetBuildStore returns the BuildStore instance.
	GetBuildStore() BuildStore

	// GetRequestLogStore returns the RequestLogStore instance.
	GetRequestLogStore() RequestLogStore

	// GetRequestSummaryStore returns the RequestSummaryStore instance.
	GetRequestSummaryStore() RequestSummaryStore

	// GetRequestQueueSummaryStore returns the RequestQueueSummaryStore instance.
	GetRequestQueueSummaryStore() RequestQueueSummaryStore

	// GetRequestURIStore returns the RequestURIStore instance.
	GetRequestURIStore() RequestURIStore

	// Close closes the storage and all underlying connections. Should only be called once at the end of the program.
	Close() error
}

Storage is a factory interface that aggregates all entity stores into a single injectable dependency.

Directories

Path Synopsis
Package mock is a generated GoMock package.
Package mock is a generated GoMock package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL