Documentation
¶
Overview ¶
Package metadata is the persistence seam for file records. The Repository interface is the only contract above this layer; the pgx implementation (Phase 1) is the sole place that issues SQL.
Index ¶
Constants ¶
const ( IdempotencyPending = "pending" IdempotencyCompleted = "completed" )
Idempotency-key lifecycle states.
const ( ReservedTagPrefix = "cm:" TagArchiveID = ReservedTagPrefix + "archive_id" // on an extracted entry: the archive it came from TagArchivePath = ReservedTagPrefix + "archive_path" // on an extracted entry: its full path inside the archive TagArchiveIndex = ReservedTagPrefix + "archive_index" // on an extracted entry: its directory index (names are not unique) TagArchive = ReservedTagPrefix + "archive" // on an archive row: TagArchiveMarker once it has been extracted )
Reserved tag keys. The cm: prefix is owned by the service — written by archive extraction, never by a client — across the whole tag namespace. The api layer rejects it on upload and PATCH, and a replace-mode UpdateMetadata carries the keys across so a client cannot strip them by accident.
const TagArchiveMarker = "true"
TagArchiveMarker is the value of TagArchive. A string, not a boolean, because the list endpoint's tag.<k>=<v> filter is string-typed: this is what makes GET /files?tag.cm:archive=true list the archives.
Variables ¶
var ErrJobNotFound = errors.New("metadata: extraction job not found")
ErrJobNotFound is returned when no extraction job matches — or, from the worker-side writes, when the row is no longer owned by the caller's attempt.
var ErrNotFound = errors.New("metadata: file not found")
ErrNotFound is returned when no file record matches the query.
Functions ¶
This section is empty.
Types ¶
type EntryScope ¶ added in v1.3.0
type EntryScope string
EntryScope selects whether extracted archive entries appear in a List. The parameter is about *entries*, not archives: the archives themselves are found by their TagArchive marker.
const ( EntriesInclude EntryScope = "include" // the default: files, archives and entries together EntriesExclude EntryScope = "exclude" // everything but extracted entries EntriesOnly EntryScope = "only" // extracted entries alone )
type File ¶
type File struct {
ID string
StorageKey string
Filename string
ContentType string
SizeBytes int64
ChecksumSHA256 string // hex sha-256
Metadata map[string]any // JSONB tags
CreatedAt time.Time
UpdatedAt *time.Time // nil until the first metadata update — a pristine, unmodified file has none
DeletedAt *time.Time // soft delete; nil while live
}
File is the metadata record for a stored object. It mirrors the files table.
type IdempotencyRecord ¶
type IdempotencyRecord struct {
Status string // IdempotencyPending | IdempotencyCompleted
FileID *string // the produced file id; nil while pending
}
IdempotencyRecord is the stored state of an upload idempotency key.
type Job ¶ added in v1.3.0
type Job struct {
ID string
ArchiveID string
Status JobStatus
// Selection is the caller's entry indexes. nil means every selectable
// entry; an empty, non-nil slice selects nothing. The distinction is
// preserved through storage (null vs []).
Selection []int
CancelRequested bool
// Attempt counts claims. It is the fencing token on every worker write.
Attempt int
// Progress of the current run: Done entries processed of Total intended.
// A job resumed after an interruption counts only what remained.
ProgressDone int
ProgressTotal int
// Summary is the extraction summary as JSON, once terminal. nil when the
// run never produced one (cancelled before it started, or failed before
// the archive could be opened).
Summary json.RawMessage
// Error is the reason when Status is JobFailed.
Error string
HeartbeatAt *time.Time // set while running; nil otherwise
CreatedAt time.Time
UpdatedAt time.Time
FinishedAt *time.Time // set once terminal
}
Job is one extraction job. It mirrors the extraction_jobs table.
type JobRepository ¶ added in v1.3.0
type JobRepository interface {
// CreateJob inserts job as pending. created=false with activeID set means
// another job for the same archive is still pending or running — the
// unique index refused the insert. activeID may be empty if that job
// finished between the conflict and the lookup; the caller simply retries.
// ErrNotFound when the archive row no longer exists.
CreateJob(ctx context.Context, job *Job) (created bool, activeID string, err error)
// GetJob returns the job by id, or ErrJobNotFound.
GetJob(ctx context.Context, id string) (*Job, error)
// RequestCancel asks a job to stop. A pending job becomes cancelled at
// once; a running one has its flag set for the worker to honour after
// the current entry; a terminal one is left as it is. Returns the job's
// state after the request, or ErrJobNotFound.
RequestCancel(ctx context.Context, id string) (*Job, error)
// ClaimJob moves the oldest pending job to running for this caller,
// bumping Attempt, and returns it — or nil, nil when nothing is pending.
// Safe across replicas: concurrent callers never claim the same row.
ClaimJob(ctx context.Context) (*Job, error)
// HeartbeatJob records progress and liveness for a running job in one
// round trip and reports whether cancellation has been requested.
// ErrJobNotFound when the row is not running under this attempt — the
// caller has lost ownership and must stop.
HeartbeatJob(ctx context.Context, id string, attempt, done, total int) (cancelRequested bool, err error)
// FinishJob records a terminal result for a running job. ErrJobNotFound
// when the row is not running under this attempt.
FinishJob(ctx context.Context, id string, attempt int, result JobResult) error
// ParkJob returns a running job to pending — a worker shutting down —
// keeping its progress. ErrJobNotFound when the row is not running under
// this attempt.
ParkJob(ctx context.Context, id string, attempt, done, total int) error
// ReapJobs returns running jobs that have gone longer than ttl without a
// heartbeat to pending, so the next pickup resumes them, and reports how
// many. Judged on the repository's own clock, which also stamps the
// heartbeat: this is the one comparison where a skew between processes
// would hand a live run to a second worker.
ReapJobs(ctx context.Context, ttl time.Duration) (int, error)
// CountPendingJobs is the queue depth.
CountPendingJobs(ctx context.Context) (int, error)
// PurgeFinishedJobs deletes terminal jobs that finished before the cutoff
// (retention), returning how many. Called by GC.
PurgeFinishedJobs(ctx context.Context, before time.Time) (int, error)
}
JobRepository persists extraction jobs. It is a sibling of Repository — the pgx and in-memory implementations satisfy both — kept separate because jobs are a distinct bounded concern and Repository is already ten methods.
type JobResult ¶ added in v1.3.0
type JobResult struct {
Status JobStatus // JobSucceeded, JobFailed or JobCancelled
Done int // entries processed
Total int // entries the run intended to write
Summary json.RawMessage // nil when there is none
Error string // the reason, for JobFailed
}
JobResult is what a worker records when a run ends. It carries the final progress itself: a short run can end before its first heartbeat has landed, and the terminal row must still read done-of-total.
type JobStatus ¶ added in v1.3.0
type JobStatus string
JobStatus is the lifecycle state of an extraction job: pending → running → succeeded | failed | cancelled. A running job goes back to pending when its worker parks it on shutdown or a reaper takes it from a worker that stopped reporting.
type ListFilter ¶
type ListFilter struct {
ContentType string
Tags map[string]any // matched against the JSONB metadata column
Limit int
Cursor string // exclusive upper-bound file id; empty means first page
Entries EntryScope // "" behaves as EntriesInclude
// IncludeDeleted also returns soft-deleted rows. Every read path stays
// live-only except extraction's resume logic, which must see the tombstone
// of an entry the user deleted so that it does not recreate it.
IncludeDeleted bool
}
ListFilter narrows a List query. Zero values mean "no constraint".
Pagination is keyset-based: Cursor is the id of the last file of the previous page (exclusive), and a page holds the next Limit files older than it. Ids are UUIDv7 — time-ordered — so paging by id descending walks newest-first without the deep-page cost of OFFSET.
type Repository ¶
type Repository interface {
// Create inserts a new record. The caller supplies the ID and StorageKey.
Create(ctx context.Context, f *File) error
// Get returns the record by ID, or ErrNotFound.
Get(ctx context.Context, id string) (*File, error)
// List returns records matching the filter, newest first.
List(ctx context.Context, filter ListFilter) ([]*File, error)
// UpdateMetadata writes tags into the record's JSONB metadata: merged into
// the existing object when merge is true, replacing it wholesale otherwise.
// A replace preserves keys under ReservedTagPrefix.
UpdateMetadata(ctx context.Context, id string, tags map[string]any, merge bool) (*File, error)
// Delete soft-deletes the record by ID (sets deleted_at). The object is
// purged asynchronously by GC. Returns ErrNotFound if no live row matched.
Delete(ctx context.Context, id string) error
// ListDeleted returns up to limit soft-deleted records, oldest first, so GC
// can purge their objects and then Purge the rows.
ListDeleted(ctx context.Context, limit int) ([]*File, error)
// Purge hard-deletes a soft-deleted record by ID, after its object is gone.
Purge(ctx context.Context, id string) error
// StorageKeys returns the set of every storage key referenced by the table
// (any state), so GC can tell which stored objects are orphans.
StorageKeys(ctx context.Context) (map[string]struct{}, error)
// ClaimIdempotencyKey atomically claims key as pending. It returns
// claimed=true if this caller now owns the key; otherwise claimed=false and
// existing describes the row already present (which may be nil if it raced
// away). Implemented as INSERT ... ON CONFLICT DO NOTHING.
ClaimIdempotencyKey(ctx context.Context, key string) (claimed bool, existing *IdempotencyRecord, err error)
// CreateForKey inserts the file row and marks the (already-claimed) key
// completed in one transaction, so the two can never disagree after a crash.
CreateForKey(ctx context.Context, f *File, key string) error
// ReleaseIdempotencyKey deletes a still-pending claim so a failed upload can
// be retried under the same key.
ReleaseIdempotencyKey(ctx context.Context, key string) error
// PurgeIdempotencyKeys deletes keys created before the cutoff (TTL), returning
// how many were removed. Called by GC.
PurgeIdempotencyKeys(ctx context.Context, before time.Time) (int, error)
}
Repository persists File records. Implementations own all SQL.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package memory is an in-memory metadata.Repository used as a test double.
|
Package memory is an in-memory metadata.Repository used as a test double. |
|
Package postgres is the pgx implementation of metadata.Repository.
|
Package postgres is the pgx implementation of metadata.Repository. |