metadata

package
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 17, 2026 License: MIT Imports: 4 Imported by: 0

Documentation

Overview

Package metadata is the persistence seam for file records. The Repository interface is the only contract above this layer; the pgx implementation (Phase 1) is the sole place that issues SQL.

Index

Constants

View Source
const (
	IdempotencyPending   = "pending"
	IdempotencyCompleted = "completed"
)

Idempotency-key lifecycle states.

View Source
const (
	ReservedTagPrefix = "cm:"
	TagArchiveID      = ReservedTagPrefix + "archive_id"    // on an extracted entry: the archive it came from
	TagArchivePath    = ReservedTagPrefix + "archive_path"  // on an extracted entry: its full path inside the archive
	TagArchiveIndex   = ReservedTagPrefix + "archive_index" // on an extracted entry: its directory index (names are not unique)
	TagArchive        = ReservedTagPrefix + "archive"       // on an archive row: TagArchiveMarker once it has been extracted
)

Reserved tag keys. The cm: prefix is owned by the service — written by archive extraction, never by a client — across the whole tag namespace. The api layer rejects it on upload and PATCH, and a replace-mode UpdateMetadata carries the keys across so a client cannot strip them by accident.

View Source
const TagArchiveMarker = "true"

TagArchiveMarker is the value of TagArchive. A string, not a boolean, because the list endpoint's tag.<k>=<v> filter is string-typed: this is what makes GET /files?tag.cm:archive=true list the archives.

Variables

View Source
var ErrJobNotFound = errors.New("metadata: extraction job not found")

ErrJobNotFound is returned when no extraction job matches — or, from the worker-side writes, when the row is no longer owned by the caller's attempt.

View Source
var ErrNotFound = errors.New("metadata: file not found")

ErrNotFound is returned when no file record matches the query.

Functions

This section is empty.

Types

type EntryScope added in v1.3.0

type EntryScope string

EntryScope selects whether extracted archive entries appear in a List. The parameter is about *entries*, not archives: the archives themselves are found by their TagArchive marker.

const (
	EntriesInclude EntryScope = "include" // the default: files, archives and entries together
	EntriesExclude EntryScope = "exclude" // everything but extracted entries
	EntriesOnly    EntryScope = "only"    // extracted entries alone
)

type File

type File struct {
	ID             string
	StorageKey     string
	Filename       string
	ContentType    string
	SizeBytes      int64
	ChecksumSHA256 string         // hex sha-256
	Metadata       map[string]any // JSONB tags
	CreatedAt      time.Time
	UpdatedAt      *time.Time // nil until the first metadata update — a pristine, unmodified file has none
	DeletedAt      *time.Time // soft delete; nil while live
}

File is the metadata record for a stored object. It mirrors the files table.

type IdempotencyRecord

type IdempotencyRecord struct {
	Status string  // IdempotencyPending | IdempotencyCompleted
	FileID *string // the produced file id; nil while pending
}

IdempotencyRecord is the stored state of an upload idempotency key.

type Job added in v1.3.0

type Job struct {
	ID        string
	ArchiveID string
	Status    JobStatus
	// Selection is the caller's entry indexes. nil means every selectable
	// entry; an empty, non-nil slice selects nothing. The distinction is
	// preserved through storage (null vs []).
	Selection       []int
	CancelRequested bool
	// Attempt counts claims. It is the fencing token on every worker write.
	Attempt int
	// Progress of the current run: Done entries processed of Total intended.
	// A job resumed after an interruption counts only what remained.
	ProgressDone  int
	ProgressTotal int
	// Summary is the extraction summary as JSON, once terminal. nil when the
	// run never produced one (cancelled before it started, or failed before
	// the archive could be opened).
	Summary json.RawMessage
	// Error is the reason when Status is JobFailed.
	Error       string
	HeartbeatAt *time.Time // set while running; nil otherwise
	CreatedAt   time.Time
	UpdatedAt   time.Time
	FinishedAt  *time.Time // set once terminal
}

Job is one extraction job. It mirrors the extraction_jobs table.

type JobRepository added in v1.3.0

type JobRepository interface {
	// CreateJob inserts job as pending. created=false with activeID set means
	// another job for the same archive is still pending or running — the
	// unique index refused the insert. activeID may be empty if that job
	// finished between the conflict and the lookup; the caller simply retries.
	// ErrNotFound when the archive row no longer exists.
	CreateJob(ctx context.Context, job *Job) (created bool, activeID string, err error)

	// GetJob returns the job by id, or ErrJobNotFound.
	GetJob(ctx context.Context, id string) (*Job, error)

	// RequestCancel asks a job to stop. A pending job becomes cancelled at
	// once; a running one has its flag set for the worker to honour after
	// the current entry; a terminal one is left as it is. Returns the job's
	// state after the request, or ErrJobNotFound.
	RequestCancel(ctx context.Context, id string) (*Job, error)

	// ClaimJob moves the oldest pending job to running for this caller,
	// bumping Attempt, and returns it — or nil, nil when nothing is pending.
	// Safe across replicas: concurrent callers never claim the same row.
	ClaimJob(ctx context.Context) (*Job, error)

	// HeartbeatJob records progress and liveness for a running job in one
	// round trip and reports whether cancellation has been requested.
	// ErrJobNotFound when the row is not running under this attempt — the
	// caller has lost ownership and must stop.
	HeartbeatJob(ctx context.Context, id string, attempt, done, total int) (cancelRequested bool, err error)

	// FinishJob records a terminal result for a running job. ErrJobNotFound
	// when the row is not running under this attempt.
	FinishJob(ctx context.Context, id string, attempt int, result JobResult) error

	// ParkJob returns a running job to pending — a worker shutting down —
	// keeping its progress. ErrJobNotFound when the row is not running under
	// this attempt.
	ParkJob(ctx context.Context, id string, attempt, done, total int) error

	// ReapJobs returns running jobs that have gone longer than ttl without a
	// heartbeat to pending, so the next pickup resumes them, and reports how
	// many. Judged on the repository's own clock, which also stamps the
	// heartbeat: this is the one comparison where a skew between processes
	// would hand a live run to a second worker.
	ReapJobs(ctx context.Context, ttl time.Duration) (int, error)

	// CountPendingJobs is the queue depth.
	CountPendingJobs(ctx context.Context) (int, error)

	// PurgeFinishedJobs deletes terminal jobs that finished before the cutoff
	// (retention), returning how many. Called by GC.
	PurgeFinishedJobs(ctx context.Context, before time.Time) (int, error)
}

JobRepository persists extraction jobs. It is a sibling of Repository — the pgx and in-memory implementations satisfy both — kept separate because jobs are a distinct bounded concern and Repository is already ten methods.

type JobResult added in v1.3.0

type JobResult struct {
	Status  JobStatus       // JobSucceeded, JobFailed or JobCancelled
	Done    int             // entries processed
	Total   int             // entries the run intended to write
	Summary json.RawMessage // nil when there is none
	Error   string          // the reason, for JobFailed
}

JobResult is what a worker records when a run ends. It carries the final progress itself: a short run can end before its first heartbeat has landed, and the terminal row must still read done-of-total.

type JobStatus added in v1.3.0

type JobStatus string

JobStatus is the lifecycle state of an extraction job: pending → running → succeeded | failed | cancelled. A running job goes back to pending when its worker parks it on shutdown or a reaper takes it from a worker that stopped reporting.

const (
	JobPending   JobStatus = "pending"
	JobRunning   JobStatus = "running"
	JobSucceeded JobStatus = "succeeded"
	JobFailed    JobStatus = "failed"
	JobCancelled JobStatus = "cancelled"
)

Job lifecycle states.

func (JobStatus) Terminal added in v1.3.0

func (s JobStatus) Terminal() bool

Terminal reports whether the job will not change state again.

type ListFilter

type ListFilter struct {
	ContentType string
	Tags        map[string]any // matched against the JSONB metadata column
	Limit       int
	Cursor      string     // exclusive upper-bound file id; empty means first page
	Entries     EntryScope // "" behaves as EntriesInclude

	// IncludeDeleted also returns soft-deleted rows. Every read path stays
	// live-only except extraction's resume logic, which must see the tombstone
	// of an entry the user deleted so that it does not recreate it.
	IncludeDeleted bool
}

ListFilter narrows a List query. Zero values mean "no constraint".

Pagination is keyset-based: Cursor is the id of the last file of the previous page (exclusive), and a page holds the next Limit files older than it. Ids are UUIDv7 — time-ordered — so paging by id descending walks newest-first without the deep-page cost of OFFSET.

type Repository

type Repository interface {
	// Create inserts a new record. The caller supplies the ID and StorageKey.
	Create(ctx context.Context, f *File) error

	// Get returns the record by ID, or ErrNotFound.
	Get(ctx context.Context, id string) (*File, error)

	// List returns records matching the filter, newest first.
	List(ctx context.Context, filter ListFilter) ([]*File, error)

	// UpdateMetadata writes tags into the record's JSONB metadata: merged into
	// the existing object when merge is true, replacing it wholesale otherwise.
	// A replace preserves keys under ReservedTagPrefix.
	UpdateMetadata(ctx context.Context, id string, tags map[string]any, merge bool) (*File, error)

	// Delete soft-deletes the record by ID (sets deleted_at). The object is
	// purged asynchronously by GC. Returns ErrNotFound if no live row matched.
	Delete(ctx context.Context, id string) error

	// ListDeleted returns up to limit soft-deleted records, oldest first, so GC
	// can purge their objects and then Purge the rows.
	ListDeleted(ctx context.Context, limit int) ([]*File, error)

	// Purge hard-deletes a soft-deleted record by ID, after its object is gone.
	Purge(ctx context.Context, id string) error

	// StorageKeys returns the set of every storage key referenced by the table
	// (any state), so GC can tell which stored objects are orphans.
	StorageKeys(ctx context.Context) (map[string]struct{}, error)

	// ClaimIdempotencyKey atomically claims key as pending. It returns
	// claimed=true if this caller now owns the key; otherwise claimed=false and
	// existing describes the row already present (which may be nil if it raced
	// away). Implemented as INSERT ... ON CONFLICT DO NOTHING.
	ClaimIdempotencyKey(ctx context.Context, key string) (claimed bool, existing *IdempotencyRecord, err error)

	// CreateForKey inserts the file row and marks the (already-claimed) key
	// completed in one transaction, so the two can never disagree after a crash.
	CreateForKey(ctx context.Context, f *File, key string) error

	// ReleaseIdempotencyKey deletes a still-pending claim so a failed upload can
	// be retried under the same key.
	ReleaseIdempotencyKey(ctx context.Context, key string) error

	// PurgeIdempotencyKeys deletes keys created before the cutoff (TTL), returning
	// how many were removed. Called by GC.
	PurgeIdempotencyKeys(ctx context.Context, before time.Time) (int, error)
}

Repository persists File records. Implementations own all SQL.

Directories

Path Synopsis
Package memory is an in-memory metadata.Repository used as a test double.
Package memory is an in-memory metadata.Repository used as a test double.
Package postgres is the pgx implementation of metadata.Repository.
Package postgres is the pgx implementation of metadata.Repository.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL