store

package
v0.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 6, 2026 License: MIT Imports: 6 Imported by: 0

Documentation

Overview

Package store is the object store the archive lives in.

The interface is small on purpose: the writer puts objects and never rewrites one, the reader gets and lists them, and nothing deletes. What makes the archive an archive is the store's own configuration — versioning, Object Lock in compliance mode, a policy that denies deletes — and not anything this code could enforce on its own. What this code does is set a retention on every object it writes and never reuse a key.

Index

Constants

View Source
const (
	MetaFormat = "format"
	MetaSHA256 = "sha256"
	MetaCount  = "count"
)

MetaFormat, MetaSHA256 and MetaCount are the user-defined metadata every record object carries. A reader needs a listing and a HEAD and never a body.

View Source
const (
	// SealsPrefix is the prefix under which every seal lives.
	SealsPrefix = "seals/"
	// KeysPrefix is the prefix of the trust anchors, the delegations and the
	// revocations.
	KeysPrefix = "keys/"
	// RootsKey is the JWK Set of root public keys: distribution, not trust.
	RootsKey = KeysPrefix + "roots.jwks"
	// DelegationsPrefix holds keys/delegations/<thumbprint>/<ULID>.jws.
	DelegationsPrefix = KeysPrefix + "delegations/"
	// RevocationsPrefix holds keys/revocations/<ULID>.jws.
	RevocationsPrefix = KeysPrefix + "revocations/"
)
View Source
const CataloguePrefix = "catalogue/"

CataloguePrefix is the prefix of the catalogues, one object per application and version.

View Source
const Format = "1"

Format is the value of the format metadata: the layout version.

View Source
const RecordsPrefix = "records/"

RecordsPrefix is the prefix under which every record object lives.

View Source
const SealExt = ".jws"

SealExt is the extension of a seal's key.

Variables

View Source
var ErrExists = errors.New("store: the key already exists")

ErrExists is returned when a key is already taken. Under Object Lock a second put would create a version rather than replace anything, so a writer that reuses a key is a writer whose objects a digest cannot account for.

View Source
var ErrNotFound = errors.New("store: no such object")

ErrNotFound is returned for a key that was never written.

View Source
var ErrNotLockable = errors.New("store: the store holds no lock")

ErrNotLockable is returned by SetLegalHold and ExtendRetention on a store that holds no lock: one configured with no Object Lock mode, or a store whose API has no Object Lock at all. It is a sentinel rather than a plain error so that a caller can tell "this store cannot" from "this call failed": the writer records the first in the trail and moves on, where the second is worth retrying.

Functions

func CatalogueKey added in v0.7.0

func CatalogueKey(app, version string) string

CatalogueKey is where an application's catalogue at a version lives.

func HourPrefix added in v0.7.0

func HourPrefix(profile, tenant string, hour time.Time) string

HourPrefix is where the objects taken in one hour of ingest time live.

func KeyComponent added in v0.7.0

func KeyComponent(s string) string

KeyComponent reports why a string cannot be a profile, a tenant, an application or a version in a key, or "" when it can. Only the slash is refused: it is the one character a key's grammar gives a meaning to.

func ProfilePrefix added in v0.7.0

func ProfilePrefix(profile string) string

ProfilePrefix is where one profile's records live.

func Profiles added in v0.7.0

func Profiles(ctx context.Context, s Store) ([]string, error)

Profiles lists the profiles that have records in the archive.

func RecordKey added in v0.7.0

func RecordKey(profile, tenant string, at time.Time, id string) string

RecordKey is the key of an ingest batch: the hour is the hour of the ULID's own moment, so that the two cannot disagree.

func SchemaDir added in v0.7.0

func SchemaDir(app, version string) string

SchemaDir is where the extension schemas of an application's catalogue at a version are kept, beside the catalogue and outside the contract: a reader that wants only the contract ignores it, and one that reads a catalogue's data columns needs it.

func SealKey added in v0.7.0

func SealKey(profile, tenant string, hour time.Time) string

SealKey is the key of the seal of one hour: seals/<profile>/<tenant>/<yyyy>/<mm>/<dd>/<hh>.jws.

func SealTenantPrefix added in v0.7.0

func SealTenantPrefix(profile, tenant string) string

SealTenantPrefix is where one profile's and tenant's seals live.

func TenantPrefix added in v0.7.0

func TenantPrefix(profile, tenant string) string

TenantPrefix is where one tenant's records under one profile live.

func Tenants added in v0.7.0

func Tenants(ctx context.Context, s Store, profile string) ([]string, error)

Tenants lists the tenants that have records under a profile, without walking the objects beneath them.

func WalkHours added in v0.7.0

func WalkHours(
	ctx context.Context, s Store, profile, tenant string, from, to time.Time, after string, fn func(Entry) error,
) error

WalkHours visits the record objects of one profile and tenant taken in the hours from `from` to `to`, both inclusive, in key order: which, because a key begins with its ingest hour and ends with a ULID, is the order the batches arrived in.

It is one listing paged to its end, started after the hour before the first the range names, and stopped at the first key past the last. The tenant is the narrowest prefix a range of hours has in common, so a longer range costs pages and not requests per hour. `after`, when it is not empty, resumes a walk: only keys sorting after it are visited, which is the cursor a follower keeps.

The range is of years 1000 to 8999: keys write the year in four digits, and a range outside them would not sort as it reads.

fn is given each entry; returning an error stops the walk.

func WalkProfile added in v0.7.0

func WalkProfile(
	ctx context.Context, s Store, profile string, from, to time.Time, fn func(tenant string, e Entry) error,
) error

WalkProfile is WalkHours for every tenant of a profile, tenant by tenant. Tenants come in key order, so a walk is repeatable; objects of different tenants are not interleaved.

func WalkSeals added in v0.7.0

func WalkSeals(
	ctx context.Context, s Store, profile, tenant string, from, to time.Time, fn func(Entry) error,
) error

WalkSeals visits the seals of one profile and tenant for the hours from `from` to `to`, both inclusive, in key order, which is time order.

Types

type Entry

type Entry struct {
	Key         string
	Size        int64
	Modified    time.Time
	RetainUntil time.Time
	// LegalHold is whether a hold is on the object. A listing does not carry
	// it — S3 does not report it per key — so it is set by Head and left false
	// by List.
	LegalHold bool
	// Metadata is the object's user-defined metadata, as Head returns it. A
	// listing carries none — S3 does not report it per key — so it is nil from
	// List. It is what lets a reader of the v1 layout learn an object's
	// sha256 and record count from a HEAD and never a body.
	Metadata map[string]string
}

Entry is one object as a listing sees it.

func LastSeal added in v0.7.0

func LastSeal(ctx context.Context, s Store, profile, tenant string) (Entry, bool, error)

LastSeal is the newest seal of one profile and tenant, or false when there is none. It descends year, month, day by common prefix and lists one day, and so costs five requests however many seals the tenant has; a key under the prefix that is not a seal's is passed over.

type Object

type Object struct {
	Key  string
	Body []byte
	// RetainUntil is when the object may first be deleted. The store is
	// expected to refuse a deletion before it, and to refuse shortening it.
	RetainUntil time.Time
	ContentType string
	// Encoding is the content encoding, "zstd" for a batch of records.
	Encoding string
	// Metadata is small, and is there so an object can say what it is without
	// being opened.
	Metadata map[string]string
	// LegalHold keeps an object undeletable for as long as it is set,
	// independently of RetainUntil and without any expiry of its own.
	//
	// It is set at write time rather than afterwards because a hold is placed
	// on a prefix and objects keep arriving under it: an object written into a
	// held prefix and only held by a later sweep is an object that was
	// deletable in between, which is the window a hold exists to close.
	LegalHold bool
}

Object is one thing written to the archive.

type Presigner

type Presigner interface {
	// Presign returns a URL to an object, valid for the given time.
	Presign(ctx context.Context, key string, valid time.Duration) (string, error)
}

Presigner is a store that can hand out a URL to one object for a while.

It is separate from Store because most of what this repository does with an archive must not be reachable by a URL anybody can hold: the records are read through a service that checks a grant and records the read. Presigning exists for exports, which are a deliberate copy of records to somewhere a person can download them, and a store that cannot do it is not a lesser store.

type RecordObject added in v0.7.0

type RecordObject struct {
	Profile, Tenant string
	// Hour is the hour of ingest time the key names.
	Hour time.Time
	ULID string
}

RecordObject is a record object's key, taken apart.

func ParseRecordKey added in v0.7.0

func ParseRecordKey(key string) (RecordObject, bool)

ParseRecordKey reads a key of the grammar

records/<profile>/<tenant>/<yyyy>/<mm>/<dd>/<hh>/<ULID>

and reports whether it is one. The ULID's own moment must fall in the hour the key names, which is what lets a listing order be an ingest order.

type SealObject added in v0.7.0

type SealObject struct {
	Profile, Tenant string
	Hour            time.Time
}

SealObject is a seal's key, taken apart.

func ParseSealKey added in v0.7.0

func ParseSealKey(key string) (SealObject, bool)

ParseSealKey reads a key of the grammar

seals/<profile>/<tenant>/<yyyy>/<mm>/<dd>/<hh>.jws

and reports whether it is one.

type Store

type Store interface {
	// Put writes an object. It fails with ErrExists rather than overwriting.
	Put(ctx context.Context, o Object) error
	// Get returns an object's body.
	Get(ctx context.Context, key string) ([]byte, error)
	// Head returns an object's listing entry without its body.
	Head(ctx context.Context, key string) (Entry, error)
	// List returns entries under a prefix, in key order, starting after a key.
	List(ctx context.Context, prefix, after string, limit int) ([]Entry, error)
	// SetLegalHold places or releases a hold on an object that already exists.
	// Placing a hold covers what an archive already holds; releasing one is
	// governed outside this interface, by whatever the deployment's break-glass
	// role is, and this only makes the call.
	SetLegalHold(ctx context.Context, key string, on bool) error
	// ExtendRetention lengthens an object's retention to until. It only ever
	// lengthens: compliance mode refuses a shorter date, and so must every
	// implementation, or a test could shorten what a real bucket would keep.
	// A caller that is not sure the new date is later checks with Head first.
	ExtendRetention(ctx context.Context, key string, until time.Time) error
	// Prefixes returns the distinct groups one level under a prefix, as S3's
	// common prefixes: listing "records/security/" with "/" gives the tenants
	// without walking the objects beneath them.
	//
	// The archive puts the tenant between the profile and the hour, so a job
	// that works a range of hours cannot build its prefix without first
	// knowing which tenants exist. Walking every object to find out would cost
	// the whole profile every time.
	Prefixes(ctx context.Context, prefix, delimiter string) ([]string, error)
}

Store is an object store.

Directories

Path Synopsis
Package s3store keeps the archive in an S3 bucket.
Package s3store keeps the archive in an S3 bucket.
Package storetest holds the store a test writes to.
Package storetest holds the store a test writes to.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL