Documentation
¶
Overview ¶
Package store is the object store the archive lives in.
The interface is small on purpose: the writer puts objects and never rewrites one, the reader gets and lists them, and nothing deletes. What makes the archive an archive is the store's own configuration — versioning, Object Lock in compliance mode, a policy that denies deletes — and not anything this code could enforce on its own. What this code does is set a retention on every object it writes and never reuse a key.
Index ¶
- Constants
- Variables
- func CatalogueKey(app, version string) string
- func HourPrefix(profile, tenant string, hour time.Time) string
- func KeyComponent(s string) string
- func ProfilePrefix(profile string) string
- func Profiles(ctx context.Context, s Store) ([]string, error)
- func RecordKey(profile, tenant string, at time.Time, id string) string
- func SchemaDir(app, version string) string
- func SealKey(profile, tenant string, hour time.Time) string
- func SealTenantPrefix(profile, tenant string) string
- func TenantPrefix(profile, tenant string) string
- func Tenants(ctx context.Context, s Store, profile string) ([]string, error)
- func WalkHours(ctx context.Context, s Store, profile, tenant string, from, to time.Time, ...) error
- func WalkProfile(ctx context.Context, s Store, profile string, from, to time.Time, ...) error
- func WalkSeals(ctx context.Context, s Store, profile, tenant string, from, to time.Time, ...) error
- type Entry
- type Object
- type Presigner
- type RecordObject
- type SealObject
- type Store
Constants ¶
const ( MetaFormat = "format" MetaSHA256 = "sha256" MetaCount = "count" )
MetaFormat, MetaSHA256 and MetaCount are the user-defined metadata every record object carries. A reader needs a listing and a HEAD and never a body.
const ( // SealsPrefix is the prefix under which every seal lives. SealsPrefix = "seals/" // KeysPrefix is the prefix of the trust anchors, the delegations and the // revocations. KeysPrefix = "keys/" // RootsKey is the JWK Set of root public keys: distribution, not trust. RootsKey = KeysPrefix + "roots.jwks" // DelegationsPrefix holds keys/delegations/<thumbprint>/<ULID>.jws. DelegationsPrefix = KeysPrefix + "delegations/" // RevocationsPrefix holds keys/revocations/<ULID>.jws. RevocationsPrefix = KeysPrefix + "revocations/" )
const CataloguePrefix = "catalogue/"
CataloguePrefix is the prefix of the catalogues, one object per application and version.
const Format = "1"
Format is the value of the format metadata: the layout version.
const RecordsPrefix = "records/"
RecordsPrefix is the prefix under which every record object lives.
const SealExt = ".jws"
SealExt is the extension of a seal's key.
Variables ¶
var ErrExists = errors.New("store: the key already exists")
ErrExists is returned when a key is already taken. Under Object Lock a second put would create a version rather than replace anything, so a writer that reuses a key is a writer whose objects a digest cannot account for.
var ErrNotFound = errors.New("store: no such object")
ErrNotFound is returned for a key that was never written.
var ErrNotLockable = errors.New("store: the store holds no lock")
ErrNotLockable is returned by SetLegalHold and ExtendRetention on a store that holds no lock: one configured with no Object Lock mode, or a store whose API has no Object Lock at all. It is a sentinel rather than a plain error so that a caller can tell "this store cannot" from "this call failed": the writer records the first in the trail and moves on, where the second is worth retrying.
Functions ¶
func CatalogueKey ¶ added in v0.7.0
CatalogueKey is where an application's catalogue at a version lives.
func HourPrefix ¶ added in v0.7.0
HourPrefix is where the objects taken in one hour of ingest time live.
func KeyComponent ¶ added in v0.7.0
KeyComponent reports why a string cannot be a profile, a tenant, an application or a version in a key, or "" when it can. Only the slash is refused: it is the one character a key's grammar gives a meaning to.
func ProfilePrefix ¶ added in v0.7.0
ProfilePrefix is where one profile's records live.
func RecordKey ¶ added in v0.7.0
RecordKey is the key of an ingest batch: the hour is the hour of the ULID's own moment, so that the two cannot disagree.
func SchemaDir ¶ added in v0.7.0
SchemaDir is where the extension schemas of an application's catalogue at a version are kept, beside the catalogue and outside the contract: a reader that wants only the contract ignores it, and one that reads a catalogue's data columns needs it.
func SealKey ¶ added in v0.7.0
SealKey is the key of the seal of one hour: seals/<profile>/<tenant>/<yyyy>/<mm>/<dd>/<hh>.jws.
func SealTenantPrefix ¶ added in v0.7.0
SealTenantPrefix is where one profile's and tenant's seals live.
func TenantPrefix ¶ added in v0.7.0
TenantPrefix is where one tenant's records under one profile live.
func Tenants ¶ added in v0.7.0
Tenants lists the tenants that have records under a profile, without walking the objects beneath them.
func WalkHours ¶ added in v0.7.0
func WalkHours( ctx context.Context, s Store, profile, tenant string, from, to time.Time, after string, fn func(Entry) error, ) error
WalkHours visits the record objects of one profile and tenant taken in the hours from `from` to `to`, both inclusive, in key order: which, because a key begins with its ingest hour and ends with a ULID, is the order the batches arrived in.
It is one listing paged to its end, started after the hour before the first the range names, and stopped at the first key past the last. The tenant is the narrowest prefix a range of hours has in common, so a longer range costs pages and not requests per hour. `after`, when it is not empty, resumes a walk: only keys sorting after it are visited, which is the cursor a follower keeps.
The range is of years 1000 to 8999: keys write the year in four digits, and a range outside them would not sort as it reads.
fn is given each entry; returning an error stops the walk.
func WalkProfile ¶ added in v0.7.0
func WalkProfile( ctx context.Context, s Store, profile string, from, to time.Time, fn func(tenant string, e Entry) error, ) error
WalkProfile is WalkHours for every tenant of a profile, tenant by tenant. Tenants come in key order, so a walk is repeatable; objects of different tenants are not interleaved.
Types ¶
type Entry ¶
type Entry struct {
Key string
Size int64
Modified time.Time
RetainUntil time.Time
// LegalHold is whether a hold is on the object. A listing does not carry
// it — S3 does not report it per key — so it is set by Head and left false
// by List.
LegalHold bool
// Metadata is the object's user-defined metadata, as Head returns it. A
// listing carries none — S3 does not report it per key — so it is nil from
// List. It is what lets a reader of the v1 layout learn an object's
// sha256 and record count from a HEAD and never a body.
Metadata map[string]string
}
Entry is one object as a listing sees it.
func LastSeal ¶ added in v0.7.0
LastSeal is the newest seal of one profile and tenant, or false when there is none. It descends year, month, day by common prefix and lists one day, and so costs five requests however many seals the tenant has; a key under the prefix that is not a seal's is passed over.
type Object ¶
type Object struct {
Key string
Body []byte
// RetainUntil is when the object may first be deleted. The store is
// expected to refuse a deletion before it, and to refuse shortening it.
RetainUntil time.Time
ContentType string
// Encoding is the content encoding, "zstd" for a batch of records.
Encoding string
// Metadata is small, and is there so an object can say what it is without
// being opened.
Metadata map[string]string
// LegalHold keeps an object undeletable for as long as it is set,
// independently of RetainUntil and without any expiry of its own.
//
// It is set at write time rather than afterwards because a hold is placed
// on a prefix and objects keep arriving under it: an object written into a
// held prefix and only held by a later sweep is an object that was
// deletable in between, which is the window a hold exists to close.
LegalHold bool
}
Object is one thing written to the archive.
type Presigner ¶
type Presigner interface {
// Presign returns a URL to an object, valid for the given time.
Presign(ctx context.Context, key string, valid time.Duration) (string, error)
}
Presigner is a store that can hand out a URL to one object for a while.
It is separate from Store because most of what this repository does with an archive must not be reachable by a URL anybody can hold: the records are read through a service that checks a grant and records the read. Presigning exists for exports, which are a deliberate copy of records to somewhere a person can download them, and a store that cannot do it is not a lesser store.
type RecordObject ¶ added in v0.7.0
type RecordObject struct {
Profile, Tenant string
// Hour is the hour of ingest time the key names.
Hour time.Time
ULID string
}
RecordObject is a record object's key, taken apart.
func ParseRecordKey ¶ added in v0.7.0
func ParseRecordKey(key string) (RecordObject, bool)
ParseRecordKey reads a key of the grammar
records/<profile>/<tenant>/<yyyy>/<mm>/<dd>/<hh>/<ULID>
and reports whether it is one. The ULID's own moment must fall in the hour the key names, which is what lets a listing order be an ingest order.
type SealObject ¶ added in v0.7.0
SealObject is a seal's key, taken apart.
func ParseSealKey ¶ added in v0.7.0
func ParseSealKey(key string) (SealObject, bool)
ParseSealKey reads a key of the grammar
seals/<profile>/<tenant>/<yyyy>/<mm>/<dd>/<hh>.jws
and reports whether it is one.
type Store ¶
type Store interface {
// Put writes an object. It fails with ErrExists rather than overwriting.
Put(ctx context.Context, o Object) error
// Get returns an object's body.
Get(ctx context.Context, key string) ([]byte, error)
// Head returns an object's listing entry without its body.
Head(ctx context.Context, key string) (Entry, error)
// List returns entries under a prefix, in key order, starting after a key.
List(ctx context.Context, prefix, after string, limit int) ([]Entry, error)
// SetLegalHold places or releases a hold on an object that already exists.
// Placing a hold covers what an archive already holds; releasing one is
// governed outside this interface, by whatever the deployment's break-glass
// role is, and this only makes the call.
SetLegalHold(ctx context.Context, key string, on bool) error
// ExtendRetention lengthens an object's retention to until. It only ever
// lengthens: compliance mode refuses a shorter date, and so must every
// implementation, or a test could shorten what a real bucket would keep.
// A caller that is not sure the new date is later checks with Head first.
ExtendRetention(ctx context.Context, key string, until time.Time) error
// Prefixes returns the distinct groups one level under a prefix, as S3's
// common prefixes: listing "records/security/" with "/" gives the tenants
// without walking the objects beneath them.
//
// The archive puts the tenant between the profile and the hour, so a job
// that works a range of hours cannot build its prefix without first
// knowing which tenants exist. Walking every object to find out would cost
// the whole profile every time.
Prefixes(ctx context.Context, prefix, delimiter string) ([]string, error)
}
Store is an object store.