Documentation
¶
Overview ¶
Package objectstore is the store several dispatchers share: the in-memory store (store/memory) is the engine in each of them, and the bucket holds the authoritative history, an append-only log of create-only objects, one per change, plus periodic snapshots. A change is committed when its log object is created, and creation is conditional on the name being free, so two dispatchers can't both take one version.
<prefix>/ log/00000000000000001043.json one change: P2's journal line, the writer and its schema snapshots/00000000000000001000.json.gz the whole store at version 1000 (P2's snapshot, gzip'd) transcripts/<run>.jsonl.gz one blob per transcript or environment log (P2's naming)
Reads are served from memory, which syncs (reads the log from its version + 1 until an entry is missing) before a read older than the sync interval, on a Get that finds nothing, after a lost race, and in the background. Locks are StoreLock records and transcript sizes are TranscriptBlob records, both written through the log.
Design: proposal P3, "The dispatcher keeps its state in object storage", and design/workstreams/dispatcher-runtime/LLD/store.md §Object store.
Index ¶
- Constants
- Variables
- func RegisterScheme(scheme string, open Opener)
- type Bucket
- type FakeBucket
- func (b *FakeBucket) Calls(op FakeOp) int
- func (b *FakeBucket) Create(ctx context.Context, name string, data []byte) error
- func (b *FakeBucket) Delete(ctx context.Context, name string) error
- func (b *FakeBucket) Get(ctx context.Context, name string) ([]byte, error)
- func (b *FakeBucket) List(ctx context.Context, prefix, startAfter string, fn func(ObjectInfo) error) error
- func (b *FakeBucket) Names(prefix string) []string
- func (b *FakeBucket) Objects() map[string][]byte
- func (b *FakeBucket) SetFaults(f func(FakeRequest) Fault)
- func (b *FakeBucket) SetObject(name string, data []byte)
- type FakeOp
- type FakeRequest
- type Fault
- type FileBucket
- func (b *FileBucket) Create(ctx context.Context, name string, data []byte) error
- func (b *FileBucket) Delete(ctx context.Context, name string) error
- func (b *FileBucket) Dir() string
- func (b *FileBucket) Get(ctx context.Context, name string) ([]byte, error)
- func (b *FileBucket) List(ctx context.Context, prefix, startAfter string, fn func(ObjectInfo) error) error
- type GCSBucket
- func (b *GCSBucket) Close() error
- func (b *GCSBucket) Create(ctx context.Context, name string, data []byte) error
- func (b *GCSBucket) Delete(ctx context.Context, name string) error
- func (b *GCSBucket) Get(ctx context.Context, name string) ([]byte, error)
- func (b *GCSBucket) List(ctx context.Context, prefix, startAfter string, fn func(ObjectInfo) error) error
- func (b *GCSBucket) String() string
- type LoadInfo
- type LogEntry
- type ObjectInfo
- type Opener
- type Option
- func WithCompactEvery(n int) Option
- func WithKind(kind string, proto store.Object) Option
- func WithMaxLag(d time.Duration) Option
- func WithRequestTimeout(d time.Duration) Option
- func WithSegmentRetention(d time.Duration) Option
- func WithSyncInterval(d time.Duration) Option
- func WithWriterID(id string) Option
- type S3Bucket
- func (b *S3Bucket) Create(ctx context.Context, name string, data []byte) error
- func (b *S3Bucket) Delete(ctx context.Context, name string) error
- func (b *S3Bucket) Get(ctx context.Context, name string) ([]byte, error)
- func (b *S3Bucket) List(ctx context.Context, prefix, startAfter string, fn func(ObjectInfo) error) error
- func (b *S3Bucket) String() string
- type Store
- func (s *Store) Close() error
- func (s *Store) Compact(ctx context.Context) error
- func (s *Store) Create(ctx context.Context, obj store.Object) error
- func (s *Store) Delete(ctx context.Context, k store.Key) error
- func (s *Store) DeleteIfUnchanged(ctx context.Context, k store.Key, resourceVersion string) error
- func (s *Store) Get(ctx context.Context, k store.Key, out store.Object) error
- func (s *Store) GetTranscript(ctx context.Context, run string) ([]byte, error)
- func (s *Store) List(ctx context.Context, kind string, opts store.ListOptions, out any) (store.ListMeta, error)
- func (s *Store) Loaded() LoadInfo
- func (s *Store) Lock(ctx context.Context, name string, ttl time.Duration) (store.Unlock, error)
- func (s *Store) PutTranscript(ctx context.Context, run string, data []byte) error
- func (s *Store) Ready() error
- func (s *Store) Sweep(ctx context.Context) (store.SweepResult, error)
- func (s *Store) Sync(ctx context.Context) error
- func (s *Store) Update(ctx context.Context, obj store.Object) error
- func (s *Store) Watch(ctx context.Context, kind string, opts store.WatchOptions) (store.Watcher, error)
- type StoreLock
- type TranscriptBlob
- type View
- func (v *View) Get(ctx context.Context, k store.Key, out store.Object) error
- func (v *View) List(ctx context.Context, kind string, opts store.ListOptions, out any) (store.ListMeta, error)
- func (v *View) Loaded() LoadInfo
- func (v *View) Log() []LogEntry
- func (v *View) Transcript(ctx context.Context, run string) ([]byte, error)
- func (v *View) Warnings() []string
Constants ¶
const ( // KindStoreLock is a held lock (store.Store's Lock), named after the lock. KindStoreLock = "StoreLock" // KindTranscriptBlob indexes one transcript blob (or environment log), named after it. KindTranscriptBlob = "TranscriptBlob" )
The object store's own internal kinds. They are written through the log like any other object, so every dispatcher on the bucket sees them. Neither holds a secret.
const ( // DefaultSyncInterval is how stale a read may be before it syncs, and the background // sync's period (--state-sync-interval). DefaultSyncInterval = 2 * time.Second // DefaultMaxLag: a store whose last successful sync started longer ago reloads from a // listing before it writes, and isn't Ready. DefaultMaxLag = time.Hour // DefaultSegmentRetention is how old a log entry covered by a snapshot, and a snapshot that // isn't one of the newest two, must be before they are deleted. DefaultSegmentRetention = 7 * 24 * time.Hour // DefaultCompactEvery is how many log entries past the newest snapshot make the sweep // compact. DefaultCompactEvery = 1000 // DefaultRequestTimeout bounds each bucket request. DefaultRequestTimeout = 5 * time.Second // MaxLostRaces is how many lost races in a row fail a write (ErrContended). MaxLostRaces = 20 )
Defaults (P3).
const DefaultGCSConfirmEvery = time.Minute
DefaultGCSConfirmEvery is how long a request that proved a GCS bucket exists lets a not-found Get or Delete skip its confirming list (GCSBucket).
const S3EndpointEnv = "FACTORY_S3_ENDPOINT"
S3EndpointEnv names the environment variable that points S3 buckets at an S3-compatible endpoint (MinIO, for the emulator tests) instead of AWS. When it is set, requests use path-style addressing (https://endpoint/bucket/key), which such servers expect. It must be https://, except on a loopback host (localhost, 127.0.0.1, ::1), which may use http://: requests carry SigV4 signatures and the state, and only a local emulator goes without TLS. Credentials and the region still come from the default chain (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, AWS_REGION).
Variables ¶
var ( // ErrNotExist: no object with that name. ErrNotExist = errors.New("object does not exist") // ErrExists: Create on a name that is taken. ErrExists = errors.New("object already exists") )
Bucket errors (test with errors.Is).
var ( // ErrClosed: a write after Close. ErrClosed = errors.New("store is closed") // ErrContended: a write lost MaxLostRaces races in a row for the next log entry. ErrContended = errors.New("lost too many races for the next log entry") // ErrSchema: the newest snapshot was written by a newer schema than this release reads, so // Open refuses to start. ErrSchema = errors.New("state written by a newer schema") // ErrCorrupt: a log entry or snapshot can't be loaded, or the log has a gap no snapshot // covers. ErrCorrupt = errors.New("state is corrupt") )
Errors (test with errors.Is).
var ErrInjected = errors.New("objectstore: injected fault")
ErrInjected is the error of a call FakeBucket was told to fail.
Functions ¶
func RegisterScheme ¶
RegisterScheme makes ParseURL open URLs of scheme with open (tests register their own). Registering a scheme again replaces it.
Types ¶
type Bucket ¶
type Bucket interface {
// Get returns the object's bytes. ErrNotExist when there is none.
Get(ctx context.Context, name string) ([]byte, error)
// Create creates the object only if the name is free (GCS ifGenerationMatch=0, S3
// If-None-Match: *, O_EXCL). ErrExists when it is taken. An error other than ErrExists
// leaves the outcome unknown: the object may have been created.
Create(ctx context.Context, name string, data []byte) error
// List calls fn with the objects whose names start with prefix and sort after startAfter
// (all of them when it is ""), in name order. An error from fn stops the listing and is
// returned.
List(ctx context.Context, prefix, startAfter string, fn func(ObjectInfo) error) error
// Delete removes the object. ErrNotExist when there is none.
Delete(ctx context.Context, name string) error
}
Bucket is the little the object store needs from object storage. Names are paths with '/' separators (log/…, snapshots/…, transcripts/…), relative to the store's prefix. Every object is created once and never overwritten (transcript blobs too: each put has its own name).
func ParseURL ¶
ParseURL opens the bucket a state URL names: file:///abs/dir (FileBucket), gs://bucket/prefix (GCSBucket), s3://bucket/prefix (S3Bucket), and the schemes registered with RegisterScheme. An unknown scheme is an error naming the known ones.
A state URL may carry a credential anywhere (s3://key:secret@…, and a secret holding '/', '?', '#' or '"' doesn't even parse as userinfo), so no error about it prints any of it but its scheme: each says which part is wrong instead.
type FakeBucket ¶
type FakeBucket struct {
// contains filtered or unexported fields
}
FakeBucket is an in-memory Bucket for tests: objects stamped with a clock, a count of each call, and a hook that can fail calls, report them as timed out after they landed, or race them (the hook may write to the bucket, through another store, before the call goes on).
func NewFakeBucket ¶
func NewFakeBucket(clk clock.Clock) *FakeBucket
NewFakeBucket returns an empty bucket whose objects' Updated times come from clk.
func (*FakeBucket) Calls ¶
func (b *FakeBucket) Calls(op FakeOp) int
Calls returns how many calls of op the bucket has had (faulted ones included).
func (*FakeBucket) Delete ¶
func (b *FakeBucket) Delete(ctx context.Context, name string) error
Delete implements Bucket.
func (*FakeBucket) List ¶
func (b *FakeBucket) List(ctx context.Context, prefix, startAfter string, fn func(ObjectInfo) error) error
List implements Bucket. fn is called without the bucket's lock, on the listing as it was when the call began.
func (*FakeBucket) Names ¶
func (b *FakeBucket) Names(prefix string) []string
Names returns the names of the objects under prefix, sorted.
func (*FakeBucket) Objects ¶
func (b *FakeBucket) Objects() map[string][]byte
Objects returns a copy of every object's bytes, by name.
func (*FakeBucket) SetFaults ¶
func (b *FakeBucket) SetFaults(f func(FakeRequest) Fault)
SetFaults sets the hook every call goes through first (nil: every call passes). The hook is called without the bucket's lock, so it may call the bucket, or a store on it.
func (*FakeBucket) SetObject ¶
func (b *FakeBucket) SetObject(name string, data []byte)
SetObject writes an object as is, bypassing the fault hook (tests plant objects with it).
type FakeRequest ¶
FakeRequest is one call, as FakeBucket's fault hook sees it.
type Fault ¶
type Fault int
Fault is what FakeBucket does with a call.
const ( // Pass does the call. Pass Fault = iota // Fail fails the call without doing it (ErrInjected). Fail // LandedTimedOut does the call, then fails it as a timeout would (ErrInjected wrapping // context.DeadlineExceeded): the caller can't tell whether it happened. LandedTimedOut // LandedTaken does a Create, then answers ErrExists: an SDK that retried a create after its // first attempt landed. LandedTaken )
Faults.
type FileBucket ¶
type FileBucket struct {
// contains filtered or unexported fields
}
FileBucket is a Bucket on a local directory, for development and for tests with two dispatchers on one machine. Create writes a tmp file, fsyncs it and links it into place (link fails when the name is taken, so the object appears whole or not at all), then fsyncs the directory. The directory is private (0700, files 0600), like a P2 state dir.
func NewFileBucket ¶
func NewFileBucket(dir string) (*FileBucket, error)
NewFileBucket opens (creating it if missing) the directory dir as a bucket. It refuses a relative path, a directory group or others can access, and a P2 state dir (snapshot.json, journal.jsonl or LOCK in it).
func (*FileBucket) Delete ¶
func (b *FileBucket) Delete(ctx context.Context, name string) error
Delete implements Bucket.
func (*FileBucket) List ¶
func (b *FileBucket) List(ctx context.Context, prefix, startAfter string, fn func(ObjectInfo) error) error
List implements Bucket.
type GCSBucket ¶
type GCSBucket struct {
// contains filtered or unexported fields
}
GCSBucket is a Bucket on Google Cloud Storage (gs://bucket/prefix). Credentials are Application Default Credentials (Cloud Run's service account); none come from a flag or the config. STORAGE_EMULATOR_HOST (the SDK's own variable) points it at an emulator such as fake-gcs-server instead, without credentials.
Every call goes through the JSON API, reads included, so one set of errors applies:
- Create uploads with ifGenerationMatch=0 in one request; a 412 is ErrExists. The SDK retries it on a 408, 429, 5xx or a broken connection (a conditional upload is idempotent), so a retry whose first attempt landed answers ErrExists, which the store's nonce check resolves.
- Get and Delete answer ErrNotExist on a missing object's 404. A missing bucket is not ErrNotExist: gcsBucketMissing's message match is the fast path, and otherwise a one-object list of the prefix confirms the bucket (notFound), at most once per DefaultGCSConfirmEvery: a request that proved the bucket exists within it (a successful Get, Create, List or Delete, a 412 on a create, or a confirmed 404) skips the list. ParseURL lists the prefix once at open, so a bucket that doesn't exist fails the start rather than reading as an empty, current log.
- List lists from startOffset, in name order, skipping startAfter itself.
Each request's deadline is its context's: the store gives each one DefaultRequestTimeout.
type LoadInfo ¶
type LoadInfo struct {
// Writer is this store's instance id.
Writer string
// Version is the store version loaded (0 for an empty bucket).
Version uint64
// Objects counts the objects loaded, by kind.
Objects map[string]int
// Snapshot is the version of the snapshot loaded (0 when there was none); LogEntries
// counts the log entries replayed above it.
Snapshot uint64
LogEntries int
// Transcripts counts the transcript blobs indexed.
Transcripts int
// ReadOnly says the log holds an entry past this release's ReadableSchema: the store
// stopped writing (Ready says why).
ReadOnly bool
}
LoadInfo is what Open loaded, for the caller's start-up log.
type LogEntry ¶
type LogEntry struct {
Version uint64
At time.Time
Op memory.Op
Kind string
Name string
// Obj is the object as stored by the change (nil for delete).
Obj json.RawMessage
}
LogEntry is one log entry a View read above its snapshot: one committed change.
type ObjectInfo ¶
type ObjectInfo struct {
Name string
Size int64
// Updated is when the object was last written, on the bucket's clock.
Updated time.Time
}
ObjectInfo is one listed object.
type Opener ¶
Opener opens the bucket a parsed state URL names. ctx bounds the opening (finding credentials, reading the SDK's config), not the bucket's later requests.
type Option ¶
type Option func(*config)
Option configures Open.
func WithCompactEvery ¶
WithCompactEvery sets how many log entries past the newest snapshot make the sweep compact (default DefaultCompactEvery).
func WithKind ¶
WithKind registers the Go type of kind, a stored kind outside v1alpha1 (the internal Lease, Session, Delivery and the like), as filestore.WithKind does: syncs and loads need each kind's type, and a write of a kind without one is ErrInvalid. StoreLock and TranscriptBlob are always registered.
func WithMaxLag ¶
WithMaxLag sets the lag past which a store reloads before writing and isn't Ready (default DefaultMaxLag). It must be under the segment retention.
func WithRequestTimeout ¶
WithRequestTimeout sets each bucket request's deadline (default DefaultRequestTimeout).
func WithSegmentRetention ¶
WithSegmentRetention sets how long covered log entries and old snapshots are kept (default DefaultSegmentRetention).
func WithSyncInterval ¶
WithSyncInterval sets how stale a read may be before it syncs, and the background sync's period (default DefaultSyncInterval).
func WithWriterID ¶
WithWriterID sets the store's instance id, which every log entry it writes and every lock it holds carry (default random). Two stores on one bucket must not share one.
type S3Bucket ¶
type S3Bucket struct {
// contains filtered or unexported fields
}
S3Bucket is a Bucket on Amazon S3 (s3://bucket/prefix), or on an S3-compatible server when FACTORY_S3_ENDPOINT is set. Credentials come from the SDK's default chain (Lambda's role, or the environment); none come from a flag or the config.
- Create is PutObject with If-None-Match: *; a 412 is ErrExists. The SDK retries a PUT that failed with a 5xx or a timeout, so a retry whose first attempt landed answers ErrExists, which the store's nonce check resolves. A 409 ConditionalRequestConflict (a conditional write to the key in flight) is re-sent up to 3 times after a short wait; when it persists, the outcome is unknown, as for a timeout.
- Get answers ErrNotExist on NoSuchKey (not on NoSuchBucket).
- List is ListObjectsV2 with StartAfter, in name order.
- Delete is HeadObject, then DeleteObject: S3's delete succeeds whether or not the key exists, and the Bucket contract answers ErrNotExist for a missing object.
Each request's deadline is its context's: the store gives each one DefaultRequestTimeout.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is the object store. Use Open, and Close when done.
func Open ¶
Open loads the state from b (the newest snapshot, then the log above it) and returns the store, with its background sync running. An empty bucket is a new, empty state. Open refuses a snapshot written by a newer schema than this release reads (ErrSchema) and a log it can't load (ErrCorrupt, saying where); an entry past ReadableSchema in the log loads what comes before it and leaves the store read-only (Ready says so). opts are the memory store's.
func (*Store) Close ¶
Close stops the background sync and compacts the log into a snapshot (when there are entries past the newest one this store knows of). Writes after Close fail with ErrClosed; reads still answer from memory. It is idempotent.
func (*Store) Compact ¶
Compact writes a snapshot of the store now, when the log has entries past the newest one this store knows of (the sweep does it every CompactEvery entries, and Close at the end).
func (*Store) Create ¶
Create implements store.Store. The change is committed by creating its log entry.
func (*Store) Delete ¶
Delete implements store.Store. A Run's transcript blobs and their records go after the Run's own delete commits.
func (*Store) DeleteIfUnchanged ¶
DeleteIfUnchanged implements store.UnchangedDeleter, through the log. A lagging store syncs before it answers ErrConflict or ErrNotFound, as any refused write does.
func (*Store) Get ¶
Get implements store.Store. It syncs first when the store's last sync is older than the sync interval, and once more before answering ErrNotFound, so an object another dispatcher just wrote is found.
func (*Store) GetTranscript ¶
GetTranscript implements store.TranscriptStore: the record (synced once when it isn't found), then the blob it names. A blob a re-put replaced between the two reads is followed once to the new record.
func (*Store) List ¶
func (s *Store) List(ctx context.Context, kind string, opts store.ListOptions, out any) (store.ListMeta, error)
List implements store.Store. It syncs first when the store's last sync is older than the sync interval. A caller that acts on an empty answer syncs and asks again first (store.SyncIfSupported).
func (*Store) Lock ¶
Lock implements store.Store with a StoreLock record written through the log, so a lock holds across every dispatcher on the bucket. Lock creates the record, or takes over an expired one with a compare-and-swap. While another holder has it, Lock waits until the record is released or expires, or ctx ends: it wakes on a local Unlock, on the store's clock reaching the expiry, on a sync that changes a lock record, and syncs every second to see other dispatchers' releases.
func (*Store) PutTranscript ¶
PutTranscript implements store.TranscriptStore. Each put creates a new blob, transcripts/<run>.<id>.jsonl.gz with a random id, create-if-absent; then the run's TranscriptBlob record {id, size, putAt} is written through the log, and the blob the record named before is deleted. So a drop (which deletes only the blob its deleted record named) and a re-put on another dispatcher can't remove each other's blob, in either order. A crash before the record leaves an orphan blob, which the daily orphan scan deletes. The caps are the memory store's: over the per-transcript cap is ErrTooLarge, and the blobs with the oldest putAt (then the lowest resourceVersion) are evicted, logged, to stay under the total.
func (*Store) Ready ¶
Ready says whether the store can serve: nil once its first sync (the load) has succeeded, while it still writes and while its last successful sync started within MaxLag. /readyz answers 503 with the error otherwise.
func (*Store) Sweep ¶
Sweep implements store.Sweeper. It deletes, through the log:
- the objects past retention (store.RetentionOf), as memory's sweep does;
- the expired lock records (a crashed holder, or an Unlock that failed);
- the transcript blobs whose putAt is past retention, with their records.
Then it compacts when the log is CompactEvery entries past the newest snapshot, and at its first run and once a day after it deletes the log entries and snapshots past the segment retention and the orphan transcript blobs. The leader runs it (store.RunSweeper).
func (*Store) Sync ¶
Sync implements store.Syncer: it brings the store up to the log's tail, reading the entries after its version until one is missing (one GET of a missing object when it is current). A sync already running when Sync is called doesn't count; one that starts after does, so concurrent callers share one. ErrReadOnly when the log holds an entry past ReadableSchema: the store can't read on.
func (*Store) Update ¶
Update implements store.Store. The compare-and-swap is checked against memory, which a lost race brings up to the log's tail before the update runs again.
func (*Store) Watch ¶
func (s *Store) Watch(ctx context.Context, kind string, opts store.WatchOptions) (store.Watcher, error)
Watch implements store.Store. Changes applied from the log fire like local writes, in version order. After a cold start the history begins at the loaded version, so an older SinceVersion is ErrWatchExpired; a reload from a snapshot expires every watcher.
type StoreLock ¶
type StoreLock struct {
v1alpha1.TypeMeta
Metadata v1alpha1.ObjectMeta `json:"metadata"`
// Holder is the holding store's instance id.
Holder string `json:"holder"`
// Token is random per Lock: it tells two holds by one instance apart in the log.
Token string `json:"token"`
ExpiresAt time.Time `json:"expiresAt"`
}
StoreLock is a lock held across dispatchers: who holds it until when, on the holder's store clock. Lock creates it, or takes it over with a compare-and-swap once it has expired; Renew updates it; Unlock deletes it at the version its holder last wrote.
func (*StoreLock) GetObjectMeta ¶
func (l *StoreLock) GetObjectMeta() *v1alpha1.ObjectMeta
GetObjectMeta implements store.MetaObject.
type TranscriptBlob ¶
type TranscriptBlob struct {
v1alpha1.TypeMeta
Metadata v1alpha1.ObjectMeta `json:"metadata"`
ID string `json:"id"`
Size int64 `json:"size"`
PutAt time.Time `json:"putAt"`
}
TranscriptBlob indexes a transcript's current blob: its id (each put writes a new blob, transcripts/<run>.<id>.jsonl.gz), its size and when it was put, on the store's clock. Reads follow the id; the caps, eviction (oldest PutAt first) and retention read the records from memory, since a bucket listing can't give them on every backend. createdAt can't serve, because an update keeps it and a replaced blob counts as new.
func (*TranscriptBlob) GetName ¶
func (b *TranscriptBlob) GetName() string
GetName returns metadata.name: the blob's run name (store.TranscriptStore's).
func (*TranscriptBlob) GetObjectMeta ¶
func (b *TranscriptBlob) GetObjectMeta() *v1alpha1.ObjectMeta
GetObjectMeta implements store.MetaObject.
type View ¶
type View struct {
// contains filtered or unexported fields
}
View is the state in a bucket, read once: the newest snapshot and the log above it. It writes nothing, takes no lock and doesn't sync, so it works while dispatchers run on the bucket (factory state --state-url). An entry past ReadableSchema ends the view before it, with a warning.
func Read ¶
Read reads the state in b, read-only. options are Open's (WithKind for the stored kinds outside v1alpha1; the timings that apply to reads).
func (*View) List ¶
func (v *View) List(ctx context.Context, kind string, opts store.ListOptions, out any) (store.ListMeta, error)
List lists objects, as store.Store's List.
func (*View) Transcript ¶
Transcript returns run's transcript blob as stored (gzip'd); store.ErrNotFound when it has none.