Documentation
¶
Overview ¶
Package s3 is the cachestore driver for in-DC S3-compatible stores. In production this targets VAST or another S3-compatible object store; in dev it targets a self-hosted S3-compatible store.
Commit uses stat-then-put: HeadObject the chunk path, and if it is already present skip the upload and report ErrCommitLost (another replica won the fill race). This needs only GET/PUT/HEAD. Correctness rests on the content-addressed layout: the ETag is part of the chunk path, so two writers racing on the same key always carry byte-identical bytes (see designs/orca/design.md s8). The boot SelfTest verifies read-after-write visibility; the versioning gate refuses versioned buckets, where retaining every overwrite of an immutable chunk would only waste space.
Index ¶
- type Config
- type Driver
- func (d *Driver) Delete(ctx context.Context, k chunk.Key) error
- func (d *Driver) GetChunk(ctx context.Context, k chunk.Key, off, n int64) (io.ReadCloser, error)
- func (d *Driver) PutChunk(ctx context.Context, k chunk.Key, size int64, r io.Reader) error
- func (d *Driver) SelfTest(ctx context.Context) error
- func (d *Driver) Stat(ctx context.Context, k chunk.Key) (cachestore.Info, error)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
Endpoint string
Bucket string
Region string
AccessKey string
SecretKey string
UsePathStyle bool
}
Config is the s3-driver configuration. Mirrors config.CachestoreS3 but kept package-local so the driver can be unit-tested without importing the whole config package.
type Driver ¶
type Driver struct {
// contains filtered or unexported fields
}
Driver implements cachestore.CacheStore against an S3-compatible endpoint.
func New ¶
New constructs a Driver. The bucket-versioning gate is run here unconditionally: a versioned bucket retains every overwrite, and because Orca's chunks are immutable and only ever re-committed with byte-identical content, versioning would accumulate redundant object versions that only waste space. The driver refuses to start against a versioned bucket.
The log receives debug-level emissions for every chunk operation (Get, Put, Stat, Delete) and step-by-step boot trace from SelfTest / versioningGate. Passing nil falls back to slog.Default().
SelfTest is a separate step (called by main after New) to keep the constructor side-effect-light.
func (*Driver) GetChunk ¶
GetChunk fetches [off, off+n) of the chunk path from the bucket.
Rejects n <= 0 with a sentinel ErrInvalidArgument: the wire-format boundary (cluster.DecodeChunkKey) already rejects object_size <= 0, so an in-process caller asking for a zero-length read is a logic bug. Forwarding the request would yield a malformed S3 Range header (bytes=0--1).
func (*Driver) PutChunk ¶
PutChunk commits the chunk with a stat-then-put step: HeadObject the chunk path, and if it is already present skip the upload and return ErrCommitLost (another replica won the fill race; the existing object is byte-identical because the ETag is part of the path). Otherwise upload via plain PutObject.
The HeadObject/PutObject pair has a benign time-of-check/time-of-use window: two replicas may both observe the key absent and both upload. That degrades to a last-writer-wins overwrite of byte-identical content, so readers always observe correct bytes.
Rejects size <= 0 with a sentinel error: a zero-byte chunk is never a legitimate fill result (the wire-format boundary already rejects object_size <= 0, and the smallest legitimate tail chunk is 1 byte), and uploading a zero-byte object would poison the path so later GetChunk(n=expected) reads return 0 bytes and break the streaming model.
func (*Driver) SelfTest ¶ added in v0.1.11
SelfTest verifies the backend provides read-after-write visibility, which the stat-then-put commit step depends on: a chunk this replica (or any peer) just wrote must be observable by a subsequent HeadObject so concurrent fills converge instead of all re-uploading. It writes a probe key, confirms it is visible via HeadObject, reads the bytes back via GetObject, and deletes the probe.
Source Files