Documentation
¶
Overview ¶
Package s3 implements a backend.Backend over an S3-compatible object store. The store-specific calls are isolated behind the small ObjectStore interface: the AWS SDK adapter (NewAWS) implements it for real S3, and tests implement it with an in-memory fake, so all of the Backend's mapping logic — key prefixing, sorted listing, 404 → backend.ErrNotExist translation, conditional-put, and existence-checked delete — is exercised by the shared backend conformance suite without a live bucket.
The object store is the durable, stateless tier: nodes hold no authoritative state, so the read path is reconstructed entirely from objects (DESIGN.md §3, §11). Whole-object Get/Put is sufficient because a part maps to one key prefix with one object per column/marks/manifest.
Index ¶
- Variables
- type AWSAPI
- type Backend
- func (b *Backend) Delete(ctx context.Context, key string) error
- func (*Backend) IsEphemeral() bool
- func (b *Backend) List(ctx context.Context, prefix string) ([]string, error)
- func (b *Backend) PutIfAbsent(ctx context.Context, key string, data []byte) (bool, error)
- func (b *Backend) Read(ctx context.Context, key string) ([]byte, error)
- func (b *Backend) Write(ctx context.Context, key string, data []byte) error
- type ObjectStore
Constants ¶
This section is empty.
Variables ¶
var ErrObjectNotFound = errors.New("s3: object not found")
ErrObjectNotFound is returned (wrapped) by ObjectStore.GetObject for an absent key. The Backend translates it to backend.ErrNotExist.
Functions ¶
This section is empty.
Types ¶
type AWSAPI ¶
type AWSAPI interface {
GetObject(ctx context.Context, in *awss3.GetObjectInput, optFns ...func(*awss3.Options)) (*awss3.GetObjectOutput, error)
PutObject(ctx context.Context, in *awss3.PutObjectInput, optFns ...func(*awss3.Options)) (*awss3.PutObjectOutput, error)
HeadObject(ctx context.Context, in *awss3.HeadObjectInput, optFns ...func(*awss3.Options)) (*awss3.HeadObjectOutput, error)
DeleteObject(ctx context.Context, in *awss3.DeleteObjectInput, optFns ...func(*awss3.Options)) (*awss3.DeleteObjectOutput, error)
ListObjectsV2(ctx context.Context, in *awss3.ListObjectsV2Input, optFns ...func(*awss3.Options)) (*awss3.ListObjectsV2Output, error)
}
AWSAPI is the subset of the aws-sdk-go-v2 *s3.Client the NewAWS adapter uses. The real client satisfies it; tests can fake it. It also satisfies s3.ListObjectsV2APIClient (used by the paginator).
type Backend ¶
type Backend struct {
// contains filtered or unexported fields
}
Backend is a backend.Backend over an ObjectStore. Keys are stored under an optional root prefix so several datasets can share one bucket.
func New ¶
func New(store ObjectStore, keyPrefix string) *Backend
New returns a Backend over store, rooting all keys under keyPrefix (which may be empty).
func (*Backend) Delete ¶
Delete removes key, returning an backend.ErrNotExist-wrapping error if it is absent. Because S3 DeleteObject is idempotent, existence is checked first to honor the contract.
func (*Backend) IsEphemeral ¶
IsEphemeral reports false: objects persist in the store.
func (*Backend) PutIfAbsent ¶
PutIfAbsent stores data under key only if it does not already exist.
func (*Backend) Read ¶
Read returns the value stored under key, or an backend.ErrNotExist-wrapping error.
type ObjectStore ¶
type ObjectStore interface {
// GetObject returns the object bytes, or an error wrapping [ErrObjectNotFound] if absent.
GetObject(ctx context.Context, key string) ([]byte, error)
// PutObject stores data under key, overwriting any existing object.
PutObject(ctx context.Context, key string, data []byte) error
// PutObjectIfAbsent stores data under key only if it does not exist, returning whether it
// was created (the conditional write — S3 If-None-Match: *).
PutObjectIfAbsent(ctx context.Context, key string, data []byte) (bool, error)
// HeadObject reports whether key exists.
HeadObject(ctx context.Context, key string) (bool, error)
// DeleteObject removes key. It is idempotent (no error if the key is absent), mirroring
// S3 DeleteObject.
DeleteObject(ctx context.Context, key string) error
// ListObjects returns every key under prefix (the implementation paginates internally).
// Order is unspecified; the [Backend] sorts.
ListObjects(ctx context.Context, prefix string) ([]string, error)
}
ObjectStore is the minimal object-store API the Backend needs. It is intentionally thin and S3-shaped (delete is idempotent, list is by prefix); the Backend layers the backend.Backend contract on top. Implementations must be safe for concurrent use.
func NewAWS ¶
func NewAWS(api AWSAPI, bucket string) ObjectStore
NewAWS returns an ObjectStore backed by an aws-sdk-go-v2 S3 client over the given bucket. Compose it with New to get a backend.Backend:
store := s3.NewAWS(awss3.NewFromConfig(cfg), "my-bucket") b := s3.New(store, "oteldb/")
This adapter is verified by compilation and exercised against real/MinIO S3 in integration tests; the Backend's contract logic is covered by the conformance suite over the in-memory fake.
Source Files