Documentation
¶
Overview ¶
Package objectstore provides an instrumented, backend-agnostic object store client built on gocloud.dev/blob.
It wraps a blob.Bucket in a labkit-owned Bucket that mirrors the upstream method surface and adds tracing, Prometheus metrics, and structured logging around every operation, so telemetry is identical across S3, GCS, Azure and other backends.
It is a decorator: the blob.* option and result types are re-exported and the As / ErrorAs escape hatches are preserved, so callers keep full access to backend-specific behaviour. Backend settings are configured when the blob.Bucket is opened; objectstore adds only observability on top.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrMissingConfigField = errors.New("objectstore: required config field missing")
ErrMissingConfigField is returned when a required Config field is unset.
var ErrMissingSecrets = errors.New("objectstore: secret provider missing")
ErrMissingSecrets is returned by New when the infrastructure config carries no secret provider to resolve credentials.
var ErrUnknownObjectStore = errors.New("objectstore: unknown object store")
ErrUnknownObjectStore is returned by New when the infrastructure config selects an unrecognised object store backend.
var ErrUnsupportedBackend = errors.New("objectstore: unsupported backend")
ErrUnsupportedBackend is returned when no provider is registered for the configured Backend.
Functions ¶
Types ¶
type AzureCredentials ¶
type AzureCredentials struct {
// ConnectionString is a full Azure Storage connection string. It carries its
// own account name, key, and endpoint, so Config.AccountName is not required
// when it is set. Also the simplest way to target the Azurite emulator.
ConnectionString string
// AccountKey is the storage account shared key, used with Config.AccountName.
AccountKey string
// SASToken authorizes requests via a shared access signature appended to the
// service URL; the client is otherwise unauthenticated.
SASToken string
}
AzureCredentials carries resolved Azure Blob Storage credentials. A nil *AzureCredentials (or one with all fields empty) uses the SDK's default credential chain (Managed Identity, Workload Identity, environment).
At most one field should be set; they are tried in the order below.
type Bucket ¶ added in v2.38.0
type Bucket interface {
ReadAll(ctx context.Context, key string) ([]byte, error)
WriteAll(ctx context.Context, key string, p []byte, opts *blob.WriterOptions) error
Download(ctx context.Context, key string, w io.Writer, opts *blob.ReaderOptions) error
Upload(ctx context.Context, key string, r io.Reader, opts *blob.WriterOptions) error
NewReader(ctx context.Context, key string, opts *blob.ReaderOptions) (*Reader, error)
NewRangeReader(ctx context.Context, key string, offset, length int64, opts *blob.ReaderOptions) (*Reader, error)
NewWriter(ctx context.Context, key string, opts *blob.WriterOptions) (*Writer, error)
Copy(ctx context.Context, dstKey, srcKey string, opts *blob.CopyOptions) error
Delete(ctx context.Context, key string) error
Exists(ctx context.Context, key string) (bool, error)
Attributes(ctx context.Context, key string) (*blob.Attributes, error)
SignedURL(ctx context.Context, key string, opts *blob.SignedURLOptions) (string, error)
IsAccessible(ctx context.Context) (bool, error)
List(opts *blob.ListOptions) *ListIterator
ListPage(ctx context.Context, pageToken []byte, pageSize int, opts *blob.ListOptions) ([]*blob.ListObject, []byte, error)
As(i any) bool
ErrorAs(err error, i any) bool
Close() error
}
Bucket is an instrumented object store handle mirroring gocloud.dev/blob.Bucket. The As and ErrorAs escape hatches expose the underlying driver types.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client opens a backend bucket and exposes it as an instrumented Bucket. It implements app.Component: the bucket is opened in Start, so Bucket returns nil until Start has run.
func New ¶ added in v2.25.0
New returns a Client configured from the environment, reading the infrastructure configuration from the standard config path and resolving credentials from its secret provider. Returns infrastructure.ErrNotConfigured when no ObjectStore config is present.
The backend is not opened here; that happens in Start (see NewWithConfig).
func NewWithConfig ¶
NewWithConfig validates cfg, resolves the backend provider, and registers metrics. It does not open the backend; that happens in Start so the lifecycle context governs the open and app.App's start-time retry applies. The backend's provider package must be blank-imported so it registers itself.
func (*Client) Bucket ¶
Bucket returns the instrumented object store handle. It is nil until Start has been called.
type Config ¶
type Config struct {
Name string
Namespace string
Subsystem string
Backend Backend
BucketName string
// S3 and S3-compatible backends. Endpoint overrides the default service
// URL (S3-compatibles, emulators); PathStyle forces path-style addressing.
Region string
Endpoint string
PathStyle bool
S3Credentials *S3Credentials
// GCS backends. UniverseDomain selects a non-default universe (sovereign
// clouds); Endpoint (above) may point at an emulator.
UniverseDomain string
GCSCredentials *GCSCredentials
// Azure Blob backends. AccountName identifies the storage account (not
// required when AzureCredentials.ConnectionString is set). StorageDomain
// overrides the default blob.core.windows.net (sovereign clouds) or points at
// an emulator host such as "127.0.0.1:10000". Protocol defaults to "https";
// set it to "http" for an emulator served over plain HTTP. AzureEmulator
// selects local-emulator (Azurite) addressing; leave it false for real Azure.
AccountName string
StorageDomain string
Protocol string
AzureEmulator bool
AzureCredentials *AzureCredentials
Tracer *trace.Tracer
Registerer prometheus.Registerer
Logger *slog.Logger
}
Config configures a Client. Backend and BucketName are required; identity fields default to objectstore/gitlab/objectstore, and each observability collaborator is optional.
type GCSCredentials ¶
type GCSCredentials struct {
// JSON is the raw service-account key JSON. When empty (or GCSCredentials
// is nil), the SDK falls back to Application Default Credentials (Workload
// Identity, metadata server, GOOGLE_APPLICATION_CREDENTIALS).
JSON []byte
}
GCSCredentials carries resolved GCS credentials.
type ListIterator ¶ added in v2.38.0
type ListIterator struct {
// contains filtered or unexported fields
}
ListIterator wraps *blob.ListIterator, instrumenting each Next.
func (*ListIterator) Next ¶ added in v2.38.0
func (i *ListIterator) Next(ctx context.Context) (*blob.ListObject, error)
type Option ¶ added in v2.25.0
type Option func(*clientOptions)
Option configures a Client created via New.
func WithInfraConfig ¶ added in v2.25.0
func WithInfraConfig(c *infrastructure.Config) Option
WithInfraConfig sets the infrastructure.Config used to read configuration and secrets. When nil, the default config is loaded.
func WithLogger ¶ added in v2.38.0
WithLogger sets the structured logger for operation logs.
func WithRegisterer ¶ added in v2.38.0
func WithRegisterer(r prometheus.Registerer) Option
WithRegisterer sets the Prometheus registerer for operation metrics.
func WithTracer ¶ added in v2.38.0
WithTracer sets the tracer used for operation spans.
type Reader ¶ added in v2.38.0
Reader wraps a *blob.Reader so the operation span covers the whole read (NewReader to Close) and the transferred byte count is recorded. It embeds *blob.Reader, inheriting Seek, Size, ContentType, ModTime and As; only the byte-transfer methods and Close are overridden. The operation is finalised on Close, so the caller must Close the Reader for it to be recorded.
Both Read and WriteTo are overridden: *blob.Reader implements io.WriterTo, so io.Copy would otherwise drain through the promoted WriteTo and bypass the counter.
type S3Credentials ¶
type S3Credentials struct {
AccessKeyID string
SecretAccessKey string
SessionToken string // optional; STS / temporary credentials
}
S3Credentials carries resolved S3 credentials. A nil *S3Credentials uses the SDK's default credential chain (IRSA, environment, instance profile).
type Writer ¶ added in v2.38.0
Writer wraps a *blob.Writer with the same lifecycle and byte-counting treatment. It embeds *blob.Writer; only Write, ReadFrom and Close are overridden.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
providers
|
|
|
azure
Package azure registers the Azure Blob Storage backend for objectstore.
|
Package azure registers the Azure Blob Storage backend for objectstore. |
|
gcs
Package gcs registers the Google Cloud Storage backend for objectstore.
|
Package gcs registers the Google Cloud Storage backend for objectstore. |
|
mem
Package mem registers an in-memory object store backend for objectstore.
|
Package mem registers an in-memory object store backend for objectstore. |
|
s3
Package s3 registers the S3 (and S3-compatible) object store backend for objectstore.
|
Package s3 registers the S3 (and S3-compatible) object store backend for objectstore. |