objectstore

package
v2.38.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 28, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package objectstore provides an instrumented, backend-agnostic object store client built on gocloud.dev/blob.

It wraps a blob.Bucket in a labkit-owned Bucket that mirrors the upstream method surface and adds tracing, Prometheus metrics, and structured logging around every operation, so telemetry is identical across S3, GCS, Azure and other backends.

It is a decorator: the blob.* option and result types are re-exported and the As / ErrorAs escape hatches are preserved, so callers keep full access to backend-specific behaviour. Backend settings are configured when the blob.Bucket is opened; objectstore adds only observability on top.

Index

Constants

This section is empty.

Variables

View Source
var ErrMissingConfigField = errors.New("objectstore: required config field missing")

ErrMissingConfigField is returned when a required Config field is unset.

View Source
var ErrMissingSecrets = errors.New("objectstore: secret provider missing")

ErrMissingSecrets is returned by New when the infrastructure config carries no secret provider to resolve credentials.

View Source
var ErrUnknownObjectStore = errors.New("objectstore: unknown object store")

ErrUnknownObjectStore is returned by New when the infrastructure config selects an unrecognised object store backend.

View Source
var ErrUnsupportedBackend = errors.New("objectstore: unsupported backend")

ErrUnsupportedBackend is returned when no provider is registered for the configured Backend.

Functions

func Register

func Register(p provider)

Register adds p to the dispatch table. It panics on duplicate registration, which is always a build-time mistake.

Types

type AzureCredentials

type AzureCredentials struct {
	// ConnectionString is a full Azure Storage connection string. It carries its
	// own account name, key, and endpoint, so Config.AccountName is not required
	// when it is set. Also the simplest way to target the Azurite emulator.
	ConnectionString string

	// AccountKey is the storage account shared key, used with Config.AccountName.
	AccountKey string

	// SASToken authorizes requests via a shared access signature appended to the
	// service URL; the client is otherwise unauthenticated.
	SASToken string
}

AzureCredentials carries resolved Azure Blob Storage credentials. A nil *AzureCredentials (or one with all fields empty) uses the SDK's default credential chain (Managed Identity, Workload Identity, environment).

At most one field should be set; they are tried in the order below.

type Backend

type Backend string

Backend selects the object store SDK that backs a Client.

const (
	BackendMem   Backend = "mem"
	BackendS3    Backend = "s3"
	BackendGCS   Backend = "gcs"
	BackendAzure Backend = "azure"
)

type Bucket added in v2.38.0

type Bucket interface {
	ReadAll(ctx context.Context, key string) ([]byte, error)
	WriteAll(ctx context.Context, key string, p []byte, opts *blob.WriterOptions) error
	Download(ctx context.Context, key string, w io.Writer, opts *blob.ReaderOptions) error
	Upload(ctx context.Context, key string, r io.Reader, opts *blob.WriterOptions) error

	NewReader(ctx context.Context, key string, opts *blob.ReaderOptions) (*Reader, error)
	NewRangeReader(ctx context.Context, key string, offset, length int64, opts *blob.ReaderOptions) (*Reader, error)
	NewWriter(ctx context.Context, key string, opts *blob.WriterOptions) (*Writer, error)

	Copy(ctx context.Context, dstKey, srcKey string, opts *blob.CopyOptions) error
	Delete(ctx context.Context, key string) error
	Exists(ctx context.Context, key string) (bool, error)
	Attributes(ctx context.Context, key string) (*blob.Attributes, error)
	SignedURL(ctx context.Context, key string, opts *blob.SignedURLOptions) (string, error)
	IsAccessible(ctx context.Context) (bool, error)

	List(opts *blob.ListOptions) *ListIterator
	ListPage(ctx context.Context, pageToken []byte, pageSize int, opts *blob.ListOptions) ([]*blob.ListObject, []byte, error)

	As(i any) bool
	ErrorAs(err error, i any) bool
	Close() error
}

Bucket is an instrumented object store handle mirroring gocloud.dev/blob.Bucket. The As and ErrorAs escape hatches expose the underlying driver types.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client opens a backend bucket and exposes it as an instrumented Bucket. It implements app.Component: the bucket is opened in Start, so Bucket returns nil until Start has run.

func New added in v2.25.0

func New(ctx context.Context, opts ...Option) (*Client, error)

New returns a Client configured from the environment, reading the infrastructure configuration from the standard config path and resolving credentials from its secret provider. Returns infrastructure.ErrNotConfigured when no ObjectStore config is present.

The backend is not opened here; that happens in Start (see NewWithConfig).

func NewWithConfig

func NewWithConfig(cfg *Config) (*Client, error)

NewWithConfig validates cfg, resolves the backend provider, and registers metrics. It does not open the backend; that happens in Start so the lifecycle context governs the open and app.App's start-time retry applies. The backend's provider package must be blank-imported so it registers itself.

func (*Client) Backend

func (c *Client) Backend() Backend

func (*Client) Bucket

func (c *Client) Bucket() Bucket

Bucket returns the instrumented object store handle. It is nil until Start has been called.

func (*Client) Name

func (c *Client) Name() string

func (*Client) Shutdown

func (c *Client) Shutdown(_ context.Context) error

Shutdown closes the bucket and releases the backend client. It is a no-op if Start never ran.

func (*Client) Start

func (c *Client) Start(ctx context.Context) error

Start opens the backend bucket under ctx and wraps it with observability. It is idempotent.

type Config

type Config struct {
	Name      string
	Namespace string
	Subsystem string

	Backend    Backend
	BucketName string

	// S3 and S3-compatible backends. Endpoint overrides the default service
	// URL (S3-compatibles, emulators); PathStyle forces path-style addressing.
	Region        string
	Endpoint      string
	PathStyle     bool
	S3Credentials *S3Credentials

	// GCS backends. UniverseDomain selects a non-default universe (sovereign
	// clouds); Endpoint (above) may point at an emulator.
	UniverseDomain string
	GCSCredentials *GCSCredentials

	// Azure Blob backends. AccountName identifies the storage account (not
	// required when AzureCredentials.ConnectionString is set). StorageDomain
	// overrides the default blob.core.windows.net (sovereign clouds) or points at
	// an emulator host such as "127.0.0.1:10000". Protocol defaults to "https";
	// set it to "http" for an emulator served over plain HTTP. AzureEmulator
	// selects local-emulator (Azurite) addressing; leave it false for real Azure.
	AccountName      string
	StorageDomain    string
	Protocol         string
	AzureEmulator    bool
	AzureCredentials *AzureCredentials

	Tracer     *trace.Tracer
	Registerer prometheus.Registerer
	Logger     *slog.Logger
}

Config configures a Client. Backend and BucketName are required; identity fields default to objectstore/gitlab/objectstore, and each observability collaborator is optional.

type GCSCredentials

type GCSCredentials struct {
	// JSON is the raw service-account key JSON. When empty (or GCSCredentials
	// is nil), the SDK falls back to Application Default Credentials (Workload
	// Identity, metadata server, GOOGLE_APPLICATION_CREDENTIALS).
	JSON []byte
}

GCSCredentials carries resolved GCS credentials.

type ListIterator added in v2.38.0

type ListIterator struct {
	// contains filtered or unexported fields
}

ListIterator wraps *blob.ListIterator, instrumenting each Next.

func (*ListIterator) Next added in v2.38.0

func (i *ListIterator) Next(ctx context.Context) (*blob.ListObject, error)

type Option added in v2.25.0

type Option func(*clientOptions)

Option configures a Client created via New.

func WithInfraConfig added in v2.25.0

func WithInfraConfig(c *infrastructure.Config) Option

WithInfraConfig sets the infrastructure.Config used to read configuration and secrets. When nil, the default config is loaded.

func WithLogger added in v2.38.0

func WithLogger(l *slog.Logger) Option

WithLogger sets the structured logger for operation logs.

func WithRegisterer added in v2.38.0

func WithRegisterer(r prometheus.Registerer) Option

WithRegisterer sets the Prometheus registerer for operation metrics.

func WithTracer added in v2.38.0

func WithTracer(t *trace.Tracer) Option

WithTracer sets the tracer used for operation spans.

type Reader added in v2.38.0

type Reader struct {
	*blob.Reader
	// contains filtered or unexported fields
}

Reader wraps a *blob.Reader so the operation span covers the whole read (NewReader to Close) and the transferred byte count is recorded. It embeds *blob.Reader, inheriting Seek, Size, ContentType, ModTime and As; only the byte-transfer methods and Close are overridden. The operation is finalised on Close, so the caller must Close the Reader for it to be recorded.

Both Read and WriteTo are overridden: *blob.Reader implements io.WriterTo, so io.Copy would otherwise drain through the promoted WriteTo and bypass the counter.

func (*Reader) Close added in v2.38.0

func (r *Reader) Close() error

Close finalises the operation and returns the underlying Close error.

func (*Reader) Read added in v2.38.0

func (r *Reader) Read(p []byte) (int, error)

func (*Reader) WriteTo added in v2.38.0

func (r *Reader) WriteTo(w io.Writer) (int64, error)

type S3Credentials

type S3Credentials struct {
	AccessKeyID     string
	SecretAccessKey string
	SessionToken    string // optional; STS / temporary credentials
}

S3Credentials carries resolved S3 credentials. A nil *S3Credentials uses the SDK's default credential chain (IRSA, environment, instance profile).

type Writer added in v2.38.0

type Writer struct {
	*blob.Writer
	// contains filtered or unexported fields
}

Writer wraps a *blob.Writer with the same lifecycle and byte-counting treatment. It embeds *blob.Writer; only Write, ReadFrom and Close are overridden.

func (*Writer) Close added in v2.38.0

func (w *Writer) Close() error

Close flushes the write, finalises the operation, and returns the Close error.

func (*Writer) ReadFrom added in v2.38.0

func (w *Writer) ReadFrom(r io.Reader) (int64, error)

func (*Writer) Write added in v2.38.0

func (w *Writer) Write(p []byte) (int, error)

Directories

Path Synopsis
providers
azure
Package azure registers the Azure Blob Storage backend for objectstore.
Package azure registers the Azure Blob Storage backend for objectstore.
gcs
Package gcs registers the Google Cloud Storage backend for objectstore.
Package gcs registers the Google Cloud Storage backend for objectstore.
mem
Package mem registers an in-memory object store backend for objectstore.
Package mem registers an in-memory object store backend for objectstore.
s3
Package s3 registers the S3 (and S3-compatible) object store backend for objectstore.
Package s3 registers the S3 (and S3-compatible) object store backend for objectstore.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL