storage

package
v0.8.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 8, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Package storage defines the backend-neutral data access contract. No database driver types cross this boundary.

Index

Constants

View Source
const (
	CheckOK   = "ok"
	CheckWarn = "warn" // worth knowing, doesn't fail the check
	CheckFail = "fail"
	CheckSkip = "skipped"
)

Check levels.

View Source
const ErasedBy = "backd:erase"

ErasedBy is what an erase records in _meta.updated_by.

View Source
const UploadPartSize = 8 << 20

UploadPartSize is the size of the parts a streamed upload is cut into; an object smaller than one is a single PUT.

Variables

View Source
var (
	ErrObjectNotFound   = errors.New("object not found")
	ErrAccessDenied     = errors.New("the storage refused the credentials or the operation")
	ErrBucketNotFound   = errors.New("the bucket doesn't exist")
	ErrChecksumMismatch = errors.New("the storage rejected the content: its checksum doesn't match")
)

Errors the object operations map the storage's answers to.

View Source
var (
	// ErrNotFound means no document has the requested id.
	ErrNotFound = errors.New("document not found")
	// ErrVersionMismatch means a conditional write found a different version.
	ErrVersionMismatch = errors.New("document version mismatch")
	// ErrUnavailable means the storage backend could not be reached in time.
	ErrUnavailable = errors.New("storage unavailable")
	// ErrConflict means a write violated a unique index. The concrete error
	// is a *ConflictError.
	ErrConflict = errors.New("unique constraint violated")
)
View Source
var ErrNoSuchUpload = errors.New("storage: no such multipart upload")

ErrNoSuchUpload means the multipart upload does not exist (any more): it was completed, aborted, or never made.

Functions

func CompositeSHA256 added in v0.8.1

func CompositeSHA256(partSHA256 []string) (reported, hexDigest string, err error)

CompositeSHA256 is what a storage reports as the checksum of an object made of parts with the given SHA-256 digests (hex): the SHA-256 of the digests joined, in base64, then "-" and the number of parts. The second result is the same digest in hex, which backd records as the file's sha256.

func IsCompositeOf added in v0.8.1

func IsCompositeOf(reported string, partSHA256 []string) bool

IsCompositeOf reports whether an object's checksum as the storage returns it ("<base64>-<parts>") is the composite of the given part digests.

func Match added in v0.5.0

func Match(doc Document, f Filter) (bool, error)

Match evaluates a filter against a document held in memory, with the semantics MongoDB gives the same filter, for the operators access rules produce (eq, ne, in, nin, the comparisons and isNull), and between:

  • a missing field is null: it equals nil, is not equal to anything else, and an ordering comparison with it matches nothing;
  • on an array, equality matches any element (and so do comparisons);
  • numbers compare by value whether they are int64 or float64, and a comparison only matches values of its own kind (numbers, strings, dates), never across kinds.

It lets rules be tested without a database. Between (inclusive) is also understood; the text operators belong to the query language and report an error.

func NewHTTPClient added in v0.8.0

func NewHTTPClient(endpoint, bucket string, addressing Addressing) (*http.Client, error)

NewHTTPClient returns the HTTP client of a realm's storage connection.

func ObjectKey added in v0.8.0

func ObjectKey(prefix, realm, database, collection, fileID string) string

ObjectKey is the key of a file's object under a prefix: it needs no connection, so what only has to name an object (a deletion to queue) works while the storage is down.

func ProviderNames added in v0.8.0

func ProviderNames() []string

ProviderNames lists the supported providers, sorted.

Types

type Addressing added in v0.8.0

type Addressing string

Addressing is how a bucket is reached.

const (
	// VirtualHosted puts the bucket in the host name: <bucket>.<endpoint host>.
	VirtualHosted Addressing = "virtual-hosted"
	// PathStyle puts it in the path: <endpoint>/<bucket>.
	PathStyle Addressing = "path"
)

type And

type And []Filter

And matches documents matching every filter; And{} matches everything.

type CORSRule added in v0.8.0

type CORSRule struct {
	Origins, Methods, Headers []string
}

CORSRule is one rule of the bucket's CORS configuration.

type Capability added in v0.8.0

type Capability string

Capability says whether a provider supports something, as the smoke test found.

const (
	Supported   Capability = "yes"
	Unsupported Capability = "no"
	// Unverified is documented or expected but not confirmed by the smoke test yet:
	// it is treated as unsupported until it is.
	Unverified Capability = "unverified"
)

type CheckReport added in v0.8.0

type CheckReport struct {
	Provider       string
	Endpoint       string
	PublicEndpoint string
	Bucket         string
	Prefix         string
	Steps          []CheckStep
	// ChecksumSHA256 is what the storage does with a signed x-amz-checksum-sha256:
	// "verified" (it rejects content that differs), "ignored" (it accepts it), or
	// "not tested".
	ChecksumSHA256 string
	// Encryption is the bucket's default encryption ("none", an algorithm, or
	// "unknown" when the provider doesn't expose it).
	Encryption string
	CORS       []CORSRule
	// LinkHost is the host a signed link is for (the public endpoint's when set).
	LinkHost string
}

CheckReport is what a check of a realm's storage found.

func (CheckReport) OK added in v0.8.0

func (r CheckReport) OK() bool

OK reports whether nothing failed.

type CheckStep added in v0.8.0

type CheckStep struct {
	Name   string
	Level  string
	Detail string
}

CheckStep is one thing `backd storage check` tried.

type Condition

type Condition struct {
	Field string
	Op    Operator
	Value any
}

Condition is one filter clause. Field is a dot path in API terms ("id", "_meta.created_at", "address.city"). Values are already type-checked: numbers are int64 or float64, _meta dates are time.Time.

type ConflictError

type ConflictError struct {
	Fields []string // fields of the violated index, in API terms; may be empty
}

ConflictError reports a unique index violation.

func (*ConflictError) Error

func (e *ConflictError) Error() string

func (*ConflictError) Is

func (e *ConflictError) Is(target error) bool

Is makes errors.Is(err, ErrConflict) match.

type Const

type Const bool

Const matches every document (true) or none (false).

type Document

type Document = map[string]any

Document is a stored document in API shape: "id" (string), "_meta" (map with time.Time values) and the user fields. Numbers are int64 or float64, nested objects are Documents or map[string]any, arrays []any.

func Fetch

func Fetch(ctx context.Context, repo Repository, id string, filter Filter) (Document, error)

Fetch returns the document with id if the access filter (nil: none) lets the caller read it, else ErrNotFound. Single reads use it, so a read rule applies to them exactly as it does to lists.

type ErasedFile added in v0.8.0

type ErasedFile struct {
	ID   string
	Size int64
	// Versions are the made copies of an image the file had, by version name, with their sizes.
	Versions map[string]int64
}

ErasedFile is a file an erase took out of a document.

type Eraser added in v0.4.0

type Eraser interface {
	// CountOwned counts the documents whose _meta.owner is the user.
	CountOwned(ctx context.Context, owner string) (int64, error)
	// CountReferences counts the documents where field (an array of strings
	// when array is true, else a string) holds value.
	CountReferences(ctx context.Context, field, value string, array bool) (int64, error)
	// DeleteOwned deletes up to limit documents the user owns. It returns the files
	// the documents held in fileFields (the collection's file fields), exactly the
	// ones of the documents it deleted, so their objects can be queued for deletion.
	DeleteOwned(ctx context.Context, owner string, fileFields []string, limit int) (int64, []ErasedFile, error)
	// AnonymizeOwned removes the fields in remove, sets those in replace and
	// clears _meta.owner on up to limit documents the user owns. It returns the
	// files the documents held in fileFields (the file fields among remove) at the
	// moment they were removed.
	AnonymizeOwned(ctx context.Context, owner string, remove []string, replace map[string]any, fileFields []string, limit int, now time.Time) (int64, []ErasedFile, error)
	// PullReference removes value from the array field of up to limit
	// documents that contain it.
	PullReference(ctx context.Context, field, value string, limit int, now time.Time) (int64, error)
	// ClearReference unsets the string field of up to limit documents where
	// it equals value.
	ClearReference(ctx context.Context, field, value string, limit int, now time.Time) (int64, error)
}

Eraser is what erasing a user needs of a collection's storage. Every method works in batches and acts only on documents that still match, so a batch that was interrupted is simply repeated: call each until it returns 0. Updates are atomic per document (never read-then-write) and are system writes: they bump _meta.version and _meta.updated_at, and record _meta.updated_by as ErasedBy.

type Filter

type Filter interface {
	// contains filtered or unexported methods
}

Filter is a boolean combination of conditions: a Condition, And, Or, Not or Const.

func Simplify

func Simplify(f Filter) Filter

Simplify folds constants, so the result is a Const or contains none.

type Link struct {
	URL       string
	ExpiresAt time.Time
	// Headers must be sent with the request (a signed PUT's content type, length
	// and checksum).
	Headers map[string]string
}

Link is a signed URL.

type Not

type Not struct{ Filter Filter }

Not matches documents not matching Filter.

type ObjectInfo added in v0.8.0

type ObjectInfo struct {
	Key         string
	Size        int64
	ContentType string
	ETag        string
	SHA256      string // hex; "" when the storage keeps none (or keeps a composite one)
	// Checksum is the SHA-256 as the storage reports it: base64, and for an object made by a
	// multipart upload "<base64>-<parts>" (a composite checksum, see CompositeSHA256).
	Checksum     string
	LastModified time.Time
}

ObjectInfo describes a stored object.

type Objects added in v0.8.0

type Objects struct {
	// contains filtered or unexported fields
}

Objects stores and reads a realm's files in its bucket: only operations every supported provider has, all signed with AWS Signature Version 4.

func NewObjects added in v0.8.0

func NewObjects(cfg ObjectsConfig) (*Objects, error)

NewObjects connects to the storage a realm declares. It makes no request.

func (*Objects) AbortMultipart added in v0.8.1

func (o *Objects) AbortMultipart(ctx context.Context, key, uploadID string) error

AbortMultipart drops the multipart upload and the parts it holds. One that is gone is not an error.

func (*Objects) CORS added in v0.8.0

func (o *Objects) CORS(ctx context.Context) ([]CORSRule, error)

CORS reads the bucket's CORS rules (none: an empty list, nil error).

func (*Objects) Check added in v0.8.0

func (o *Objects) Check(ctx context.Context, realm string) CheckReport

Check verifies the realm's storage the way a file would use it: the bucket and the credentials, every operation backd needs on the realm's prefix, the signed checksum and signed links, and (where the provider exposes them) the bucket's CORS rules and default encryption. It uploads a few small objects under <prefix>/<realm>/_check/ and deletes them again, and never touches anything else.

func (*Objects) CompleteMultipart added in v0.8.1

func (o *Objects) CompleteMultipart(ctx context.Context, key, uploadID string, parts []Part) error

CompleteMultipart assembles the parts into the object. The storage checks that each part's SHA-256 is the one it recorded.

func (*Objects) Config added in v0.8.0

func (o *Objects) Config() ObjectsConfig

Config returns the connection's settings.

func (*Objects) CreateMultipart added in v0.8.1

func (o *Objects) CreateMultipart(ctx context.Context, key, contentType string) (string, error)

CreateMultipart starts a multipart upload of key whose parts carry a SHA-256, and returns its id.

func (*Objects) Delete added in v0.8.0

func (o *Objects) Delete(ctx context.Context, key string) error

Delete removes an object; a missing one is not an error (S3 answers success).

func (*Objects) Encryption added in v0.8.0

func (o *Objects) Encryption(ctx context.Context) (string, error)

Encryption reads the bucket's default encryption algorithm ("" when none).

func (*Objects) Get added in v0.8.0

func (o *Objects) Get(ctx context.Context, key, rng string) (io.ReadCloser, ObjectInfo, error)

Get opens an object; rng is an HTTP Range value ("bytes=0-99") or "". The caller closes the body.

func (*Objects) Head added in v0.8.0

func (o *Objects) Head(ctx context.Context, key string) (ObjectInfo, error)

Head returns an object's details, including its stored SHA-256 when it has one.

func (*Objects) HeadBucket added in v0.8.0

func (o *Objects) HeadBucket(ctx context.Context) error

HeadBucket reports whether the bucket exists and the credentials can reach it.

func (*Objects) Key added in v0.8.0

func (o *Objects) Key(realm, database, collection, fileID string) string

Key is the object key of a file: <prefix>/<realm>/<database>/<collection>/<file id>.

func (*Objects) List added in v0.8.0

func (o *Objects) List(ctx context.Context, prefix string, max int) ([]ObjectInfo, error)

List returns up to max objects whose key starts with prefix. It is for `backd storage check` and the manual reconcile only: backd never lists the bucket in normal operation.

func (*Objects) ListAll added in v0.8.0

func (o *Objects) ListAll(ctx context.Context, prefix string, fn func(ObjectInfo) bool) error

ListAll calls fn for every object under prefix, page by page, until fn returns false. It is for the manual reconcile: nothing else lists the bucket.

func (*Objects) ListParts added in v0.8.1

func (o *Objects) ListParts(ctx context.Context, key, uploadID string) ([]Part, error)

ListParts returns the parts uploaded so far, in order. ErrNoSuchUpload when the upload is gone.

func (*Objects) PresignGet added in v0.8.0

func (o *Objects) PresignGet(ctx context.Context, key string, ttl time.Duration, filename, contentType string) (Link, error)

PresignGet signs a download link valid for ttl that forces the browser to save the file as filename (an attachment) with the stored type. It makes no request.

func (*Objects) PresignPut added in v0.8.0

func (o *Objects) PresignPut(ctx context.Context, key string, ttl time.Duration, size int64, contentType, sha256 string) (Link, error)

PresignPut signs an upload link valid for ttl for exactly size bytes of contentType with the given SHA-256 (hex): the storage rejects a PUT that differs from what was signed. It makes no request.

func (*Objects) PresignUploadPart added in v0.8.1

func (o *Objects) PresignUploadPart(ctx context.Context, key, uploadID string, number int32, ttl time.Duration, size int64, sha256hex string) (Link, error)

PresignUploadPart signs the upload of part number of the multipart upload, valid for ttl, for exactly size bytes with the given SHA-256 (hex): the storage rejects a part that differs. It makes no request.

func (*Objects) Provider added in v0.8.0

func (o *Objects) Provider() Provider

Provider returns the connection's provider entry.

func (*Objects) Put added in v0.8.0

func (o *Objects) Put(ctx context.Context, key string, body io.Reader, size int64, contentType, sha256 string) error

Put stores body (size bytes) under key. With sha256 (hex) the storage verifies the content against it and rejects the object when it differs.

func (*Objects) PutStream added in v0.8.0

func (o *Objects) PutStream(ctx context.Context, key string, body io.Reader, contentType string) error

PutStream stores a stream of unknown length under key, in parts when it is large (a multipart upload, aborted if the stream fails), holding at most one part in memory. It does not verify a checksum: the caller hashes the stream as it passes.

type ObjectsConfig added in v0.8.0

type ObjectsConfig struct {
	Provider       string
	Endpoint       string
	PublicEndpoint string // "" unless the provider allows it; only signed into links
	Region         string
	Bucket         string
	Prefix         string
	AccessKey      string
	SecretKey      string
}

ObjectsConfig is a realm's connection to its object storage, as registry.StorageSettings resolves it, with the keys read from the realm's secrets.

type Operator

type Operator string

Operator is a filter operator. Operators are backend-neutral; each storage implementation translates them.

const (
	OpEq          Operator = "$eq"
	OpNe          Operator = "$ne"
	OpGt          Operator = "$gt"
	OpGte         Operator = "$gte"
	OpLt          Operator = "$lt"
	OpLte         Operator = "$lte"
	OpIn          Operator = "$in"      // Value: []any
	OpNin         Operator = "$nin"     // Value: []any
	OpBetween     Operator = "$between" // Value: []any{min, max}, inclusive
	OpIsNull      Operator = "$isNull"  // Value: bool; true matches null or missing
	OpLike        Operator = "$like"    // Value: string; SQL-style % and _ wildcards
	OpILike       Operator = "$ilike"
	OpStartsWith  Operator = "$startsWith"
	OpIStartsWith Operator = "$istartsWith"
	OpEndsWith    Operator = "$endsWith"
	OpIEndsWith   Operator = "$iendsWith"
	OpContains    Operator = "$contains"
	OpIContains   Operator = "$icontains"
)

type Or

type Or []Filter

Or matches documents matching any filter; Or{} matches nothing.

type Page

type Page struct {
	Items   []Document
	HasMore bool
	Total   *int64 // set only when Query.Count is true
}

Page is one page of a list query.

type Part added in v0.8.1

type Part struct {
	Number int32
	ETag   string
	Size   int64
	SHA256 string // hex
}

Part is one uploaded part of a multipart upload.

type Provider added in v0.8.0

type Provider struct {
	Name  string // the value of storage.provider
	Label string // for people

	Addressing Addressing

	// EndpointRequired says whether storage.endpoint must be given; without it
	// DefaultEndpoint(region) is used.
	EndpointRequired bool
	DefaultEndpoint  func(region string) string
	// EndpointPattern is what an endpoint (scheme and host, with no path) must
	// match; Group 1, when the entry has RegionFromEndpoint, is the region.
	EndpointPattern *regexp.Regexp
	EndpointHelp    string // the shape, for error messages and the template
	// AllowHTTP lets the endpoint be plain http (development only: startup
	// refuses it unless BACKD_DEV=true).
	AllowHTTP bool

	// Region rules: FixedRegion is the only valid value ("" = free); with
	// RegionFromEndpoint the region is read from the endpoint and must agree with
	// one given; DefaultRegion is used when none is given.
	FixedRegion        string
	RegionFromEndpoint bool
	DefaultRegion      string

	// ChecksumSHA256 is whether the x-amz-checksum-sha256 header is accepted and
	// verified on a PUT, which direct uploads need.
	ChecksumSHA256 Capability
	// MaxSinglePut is the largest object one PUT can carry.
	MaxSinglePut int64
	// MultipartSHA256 is whether a multipart upload can carry a SHA-256 per part, which the
	// storage verifies and combines into the object's composite checksum: what direct uploads
	// above MaxSinglePut need. Unverified (not confirmed by a smoke test) counts as unsupported.
	MultipartSHA256 Capability
	// ReadsCORS and ReadsEncryption say whether `backd storage check` can read
	// the bucket's CORS rules and default encryption.
	ReadsCORS, ReadsEncryption bool
	// ChecksumsWhenRequired makes the client send request checksums only where an
	// operation needs them: recent AWS SDKs send CRC checksums on every upload,
	// which some S3-compatible services reject.
	ChecksumsWhenRequired bool
	// PublicEndpointAllowed lets a realm set public_endpoint: only for providers
	// whose address differs between backd and browsers (a self-hosted MinIO).
	PublicEndpointAllowed bool
	// OldestTested is the oldest version of a self-hosted provider the tests ran
	// against, stated in the docs ("" for hosted providers).
	OldestTested string
}

Provider is what backd knows about one S3-compatible storage service.

func LookupProvider added in v0.8.0

func LookupProvider(name string) (Provider, bool)

LookupProvider returns the table's entry for name.

func (Provider) Resolve added in v0.8.0

func (p Provider) Resolve(endpoint, region string) (Resolved, error)

Resolve applies the provider's endpoint and region rules to what a realm wrote and returns what to connect to, or says what is wrong.

func (Provider) SupportsDirectUploads added in v0.8.0

func (p Provider) SupportsDirectUploads() bool

SupportsDirectUploads reports whether direct uploads work with this provider: they sign a SHA-256 into the PUT, so the provider must have been confirmed to verify it.

func (Provider) SupportsMultipartUploads added in v0.8.1

func (p Provider) SupportsMultipartUploads() bool

SupportsMultipartUploads reports whether direct uploads above MaxSinglePut work: the provider must verify a SHA-256 per part and report the composite checksum.

type Query

type Query struct {
	// Filter is the client's where: a tree of conditions combined with
	// And and Or. nil matches everything.
	Filter Filter
	Sort   []SortField // applied in order; ends with "id" as a tiebreaker
	Limit  int
	Skip   int
	Count  bool // also compute the total number of matching documents
	// Access restricts the query to documents the caller may read
	// (from access rules); nil means no restriction.
	Access Filter
	// After restricts the page to the documents positioned after a cursor in
	// Sort (query.ParseCursor builds it). It narrows the page, not the Total.
	After Filter
}

Query selects a page of documents.

type Repository

type Repository interface {
	// Create stores doc, which must already carry "id" and "_meta".
	Create(ctx context.Context, doc Document) error
	// Get returns the document with the given id, or ErrNotFound.
	Get(ctx context.Context, id string) (Document, error)
	// List returns a page of documents.
	List(ctx context.Context, q Query) (Page, error)
	// Replace overwrites the document with doc["id"] only if its stored
	// _meta.version equals ifVersion (0 means "stored without a version").
	// It returns ErrNotFound or ErrVersionMismatch otherwise.
	Replace(ctx context.Context, doc Document, ifVersion int64) error
	// Delete removes the document with the given id. When ifVersion is not
	// nil the stored version must equal it. It returns ErrNotFound or
	// ErrVersionMismatch otherwise.
	Delete(ctx context.Context, id string, ifVersion *int64) error
}

Repository gives access to the documents of one collection.

type Resolved added in v0.8.0

type Resolved struct {
	Endpoint string // no trailing slash
	Region   string
	HTTP     bool // the endpoint is plain http
}

Resolved is the endpoint and region a configuration comes to once the provider's rules are applied.

type Scanner added in v0.8.0

type Scanner interface {
	// ScanAfter returns up to limit documents whose id sorts after afterID
	// ("" starts from the first), ordered by id.
	ScanAfter(ctx context.Context, afterID string, limit int) ([]Document, error)
	// EstimatedCount is the collection's approximate size, cheap to ask.
	EstimatedCount(ctx context.Context) (int64, error)
}

Scanner reads every stored document of a collection, in id order, for the schema check: soft-deleted ones included, with no access rules and no writes. A repository that doesn't implement it can't be checked.

type SortField

type SortField struct {
	Field string
	Desc  bool
}

SortField orders results by one field.

Directories

Path Synopsis
Package storagetest is a small in-memory S3 server for tests that need an object storage but not a real one: path-style addressing, the operations backd uses (bucket HEAD, PUT, GET with Range, HEAD, DELETE, ListObjectsV2) and the signed SHA-256 check a PUT carries.
Package storagetest is a small in-memory S3 server for tests that need an object storage but not a real one: path-style addressing, the operations backd uses (bucket HEAD, PUT, GET with Range, HEAD, DELETE, ListObjectsV2) and the signed SHA-256 check a PUT carries.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL