Documentation
¶
Overview ¶
Package storage defines the backend-neutral data access contract. No database driver types cross this boundary.
Index ¶
- Constants
- Variables
- func CompositeSHA256(partSHA256 []string) (reported, hexDigest string, err error)
- func IsCompositeOf(reported string, partSHA256 []string) bool
- func Match(doc Document, f Filter) (bool, error)
- func NewHTTPClient(endpoint, bucket string, addressing Addressing) (*http.Client, error)
- func ObjectKey(prefix, realm, database, collection, fileID string) string
- func ProviderNames() []string
- type Addressing
- type And
- type CORSRule
- type Capability
- type CheckReport
- type CheckStep
- type Condition
- type ConflictError
- type Const
- type Document
- type ErasedFile
- type Eraser
- type Filter
- type Link
- type Not
- type ObjectInfo
- type Objects
- func (o *Objects) AbortMultipart(ctx context.Context, key, uploadID string) error
- func (o *Objects) CORS(ctx context.Context) ([]CORSRule, error)
- func (o *Objects) Check(ctx context.Context, realm string) CheckReport
- func (o *Objects) CompleteMultipart(ctx context.Context, key, uploadID string, parts []Part) error
- func (o *Objects) Config() ObjectsConfig
- func (o *Objects) CreateMultipart(ctx context.Context, key, contentType string) (string, error)
- func (o *Objects) Delete(ctx context.Context, key string) error
- func (o *Objects) Encryption(ctx context.Context) (string, error)
- func (o *Objects) Get(ctx context.Context, key, rng string) (io.ReadCloser, ObjectInfo, error)
- func (o *Objects) Head(ctx context.Context, key string) (ObjectInfo, error)
- func (o *Objects) HeadBucket(ctx context.Context) error
- func (o *Objects) Key(realm, database, collection, fileID string) string
- func (o *Objects) List(ctx context.Context, prefix string, max int) ([]ObjectInfo, error)
- func (o *Objects) ListAll(ctx context.Context, prefix string, fn func(ObjectInfo) bool) error
- func (o *Objects) ListParts(ctx context.Context, key, uploadID string) ([]Part, error)
- func (o *Objects) PresignGet(ctx context.Context, key string, ttl time.Duration, ...) (Link, error)
- func (o *Objects) PresignPut(ctx context.Context, key string, ttl time.Duration, size int64, ...) (Link, error)
- func (o *Objects) PresignUploadPart(ctx context.Context, key, uploadID string, number int32, ttl time.Duration, ...) (Link, error)
- func (o *Objects) Provider() Provider
- func (o *Objects) Put(ctx context.Context, key string, body io.Reader, size int64, ...) error
- func (o *Objects) PutStream(ctx context.Context, key string, body io.Reader, contentType string) error
- type ObjectsConfig
- type Operator
- type Or
- type Page
- type Part
- type Provider
- type Query
- type Repository
- type Resolved
- type Scanner
- type SortField
Constants ¶
const ( CheckOK = "ok" CheckWarn = "warn" // worth knowing, doesn't fail the check CheckFail = "fail" CheckSkip = "skipped" )
Check levels.
const ErasedBy = "backd:erase"
ErasedBy is what an erase records in _meta.updated_by.
const UploadPartSize = 8 << 20
UploadPartSize is the size of the parts a streamed upload is cut into; an object smaller than one is a single PUT.
Variables ¶
var ( ErrObjectNotFound = errors.New("object not found") ErrAccessDenied = errors.New("the storage refused the credentials or the operation") ErrBucketNotFound = errors.New("the bucket doesn't exist") ErrChecksumMismatch = errors.New("the storage rejected the content: its checksum doesn't match") )
Errors the object operations map the storage's answers to.
var ( // ErrNotFound means no document has the requested id. ErrNotFound = errors.New("document not found") // ErrVersionMismatch means a conditional write found a different version. ErrVersionMismatch = errors.New("document version mismatch") ErrUnavailable = errors.New("storage unavailable") // ErrConflict means a write violated a unique index. The concrete error // is a *ConflictError. ErrConflict = errors.New("unique constraint violated") )
var ErrNoSuchUpload = errors.New("storage: no such multipart upload")
ErrNoSuchUpload means the multipart upload does not exist (any more): it was completed, aborted, or never made.
Functions ¶
func CompositeSHA256 ¶ added in v0.8.1
CompositeSHA256 is what a storage reports as the checksum of an object made of parts with the given SHA-256 digests (hex): the SHA-256 of the digests joined, in base64, then "-" and the number of parts. The second result is the same digest in hex, which backd records as the file's sha256.
func IsCompositeOf ¶ added in v0.8.1
IsCompositeOf reports whether an object's checksum as the storage returns it ("<base64>-<parts>") is the composite of the given part digests.
func Match ¶ added in v0.5.0
Match evaluates a filter against a document held in memory, with the semantics MongoDB gives the same filter, for the operators access rules produce (eq, ne, in, nin, the comparisons and isNull), and between:
- a missing field is null: it equals nil, is not equal to anything else, and an ordering comparison with it matches nothing;
- on an array, equality matches any element (and so do comparisons);
- numbers compare by value whether they are int64 or float64, and a comparison only matches values of its own kind (numbers, strings, dates), never across kinds.
It lets rules be tested without a database. Between (inclusive) is also understood; the text operators belong to the query language and report an error.
func NewHTTPClient ¶ added in v0.8.0
func NewHTTPClient(endpoint, bucket string, addressing Addressing) (*http.Client, error)
NewHTTPClient returns the HTTP client of a realm's storage connection.
func ObjectKey ¶ added in v0.8.0
ObjectKey is the key of a file's object under a prefix: it needs no connection, so what only has to name an object (a deletion to queue) works while the storage is down.
func ProviderNames ¶ added in v0.8.0
func ProviderNames() []string
ProviderNames lists the supported providers, sorted.
Types ¶
type Addressing ¶ added in v0.8.0
type Addressing string
Addressing is how a bucket is reached.
const ( // VirtualHosted puts the bucket in the host name: <bucket>.<endpoint host>. VirtualHosted Addressing = "virtual-hosted" // PathStyle puts it in the path: <endpoint>/<bucket>. PathStyle Addressing = "path" )
type CORSRule ¶ added in v0.8.0
type CORSRule struct {
Origins, Methods, Headers []string
}
CORSRule is one rule of the bucket's CORS configuration.
type Capability ¶ added in v0.8.0
type Capability string
Capability says whether a provider supports something, as the smoke test found.
const ( Supported Capability = "yes" Unsupported Capability = "no" // Unverified is documented or expected but not confirmed by the smoke test yet: // it is treated as unsupported until it is. Unverified Capability = "unverified" )
type CheckReport ¶ added in v0.8.0
type CheckReport struct {
Provider string
Endpoint string
PublicEndpoint string
Bucket string
Prefix string
Steps []CheckStep
// ChecksumSHA256 is what the storage does with a signed x-amz-checksum-sha256:
// "verified" (it rejects content that differs), "ignored" (it accepts it), or
// "not tested".
ChecksumSHA256 string
// Encryption is the bucket's default encryption ("none", an algorithm, or
// "unknown" when the provider doesn't expose it).
Encryption string
CORS []CORSRule
// LinkHost is the host a signed link is for (the public endpoint's when set).
LinkHost string
}
CheckReport is what a check of a realm's storage found.
func (CheckReport) OK ¶ added in v0.8.0
func (r CheckReport) OK() bool
OK reports whether nothing failed.
type Condition ¶
Condition is one filter clause. Field is a dot path in API terms ("id", "_meta.created_at", "address.city"). Values are already type-checked: numbers are int64 or float64, _meta dates are time.Time.
type ConflictError ¶
type ConflictError struct {
Fields []string // fields of the violated index, in API terms; may be empty
}
ConflictError reports a unique index violation.
func (*ConflictError) Error ¶
func (e *ConflictError) Error() string
func (*ConflictError) Is ¶
func (e *ConflictError) Is(target error) bool
Is makes errors.Is(err, ErrConflict) match.
type Document ¶
Document is a stored document in API shape: "id" (string), "_meta" (map with time.Time values) and the user fields. Numbers are int64 or float64, nested objects are Documents or map[string]any, arrays []any.
type ErasedFile ¶ added in v0.8.0
type ErasedFile struct {
ID string
Size int64
// Versions are the made copies of an image the file had, by version name, with their sizes.
Versions map[string]int64
}
ErasedFile is a file an erase took out of a document.
type Eraser ¶ added in v0.4.0
type Eraser interface {
// CountOwned counts the documents whose _meta.owner is the user.
CountOwned(ctx context.Context, owner string) (int64, error)
// CountReferences counts the documents where field (an array of strings
// when array is true, else a string) holds value.
CountReferences(ctx context.Context, field, value string, array bool) (int64, error)
// DeleteOwned deletes up to limit documents the user owns. It returns the files
// the documents held in fileFields (the collection's file fields), exactly the
// ones of the documents it deleted, so their objects can be queued for deletion.
DeleteOwned(ctx context.Context, owner string, fileFields []string, limit int) (int64, []ErasedFile, error)
// AnonymizeOwned removes the fields in remove, sets those in replace and
// clears _meta.owner on up to limit documents the user owns. It returns the
// files the documents held in fileFields (the file fields among remove) at the
// moment they were removed.
AnonymizeOwned(ctx context.Context, owner string, remove []string, replace map[string]any, fileFields []string, limit int, now time.Time) (int64, []ErasedFile, error)
// PullReference removes value from the array field of up to limit
// documents that contain it.
PullReference(ctx context.Context, field, value string, limit int, now time.Time) (int64, error)
// ClearReference unsets the string field of up to limit documents where
// it equals value.
ClearReference(ctx context.Context, field, value string, limit int, now time.Time) (int64, error)
}
Eraser is what erasing a user needs of a collection's storage. Every method works in batches and acts only on documents that still match, so a batch that was interrupted is simply repeated: call each until it returns 0. Updates are atomic per document (never read-then-write) and are system writes: they bump _meta.version and _meta.updated_at, and record _meta.updated_by as ErasedBy.
type Filter ¶
type Filter interface {
// contains filtered or unexported methods
}
Filter is a boolean combination of conditions: a Condition, And, Or, Not or Const.
type Link ¶ added in v0.8.0
type Link struct {
URL string
ExpiresAt time.Time
// Headers must be sent with the request (a signed PUT's content type, length
// and checksum).
Headers map[string]string
}
Link is a signed URL.
type ObjectInfo ¶ added in v0.8.0
type ObjectInfo struct {
Key string
Size int64
ContentType string
ETag string
SHA256 string // hex; "" when the storage keeps none (or keeps a composite one)
// Checksum is the SHA-256 as the storage reports it: base64, and for an object made by a
// multipart upload "<base64>-<parts>" (a composite checksum, see CompositeSHA256).
Checksum string
LastModified time.Time
}
ObjectInfo describes a stored object.
type Objects ¶ added in v0.8.0
type Objects struct {
// contains filtered or unexported fields
}
Objects stores and reads a realm's files in its bucket: only operations every supported provider has, all signed with AWS Signature Version 4.
func NewObjects ¶ added in v0.8.0
func NewObjects(cfg ObjectsConfig) (*Objects, error)
NewObjects connects to the storage a realm declares. It makes no request.
func (*Objects) AbortMultipart ¶ added in v0.8.1
AbortMultipart drops the multipart upload and the parts it holds. One that is gone is not an error.
func (*Objects) CORS ¶ added in v0.8.0
CORS reads the bucket's CORS rules (none: an empty list, nil error).
func (*Objects) Check ¶ added in v0.8.0
func (o *Objects) Check(ctx context.Context, realm string) CheckReport
Check verifies the realm's storage the way a file would use it: the bucket and the credentials, every operation backd needs on the realm's prefix, the signed checksum and signed links, and (where the provider exposes them) the bucket's CORS rules and default encryption. It uploads a few small objects under <prefix>/<realm>/_check/ and deletes them again, and never touches anything else.
func (*Objects) CompleteMultipart ¶ added in v0.8.1
CompleteMultipart assembles the parts into the object. The storage checks that each part's SHA-256 is the one it recorded.
func (*Objects) Config ¶ added in v0.8.0
func (o *Objects) Config() ObjectsConfig
Config returns the connection's settings.
func (*Objects) CreateMultipart ¶ added in v0.8.1
CreateMultipart starts a multipart upload of key whose parts carry a SHA-256, and returns its id.
func (*Objects) Delete ¶ added in v0.8.0
Delete removes an object; a missing one is not an error (S3 answers success).
func (*Objects) Encryption ¶ added in v0.8.0
Encryption reads the bucket's default encryption algorithm ("" when none).
func (*Objects) Get ¶ added in v0.8.0
func (o *Objects) Get(ctx context.Context, key, rng string) (io.ReadCloser, ObjectInfo, error)
Get opens an object; rng is an HTTP Range value ("bytes=0-99") or "". The caller closes the body.
func (*Objects) Head ¶ added in v0.8.0
Head returns an object's details, including its stored SHA-256 when it has one.
func (*Objects) HeadBucket ¶ added in v0.8.0
HeadBucket reports whether the bucket exists and the credentials can reach it.
func (*Objects) Key ¶ added in v0.8.0
Key is the object key of a file: <prefix>/<realm>/<database>/<collection>/<file id>.
func (*Objects) List ¶ added in v0.8.0
List returns up to max objects whose key starts with prefix. It is for `backd storage check` and the manual reconcile only: backd never lists the bucket in normal operation.
func (*Objects) ListAll ¶ added in v0.8.0
ListAll calls fn for every object under prefix, page by page, until fn returns false. It is for the manual reconcile: nothing else lists the bucket.
func (*Objects) ListParts ¶ added in v0.8.1
ListParts returns the parts uploaded so far, in order. ErrNoSuchUpload when the upload is gone.
func (*Objects) PresignGet ¶ added in v0.8.0
func (o *Objects) PresignGet(ctx context.Context, key string, ttl time.Duration, filename, contentType string) (Link, error)
PresignGet signs a download link valid for ttl that forces the browser to save the file as filename (an attachment) with the stored type. It makes no request.
func (*Objects) PresignPut ¶ added in v0.8.0
func (o *Objects) PresignPut(ctx context.Context, key string, ttl time.Duration, size int64, contentType, sha256 string) (Link, error)
PresignPut signs an upload link valid for ttl for exactly size bytes of contentType with the given SHA-256 (hex): the storage rejects a PUT that differs from what was signed. It makes no request.
func (*Objects) PresignUploadPart ¶ added in v0.8.1
func (o *Objects) PresignUploadPart(ctx context.Context, key, uploadID string, number int32, ttl time.Duration, size int64, sha256hex string) (Link, error)
PresignUploadPart signs the upload of part number of the multipart upload, valid for ttl, for exactly size bytes with the given SHA-256 (hex): the storage rejects a part that differs. It makes no request.
func (*Objects) Put ¶ added in v0.8.0
func (o *Objects) Put(ctx context.Context, key string, body io.Reader, size int64, contentType, sha256 string) error
Put stores body (size bytes) under key. With sha256 (hex) the storage verifies the content against it and rejects the object when it differs.
func (*Objects) PutStream ¶ added in v0.8.0
func (o *Objects) PutStream(ctx context.Context, key string, body io.Reader, contentType string) error
PutStream stores a stream of unknown length under key, in parts when it is large (a multipart upload, aborted if the stream fails), holding at most one part in memory. It does not verify a checksum: the caller hashes the stream as it passes.
type ObjectsConfig ¶ added in v0.8.0
type ObjectsConfig struct {
Provider string
Endpoint string
PublicEndpoint string // "" unless the provider allows it; only signed into links
Region string
Bucket string
Prefix string
AccessKey string
SecretKey string
}
ObjectsConfig is a realm's connection to its object storage, as registry.StorageSettings resolves it, with the keys read from the realm's secrets.
type Operator ¶
type Operator string
Operator is a filter operator. Operators are backend-neutral; each storage implementation translates them.
const ( OpEq Operator = "$eq" OpNe Operator = "$ne" OpGt Operator = "$gt" OpGte Operator = "$gte" OpLt Operator = "$lt" OpLte Operator = "$lte" OpIn Operator = "$in" // Value: []any OpNin Operator = "$nin" // Value: []any OpBetween Operator = "$between" // Value: []any{min, max}, inclusive OpIsNull Operator = "$isNull" // Value: bool; true matches null or missing OpLike Operator = "$like" // Value: string; SQL-style % and _ wildcards OpILike Operator = "$ilike" OpStartsWith Operator = "$startsWith" OpIStartsWith Operator = "$istartsWith" OpEndsWith Operator = "$endsWith" OpIEndsWith Operator = "$iendsWith" OpContains Operator = "$contains" OpIContains Operator = "$icontains" )
type Page ¶
type Page struct {
Items []Document
HasMore bool
Total *int64 // set only when Query.Count is true
}
Page is one page of a list query.
type Provider ¶ added in v0.8.0
type Provider struct {
Name string // the value of storage.provider
Label string // for people
Addressing Addressing
// EndpointRequired says whether storage.endpoint must be given; without it
// DefaultEndpoint(region) is used.
EndpointRequired bool
DefaultEndpoint func(region string) string
// EndpointPattern is what an endpoint (scheme and host, with no path) must
// match; Group 1, when the entry has RegionFromEndpoint, is the region.
EndpointPattern *regexp.Regexp
EndpointHelp string // the shape, for error messages and the template
// AllowHTTP lets the endpoint be plain http (development only: startup
// refuses it unless BACKD_DEV=true).
AllowHTTP bool
// Region rules: FixedRegion is the only valid value ("" = free); with
// RegionFromEndpoint the region is read from the endpoint and must agree with
// one given; DefaultRegion is used when none is given.
FixedRegion string
RegionFromEndpoint bool
DefaultRegion string
// ChecksumSHA256 is whether the x-amz-checksum-sha256 header is accepted and
// verified on a PUT, which direct uploads need.
ChecksumSHA256 Capability
// MaxSinglePut is the largest object one PUT can carry.
MaxSinglePut int64
// MultipartSHA256 is whether a multipart upload can carry a SHA-256 per part, which the
// storage verifies and combines into the object's composite checksum: what direct uploads
// above MaxSinglePut need. Unverified (not confirmed by a smoke test) counts as unsupported.
MultipartSHA256 Capability
// ReadsCORS and ReadsEncryption say whether `backd storage check` can read
// the bucket's CORS rules and default encryption.
ReadsCORS, ReadsEncryption bool
// ChecksumsWhenRequired makes the client send request checksums only where an
// operation needs them: recent AWS SDKs send CRC checksums on every upload,
// which some S3-compatible services reject.
ChecksumsWhenRequired bool
// PublicEndpointAllowed lets a realm set public_endpoint: only for providers
// whose address differs between backd and browsers (a self-hosted MinIO).
PublicEndpointAllowed bool
// OldestTested is the oldest version of a self-hosted provider the tests ran
// against, stated in the docs ("" for hosted providers).
OldestTested string
}
Provider is what backd knows about one S3-compatible storage service.
func LookupProvider ¶ added in v0.8.0
LookupProvider returns the table's entry for name.
func (Provider) Resolve ¶ added in v0.8.0
Resolve applies the provider's endpoint and region rules to what a realm wrote and returns what to connect to, or says what is wrong.
func (Provider) SupportsDirectUploads ¶ added in v0.8.0
SupportsDirectUploads reports whether direct uploads work with this provider: they sign a SHA-256 into the PUT, so the provider must have been confirmed to verify it.
func (Provider) SupportsMultipartUploads ¶ added in v0.8.1
SupportsMultipartUploads reports whether direct uploads above MaxSinglePut work: the provider must verify a SHA-256 per part and report the composite checksum.
type Query ¶
type Query struct {
// Filter is the client's where: a tree of conditions combined with
// And and Or. nil matches everything.
Filter Filter
Sort []SortField // applied in order; ends with "id" as a tiebreaker
Limit int
Skip int
Count bool // also compute the total number of matching documents
// Access restricts the query to documents the caller may read
// (from access rules); nil means no restriction.
Access Filter
// After restricts the page to the documents positioned after a cursor in
// Sort (query.ParseCursor builds it). It narrows the page, not the Total.
After Filter
}
Query selects a page of documents.
type Repository ¶
type Repository interface {
// Create stores doc, which must already carry "id" and "_meta".
Create(ctx context.Context, doc Document) error
// Get returns the document with the given id, or ErrNotFound.
Get(ctx context.Context, id string) (Document, error)
// List returns a page of documents.
List(ctx context.Context, q Query) (Page, error)
// Replace overwrites the document with doc["id"] only if its stored
// _meta.version equals ifVersion (0 means "stored without a version").
// It returns ErrNotFound or ErrVersionMismatch otherwise.
Replace(ctx context.Context, doc Document, ifVersion int64) error
// Delete removes the document with the given id. When ifVersion is not
// nil the stored version must equal it. It returns ErrNotFound or
// ErrVersionMismatch otherwise.
Delete(ctx context.Context, id string, ifVersion *int64) error
}
Repository gives access to the documents of one collection.
type Resolved ¶ added in v0.8.0
type Resolved struct {
Endpoint string // no trailing slash
Region string
HTTP bool // the endpoint is plain http
}
Resolved is the endpoint and region a configuration comes to once the provider's rules are applied.
type Scanner ¶ added in v0.8.0
type Scanner interface {
// ScanAfter returns up to limit documents whose id sorts after afterID
// ("" starts from the first), ordered by id.
ScanAfter(ctx context.Context, afterID string, limit int) ([]Document, error)
// EstimatedCount is the collection's approximate size, cheap to ask.
EstimatedCount(ctx context.Context) (int64, error)
}
Scanner reads every stored document of a collection, in id order, for the schema check: soft-deleted ones included, with no access rules and no writes. A repository that doesn't implement it can't be checked.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package storagetest is a small in-memory S3 server for tests that need an object storage but not a real one: path-style addressing, the operations backd uses (bucket HEAD, PUT, GET with Range, HEAD, DELETE, ListObjectsV2) and the signed SHA-256 check a PUT carries.
|
Package storagetest is a small in-memory S3 server for tests that need an object storage but not a real one: path-style addressing, the operations backd uses (bucket HEAD, PUT, GET with Range, HEAD, DELETE, ListObjectsV2) and the signed SHA-256 check a PUT carries. |