Documentation
¶
Overview ¶
Package docstore keeps the files processes receive, so that a process variable can hold a reference to a file instead of the file (ADR-0543).
A variable is a business record: every write of one is an event in the log, a value in state and a full copy in the per-step variable history, and the variable budget is a mebibyte (ADR-0294). A CSV upload or a PDF does not belong there. This package stores it once, outside the log, and hands back a small reference the process carries instead:
{"$document": "doc_…", "name": "…", "contentType": "…", "size": 1234, "sha256": "…"}
What is where ¶
The metadata of every document is a small JSON record under <dir>/meta, written with the sidecar discipline (temp, fsync, rename, fsync the directory). The bytes are kept by a Backend: the local one stores them under <dir>/blobs, one file per SHA-256, and an installation may choose an S3 bucket instead. A record names the backend its bytes were written to, so changing the default never strands a document.
What it never does ¶
It is written by the API server, never by the processor, and nothing in it is replayed: no event carries a document's bytes and applyToState never opens this store (I4). A document's durability is the write discipline here, which is why a reference is only handed out once its bytes and its record are both on disk.
Lifecycle ¶
A document is pending until a variable that names it enters an instance; then it is bound to that instance's root. Store.Sweep removes a pending document after its time to live and a bound one once no instance it is bound to exists any more, and deletes the bytes when no record names them. Deciding *who* may read a document is the API server's job (it knows the caller and the instance rules); this package only records the facts that decision needs.
Index ¶
- Constants
- Variables
- func DetectType(head []byte, name string) string
- func FindRefs(vars map[string]any) []string
- func FindRefsInJSON(text string) []string
- func RefID(v any) string
- func RendersAsPage(contentType string) bool
- func ValidID(id string) bool
- type Backend
- type Bucket
- type Cipher
- type Local
- func (l *Local) Delete(_ context.Context, hash string) error
- func (l *Local) Has(_ context.Context, hash string) (bool, error)
- func (l *Local) Hashes() ([]string, error)
- func (l *Local) Name() string
- func (l *Local) Open(_ context.Context, hash string) (io.ReadCloser, error)
- func (l *Local) Put(_ context.Context, hash, path string, _ int64, _ string) error
- type ObjectStore
- type Options
- type Record
- type Store
- func (s *Store) All() ([]Record, error)
- func (s *Store) Available(ctx context.Context, rec Record) (bool, error)
- func (s *Store) Bind(id string, root uint64) error
- func (s *Store) DefaultBackend() string
- func (s *Store) Get(id string) (Record, error)
- func (s *Store) MaxBytes() int64
- func (s *Store) OpenBytes(ctx context.Context, rec Record) (io.ReadCloser, error)
- func (s *Store) Put(ctx context.Context, name, declaredType, uploader string, body io.Reader) (Record, error)
- func (s *Store) Seal(ctx context.Context, id, subject string) error
- func (s *Store) Sweep(ctx context.Context, pendingTTL time.Duration, ...) (SweepResult, error)
- type SweepResult
Constants ¶
const RefKey = "$document"
RefKey is the key that marks a JSON object as a document reference.
Variables ¶
var ( // ErrTooLarge is returned by Put for a document over the size limit. ErrTooLarge = errors.New("document is larger than the document limit") // ErrNotFound is returned for an id no record exists for. ErrNotFound = errors.New("no such document") // ErrEmpty is returned by Put for a document with no bytes. ErrEmpty = errors.New("document is empty") // a restore from a snapshot taken without documents, or a bucket that lost them. ErrUnavailable = errors.New("document is not available") )
var ErrErased = errors.New("the document belonged to a data subject who was erased; it is permanently unreadable")
ErrErased is OpenBytes's answer for a document whose data subject was erased: its bytes are permanently unreadable, which is the erasure working.
Functions ¶
func DetectType ¶
DetectType says what a document is from its first bytes. The extension refines a text the sniffer can only call text — a CSV is text/csv, not text/plain — and never overrides a binary format the bytes announce.
func FindRefs ¶
FindRefs returns the ids of every document reference in a set of decoded variables (maps, slices, scalars), each once and in no particular order. For a value held as JSON text use FindRefsInJSON.
func FindRefsInJSON ¶
FindRefsInJSON is FindRefs over one JSON text, for a value stored as text (a JSON variable's stored form). Text that is not JSON, or holds no reference, yields none.
func RefID ¶
RefID returns the document id a value names when it is a reference — a JSON object whose "$document" is a well-formed id — and "" otherwise.
func RendersAsPage ¶
RendersAsPage reports whether a browser would run a document of this type as a page rather than show it, so it must be served as an attachment (as ADR-0478 does for exported mail attachments).
Types ¶
type Backend ¶
type Backend interface {
// Name is the backend's identity in a record ("local", "s3").
Name() string
// Put stores the bytes of the file at path under hash. It may be called for a
// hash it already holds, and must then leave the stored bytes as they are.
Put(ctx context.Context, hash, path string, size int64, contentType string) error
// Open reads the bytes stored under hash. A hash it does not hold is os.ErrNotExist.
Open(ctx context.Context, hash string) (io.ReadCloser, error)
// Delete removes the bytes stored under hash; a hash it does not hold is no error.
Delete(ctx context.Context, hash string) error
}
Backend keeps a document's bytes, addressed by their SHA-256 (hex).
type Bucket ¶
type Bucket struct {
// contains filtered or unexported fields
}
Bucket keeps documents' bytes in an S3 bucket, under prefix + hash. It does not list the bucket — the store keeps a journal of what it wrote instead — because a bucket may hold other objects and listing one can be slow and costly.
func NewBucket ¶
func NewBucket(objects ObjectStore, worker, bucket, prefix string, notFound error) (*Bucket, error)
NewBucket returns the bucket backend. notFound is the error the object store answers for a missing key, so Open can say "not available" rather than "failed".
type Cipher ¶
type Cipher interface {
SealBytes(subject, name string, plain []byte) ([]byte, error)
OpenBytes(subject, name string, sealed []byte) ([]byte, error)
Erased(err error) bool
// HasKey reports whether a subject's key still exists, so a sweep can remove the
// documents an erasure made unreadable.
HasKey(subject string) (bool, error)
}
Cipher enciphers a document under a data subject's key and opens it again — the vault's SealBytes and OpenBytes. Opening an erased subject's document answers an error for which Erased is true.
type Local ¶
type Local struct {
// contains filtered or unexported fields
}
Local keeps documents' bytes in a directory, one file per SHA-256. It is the default backend: a bare installation needs nothing else.
func (*Local) Hashes ¶
Hashes lists the hashes the backend holds, which is what lets a sweep find bytes no record names any more.
type ObjectStore ¶
type ObjectStore interface {
PutDocument(ctx context.Context, worker, bucket, key, contentType, disposition string, data []byte, metadata map[string]string) (map[string]any, error)
GetDocument(ctx context.Context, worker, bucket, key string) ([]byte, error)
DeleteDocument(ctx context.Context, worker, bucket, key string) error
}
ObjectStore is what the bucket backend needs of an S3 Worker: put, read back and delete one object, through a Worker the installation already defines, so the store never holds the Worker's access key (connector/s3.Documenter satisfies it).
type Options ¶
type Options struct {
// MaxBytes is the largest document accepted, the installation's limits.Document;
// zero or less means the default document budget.
MaxBytes int64
// Default is the backend new documents are written to; nil is the local backend.
Default Backend
// Others are further backends existing records may name (the local backend is
// always among them, so a document written before a switch stays readable).
Others []Backend
// Now is the clock; nil is time.Now.
Now func() time.Time
// Cipher enciphers the documents of personal variables; nil leaves Seal refusing.
Cipher Cipher
}
Options configure a Store.
type Record ¶
type Record struct {
ID string `json:"id"`
// Name is the file name the uploader gave, without any directory part.
Name string `json:"name"`
// DeclaredType is the content type the uploader claimed; ContentType is what the
// first bytes say, which is what the document is served as.
DeclaredType string `json:"declaredType,omitempty"`
ContentType string `json:"contentType"`
Size int64 `json:"size"`
SHA256 string `json:"sha256"`
// Backend names where the bytes are, so a later change of the default does not
// strand this document.
Backend string `json:"backend"`
// Uploader is the principal that uploaded it, "" with authentication off.
Uploader string `json:"uploader,omitempty"`
// CreatedAt is unix nanoseconds.
CreatedAt int64 `json:"createdAt"`
// BoundTo are the root process instances the document entered, ascending. Empty
// means pending.
BoundTo []uint64 `json:"boundTo,omitempty"`
// Subject is the data subject whose key enciphers the bytes, when a variable the
// process declares personal named the document (ADR-0314); "" when the bytes are
// plain. Blob is then the SHA-256 of the ciphertext, the key the backend holds it
// under — SHA256 stays the content's hash, which is what the reference states.
Subject string `json:"subject,omitempty"`
Blob string `json:"blob,omitempty"`
}
Record is what the store knows about one document.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is the document store rooted at one directory.
func (*Store) Available ¶
Available reports whether a document's bytes are where its record says. A backend that can tell without reading them (the local one) is asked; for one that cannot, the answer is true — reading a whole object to answer a yes-or-no question is the cost this exists to avoid, and a download that finds the bytes missing says so itself.
func (*Store) Bind ¶
Bind records that the document entered the root instance root. Binding a document to an instance it is already bound to changes nothing.
func (*Store) DefaultBackend ¶
DefaultBackend names the backend new documents are written to.
func (*Store) Put ¶
func (s *Store) Put(ctx context.Context, name, declaredType, uploader string, body io.Reader) (Record, error)
Put reads one document from body and stores it. name is the uploader's file name, declaredType the content type the uploader claimed. It returns the record once the bytes and the record are both durable.
func (*Store) Seal ¶
Seal enciphers a document's bytes under a data subject's key: a variable the process declares personal names it (ADR-0314, ADR-0543). The plain bytes stay where they were until a sweep finds no record naming them — at once, unless another document holds the same content. Sealing a document already sealed for the same subject changes nothing; for another subject it is refused, since one key cannot be erased for two people.
func (*Store) Sweep ¶
func (s *Store) Sweep(ctx context.Context, pendingTTL time.Duration, exists func(root uint64) (bool, error)) (SweepResult, error)
Sweep removes pending documents older than pendingTTL, bound documents none of whose instances exists any more (exists answers that), and then the bytes no remaining record names. It is safe to run at any time and again: a removal already made is not made twice, and a document an upload is writing is not seen until its record is.
type SweepResult ¶
type SweepResult struct {
Expired int // pending documents past their time to live
Released int // bound documents whose instances are all gone
Erased int // personal documents whose data subject was erased
Blobs int // byte blobs no record named any more
}
SweepResult says what one sweep removed.