blobfs

package module
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 15, 2026 License: Apache-2.0 Imports: 8 Imported by: 1

README

blobfs

A virtual tree of directories and files in SQL over any object store: the metadata lives in the database, the bytes live in the store under opaque keys, and the library imports no object store and reaches the consumer's only through interfaces the consumer implements.

github.com/standards-lab/blobfs is the base module: the root package blobfs, the persistence package data, and its conformance suite data/datatest, over sqlate and golang.org/x/text alone. The PostgreSQL engine, with its native statements and the schema's migration set, is the postgres sub-module, github.com/standards-lab/blobfs/postgres, released on its own tags. example is a module of its own that composes the library with a real object store; it is never imported.

The problem

An object store is flat and has no atomic rename, so a tree encoded in its keys turns a move of a directory into a copy and a delete of everything beneath it. blobfs keeps the tree in SQL, where a move is one update, and leaves each key opaque. The rows and the objects share no transaction, so blobfs exposes each protocol that touches an object as steps the consumer runs its own store's call between. The data package's store also runs each protocol end to end, calling the put and delete the consumer supplies; concepts explains the model.

Documentation

  1. Concepts: the tree over opaque keys, the write and delete protocols, moves, listings, the delete of a branch and its sweep, and how engines and ownership fit around the library.
  2. Quick start: a working program from go get to a composition against PostgreSQL and an object store, tested without a database and against one.
  3. Features: every operation, refusal, and schema object, package by package, in the detail needed to use them without reading the source.
  4. Glossary: the vocabulary the library introduces, by layer.

The package documentation, go doc github.com/standards-lab/blobfs/..., is the reference for each exported name.

How it works

The library has three layers, and a consumer takes as many as it needs. The root package is Go only: the entities, statuses, key and name rules, and errors. The persistence package, data, holds blobfs's standard-tier SQL and runs it through sqlate on any engine sqlate has a dialect for. An engine sub-module adds the engine's native forms and ships the schema as a migration set.

A consumer builds one pattern catalog, compiles the store against it with the engine installed, and runs the two-phase write through the store's protocol. objects is the consumer's adapter over its object store, a data.ObjectStore; error handling is elided.

catalog, err := query.NewCatalog(sqlatepg.Patterns(), data.Patterns())
store, err := data.New(catalog, db.Dialect(), data.WithEngine(blobfspg.Engine))

// The write: the pending row, the object under the row's key, the row available.
file, err := store.WriteFile(ctx, db, objects, body, size, func(tx *sqlate.Tx) (blobfs.File, error) {
	return store.Files.Create(ctx, tx, objects, dirID, "Q3 summary.txt", "text/plain")
})

// One page of the directory's files, with the total counted in the page's own statement.
page, err := store.Files.List(ctx, db, dirID, query.Directives{}, query.Page{Number: 1, Size: 20})

The callback runs in the transaction that inserts the pending row, where the consumer checks its own scope. file.Key is <id>/Q3 summary.txt. A later rename of the file, or a move of its directory, updates rows and leaves the object where it is. The protocol's steps are exported too: a write that does not fit its shape runs Files.Create, its own put, and Files.Complete itself.

Conventions

Four conventions the library keeps are stricter than a reader might expect:

  • The base module depends on sqlate and golang.org/x/text and nothing else: no driver, no dialect module, no object store. An engine's driver and dialect enter only through its sub-module, and an object store only through the consumer's own adapter.
  • The schema is public API under semantic versioning: from v1.0, the tables, columns, constraint names, referential actions, and migration set change only in a major release. Before v1.0, a minor release may change them, and its changelog marks the change breaking. A released migration never changes.
  • The library imports no object store. It reaches the consumer's through three one-method interfaces, which one adapter implements as data.ObjectStore: every write asks whether the store accepts a key, the store's write protocols put the object, and its delete protocols and the sweep delete it. The protocols' steps, which a consumer may run itself, never touch the store.
  • Every statement in the base module is standard tier, and the persistence package is complete on any engine. A native form lives only in an engine sub-module, and each native file names the feature it uses and what a port to another engine must provide.

Packages

  • blobfs (the root) is the model: Directory, File, and Object, RootID, the status vocabulary and its transitions, NewKey and SanitizeFilename, NormalizeName and ValidateName, KeyValidator, the sentinel errors, ViolationError, and the constraint names.
  • data is the persistence package. New compiles the Store. The store's Directories and Files handles run the operations and listings. Its protocols, WriteFile, EnsureFile, RemoveFile, RemoveFileID, and PurgeFile, run the write and the delete end to end over the consumer's ObjectStore; OnComplete and OnDelete run the consumer's function in the transaction that completes a write or begins a delete. Its Sweep finishes the deletes of marked branches and stopped protocols. The package also holds the Variant interface, the Engine type, and the published patterns.
  • data/datatest is the conformance suite, Run, which an engine or a consumer's own variant runs against a live database.
  • postgres (sub-module) is the PostgreSQL engine: Engine, whose variant takes an advisory tree lock, resolves a path in one statement, and holds a file without writing a row version, and Migrations, the schema's migration set.
  • example (module) composes the library with PostgreSQL and go-storage's Azure Blob provider through one adapter, and runs one file's whole life.

Development

The repository is a Go workspace of three modules, ., postgres, and example. Tasks run through mise:

mise run check         # build, vet, format, fix, tidy, test, and lint every module, then sqlint; writes nothing
mise run currency      # report requirements, Go, tools, actions, and images behind their latest
mise run upgrade       # upgrade every module's go directive and requirements, and the tools, to their latest
mise run test          # the unit tier, every module, nothing installed
mise run lint          # golangci-lint over every module, then sqlint over the repository
mise run build         # each module built with the workspace off, against its own pins
mise run db:up         # build and start the compose PostgreSQL and Azurite and wait until both are healthy
mise run db:down       # stop the stack, keeping its data
mise run db:reset      # stop the stack and drop its data
mise run integration   # the conformance suite and the engine's proofs against the stack
mise run example       # the composition proof against the stack
mise run acceptance    # run db:up, integration, and example, then db:reset whatever the result

compose.yml runs PostgreSQL on port 5434 and Azurite's blob service on port 10000; POSTGRES_PORT and AZURITE_BLOB_PORT move them. Each service builds from compose/<service>/Dockerfile, whose FROM line is the service's one image pin and which carries its configuration and health check; mise run currency reports a pin behind its latest. mise.toml sets the connection settings the integration tier and the example read, their ports following the same two variables. The integration tier is not part of CI.

License

Apache License 2.0.

Documentation

Overview

Package blobfs is the root package of blobfs, a SQL-backed tree of directories and file metadata over an object store the library does not import.

Ids and names:

  • NewID mints a row id, and ParseID checks one a caller supplies.
  • RootID is the id of the one root directory of an install.
  • NormalizeName puts a name in the form uniqueness compares, and ValidateName checks it; MaxNameLength is the longest name allowed.

The entities and their statuses:

  • Directory is a node of the tree, File a file's metadata, and Object what the store reports about a stored object.
  • Status is a file's place in the two-phase write and delete: StatusPending, StatusAvailable, or StatusDeleting. Transition checks a change of Status.
  • DirectoryStatus is a directory's: DirectoryStatusActive or DirectoryStatusDeleting.

Keys:

  • KeyValidator is the one question a write asks of the object store.
  • NewKey builds a file's key, <id>/<sanitized name>, and validates it; SanitizeFilename builds the name's segment. A key is opaque: nothing in blobfs parses it.

Errors:

  • The sentinels, ErrNotFound through ErrCycle, are what package data maps its outcomes onto, matched with errors.Is.
  • IDError, NameError, KeyError, and TransitionError report a refused id, name, key, and status change, each matching its sentinel.
  • ViolationError reports a violation of one of blobfs's constraints, naming the sentinel and the constraint; DeletingError reports a mutation refused because a row's delete has begun, naming whose.
  • The Constraint constants, ConstraintPrimaryKeyDirectory and its siblings, name the constraints and unique indexes blobfs's DDL declares, as ViolationError.Constraint carries them.

The package is Go only; its one dependency outside the standard library is golang.org/x/text, for Unicode normalization. A consumer that keeps its own persistence takes only this package.

docs/concepts.md explains the model and docs/features.md states every rule, the schema included.

Index

Constants

View Source 
const (
	// ConstraintPrimaryKeyDirectory is the primary key of blobfs_directory.
	// A violation is ErrIDTaken: a caller-supplied id another directory
	// already carries.
	ConstraintPrimaryKeyDirectory = "blobfs_pk_directory"

	// ConstraintPrimaryKeyFile is the primary key of blobfs_file. A
	// violation is ErrIDTaken: a caller-supplied id another file already
	// carries.
	ConstraintPrimaryKeyFile = "blobfs_pk_file"

	// ConstraintUniqueDirectoryRoot is the partial unique index that allows
	// one directory row with no parent: the one root per install. A
	// violation is ErrRootDirectory.
	ConstraintUniqueDirectoryRoot = "blobfs_uq_directory_root"

	// ConstraintUniqueDirectoryParentName is the unique constraint on
	// blobfs_directory (parent_id, name). A violation is ErrNameTaken. A
	// name a deleting row holds is refused before the constraint is reached.
	ConstraintUniqueDirectoryParentName = "blobfs_uq_directory_parent_name"

	// ConstraintUniqueFileDirectoryName is the unique constraint on
	// blobfs_file (directory_id, name). A violation is ErrNameTaken. A name
	// a deleting row holds is refused before the constraint is reached.
	ConstraintUniqueFileDirectoryName = "blobfs_uq_file_directory_name"

	// ConstraintForeignKeyDirectoryParent is the foreign key from
	// blobfs_directory.parent_id to blobfs_directory.id. On an insert or a
	// move a violation is ErrNotFound; on a delete it means the directory
	// still has child directories.
	ConstraintForeignKeyDirectoryParent = "blobfs_fk_directory_parent"

	// ConstraintForeignKeyFileDirectory is the foreign key from
	// blobfs_file.directory_id to blobfs_directory.id. On an insert or a
	// move a violation is ErrNotFound; on a delete it means the directory
	// still has files.
	ConstraintForeignKeyFileDirectory = "blobfs_fk_file_directory"
)

The names of the constraints and unique indexes blobfs's DDL declares, which package data maps to sentinels. They are public API: a violation reaches a consumer as ViolationError.Constraint.

View Source 
const MaxNameLength = 255

MaxNameLength is the longest name a directory or file may carry, in runes: the per-component limit ext4, NTFS, and APFS share.

View Source 
const RootID = "00000000-0000-0000-0000-000000000000"

RootID is the id of the one root directory of an install: the nil UUID, seeded by the directory migration. Every path starts at the root, and a consumer that needs the root reads it by this id instead of searching for the row with no parent.

Variables

View Source 
var (
	// ErrNotFound reports a directory or file that does not exist, including
	// a parent or directory id that an insert or move referenced.
	ErrNotFound = errors.New("blobfs: not found")

	// ErrNameTaken reports a name a row of the same kind that is not deleting
	// already holds in the target directory. A deleting holder is refused with
	// its DeletingError instead; see Errors and constraints in
	// docs/features.md.
	ErrNameTaken = errors.New("blobfs: name taken")

	// ErrInvalidName reports a name ValidateName refused. A NameError
	// carries the reason.
	ErrInvalidName = errors.New("blobfs: invalid name")

	// ErrInvalidPath reports a path package data refused: one that starts
	// with a slash, or has a segment ValidateName refuses, which also matches
	// ErrInvalidName.
	ErrInvalidPath = errors.New("blobfs: invalid path")

	// ErrRootDirectory reports an operation refused because it targets the
	// root: creating a second row with no parent, or deleting, moving, or
	// renaming the root. There is exactly one root per install, seeded by
	// the schema with the id RootID.
	ErrRootDirectory = errors.New("blobfs: the root directory")

	// ErrInvalidKey reports a key the store refused. A KeyError carries the
	// key and the store's reason.
	ErrInvalidKey = errors.New("blobfs: invalid key")

	// ErrInvalidID reports a caller-supplied row id that ParseID refused:
	// text that is not a UUID, or the nil UUID, which is RootID. An IDError
	// carries the reason.
	ErrInvalidID = errors.New("blobfs: invalid id")

	// ErrIDTaken reports a caller-supplied row id that a row of the same
	// table already carries: the primary key refused the insert.
	ErrIDTaken = errors.New("blobfs: id taken")

	// ErrNotEmpty reports a directory delete refused because the directory
	// still has child directories or files.
	ErrNotEmpty = errors.New("blobfs: directory not empty")

	// ErrInvalidTransition reports a status change the transition table
	// does not allow. A TransitionError carries the two statuses.
	ErrInvalidTransition = errors.New("blobfs: invalid status transition")

	// ErrDeleting reports a mutation refused because the row is deleting or a
	// directory it reaches is, so nothing enters or leaves a branch being
	// deleted. Package data reports it as a DeletingError.
	ErrDeleting = errors.New("blobfs: row is deleting")

	// ErrNotDeleting reports a purge, the last step of the two-phase
	// delete, refused because the row exists and is not deleting: its
	// delete has not begun. The caller runs the delete's first step first.
	ErrNotDeleting = errors.New("blobfs: the file is not deleting")

	// ErrReferenced reports a delete refused by a foreign key blobfs does
	// not own: a consumer's row still references the file or directory.
	// The sqlate.ConstraintError stays reachable, so the consumer matches
	// the constraint's name against its own and classifies further.
	ErrReferenced = errors.New("blobfs: the row is referenced by a consumer's row")

	// ErrCycle reports a directory move whose new parent sits inside the
	// directory's own subtree, the directory itself included.
	ErrCycle = errors.New("blobfs: move would create a cycle")
)

The sentinels package data maps its outcomes onto. A database violation of one of blobfs's own documented constraints becomes one of these, carried by a ViolationError; a violation of a consumer's constraint is left unclassified.

Functions

func NewID 

func NewID() string

NewID mints a new row id: a version 7 UUID in its canonical string form, so ids sort in insertion order and bind to a uuid column as text.

func NewKey 

func NewKey(store KeyValidator, id, name string) (string, error)

NewKey builds a file's key from its id and display name and validates it against the store. The key is id, a slash, and SanitizeFilename(name). A refusal is a KeyError carrying the store's reason.

func NormalizeName 

func NormalizeName(name string) string

NormalizeName returns name in Unicode normalization form C. Uniqueness is an exact match on the stored name, so package data normalizes every name before an insert or a rename: a composed and a decomposed spelling of the same name then collide instead of coexisting.

func ParseID 

func ParseID(id string) (string, error)

ParseID checks a row id a caller supplies in place of a minted one and returns it in the canonical form NewID mints, so the row and the key built from its id carry the same text whatever form the caller wrote. Any form uuid.Parse accepts is taken. The nil UUID is refused, because it is RootID and belongs to the seeded root. A refusal is an IDError.

func SanitizeFilename 

func SanitizeFilename(name string) string

SanitizeFilename turns a display name into the filename segment of a key. The segment is frozen when the pending row is inserted, is never kept in step with a later rename, and is never parsed back out of the key. It exists so an operator browsing the raw container can read it.

The result satisfies the key rules of the providers blobfs targets: each invalid UTF-8 sequence, slash, backslash, and control character becomes an underscore, trailing dots and whitespace are removed, and a name that nothing survives of becomes "file".

func Transition 

func Transition(from, to Status) error

Transition returns nil when the transition table allows the change from one status to another and a TransitionError otherwise.

func ValidateName 

func ValidateName(name string) error

ValidateName reports whether name may be a directory or file name. A name is non-empty, valid UTF-8, at most MaxNameLength runes, contains no slash and no control character, and is neither "." nor "..". The rejection is a NameError. Validate the normalized form: the check counts runes, and normalization can change the count.

Types

type DeletingError added in v0.3.0 

type DeletingError struct {
	Directory bool
	ID        string
	Err       error
}

DeletingError reports a mutation refused with ErrDeleting and names whose delete refused it: a file's own, with Directory false and ID the file, or a directory's, with Directory true and ID the directory. Err is the refusal's cause when it has one. See Errors and constraints in docs/features.md.

func (*DeletingError) Error added in v0.3.0 

func (e *DeletingError) Error() string

func (*DeletingError) Is added in v0.3.0 

func (e *DeletingError) Is(target error) bool

Is reports whether target is ErrDeleting.

func (*DeletingError) Unwrap added in v0.3.0 

func (e *DeletingError) Unwrap() error

Unwrap returns the cause, or nil when there is none.

type Directory 

type Directory struct {
	ID        string          `json:"id"`
	ParentID  *string         `json:"parent_id"`
	Name      string          `json:"name"`
	Status    DirectoryStatus `json:"status"`
	Version   int64           `json:"version"`
	CreatedAt time.Time       `json:"created_at"`
	UpdatedAt time.Time       `json:"updated_at"`
}

Directory is one node of the tree, a row of blobfs_directory. The root alone has a nil ParentID and the Name "/". Version is the concurrency token the guarded commands check. CreatedAt and UpdatedAt are in time.UTC, whatever time.Local is. The json tags are the scan and binding contract.

func (Directory) IsRoot 

func (d Directory) IsRoot() bool

IsRoot reports whether d is the root directory: the row with no parent.

type DirectoryStatus added in v0.2.0 

type DirectoryStatus string

DirectoryStatus is a directory row's status: active until the delete of its branch marks it deleting, which nothing undoes. It binds and scans as the status column's text. See Deleting a branch in docs/concepts.md. 

const (
	// DirectoryStatusActive marks a directory that accepts children, files,
	// and moves: every directory, the root included, until a mark.
	DirectoryStatusActive DirectoryStatus = "active"

	// DirectoryStatusDeleting marks a directory whose branch is being
	// removed. Nothing enters or leaves it, and its rows keep their names
	// until they are removed.
	DirectoryStatusDeleting DirectoryStatus = "deleting"
)

func (DirectoryStatus) Mutable added in v0.2.0 

func (s DirectoryStatus) Mutable() bool

Mutable reports whether a directory in status s accepts a create or a move beneath it, and a move of itself or of anything in it: every status but deleting.

func (DirectoryStatus) String added in v0.2.0 

func (s DirectoryStatus) String() string

type File 

type File struct {
	ID          string    `json:"id"`
	DirectoryID string    `json:"directory_id"`
	Name        string    `json:"name"`
	Status      Status    `json:"status"`
	Key         string    `json:"key"`
	Size        *int64    `json:"size"`
	ContentType string    `json:"content_type"`
	ETag        *string   `json:"etag"`
	Version     int64     `json:"version"`
	CreatedAt   time.Time `json:"created_at"`
	UpdatedAt   time.Time `json:"updated_at"`
}

File is one file's metadata, a row of blobfs_file. Key is the object's key, built once by NewKey and never parsed back. Size and ETag are nil until the row is available, and ContentType is the declared type until the completion records the store's. CreatedAt and UpdatedAt are in time.UTC, whatever time.Local is. The json tags are the scan and binding contract.

type IDError 

type IDError struct {
	ID     string
	Reason string
}

IDError reports a row id that ParseID refused, with the reason. It matches ErrInvalidID under errors.Is.

func (*IDError) Error 

func (e *IDError) Error() string

func (*IDError) Unwrap 

func (e *IDError) Unwrap() error

Unwrap returns ErrInvalidID, so errors.Is matches the sentinel.

type KeyError 

type KeyError struct {
	Key string
	Err error
}

KeyError reports a key the store refused, with the store's reason. It matches ErrInvalidKey under errors.Is, and errors.As reaches the store's own error through Unwrap.

func (*KeyError) Error 

func (e *KeyError) Error() string

func (*KeyError) Unwrap 

func (e *KeyError) Unwrap() []error

Unwrap returns ErrInvalidKey and the store's error, so errors.Is matches the sentinel and errors.As finds the cause.

type KeyValidator 

type KeyValidator interface {
	// ValidateKey returns a non-nil error that says why when the store
	// refuses key.
	ValidateKey(key string) error
}

KeyValidator is the one question a write asks of the object store: whether it accepts a key. It carries no maximum length; the store's own validation enforces its limit, counted the way the store counts.

type NameError 

type NameError struct {
	Name   string
	Reason string
}

NameError reports a name that ValidateName refused, with the reason. It matches ErrInvalidName under errors.Is.

func (*NameError) Error 

func (e *NameError) Error() string

func (*NameError) Unwrap 

func (e *NameError) Unwrap() error

Unwrap returns ErrInvalidName, so errors.Is matches the sentinel.

type Object 

type Object struct {
	Size        int64
	ContentType string
	ETag        string
}

Object is what the store reports about a stored object, which the completion of a write records.

type Status 

type Status string

Status is a file row's place in the two-phase write and delete: pending until its object exists, then available, and deleting once its delete begins. It binds and scans as the status column's text. 

const (
	// StatusPending marks a row whose object has not been written yet.
	StatusPending Status = "pending"

	// StatusAvailable marks a row whose object exists in the store.
	StatusAvailable Status = "available"

	// StatusDeleting marks a row whose object is being removed. The row
	// keeps its name until it is purged.
	StatusDeleting Status = "deleting"
)

func (Status) Mutable 

func (s Status) Mutable() bool

Mutable reports whether a row in status s accepts a move or a rename: every status but deleting. See Deleting outranks the version in docs/concepts.md.

func (Status) String 

func (s Status) String() string

type TransitionError 

type TransitionError struct {
	From Status
	To   Status
}

TransitionError reports a refused status change. It matches ErrInvalidTransition, and ErrDeleting too when From is deleting.

func (*TransitionError) Error 

func (e *TransitionError) Error() string

func (*TransitionError) Is 

func (e *TransitionError) Is(target error) bool

Is reports whether e stands for target: ErrInvalidTransition always, and ErrDeleting when the refused change started from a deleting row.

type ViolationError 

type ViolationError struct {
	Sentinel   error
	Constraint string
	Err        error
}

ViolationError reports a database constraint violation a classifier mapped to a sentinel: the sentinel, the constraint's name, and the database's error as the cause. Unwrap yields the sentinel and the cause, and the message never prints the driver's text.

func (*ViolationError) Error 

func (e *ViolationError) Error() string

func (*ViolationError) Unwrap 

func (e *ViolationError) Unwrap() []error

Unwrap returns the sentinel and the cause, so errors.Is matches the sentinel and errors.As finds the cause's types.

Source Files

View all Source files

Directories

Path Synopsis
Package data is blobfs's persistence: its standard-tier SQL statements, the pattern namespace it publishes, and the Store that runs the statements through sqlate on any engine sqlate has a dialect for.
 Package data is blobfs's persistence: its standard-tier SQL statements, the pattern namespace it publishes, and the Store that runs the statements through sqlate on any engine sqlate has a dialect for.
datatest
Package datatest is the conformance suite of the data package: the checks every data.Store must pass against a live database, whatever the engine, the returning commands' form, and the variant.
 Package datatest is the conformance suite of the data package: the checks every data.Store must pass against a live database, whatever the engine, the returning commands' form, and the variant.
postgres module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.