records

package module
v0.0.0-...-3ac6500 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 24, 2026 License: Apache-2.0 Imports: 29 Imported by: 0

README

github.com/ubgo/filesystem/records

Stores files as database rows and attaches them to any row of the application: a user's avatar, a post's gallery, a message's attachments. Bytes live on a filesystem.Disk; rows live in the application's own *sql.DB on SQLite, Postgres or MySQL. The guides start at docs/records; this page is the reference.

Install

go get github.com/ubgo/filesystem/records

A module of its own. It requires the core module github.com/ubgo/filesystem and github.com/ubgo/sqlb, and no database driver: the application opens the *sql.DB (modernc.org/sqlite, github.com/jackc/pgx/v5/stdlib, github.com/go-sql-driver/mysql with parseTime=true) and passes it in. deps_test.go enforces the dependency list.

Wire it

newStore · examples/records-files/main.go:90-99

func newStore(ctx context.Context, db *sql.DB, disks *filesystem.Manager) (*records.Store, error) {
	s, err := records.New(db, sqlb.SQLiteDialect{}, disks)
	if err != nil {
		return nil, err
	}
	if _, err := s.Migrate(ctx); err != nil {
		return nil, err
	}
	return s, nil
}

records.New(db, dialect, disks, opts...) takes the database, one of sqlb.SQLiteDialect{}, sqlb.PostgresDialect{}, sqlb.MySQLDialect{}, and the filesystem.Manager holding the disks files are stored on. Migrate creates the tables; call it on every start. A full walk-through is Files.

Consistency rules

  • Object first, row second. CreateFile records only an object that exists; Upload deletes its object when the row cannot be inserted or an enclosing Tx rolls back.
  • Row first, object last. Purge, ReplaceObject, SetVisibility (between disks) and plugins delete objects only after the transaction commits.
  • What can be left behind is an orphan object, never a row without bytes. Failed post-commit deletions go to WithObjectErrorHandler; SweepOrphans deletes orphans after checking each object against the rows and the plugins that own objects.
  • Reads are open, writes are closed: SELECT and JOIN the fs_ tables freely; change them only through the Store.

Options

Option What it does Default
WithOwnerTypes(types...) Owner types files can attach to; any other is invalid_argument. Pattern ^[a-z][a-z0-9_.]{0,63}$. none
WithForeignOwnerTypes(types...) Owner types spelled the way another framework stores them in a shared database (User, Admin::User, App\Models\Post), registered beside WithOwnerTypes'. Pattern ^[A-Za-z][A-Za-z0-9_.:\\-]{0,63}$: no /, spaces, quotes or control characters. none
WithActorFunc(fn) Replaces ActorFrom as the way the Store and its plugins name who a request acts for (Store.Actor), for an app that keeps its user under its own context key; when fn returns "" the context's WithActor value is read. ActorFrom
WithSingleRoles(roles...) Roles holding at most one file per owner, enforced by a derived primary key even under concurrency. none
WithVisibilityDisks(public, private) Public and private files on two disks; Upload with disk "" chooses, SetVisibility moves. Both must be in the manager. off
WithPlugins(plugins...) Installs plugins; unique names, records reserved. none
WithObjectErrorHandler(fn) Receives object deletions that failed after a commit. dropped
WithIDs(fn) ID generator; receives IDPrefixFile (fil_), IDPrefixAttachment (att_) or a plugin's own prefix (acl_, fld_, ver_, shr_, sev_, whe_, whd_, qrs_, through Store.NewID), returns a unique string with that prefix, at most 191 bytes. Not used for single-role attachments, event IDs or audit rows (sequence-prefixed by design). prefix + 128 random bits, base32
WithClock(now) Replaces time.Now, for the Store and every plugin without its own WithClock (acl, folders, versions, locks, share, quota, webhooks) and the recordshttp and share handlers. time.Now
WithSourceIP() Keeps Source.IP (personal data) when a file is recorded; without it the IP is dropped. off
WithRetention(Retention{OwnerType, Role, MaxAge}...) Retention rules for SweepRetention, in every scope; the owner type must be registered, Role "" is every role, MaxAge positive. Rules per tenant go to SweepRetentionIn. none
WithScanFailure(ScanFailureKeep|ScanFailurePurge) What FailScan does with a file whose scan failed: keep it quarantined with the verdict, or purge it. ScanFailureKeep
WithTablePrefix(prefix), WithTableNames(fn) Every table and index of the core and its plugins under another name: WithTablePrefix("app_") makes fs_files app_fs_files; WithTableNames maps each fs_ name with the application's function (pure, injective, an identifier of at most MaxObjectNameBytes, checked by Migrate). Queries, migrations and the ledger all follow; Store.TableName(name) gives the mapped name for hand-written SQL. Pick it before the first Migrate. the fs_ names
WithFileOrder(name, Ordering), KeysetOrder(key, dir, parse, format) An order of your own for Files (FileQuery.Order: name): any expression over fs_files as a keyset on (key, id), such as sqlb.Lower(FileFilename) for a case-insensitive name sort. The cursor holds the key as the database computed it; name is [a-z][a-z0-9_]*, not built in, added once. the built-in FileOrderValues
WithMigrationLockWait(d) MySQL migration lock wait, whole seconds; 0 fails at once; negative is invalid_argument. DefaultMigrationLockWait (60s)

WithActor(ctx, actor) / ActorFrom(ctx) carry the actor of a request in the context; Store.Actor(ctx) reads it (or WithActorFunc's lookup), and it becomes File.UploadedBy (at most MaxActorBytes, 191) and the share plugin's viewers and depositors.

Action (ActionValues: upload, list, reorder, detach, read, replace, trash, restore, purge, set_attachment, set_visibility, set_text) and ActionTarget{Scope, Owner, Role, FileID, File, Change} are the vocabulary of authorization rules: recordshttp.Action and recordshttp.Target are aliases of them, and permission plugins such as acl decide on them without importing the HTTP layer. Change{Visibility, Text, Previous} is the change a visibility, text or set-attachment request asks for, read before the rule is asked, so "only admins publish" can be written.

Go API

Files (guide):

Method What it does
CreateFile(ctx, NewFile) Records an existing object; size, type, digest and visibility default to the disk's. NewFile.Checksums (extra digests by algorithm, at most MaxChecksums, each hex of at most MaxChecksumBytes) land on File.Checksums. Runs FileHooks, emits file.created.
Upload(ctx, scope, disk, body, filename, upload.Policy, opts...) upload.Receive then CreateFile, keeping the digests Policy.Checksums asked for as File.Checksums (Replace sets the new content's); deletes the object on failure or rollback. Dedup() records the file on an existing live file's object with the same SHA-256, disk and visibility in the scope (deleting the upload's copy); Expires(at) and ViewOnce() make it temporary.
WithSource(ctx, Source{Channel, Client, IP}), SourceFrom(ctx) Where files recorded with ctx came from, kept as File.Source (NewFile.Source overrides); widths MaxSourceChannelBytes, MaxSourceClientBytes, MaxSourceIPBytes, wider is invalid_argument.
File(ctx, scope, id) One file, trashed or not; not_found otherwise.
Files(ctx, FileQuery{Scope, Trashed, Limit, Cursor, Filters, Order}) A page newest first, keyset on (created_at, id); Limit 0 is DefaultPageSize (50), at most MaxPageSize (1000). Order (FileOrderValues) sorts by OrderOldest, OrderName, OrderNameDesc, OrderLargest or OrderSmallest instead, or by a WithFileOrder name, each a keyset on (key, id); a cursor is valid only for the order that issued it.
SetText(ctx, scope, id, Text{Alt, Caption, Title}), File.Text() A file's alt text, caption and title, kept in Metadata under MetaAlt, MetaCaption, MetaTitle and limited to MaxAltChars, MaxCaptionChars, MaxTitleChars on every write path. SetText replaces all three ("" removes one), keeps the other metadata, and refuses with unavailable if the metadata changed concurrently. BeforeReplace hooks are held to the same limits.
AttachedTo(scope, owner, role) A FileQuery.Filters condition keeping an owner's files, under one role or any ("").
Facets(ctx, fq, facet) Counts (files and bytes) of what fq selects by FacetContentType or FacetVisibility; folders plugin: FolderFacet.
SetStatus(ctx, scope, id, Status, detail) Records where a file's processing stands (StatusReady, StatusProcessing, StatusFailed, StatusQuarantined) with a short note; emits file.status_changed with the file. NewFile.Status sets it at creation; filter with sqlb.Eq(FileStatus, StatusFailed). It never moves a file out of StatusQuarantined (ErrQuarantined) and never quarantines a public file (ErrPublicQuarantine).
Quarantine() (upload option), PassScan(ctx, scope, id), FailScan(ctx, scope, id, detail), Scan(ctx, scope, id, scan.Scanner), File.Quarantined() Quarantine until scanned (guide): a quarantined file gets no link (URL, DownloadURL), stream (Served), view (Viewed), share or public visibility (ErrQuarantined, conflict) until PassScan releases it; FailScan keeps it quarantined with the verdict or purges it (WithScanFailure); Scan reads the object, scans it and records the verdict, failing closed; PassScan runs ReleaseHooks and emits file.status_changed then file.released; Dedup never reuses a quarantined object; QuarantineContent() quarantines a file with its replacement's new content.
FileBySHA256(ctx, scope, sum) The newest live file of the scope with that content digest (hex, either case), or not_found: the deduplication lookup, on the (scope, sha256) index. A malformed digest is invalid_argument.
URL(ctx, f, ttl) Public URL for a public file when the disk has one, else a signed GET.
References(ctx, disk, key) How many file rows name an object (shared objects). Every deletion of an object (purge, replace, move, DeleteObjectAfterCommit) keeps it while another row or an ObjectOwner plugin still names it; an in-place visibility change of a shared object is conflict.
UsagesOf(ctx, UsageQuery{Scope, FileID, Limit, Cursor}) Every attachment of a file (owner and role), newest first, keyset-paginated.
Embed(f), Unembed(raw) A file as versioned JSON (EmbedVersion) to keep on its owner's row, and back; a newer version is unsupported.
DownloadURL(ctx, f, ttl) A signed GET that saves under the file's name (AttachmentDisposition).

Attachments (guide):

Method What it does
Attach(ctx, scope, fileID, owner, role) Links a live file at the last position; exists when already attached or a single role is occupied.
Attachment(ctx, scope, id) One attachment.
Attachments(ctx, scope, owner, role) []AttachedFile in position order, trashed files left out.
Reorder(ctx, scope, owner, role, ids) New order; ids must list every attachment Attachments returns (live files) exactly once. Trashed files' attachments keep their relative order after them.
AttachmentsPage(ctx, AttachmentQuery{Scope, Owner, Role, Limit, Cursor}) Attachments a page at a time, keyset on (position, id).
WithRoleLimit(role, max) At most max files per owner under role; past it Attach is quota_exceeded, enforced by derived IDs under concurrency.
Detach(ctx, scope, id) Removes the link; unknown IDs succeed.
SetAttachment(ctx, scope, fileID, owner, role, Previous) Makes the file the role's only one; detached files are kept, trashed or purged (PreviousKeep, PreviousTrash, PreviousPurge) unless still used elsewhere.

Lifecycle (guide):

Method What it does
Trash(ctx, scope, id), Restore(ctx, scope, id) Set or clear DeletedAt; no-ops emit nothing. A trashed public file's object is hidden (moved to the private disk with WithVisibilityDisks, else made private in place where the disk can) and shown again by Restore.
Purge(ctx, scope, id, opts...) Rows in a transaction (after BeforePurge), object after the commit, with the file's row locked so a racing Attach cannot leave an attachment behind. Outside a Tx it returns an object deletion error; inside one the handler gets it. IfTrashed(bool) purges only if the file is still in that trash state (else unavailable).
PurgeOwner(ctx, scope, owner) Detaches every attachment of the owner and purges the files only it used; returns the count.
PurgeUnattached(ctx, scope, uploadedBy, filters...) Purges the files of the scope that uploadedBy recorded and no attachment uses (live or trashed), which PurgeOwner cannot find; filters narrow it; "" is refused. Returns the count.
PurgeScope(ctx, scope) Offboards a tenant: purges every file of the scope (live and trashed) through Purge, 100 IDs at a time, each in its own transaction, then runs every ScopePurger plugin in one transaction. Stops at the first file that cannot be purged; rerun to continue. Leaves the audit log. Refuses "". Returns the files purged.
TrashMany, RestoreMany, PurgeMany(ctx, scope, ids), SetVisibilityMany(ctx, scope, ids, v) One change for up to MaxBulk files, each in its own transaction, one BulkResult per distinct ID; refused inside a Tx.
FindOrphans(ctx, disk, prefix, olderThan, found) What SweepOrphans would delete, deleting nothing.
MissingObjects(ctx, disk, missing) File rows whose object the disk does not have; reports only.
Viewed(ctx, scope, id), File.Expired(now) Asked before serving: false for an expired or trashed file; a view-once file is served exactly once (its first view sets ExpiresAt).
SweepExpired(ctx, now) Purges every file whose ExpiresAt has passed, each in its own transaction; returns the count.
SweepRetention(ctx, now, dryRun) Detaches attachments older than their WithRetention rule and purges files left with none; dryRun reports the same RetentionReport{Detached, Purged} changing nothing.
SweepRetentionIn(ctx, scope, rules, now, dryRun) SweepRetention with the given rules in one scope only: retention by plan ("free 30 days, pro 1 year"), the app passing each tenant's rules. Rules are checked as WithRetention's; a bad one sweeps nothing.
SweepTrash(ctx, cutoff) Purges every file in every scope trashed before cutoff, each in its own transaction; returns the count. A file that cannot be purged does not stop it: the error joins one error per failed file.
FindTrash(ctx, cutoff, found) SweepTrash's dry run: calls found with every file SweepTrash would purge for the same cutoff, changing nothing; a real sweep may purge fewer (a plugin can refuse), never more.
SweepOrphans(ctx, disk, prefix, olderThan) Deletes the objects under prefix that no file row names and no ObjectOwner plugin holds, older than olderThan; returns the count. Never deletes a recorded file.

Replace and visibility (guide):

Method What it does
Replace(ctx, scope, id, body, filename, upload.Policy, opts...) New content under the file's visibility; same ID. IfUnchanged(sha256) replaces only the content the caller read, else records.ErrChanged (conflict), checked before the upload and again under the row lock.
ChangeOption: IfUnchanged(sha256), IfTrashed(bool), QuarantineContent() Narrow Replace, ReplaceObject, Trash, Restore, SetVisibility and Purge to the state the caller read, re-checked with the row locked inside the change's transaction (an object change already made is undone). IfUnchanged refuses with ErrChanged; IfTrashed is Purge's only (unavailable on mismatch, invalid_argument on any other call); QuarantineContent is a replacement's only (invalid_argument on any other call). PurgeOption is an alias.
ReplaceObject(ctx, scope, id, NewContent, opts...) An existing object on the file's disk becomes its content, and gets the file's visibility where the disk can set it.
MoveFile(ctx, scope, id, to) Moves one live file's object to disk to (copy checked against the row, row and FileDiskMover plugins in one transaction, old object deleted after the commit); rerunnable; emits file.moved with Detail from/to.
RenameDisk(ctx, from, to) Points every record on disk from at disk to (a name the Manager has), in every scope and one transaction: fs_files.disk, then each DiskRenamer plugin. Rows only; the objects must already be reachable on to. Returns the file rows changed.
SetVisibility(ctx, scope, id, v) In place (needs the disk's SetVisibility capability) or by moving between the WithVisibilityDisks disks. A trashed file changes only its row; Restore applies it.

Transactions and plugins (events and hooks):

Method What it does
Tx(ctx, fn(ctx, *sql.Tx) error) One transaction; Store calls with its ctx join it; nested calls join the outer one.
AfterCommit(ctx, fn), OnRollback(ctx, fn) Work after a commit (at once without a transaction) or after a rollback.
DeleteObjectAfterCommit(ctx, f) Deletes f.Disk/f.Key after the commit, failures to the handler.
Conn(ctx), Exec, One, Each, Build, Dialect(), Disks() What plugins run statements and reach bytes with; Build renders a statement with the Store's table names for a plugin that runs it another way (a locking read).
TableName(name) The name a default table or index name (TableNameFiles, acl.TableNameGrants) has in this Store's database (WithTableNames); name itself by default.
Hold(ctx, HeldObject), Release(ctx, HeldObject), HeldBytes(ctx, scope) A plugin reports an object it keeps beside a file (a version, a variant) to every UsageHooks plugin, in install order, first error wins; HeldBytes sums every HeldCounter.
Emit(ctx, plugin, Event) A plugin's own event (<name>.<action>, with Subject and Detail) to every EventHooks plugin, inside the change's transaction; a core kind, another plugin's kind or no transaction is invalid_argument.

Migrations (guide):

Name What it does
Migrate(ctx) Applies new migrations under a cross-process lock; returns owner/name of each.
Migrations() Every migration in apply order.
MigrationSQL() Every migration as one script, with the fs_migrations ledger and its inserts, so a later Migrate sees the script's migrations as applied.
NewDDL(engine) with CreateTable(Table{…}), CreateIndex, DropIndex, AddColumn, ModifyColumn, Raw, then Statements() A migration's statements built with github.com/ubgo/sqlb/ddl from typed columns (Column(col, spelling), Key(col)), re-runnable on every engine.
TypesFor(engine) Each engine's spellings of the types fs_ tables use.

Helpers: IsUniqueViolation(err), EncodeCursor(at, id) / DecodeCursor(c), AttachmentDisposition(name).

Plugin interfaces

Interface Methods
Plugin Name() string
Migrator Migrations(engine) ([]Migration, error)
FileHooks BeforeCreate(ctx, s, *File), AfterCreate(ctx, s, File), BeforePurge(ctx, s, File)
ReplaceHooks BeforeReplace(ctx, s, current, *next)
ReleaseHooks BeforeRelease(ctx, s, *f): a quarantined file being released by PassScan, inside its transaction; may set metadata only
Versioner KeepVersion(ctx, s, previous) (bool, error)
EventHooks OnEvent(ctx, s, Event)
ServeHooks OnServed(ctx, s, Served): told of every read a handler reports with Store.Served (egress metering, an access log), outside any transaction
ObjectOwner OwnedObjects(ctx, s, disk, keys) (map[string]bool, error): the plugin's own objects, which SweepOrphans keeps
UsageHooks OnHold(ctx, s, HeldObject), OnRelease(ctx, s, HeldObject): told of objects other plugins keep beside files, inside their transaction
HeldCounter HeldBytes(ctx, s, scope): the bytes the plugin holds in a scope
ScopePurger PurgeScope(ctx, s, scope): deletes the plugin's rows of a scope once Store.PurgeScope has purged its files
FileDiskMover MoveFileDisk(ctx, s, f, to): moves the plugin's own objects for one file inside Store.MoveFile's transaction
DiskRenamer RenameDisk(ctx, s, from, to): rewrites the plugin's own disk names inside Store.RenameDisk's transaction

Event kinds (EventKindValues): file.created, file.trashed, file.restored, file.replaced, file.visibility_changed, file.status_changed, file.released, file.moved, file.purged, attachment.created, attachment.deleted, attachments.reordered. Plugins add their own through Emit: folders, tags and share report their changes, so audit and webhooks see them.

Every event has an ID (evt_ and random, or one the emitting plugin chose, at most MaxEventIDBytes), kept by every redelivery, so at-least-once consumers (webhooks, billing) can drop repeats.

Reads are reported with Store.Served(ctx, Served{Kind, Scope, FileID, Subject, Bytes}), kinds ServeKindValues: access.link, access.share, access.variant, access.stream, or a custom access.<name>.<action> (ServeKind.Valid) for reads the built-in kinds do not name, such as a zip download. Store.URL and Store.DownloadURL call it for every link they issue (records.ServedAs(kind, subject) names a share link or variant) and return no link when it fails, so recordshttp, share links, variants and application code are all metered, and a read of a file quarantined now (read from the database) is refused with ErrQuarantined before any hook; for public files that counts links issued, not fetches. IsLockConflict(err) recognises a transaction the database aborted over a lock conflict on every engine. Store.Now() is the Store's clock, for plugins.

Bundled plugins: acl, audit, egress, folders, locks, quota, share, tags, uploadsessions, versions, webhooks, and recordshttp for the HTTP routes. In the imaging module: variants, dimensions.

Tables

Table Holds
fs_files One row per file. Typed columns FileID, FileScope, FileDisk, FileKey, FileFilename, FileContentType, FileSize, FileSHA256, FileVisibility, FileMetadata, FileCreatedAt, FileDeletedAt, FileUploadedBy, FileChecksums (JSON, NULL for none, core migration 0007) over TableFiles.
fs_attachments One row per link. AttachmentID, AttachmentScope, AttachmentFileID, AttachmentOwnerType, AttachmentOwnerID, AttachmentRole, AttachmentPosition, AttachmentCreatedAt over TableAttachments.
fs_migrations The ledger: owner, name, checksum, applied_at.

Every statement names the tables unqualified, so they live wherever the Store's connection points: a Postgres schema of your own (search_path=storage alone in the pgx DSN), one MySQL database per connection, one SQLite file per Store (the Postgres and MySQL test suites run every test this way, in a schema or database of its own). To keep two Stores in one schema, or to follow a house naming rule, rename them with WithTablePrefix or WithTableNames: every name starting with fs_ (ObjectPrefix) in what the Store runs, tables and indexes, is mapped, plugins' included, and nothing else is.

Errors

Every error the Store creates is a *filesystem.Error; read its code with filesystem.CodeOf(err): invalid_argument (bad input, unregistered owner type, stale reorder, edited migration), not_found (unknown or trashed file where a live one is needed, other scope), conflict (ErrChanged; ErrQuarantined for a quarantined file asked to be served or released by SetStatus; ErrNotQuarantined from FailScan), exists (duplicate attachment, occupied single role), unsupported (engine, or a disk that cannot sign or change visibility), unavailable (MySQL migration lock). Storage and plugin errors pass through unchanged. Errors reference maps codes to HTTP statuses.

Documentation

Overview

Package records stores files as database rows and attaches them to any model of the application: a user's avatar, a post's gallery, a message's attachments. It is the layer above filesystem.Disk.

It is a separate module that requires only github.com/ubgo/sqlb (and the core filesystem module), and runs on SQLite, Postgres and MySQL. The application owns the *sql.DB and the driver; the library never imports a database driver.

Two rules keep bytes and rows consistent, because an object store and a database can never share a transaction:

  • Object first, row second. A file is recorded only after its object is stored (CreateFile checks it exists); Upload deletes the object again when the row cannot be inserted.
  • Row first, object last. Purge deletes the rows inside the transaction and the object only after the transaction commits, so a rollback never leaves a row pointing at deleted bytes.

What can still be left behind (an object whose row insert failed and whose cleanup also failed, or one whose deletion after a purge failed) is an orphan object, never a row without bytes; Store.SweepOrphans removes it, checking every object against the rows (and the plugins that own objects) first. Do not point upload.Sweep or an age-only bucket lifecycle rule at recorded files: they delete by age, recorded or not.

Reads are open, writes are closed: applications may SELECT and JOIN the fs_ tables freely, but must change them only through this package, where the invariants live.

Index

Constants

View Source
const (
	// DetailFrom and DetailTo are the old and new disk of file.moved.
	DetailFrom = "from"
	DetailTo   = "to"
)

Detail keys of the Store's own events.

View Source
const (
	// MaxChecksums caps the digests of one file: generous for every
	// algorithm an application keeps, small enough that the JSON column
	// never grows unbounded.
	MaxChecksums = 16
	// MaxChecksumBytes is the longest digest, in bytes (SHA-512's 64).
	MaxChecksumBytes = 64
)

Limits of File.Checksums.

View Source
const (
	EnginePostgres = sqlb.DialectPostgres
	EngineMySQL    = sqlb.DialectMySQL
	EngineSQLite   = sqlb.DialectSQLite
)

Engine names, as sqlb reports them.

View Source
const (
	TableNameFiles       = "fs_files"
	TableNameAttachments = "fs_attachments"
	TableNameMigrations  = "fs_migrations"
)

Table names. Prefixed so they never collide with application tables.

View Source
const (
	ColFileID          = "id"
	ColFileScope       = "scope"
	ColFileDisk        = "disk"
	ColFileKey         = "storage_key"
	ColFileFilename    = "filename"
	ColFileContentType = "content_type"
	ColFileSize        = "size"
	ColFileSHA256      = "sha256"
	ColFileVisibility  = "visibility"
	ColFileMetadata    = "metadata"
	ColFileCreatedAt   = "created_at"
	ColFileDeletedAt   = "deleted_at"
	ColFileUploadedBy  = "uploaded_by"
	// ColFileStatus and ColFileStatusDetail hold the processing status
	// (core migration 0003).
	ColFileStatus       = "status"
	ColFileStatusDetail = "status_detail"
	// ColFileSourceChannel, ColFileSourceClient and ColFileSourceIP hold
	// File.Source (core migration 0005).
	ColFileSourceChannel = "source_channel"
	ColFileSourceClient  = "source_client"
	ColFileSourceIP      = "source_ip"
	// ColFileExpiresAt and ColFileViewOnce hold File.ExpiresAt and
	// File.ViewOnce (core migration 0006).
	ColFileExpiresAt = "expires_at"
	ColFileViewOnce  = "view_once"
	// ColFileChecksums holds File.Checksums as a JSON object (core
	// migration 0007); NULL for a file without extra digests.
	ColFileChecksums = "checksums"
)

Column names of fs_files.

View Source
const (
	ColAttachmentID        = "id"
	ColAttachmentScope     = "scope"
	ColAttachmentFileID    = "file_id"
	ColAttachmentOwnerType = "owner_type"
	ColAttachmentOwnerID   = "owner_id"
	ColAttachmentRole      = "role"
	ColAttachmentPosition  = "position"
	ColAttachmentCreatedAt = "created_at"
)

Column names of fs_attachments.

View Source
const (
	MaxSourceChannelBytes = 64
	MaxSourceClientBytes  = 255
	// MaxSourceIPBytes fits any IPv6 address with a zone.
	MaxSourceIPBytes = 64
)

Limits of Source, the widths of their columns.

View Source
const (
	IDPrefixFile       = "fil_"
	IDPrefixAttachment = "att_"
)

ID prefixes, so an ID says what it identifies in logs and URLs.

View Source
const (
	MetaAlt     = "alt"
	MetaCaption = "caption"
	MetaTitle   = "title"
)

Reserved Metadata keys of a file's describing text (Text). They live in the metadata column rather than in columns of their own because nothing filters or sorts by them: they are read with the file and shown beside it. Writers other than SetText (CreateFile's Metadata, a BeforeCreate hook) are checked against the same limits.

View Source
const (
	MaxAltChars     = 1000
	MaxCaptionChars = 2000
	MaxTitleChars   = 255
)

Limits of the describing text, in characters. Alt text is read aloud by screen readers and belongs short; a caption may be a paragraph.

View Source
const CoreOwner = "records"

CoreOwner owns the core migrations in the ledger; no plugin may use it.

View Source
const DefaultMigrationLockWait = 60 * time.Second

DefaultMigrationLockWait is how long Migrate waits on MySQL for another instance to finish migrating: long enough for a normal migration, short enough that a stuck lock fails the deploy instead of hanging it.

View Source
const DefaultPageSize = 50

DefaultPageSize is the page size when FileQuery.Limit is 0.

View Source
const EmbedVersion = 1

EmbedVersion is the version of the JSON Embed writes. Unembed reads every version up to it and refuses a newer one (written by a newer library), so an owner row is never read wrongly after a downgrade.

View Source
const IDPrefixEvent = "evt_"

IDPrefixEvent starts the IDs the Store gives events (Event.ID).

View Source
const MaxActorBytes = 191

MaxActorBytes is the longest actor, the indexed width of the columns that store one.

View Source
const MaxBulk = 1000

MaxBulk caps the IDs of one bulk call: a selection a user can make in a file manager, never a whole scope (PurgeScope is for that).

View Source
const MaxEventIDBytes = 191

MaxEventIDBytes is the longest Event.ID: the width of an indexed key column (SQLTypes.Key), where outboxes such as the webhooks plugin's store it.

View Source
const MaxEventKindBytes = 64

MaxEventKindBytes is the longest event kind: the width of an indexed short column (SQLTypes.Short), where a log such as the audit plugin stores it.

View Source
const MaxObjectNameBytes = 63

MaxObjectNameBytes is the longest table or index name a mapping may produce: Postgres keeps 63 bytes of an identifier and cuts the rest silently, which would make two long names one.

View Source
const MaxPageSize = 1000

MaxPageSize caps FileQuery.Limit, so a client cannot ask for every row.

View Source
const MaxRoleLimit = 1000

MaxRoleLimit is the largest WithRoleLimit: an Attach reads every slot of the owner's role, so the bound keeps that one short query.

View Source
const MaxStatusDetailChars = 255

MaxStatusDetailChars is the longest StatusDetail, the width of its column.

View Source
const MaxTablePrefixBytes = 32

MaxTablePrefixBytes is the longest WithTablePrefix: the longest name the module ships (fs_webhook_deliveries_endpoint, 30 bytes) plus the prefix stays within MaxObjectNameBytes.

View Source
const ObjectPrefix = "fs_"

ObjectPrefix starts the name of every table and index this module and its plugins create (fs_files, fs_acl_grants, fs_files_scope_created), and no other word of the SQL they write: no column, alias or keyword starts with it. WithTableNames relies on that invariant, so a plugin's own tables follow it too (a table named otherwise keeps its name under any mapping).

View Source
const PurgeScopeBatch = 100

PurgeScopeBatch is how many file IDs PurgeScope reads at a time: few enough that a scope of millions never loads into memory at once.

Variables

View Source
var (
	TableFiles       = sqlb.Table{Name: TableNameFiles, Alias: "f"}
	TableAttachments = sqlb.Table{Name: TableNameAttachments, Alias: "a"}
)

Tables, with aliases used in joins.

View Source
var (
	FileID          = sqlb.Col[string](TableFiles, ColFileID)
	FileScope       = sqlb.Col[string](TableFiles, ColFileScope)
	FileDisk        = sqlb.Col[string](TableFiles, ColFileDisk)
	FileKey         = sqlb.Col[string](TableFiles, ColFileKey)
	FileFilename    = sqlb.Col[string](TableFiles, ColFileFilename)
	FileContentType = sqlb.Col[string](TableFiles, ColFileContentType)
	FileSize        = sqlb.Col[int64](TableFiles, ColFileSize)
	FileSHA256      = sqlb.Col[string](TableFiles, ColFileSHA256)
	FileVisibility  = sqlb.Col[filesystem.Visibility](TableFiles, ColFileVisibility)
	FileMetadata    = sqlb.Col[string](TableFiles, ColFileMetadata)
	FileCreatedAt   = sqlb.Col[time.Time](TableFiles, ColFileCreatedAt)
	// FileDeletedAt is nil for a live file and set when it is trashed.
	FileDeletedAt = sqlb.Col[*time.Time](TableFiles, ColFileDeletedAt)
	// FileUploadedBy is "" when no actor was known.
	FileUploadedBy = sqlb.Col[string](TableFiles, ColFileUploadedBy)
	// FileStatus is the processing status; filter with it to list the
	// files still processing or failed.
	FileStatus       = sqlb.Col[Status](TableFiles, ColFileStatus)
	FileStatusDetail = sqlb.Col[string](TableFiles, ColFileStatusDetail)
	// FileSourceChannel, FileSourceClient and FileSourceIP are where the
	// file came from (File.Source); "" when not known.
	FileSourceChannel = sqlb.Col[string](TableFiles, ColFileSourceChannel)
	FileSourceClient  = sqlb.Col[string](TableFiles, ColFileSourceClient)
	FileSourceIP      = sqlb.Col[string](TableFiles, ColFileSourceIP)
	// FileExpiresAt is nil for a file that never expires (SweepExpired).
	FileExpiresAt = sqlb.Col[*time.Time](TableFiles, ColFileExpiresAt)
	// FileViewOnce is 0 for an ordinary file, viewOnceUnviewed for a
	// view-once file not yet viewed and viewOnceViewed once Viewed served
	// it (an integer, so every engine stores it the same way).
	FileViewOnce = sqlb.Col[int64](TableFiles, ColFileViewOnce)
	// FileChecksums is File.Checksums as JSON text, nil for none.
	FileChecksums = sqlb.Col[*string](TableFiles, ColFileChecksums)
)

Typed columns of fs_files. Nullable columns are pointers, so null is visible in the type system.

Typed columns of fs_attachments.

ActionValues lists every Action.

View Source
var ErrChanged = filesystem.Errorf(filesystem.CodeConflict, "records: the file changed since it was read")

ErrChanged is the error of a change refused by IfUnchanged (CodeConflict): the file's content is no longer the one the caller read. Match it with errors.Is, as recordshttp does to answer 412.

View Source
var ErrNotQuarantined = filesystem.Errorf(filesystem.CodeConflict, "records: the file is not quarantined")

ErrNotQuarantined is the error of FailScan on a file that is not quarantined (CodeConflict): a verdict arriving for a file another worker already released, or one never quarantined. Nothing is changed.

View Source
var ErrPublicQuarantine = filesystem.Errorf(filesystem.CodeInvalidArgument, "records: a public file cannot be quarantined; make it private first")

ErrPublicQuarantine refuses quarantining a public file, or recording a quarantined file as public (CodeInvalidArgument): its object would be reachable without the library while unscanned. Make it private first, and public after PassScan.

View Source
var ErrQuarantined = filesystem.Errorf(filesystem.CodeConflict, "records: the file is quarantined until its scan passes")

ErrQuarantined is the error of an operation refused because the file is quarantined (CodeConflict): it cannot be served, linked, shared or made public until PassScan releases it. Match it with errors.Is; recordshttp answers it 409 so a client keeps polling the status route.

EventKindValues lists every EventKind.

FacetValues lists every Facet.

FileOrderValues lists every built-in FileOrder; WithFileOrder adds a Store's own.

PreviousValues lists every Previous.

ScanFailureValues lists every ScanFailure.

ServeKindValues lists every built-in ServeKind.

StatusValues lists every Status.

Functions

func ActorFrom

func ActorFrom(ctx context.Context) string

ActorFrom returns the actor set with WithActor, or "". It is the Store's default actor lookup; Store.Actor is the one to read, since WithActorFunc may replace it.

func AttachmentDisposition

func AttachmentDisposition(name string) string

AttachmentDisposition renders a Content-Disposition that saves a file as name, with an ASCII fallback and the exact UTF-8 name (RFC 6266/5987), so non-ASCII names survive every browser.

func Column

func Column[T any](col sqlb.Expr[T], sqlType string) ddl.ColumnDef[T]

Column is a column definition for a migration: the typed column the queries use, spelled for the engine with sqlType (from TypesFor), so a table cannot be created under one name or type and read under another. Nullability follows the Go type: a pointer column is NULL, any other NOT NULL. Give a default with Default on the result.

func DecodeCursor

func DecodeCursor(c string) (time.Time, string, error)

DecodeCursor reverses EncodeCursor; anything else is CodeInvalidArgument.

func Embed

func Embed(f File) (json.RawMessage, error)

Embed returns f as JSON to store on the row that owns it (a user's avatar column, a post's cover), so a page can render the file (its URL through Store.URL, its name, size and type) without reading fs_files. It holds what a reader needs and carries EmbedVersion. The copy goes stale when the file changes: keep it in sync with an EventHooks plugin that re-embeds on file.replaced, file.visibility_changed, file.moved and file.released (whose release hooks may set metadata), and clears it on file.trashed and file.purged. fs_files stays the source of truth.

func EncodeCursor

func EncodeCursor(at time.Time, id string) string

EncodeCursor is the keyset cursor after the row (at, id) of a listing ordered by (time, id) descending. It is exported so plugins with their own listings (an audit log, share links) paginate the way Files does.

func IsLockConflict

func IsLockConflict(err error) bool

IsLockConflict reports whether err is the database aborting a transaction over a lock conflict (a deadlock, a serialization failure, a lock wait that timed out) on any supported engine: the transaction rolled back as a whole and is safe to run again.

func IsUniqueViolation

func IsUniqueViolation(err error) bool

IsUniqueViolation reports whether err is a unique-constraint failure on any supported engine.

func PurgingScope

func PurgingScope(ctx context.Context) (scope string, ok bool)

PurgingScope reports whether ctx is PurgeScope's work, and for which scope. It exists so hooks that refuse ordinary purges (a check-out lock, a hold) can let tenant offboarding through: deleting a tenant beats any one file's guard. A hook should compare the scope with the file's.

func WithActor

func WithActor(ctx context.Context, actor string) context.Context

WithActor returns ctx carrying the actor of the changes made with it: a user ID, "system:cron", "api-key:…". Set it once per request, in the middleware that authenticates; the Store records it as File.UploadedBy, and plugins (audit) log it.

func WithSource

func WithSource(ctx context.Context, src Source) context.Context

WithSource returns ctx carrying the source of the files recorded with it. Set it once per request, beside WithActor, in the middleware that knows the channel and client; CreateFile and Upload record it as File.Source unless NewFile.Source is set.

Types

type Action

type Action string

Action names what a request wants to do to files or attachments, for an authorization rule: recordshttp's Config.Authorize receives one, and permission plugins (acl) map them to their own levels.

Why it lives here and not in recordshttp: a permission plugin decides on actions without serving HTTP, so the vocabulary belongs to the layer both import. recordshttp.Action is an alias of this type, so code that names recordshttp.ActionRead keeps compiling and the two are one type.

const (
	ActionUpload  Action = "upload"  // add files to an owner's role
	ActionList    Action = "list"    // list an owner's files in a role
	ActionReorder Action = "reorder" // change the order of an owner's files
	ActionDetach  Action = "detach"  // remove an attachment
	ActionRead    Action = "read"    // get a link to a file
	ActionReplace Action = "replace" // upload new content for a file
	ActionTrash   Action = "trash"   // move a file to the trash
	ActionRestore Action = "restore" // take a file out of the trash
	ActionPurge   Action = "purge"   // delete a file for good
	// ActionSetAttachment makes an existing file the only one of an
	// owner's role (Store.SetAttachment). recordshttp also authorizes the
	// same request for ActionRead on the file, because attaching it
	// exposes it through the owner's list.
	ActionSetAttachment Action = "set_attachment"
	ActionSetVisibility Action = "set_visibility" // make a file public or private
	ActionSetText       Action = "set_text"       // set a file's alt text, caption and title
)

The closed set of actions.

type ActionTarget

type ActionTarget struct {
	Scope  string
	Owner  Owner
	Role   string
	FileID string
	File   *File
	Change *Change
}

ActionTarget is what an action applies to. Owner and Role are set for the owner routes (and for a detach, from the attachment); FileID for the file routes, the detach and the set-attachment routes. File is the file loaded before the rule is asked, set on every route that names a file, nil elsewhere: a rule can decide on its UploadedBy, Visibility or DeletedAt without another query. Loading first means a file of another scope is not found and never reaches the rule; a rule must not modify *File. Change is the change the request asks for, on the routes that carry one in their body (see Change). recordshttp.Target is an alias of this type.

type AttachedFile

type AttachedFile struct {
	Attachment Attachment
	File       File
}

AttachedFile is an attachment together with its file.

type Attachment

type Attachment struct {
	ID        string
	Scope     string
	FileID    string
	Owner     Owner
	Role      string
	Position  int64
	CreatedAt time.Time
}

Attachment links a file to an owner under a role ("avatar", "gallery"). Position orders the attachments of one owner and role, from 1.

type AttachmentPage

type AttachmentPage struct {
	Files      []AttachedFile
	NextCursor string
}

AttachmentPage is one page of attached live files in position order.

type AttachmentQuery

type AttachmentQuery struct {
	Scope string
	Owner Owner
	Role  string
	// Limit caps the page; 0 means DefaultPageSize, at most MaxPageSize.
	Limit int
	// Cursor continues from AttachmentPage.NextCursor.
	Cursor string
}

AttachmentQuery selects a page of an owner's attachments under a role.

type BulkResult

type BulkResult struct {
	ID  string
	Err error
}

BulkResult is what a bulk call did to one file: Err is nil when the file was changed (or was already in the wanted state), else why not.

type Change

type Change struct {
	// Visibility is the visibility asked for (ActionSetVisibility).
	Visibility filesystem.Visibility
	// Text is the whole describing text asked for (ActionSetText).
	Text *Text
	// Previous is what happens to the files a set-attachment detaches
	// (ActionSetAttachment), records.PreviousKeep when the body named none.
	Previous Previous
}

Change is what a request asks to change, read from its size-capped body before the rule is asked, so a rule can decide on the change as well as on the file: "only admins publish", "a caption needs only Read". It is set on ActionSetVisibility (Visibility), ActionSetText (Text) and ActionSetAttachment (Previous), and nil on every other action. On the visibility and text routes it is also nil when the body could not be read: the request is then refused with 400 once the rule allows it, and a rule refusing it still answers 403 first, so a refused caller learns nothing from the body's shape. A rule must not modify it.

Why the body is read first: without it, a rule sees only that the file is changing, so "publishing needs Admin, unpublishing needs Write" could not be written in any Authorize, OpenFGA and Cedar included.

type ChangeOption

type ChangeOption func(*changeOptions)

ChangeOption narrows a change to one file (Replace, ReplaceObject, Trash, Restore, SetVisibility, Purge) to the state the caller read, so a decision made on that state cannot be applied to a file that changed since. Each option is checked with the file's row locked, inside the change's transaction.

func IfTrashed

func IfTrashed(trashed bool) ChangeOption

IfTrashed makes Purge go ahead only if the file is in the trash (trashed true) or live (false) when its row is locked, and fail with CodeUnavailable, purging nothing, otherwise. A caller that decided to purge from a file it read earlier (an authorization rule allowing only trashed files to be purged) passes the state it saw, so a Restore or Trash that landed in between cannot slip a different file state past that decision. Only Purge takes it; any other change refuses it with CodeInvalidArgument.

func IfUnchanged

func IfUnchanged(sha256 string) ChangeOption

IfUnchanged makes a change go ahead only while the file's content is still the one with digest sha256 (File.SHA256 as the caller read it, either case), and refuse with ErrChanged (CodeConflict), changing nothing, when a replacement landed since: optimistic concurrency, so an editor saving what it loaded, or a user hiding or deleting the version they looked at, cannot act on a newer content they never saw. The check runs with the file's row locked. A file recorded without a digest has SHA256 "", and IfUnchanged("") matches only such a file.

func QuarantineContent

func QuarantineContent() ChangeOption

QuarantineContent makes Replace (or ReplaceObject) quarantine the file with its new content (StatusQuarantined, detail cleared): bytes that replace scanned ones are unscanned again, and must not be served under the file's old verdict. The file must not be public (ErrPublicQuarantine). Only a replacement takes it; any other change refuses it with CodeInvalidArgument, so it is never silently ignored.

type DDL

type DDL struct {
	// contains filtered or unexported fields
}

DDL collects the statements of one migration for one engine, built with github.com/ubgo/sqlb/ddl from the same typed columns the queries use. Every statement it writes can run again after a crash between it and the ledger write, which matters on MySQL, whose DDL commits at once: a table and a Postgres or SQLite index say IF NOT EXISTS, and a MySQL index or added column runs only when the catalogue lacks it. The first build error is kept and returned by Statements, so a migration function reads as a list of steps with one error check at the end.

func NewDDL

func NewDDL(engine string) *DDL

NewDDL starts the statements of a migration for engine (EnginePostgres, EngineMySQL or EngineSQLite).

func (*DDL) AddColumn

func (d *DDL) AddColumn(t sqlb.Table, c ddl.ColumnSpec) *DDL

AddColumn adds a column to an existing table, re-runnably: Postgres says ADD COLUMN IF NOT EXISTS; SQLite's DDL is transactional, so a migration never half applies there; MySQL has no ADD COLUMN IF NOT EXISTS, so its ALTER runs only when information_schema lacks the column. A NOT NULL column needs a default (Default), for the existing rows.

func (*DDL) CreateIndex

func (d *DDL) CreateIndex(t sqlb.Table, ix Index) *DDL

CreateIndex creates an index on an existing table if it does not exist: CREATE INDEX IF NOT EXISTS on Postgres and SQLite, and on MySQL, which has none, a CREATE INDEX that runs only when the catalogue lacks the index (sqlb/ddl's SkipIf, run by a guard: see guarded).

func (*DDL) CreateTable

func (d *DDL) CreateTable(t Table) *DDL

CreateTable creates t if it does not exist. On MySQL its indexes are declared inside the CREATE TABLE (MySQL has no CREATE INDEX IF NOT EXISTS), and the table is InnoDB with utf8mb4 and the byte-exact utf8mb4_bin collation, so names compare byte for byte as on the other engines; on Postgres and SQLite each index is a CREATE INDEX IF NOT EXISTS after the table.

func (*DDL) DropIndex

func (d *DDL) DropIndex(t sqlb.Table, name string) *DDL

DropIndex drops the index named name from t if it exists: DROP INDEX IF EXISTS on Postgres and SQLite, and on MySQL, which has none, a DROP INDEX that runs only when the catalogue still has the index (the same guard CreateIndex uses). A migration that widens a unique key creates the new index first and drops the old one after, so the table is never left without one.

func (*DDL) ModifyColumn

func (d *DDL) ModifyColumn(t sqlb.Table, c ddl.ColumnSpec) *DDL

ModifyColumn changes a MySQL column's definition in place (MODIFY), for a step MySQL needs and the other engines do not (a TEXT column made NOT NULL after a backfill). It writes nothing on the other engines.

func (*DDL) Raw

func (d *DDL) Raw(stmts ...string) *DDL

Raw appends statements as written, for what sqlb/ddl does not build (a data backfill between two schema steps). Keep them re-runnable.

func (*DDL) Statements

func (d *DDL) Statements() ([]string, error)

Statements returns the statements, or none and the first error a step met, so a migration is never half built.

type DiskRenamer

type DiskRenamer interface {
	RenameDisk(ctx context.Context, s *Store, from, to string) error
}

DiskRenamer is a plugin that stores disk names of its own beside fs_files.disk (image variants, kept versions). RenameDisk must rewrite every one of them that is from to to, with Store.Conn(ctx) so the change joins the transaction of Store.RenameDisk; an error rolls the whole rename back. A plugin that stores disk names and does not implement it is left pointing at the old name.

type Event

type Event struct {
	// ID identifies the event for consumers that may see it more than once
	// (webhook deliveries, billing feeds): a consumer that keeps the IDs it
	// processed can drop a redelivery. The Store gives every event a random
	// ID with IDPrefixEvent; a plugin may set its own, deterministic one
	// with Emit when it could report the same fact twice (a daily total
	// reported again after a crash), at most MaxEventIDBytes long.
	ID           string
	Kind         EventKind
	Scope        string
	At           time.Time
	FileID       string
	AttachmentID string
	Owner        Owner
	Role         string
	// File is the file the event is about as the change left it (as it
	// was, for file.purged, whose row is gone), so a hook, a webhook or a
	// log has its name, size, visibility and status without a query; nil
	// for attachments.reordered, and for a plugin's event unless that
	// plugin sets it. Hooks must not change it.
	File *File
	// Subject is the ID of what a plugin's event is about when that is
	// not a file: a folder, a share link. "" otherwise.
	Subject string
	// Detail carries a plugin event's particulars as documented by that
	// plugin, with keys it names as constants (a folder's new parent, the
	// tag added). Hooks must not change it. nil when there are none.
	Detail map[string]string
}

Event is one change. Which fields are set depends on Kind:

  • file.created, file.trashed, file.restored, file.replaced, file.visibility_changed, file.status_changed, file.released, file.purged: FileID and File.
  • file.moved: FileID, File (on its new disk) and Detail with DetailFrom and DetailTo.
  • attachment.created, attachment.deleted: FileID, File, AttachmentID, Owner and Role.
  • attachments.reordered: Owner and Role.
  • a plugin's kinds ("<plugin name>.<action>", sent with Store.Emit): whatever that plugin documents, usually FileID, Subject and Detail.

ID, Kind, Scope and At are always set. The Store's own kinds never set Subject, and only file.moved sets Detail.

type EventHooks

type EventHooks interface {
	OnEvent(ctx context.Context, s *Store, e Event) error
}

EventHooks is a plugin told of every change the Store makes: audit logs, search indexes, webhooks, cache invalidation. OnEvent runs inside the change's transaction, after the change, so its own writes (via Store.Conn) commit or roll back with it, and an error rolls the change back. Work that must not run for a rolled-back change, such as calling a webhook, belongs in Store.AfterCommit. Trashing a trashed file, restoring a live one and detaching an unknown attachment change nothing and have no event.

type EventKind

type EventKind string

EventKind names a change the Store made.

const (
	EventFileCreated  EventKind = "file.created"
	EventFileTrashed  EventKind = "file.trashed"
	EventFileRestored EventKind = "file.restored"
	// EventFileReplaced is a new content for the same file (Replace,
	// ReplaceObject).
	EventFileReplaced EventKind = "file.replaced"
	// EventFileVisibilityChanged is a file made public or private
	// (SetVisibility).
	EventFileVisibilityChanged EventKind = "file.visibility_changed"
	// EventFileStatusChanged is a file's processing status or detail
	// changed (SetStatus).
	EventFileStatusChanged EventKind = "file.status_changed"
	// EventFileReleased is a quarantined file released by its scan
	// (PassScan, Scan), right after its file.status_changed: the signal
	// for a plugin that skipped the file while it was quarantined to do
	// its work now (thumbnails, text extraction, labelling).
	EventFileReleased EventKind = "file.released"
	// EventFileMoved is a file's object moved to another disk (MoveFile);
	// Detail has DetailFrom and DetailTo, the disk names.
	EventFileMoved EventKind = "file.moved"
	// EventFilePurged also stands for the purged file's attachments, which
	// go with it without events of their own.
	EventFilePurged         EventKind = "file.purged"
	EventAttachmentCreated  EventKind = "attachment.created"
	EventAttachmentDeleted  EventKind = "attachment.deleted"
	EventAttachmentsOrdered EventKind = "attachments.reordered"
)

The closed set of event kinds.

type Facet

type Facet string

Facet is a column files can be counted by.

const (
	FacetContentType Facet = "content_type"
	FacetVisibility  Facet = "visibility"
)

The closed set of facets: the columns that hold a small set of values, so a count by them is a short list. A folder facet is the folders plugin's (FileCounts), since folders are its rows.

type FacetCount

type FacetCount struct {
	Value string
	Files int64
	Bytes int64
}

FacetCount is one value of a facet with how many files have it and their bytes.

type File

type File struct {
	ID string
	// Scope is the tenant, workspace or user the file belongs to, opaque to
	// the library ("" for an application without tenants). Every query
	// filters by it.
	Scope       string
	Disk        string
	Key         string
	Filename    string
	ContentType string
	Size        int64
	SHA256      string
	Visibility  filesystem.Visibility
	Metadata    map[string]string
	CreatedAt   time.Time
	// DeletedAt is set while the file is in the trash; nil otherwise.
	DeletedAt *time.Time
	// UploadedBy is who recorded the file: NewFile.UploadedBy, else the
	// actor in the context (WithActor), else "". A replacement keeps it.
	UploadedBy string
	// Status is where the file's processing stands (StatusReady unless a
	// plugin or the application said otherwise, see SetStatus), and
	// StatusDetail the reason or progress note that came with it.
	Status       Status
	StatusDetail string
	// Source is where the file came from (see WithSource); zero when not
	// known. Source.IP is kept only with WithSourceIP.
	Source Source
	// ExpiresAt, when set, is when SweepExpired purges the file; a
	// view-once file gets it at its first view (Viewed).
	ExpiresAt *time.Time
	// ViewOnce marks a file to be served once: Viewed reports the first
	// view and expires the file at that moment.
	ViewOnce bool
	// Checksums are the extra digests of the content beside SHA256, by
	// algorithm name ("md5", "crc32c"), lowercase hex: what
	// upload.Policy.Checksums asked for (Upload and Replace keep
	// upload.Result.Checksums), or NewFile.Checksums. nil when none was
	// asked for. A replacement sets the new content's, so they never
	// describe bytes the file no longer has.
	Checksums map[string]string
}

File is one stored file. Its bytes live on Disk under Key; the row holds what the application needs without touching the disk.

func Unembed

func Unembed(raw json.RawMessage) (File, error)

Unembed reads JSON written by Embed back into a File (only the fields Embed keeps). JSON that is not an embedded file, has no version, or has no file ID is CodeInvalidArgument; a version newer than EmbedVersion is CodeUnsupported.

func (File) Expired

func (f File) Expired(now time.Time) bool

Expired reports whether f has expired at now: its ExpiresAt is set and not after now.

func (File) Quarantined

func (f File) Quarantined() bool

Quarantined reports whether f is quarantined (StatusQuarantined). It reads the value f holds; the serving paths re-read the database.

func (File) Text

func (f File) Text() Text

Text returns the file's describing text from its Metadata.

type FileDiskMover

type FileDiskMover interface {
	MoveFileDisk(ctx context.Context, s *Store, f File, to string) error
}

FileDiskMover is a plugin that keeps objects of its own beside a file (image variants, kept versions) and moves them when the file moves to another disk (MoveFile). MoveFileDisk runs inside MoveFile's transaction after the file's row changed: f is the file as it was (f.Disk the old disk) and to the new disk. It must copy its objects to to, update its rows with Store.Conn(ctx), delete its copies again on rollback (Store.OnRollback) and the old objects only after the commit (Store.DeleteObjectAfterCommit), so a failed move leaves everything as it was. An error rolls the whole move back. A plugin that keeps objects on the file's disk and does not implement it is left pointing at the old disk, whose objects SweepOrphans then keeps only while the plugin reports them (ObjectOwner).

type FileHooks

type FileHooks interface {
	// BeforeCreate runs before a file row is inserted and may change it
	// (metadata, visibility), but not its ID, Disk or Key.
	BeforeCreate(ctx context.Context, s *Store, f *File) error
	// AfterCreate runs after the row is inserted.
	AfterCreate(ctx context.Context, s *Store, f File) error
	// BeforePurge runs before a file's rows are deleted for good.
	BeforePurge(ctx context.Context, s *Store, f File) error
}

FileHooks lets a plugin take part in the file lifecycle. Each hook runs inside the operation's transaction (use Store.Conn(ctx) for statements), and an error aborts the operation and rolls it back. Typical uses: quota counters, audit rows, search indexing, refusing a purge under legal hold.

type FileOrder

type FileOrder string

FileOrder is the order Files lists a page in (FileQuery.Order). Every order is a keyset on (key, id), so pagination stays stable while files are added and fast at any depth.

const (
	// OrderNewest is created_at then ID, descending: the default ("").
	OrderNewest FileOrder = ""
	// OrderOldest is created_at then ID, ascending.
	OrderOldest FileOrder = "oldest"
	// OrderName is filename then ID, ascending, in the database's
	// collation (case-insensitive on MySQL's default, byte order on
	// SQLite, the column's collation on Postgres).
	OrderName FileOrder = "name"
	// OrderNameDesc is filename then ID, descending.
	OrderNameDesc FileOrder = "name_desc"
	// OrderLargest is size then ID, descending.
	OrderLargest FileOrder = "largest"
	// OrderSmallest is size then ID, ascending.
	OrderSmallest FileOrder = "smallest"
)

The closed set of file orders.

type FilePage

type FilePage struct {
	Files      []File
	NextCursor string
}

FilePage is one page of files, in the query's Order.

type FileQuery

type FileQuery struct {
	Scope string
	// Trashed lists only trashed files; otherwise only live ones.
	Trashed bool
	// Limit caps the page; 0 means DefaultPageSize.
	Limit int
	// Cursor continues from FilePage.NextCursor.
	Cursor string
	// Filters are extra conditions on the files table (TableFiles), for
	// plugins that narrow a listing (a folder's files, a tag's files)
	// without re-implementing its order and pagination. A cursor is only
	// valid with the filters it was issued for.
	Filters []sqlb.Predicate
	// Order is the listing's order, OrderNewest ("") by default: by name
	// or by size for a drive's sort menu, or a name the Store added with
	// WithFileOrder. A cursor is only valid with the
	// order it was issued for; one from another order is
	// CodeInvalidArgument. Orders other than the default and OrderOldest
	// are not backed by an index of their own: the scope index narrows
	// the rows and the database sorts them.
	Order FileOrder
}

FileQuery selects a page of files.

type HeldCounter

type HeldCounter interface {
	HeldBytes(ctx context.Context, s *Store, scope string) (int64, error)
}

HeldCounter is a plugin that can report the bytes it holds in a scope, so a recount (quota) can add them to the files' own sizes.

type HeldObject

type HeldObject struct {
	// Plugin is the Name of the plugin holding the object.
	Plugin string
	Scope  string
	// FileID is the file the object belongs to.
	FileID string
	Disk   string
	Key    string
	Size   int64
}

HeldObject is an object a plugin stores beside a file's content: a kept version, an image variant. Such objects take space a file's Size does not show, so a plugin that holds one reports it (Store.Hold) and its end (Store.Release), and plugins that account for space (quota) hear of both through UsageHooks.

type Index

type Index struct {
	// Name is unique in the database (Postgres and SQLite share one index
	// namespace per schema), so prefix it with the table's name.
	Name    string
	Unique  bool
	Columns []IndexColumn
}

Index describes a non-primary index of a table.

type IndexColumn

type IndexColumn struct {
	// contains filtered or unexported fields
}

IndexColumn is one column of an index, for every engine at once.

func Key

func Key[T any](col sqlb.Expr[T]) IndexColumn

Key makes an index column of col.

func KeyPrefix

func KeyPrefix[T ~string](col sqlb.Expr[T], n int) IndexColumn

KeyPrefix makes an index column of the first n characters of a text column on MySQL, whose index keys are capped at 3072 bytes (a Long column is 4096 bytes in utf8mb4), and of the whole column on Postgres and SQLite, which have no such cap. Lookups by the full value still use the index; MySQL rechecks the rest of the value on the rows it finds.

type LinkOption

type LinkOption func(*Served)

LinkOption describes a link Store.URL or DownloadURL issues, for the Served report they make.

func ServedAs

func ServedAs(kind ServeKind, subject string) LinkOption

ServedAs reports the link as kind with subject (ServeShare and the link ID, ServeVariant and the variant name) instead of ServeLink.

type Migration

type Migration struct {
	Owner      string
	Name       string
	Statements []string
}

Migration is one named schema change owned by the core or a plugin. Name identifies it for ever; changing its Statements after it ran makes the next Migrate fail with CodeInvalidArgument rather than skip or re-run it.

The ledger is this package's own rather than sqlb's contrib/migrate, which cannot be imported until sqlb is tagged (its go.mod requires an untagged v0.1.0); ubgo/auth carries one for the same reason.

func (Migration) Checksum

func (m Migration) Checksum() string

Checksum identifies the migration's statements.

type Migrator

type Migrator interface {
	Migrations(engine string) ([]Migration, error)
}

Migrator is a plugin with tables of its own. Migrations must be written for engine (EnginePostgres, EngineMySQL, EngineSQLite) and owned by the plugin's name.

type NewContent

type NewContent struct {
	// Key is the object, on the file's own disk; it must exist and differ
	// from the current one.
	Key string
	// Filename replaces the display name; "" keeps the current one.
	Filename string
	// ContentType, Size and SHA256 default to what the disk reports;
	// SHA256 is 64 hex digits in either case, stored lowercase.
	ContentType string
	Size        int64
	SHA256      string
	// Checksums are the new content's extra digests, as NewFile.Checksums
	// takes them; they replace the file's, so none leaves it none.
	Checksums map[string]string
}

NewContent is the object that becomes a file's content in ReplaceObject.

type NewFile

type NewFile struct {
	Scope string
	// Disk names a disk of the Store's manager; Key is the object on it.
	Disk string
	Key  string
	// Filename is the display name (use upload.SafeFilename on client
	// input).
	Filename string
	// ContentType, Size and Visibility default to what the disk reports.
	ContentType string
	Size        int64
	Visibility  filesystem.Visibility
	// SHA256 is the hex digest, when known (upload.Result has it): 64 hex
	// digits in either case, stored lowercase, so FileBySHA256 finds it.
	SHA256   string
	Metadata map[string]string
	// UploadedBy overrides the actor in the context (WithActor).
	UploadedBy string
	// Status is the file's processing status at creation: "" is
	// StatusReady; StatusProcessing for a file whose processing is about
	// to start (see SetStatus); StatusQuarantined for a file unusable
	// until its scan passes (see PassScan), which must not be public.
	// StatusDetail is its note.
	Status       Status
	StatusDetail string
	// Source overrides the source in the context (WithSource); its IP is
	// kept only with WithSourceIP.
	Source Source
	// ExpiresAt makes the file temporary: SweepExpired purges it once the
	// time has passed. ViewOnce makes it expire at its first view (see
	// Viewed). Both are for ephemeral media; leave them zero otherwise.
	ExpiresAt *time.Time
	ViewOnce  bool
	// Checksums are extra digests of the object beside SHA256, by
	// algorithm name (upload.Result.Checksums has them): names of 1 to 32
	// lowercase letters, digits or dashes, values hex of 1 to
	// MaxChecksumBytes bytes in either case, stored lowercase, at most
	// MaxChecksums of them. The Store records them as given; it does not
	// read the object to check them.
	Checksums map[string]string
}

NewFile describes an already-stored object to record.

type ObjectOwner

type ObjectOwner interface {
	// OwnedObjects reports which of keys, all on the disk named disk, the
	// plugin holds. Keys missing from the map, or mapped to false, are not
	// the plugin's.
	OwnedObjects(ctx context.Context, s *Store, disk string, keys []string) (map[string]bool, error)
}

ObjectOwner is a plugin that stores objects of its own beside the files' (kept versions, image variants). SweepOrphans asks every installed ObjectOwner before deleting an object no fs_files row names, so it never deletes an object a plugin still holds. A plugin that stores objects on a disk SweepOrphans may sweep must implement it.

type Option

type Option func(*Store) error

Option configures a Store.

func WithActorFunc

func WithActorFunc(fn func(ctx context.Context) string) Option

WithActorFunc replaces ActorFrom as the Store's way of naming who a request acts for, for applications that keep their user under their own context key or encode a structured actor ("user:42", "admin:7 as user:42") in their own type. Store.Actor calls fn, and every place the Store and its plugins record an actor reads Store.Actor: File.UploadedBy and the share plugin's viewers and depositors.

Invariant: when fn returns "", Store.Actor falls back to ActorFrom, so an actor the library itself sets with WithActor (a share link's depositor) is still recorded. An actor longer than MaxActorBytes fails the write that records it, as one set with WithActor does. A nil fn is CodeInvalidArgument.

func WithClock

func WithClock(now func() time.Time) Option

WithClock replaces time.Now, for tests.

func WithFileOrder

func WithFileOrder(name FileOrder, o Ordering) Option

WithFileOrder adds a named order to the Store's Files: FileQuery.Order set to name lists in o (see KeysetOrder), with cursors that name it, so a cursor of another order is refused rather than skipping rows. name follows [a-z][a-z0-9_]*, at most 64 bytes, and is neither a built-in FileOrder nor added twice; o must come from KeysetOrder with a key, parse and format. Any of these wrong fails New with CodeInvalidArgument. The built-in orders are unchanged.

Example, names case-insensitively in the database's own lowering:

records.WithFileOrder("name_ci", records.KeysetOrder(sqlb.Lower(records.FileFilename),
	sqlb.SortAsc, func(v string) (string, error) { return v, nil }, func(v string) string { return v }))

func WithForeignOwnerTypes

func WithForeignOwnerTypes(types ...string) Option

WithForeignOwnerTypes registers owner types spelled the way another framework already stores them, so an application sharing a database with another framework's app can attach files to "User", "Admin::User" or "App\Models\Post" and JOIN its polymorphic columns without a mapping table. Types match ^[A-Za-z][A-Za-z0-9_.:\\-]{0,63}$ (letters of either case, digits, "_", ".", ":", a backslash and "-"); anything else is CodeInvalidArgument. WithOwnerTypes keeps its stricter lowercase rule, the default for new applications. Registration is still the allowlist: Attach refuses any type neither option registered.

func WithIDs

func WithIDs(fn func(prefix string) string) Option

WithIDs replaces the ID generator. It receives a prefix (IDPrefixFile, IDPrefixAttachment, or a plugin's own such as "shr_" or "acl_", through Store.NewID) and must return a unique string starting with it, at most 191 bytes (the indexed width on MySQL). Use it to plug in github.com/ubgo/publicid or a UUID scheme: every row a plugin creates carries the same scheme as the files. Event IDs (IDPrefixEvent) and the audit plugin's sequence-prefixed IDs stay random by design.

func WithMigrationLockWait

func WithMigrationLockWait(d time.Duration) Option

WithMigrationLockWait sets how long Migrate waits on MySQL for another instance's migration lock (whole seconds; 0 fails at once if the lock is held). Postgres waits for its advisory lock without a bound; SQLite has one writer and needs no lock.

func WithObjectErrorHandler

func WithObjectErrorHandler(fn func(ctx context.Context, f File, err error)) Option

WithObjectErrorHandler receives the failures of object deletions that run after an enclosing Tx commits (see Purge), when there is no caller to return them to; log them, or queue the key for a retry. Without it they are dropped and the object is left for a sweep: a purged row never comes back, so the failure cannot corrupt the records.

func WithOwnerTypes

func WithOwnerTypes(types ...string) Option

WithOwnerTypes registers the owner types files can be attached to ("user", "post", "product"). Attach refuses any other type, so a typo is an error instead of an attachment to nothing.

func WithPlugins

func WithPlugins(plugins ...Plugin) Option

WithPlugins installs plugins. Their names must be unique.

func WithRetention

func WithRetention(rules ...Retention) Option

WithRetention installs retention rules for SweepRetention. Without it the sweep does nothing. A rule needs a valid owner type (registered with WithOwnerTypes, in any option order), a valid or empty role and a positive MaxAge; a bad rule fails New.

func WithRoleLimit

func WithRoleLimit(role string, max int) Option

WithRoleLimit caps how many files one owner may have under role (a gallery of at most 20 photos, 5 attachments per message), counting attachments of trashed files, which still take their place. Attach past the cap is CodeQuotaExceeded naming the role and the cap. The database enforces it under concurrent requests: the role's attachment IDs are derived from scope, owner, role and a slot number 1..max, so at most max rows can exist, and two Attach calls racing for the last slot make one fail with CodeUnavailable, to retry. WithIDs does not apply to such roles. max must be in [2, MaxRoleLimit] (one file is WithSingleRoles), and a role cannot be both.

func WithScanFailure

func WithScanFailure(a ScanFailure) Option

WithScanFailure chooses what FailScan does with a file whose scan failed; without it, ScanFailureKeep. An unknown action is CodeInvalidArgument.

func WithSingleRoles

func WithSingleRoles(roles ...string) Option

WithSingleRoles declares roles that hold at most one file per owner: "avatar", "cover", "logo". The database enforces it, even under concurrent requests (see SetAttachment), so a second Attach to an occupied single role is CodeExists; SetAttachment replaces the file. Their attachment IDs are derived from scope, owner and role, which is how the database enforces it; WithIDs does not apply to them.

func WithSourceIP

func WithSourceIP() Option

WithSourceIP keeps Source.IP when a file is recorded. Without it the IP is dropped before the row is written, whatever the context or NewFile carries, so a Store never holds addresses its application did not decide to keep.

func WithTableNames

func WithTableNames(fn TableNamer) Option

WithTableNames gives every table and index of the module and its plugins the name fn maps it to: two Stores in one database (one per product, or a test beside a live one), a house naming rule ("storage_files"), or a database whose fs_ names are taken. Queries, migrations and the migration ledger all use the mapped names; without it the names are the defaults, unchanged.

Why a mapping applied to rendered SQL rather than to the typed table values: the typed columns (FileID, acl.Subject) are package values that carry their table, shared by every Store in the process, so a per-Store name can only be applied where a Store renders a statement. Every statement the module runs passes Store.Build, Exec, One or Each, and every migration passes Migrations, so the mapping reaches them all, plugins' included. SQL an application writes itself against these tables names them with Store.TableName.

Pick the names before the first Migrate and keep them: the tables are not renamed for an existing database. A nil fn is CodeInvalidArgument.

func WithTablePrefix

func WithTablePrefix(prefix string) Option

WithTablePrefix is WithTableNames putting prefix before every name: WithTablePrefix("app_") makes fs_files app_fs_files and every index app_fs_…. prefix is 1 to MaxTablePrefixBytes of [a-z0-9_], not starting with a digit (CodeInvalidArgument otherwise).

func WithVisibilityDisks

func WithVisibilityDisks(public, private string) Option

WithVisibilityDisks keeps public and private files on different disks: typically a public bucket behind a CDN and a private one reached only through signed URLs, the usual layout now that S3 disables object ACLs by default. Upload with an empty disk name chooses by the policy's visibility (private when it has none), and SetVisibility moves a file between them. Both disks must be in the Store's manager.

type Ordering

type Ordering struct {
	// contains filtered or unexported fields
}

Ordering is a keyset order of a Store's own for Files (WithFileOrder): a case-insensitive name sort (sqlb.Lower(FileFilename)), a natural sort key, a collation, or any expression over the files table. Build it with KeysetOrder.

Why it exists: the built-in orders sort a name in the database's collation, which is byte order on SQLite and case-sensitive on most Postgres collations; what a drive's users expect varies by product and language, so the expression is the application's.

func KeysetOrder

func KeysetOrder[T any](key sqlb.Expr[T], dir sqlb.SortDirection, parse func(string) (T, error), format func(T) string) Ordering

KeysetOrder is an Ordering on (key, file ID) in dir: rows sort by key, ties by ID, and a page continues after the last row's (key, ID), so pagination stays stable at any depth. key is any expression over TableFiles' columns. The cursor holds the last row's key as the database computed it (selected beside the row), written with format and read back with parse, which must reverse it exactly: a cursor whose value parse refuses is CodeInvalidArgument. Index the expression (a functional index of the application's migration) when listings are large; without one the scope index narrows the rows and the database sorts them.

type Owner

type Owner struct {
	Type string
	ID   string
}

Owner identifies the application row a file is attached to: a type the application registered (WithOwnerTypes) and that row's ID.

type Plugin

type Plugin interface {
	// Name identifies the plugin in the migration ledger and errors. It
	// must be unique and match [a-z][a-z0-9_.]*.
	Name() string
}

Plugin extends a Store. The only required method is Name; every capability is an optional interface the Store detects: Migrator for the plugin's own tables, FileHooks for the file lifecycle.

type Previous

type Previous string

Previous says what SetAttachment does with the files it detaches.

const (
	// PreviousKeep only detaches them.
	PreviousKeep Previous = "keep"
	// PreviousTrash moves them to the trash (restorable).
	PreviousTrash Previous = "trash"
	// PreviousPurge deletes them for good, objects after the commit.
	PreviousPurge Previous = "purge"
)

The closed set of Previous values.

type PurgeOption

type PurgeOption = ChangeOption

PurgeOption is ChangeOption, the name Purge's options had first.

type ReleaseHooks

type ReleaseHooks interface {
	BeforeRelease(ctx context.Context, s *Store, f *File) error
}

ReleaseHooks lets a plugin do, when a quarantined file is released, what its FileHooks.BeforeCreate skipped for the file while it was unscanned: read the bytes and describe them (dimensions, duration, document information). BeforeRelease runs inside PassScan's transaction, before the row changes, with the file as it will be released (StatusReady), and may change its Metadata only; an error aborts the release and the file stays quarantined. Plugins that work after the commit (thumbnails, text extraction) react to EventFileReleased instead.

type ReplaceHooks

type ReplaceHooks interface {
	BeforeReplace(ctx context.Context, s *Store, current File, next *File) error
}

ReplaceHooks lets a plugin take part in a content replacement: quota charges the size difference. BeforeReplace runs inside the transaction before the row changes and may change next's metadata, not its ID, Scope, Disk or Key; an error aborts the replacement.

type Retention

type Retention struct {
	// OwnerType is a registered owner type (WithOwnerTypes).
	OwnerType string
	// Role limits the rule to one role; "" applies it to every role of
	// the owner type.
	Role string
	// MaxAge is how long an attachment is kept, from its creation.
	MaxAge time.Duration
}

Retention is a rule of how long files stay attached under an owner type and role: an attachment older than MaxAge is removed by SweepRetention, and its file purged once nothing else uses it ("chat attachments 90 days", "support uploads 1 year"). Which rules apply is the application's decision; the library only applies them.

type RetentionReport

type RetentionReport struct {
	// Detached are the attachments removed, oldest first per rule.
	Detached []Attachment
	// Purged are the IDs of the files purged because their last
	// attachment was removed (a file still used elsewhere is kept).
	Purged []string
}

RetentionReport is what SweepRetention did, or would do in a dry run.

type SQLTypes

type SQLTypes struct {
	// Key is an ID or scope: indexed, up to 191 characters (MySQL's indexed
	// utf8mb4 width).
	Key string
	// Short is an owner type or role: indexed, up to 64 characters, so a
	// composite unique index stays under MySQL's 3072-byte limit.
	Short string
	// Text is a display string up to 255 characters.
	Text string
	// Long is up to 1024 characters (a storage key), not indexed.
	Long string
	// Int is a 64-bit integer.
	Int string
	// Timestamp is a UTC instant with microsecond precision.
	Timestamp string
	// Doc is large text such as JSON, not indexed.
	Doc string
}

SQLTypes are each engine's spellings of the types fs_ tables use, so a plugin writing its own migration uses the same ones.

func TypesFor

func TypesFor(engine string) SQLTypes

TypesFor returns the spellings for a supported engine.

type ScanFailure

type ScanFailure string

ScanFailure is what FailScan does with a file whose scan failed.

const (
	// ScanFailureKeep keeps the file quarantined, with the verdict in
	// StatusDetail, for an administrator to inspect or purge. The default:
	// nothing is lost, and the file stays unusable.
	ScanFailureKeep ScanFailure = "keep"
	// ScanFailurePurge purges the file (Store.Purge: its rows, then its
	// object once no other file names it).
	ScanFailurePurge ScanFailure = "purge"
)

The closed set of scan failure actions.

type ScopePurger

type ScopePurger interface {
	PurgeScope(ctx context.Context, s *Store, scope string) error
}

ScopePurger is a plugin with rows that belong to a scope rather than to one file: a folder tree, a usage counter, a scope's leftover links. The rows tied to a file go through FileHooks.BeforePurge as each file is purged; PurgeScope then calls PurgeScope once, in one transaction, to delete the rest. It must delete only rows of scope, through Store.Conn(ctx), and succeed when there are none, because PurgeScope may run again after a failure.

type ServeHooks

type ServeHooks interface {
	OnServed(ctx context.Context, s *Store, sv Served) error
}

ServeHooks is a plugin told of every read a handler reports with Store.Served: egress metering, an access log. OnServed runs outside any transaction of the Store's (a read changes nothing), so a hook that writes more than one statement opens its own with Store.Tx.

type ServeKind

type ServeKind string

ServeKind names how a read handed a file, or something derived from it, to a client.

const (
	// ServeLink is a URL to a file issued by Store.URL or DownloadURL:
	// recordshttp's file route and listings, and any application code.
	ServeLink ServeKind = "access.link"
	// ServeShare is a file handed out through a share link (the share
	// plugin's handler); Subject is the link's ID.
	ServeShare ServeKind = "access.share"
	// ServeVariant is a URL to a derived image (the imaging variants
	// handler, URL and Srcset); Subject is the variant's name.
	ServeVariant ServeKind = "access.variant"
	// ServeStream is bytes the application streamed itself, Bytes being
	// what it wrote.
	ServeStream ServeKind = "access.stream"
)

The closed set of serve kinds. They share the "access." prefix, which no EventKind uses, so a log that keeps both (the audit plugin) can tell reads from changes.

func (ServeKind) Valid

func (k ServeKind) Valid() bool

Valid reports whether k is a kind Store.Served accepts: one of ServeKindValues, or a custom kind "access.<name>.<action>" (each part [a-z][a-z0-9_]*) of at most MaxEventKindBytes, for reads the built-in kinds do not name (a zip download, a rendered PDF page). Why custom kinds keep the "access." prefix: the audit plugin stores reads beside changes and tells them apart by it.

type Served

type Served struct {
	Kind   ServeKind
	Scope  string
	FileID string
	// Subject is the share link ID (ServeShare) or variant name
	// (ServeVariant); "" otherwise.
	Subject string
	Bytes   int64
	// At is stamped by Store.Served from the Store's clock.
	At time.Time
}

Served is one read: what was handed out, to which scope, and how many bytes it can carry. Bytes is the size of the object a link points at (an upper bound: a client may not follow the link, or fetch a range), or, for ServeStream, the bytes actually written. A public URL never expires and can be fetched any number of times without the application seeing it, so for public files Served counts links issued; the bytes actually fetched are in the CDN's or the backend's access logs.

type Source

type Source struct {
	// Channel is up to MaxSourceChannelBytes.
	Channel string
	// Client is up to MaxSourceClientBytes.
	Client string
	// IP is up to MaxSourceIPBytes, and kept only when the Store was made
	// WithSourceIP: an IP address is personal data, so storing it is the
	// application's explicit choice.
	IP string
}

Source is where a file came from: the channel it arrived through, the client that sent it, and the client's IP address. The library never interprets the values; the application chooses them (channel "web", "ios", "api", "import"; client a user agent or an API key's name).

func SourceFrom

func SourceFrom(ctx context.Context) Source

SourceFrom returns the source set with WithSource, or the zero Source.

type Statement

type Statement interface {
	Build() (sqlb.Query, error)
}

Statement is a sqlb builder ready to render.

type Status

type Status string

Status is where a file's processing stands: the work that runs after its bytes are stored (image variants, a virus scan, a transcode). A file is usable as stored whatever its status, except StatusQuarantined: the status tells a client whether the derived things it may want (a thumbnail, a scan verdict) are there yet.

const (
	// StatusReady: nothing is pending. Every file starts ready unless
	// NewFile.Status says otherwise, and files recorded before the status
	// existed are ready.
	StatusReady Status = "ready"
	// StatusProcessing: work on the file is under way.
	StatusProcessing Status = "processing"
	// StatusFailed: the work failed; StatusDetail says why.
	StatusFailed Status = "failed"
	// StatusQuarantined: the file is unusable until a scan passes (see
	// quarantine.go): no link, view, share or stream is handed out for
	// it, and it cannot be made public. Only PassScan (or Scan) leaves it;
	// SetStatus refuses to, so a plugin marking its own work done can
	// never release a file nobody scanned. A failed scan keeps it here
	// with the verdict in StatusDetail, or purges it (WithScanFailure).
	StatusQuarantined Status = "quarantined"
)

The closed set of statuses.

func (Status) Valid

func (st Status) Valid() bool

Valid reports whether st is one of StatusValues.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store records files and attachments in a database and reaches their bytes through a filesystem.Manager. It is safe for concurrent use.

func New

func New(db *sql.DB, dialect sqlb.Dialect, disks *filesystem.Manager, opts ...Option) (*Store, error)

New returns a Store over db, rendering SQL for dialect (one of sqlb.PostgresDialect{}, sqlb.SQLiteDialect{}, sqlb.MySQLDialect{}) and reaching bytes through disks. Run Migrate before first use.

func (*Store) Actor

func (s *Store) Actor(ctx context.Context) string

Actor returns who ctx acts for: WithActorFunc's function when it names someone, else ActorFrom(ctx). Plugins read the actor here, so one option rules the Store and every plugin alike.

func (*Store) AfterCommit

func (s *Store) AfterCommit(ctx context.Context, fn func(context.Context))

AfterCommit runs fn once the transaction in ctx commits, or at once when there is none. A rolled-back transaction runs nothing. fn gets ctx's values but never its cancellation, so cleanup survives a client that went away.

func (*Store) Attach

func (s *Store) Attach(ctx context.Context, scope, fileID string, owner Owner, role string) (Attachment, error)

Attach links a live file of scope to owner under role, as the last position. Attaching the same file twice to the same owner and role is CodeExists; a trashed or unknown file is CodeNotFound.

func (*Store) AttachedTo

func (s *Store) AttachedTo(scope string, owner Owner, role string) (sqlb.Predicate, error)

AttachedTo is the condition on TableFiles that keeps the files of scope attached to owner, under role or, with role "", under any role. Put it in FileQuery.Filters to list, page and filter an owner's files like any listing ("this post's PDFs over 1 MB, newest first"), or combine it with a plugin's filter (a folder, a tag). The owner type must be registered (WithOwnerTypes); a bad owner or role is CodeInvalidArgument.

func (*Store) Attachment

func (s *Store) Attachment(ctx context.Context, scope, id string) (Attachment, error)

Attachment returns one attachment of scope, or CodeNotFound.

func (*Store) Attachments

func (s *Store) Attachments(ctx context.Context, scope string, owner Owner, role string) ([]AttachedFile, error)

Attachments returns the live files attached to owner under role, in position order. Trashed files are left out.

func (*Store) AttachmentsPage

func (s *Store) AttachmentsPage(ctx context.Context, aq AttachmentQuery) (AttachmentPage, error)

AttachmentsPage is Attachments a page at a time, for owners with many files (a gallery of thousands): the same live files in the same (position, id) order, with keyset pagination on it. Reorder changes positions, so a cursor taken before a reorder may skip or repeat items.

func (*Store) Build

func (s *Store) Build(st Statement) (sqlb.Query, error)

Build renders st with the Store's table names (WithTableNames): what Exec, One and Each run. A plugin that runs a statement any other way (a locking read with its own clause) builds it here, so its SQL names the same tables.

func (*Store) Conn

func (s *Store) Conn(ctx context.Context) sqlb.DBTX

Conn returns what to run statements on for ctx: the open transaction, or the database. Plugins use it for their own tables so their writes join the transaction.

func (*Store) CreateFile

func (s *Store) CreateFile(ctx context.Context, in NewFile) (File, error)

CreateFile records an object that already exists on its disk. It checks the object exists (the object-first rule), fills size, content type and visibility from the disk when not given, runs the plugins' BeforeCreate and AfterCreate hooks and inserts the row, all in one transaction.

func (*Store) DeleteObjectAfterCommit

func (s *Store) DeleteObjectAfterCommit(ctx context.Context, f File)

DeleteObjectAfterCommit deletes the object of f (its Disk and Key) once the transaction in ctx commits, at once when there is none, and never when it rolls back, unless a file row or an ObjectOwner plugin still names it (a shared object, see References), which is then kept. Call it after removing your own reference (your row). A failure, of the lookup or the deletion, keeps the object and goes to WithObjectErrorHandler. Plugins that own objects (kept versions, image variants) use it, so their deletions follow the object-last rule and report like the Store's own.

func (*Store) Detach

func (s *Store) Detach(ctx context.Context, scope, attachmentID string) error

Detach removes an attachment; the file itself stays. Detaching an unknown attachment succeeds, so cleanup code can call it unconditionally.

func (*Store) Dialect

func (s *Store) Dialect() sqlb.Dialect

Dialect returns the dialect statements must be built for.

func (*Store) Disks

func (s *Store) Disks() *filesystem.Manager

Disks returns the disk manager, for plugins that work on bytes.

func (*Store) DownloadURL

func (s *Store) DownloadURL(ctx context.Context, f File, ttl time.Duration, opts ...LinkOption) (string, error)

DownloadURL returns a signed GET valid for ttl that makes browsers save the file under its original name instead of displaying it, reported with Served as URL's links are.

func (*Store) Each

func (s *Store) Each(ctx context.Context, st Statement, fn func() error) error

Each runs a read and calls fn after scanning each row.

func (*Store) Emit

func (s *Store) Emit(ctx context.Context, from Plugin, e Event) error

Emit tells every EventHooks plugin of e on behalf of the installed plugin from, so a plugin's own changes (a folder moved, a tag added, a share link created) reach audit logs and webhooks like the Store's. e.Kind must be from's name, a dot and an action of [a-z][a-z0-9_]*, at most MaxEventKindBytes in all, and never one of EventKindValues: a plugin cannot speak for the Store or for another plugin. Emit must run inside the transaction of the change it reports (Store.Tx), so the hooks' writes commit or roll back with that change; called outside one, or with a plugin that is not installed, it is CodeInvalidArgument. At is stamped by the Store. An error from a hook is returned, and the caller must fail its transaction with it, as the Store does.

func (*Store) Exec

func (s *Store) Exec(ctx context.Context, st Statement) (int64, error)

Exec runs a write on Conn(ctx) and returns the rows affected.

func (*Store) Facets

func (s *Store) Facets(ctx context.Context, fq FileQuery, facet Facet) ([]FacetCount, error)

Facets counts the files FileQuery selects (its Scope, Trashed and Filters; Limit and Cursor must be unset, as a count has no pages) by facet, most files first, then by value: the sidebar of a media library ("images 120, PDFs 14"). Content types are counted as stored, with any parameters ("text/plain; charset=utf-8"). An unknown facet, or a query with Limit or Cursor, is CodeInvalidArgument.

func (*Store) FailScan

func (s *Store) FailScan(ctx context.Context, scope, id, detail string) (File, error)

FailScan records a failed scan of a quarantined file, detail being the verdict ("infected: Win.Test.EICAR_HDB-1"; the StatusDetail rules apply). With ScanFailureKeep (the default) the file stays quarantined with detail as its StatusDetail; with ScanFailurePurge it is purged, and the File returned is the one purged. A file that is not quarantined is ErrNotQuarantined and nothing changes: a late verdict never purges a file another worker released. An unknown file is CodeNotFound.

func (*Store) File

func (s *Store) File(ctx context.Context, scope, id string) (File, error)

File returns the file with id in scope, including a trashed one.

func (*Store) FileBySHA256

func (s *Store) FileBySHA256(ctx context.Context, scope, sum string) (File, error)

FileBySHA256 returns the newest live file of scope whose content has the SHA-256 digest sum (hex, either case), or CodeNotFound: the lookup for deduplication, so an upload of bytes the scope already holds can reuse that file (attach it again) instead of storing a copy. It reads the (scope, sha256) index. Trashed files are left out, since they are on their way out, and so are files recorded without a digest: a sum that is not 64 hex digits is CodeInvalidArgument, never a match for them. Every file with the digest, a page at a time, is Files with the filter sqlb.Eq(FileSHA256, sum) (lowercase).

func (*Store) Files

func (s *Store) Files(ctx context.Context, fq FileQuery) (FilePage, error)

Files lists a scope's files newest first (or in fq.Order), with keyset pagination on (key, id): stable while files are added, and fast at any depth, unlike OFFSET.

func (*Store) FindOrphans

func (s *Store) FindOrphans(ctx context.Context, diskName, prefix string, olderThan time.Time, found func(key string)) (int, error)

FindOrphans is SweepOrphans without the deletions: it calls found with the key of every object SweepOrphans would delete, in listing order, and returns how many it found. Run it first on a disk you have not swept before, to see what a sweep would take. A listing or database failure stops it with the count so far.

func (*Store) FindTrash

func (s *Store) FindTrash(ctx context.Context, cutoff time.Time, found func(File)) (int, error)

FindTrash is SweepTrash's dry run: it calls found with every file, in any scope, trashed before cutoff, in ID order, and returns how many it found, changing nothing. A real sweep may purge fewer (a plugin can refuse a purge, such as a legal hold), never more for the same cutoff. found runs after the rows were read, so it may query the Store.

func (*Store) HeldBytes

func (s *Store) HeldBytes(ctx context.Context, scope string) (int64, error)

HeldBytes sums the bytes every installed HeldCounter plugin holds in scope; 0 when none is installed. The first error is returned.

func (*Store) Hold

func (s *Store) Hold(ctx context.Context, o HeldObject) error

Hold tells every installed UsageHooks plugin, in install order, that o is now held, and returns the first error, which stops the rest. Call it inside the transaction that records the object, and fail that transaction with the error, so a refused hold leaves no row behind.

func (*Store) Migrate

func (s *Store) Migrate(ctx context.Context) ([]string, error)

Migrate applies every migration not yet in the ledger and returns the ones it applied, as "owner/name". It takes a cross-process lock (Postgres advisory lock, MySQL GET_LOCK; SQLite has one writer), so several instances starting together apply each migration once. A migration whose statements changed after it ran stops Migrate with CodeInvalidArgument.

func (*Store) MigrationSQL

func (s *Store) MigrationSQL() (string, error)

MigrationSQL renders every migration as one SQL script, for review or for applying with an external tool. The script creates the fs_migrations ledger first and records each migration in it after its statements, exactly as Migrate does, so a later Migrate sees the script's migrations as applied instead of running them again. The ledger inserts skip a migration already recorded, but a migration's own statements are not all repeatable (SQLite has no ADD COLUMN IF NOT EXISTS): apply the script once, to a database Migrate has not set up.

func (*Store) Migrations

func (s *Store) Migrations() ([]Migration, error)

Migrations returns every migration in apply order: the core's, then each plugin's in installation order.

func (*Store) MissingObjects

func (s *Store) MissingObjects(ctx context.Context, diskName string, missing func(File)) (int, error)

MissingObjects calls missing with every file row, in any scope, live or trashed, on the named disk whose object the disk does not have, and returns how many it found: the inverse of FindOrphans, for an audit after a restore from backup or a hand-edited bucket. It only reports; it never changes a row, because whether to purge the row, restore the object from a backup or re-upload it is the application's decision. It checks each object with one Exists call, reading rows a batch at a time in ID order. Objects plugins keep (versions, variants) are not checked. A database or storage failure stops it with the count so far.

func (*Store) MoveFile

func (s *Store) MoveFile(ctx context.Context, scope, id, to string) (File, error)

MoveFile moves one live file's object to the disk named to, under the same key, and records it there, for storage tiering and for "accept fast, store later" (an upload landed on a local disk, promoted to the bucket by a job). It is object-first like every change of the Store:

  1. the object is copied to to and the copy checked against the row (size, and SHA-256 when both are known);
  2. in one transaction, with the file's row locked, the row's disk changes and every FileDiskMover plugin moves its own objects;
  3. the old object is deleted after the commit (a failure goes to WithObjectErrorHandler and leaves an orphan for SweepOrphans).

A failure before the commit deletes the copy again, so the file stays where it was, whole. A crash leaves at most an extra copy on one disk, never a row without its object, and running MoveFile again finishes the move (it overwrites a copy left behind); a file already on to is returned as it is, with no event. It returns the file as moved and emits EventFileMoved inside the transaction, with the file on its new disk and Detail naming the old and new disk.

Refused with nothing changed: a trashed file (CodeNotFound, as Replace does), another file recorded under the same key on to (CodeExists), a disk the Manager does not have (CodeNotFound), a disk that cannot give the file its visibility (CodeUnsupported, as CreateFile refuses it), and, with WithVisibilityDisks, a move that would leave the file off the disk its visibility routes to (CodeInvalidArgument): out of one of those disks to any other, or into one that is not its visibility's. A row that changed during the copy (replaced, moved) is CodeUnavailable, to retry.

func (*Store) NewID

func (s *Store) NewID(prefix string) string

NewID returns a new row ID starting with prefix from the Store's ID generator (WithIDs, else prefix plus 128 random bits in lowercase base32). Plugins create their rows' IDs with it, so an application's ID scheme (UUIDv7, a publicid) covers every table and not only files and attachments. The generator's contract applies: unique, starting with prefix, at most 191 bytes.

func (*Store) Now

func (s *Store) Now() time.Time

Now is the Store's clock (WithClock) in UTC at microsecond precision, the precision every engine keeps: plugins stamp and compare times with it, so one clock rules a deployment and its tests.

func (*Store) OnRollback

func (s *Store) OnRollback(ctx context.Context, fn func(context.Context))

OnRollback runs fn if the transaction in ctx rolls back, and never when there is no transaction. Upload uses it to delete the object of a file whose row a rollback undid, so a failed request leaves no orphan bytes.

func (*Store) One

func (s *Store) One(ctx context.Context, st Statement) error

One runs a read expecting one row and scans it into the statement's bindings; sqlb.ErrNoRows when there is none.

func (*Store) PassScan

func (s *Store) PassScan(ctx context.Context, scope, id string) (File, error)

PassScan releases a quarantined file: every ReleaseHooks plugin runs on it, then its status becomes StatusReady with no detail, with the metadata the hooks set, and EventFileStatusChanged and then EventFileReleased are emitted, all in one transaction; after it the file can be served, shared and made public. A file that is not quarantined is returned unchanged, so a verdict delivered twice is harmless. The row is locked for the change. An unknown file is CodeNotFound.

func (*Store) Purge

func (s *Store) Purge(ctx context.Context, scope, id string, opts ...ChangeOption) error

Purge deletes a file for good: its attachments and row in one transaction (after the plugins' BeforePurge hooks), then its object once the transaction commits. Called outside a Tx it returns an object deletion failure; inside one the deletion happens at the outer commit and a failure is passed to the handler set with WithObjectErrorHandler.

Options narrow the purge to the state the caller read (IfTrashed, IfUnchanged), checked under the row lock.

func (*Store) PurgeMany

func (s *Store) PurgeMany(ctx context.Context, scope string, ids []string) ([]BulkResult, error)

PurgeMany purges each file of ids, as Purge, each in its own transaction; an object whose deletion fails after its rows went is reported to WithObjectErrorHandler, not in the results, as with SweepTrash.

func (*Store) PurgeOwner

func (s *Store) PurgeOwner(ctx context.Context, scope string, owner Owner) (int, error)

PurgeOwner deletes what an owner leaves behind when the application deletes it (a user closing their account, a post removed): every attachment of owner, in every role, and every file attached only to it, purged for good with its objects deleted after the commit. A file another owner still uses is only detached. Call it in the transaction that deletes the owner's own row, so both happen or neither. Returns how many files were purged.

func (*Store) PurgeScope

func (s *Store) PurgeScope(ctx context.Context, scope string) (int, error)

PurgeScope deletes everything the Store and its plugins keep for one scope, to offboard a tenant: every file (trashed or not) with its attachments, through Purge, so every plugin's BeforePurge runs and the objects are deleted after each file's commit; then, in one transaction, every ScopePurger plugin's rows of the scope. Each file is purged in its own transaction, a batch of IDs at a time, so it can run on a scope of any size; call it outside a Tx. It stops at the first file that cannot be purged (a database failure, a plugin refusing, a cancelled ctx) and returns how many it purged before; nothing purged comes back, so run it again once the cause is fixed and it continues where it stopped. The scope's own rows go only when every file is gone: the last transaction counts the scope's files after the ScopePurgers ran and, if an upload landed meanwhile, rolls their work back and purges the new files first, up to a few rounds, then gives up with CodeUnavailable, so stop uploads to the scope before offboarding it. An object whose deletion fails goes to WithObjectErrorHandler and is left for SweepOrphans. Every purged file emits file.purged. A log that outlives files (the audit plugin) is not erased: prune it yourself if the tenant's history must go too. An empty scope is CodeInvalidArgument: it is the scope of an application without tenants, which is every file.

func (*Store) PurgeUnattached

func (s *Store) PurgeUnattached(ctx context.Context, scope, uploadedBy string, filters ...sqlb.Predicate) (int, error)

PurgeUnattached purges, for good and with their objects deleted after the commit, the files of scope that uploadedBy recorded (File.UploadedBy) and that no attachment uses, live or trashed: the uploads PurgeOwner cannot find, because it works from attachments (a file recorded and never attached, or detached with PreviousKeep). An owner is an application row and an uploader is an actor, and only the application knows which actor an owner was, so account deletion calls both in one transaction: PurgeOwner(ctx, scope, owner), then PurgeUnattached(ctx, scope, actor). Files the actor uploaded that another owner uses are attached, so they are kept.

"Unattached" means no fs_attachments row: a file that a plugin places somewhere (a folder of a shared drive) is still unattached. filters, as in FileQuery.Filters, narrow the files further (records.FileCreatedAt before a date, a plugin's condition); a file a plugin refuses to purge (BeforePurge) fails the whole call. An empty uploadedBy is CodeInvalidArgument: it stands for every upload made without an actor, which is never one person's. Returns how many files were purged.

func (*Store) References

func (s *Store) References(ctx context.Context, disk, key string) (int64, error)

References returns how many file rows, in every scope and trashed ones included, name the object key on disk: 0 for an orphan, more than 1 for a shared object. Objects that plugins hold (kept versions) are not counted; see ObjectOwner.

func (*Store) Release

func (s *Store) Release(ctx context.Context, o HeldObject) error

Release tells every installed UsageHooks plugin, in install order, that o is no longer held, and returns the first error, which stops the rest. Call it inside the transaction that removes the object's row; the object itself is deleted after the commit (DeleteObjectAfterCommit).

func (*Store) RenameDisk

func (s *Store) RenameDisk(ctx context.Context, from, to string) (int64, error)

RenameDisk points every record on the disk named from at the disk named to instead, in every scope and in one transaction: fs_files.disk, then each DiskRenamer plugin's own columns. It is the records side of renaming a disk in the Manager (moving a store to S3 under a new name): it changes rows only and copies no bytes, so the objects must already be reachable on to under the same keys. to must be a disk of the Store's Manager; from need not be any more. Trashed files move too. It emits no events (it can touch every row), so pause uploads to from while it runs and keep both names registered until it has committed. Returns how many fs_files rows changed. from == to, or an empty name, is CodeInvalidArgument; a to the Manager does not have is CodeNotFound.

With WithVisibilityDisks, renaming from one of those disks to a disk that is not one is CodeInvalidArgument: the Store would keep uploading to, and moving files between, the old names. Rebuild the Store with WithVisibilityDisks naming the new disk first, then rename.

func (*Store) Reorder

func (s *Store) Reorder(ctx context.Context, scope string, owner Owner, role string, ids []string) error

Reorder sets the order of owner's attachments under role: ids must list every attachment of that owner and role that Attachments returns (the live files) exactly once, in the new order. Anything else is CodeInvalidArgument and changes nothing, so a client working from a stale list cannot silently drop an item. Attachments of trashed files, which clients do not see, keep their relative order after the listed ones, so a restored file comes back at the end.

func (*Store) Replace

func (s *Store) Replace(ctx context.Context, scope, id string, body io.Reader, filename string, p upload.Policy, opts ...ChangeOption) (File, error)

Replace stores body as the new content of a live file, under policy p (see upload.Receive), and records it with ReplaceObject. The new object gets the file's visibility, whatever p says, so a private file cannot become public by being replaced. filename "" keeps the display name. If the replacement fails, or an enclosing Tx rolls back, the new object is deleted again.

func (*Store) ReplaceObject

func (s *Store) ReplaceObject(ctx context.Context, scope, id string, in NewContent, opts ...ChangeOption) (File, error)

ReplaceObject makes an existing object the content of a live file. The file keeps its ID, so its attachments, and whatever plugins tie to it, stay. Like CreateFile it is object-first: the new object must exist, and is given the file's visibility where its disk can set it (put back if the transaction fails), and the previous one is deleted only after the transaction commits (failures go to WithObjectErrorHandler), unless a Versioner keeps it. Emits EventFileReplaced. With IfUnchanged it replaces only the content the caller read (optimistic concurrency); with QuarantineContent the file is quarantined with its new content. A quarantined file stays quarantined whatever replaces it.

func (*Store) Restore

func (s *Store) Restore(ctx context.Context, scope, id string, opts ...ChangeOption) error

Restore takes a file out of the trash and, for a public file, makes its object public again (see Trash). IfUnchanged restores it only while its content is the one the caller read.

func (*Store) RestoreMany

func (s *Store) RestoreMany(ctx context.Context, scope string, ids []string) ([]BulkResult, error)

RestoreMany restores each file of ids, as Restore, each in its own transaction.

func (*Store) RoutesVisibility

func (s *Store) RoutesVisibility() bool

RoutesVisibility reports whether the Store was built WithVisibilityDisks, so an Upload with an empty disk name chooses the disk itself: handlers that take an optional disk check it once at construction instead of failing every request.

func (*Store) Scan

func (s *Store) Scan(ctx context.Context, scope, id string, sc scan.Scanner) (File, error)

Scan scans a quarantined file's object with sc and records the verdict: clean is PassScan, infected is FailScan with "infected: <signature>". It is what a worker runs for each quarantined file (list them with Files and FileStatus). A scan that does not complete (sc's error, the object unreadable) is returned and changes nothing, so the file stays quarantined: scanning fails closed. A file that is not quarantined is returned unchanged without being read.

func (*Store) Served

func (s *Store) Served(ctx context.Context, sv Served) error

Served reports a read to every installed ServeHooks plugin, in install order. Store.URL and DownloadURL call it for every link they issue, so handlers built on them are metered without calling it; call it yourself only for bytes the application streams (ServeStream), before writing them, and refuse the request when it fails, so nothing is served that metering or an access log did not see. Every hook runs even when an earlier one fails, and their errors are joined: one plugin's failure must not hide the read from another. Kind must be valid (ServeKind.Valid: a built-in kind, or a custom "access.<name>.<action>") and Bytes not negative (CodeInvalidArgument). A read of a file that is quarantined now (read from the database, not from the caller's File) is refused with ErrQuarantined before any hook runs. With no ServeHooks plugin installed it does nothing else.

func (*Store) SetAttachment

func (s *Store) SetAttachment(ctx context.Context, scope, fileID string, owner Owner, role string, previous Previous) (Attachment, error)

SetAttachment makes fileID the only file of owner under role, in one transaction: the user's new avatar replacing the old. The files it detaches are kept, trashed or purged as previous says, but only when no other attachment uses them: a file shared with another owner is only detached. Setting the file already attached alone does nothing. Two concurrent calls on a single role (WithSingleRoles) cannot both win: the loser fails and can retry.

func (*Store) SetStatus

func (s *Store) SetStatus(ctx context.Context, scope, id string, st Status, detail string) (File, error)

SetStatus records where a file's processing stands: a plugin or a worker calls it with StatusProcessing when it starts, StatusReady when it is done, and StatusFailed with the reason in detail when it gives up ("thumbnail: unsupported format"). detail is a short note for the application and its users ("" for none), never a stack trace. It runs in a transaction (joining the one in ctx) and emits EventFileStatusChanged with the file as updated; setting the current status and detail does nothing. It works on a trashed file too. An unknown status or a bad detail is CodeInvalidArgument; an unknown file is CodeNotFound.

Quarantine is guarded here because every processing plugin calls SetStatus: a quarantined file cannot be moved to any other status (ErrQuarantined; PassScan releases it), and a public file cannot be quarantined (ErrPublicQuarantine), since its object is already reachable without the library. The row is locked for the check.

func (*Store) SetText

func (s *Store) SetText(ctx context.Context, scope, id string, t Text) (File, error)

SetText replaces a file's describing text (all three fields; "" removes one) and returns the file as stored. The rest of its Metadata is kept. It works on a trashed file too. The row is written only if its metadata is still what was read, so a concurrent metadata writer is never silently overwritten: that race is CodeUnavailable, to retry. Setting the current text changes nothing. It emits no event.

func (*Store) SetVisibility

func (s *Store) SetVisibility(ctx context.Context, scope, id string, v filesystem.Visibility, opts ...ChangeOption) error

SetVisibility makes a file public or private. With WithVisibilityDisks the object moves to the other disk (copied first, the old one deleted after the commit); otherwise its own disk changes its access, which needs the SetVisibility capability. The object changes before the row, and a failed transaction puts it back, so the row never claims a visibility the object does not have. A trashed file changes only its row: its object stays hidden (see Trash) until Restore applies the new visibility. Emits EventFileVisibilityChanged; setting the current visibility does nothing. IfUnchanged changes it only while the file's content is the one the caller read.

func (*Store) SetVisibilityMany

func (s *Store) SetVisibilityMany(ctx context.Context, scope string, ids []string, v filesystem.Visibility) ([]BulkResult, error)

SetVisibilityMany sets the visibility of each file of ids, as SetVisibility, each in its own transaction.

func (*Store) SweepExpired

func (s *Store) SweepExpired(ctx context.Context, now time.Time) (int, error)

SweepExpired purges every file, in any scope, whose ExpiresAt is not after now, live or trashed, and returns how many it purged. Each file is purged in its own transaction (Purge, so plugins' hooks run and a shared object stays for its other files); a file that cannot be purged does not stop the sweep, and the returned error names each one. Schedule it; the library starts no background work.

func (*Store) SweepOrphans

func (s *Store) SweepOrphans(ctx context.Context, diskName, prefix string, olderThan time.Time) (int, error)

SweepOrphans deletes the objects under prefix on the named disk that no file row names and no ObjectOwner plugin holds, last modified before olderThan, and returns how many it deleted. Orphans are what the object-first rule can leave behind: an object whose row insert failed and whose cleanup failed too, or one whose deletion after a purge failed. Unlike upload.Sweep, which deletes by age alone, it never deletes a recorded file, so prefix may be the root ("") of a disk the Store owns.

The contract of prefix: every object under it is either a file's, a plugin's that implements ObjectOwner, or garbage. Do not point it at objects the application keeps outside records (exports, backups), or at a direct-upload area whose objects wait for a later CreateFile; for those, keep olderThan well before the oldest upload that may still be recorded. An object whose ModTime the disk does not report is never deleted, since its age is unknown.

A failed deletion does not stop the sweep; the returned error joins one error per object left. A listing or database failure stops it with the count so far, and a rerun continues.

func (*Store) SweepRetention

func (s *Store) SweepRetention(ctx context.Context, now time.Time, dryRun bool) (RetentionReport, error)

SweepRetention applies the retention rules at now, in every scope: each attachment older than its rule's MaxAge is detached, and its file is purged when no attachment is left (Purge, so plugins' hooks run and a shared object stays for its other files). Each attachment is handled in its own transaction; one that fails does not stop the sweep, and the returned error names each. With dryRun nothing changes and the report says what a real run would do, so an application can show it, or log it, before switching a new rule on. Schedule it; the library starts no background work.

func (*Store) SweepRetentionIn

func (s *Store) SweepRetentionIn(ctx context.Context, scope string, rules []Retention, now time.Time, dryRun bool) (RetentionReport, error)

SweepRetentionIn applies rules, instead of WithRetention's, to one scope only: retention that depends on the tenant ("free plan 30 days, pro 1 year"). The application decides each scope's rules from its own plans and calls it per scope, from the job that would call SweepRetention; the library keeps no table of plans. It behaves as SweepRetention in every other way (per-attachment transactions, joined errors, dryRun). Each rule is checked as WithRetention checks it, and its owner type must be registered with WithOwnerTypes; a bad rule is CodeInvalidArgument and sweeps nothing. No rules sweep nothing. WithRetention's rules still apply to every scope through SweepRetention, so a scope swept here should not also match those.

func (*Store) SweepTrash

func (s *Store) SweepTrash(ctx context.Context, cutoff time.Time) (int, error)

SweepTrash purges every file, in any scope, trashed before cutoff, and returns how many it purged. Schedule it (the library starts no background work) to give the trash a retention period. Each file is purged in its own transaction, so call it outside one. A file that cannot be purged (a database failure, or a plugin refusing, such as a legal hold) does not stop the sweep: the rest are still purged, and the returned error joins one error per failed file, naming it, so one stuck file never holds back the others. An object whose deletion fails after its row was purged goes to WithObjectErrorHandler, as with any deletion after a commit, and is left for SweepOrphans. A cancelled ctx stops the sweep. FindTrash lists what a sweep would purge without purging it.

func (*Store) TableName

func (s *Store) TableName(name string) string

TableName is the name the table or index name (a default name, such as TableNameFiles or acl.TableNameGrants) has in this Store's database: name itself unless WithTableNames maps it. Use it in SQL the application writes against the fs_ tables.

func (*Store) Trash

func (s *Store) Trash(ctx context.Context, scope, id string, opts ...ChangeOption) error

Trash moves a file to the trash: it stays stored and attached, but Attachments and Files no longer list it until Restore. A public file's object stops being reachable at its public URL while it is trashed: with WithVisibilityDisks it moves to the private disk (the row keeps its visibility, so Restore moves it back); otherwise its disk makes it private in place where it has the SetVisibility capability (a disk without it cannot hide one object, so there the object stays as it is). The object changes before the row and is put back if the transaction fails, as in SetVisibility. Trashing a trashed file does nothing. IfUnchanged trashes it only while its content is the one the caller read.

func (*Store) TrashMany

func (s *Store) TrashMany(ctx context.Context, scope string, ids []string) ([]BulkResult, error)

TrashMany trashes each file of ids, as Trash, each in its own transaction. See bulk for the rules every *Many method shares.

func (*Store) Tx

func (s *Store) Tx(ctx context.Context, fn func(ctx context.Context, tx *sql.Tx) error) error

Tx runs fn in one transaction and commits it when fn returns nil. Every Store call made with the ctx fn receives joins the transaction, and so can the application's own statements on tx. A Tx inside a Tx joins the outer one; only the outermost commits. Object deletions scheduled by Purge run after the commit, never before.

func (*Store) URL

func (s *Store) URL(ctx context.Context, f File, ttl time.Duration, opts ...LinkOption) (string, error)

URL returns a link to the file: its public URL when it is public and the disk can build one, otherwise a signed GET valid for ttl. Every link it issues is reported with Served (ServeLink unless ServedAs says otherwise, Bytes the file's Size) before it is returned, and none is returned when that fails, so every route, listing and plugin that hands out links is metered without remembering to be. A public URL never expires, so for a public file this counts issuance, not fetches.

func (*Store) Upload

func (s *Store) Upload(ctx context.Context, scope, diskName string, body io.Reader, filename string, p upload.Policy, opts ...UploadOption) (File, error)

Upload stores body on the named disk under policy p (see upload.Receive) and records it. If the row cannot be inserted, or an enclosing Tx later rolls back, the stored object is deleted again, so a failed upload leaves neither a row nor bytes. An empty diskName chooses the disk by the policy's visibility (WithVisibilityDisks). Options reuse an existing object with the same content (Dedup) or make the file temporary (Expires, ViewOnce).

func (*Store) UsagesOf

func (s *Store) UsagesOf(ctx context.Context, uq UsageQuery) (UsagePage, error)

UsagesOf lists where a file is used (every attachment of it: owner and role), newest first with keyset pagination on (created_at, id), so a "used in 3 posts" panel or a check before purging a shared asset reads it a page at a time however widely the file is used. It reads the fs_attachments_file index. A file that does not exist, or is attached nowhere, has an empty page.

func (*Store) Viewed

func (s *Store) Viewed(ctx context.Context, scope, id string) (bool, error)

Viewed is what a route serving a file asks before it serves it, and counts the view: it returns false for an expired file, true for any other file, and for a view-once file true exactly once, setting its ExpiresAt to now at that first view so SweepExpired purges it and later views are refused. Two concurrent first views of a view-once file get one true between them: the expiry is set by one conditional UPDATE. A missing file is CodeNotFound; a quarantined one is false with ErrQuarantined, and a view-once one spends no view.

type Table

type Table struct {
	Table      sqlb.Table
	Columns    []ddl.ColumnSpec
	PrimaryKey []ddl.ColumnRef
	Indexes    []Index
}

Table describes a table for CreateTable: its columns, its primary key and its indexes.

type TableNamer

type TableNamer func(name string) string

TableNamer maps the name of a table or index of this module or a plugin (always starting with ObjectPrefix: "fs_files", "fs_acl_grants") to the name it has in the application's database. It must be pure and injective (two names never map to one), and return an identifier of letters, digits and "_" not starting with a digit, of at most MaxObjectNameBytes; Migrate checks every name it creates against these rules (CodeInvalidArgument).

type Text

type Text struct {
	Alt     string
	Caption string
	Title   string
}

Text is a file's describing text: Alt for screen readers and broken images, Caption shown under it, Title as its display heading. "" means none. Caption may hold line breaks; Alt and Title are one line.

type UploadOption

type UploadOption func(*uploadOptions)

UploadOption changes how Upload records a file. Without any, Upload stores and records every body as its own object.

func Dedup

func Dedup() UploadOption

Dedup makes Upload reuse an existing object with the same content: when a live file of the same scope, on the same disk and with the same visibility, already has the body's SHA-256, the new file is recorded on that file's object and the uploaded copy is deleted after the commit. The two files then share one object (see References); each keeps its own row, name, metadata and attachments, and the object is deleted only when the last file naming it is purged. The body is still received and hashed in full (the digest is known only at the end), so Dedup saves storage, not upload bandwidth. Reuse never crosses scopes, so one tenant's upload never reveals or depends on another tenant's object.

func Expires

func Expires(at time.Time) UploadOption

Expires makes the uploaded file temporary: NewFile.ExpiresAt.

func Quarantine

func Quarantine() UploadOption

Quarantine makes Upload record the file quarantined (StatusQuarantined) until a scan passes; see PassScan, FailScan and Scan. The policy's visibility must not be public (ErrPublicQuarantine).

func ViewOnce

func ViewOnce() UploadOption

ViewOnce makes the uploaded file expire at its first view: NewFile.ViewOnce.

type UsageHooks

type UsageHooks interface {
	OnHold(ctx context.Context, s *Store, o HeldObject) error
	OnRelease(ctx context.Context, s *Store, o HeldObject) error
}

UsageHooks is a plugin told of every object another plugin holds or releases. Both run inside the holder's transaction (use Store.Conn for statements), so a counter changes with the holder's rows or not at all. An OnHold error, such as CodeQuotaExceeded, aborts the holder's change; an OnRelease error aborts it too, so a release is never half counted.

type UsagePage

type UsagePage struct {
	Usages     []Attachment
	NextCursor string
}

UsagePage is one page of a file's attachments, newest first.

type UsageQuery

type UsageQuery struct {
	Scope  string
	FileID string
	// Limit caps the page; 0 means DefaultPageSize, at most MaxPageSize.
	Limit int
	// Cursor continues from UsagePage.NextCursor of the same query.
	Cursor string
}

UsageQuery selects a page of the places one file is used: its attachments, under any owner and role.

type Versioner

type Versioner interface {
	KeepVersion(ctx context.Context, s *Store, previous File) (bool, error)
}

Versioner keeps replaced content (the versions plugin). KeepVersion runs inside the replacement's transaction with the file as it was. Returning true takes ownership of that content's object: the Store does not delete it, and the plugin deletes it when it drops the version and before the file is purged.

Directories

Path Synopsis
Package acl is a records plugin for permissions on files and folders: grants of a level (read, write, admin) to a subject on one file or on a folder, inherited by every file in the folder and its subfolders, and a Can check that resolves them.
Package acl is a records plugin for permissions on files and folders: grants of a level (read, write, admin) to a subject on one file or on a folder, inherited by every file in the folder and its subfolders, and a Can check that resolves them.
Package audit is a records plugin that keeps an append-only log of every change to files and attachments, and of every change other plugins report with records.Store.Emit (folders, tags, share links, with their Subject and Detail): who did what to which file, and when.
Package audit is a records plugin that keeps an append-only log of every change to files and attachments, and of every change other plugins report with records.Store.Emit (folders, tags, share links, with their Subject and Detail): who did what to which file, and when.
Package egress is a records plugin that meters the bytes each scope serves, in daily buckets, from the reads handlers report with records.Store.Served (recordshttp's file route, share links, image variants), so a plan's bandwidth can be shown, limited by the application and billed.
Package egress is a records plugin that meters the bytes each scope serves, in daily buckets, from the reads handlers report with records.Store.Served (recordshttp's file route, share links, image variants), so a plan's bandwidth can be shown, limited by the application and billed.
Package folders is a records plugin for virtual folders: a tree of named folders per scope, and a place in it for every file.
Package folders is a records plugin for virtual folders: a tree of named folders per scope, and a place in it for every file.
internal
testdb
Package testdb is the records module's own name for recordstest, the exported engine-aware test helper: it keeps the module's tests' imports short and stable while the implementation lives in one place.
Package testdb is the records module's own name for recordstest, the exported engine-aware test helper: it keeps the module's tests' imports short and stable while the implementation lives in one place.
Package locks is a records plugin for check-out: a subject locks a file for a while, and until it unlocks it or the lock expires, nobody else may replace its content (records.Store.Replace and ReplaceObject), trash it or purge it: each is refused with ErrLocked, and an administrator Breaks the lock first.
Package locks is a records plugin for check-out: a subject locks a file for a while, and until it unlocks it or the lock expires, nobody else may replace its content (records.Store.Replace and ReplaceObject), trash it or purge it: each is refused with ErrLocked, and an administrator Breaks the lock first.
Package quota is a records plugin that limits how much each scope (a tenant, a workspace, a user) may store, in bytes and in files, and keeps the usage that billing pages and admin screens show.
Package quota is a records plugin that limits how much each scope (a tenant, a workspace, a user) may store, in bytes and in files, and keeps the usage that billing pages and admin screens show.
Package recordshttp serves the records layer over net/http: upload files and attach them to an owner, list an owner's files, reorder and detach them, make an existing file an owner's only one, redirect to a file, replace it, change its visibility, trash, restore and purge it.
Package recordshttp serves the records layer over net/http: upload files and attach them to an owner, list an owner's files, reorder and detach them, make an existing file an owner's only one, redirect to a file, replace it, change its visibility, trash, restore and purge it.
Package recordstest is a test helper: it opens a fresh, empty database for one test on the engine the environment selects, so the records module's tests, its plugins' tests and any other module's tests that run SQL against the records tables (contrib/imaging, an application's own plugin) run unchanged on SQLite, Postgres and MySQL.
Package recordstest is a test helper: it opens a fresh, empty database for one test on the engine the environment selects, so the records module's tests, its plugins' tests and any other module's tests that run SQL against the records tables (contrib/imaging, an application's own plugin) run unchanged on SQLite, Postgres and MySQL.
Package share is a records plugin for share links: a URL that lets anyone holding it download one file, until it expires, is revoked, or has been used a set number of times.
Package share is a records plugin for share links: a URL that lets anyone holding it download one file, until it expires, is revoked, or has been used a set number of times.
Package tags is a records plugin for tagging files: free-form labels per scope, files listed by tag, and a tag cloud with counts.
Package tags is a records plugin for tagging files: free-form labels per scope, files listed by tag, and a tag cloud with counts.
Package uploadsessions is a records plugin that keeps upload sessions (upload.Session, UP-15) in the database: the SessionStore to use when sessions must survive a restart or be shared by several instances behind a load balancer.
Package uploadsessions is a records plugin that keeps upload sessions (upload.Session, UP-15) in the database: the SessionStore to use when sessions must survive a restart or be shared by several instances behind a load balancer.
Package versions is a records plugin that keeps the previous contents of a file when it is replaced (records.Store.Replace), lists them, and restores one: the version history of document managers and drives.
Package versions is a records plugin that keeps the previous contents of a file when it is replaced (records.Store.Replace), lists them, and restores one: the version history of document managers and drives.
Package webhooks is a records plugin that delivers chosen events (file changes, plugin events such as quota.usage_changed or egress.daily_total) to HTTPS endpoints registered per scope (or for every scope, Register with Global), signed with HMAC-SHA256 in the Standard Webhooks format (or by a Signer of the application's), retried with exponential backoff, and refused for any address outside the public internet.
Package webhooks is a records plugin that delivers chosen events (file changes, plugin events such as quota.usage_changed or egress.daily_total) to HTTPS endpoints registered per scope (or for every scope, Register with Global), signed with HMAC-SHA256 in the Standard Webhooks format (or by a Signer of the application's), retried with exponential backoff, and refused for any address outside the public internet.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL