filesystem

package module
v0.0.0-...-3ac6500 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 24, 2026 License: Apache-2.0 Imports: 33 Imported by: 0

README

ubgo/filesystem

Postgres, MySQL and SQLite

One storage library for Go applications: a small portable API over local disk, memory and every S3-compatible store, with the parts every product builds on top of files already built: safe uploads, signed URLs, files as database rows attached to your models, and test fakes.

mux := filesystem.NewMux()
mux.Register(s3.Scheme, s3.Opener)

disk, _ := mux.Open(ctx, "media", "s3://my-bucket?region=auto&endpoint=https://<account>.r2.cloudflarestorage.com")
entry, _ := disk.Write(ctx, "avatars/u1.png", body, filesystem.WriteOptions{})
link, _ := disk.SignedURL(ctx, "avatars/u1.png", filesystem.PresignOptions{Method: filesystem.MethodGet, Expires: 5 * time.Minute})

Switching from local disk in development to S3, R2 or SeaweedFS in production is a DSN change. Nothing else in the application moves. mux.OpenManager(ctx, "STORAGE_", os.LookupEnv) builds every disk from the environment (STORAGE_DISKS=media,backups, STORAGE_MEDIA_DSN=…, STORAGE_DEFAULT_DISK=media), and its errors name the variable, never the DSN, which usually holds credentials.

Status: pre-release. The API may still change. Every library package is at 94 to 100% statement coverage (COVERAGE.md lists every uncovered line and why). The S3 driver passes the conformance suite against a real SeaweedFS server, the Azure driver against Azurite, and the records module on SQLite, Postgres and MySQL, where every operation is also failed at each of its database steps in turn to prove it leaves nothing half done.

Guides

The guides start with a learning path of twelve ordered steps, a start-from-your-goal table and choosing guides (which adapter, which upload method, how to serve files, whether you need records, which plugins for which app), and the feature index links every feature to the guide that teaches it and the example that runs it. They cover everything from a first disk to production: install, your first app, production, the concepts, every backend, files as records and each plugin, recipes for avatars, galleries, direct and resumable uploads, private downloads, tenants, account deletion and more, integrations with net/http, chi, gin, echo, fiber and OpenTelemetry, extending and the reference. Every Go snippet in them is copied from an example program in examples/ that the test suite compiles and runs.

Why

Most Go products that store files end up writing their own storage layer: drivers, signed links, uploads, files as database rows, scanning, variants and cleanup jobs. This library is that layer as one set of packages, built to be imported rather than rewritten.

How it fits together

The layers of ubgo/filesystem from your app down to the adapters, with the numbered upload path (object stored first, records row saved after) and the download path back up

The picture follows one upload from start to finish; the sections below zoom into each layer.

The layers

An application holds *filesystem.Disk values, usually from a Manager it built by hand or from DSNs with the Mux. A Disk does the portable work (key cleaning, content sniffing, hashing, size caps, error codes, URLs) and runs every operation through its interceptors before it reaches the one Driver that talks to a backend. Everything else sits beside the Disk and uses it: upload decides what bytes may arrive, fshttp serves them, scan inspects them, and the records module remembers them as rows.

flowchart TB
    App["Your application<br/>(handlers, services, jobs)"]

    subgraph Edge["Getting bytes in and out"]
        Upload["upload<br/>policies, form uploads, presign + Confirm,<br/>Uppy multipart, FromURL, Process, Inspect"]
        Serve["fshttp<br/>signed and public downloads,<br/>ranges, ETag, active content as attachment"]
        Scan["scan<br/>clamd and any Scanner"]
        Tus["contrib/tus<br/>resumable uploads"]
    end

    subgraph Records["records module (files as rows on ubgo/sqlb)"]
        Store["records.Store<br/>files, attachments, trash, replace,<br/>visibility, events, migrations"]
        Plugins["plugins: audit, folders, quota, share,<br/>tags, versions, variants, dimensions"]
        RecHTTP["recordshttp<br/>HTTP endpoints"]
    end

    subgraph Core["core module (standard library only)"]
        Manager["Manager / Mux<br/>named disks, DSNs, environment"]
        Disk["Disk<br/>keys, sniffing, hashing, size caps,<br/>errors, signed and public URLs"]
        Chain["Interceptors<br/>Retry, Timeout, Limit, Log, Observe, Guard,<br/>contrib/otel"]
        Wrappers["Scoped and mirrored disks"]
    end

    subgraph Drivers["Driver implementations"]
        Local["local"]
        Memory["memory"]
        S3["adapters/s3<br/>AWS, R2, B2, Wasabi, Spaces,<br/>SeaweedFS, Garage, MinIO"]
        GCS["adapters/gcs"]
        Azure["adapters/azure"]
        SFTP["adapters/sftp"]
    end

    App --> Upload & Serve & Store & RecHTTP & Manager
    Tus --> Upload
    Upload --> Scan
    RecHTTP --> Store
    Store --> Plugins
    Store --> Upload
    Upload & Serve & Store --> Disk
    Manager --> Disk
    Disk --> Chain --> Wrappers --> Local & Memory & S3 & GCS & Azure & SFTP
The modules

Each box is a Go module, so an application downloads only what it imports. Arrows point at what a module requires. The core requires nothing outside the standard library, and a test fails the build if that changes.

flowchart LR
    core["github.com/ubgo/filesystem<br/>core: Disk, Driver, Manager, Mux, local, memory,<br/>layer, sniff, signedurl, fshttp, upload, scan,<br/>filesystemtest"]
    s3["adapters/s3"] --> core
    gcs["adapters/gcs"] --> core
    azure["adapters/azure"] --> core
    sftp["adapters/sftp"] --> core
    records["records<br/>+ audit, folders, quota, share, tags,<br/>versions, recordshttp"] --> core
    records --> sqlb["github.com/ubgo/sqlb"]
    imaging["contrib/imaging<br/>+ variants, dimensions"] --> core
    imaging --> records
    codecs["contrib/imaging/webp, avif, heic<br/>(pure Go by default)"] --> imaging
    media["contrib/video, pdf<br/>(ffmpeg, Poppler)"] --> records
    svg["contrib/svg"] --> core
    otel["contrib/otel"] --> core
    tus["contrib/tus"] --> core
    frameworks["contrib/gin, echo, fiber, chi,<br/>gqlgen"] --> records
    orm["contrib/ent, gorm, sqlc<br/>read helpers"] --> records
    diskjobs["jobs/diskjobs<br/>scheduler, disk-only jobs"] --> core
    jobs["jobs<br/>records jobs"] --> records
    jobs --> diskjobs
    queues["jobs/river, jobs/asynq"] --> diskjobs
    fs["cmd/fs"] --> s3 & gcs & azure & sftp
One operation through a Disk

Every Disk method runs the same way, whichever backend is behind it. Capabilities decide between the driver's native call, a portable fallback and CodeUnsupported, so an option is never silently dropped. Interceptors wrap the operation, not the driver, so they cannot hide a capability; the application chooses them and their order (the diagram shows the set from the interceptors example).

flowchart LR
    Call["disk.Write(ctx, key, body, opts)"] --> Clean["CleanKey<br/>traversal, control chars,<br/>length refused"]
    Clean --> Check["capability check<br/>native, fallback, or CodeUnsupported"]
    Check --> I1["Observe"] --> I2["Log"] --> I3["Retry<br/>(rewinds seekable bodies)"] --> I4["Timeout"] --> I5["Limit"]
    I5 --> Driver["Driver.Write"]
    Driver --> Count["count, hash, cap at MaxSize,<br/>sniff the content type"]
    Count --> Result["Entry or *Error with a Code<br/>(not_found, too_large, unavailable, …)"]
An upload, end to end

A server-side upload from a form, through the records module. The object is written before its row and deleted again if the transaction rolls back, so bytes and rows never drift apart; plugins take part inside the same transaction.

sequenceDiagram
    autonumber
    participant B as Browser
    participant H as upload.Handler or recordshttp
    participant P as upload.Policy
    participant D as Disk (+ interceptors)
    participant S as records.Store
    participant G as Plugins (quota, audit, dimensions, variants)
    B->>H: POST multipart/form-data
    H->>P: size cap, sniff type from bytes, name rules
    P-->>H: refused: too_large / rejected (nothing stored)
    H->>P: Process (e.g. imaging.Sanitize strips EXIF)
    H->>D: Write object (object first)
    H->>P: Inspect (e.g. scan.Clamd), a failure deletes the object
    H->>S: CreateFile + Attach in one transaction
    S->>G: BeforeCreate (quota charges, dimensions measure)
    S->>G: events: file.created, attachment.created (audit logs)
    S-->>H: rollback deletes the stored object
    S->>G: after commit: eager variants generated
    H-->>B: 201 with the stored files

Direct-to-bucket uploads skip the application for the bytes and check them afterwards:

sequenceDiagram
    autonumber
    participant B as Browser or mobile app
    participant A as Your app (upload.Presign / upload.Uppy)
    participant O as Object store (S3, GCS, Azure)
    participant S as records.Store
    B->>A: ask for a ticket (name, type, size)
    A->>A: policy checks the declared type, size and name
    A-->>B: signed PUT URL (expires) or presigned parts
    B->>O: PUT the bytes
    B->>A: done
    A->>O: Confirm: re-read, sniff, size, hash, inspect
    O-->>A: refused objects are deleted
    A->>S: CreateFile + Attach
A file's life
stateDiagram-v2
    [*] --> Live: Upload / CreateFile
    Live --> Live: Attach, Reorder, Detach, SetAttachment
    Live --> Live: Replace (versions keeps the old content, variants regenerate)
    Live --> Live: SetVisibility (moves between public and private disks)
    Live --> Trashed: Trash (public objects hidden, variants dropped)
    Trashed --> Live: Restore
    Trashed --> Purged: Purge / SweepTrash / PurgeOwner
    Live --> Purged: Purge
    Purged --> [*]: row deleted, then objects deleted after commit
Where plugins plug in

A records plugin is any value with a Name; each capability is an optional interface the Store detects, so a plugin implements only what it needs.

Interface Called Used by
Migrator Store.Migrate, for the plugin's own tables audit, folders, quota, share, tags, versions, variants
FileHooks before a file row is inserted, after it, before a purge quota, dimensions, share, tags, folders, versions, variants
ReplaceHooks before new content replaces a file's quota, dimensions
Versioner when content is replaced: keep the old object or let it go versions
EventHooks after every change, inside its transaction audit, variants, versions
ObjectOwner when SweepOrphans asks which objects a plugin owns versions, variants, video, pdf, ocr
FileDiskMover when MoveFile moves a file to another disk, for the plugin's own objects versions, variants, video, pdf, ocr
DiskRenamer when RenameDisk renames a disk the plugin recorded versions, variants, video, pdf, ocr

Packages

Package Module What it does
filesystem core Disk, the Driver contract, capabilities, interceptors, Manager, scoped and mirrored disks, DSN Mux, typed errors, Disk.FS (a read-only io/fs.FS view), ParseSize, directories (MakeDir, DirExists), the abandoned-upload sweep and its dry run (FindAbandoned)
filesystem/local core Local directory driver: os.Root sandboxing, atomic and durable writes (WithDurability), visibility as file modes, multipart, real directories
filesystem/memory core In-memory driver with every capability that needs no network
filesystem/layer core Interceptors: Retry, Timeout, Limit, Bandwidth, Hedge, Log (slog), Observe (metrics hook), Guard, Fault (injected failures for tests)
filesystem/sniff core Content-type detection from bytes, with safe refinement by extension, and a Detector hook for formats it does not know (DICOM, STL, Parquet) that can never relabel a known type
filesystem/scan core Malware scanning: a Scanner contract, a clamd (ClamAV) client over TCP or a Unix socket, and Inspect to scan every upload before it is accepted
filesystem/signedurl core HMAC-signed URLs with key rotation, for disks whose backend cannot sign
filesystem/fshttp core net/http handlers for signed downloads and uploads (ranges, ETag, 304) and public files, Cache-Control by handler, ServeEntry (the same serve path for your own handler after its access check), ServeZip (many files streamed as one zip), ClientMessage (client-safe error text)
filesystem/upload core Upload policies, streaming form uploads, presigned uploads with server-side confirmation, Uppy S3 multipart endpoints, imports from a URL with an SSRF guard, sweeping of abandoned uploads, upload sessions (a server-issued token for one key or prefix, valid for a window), public direct uploads where the policy allows (AllowPublic), your own success envelope (Respond)
filesystem/filesystemtest core Conformance suite for drivers, and a Fake disk with assertions for application tests
filesystem/adapters/s3 own module S3 and every S3-compatible store (R2, B2, Wasabi, Spaces, SeaweedFS, Garage, MinIO), on aws-sdk-go-v2 and its transfer manager; a customer's bucket through an assumed IAM role (role_arn), a KMS key per tenant (Config.SSEKMSKeyFor), prefix-scoped delegated credentials (DelegatePrefix) in any partition and on SSE-KMS buckets
filesystem/adapters/gcs own module Google Cloud Storage: V4 signed URLs, optional fine-grained ACLs, resumable uploads, a service account per disk (credentials_file, credentials_env), a KMS key or storage class per call (WithRequestOptions)
filesystem/adapters/azure own module Azure Blob Storage and Azurite: SAS URLs, block-based multipart, atomic commits, configurable block size and concurrency (block_size, concurrency) for blobs up to 190 TiB, an encryption scope, tier, tags or hold per call (WithRequestOptions), an endpoint behind a custom domain or Front Door (AccountName)
filesystem/adapters/sftp own module SFTP servers: atomic writes via posix-rename, conditional writes via hard links, known_hosts verification, ssh-agent and encrypted keys
filesystem/adapters/ftp own module FTP and FTPS servers (explicit or implicit, a private CA with ca_file): a connection pool and atomic writes through a staged upload and a rename
filesystem/adapters/webdav own module WebDAV servers over net/http (Nextcloud, ownCloud, many NAS): staged PUT then MOVE
filesystem/adapters/sqlblob own module Objects as rows of a table on SQLite, Postgres or MySQL through ubgo/sqlb: ranged reads, compare-and-swap, a size cap
filesystem/adapters/gdrive, dropbox, onedrive own modules Consumer drives over their REST APIs with the app's OAuth token; honest capabilities (no presign)
filesystem/adapters/vercelblob, cloudinary own modules Hosted storage APIs: Vercel Blob (compare-and-swap, batch delete) and Cloudinary raw resources (signed reads)
filesystem/records own module Files as rows (SQLite, Postgres, MySQL via ubgo/sqlb), polymorphic attachments, trash and purge, quarantine until scanned (PassScan, FailScan, Scan), migrations, plugins, your own table names (WithTablePrefix, WithTableNames)
filesystem/records/audit records Append-only log of every file and attachment change with its actor, written in the change's own transaction; per-file history, per-actor activity, retention, an HMAC hash chain with key rotation (WithHashChainKeyring), the request's source (WithSource)
filesystem/records/folders records Virtual folders per scope: create, rename, move (cycle- and depth-checked), delete when empty, file placement, paginated folder listings, case- or Unicode-insensitive sibling names (WithNameKey), one file shown in several folders (WithShortcuts)
filesystem/records/quota records Per-scope byte and file limits, charged race-free in the upload's transaction, plus counters of your own beside the scope's (an organisation's total, a content class) through WithDimensions
filesystem/records/egress records Bytes served per scope per UTC day (or your own buckets, hourly or in each tenant's time zone, with WithBucket), optionally broken down per share link or other subject (WithPerSubject), reported as egress.daily_total events with stable IDs for billing
filesystem/records/webhooks records Outbound webhooks: an outbox written in the change's own transaction, Standard Webhooks HMAC signatures (or your own Signer), secret rotation, global endpoints for every scope, your own payload shape, retries with backoff, an SSRF guard
filesystem/records/recordshttp records HTTP endpoints for attachments: upload and attach (all or nothing), list, reorder, detach, redirect to a file, replace its content, trash; authorization through two required hooks; the disk fixed, chosen per request (DiskFor) or by visibility; your own success envelope (Respond)
filesystem/records/tags records Tags on files: normalized labels, files by tag (all of several), tag counts, rename and delete across a scope
filesystem/records/versions records Previous contents of replaced files: history per file with a cap or a per-file policy (WithKeepFunc, Trim), restore (itself undoable), delete
filesystem/records/share records Expiring, download-limited share links with hashed tokens and a redirecting handler; your own access check, delivery page, error pages, success envelope (WithRespond) and per-request upload policy
filesystem/records/acl records Read, write and admin grants (or your own levels, WithLevels) on files, folders and your own containers (WithContainers), inherited down a folder tree, optionally expiring, deny grants (WithDeny), with an authorizer for recordshttp
filesystem/records/uploadsessions records Upload sessions kept in the database (only a hash of each token's secret), so they survive a restart and are shared by every instance
filesystem/records/locks records Check-out locks with expiry: only the holder may replace, trash or purge a locked file until it unlocks, the lock expires or it is broken; WebDAV-style lock tokens (TokenHolder); shared locks with several holders (WithSharedLocks)
filesystem/contrib/imaging own module Safe pure-Go resizing (pixel-bomb guard, EXIF orientation, re-encoding), fit, fill (with gravity), pad and user-chosen crops, header-only Probe, Sanitize (an upload processor that strips EXIF and GPS), and records plugins for named, cached image variants (generated on first request, or eagerly after an upload commits, with 2x/3x densities and their srcset) and for stored image dimensions; variants on any disk (WithLocation)
filesystem/contrib/otel own module OpenTelemetry tracing and metrics interceptors
filesystem/contrib/keynorm own module Unicode NFC key normalisation for WithKeyNormalizer, so composed and decomposed spellings name one object
filesystem/contrib/imaging/placeholder contrib/imaging Generated fallback avatars: initials or an identicon, with a cache-friendly handler
filesystem/contrib/imaging/webp, avif, heic own modules WebP and AVIF encoding and decoding, HEIC decoding, behind the imaging processor; pure Go by default, a system library when one is installed (-tags nodynamic forces pure Go)
filesystem/contrib/imaging/icc own module ICC colour profiles: parsing and conversion to sRGB, standard library only
filesystem/contrib/video own module Video metadata, poster frames and metadata stripping through ffprobe and ffmpeg, plus transcoding to a browser-ready MP4 and storyboards with WebVTT, as upload hooks and a records plugin; posters on any disk (WithLocation); an HLS ladder for adaptive streaming (Tool.HLS)
filesystem/contrib/audio own module Audio metadata, waveforms and Opus or AAC transcoding through ffprobe and ffmpeg, as upload hooks and a records plugin
filesystem/contrib/videoplatform own module Hand streaming to Mux or Cloudflare Stream: a records plugin that publishes a video, tracks its state from signed webhooks and deletes the platform copy after the file goes
filesystem/contrib/pdf own module PDF page count, page size and first-page previews through Poppler, as upload hooks and a records plugin; previews and text on any disk (WithLocation)
filesystem/contrib/office, ocr, vision own modules Office documents to PDF through sandboxed LibreOffice, OCR through Tesseract, and auto-tagging through a Labeler (Google Cloud Vision)
filesystem/contrib/pickers own module Imports a file picked in Google Drive, Dropbox or OneDrive through upload.Receive, the token sent only to the source's own hosts
filesystem/contrib/svg own module An allow-list SVG sanitiser and an upload policy that accepts SVG safely
filesystem/contrib/svg/raster own module SVG rasterising to an image, bounded in size and concurrent renders
filesystem/contrib/archive own module Zip and tar listing and extraction: limits enforced while decompressing, every name cleaned, links refused, an upload inspector that denies executables
filesystem/contrib/crypt own module A wrapper disk that encrypts at rest: chunked AES-GCM with ranged reads, data keys wrapped by a keyring or KMS, per-tenant keys and rotation
filesystem/contrib/gin, echo, fiber, chi own modules Framework adapters: mount the handlers, actor and tenant middleware, client-safe errors, zip downloads
filesystem/contrib/gqlgen own module GraphQL uploads: the Upload scalar through an upload policy, records, a safe error presenter
filesystem/contrib/ent, gorm, sqlc own modules Read helpers over the records tables for apps on ent, GORM or sqlc; writes stay with the Store; each follows a Store's table names (ent Names, fsgorm.NewNamer, sqlc.FSWith)
filesystem/jobs own module The maintenance jobs (sweeps, recounts, prunes, promote) and an in-process scheduler
filesystem/jobs/diskjobs own module The jobs that need no database (UploadSweep, SweepAbandoned, Promote) and the scheduler, importing the core module only
filesystem/jobs/river, jobs/asynq own modules The same jobs on River (Postgres) or Asynq (Redis), for several instances
filesystem/contrib/tus own module Resumable uploads (tus protocol) by embedding tusd, promoted through the same upload policy, optionally narrowed by an upload session (Config.Resolve); a shared session store and locker for several instances (Config.Store, Config.Locker), CORS policy (Config.Cors)
filesystem/contrib/cloudfront own module CloudFront signed URLs and signed cookies as a filesystem.URLSigner, keys redacted; signs downloads even on an S3 disk that presigns its own
filesystem/contrib/cloudflare own module Cloudflare token-authentication signed URLs, standard library only
filesystem/contrib/remote own module Remote media: link previews (with oEmbed from providers you allow), an HMAC-signed image proxy and a capped media cache, every fetch through the SSRF guard
filesystem/contrib/notify (+ eventgrid, sqs, pubsub) own modules Bucket events turned into callbacks: Azure Event Grid (authenticated handshake), S3/R2 through SQS, GCS through Pub/Sub pull; a bucket allowlist on each
filesystem/cmd/fs own module A command-line tool: ls, stat, cat, put, cp and mv across backends, rm, url, check, with -json

The core module depends on the standard library only, and a test enforces this. Anything that needs a third-party library is its own module, so you download only what you import.

Capabilities, never lies

Some backends can do things others cannot: presigned URLs, server-side copy, per-object ACLs, multipart uploads, stored metadata, conditional writes. A driver declares what it supports in one Capabilities struct. NewDisk refuses a driver that declares an interface it doesn't implement, and the conformance suite exercises every declared capability against the real backend. A Disk method whose capability is missing either falls back to a portable implementation (Copy, Move, DeletePrefix) or returns an error with CodeUnsupported. It never silently drops your option.

Capability local memory s3 gcs azure sftp
Server-side copy streams yes yes yes yes streams
Atomic move yes yes copy + delete copy + delete copy + delete yes
Per-object visibility file mode yes acl=true (AWS) acl=true no (container level) file mode
Bulk prefix delete yes yes yes list + delete list + delete yes
Batch delete of keys one by one yes 1,000 per request one by one one by one one by one
Signed URLs via signedurl + fshttp via signedurl SigV4 V4 with a key SAS with a shared key via signedurl + fshttp
Multipart (server) yes yes yes no (resumable in Write) yes no
Presigned parts (browser) no no yes no no no
Sweep of abandoned uploads (SweepAbandoned) yes yes opt-in (sweep_uploads=true); a lifecycle rule is preferred no (sessions expire) yes yes
Its dry run (FindAbandoned) yes yes with sweep_uploads=true no yes yes
Real directories (MakeDir, DirExists) yes yes no (a directory exists while a key is under it) no no yes
Stored metadata / content type no / no yes / yes yes / yes yes / yes yes / yes no / no
Stored HTTP headers (Cache-Control, Content-Disposition, Content-Encoding) no yes yes yes yes no
Presigned PUT pins stored headers (PresignHeaders) no (stores no headers) via signedurl + fshttp yes (presign_headers=true, the default) yes, with a signing account no (SAS cannot pin them) no
Presigned upload pins visibility (PresignVisibility) no no acl=true with signed headers checked acl=true, with a signing account no no
Presigned POST form uploads (PresignPost) no no yes yes, with a signing key or IAM no no
Backend-verified SHA-256 (ChecksumSHA256; the Disk verifies on every backend) no no yes no no no
Conditional write yes yes conditional_write=true yes yes with hard links

Interceptors

Retry, timeouts, concurrency limits, logging, metrics and policy checks run around every operation without wrapping the driver, so they can never hide a capability. This is the usual failure of Go decorators, and this design avoids it.

disk, _ := filesystem.NewDisk("media", drv, filesystem.WithInterceptors(
	layer.Observe(recordMetrics),
	layer.Log(logger),
	layer.Retry(layer.RetryOptions{}),
	layer.Timeout(10*time.Second),
	layer.Limit(32),
))

A retried write rewinds its body when the body is seekable, and is never retried when it is a one-shot stream.

Uploads

policy := upload.Policy{MaxSize: 10 << 20, Allowed: []string{"image/*", "application/pdf"}}
http.Handle("/upload", upload.Handler(disk, policy, upload.HandlerOptions{MaxFiles: 5}))

The type is detected from the content, never taken from the client's header or file name. HTML, SVG, XML, executables and script file names (.php, also hidden as photo.php.jpg) are refused unless explicitly allowed, and fshttp and signed downloads serve active content as an attachment, never inline. Size is counted while streaming, and nothing larger than the limit is ever stored. A request stores all its files or none of them; with HandlerOptions{Partial: true} a bulk upload stores every file it can and lists the others, each with its own error code. Direct-to-bucket uploads use upload.Presign followed by upload.Confirm, which re-checks the stored bytes. Browser multipart uploads use upload.Uppy, which speaks the protocol of Uppy's AWS S3 plugin. Policy.Process transforms each file before it is stored; imaging.Sanitize re-encodes photos so the GPS position, camera details and anything smuggled after the image never reach storage. Policy.Inspect runs on every stored upload before success is reported, and its error deletes the file: scan.Inspect(scan.Clamd{Network: "tcp", Address: "127.0.0.1:3310"}) scans every upload path with ClamAV, failing closed when the scanner is down. Policy.Extensions limits file-name extensions and Policy.MatchExtension refuses a file whose name claims another type than its bytes (a PNG named invoice.pdf); upload.Inspectors chains several inspections. Imports from a URL use upload.FromURL, which goes through the same policy and refuses any connection to a non-public address (loopback, private networks, cloud metadata) at dial time, so DNS rebinding and redirects cannot reach inside.

Files as records

store, _ := records.New(db, sqlb.PostgresDialect{}, manager, records.WithOwnerTypes("user", "post"))
store.Migrate(ctx)

f, _ := store.Upload(ctx, tenantID, "media", body, filename, policy)
store.Attach(ctx, tenantID, f.ID, records.Owner{Type: "post", ID: postID}, "gallery")
gallery, _ := store.Attachments(ctx, tenantID, records.Owner{Type: "post", ID: postID}, "gallery")

records.WithSingleRoles("avatar") lets the database hold one file per owner in a role, even under concurrent requests; store.SetAttachment replaces it and keeps, trashes or purges the previous file, and store.PurgeOwner deletes what a deleted user or post leaves behind (a file another owner still uses is only detached). records.WithVisibilityDisks("cdn", "vault") keeps public files on a public bucket behind a CDN and private ones on another, and store.SetVisibility moves a file between them with its versions, regenerating its thumbnails, so nothing of a file made private stays public. records.WithActor(ctx, userID), set once in the authentication middleware, is recorded as each file's UploadedBy and logged by the audit plugin. store.Replace gives a file new content and keeps its ID, so its attachments, folder, tags and links stay; the old object is deleted after the commit unless the versions plugin keeps it, quota charges the difference, and image variants are regenerated. Bytes and rows stay consistent without a shared transaction. Objects are written before rows and deleted only after the row deletion commits. A rolled-back upload deletes its object. Plugins add their own tables (with migrations) and lifecycle hooks. An EventHooks plugin hears of every change (created, trashed, restored, purged, attached, detached, reordered, released from quarantine) inside the change's transaction, which is how audit never logs a change that rolled back; search indexing and webhooks plug in the same way.

recordshttp.Handler serves these operations over net/http for any router. The library knows nothing about users: Scope maps a request to its tenant and Authorize decides each action, and both are required.

h, _ := recordshttp.Handler(recordshttp.Config{
	Store: store, Disk: "media", Policy: policy,
	Scope:     func(r *http.Request) (string, error) { return tenantOf(r) },
	Authorize: func(r *http.Request, a recordshttp.Action, t recordshttp.Target) error { return can(r, a, t) },
})
mux.Handle("/media/", http.StripPrefix("/media", h))

Examples

examples/drive is a team drive in about 300 lines: workspaces with a quota, files attached to projects, folders, tags, version history with restore, expiring share links, an activity feed and image dimensions, all plugins of one Store, with a test that walks a file through its whole life over HTTP. examples/mediaserver is the smaller avatar flow written by hand, and examples/basics the Disk API alone.

Testing your code

disk := filesystemtest.NewFake(t, "media")
service := NewAvatarService(disk)
service.Save(ctx, userID, image)
disk.AssertExists("avatars/" + userID + ".png")

A driver you write is checked with filesystemtest.Run(t, newDisk). This is the same suite the built-in drivers pass, and its own tests prove that every case catches the bug it is meant to catch.

Development

Command What it does
task check gofmt, vet (host plus Linux, Windows and FreeBSD), every module's tests, the Markdown wrap check
task test:rustfs The S3 conformance suite against RustFS
task test:s3 The S3 conformance suite against SeaweedFS (prune_prefixes=true)
task test:azure The Azure driver's tests against Azurite
task test:clamav The scan package against a real clamd
task test:engines The records suite and the imaging plugins on SQLite, Postgres and MySQL

Each Docker-backed task starts a throwaway container from compose.yml, runs, and removes the container and its data afterwards, even when a test fails; Postgres is the host's. | task cover, task cover:records | Coverage per module, and records merged across the three engines |

License

Apache-2.0. See LICENSE and NOTICE. Contributions: CONTRIBUTING.md; security reports: SECURITY.md; changes: CHANGELOG.md.

Documentation

Overview

Package filesystem is a storage library for Go applications: one portable API over local disk, memory, S3-compatible object stores and any backend a Driver is written for.

Layers

The package is the bottom of five layers, each usable without the ones above it:

  • Driver: the small interface one backend implements (Stat, Read, Write, Delete, List, Capabilities). Drivers live in their own packages; the ones with third-party dependencies live in their own modules.
  • Disk: the concrete type applications hold. Every portable behaviour is written once here rather than per driver: key normalisation and traversal rejection, content-type detection, SHA-256 and byte counting on writes, size limits, error classification, URL generation, and the interceptor chain.
  • Manager: named disks, scoped and mirrored disks, cross-disk copy.
  • upload (sub-package): how bytes arrive from browsers and phones.
  • records (separate module): files as database rows attached to models.

Capabilities

Anything only some backends can do (presigned URLs, server-side copy, per-object visibility, multipart uploads, stored metadata) is a capability. A driver declares its capabilities in one struct, and NewDisk refuses a driver whose declaration does not match the interfaces it implements, so no driver can claim a feature it does not have. Disk methods for a missing capability either fall back to a portable implementation (Copy, Move, DeletePrefix) or return an error whose Code is CodeUnsupported.

Interceptors

Retry, timeouts, logging, metrics, tracing and read-only enforcement run as Interceptors around every Disk operation. They never wrap the driver, so they cannot hide its capabilities (the usual failure of Go decorators). See the layer sub-package for the built-in ones.

Keys

Keys are slash-separated, relative and normalised by CleanKey before any driver sees them. A key containing a ".." segment is rejected, never resolved. Directories do not exist as objects: a "directory" is the set of keys sharing a prefix, which is how every object store behaves.

Index

Constants

View Source
const (
	MinPartNumber = 1
	MaxPartNumber = 10000
)

MinPartNumber and MaxPartNumber bound part numbers. They are S3's limits, adopted by every S3-compatible backend and by browser uploaders.

View Source
const (
	PostFieldKey                = "key"
	PostFieldContentType        = "Content-Type"
	PostFieldCacheControl       = HeaderCacheControl
	PostFieldContentDisposition = HeaderContentDisposition
	PostFieldContentEncoding    = HeaderContentEncoding
	// PostFileField is the form field holding the file; backends require
	// it to be the last field.
	PostFileField = "file"
	// PostFilenamePlaceholder in a key prefix upload's key field is
	// replaced by the backend with the uploaded file's name.
	PostFilenamePlaceholder = "${filename}"
)

Form field names of a presigned POST, the HTML-form upload S3 and GCS accept (UP-04). A driver's PresignPost returns its fields under these names where the backend defines them, so a client builds the form the same way for every backend: every field of PresignedPost.Fields, in any order, then the file in PostFileField, last.

View Source
const (
	HeaderCacheControl       = "Cache-Control"
	HeaderContentDisposition = "Content-Disposition"
	HeaderContentEncoding    = "Content-Encoding"
)

The HTTP names of the HTTPHeaders fields.

View Source
const (
	// ParamPrefix scopes the disk to a key prefix (see Disk.Scope).
	ParamPrefix = "prefix"
	// ParamPublicURL sets WithPublicURL.
	ParamPublicURL = "public_url"
	// ParamVisibility sets WithDefaultVisibility ("public" or "private").
	ParamVisibility = "visibility"
	// ParamReadOnly ("true" or "false") adds the ReadOnly interceptor.
	ParamReadOnly = "readonly"
)

Query parameters every DSN accepts, handled by the Mux itself.

View Source
const (
	// EnvDisks lists the disk names, comma-separated ("media,backups").
	EnvDisks = "DISKS"
	// EnvDefaultDisk names the default disk; empty means the first listed.
	EnvDefaultDisk = "DEFAULT_DISK"
	// EnvDSNSuffix ends each disk's DSN variable: <PREFIX><NAME>_DSN.
	EnvDSNSuffix = "_DSN"
)

Variable names OpenManager reads, each after the caller's prefix.

View Source
const (
	Byte int64 = 1
	KB   int64 = 1000
	MB         = 1000 * KB
	GB         = 1000 * MB
	TB         = 1000 * GB
	KiB  int64 = 1 << 10
	MiB  int64 = 1 << 20
	GiB  int64 = 1 << 30
	TiB  int64 = 1 << 40
)

Size units for ParseSize. Decimal units are powers of 1000 and binary units powers of 1024, as the SI and IEC define them: "10MB" is 10,000,000 bytes and "10MiB" is 10,485,760. Units are case-sensitive, so "mb" (millibits, to the strict reader) is refused rather than guessed.

View Source
const CheckDir = ".filesystem-check"

CheckDir is the directory Check writes its probe object into. It is a fixed, recognisable name so an operator who finds a leftover probe (after a crash mid-check) knows what it is and that it is safe to delete.

View Source
const DefaultPageSize = 1000

DefaultPageSize is the page size used when ListOptions.Limit is 0. It is S3's maximum, so one page is one request on the most common backend.

View Source
const MaxKeyLength = 1024

MaxKeyLength is the longest key, in bytes, that CleanKey accepts. It is S3's limit, the strictest of the common object stores, so a key that is valid on one disk is valid on every disk.

View Source
const MaxPresignExpiry = 7 * 24 * time.Hour

MaxPresignExpiry is the longest lifetime a signed URL may have. It is the S3 SigV4 limit; a longer-lived URL is effectively a public link and should be one, through VisibilityPublic.

View Source
const OverlayWhiteouts = ".overlay-whiteouts"

OverlayWhiteouts is the directory of the upper disk where an overlay records deleted lower keys: deleting "a/b.txt" while the lower disk holds it writes an empty "<OverlayWhiteouts>/a/b.txt" on the upper disk. Keys under it, in any letter case, are reserved on an overlay disk and refused with CodeInvalidKey: on an upper disk that folds case, a key such as ".OVERLAY-WHITEOUTS/a.txt" would otherwise forge a whiteout.

Variables

View Source
var (
	ErrNotFound        error = &codeSentinel{CodeNotFound}
	ErrExists          error = &codeSentinel{CodeExists}
	ErrInvalidKey      error = &codeSentinel{CodeInvalidKey}
	ErrInvalidArgument error = &codeSentinel{CodeInvalidArgument}
	ErrUnsupported     error = &codeSentinel{CodeUnsupported}
	ErrReadOnly        error = &codeSentinel{CodeReadOnly}
	ErrTooLarge        error = &codeSentinel{CodeTooLarge}
	ErrRejected        error = &codeSentinel{CodeRejected}
	ErrQuotaExceeded   error = &codeSentinel{CodeQuotaExceeded}
	ErrPermission      error = &codeSentinel{CodePermission}
	ErrUnavailable     error = &codeSentinel{CodeUnavailable}
	ErrConflict        error = &codeSentinel{CodeConflict}
	ErrNotModified     error = &codeSentinel{CodeNotModified}
	ErrUnknown         error = &codeSentinel{CodeUnknown}
)

Sentinels for errors.Is. Each matches every *Error with the same Code.

AbandonedKindValues lists every AbandonedKind.

CodeValues lists every Code, for exhaustive tests and documentation.

View Source
var ErrSuperseded = Errorf(CodeConflict, "a concurrent attempt delivered the result")

ErrSuperseded is what a concurrent call of a hedgeable operation (see Op.Concurrent) returns when another call already delivered the result: the operation succeeded, through the other call. A hedging interceptor waits for the other call's answer instead of taking this one. Should it reach a caller anyway, its code is CodeConflict.

View Source
var HTTPMethodValues = []HTTPMethod{MethodGet, MethodPut}

HTTPMethodValues lists every HTTPMethod.

OpKindValues lists every OpKind.

View Source
var SyncActions = []SyncAction{SyncCopy, SyncDelete}

SyncActions lists every SyncAction.

VisibilityValues lists every Visibility.

Functions

func As

func As[T any](d *Disk) (T, bool)

As is the typed escape hatch to a backend's native client: it returns the value of the first driver in the wrapper chain (following Unwrapper) that has a method Native() T. For example, As[*s3.Client](disk) on an S3 disk. It returns false when no driver offers a T.

func Batch

func Batch(ctx context.Context, keys []string, parallel int, fn func(ctx context.Context, key string) error) error

Batch runs fn for every key with at most parallel calls at a time and collects the failures: nil when every call succeeded, otherwise a *BatchError listing each failed key with its error, in the order of keys. It is the helper behind the Disk's per-key fallbacks, and the shape to use for any per-key work an application fans out (copying a tree, re-encoding every image under a prefix).

A parallel below 1 runs one call at a time. Keys are passed as given, duplicates included. Once ctx ends no further call starts, and every key not yet started fails with the context's error, so the result still names every key that was not processed.

func CleanDir

func CleanDir(raw string) (string, error)

CleanDir normalises a directory argument (for List and DeletePrefix) the same way as CleanKey, except that the empty string, "/" and "." are valid and mean the root of the disk. The result never has a trailing slash.

func CleanKey

func CleanKey(raw string) (string, error)

CleanKey normalises a caller-supplied key and rejects unsafe ones. Every Disk method runs it before a driver sees the key, which is what makes path traversal impossible regardless of the driver's own care.

Normalisation: backslashes become slashes, leading slashes are dropped, empty and "." segments are removed, and runs of slashes collapse.

Rejection (CodeInvalidKey): an empty result, any ".." segment (rejected rather than resolved, so "a/../b" is an error and never silently "b"), invalid UTF-8, a control character (including NUL), or more than MaxKeyLength bytes.

func EnvName

func EnvName(prefix, name string) string

EnvName is the variable that holds the DSN of disk name under prefix: the name upper-cased, with "-" and "." as "_" ("STORAGE_" and "media-cdn" give "STORAGE_MEDIA_CDN_DSN").

func EscapeKey

func EscapeKey(key string) string

EscapeKey percent-escapes each segment of a clean key for use in a URL path, keeping the slashes between segments.

func JoinKey

func JoinKey(parts ...string) string

JoinKey joins already-clean key parts with the separator, skipping empty parts. It does not clean its input; wrapper drivers use it to prefix keys that the Disk has already cleaned.

func KeyInDir

func KeyInDir(key, dir string) bool

KeyInDir reports whether key lies under dir (a clean directory as returned by CleanDir, where "" is the root). A key equal to dir is not inside it.

func NotModified

func NotModified(e Entry, opts ReadOptions) bool

NotModified reports whether opts say the caller's copy of the object e describes is current, by the HTTP rules: If-None-Match first (an exact ETag, or "*" for any object), else If-Modified-Since to the second. It is exported for drivers that implement ConditionalReader over a Stat.

func ObjectSHA256

func ObjectSHA256(ctx context.Context, d *Disk, key string) (string, error)

ObjectSHA256 reads the object at key on d in full and returns the hex SHA-256 of its content: the digest Disk.Write records in Entry.SHA256, computed from the stored bytes rather than trusted from metadata. It is for integrity checks (Sync's Checksum, scrubbing records against the digest they stored), and costs a full read.

func ParseAllowedIP

func ParseAllowedIP(s string) (netip.Prefix, error)

ParseAllowedIP parses PresignOptions.AllowedIP: a CIDR prefix in canonical form. Drivers and signers that enforce the restriction parse it with this one function, so every backend accepts and refuses the same values. An IPv4-mapped IPv6 prefix (::ffff:203.0.113.0/120) is returned as the IPv4 prefix it covers (203.0.113.0/24), because client addresses are compared unmapped and the mapped form would match none of them; one shorter than /96 also covers non-IPv4 addresses and is refused.

func ParseSize

func ParseSize(s string) (int64, error)

ParseSize reads a human size such as "10MB", "1.5GiB", "512KiB", "64B" or "1024" (bytes) into a byte count, for configuration values like an upload limit. It is strict: the number is a plain decimal (digits, at most one ".", no sign, exponent, separator or space), the unit is one of B, KB, MB, GB, TB (powers of 1000) or KiB, MiB, GiB, TiB (powers of 1024) written exactly so, nothing may follow it, and the result must be a whole number of bytes that fits an int64 ("1.5B" is refused). Anything else is CodeInvalidArgument, never a guess.

func RejectParams

func RejectParams(u *url.URL, allowed ...string) error

RejectParams returns CodeInvalidArgument naming the first query parameter of u that is not in allowed. Openers call it to refuse typos.

func RelativeKey

func RelativeKey(key, dir string) string

RelativeKey returns key with the dir prefix removed. Callers must have checked KeyInDir first; for a key outside dir it returns key unchanged.

func RetryAfterOf

func RetryAfterOf(err error) (time.Duration, bool)

RetryAfterOf returns the wait the backend asked for, from the outermost *Error in err's chain that carries one, and false when none does.

func SameContent

func SameContent(src, dst Entry) bool

SameContent reports whether dst holds src's content as far as the listings can tell: equal sizes, and equal SHA-256 digests when both have one, otherwise equal ETags when both have one. Anything less counts as different, so an object whose content cannot be compared is copied again rather than wrongly skipped. ETags are compared as strings: two backends that compute them differently (an S3 multipart ETag and a local one) never match, which costs a copy, never a skip.

func TrimCache

func TrimCache(ctx context.Context, cache *Disk, maxBytes int64) (int, error)

TrimCache deletes the oldest copies on cache (by modification time, so by when they were copied or last revalidated) until the bytes it holds are at most maxBytes, and returns how many it deleted. Run it from a job (jobs.Func) to bound a CachedRead cache; deleting a copy only costs a fetch from remote on its next read. It lists the whole cache disk, so point it only at a disk used as a cache.

func ValidateChecksums

func ValidateChecksums(list []Checksum) error

ValidateChecksums checks list the way Disk.Write does before it reads a byte: every Name is 1 to 32 lowercase letters, digits or dashes and unique, every New is set, and every Want is hex of New's size. Layers that compute the same digests themselves (upload.Confirm reading an assembled object) call it so a bad list fails the same way on every path, with CodeInvalidArgument.

func ValidatePresign

func ValidatePresign(opts PresignOptions) error

ValidatePresign checks PresignOptions: a known method, an expiry in (0, MaxPresignExpiry], Headers only on a PUT and free of CR and LF, AllowedIP empty or a canonical CIDR prefix (ParseAllowedIP), and Visibility empty or a known value on a PUT. Signers call it too, so the rule is enforced the same way on every path.

Types

type Abandoned

type Abandoned struct {
	Kind AbandonedKind
	// ID is the multipart upload ID, or the staging file's name inside
	// the driver's staging directory.
	ID string
	// Key is the key a multipart upload was started for, as the backend
	// driver received it (a scoped disk's prefix included). It is empty
	// for a staging file, and for an upload whose record no longer names
	// its key.
	Key string
	// LastActivity is the time the sweep compares with its cutoff: the
	// last part received, or, where the backend records only that, when
	// the upload started (S3, Azure); a staging file's last write.
	LastActivity time.Time
}

Abandoned is one leftover a sweep would remove, as FindAbandoned reports it.

type AbandonedFinder

type AbandonedFinder interface {
	FindAbandoned(ctx context.Context, before time.Time) ([]Abandoned, error)
}

AbandonedFinder is the dry run of AbandonedSweeper: FindAbandoned returns what SweepAbandoned(ctx, before) would remove at this moment, applying exactly the same rules (the same cutoff, the same names it leaves alone), and changes nothing. Work that starts, completes or is aborted between the two calls makes them differ, so a sweep's count can be lower or higher than the preview's length. A driver implements both from one scan, so the two cannot drift apart.

type AbandonedKind

type AbandonedKind string

AbandonedKind names what kind of leftover an Abandoned is.

const (
	// AbandonedMultipart is a multipart upload never completed or aborted;
	// sweeping it aborts it and frees its parts.
	AbandonedMultipart AbandonedKind = "multipart"
	// AbandonedStaging is a staging file of a write interrupted before
	// it was renamed into place (local, sftp, ftp, webdav).
	AbandonedStaging AbandonedKind = "staging"
)

The closed set of leftovers a sweep removes.

type AbandonedSweeper

type AbandonedSweeper interface {
	SweepAbandoned(ctx context.Context, before time.Time) (int, error)
}

AbandonedSweeper removes what unfinished work left behind in the backend: multipart uploads that were never completed or aborted, and any staging files of writes that were interrupted. Those cost storage and are invisible to List, so nothing else ever removes them.

Work counts as abandoned when its last activity (a part received, a staging file written) is before before; a backend that records only when an upload started uses that. Work at or after before is left alone, so a caller must choose before well beyond the longest upload it allows; a sweep with a recent before aborts uploads in progress. Stored objects are never touched. It returns how many uploads and staging files it removed; a failure part-way returns the count so far with the error, and a rerun continues.

type AppendOptions

type AppendOptions struct {
	// Visibility of an object Append creates; empty means the disk's
	// default. An existing object keeps its own.
	Visibility Visibility
	// MaxSize, when positive, caps the object's size after the append: an
	// append that would take it past MaxSize is refused with CodeTooLarge
	// and appends nothing. A log or export then cannot grow without bound
	// however many appends arrive.
	MaxSize int64
}

AppendOptions control an append.

type Appender

type Appender interface {
	Append(ctx context.Context, key string, body io.Reader, opts AppendOptions) (Entry, error)
}

Appender adds bytes to the end of an object, creating it when missing. Contract, checked by the conformance suite: a body that fails to read appends nothing (the object is as it was), two appends to one key never interleave their bytes, and an append that would take the object past a positive opts.MaxSize is refused with CodeTooLarge without appending (the check and the append are one step, so concurrent appends cannot overshoot together).

type BatchError

type BatchError struct {
	Failures []KeyError
}

BatchError reports the keys a batch operation (Disk.DeleteKeys) could not process; every other key succeeded. errors.Is and CodeOf see the per-key errors, so errors.Is(err, ErrPermission) asks whether any key failed for lack of permission.

func (*BatchError) Error

func (e *BatchError) Error() string

Error names the failure count and the first failure. A BatchError with no failures is a caller's bug (a driver built one by hand), but Error must not panic inside a log line, so it says so instead.

func (*BatchError) Unwrap

func (e *BatchError) Unwrap() []error

Unwrap returns the per-key errors.

type ByteRange

type ByteRange struct {
	Offset int64
	Length int64
}

ByteRange selects part of an object for Read. Offset is the first byte; Length is the number of bytes, where 0 means "to the end of the object" (reading zero bytes is never useful, so the zero value reads everything from Offset). A range past the end of the object yields an empty read, not an error.

type CacheOptions

type CacheOptions struct {
	// TTL is how long a copy is served without asking remote. Required,
	// positive.
	TTL time.Duration
	// OnCacheError, when set, is told of every cache failure the disk
	// absorbs: a copy that could not be made, read, refreshed or dropped.
	// No such failure fails the call, so a copy that could not be dropped
	// after a write stays stale until TTL passes; this hook is how the
	// application finds out (log it, count it, alert on it).
	OnCacheError func(ctx context.Context, key string, err error)
	// Now is the clock a copy's age is judged by against TTL, and the time
	// a revalidated copy is marked fresh at; nil means time.Now. A fake
	// clock makes TTL expiry testable without sleeping. A copy's age is
	// Now minus its ModTime on cache, so cache's clock and this one must
	// agree.
	Now func() time.Time
}

CacheOptions configure CachedRead.

type Call

type Call func(ctx context.Context) error

Call runs the operation (or the rest of the chain). Each invocation is one attempt.

type Capabilities

type Capabilities struct {
	// Interface-backed capabilities.
	Copy          bool // implements Copier: server-side copy
	Move          bool // implements Mover: server-side or atomic rename
	SetVisibility bool // implements VisibilitySetter; Stat reports Visibility
	DeletePrefix  bool // implements PrefixDeleter: bulk delete in few calls
	DeleteKeys    bool // implements KeysDeleter: many keys deleted in few calls
	Presign       bool // implements Presigner: backend-signed URLs
	PublicURL     bool // implements PublicURLer
	Multipart     bool // implements MultipartUploader: parts uploaded through the server
	PresignPart   bool // implements PartPresigner: parts uploaded directly by clients
	PresignPost   bool // implements PostPresigner: browser form uploads the backend checks
	// SweepAbandoned: implements AbandonedSweeper, removing what unfinished
	// work left in the backend (multipart uploads never completed or
	// aborted, staging files of interrupted writes). Wrapper drivers that
	// narrow a disk to part of a backend (scoped, mirror) never forward it,
	// because the leftovers are backend-wide.
	SweepAbandoned bool
	// FindAbandoned: implements AbandonedFinder, listing what
	// SweepAbandoned would remove without removing it (a dry run). Needs
	// SweepAbandoned, and like it is never forwarded by a wrapper driver.
	FindAbandoned bool
	// Dirs: implements DirMaker. The backend has real directories, so
	// MakeDir creates an empty one that DirExists then reports. Without
	// it, MakeDir does nothing and a directory exists exactly while a key
	// is under it (see Disk.MakeDir).
	Dirs       bool
	Append     bool // implements Appender: add bytes to the end of an object
	SetModTime bool // implements ModTimeSetter: set an object's modification time
	// ConditionalRead: implements ConditionalReader, answering
	// If-None-Match and If-Modified-Since in the backend; without it
	// Disk.ReadIf compares against a Stat.
	ConditionalRead bool
	Tags            bool // implements Tagger: key-value tags stored with an object
	Hold            bool // implements Holder: legal holds and retention dates
	Versions        bool // implements VersionLister: an object's stored versions
	// VersionRead: implements VersionReader, reading one stored version and
	// making it the current content again. A separate bit from Versions, so
	// a backend that can only list versions stays honest. Needs Versions,
	// which is where version IDs come from.
	VersionRead  bool
	StorageClass bool // implements StorageClasser: change an object's storage class
	Restore      bool // implements Restorer: bring an archived object back online

	// Attribute capabilities, with no interface of their own.
	Metadata    bool // stores WriteOptions.Metadata and returns it from Stat
	ContentType bool // stores WriteOptions.ContentType and returns it from Stat
	Headers     bool // stores WriteOptions.Headers and returns them from Stat
	// PresignHeaders: the driver's presigned PUT (Presign) pins
	// PresignOptions.Headers, so the backend refuses an upload without them
	// and stores them. Needs Presign and Headers.
	PresignHeaders bool
	// PresignIP: the driver's presigned URLs (Presign) carry
	// PresignOptions.AllowedIP, so the backend refuses a request from an
	// address outside it. Needs Presign.
	PresignIP bool
	// PresignVisibility: the driver's presigned PUT (Presign) and POST
	// form (PresignPost), whichever it declares, pin
	// PresignOptions.Visibility and PostPolicy.Visibility, so the backend
	// stores the upload with that visibility and refuses a request that
	// changes it. Needs SetVisibility and one of Presign or PresignPost;
	// a driver whose backend has no per-object ACL (S3 with bucket owner
	// enforced, GCS with uniform access) does not declare it.
	PresignVisibility bool
	ConditionalWrite  bool // honours WriteOptions.IfNotExists atomically
	// ChecksumSHA256: the driver sends WriteOptions.SHA256 to the backend,
	// which refuses a body that does not match it. The Disk verifies the
	// digest itself on every driver; this adds the backend's own check.
	ChecksumSHA256 bool
	// CompareAndSwap: honours WriteOptions.IfMatch atomically, replacing an
	// object only while its ETag is still the one the caller read.
	CompareAndSwap bool
	// CaseInsensitive: keys differing only in letter case address one
	// object ("A.txt" and "a.txt"). Keys are case-sensitive on every other
	// backend; a driver over a case-insensitive filesystem (macOS, Windows
	// by default) reports it so callers can refuse such pairs themselves.
	CaseInsensitive bool
}

Capabilities declares what a driver can do beyond the Driver interface. Interface-backed fields must match the optional interfaces the driver implements; NewDisk checks this. The Disk calls an optional interface only when its field is true, which is what lets a wrapper driver implement every interface yet report only what the driver underneath supports.

func (Capabilities) String

func (c Capabilities) String() string

String renders the true capabilities, for logs and diagnostics.

type Checksum

type Checksum struct {
	// Name labels the digest in Entry.Checksums ("md5", "crc32c"):
	// lowercase letters, digits and dashes, 1 to 32 characters, unique
	// within one write.
	Name string
	// New returns a fresh hasher; it is called once per write attempt, so
	// a retried write starts from zero. Required.
	New func() hash.Hash
	// Want, when set, is the hex digest the body must have (any case).
	// A body that ends with another one fails the write with
	// CodeInvalidArgument before the driver commits, exactly as
	// WriteOptions.SHA256 does, so a corrupted body never replaces what
	// was stored. Its length must match New's Size.
	Want string
}

Checksum asks Disk.Write for one more digest of the body, computed while it streams, beside the SHA-256 it always computes (Entry.SHA256), and optionally verified: MD5 for an existing checksum column, CRC32C to compare with what GCS reports, BLAKE3 or xxHash for fast content addressing. The library names no algorithm and imports no hash package for it: the caller brings the hash.Hash.

Why a list of hashers rather than a fixed set: every application keeps the digest its other systems already use, and adding one should not need a release of this library. SHA-256 stays the one the Disk always computes, so Entry.SHA256, upload.Result.SHA256 and the records schema keep their meaning.

type Code

type Code string

Code classifies an error independently of the backend that produced it. Callers branch on the Code (with errors.Is against the Err* sentinels, or with CodeOf) and never on a vendor error type, which is what lets an application swap S3 for local disk without touching its error handling.

const (
	// CodeNotFound: the key does not exist (or the directory has no keys).
	CodeNotFound Code = "not_found"
	// CodeExists: a conditional write (WriteOptions.IfNotExists) found the
	// key already present, or a registry already holds the name.
	CodeExists Code = "exists"
	// CodeInvalidKey: the key is empty, too long, not valid UTF-8, contains a
	// control character, or contains a ".." segment.
	CodeInvalidKey Code = "invalid_key"
	// CodeInvalidArgument: an option or argument other than the key is
	// invalid (an unknown visibility, a non-positive expiry, …).
	CodeInvalidArgument Code = "invalid_argument"
	// CodeUnsupported: the driver does not have the capability the call
	// needs, and there is no portable fallback.
	CodeUnsupported Code = "unsupported"
	// CodeReadOnly: a mutating operation reached a read-only disk.
	CodeReadOnly Code = "read_only"
	// CodeTooLarge: a write exceeded WriteOptions.MaxSize. Nothing is stored.
	CodeTooLarge Code = "too_large"
	// CodeRejected: the content was refused by a policy, for example an
	// upload whose detected type is not allowed. Nothing is stored.
	CodeRejected Code = "rejected"
	// CodeQuotaExceeded: the owner of the scope has no storage left (see
	// records/quota). Distinct from CodeTooLarge, which is about one file.
	CodeQuotaExceeded Code = "quota_exceeded"
	// CodePermission: the backend refused the operation (credentials, ACL,
	// file mode).
	CodePermission Code = "permission"
	// CodeUnavailable: a transient failure (timeout, throttling, connection
	// reset). The only code the retry interceptor retries. A context that
	// ended (context.DeadlineExceeded or context.Canceled anywhere in the
	// chain) is classified here too, so a per-attempt deadline from
	// layer.Timeout is retried; layer.Retry still stops once the caller's
	// own context has ended.
	CodeUnavailable Code = "unavailable"
	// CodeConflict: a compare-and-swap write (WriteOptions.IfMatch) found
	// the object changed, or gone, since the caller read it. Nothing is
	// written; read it again and retry.
	CodeConflict Code = "conflict"
	// CodeNotModified: a conditional read (Disk.ReadIf) found the object
	// unchanged, so no body is returned. Not a failure: the caller's copy
	// is current.
	CodeNotModified Code = "not_modified"
	// CodeUnknown: an error the driver could not classify.
	CodeUnknown Code = "unknown"
)

The closed set of error codes. Every driver maps its native errors into these; an error it cannot classify becomes CodeUnknown, never a new code.

func CodeOf

func CodeOf(err error) Code

CodeOf returns the Code of err, classified exactly as the Disk classifies a driver's error: the Code of the outermost *Error in its chain (an unknown one becomes CodeUnavailable when the chain holds a context error), the code of a bare sentinel, CodeNotFound, CodeExists or CodePermission for the io/fs sentinels, CodeUnavailable for a context that ended, or CodeUnknown for any other non-nil error. It returns "" for a nil error.

Interceptors see a driver's error before the Disk annotates it, so this is what lets layer.Retry recognise a bare context.DeadlineExceeded from an attempt that layer.Timeout cut short.

type ConditionalReader

type ConditionalReader interface {
	ReadIf(ctx context.Context, key string, opts ReadOptions) (io.ReadCloser, Entry, error)
}

ConditionalReader answers a conditional read in the backend (S3, GCS and Azure send the conditions with the GET), returning CodeNotModified and no body when the condition holds.

type Copier

type Copier interface {
	Copy(ctx context.Context, src, dst string) (Entry, error)
}

Copier copies an object inside the backend without streaming it through the application. dst is overwritten if present.

type DirMaker

type DirMaker interface {
	MakeDir(ctx context.Context, dir string) error
	DirExists(ctx context.Context, dir string) (bool, error)
}

DirMaker is implemented by backends with real directories (local disk, SFTP, FTP, WebDAV). Directories arrive cleaned and never empty (the root always exists).

MakeDir creates dir and any missing parents, and succeeds when dir already exists; when an object already has dir's key it fails with CodeExists. DirExists reports whether dir exists as a directory, empty or not; an object at dir's key is not a directory.

type Disk

type Disk struct {
	// contains filtered or unexported fields
}

Disk is the handle applications use for one storage location. It is safe for concurrent use. Construct it with NewDisk; the zero value is not usable.

Every method cleans its key with CleanKey before the driver sees it, runs the driver call through the interceptor chain as one Op, and returns errors as *Error with the operation, disk name and key filled in.

func CachedRead

func CachedRead(name string, remote, cache *Disk, opts CacheOptions) (*Disk, error)

CachedRead returns a Disk named name over remote whose reads are served from a copy on cache, typically a local disk in front of a slow or metered bucket. A read of a key not on cache copies the whole object there first, then serves the read (ranges included) from the copy; a copy younger than opts.TTL is served without asking remote. An older copy is revalidated with a Stat of remote: when cache stores metadata and remote reports the same ETag it was copied with, the copy is served (and, where cache can set modification times, made fresh again); otherwise it is copied anew. Writes, deletes, copies, moves, appends and completed multipart uploads go to remote and then drop cache's copy.

Semantics, which callers must accept:

  • Freshness is bounded by TTL, not exact: a copy made while another process writes the key may be stale until it is revalidated. Use it for content that changes rarely or under new keys.
  • Stat, List, URLs and every other capability are remote's and are never cached. Signed PUT URLs and form uploads are refused or not offered (an upload straight to remote would leave a stale copy).
  • A conditional read (ReadIf) asks remote for the current ETag and serves the copy only when it was made from that ETag, else reads remote, so its entry and body always match.
  • A cache that fails (full, unavailable) never fails a call: reads are served from remote, writes and deletes return remote's result, and OnCacheError is told.
  • cache grows with every key read; bound it with TrimCache, run as a job.

cache must be a disk of its own: keys on it are remote's keys, and TrimCache deletes from it.

func Failover

func Failover(name string, primary *Disk, secondaries []*Disk, opts FailoverOptions) (*Disk, error)

Failover returns a Disk named name that reads (Stat and Read) from primary and, when primary fails as ShouldFailover decides, from each secondary in turn, so reads survive an outage of the primary backend. Everything else (writes, deletes, listings, URLs, every capability) goes to primary alone.

Semantics, which callers must accept:

  • Secondaries are only read. Keeping them in step is the application's job, typically with Mirror (write to all) or a backfill; a read that fails over returns whatever the secondary holds, which may be older than the primary's copy or missing (CodeNotFound).
  • Listings are not failed over: a cursor from one backend means nothing to another.
  • The disk has primary's capabilities and interceptors, except ConditionalRead, so conditional reads run as a Stat and a Read that both fail over, and SweepAbandoned and FindAbandoned (sweep the primary directly). A secondary runs its own interceptors.
  • When every disk fails, the last secondary's error is returned.

func Mirror

func Mirror(name string, primary *Disk, mirrors ...*Disk) (*Disk, error)

Mirror returns a Disk named name that reads from primary and writes to primary and then to every mirror. It is how an application moves to a new backend without downtime (mirror old to new, backfill, then switch the primary) or keeps a warm copy.

Semantics, which callers must accept:

  • Not atomic across disks. A write succeeds on primary first; if a mirror then fails, the error is returned (CodeUnavailable wrapping the cause) and the object exists on primary only. Writes are overwrites, so retrying converges.
  • Mirrors receive the bytes by reading the object back from primary, because the caller's body can be consumed only once.
  • A mirror that folds letter case where the primary does not (local disk on macOS behind a memory or S3 primary) holds one object for keys that differ only in case; the primary holds two. Compare Capabilities().CaseInsensitive before mixing them.
  • When the primary's driver signs its own URLs (S3, GCS, Azure), signed PUT URLs and presigned parts are refused: a client uploading straight to the backend would bypass the mirrors. A primary that cannot sign uses the disk's URLSigner (WithURLSigner), and those PUT URLs are issued: they name the mirror disk and are served by the application (fshttp.Signed), so the upload is written through the mirror as long as the application resolves that name to the mirror disk. Server-side multipart is allowed and replicated on completion.
  • The mirror disk's capabilities are primary's, minus PresignPart, PresignPost, PresignHeaders, PresignVisibility (signed PUTs are refused, as above), Append, SetModTime, ConditionalRead, CompareAndSwap, Tags, Hold, Versions, VersionRead, StorageClass, Restore (in-place changes would skip the mirrors) and SweepAbandoned and FindAbandoned (each disk keeps its own leftovers; sweep them directly). MakeDir creates the directory on the primary, then on every mirror.

func NewDisk

func NewDisk(name string, driver Driver, opts ...Option) (*Disk, error)

NewDisk builds a Disk over driver. name identifies the disk in errors, logs and signed URLs, and must match [A-Za-z0-9][A-Za-z0-9._-]{0,63}.

It fails with CodeInvalidArgument when the driver declares a capability it does not implement (see Capabilities), so a misdeclared driver is caught at start-up rather than at the first call.

func Overlay

func Overlay(name string, upper, lower *Disk, opts ...Option) (*Disk, error)

Overlay returns a Disk named name that layers upper over lower, copy on write: reads find a key on upper first and fall through to lower, writes go to upper only, and deleting a key lower holds leaves a whiteout on upper that hides it. lower is never changed through the overlay. It is how a preview environment, a test or a tenant sandbox starts from shared read-only data and keeps its changes to itself. opts configure the overlay disk itself; upper and lower keep their own interceptors.

Semantics, which callers must accept:

  • Listings merge both disks in key order, upper winning a key both hold, and hide whited-out keys and directories whose lower keys are all whited out (checking such a directory costs one more listing).
  • Not atomic across the two disks: a delete removes upper's copy, then writes the whiteout; if that fails, lower's copy shows again and the error is returned, and retrying converges.
  • The overlay declares only what it honours on both layers: the write attributes upper stores (Metadata, ContentType, Headers, ChecksumSHA256) and CaseInsensitive when either folds case. Copy and Move stream through the Disk (a move of a lower key copies it up and whites it out), DeletePrefix and DeleteKeys go key by key, and the rest (URLs, visibility, conditional and in-place writes, multipart) is not offered: an object can sit on either layer.

func Route

func Route(name string, mounts []Mount, opts ...Option) (*Disk, error)

Route returns a Disk named name that dispatches each key to the mount with the longest Prefix containing it, so one handle can put "avatars/…" on a CDN-backed bucket, "exports/…" on cold storage and everything else on a default disk. opts configure the routed disk itself (interceptors around every routed call, a URL signer, …); each mounted disk keeps its own.

Semantics, which callers must accept:

  • A key under no prefix, with no catch-all mount, does not exist: Stat and Read report CodeNotFound, Delete succeeds, and Write fails with CodeInvalidArgument.
  • Keys of the catch-all under a mounted prefix are shadowed: unreachable and not listed, as with a filesystem mount.
  • Copy and Move inside one mount use that disk (server-side where it can); across mounts they stream (CopyAcross, MoveAcross), and a move across mounts is not atomic.
  • Listing a directory above mounts (the root, or "a" over a mount at "a/b") merges the catch-all's entries with the mounts, shown as directories, or with their keys when recursive.
  • The routed disk declares a capability only when every mount has it: Copy, Move, SetVisibility, Presign (the mount can sign both GET and PUT, each through its driver or its URLSigner as WithURLSigner routes that method; signer URLs name the mount), PublicURL (the mount's driver or a WithPublicURL base), PresignHeaders (whichever of the two signs the mount's PUTs can pin headers), PresignIP (whichever signs each method enforces it), PresignVisibility (every mount's PUTs go to a driver that pins it), and the write attributes (Metadata, ContentType, Headers, ConditionalWrite, ChecksumSHA256, CompareAndSwap). DeleteKeys is always declared, each mount deleting its share its own way; CaseInsensitive when any mount folds case. Multipart, form uploads, appends, tags, holds, versions, version reads, storage classes, restores, conditional reads, modification times, bulk DeletePrefix, SweepAbandoned, FindAbandoned and real directories (Dirs) are not routed: use the mounted disk for those (the Disk falls back for DeletePrefix, conditional reads and DirExists, and MakeDir does nothing).

func (*Disk) AbortMultipart

func (d *Disk) AbortMultipart(ctx context.Context, key, uploadID string) error

AbortMultipart discards an unfinished upload and its parts. Aborting an unknown upload succeeds, so it is safe in cleanup paths.

func (*Disk) Append

func (d *Disk) Append(ctx context.Context, key string, body io.Reader, opts AppendOptions) (Entry, error)

Append adds body to the end of key, creating the object (with opts.Visibility, or the disk's default) when it does not exist, and returns the object's entry afterwards: logs, audit trails, chunked exports. With opts.MaxSize the object never grows past it: a body that alone exceeds it is cut off as it streams, and the driver refuses an append that would exceed it, both with CodeTooLarge and nothing appended. Needs the Append capability (local, memory, SFTP, Azure append blobs); object stores without appends answer CodeUnsupported. The body is a stream, so an append is never retried. Size and hash in the entry are the whole object's as the driver reports them, not the appended part's.

func (*Disk) Capabilities

func (d *Disk) Capabilities() Capabilities

Capabilities returns what the disk's driver can do. Methods backed by a missing capability either fall back (Copy, Move, DeletePrefix) or return CodeUnsupported; this lets callers check up front.

func (*Disk) Check

func (d *Disk) Check(ctx context.Context) error

Check proves the disk works end to end: it writes a small random object, reads it back, compares the bytes and deletes it. Run it at start-up so a wrong credential or a missing bucket fails the deploy instead of the first user upload. A read-only disk cannot be checked this way and reports CodeReadOnly.

func (*Disk) CleanKey

func (d *Disk) CleanKey(key string) (string, error)

CleanKey returns key as this disk addresses it: passed through the key normalizer (WithKeyNormalizer), if any, then cleaned and checked like the package-level CleanKey. Code that compares keys it built with keys the disk returns (List, Stat) should clean them with this method.

func (*Disk) Close

func (d *Disk) Close() error

Close releases the driver's resources when it holds any (it implements io.Closer); otherwise it does nothing.

func (*Disk) CompleteMultipart

func (d *Disk) CompleteMultipart(ctx context.Context, key, uploadID string, parts []Part) (Entry, error)

CompleteMultipart assembles the parts into the final object. parts must list every part with the ETag the backend returned for it.

func (*Disk) Copy

func (d *Disk) Copy(ctx context.Context, src, dst string) (Entry, error)

Copy copies src to dst on this disk, replacing dst. With the Copy capability the backend copies server-side; otherwise the object streams through the application as a Read followed by a Write, keeping its content type, metadata and visibility where the driver stores them.

func (*Disk) CreateMultipart

func (d *Disk) CreateMultipart(ctx context.Context, key string, opts WriteOptions) (string, error)

CreateMultipart starts a multipart upload of key and returns its upload ID. Metadata, content type and visibility are taken from opts and checked against the capabilities as for Write. Needs the Multipart capability.

func (*Disk) DataURI

func (d *Disk) DataURI(ctx context.Context, key string, maxBytes int64) (string, error)

DataURI returns key as an RFC 2397 data URI, "data:<type>;base64,<bytes>", for inlining a tiny file (an icon, a placeholder image) into HTML, CSS or JSON without a second request.

It refuses, before reading a byte, anything Stat reports larger than maxBytes (CodeTooLarge) and stored active content (HTML, SVG, XML, JavaScript; CodeRejected), because a data URI of an active type runs script wherever it is embedded. The read itself stops at maxBytes + 1, since a backend's reported size can be stale, and the bytes are sniffed: content that is active whatever its stored type is refused too. The type is the stored one (only its charset parameter is kept), else the sniffed one. maxBytes must be positive (CodeInvalidArgument); the caller chooses it, since what counts as tiny depends on where the URI is embedded, and the base64 output is a third larger than the file.

func (*Disk) DefaultVisibility

func (d *Disk) DefaultVisibility() Visibility

DefaultVisibility returns the visibility writes get when they do not choose one. For a driver without per-object visibility it is the visibility of every object on the disk.

func (*Disk) Delete

func (d *Disk) Delete(ctx context.Context, key string) error

Delete removes key. Deleting a missing key succeeds, so deletes are idempotent and safe to retry.

func (*Disk) DeleteKeys

func (d *Disk) DeleteKeys(ctx context.Context, keys []string) error

DeleteKeys deletes many keys: in few calls with the DeleteKeys capability (S3 deletes 1000 per request), otherwise one Delete per key. Missing keys succeed, as with Delete. Keys that could not be deleted, invalid ones included, come back in a *BatchError while every other key is deleted, so one bad key never stops a cleanup; nil means all are gone. Duplicate keys are deleted once.

func (*Disk) DeletePrefix

func (d *Disk) DeletePrefix(ctx context.Context, dir string) error

DeletePrefix deletes every key under dir. With the DeletePrefix capability the backend does it in bulk; otherwise the Disk lists and deletes key by key. dir must not be the root: emptying a whole disk is never a prefix operation, so it is refused with CodeInvalidArgument.

func (*Disk) DirExists

func (d *Disk) DirExists(ctx context.Context, dir string) (bool, error)

DirExists reports whether the directory dir exists. dir is cleaned like a key; the root always exists. An object at dir's key is not a directory.

With the Dirs capability the backend answers, so an empty directory made by MakeDir counts. Without it a directory exists exactly while a key is under it, and DirExists lists dir with a limit of one entry (one OpList).

func (*Disk) DriverKey

func (d *Disk) DriverKey(key string) (string, error)

DriverKey returns the key the backend driver receives for key: the key cleaned as the Disk cleans it, then rewritten by every key-changing wrapper on the way down (KeyMapper, following Unwrapper). On a disk scoped to "tenant", DriverKey("a.png") is "tenant/a.png". It exists so options a backend binds to one exact key can name that key without guessing at prefixes. It does not follow wrappers that send a key to one of several disks (Route, Overlay), which implement neither interface; there it returns the key at that level.

func (*Disk) Exists

func (d *Disk) Exists(ctx context.Context, key string) (bool, error)

Exists reports whether key exists. A CodeNotFound error is false, nil; any other error is returned.

func (*Disk) FS

func (d *Disk) FS(ctx context.Context) fs.FS

FS returns a read-only io/fs view of the disk, so code written against the standard interfaces (html/template.ParseFS, http.FileServerFS, fs.WalkDir, testing/fstest) works on any backend. It implements fs.FS, fs.StatFS, fs.ReadDirFS and fs.ReadFileFS; fs.Sub gives a view of a directory.

io/fs has no context, so ctx governs every call made through the view: cancel it to stop them. Every call goes through the Disk, so keys are cleaned and interceptors run as for any other read.

Names are io/fs paths (fs.ValidPath: slash-separated, no "." or ".." elements, "." for the root). A name the Disk would clean into a different key (a backslash, a control character) does not exist in the view, so two names never reach one object. Directories are implied by keys, as in Disk.List: a directory exists while some key is below it, and the root always exists. Files report mode 0444 and directories fs.ModeDir|0555; FileInfo.Sys returns the Entry.

func (*Disk) FindAbandoned

func (d *Disk) FindAbandoned(ctx context.Context, before time.Time) ([]Abandoned, error)

FindAbandoned is the dry run of SweepAbandoned: it returns what SweepAbandoned(ctx, before) would remove now, under the same rules, and removes nothing, so a maintenance job can report before it acts. The result is ordered by LastActivity, oldest first, then ID. Work that starts, completes or is aborted between the preview and the sweep makes them differ (see AbandonedFinder).

Needs the FindAbandoned capability; without it the call fails with CodeUnsupported. Like SweepAbandoned, a scoped or mirrored disk never has it: ask the disk it was derived from.

func (*Disk) Hold

func (d *Disk) Hold(ctx context.Context, key string) (Hold, error)

Hold returns key's hold. Needs the Hold capability.

func (*Disk) List

func (d *Disk) List(ctx context.Context, dir string, opts ListOptions) iter.Seq2[Entry, error]

List iterates over every entry under dir, fetching pages lazily; each page is one intercepted (and so retryable) driver call. Breaking out of the loop stops fetching. A failure is yielded once as (Entry{}, err) and ends the iteration.

for e, err := range disk.List(ctx, "avatars", filesystem.ListOptions{Recursive: true}) {
	if err != nil { return err }
	…
}

func (*Disk) ListPage

func (d *Disk) ListPage(ctx context.Context, dir string, opts ListOptions) (Page, error)

ListPage returns one page of the entries under dir (see ListOptions). dir is cleaned like a key, except that "" means the root.

func (*Disk) ListParts

func (d *Disk) ListParts(ctx context.Context, key, uploadID string) ([]Part, error)

ListParts lists the parts uploaded so far, in part-number order, so a client can resume an interrupted upload.

func (*Disk) MakeDir

func (d *Disk) MakeDir(ctx context.Context, dir string) error

MakeDir creates the directory dir (and any missing parents), succeeding when it already exists. dir is cleaned like a key; "" (the root) always exists, so it does nothing.

With the Dirs capability (local disk, and backends with real directories) the backend creates it: an empty directory DirExists reports until it is deleted. An object store has no directories, only keys that share a prefix, so without the capability MakeDir checks dir and does nothing else: no marker object is written, the directory appears when a key is written under it, and DirExists reports false until then. That is deliberate: a marker would be an object that List, DeletePrefix, quotas, mirrors and the records layer would all see, and that other tools reading the bucket would take for a file. Code that must see an empty directory checks Capabilities().Dirs.

Either way the call runs as one OpMakeDir, a mutation, so a read-only disk refuses it on every backend.

func (*Disk) Move

func (d *Disk) Move(ctx context.Context, src, dst string) (Entry, error)

Move renames src to dst on this disk, replacing dst. With the Move capability it is a backend rename; otherwise it is Copy then Delete of src, which is not atomic: a failure between the two leaves both keys.

func (*Disk) Name

func (d *Disk) Name() string

Name returns the disk's name.

func (*Disk) PresignPart

func (d *Disk) PresignPart(ctx context.Context, key, uploadID string, number int, expires time.Duration) (SignedRequest, error)

PresignPart returns a signed PUT for one part, for clients that upload parts directly to the backend. Needs the PresignPart capability.

func (*Disk) PresignPost

func (d *Disk) PresignPost(ctx context.Context, p PostPolicy) (PresignedPost, error)

PresignPost returns a form that lets a client upload straight to the backend under p, with every condition enforced by the backend. It needs the PresignPost capability (S3 and GCS); without it, CodeUnsupported. It counts as a mutation, so a read-only disk refuses it. The upload is still the client's: confirm it (upload.Confirm) before trusting it.

func (*Disk) PublicURL

func (d *Disk) PublicURL(ctx context.Context, key string) (string, error)

PublicURL returns a URL anyone can use to read key. With WithPublicURL it is the configured base plus the escaped key (no backend call); otherwise it needs the driver's PublicURL capability. It does not check that the object exists or is public: a URL for a private object simply fails when fetched.

func (*Disk) Read

func (d *Disk) Read(ctx context.Context, key string) (io.ReadCloser, error)

Read opens key for reading. The caller must close the reader.

func (*Disk) ReadAll

func (d *Disk) ReadAll(ctx context.Context, key string) ([]byte, error)

ReadAll reads the whole of key into memory. Use Read for large objects.

func (*Disk) ReadIf

func (d *Disk) ReadIf(ctx context.Context, key string, opts ReadOptions) (io.ReadCloser, Entry, error)

ReadIf reads key unless the caller's copy is current: it answers CodeNotModified, with no body, when opts.IfNoneMatch matches the object's ETag, or (with no IfNoneMatch) when the object was not modified after opts.IfModifiedSince. It returns the object's entry with the body, for the ETag and time the caller keeps for its next read. Drivers with the ConditionalRead capability decide in the backend; on others the Disk compares against a Stat before reading, which a concurrent overwrite between the two can outrun (the body is then newer than the entry). An object without an ETag never matches IfNoneMatch.

func (*Disk) ReadRange

func (d *Disk) ReadRange(ctx context.Context, key string, rng ByteRange) (io.ReadCloser, error)

ReadRange opens part of key for reading (see ByteRange). The caller must close the reader. Offset and Length must not be negative.

func (*Disk) ReadVersion

func (d *Disk) ReadVersion(ctx context.Context, key, id string) (io.ReadCloser, error)

ReadVersion opens version id of key, an ID Versions reported. The caller must close the reader. An empty id is CodeInvalidArgument. Needs the VersionRead capability.

func (*Disk) Restore

func (d *Disk) Restore(ctx context.Context, key string, opts RestoreOptions) error

Restore starts bringing archived key back online for opts.Days. Needs the Restore capability; Days must be positive (CodeInvalidArgument).

func (*Disk) RestoreVersion

func (d *Disk) RestoreVersion(ctx context.Context, key, id string) error

RestoreVersion makes version id of key its current content again, as a new latest version; no version is lost. An empty id is CodeInvalidArgument. Needs the VersionRead capability.

func (*Disk) Scope

func (d *Disk) Scope(name, prefix string) (*Disk, error)

Scope returns a Disk named name that sees only the keys under prefix on d: key "a.png" on the scoped disk is "<prefix>/a.png" on d. It shares d's driver, interceptors, URL signer and settings; a public URL base, if set, gains the prefix. Typical use is one scoped disk per tenant or feature.

The scoped disk needs its own name because signed URLs and logs identify a disk by name, and its keys differ from d's.

func (*Disk) SetHold

func (d *Disk) SetHold(ctx context.Context, key string, h Hold) error

SetHold sets key's hold: a legal hold, a retention date, both, or (the zero Hold) neither, where the backend allows lifting them. Needs the Hold capability.

func (*Disk) SetModTime

func (d *Disk) SetModTime(ctx context.Context, key string, t time.Time) error

SetModTime sets key's modification time to t, which Stat then reports (to the backend's precision, a second at worst). Needs the SetModTime capability (local, memory, SFTP); object stores, which stamp the upload time themselves, answer CodeUnsupported. A zero t is CodeInvalidArgument.

func (*Disk) SetStorageClass

func (d *Disk) SetStorageClass(ctx context.Context, key, class string) error

SetStorageClass moves key to storage class class, a name of the backend's own (S3 "STANDARD_IA", GCS "NEARLINE", Azure "Cool"); Stat then reports it in Entry.StorageClass. An empty class is CodeInvalidArgument; a class the backend does not know comes back from the driver. Needs the StorageClass capability.

func (*Disk) SetTags

func (d *Disk) SetTags(ctx context.Context, key string, tags map[string]string) error

SetTags replaces key's tags with tags (nil or empty removes them all). A tag with an empty key, or a key or value holding a control character, is CodeInvalidArgument; backend limits (S3 allows ten tags) come back from the driver. Needs the Tags capability.

func (*Disk) SetVisibility

func (d *Disk) SetVisibility(ctx context.Context, key string, v Visibility) error

SetVisibility changes the visibility of an existing key. It needs the SetVisibility capability.

func (*Disk) SignedURL

func (d *Disk) SignedURL(ctx context.Context, key string, opts PresignOptions) (SignedRequest, error)

SignedURL returns a pre-authorised URL for downloading (MethodGet) or uploading (MethodPut) key. It uses the disk's URLSigner for the methods WithURLSigner routes to it (by default every download, and uploads when the driver cannot presign), else the driver's own signing when it has the Presign capability, else fails with CodeUnsupported. A method routed to the signer never falls back to the driver. A PUT URL counts as a mutation, so a read-only disk refuses to issue one.

func (*Disk) Stat

func (d *Disk) Stat(ctx context.Context, key string) (Entry, error)

Stat returns the entry for key, or an error with CodeNotFound.

func (*Disk) SweepAbandoned

func (d *Disk) SweepAbandoned(ctx context.Context, before time.Time) (int, error)

SweepAbandoned removes what unfinished work left in the backend before before: multipart uploads never completed or aborted, and staging files of interrupted writes (see AbandonedSweeper for exactly what counts). It returns how many it removed. Stored objects are never touched; to delete uploaded objects nobody confirmed, use upload.Sweep on their prefix.

Choose before well beyond the longest upload the application allows (a day ago is usual): a recent before aborts uploads still in progress. The library starts no background goroutine, so schedule it like upload.Sweep.

Needs the SweepAbandoned capability; without it the call fails with CodeUnsupported. A scoped or mirrored disk never has it: sweep the disk it was derived from.

func (*Disk) Tags

func (d *Disk) Tags(ctx context.Context, key string) (map[string]string, error)

Tags returns key's tags. Needs the Tags capability.

func (*Disk) TempFile

func (d *Disk) TempFile(ctx context.Context, key string, opts TempFileOptions) (string, error)

TempFile copies key to a new file on local disk and returns its path, for tools that need a path rather than a stream (ffmpeg, libvips, pdftotext). The file keeps the key's extension when it is short and alphanumeric, since such tools often go by it, and is readable by the current user only.

The caller owns the file and must remove it (defer os.Remove(path)). On any error nothing is left behind: a failed or refused copy removes its partial file. Errors carry the read operation, this disk and the key.

func (*Disk) UploadPart

func (d *Disk) UploadPart(ctx context.Context, key, uploadID string, number int, body io.Reader, size int64) (Part, error)

UploadPart uploads one part through the server. A retry interceptor may retry it only when body implements io.Seeker and can actually seek; a body whose Seek fails (an *os.File on a pipe) is uploaded once as a stream, as in Write.

func (*Disk) Versions

func (d *Disk) Versions(ctx context.Context, key string) ([]Version, error)

Versions returns key's stored versions, newest first. Needs the Versions capability (a versioned bucket).

func (*Disk) Write

func (d *Disk) Write(ctx context.Context, key string, body io.Reader, opts WriteOptions) (Entry, error)

Write stores body under key, replacing any existing object, and returns the stored entry with Size and SHA256 computed by the Disk while streaming.

The Disk validates opts against the driver's capabilities first, so nothing is silently dropped: Metadata needs Capabilities.Metadata, Headers need Capabilities.Headers, IfNotExists needs Capabilities.ConditionalWrite, and a Visibility other than the disk default needs Capabilities.SetVisibility. When ContentType is empty it is detected from the first bytes and the key's extension.

A write is retried by a retry interceptor only when body implements io.Seeker and can actually seek (a pipe's *os.File cannot, so it is written once as a stream), because a consumed stream cannot be replayed.

func (*Disk) WriteBytes

func (d *Disk) WriteBytes(ctx context.Context, key string, b []byte, opts WriteOptions) (Entry, error)

WriteBytes stores b under key. It is Write with a rewindable body, so it is always retryable.

type Driver

type Driver interface {
	Stat(ctx context.Context, key string) (Entry, error)
	Read(ctx context.Context, key string, rng ByteRange) (io.ReadCloser, error)
	Write(ctx context.Context, key string, body io.Reader, opts WriteOptions) (Entry, error)
	Delete(ctx context.Context, key string) error
	List(ctx context.Context, dir string, opts ListOptions) (Page, error)
	Capabilities() Capabilities
}

Driver is the contract a storage backend implements. It is deliberately small: everything portable (key cleaning, content-type detection, hashing, size limits, fallbacks, URLs, interceptors) is the Disk's job, so a driver only moves bytes.

Contract every driver must honour, verified by the filesystemtest conformance suite:

  • Keys and directories arrive already cleaned (see CleanKey).
  • Stat and Read of a missing key return an error with CodeNotFound.
  • Write streams body to completion and never closes it; the caller owns it. If reading body fails, nothing is stored and the read error is returned. A successful Write replaces any existing object atomically: readers see the old bytes or the new bytes, never a mix.
  • Delete of a missing key succeeds.
  • List returns entries in ascending key order and honours Cursor and Limit; a directory with no keys lists as empty, not CodeNotFound.
  • Capabilities is constant for the driver's lifetime and true only for what the driver really does.

type Entry

type Entry struct {
	Key         string
	IsDir       bool
	Size        int64
	ModTime     time.Time
	ContentType string
	ETag        string
	SHA256      string
	// Checksums holds the extra digests WriteOptions.Checksums asked for,
	// by Checksum.Name, in lowercase hex; nil when none were asked for.
	Checksums  map[string]string
	Visibility Visibility
	Metadata   map[string]string
	// Headers are empty when the driver does not store them.
	Headers HTTPHeaders
	// StorageClass is the backend's name for the object's storage class
	// (S3 "STANDARD_IA", GCS "NEARLINE", Azure "Cool"), empty when the
	// driver does not report one or the backend leaves its default unnamed.
	// SetStorageClass changes it.
	StorageClass string
}

Entry describes one stored object, or one directory in a non-recursive listing.

Field presence depends on the source:

  • From Stat and Write: every field the backend can report. ContentType, ETag, Visibility and Metadata are empty when the driver does not store them (see Capabilities).
  • From a non-recursive List: a directory appears once with IsDir set, Key set to the directory's key, and every other field zero.
  • SHA256 is set by the Disk on a Write it streamed, and by drivers that store a SHA-256 (none of the built-in ones do on Stat).
  • Checksums is set only by the Disk on a Write that asked for more digests (WriteOptions.Checksums), never by Stat or List.

func CopyAcross

func CopyAcross(ctx context.Context, src *Disk, srcKey string, dst *Disk, dstKey string) (Entry, error)

CopyAcross copies srcKey on src to dstKey on dst. On the same disk it is Disk.Copy (server-side when supported); across disks it streams the bytes, keeping content type, and metadata and visibility where dst stores them.

func MoveAcross

func MoveAcross(ctx context.Context, src *Disk, srcKey string, dst *Disk, dstKey string) (Entry, error)

MoveAcross is CopyAcross followed by deleting the source. It is not atomic: if the delete fails, the object exists on both disks and the error is returned.

type Error

type Error struct {
	Code Code
	Op   OpKind
	Disk string
	Key  string
	Err  error
	// RetryAfter is how long the backend asked the caller to wait before
	// trying again (an HTTP Retry-After header on a 429 or 503, or the
	// vendor's equivalent); 0 means it did not say. A driver sets it on a
	// CodeUnavailable error with WithRetryAfter. layer.Retry waits at least
	// this long before the next attempt, and fshttp and upload pass it on to
	// their clients as a Retry-After header.
	RetryAfter time.Duration
}

Error is the one error type the library returns. Drivers create it with NewError; the Disk fills in Op, Disk and Key before returning it, so a log line always says which operation on which disk and key failed.

Err is the underlying cause and may be a native SDK error: errors.As reaches it through Unwrap, which is the escape hatch for vendor-specific error handling.

func Errorf

func Errorf(code Code, format string, args ...any) *Error

Errorf builds a classified error from a format string, for failures that have no underlying cause (validation in the Disk or a driver).

func NewError

func NewError(code Code, cause error) *Error

NewError is how a driver reports a classified failure, keeping the native cause (which may be nil) reachable through errors.As.

func (*Error) Error

func (e *Error) Error() string

Error renders "filesystem: <op> <disk>:<key>: <code>: <cause>", omitting the parts that are empty.

func (*Error) Is

func (e *Error) Is(target error) bool

Is makes errors.Is(err, ErrNotFound) true for every *Error whose Code is CodeNotFound, and likewise for each sentinel.

func (*Error) Unwrap

func (e *Error) Unwrap() error

Unwrap exposes the cause to errors.Is and errors.As.

func (*Error) WithRetryAfter

func (e *Error) WithRetryAfter(d time.Duration) *Error

WithRetryAfter records the backend's requested wait on e and returns e, so a driver can write

return filesystem.NewError(filesystem.CodeUnavailable, err).WithRetryAfter(d)

A non-positive d leaves e unchanged.

type FailoverEvent

type FailoverEvent struct {
	// Op is OpStat or OpRead.
	Op OpKind
	// Key is the key read.
	Key string
	// From names the disk that failed, To the disk tried next.
	From, To string
	// Err is From's error.
	Err error
}

FailoverEvent describes one read that moved from a disk that failed to the next one.

type FailoverOptions

type FailoverOptions struct {
	// ShouldFailover decides which errors move a read to the next disk.
	// nil means CodeUnavailable or CodeUnknown: a backend that is down or
	// answered with something unclassifiable. An answer such as
	// CodeNotFound or CodePermission is the disk's answer, not an outage,
	// and is returned as it is.
	ShouldFailover func(error) bool
	// OnFailover, when set, is called each time a read moves on, to count
	// or log outages of the primary.
	OnFailover func(ctx context.Context, e FailoverEvent)
}

FailoverOptions configure Failover. The zero value fails over on CodeUnavailable and CodeUnknown and reports nothing.

type HTTPHeaders

type HTTPHeaders struct {
	// CacheControl, such as "public, max-age=31536000, immutable" for
	// content-addressed keys.
	CacheControl string
	// ContentDisposition, such as `attachment; filename="report.pdf"`.
	ContentDisposition string
	// ContentEncoding, such as "gzip" for a pre-compressed object, which is
	// stored and read back exactly as written.
	ContentEncoding string
}

HTTPHeaders are stored with an object and sent with it when it is served (by the backend through its own URLs, or by fshttp), so a CDN caches it and a browser handles it as intended. Needs Capabilities.Headers. Values may not contain CR or LF, which would let a value inject another header when it is served.

func HTTPHeadersFrom

func HTTPHeadersFrom(src http.Header) HTTPHeaders

HTTPHeadersFrom reads the HTTPHeaders fields from an HTTP header set, such as the request of a signed PUT.

func (HTTPHeaders) IsZero

func (h HTTPHeaders) IsZero() bool

IsZero reports whether no header is set.

func (HTTPHeaders) SetOn

func (h HTTPHeaders) SetOn(dst http.Header)

SetOn writes every non-empty field into dst under its HTTP name, leaving the others as they are. Signers use it to list the headers a client must send with a presigned PUT (SignedRequest.Header); servers use it to send stored headers with an object.

type HTTPMethod

type HTTPMethod string

HTTPMethod is the method a signed request authorises.

const (
	MethodGet HTTPMethod = http.MethodGet
	MethodPut HTTPMethod = http.MethodPut
)

The closed set of methods a signed URL can authorise.

type HeaderPinner

type HeaderPinner interface {
	PinsHeaders() bool
}

HeaderPinner is implemented by a URLSigner whose PUT URLs pin PresignOptions.Headers: the URL carries them under its signature, and the server that accepts the upload refuses a request without them and stores them (signedurl.Signer with fshttp.Signed does). The Disk gives headers only to a signer whose PinsHeaders reports true, so a signer that does not know about them can never drop them silently.

type Hold

type Hold struct {
	// Legal protects the object until the hold is lifted, with no date.
	Legal bool
	// RetainUntil protects the object until this time; zero means none.
	// Backends refuse to shorten a retention already set in compliance
	// mode.
	RetainUntil time.Time
}

Hold is an object's protection from deletion and overwrite, as object lock implements it (S3 Object Lock, GCS holds and retention, Azure immutability policies).

type Holder

type Holder interface {
	Hold(ctx context.Context, key string) (Hold, error)
	SetHold(ctx context.Context, key string, h Hold) error
}

Holder is implemented by drivers that declare the Hold capability, so compliance code can protect an object without reaching for the backend's SDK. SetHold replaces the whole Hold (both fields); the Disk cleans the key and counts it as a mutation, so a read-only disk refuses it before the driver is called. A backend that refuses to lift or shorten a hold returns its error unchanged.

type IPRestricter

type IPRestricter interface {
	RestrictsIP() bool
}

IPRestricter is implemented by a URLSigner whose URLs carry PresignOptions.AllowedIP under their signature, and whose server refuses a request from an address outside it (signedurl.Signer with fshttp.Signed does). The Disk gives AllowedIP only to a signer whose RestrictsIP reports true, so a signer that does not know about it can never issue a URL that works from anywhere.

type Interceptor

type Interceptor func(ctx context.Context, op *Op, call Call) error

Interceptor runs around an operation. It must call call to proceed (zero, one or several times) and return its error or its own. It may inspect op before and after the call, but must not change Kind, Disk, Key or Target.

func ReadOnly

func ReadOnly() Interceptor

ReadOnly is an interceptor that refuses every mutating operation (see Op.Mutating) with CodeReadOnly before it reaches the driver. Signed GET URLs and public URLs still work; signed PUT URLs do not, because they authorise a write.

It lives in the core rather than the layer package because configuration (Mux's readonly parameter) needs it and the core cannot import layer.

type KeyEncoder

type KeyEncoder interface {
	EncodeKey(key string) string
	DecodeKey(name string) (string, error)
}

KeyEncoder maps keys to names a backend can store and back. Some backends refuse characters a key may hold (SFTP servers on Windows refuse : * ? < > | ", Azure refuses a blob name ending in a dot); a driver for such a backend holds a KeyEncoder, encodes every key it sends and decodes every name it lists. Keys reaching and leaving the Disk are never encoded, so applications and other drivers see the same keys.

DecodeKey(EncodeKey(k)) must return k for every clean key, and EncodeKey must not produce "/" or "..", so the encoded name keeps the key's segments.

type KeyError

type KeyError struct {
	Key string
	Err error
}

KeyError is one key a batch operation failed on.

type KeyEscaper

type KeyEscaper struct {
	Chars      string
	SegmentEnd string
}

KeyEscaper is the stock KeyEncoder: it percent-escapes (as "%3A") every character in Chars anywhere in a key, every character in SegmentEnd when it ends a segment (Windows and Azure refuse names ending in "." or " "), and '%' itself. The zero value escapes only '%'.

func (KeyEscaper) DecodeKey

func (e KeyEscaper) DecodeKey(name string) (string, error)

DecodeKey implements KeyEncoder. It accepts only names EncodeKey produces: a malformed escape, an escape EncodeKey would not write ("%41" for "A", lower-case hex, "%2F" for "/"), or a raw character it would have escaped is CodeInvalidKey, so two names can never decode to one key and a name can never gain a segment.

func (KeyEscaper) EncodeKey

func (e KeyEscaper) EncodeKey(key string) string

EncodeKey implements KeyEncoder.

type KeyMapper

type KeyMapper interface {
	InnerKey(key string) string
}

KeyMapper is implemented by wrapper drivers that change keys on the way to the driver they wrap (a scoped disk adds its prefix). Disk.DriverKey follows it, so a backend-specific option bound to one key (an S3 request option, say) can name the key the backend really receives.

type KeysDeleter

type KeysDeleter interface {
	DeleteKeys(ctx context.Context, keys []string) error
}

KeysDeleter deletes many keys in as few calls as the backend allows (S3 deletes 1000 per request). Keys are already cleaned. Deleting a missing key succeeds. A partial failure is a *BatchError listing the keys that were not deleted, with every other key deleted; any other error means the call failed and no key can be assumed deleted.

type ListOptions

type ListOptions struct {
	// Recursive lists every key under the directory. When false, keys in
	// sub-directories are folded into one IsDir entry per sub-directory.
	Recursive bool
	// Cursor resumes after a previous page: pass Page.NextCursor. Empty
	// starts from the beginning.
	Cursor string
	// Limit caps the entries per page. 0 means DefaultPageSize.
	Limit int
}

ListOptions control one page of a listing.

type LookupFunc

type LookupFunc func(key string) (string, bool)

LookupFunc reads one configuration value and reports whether it is set: os.LookupEnv in production, a map in tests. Taking it as an argument, rather than reading the process environment, keeps OpenManager free of hidden state.

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager holds an application's named disks and its default disk. It is a value the application owns, not a global, so tests and multiple apps in one process never share disks. It is safe for concurrent use.

func NewManager

func NewManager(disks ...*Disk) (*Manager, error)

NewManager returns a Manager holding disks. The first disk is the default until SetDefault changes it. Names must be unique.

func (*Manager) Add

func (m *Manager) Add(d *Disk) error

Add registers d under its name. A second disk with the same name is CodeExists. The first disk added becomes the default.

func (*Manager) Check

func (m *Manager) Check(ctx context.Context) error

Check runs Disk.Check on every disk and joins the failures, for a start-up or health probe.

func (*Manager) Close

func (m *Manager) Close() error

Close closes every disk and joins the failures. Disks that share a driver (scoped and mirror disks) may close it more than once; built-in drivers tolerate that.

func (*Manager) Default

func (m *Manager) Default() (*Disk, error)

Default returns the default disk, or CodeNotFound when none is registered.

func (*Manager) Disk

func (m *Manager) Disk(name string) (*Disk, error)

Disk returns the named disk, or CodeNotFound.

func (*Manager) Names

func (m *Manager) Names() []string

Names returns the registered disk names in sorted order.

func (*Manager) SetDefault

func (m *Manager) SetDefault(name string) error

SetDefault makes the named disk the default.

type ModTimeSetter

type ModTimeSetter interface {
	SetModTime(ctx context.Context, key string, t time.Time) error
}

ModTimeSetter sets an object's modification time, for tools that sync or archive by date (rsync-style mirroring, restoring a backup).

type Mount

type Mount struct {
	// Prefix is the directory the disk appears under: key "a.png" on Disk
	// is "<Prefix>/a.png" on the routed disk. "" mounts Disk at the root as
	// the catch-all for keys under no other prefix.
	Prefix string
	// Disk serves the keys under Prefix, with its own interceptors,
	// settings and capabilities.
	Disk *Disk
}

Mount places a disk at a directory of a routed disk (see Route).

type Mover

type Mover interface {
	Move(ctx context.Context, src, dst string) (Entry, error)
}

Mover renames an object inside the backend. dst is overwritten if present; src no longer exists afterwards.

type MultipartUploader

type MultipartUploader interface {
	CreateMultipart(ctx context.Context, key string, opts WriteOptions) (uploadID string, err error)
	UploadPart(ctx context.Context, key, uploadID string, number int, body io.Reader, size int64) (Part, error)
	ListParts(ctx context.Context, key, uploadID string) ([]Part, error)
	CompleteMultipart(ctx context.Context, key, uploadID string, parts []Part) (Entry, error)
	AbortMultipart(ctx context.Context, key, uploadID string) error
}

MultipartUploader uploads large objects in parts. Parts uploaded through the server use UploadPart; a driver that can also sign part uploads for clients implements PartPresigner, which together with this interface is the contract browser uploaders such as Uppy's S3 multipart plugin expect.

type Mux

type Mux struct {
	// contains filtered or unexported fields
}

Mux opens disks from DSN strings such as "s3://bucket?region=auto&prefix=tenant-a&visibility=public". Each driver package exports an Opener; the application registers the ones it imports.

Unlike a global registry filled by init functions, a Mux is a value: two parts of a program (or two tests) can hold different sets of drivers, and nothing is registered by importing a package.

func NewMux

func NewMux() *Mux

NewMux returns an empty Mux.

func (*Mux) Open

func (m *Mux) Open(ctx context.Context, name, dsn string, opts ...Option) (*Disk, error)

Open parses dsn, builds its driver with the registered opener and returns a Disk named name. opts are applied after the DSN's own settings, so code can add interceptors or a signer to a configured disk.

Every returned error names the disk ("disk media: …") and never contains the DSN itself, because DSNs often carry credentials. When the driver was built but the Disk could not be (an invalid Mux parameter value), the driver is closed so no connection leaks.

func (*Mux) OpenManager

func (m *Mux) OpenManager(ctx context.Context, prefix string, lookup LookupFunc, opts ...Option) (*Manager, error)

OpenManager builds a Manager from configuration, twelve-factor style:

STORAGE_DISKS=media,backups
STORAGE_MEDIA_DSN=s3://media?region=auto&visibility=public
STORAGE_BACKUPS_DSN=file:///var/backups
STORAGE_DEFAULT_DISK=media

with prefix "STORAGE_". opts apply to every disk. A missing or invalid setting is CodeInvalidArgument naming the variable, never its value (DSNs carry credentials), and the disks opened so far are closed. An error from opening a disk (an unknown scheme, a bad DSN parameter, a failed connection) keeps its code and names both the variable and the disk: "STORAGE_MEDIA_DSN: disk media: unknown DSN parameter …".

func (*Mux) Register

func (m *Mux) Register(scheme string, o Opener) error

Register adds an opener for a URL scheme. Registering a scheme twice is CodeExists.

type Op

type Op struct {
	// Kind is the operation.
	Kind OpKind
	// Disk is the name of the Disk running it.
	Disk string
	// Key is the object key, the directory for List, DeletePrefix,
	// MakeDir and DirExists, or empty for DeleteKeys and the abandoned
	// sweeps.
	Key string
	// Keys are the keys of a DeleteKeys, else nil.
	Keys []string
	// Target is the destination key of a Copy or Move, else empty.
	Target string
	// Mutating is true when the operation changes data or authorises a
	// change: Kind.Mutates(), plus a presign for MethodPut. Read-only
	// enforcement keys off this.
	Mutating bool
	// Replayable is true when the call may safely run more than once. False
	// for a write whose body cannot be rewound, so a retry interceptor must
	// not retry it.
	Replayable bool
	// Streaming is true when the call returns a stream that outlives it (a
	// Read): cancelling the context after the call returns would break the
	// stream, so timeout interceptors leave such calls alone.
	Streaming bool
	// Attempt counts how many times the call has started, from 1.
	Attempt int
	// Bytes is set after a successful Write to the number of bytes stored.
	Bytes int64
	// Hedgeable is true when the call may run concurrently with itself
	// (Stat, Read). An interceptor may call call again before an earlier
	// call returned only on a hedgeable operation, after setting
	// Concurrent, and must wait for every call it started before it
	// returns.
	Hedgeable bool
	// Concurrent is set by an interceptor (layer.Hedge) before it starts
	// calls of a hedgeable operation that may overlap. The Disk then keeps
	// the result of the first call to succeed, and a later call that also
	// succeeds discards its own result (closing its reader) and returns
	// ErrSuperseded, so the call that returned nil is always the one whose
	// result the Disk returns. Without it, calls are taken to run one
	// after another, and each success replaces the previous result,
	// closing its reader.
	Concurrent bool
	// contains filtered or unexported fields
}

Op describes one operation as it passes through the interceptor chain. Interceptors read it; the Disk owns it and updates the result fields.

func (*Op) OnClose

func (op *Op) OnClose(fn func())

OnClose registers fn to run once when the caller closes the reader the operation returns (a Read), so an interceptor can release what the stream needed until then, such as an attempt's context. If the operation fails, or returns no reader, fn runs when it returns. Call it before returning from the interceptor, never from a concurrent attempt.

func (*Op) WrapBody

func (op *Op) WrapBody(fn func(io.Reader) io.Reader)

WrapBody asks the Disk to pass the operation's byte stream through fn: the body of a Write, Append or UploadPart before the driver reads it, and the reader a Read returns before the caller reads it (its Close still closes the driver's reader). It is how an interceptor reaches bytes it otherwise never sees, to throttle or count them. Operations without a byte stream ignore it. fn must return a reader that yields exactly the bytes it was given; call WrapBody before calling call, because a body is opened when an attempt starts. An interceptor placed inside Retry runs, and registers fn, once per attempt, so fn should return a reader it already wrapped unchanged; one placed after a hedging interceptor must not call WrapBody at all, as hedged attempts run concurrently.

type OpKind

type OpKind string

OpKind names a Disk operation. Interceptors switch on it, errors carry it, and metrics label by it.

const (
	OpStat               OpKind = "stat"
	OpRead               OpKind = "read"
	OpWrite              OpKind = "write"
	OpDelete             OpKind = "delete"
	OpDeletePrefix       OpKind = "delete_prefix"
	OpDeleteKeys         OpKind = "delete_keys"
	OpList               OpKind = "list"
	OpCopy               OpKind = "copy"
	OpMove               OpKind = "move"
	OpSetVisibility      OpKind = "set_visibility"
	OpPresign            OpKind = "presign"
	OpPublicURL          OpKind = "public_url"
	OpMultipartCreate    OpKind = "multipart_create"
	OpMultipartUpload    OpKind = "multipart_upload_part"
	OpMultipartPresign   OpKind = "multipart_presign_part"
	OpMultipartListParts OpKind = "multipart_list_parts"
	OpMultipartComplete  OpKind = "multipart_complete"
	OpMultipartAbort     OpKind = "multipart_abort"
	OpSweepAbandoned     OpKind = "sweep_abandoned"
	OpFindAbandoned      OpKind = "find_abandoned"
	OpMakeDir            OpKind = "make_dir"
	OpDirExists          OpKind = "dir_exists"
	OpPresignPost        OpKind = "presign_post"
	OpAppend             OpKind = "append"
	OpSetModTime         OpKind = "set_mod_time"
	OpTags               OpKind = "tags"
	OpSetTags            OpKind = "set_tags"
	OpHold               OpKind = "hold"
	OpSetHold            OpKind = "set_hold"
	OpVersions           OpKind = "versions"
	OpRestore            OpKind = "restore"
	OpSetStorageClass    OpKind = "set_storage_class"
	OpReadVersion        OpKind = "read_version"
	OpRestoreVersion     OpKind = "restore_version"
)

The closed set of operations. Every Disk method that reaches a driver runs as exactly one of these (a portable fallback such as Copy without server side copy runs as the operations it is made of).

func (OpKind) Mutates

func (k OpKind) Mutates() bool

Mutates reports whether the operation changes stored data or grants a write. A presign is a mutation only when it authorises a PUT, so the Disk marks that on the Op (see Op.Mutating) rather than here; this method gives the answer for the kind alone.

type Opener

type Opener func(ctx context.Context, u *url.URL) (Driver, error)

Opener builds a driver from a parsed DSN. It receives the URL with the Mux's own query parameters already removed, and must reject any parameter it does not understand, so a typo in configuration fails at start-up instead of being ignored.

type Option

type Option func(*Disk) error

Option configures a Disk in NewDisk.

func WithContentDetector

func WithContentDetector(detect sniff.Detector) Option

WithContentDetector adds detect to the content-type detection of writes that do not declare a type, for formats the built-in tables do not know (DICOM, STL, camera RAW, Parquet). It is consulted only when the built-in answer is application/octet-stream, so it can never relabel a type sniff.Detect already recognises (HTML stays HTML); see sniff.DetectWith. It has no effect under WithoutContentTypeDetection. Uploads detect with upload.Policy.Detector, so set it there too.

func WithDefaultVisibility

func WithDefaultVisibility(v Visibility) Option

WithDefaultVisibility sets the visibility of writes that do not choose one. The default is VisibilityPrivate.

func WithInterceptors

func WithInterceptors(interceptors ...Interceptor) Option

WithInterceptors appends interceptors; the first given runs outermost.

func WithKeyNormalizer

func WithKeyNormalizer(fn func(string) (string, error)) Option

WithKeyNormalizer makes the Disk pass every key and directory through fn before cleaning it (CleanKey), so spellings fn treats as equal address one object: contrib/keynorm.NFC makes the NFC and NFD forms of "café.txt" one key, as macOS and Linux clients produce them. fn runs before the traversal check, so a normaliser that turns other characters into "." or "/" cannot smuggle a ".." past it. An error from fn is CodeInvalidKey. Every key this disk is given is normalised, so objects already stored under another spelling cannot be reached through it, not even to move them: set it on an empty disk, or first rename the old keys through a Disk over the same driver without the normaliser.

func WithPublicURL

func WithPublicURL(base string) Option

WithPublicURL makes PublicURL return base + "/" + key, typically a CDN origin. It takes precedence over the driver's own public URL. base must be an absolute http or https URL; a trailing slash is ignored.

func WithURLSigner

func WithURLSigner(s URLSigner, methods ...HTTPMethod) Option

WithURLSigner sets the signer SignedURL uses, typically a CDN's (contrib/cloudfront, contrib/cloudflare) or the HMAC signer of package signedurl.

Which methods it signs: with no methods given, the signer signs every download (MethodGet), even on a disk whose driver presigns (S3, GCS, Azure), so a CDN in front of a bucket hands out CDN links rather than raw bucket links; uploads (MethodPut) stay with the driver's own presigning, which sends the bytes straight to the backend, and go to the signer only when the driver cannot presign. Naming methods replaces that rule: the signer signs exactly those, and the driver (when it presigns) the rest, so WithURLSigner(s, MethodGet, MethodPut) routes uploads through the signer too, and WithURLSigner(s, MethodPut) keeps downloads on the driver.

Invariant: a method routed to the signer never falls back to the driver or to an unsigned link. A signer error, or a signer that cannot honour PresignOptions.Headers or AllowedIP, fails the call. An unknown method is CodeInvalidArgument.

func WithoutContentTypeDetection

func WithoutContentTypeDetection() Option

WithoutContentTypeDetection stops the Disk from detecting the content type of writes that do not declare one; they are stored without a type.

type Page

type Page struct {
	Entries    []Entry
	NextCursor string
}

Page is one page of a listing, in ascending key order. NextCursor is empty on the last page.

func PageOf

func PageOf(objects []Entry, dir string, opts ListOptions) Page

PageOf builds one listing page from every object under dir, for drivers that can enumerate their keys (memory, local disk) and so do not need to implement directory folding and cursors themselves. objects must hold only keys inside dir (see KeyInDir); their order does not matter.

It sorts by key, folds sub-directories into IsDir entries when the listing is not recursive, skips everything up to and including opts.Cursor, and cuts the page at opts.Limit (DefaultPageSize when 0). The cursor is the key of the last entry returned, so a listing stays correct when keys are added or deleted between pages.

type Part

type Part struct {
	Number int
	ETag   string
	Size   int64
}

Part identifies one uploaded part of a multipart upload. Number starts at 1; ETag is what the backend returned for the part and must be passed back unchanged to CompleteMultipart.

type PartPresigner

type PartPresigner interface {
	PresignPart(ctx context.Context, key, uploadID string, number int, expires time.Duration) (SignedRequest, error)
}

PartPresigner signs a PUT for one part of a multipart upload so a client can send it straight to the backend.

type PostPolicy

type PostPolicy struct {
	// Key is the exact key the upload is stored under. Set Key or
	// KeyPrefix, not both.
	Key string
	// KeyPrefix lets the client choose any key under this directory; the
	// key field is then KeyPrefix + "/" + PostFilenamePlaceholder, which
	// the client may keep (the backend substitutes the file's name) or
	// replace with any key under the prefix. Never the root.
	KeyPrefix string
	// ContentType, when set, is the only Content-Type accepted; the
	// client sends it as the PostFieldContentType field. Empty accepts any.
	ContentType string
	// MinSize and MaxSize bound the upload in bytes: MaxSize is required,
	// and 0 ≤ MinSize ≤ MaxSize.
	MinSize, MaxSize int64
	// Expires is the form's lifetime, in (0, MaxPresignExpiry].
	Expires time.Duration
	// Headers are stored with the object (and pinned by the policy); they
	// need Capabilities.Headers.
	Headers HTTPHeaders
	// Visibility is the visibility the uploaded object is stored with,
	// pinned by the policy (S3's and GCS's acl field), under the rules of
	// PresignOptions.Visibility: empty pins nothing (unchanged), and a
	// value the disk must pin needs Capabilities.PresignVisibility.
	Visibility Visibility
}

PostPolicy is what a presigned POST lets a client upload. Unlike a presigned PUT, the backend itself enforces every condition: the size range, the key, the content type, the stored headers and the expiry, so an oversized or mistyped upload is refused before it is stored.

type PostPresigner

type PostPresigner interface {
	PresignPost(ctx context.Context, p PostPolicy) (PresignedPost, error)
}

PostPresigner signs presigned POST forms. Options arrive validated and cleaned by the Disk. The returned Fields must include PostFieldKey, and PostFieldContentType when the policy fixes a type, plus whatever the backend needs (policy document, signature, credential).

type PrefixDeleter

type PrefixDeleter interface {
	DeletePrefix(ctx context.Context, dir string) error
}

PrefixDeleter deletes every key under a directory in as few calls as the backend allows. dir is clean and never the root: the Disk refuses to delete a whole disk through DeletePrefix.

type PresignOptions

type PresignOptions struct {
	// Method is MethodGet (download) or MethodPut (direct upload).
	Method HTTPMethod
	// Expires is the URL's lifetime: required, positive, at most
	// MaxPresignExpiry. There is deliberately no default, so every caller
	// decides how long a link may live.
	Expires time.Duration
	// ContentType: for a PUT, the type the upload must declare; for a GET,
	// an override for the response's Content-Type.
	ContentType string
	// ContentDisposition overrides the response's Content-Disposition on a
	// GET, e.g. `attachment; filename="report.pdf"`.
	ContentDisposition string
	// Headers, for a PUT only, are stored with the uploaded object, exactly
	// as WriteOptions.Headers are for a Write. The signed request pins
	// them: SignedRequest.Header lists them, the client must send them
	// unchanged, and the backend refuses an upload without them. The Disk
	// issues such a URL only when that is guaranteed: through the driver
	// when it declares Capabilities.PresignHeaders, through its URLSigner
	// when the driver stores headers (Capabilities.Headers) and the signer
	// implements HeaderPinner reporting true. Otherwise it is
	// CodeUnsupported, never a URL that silently loses them. Non-empty on
	// a GET is CodeInvalidArgument: a download overrides the response with
	// ContentDisposition instead.
	Headers HTTPHeaders
	// AllowedIP, when set, restricts the URL to clients whose address is
	// inside this CIDR prefix: "203.0.113.0/24", or "203.0.113.7/32" for
	// one address. It must be in canonical form (no bits set past the
	// prefix length), so what is signed is exactly what is meant. The Disk
	// issues such a URL only when the restriction is enforced: through the
	// driver when it declares Capabilities.PresignIP, through its URLSigner
	// when the signer implements IPRestricter reporting true. Otherwise it
	// is CodeUnsupported, never a URL that silently works from anywhere.
	AllowedIP string
	// Visibility, for a PUT only, is the visibility the uploaded object is
	// stored with, signed into the request (S3's x-amz-acl, GCS's
	// x-goog-acl), so the client cannot change it. Empty, the default,
	// signs nothing and the object gets whatever the backend gives an
	// upload that names none, exactly as before this field existed.
	//
	// It is checked like WriteOptions.Visibility: a value other than the
	// disk's default needs Capabilities.SetVisibility, and on a disk
	// without SetVisibility the default itself is accepted and signs
	// nothing (that backend stores every object at the default). On a
	// disk with SetVisibility any value needs a driver that pins it,
	// Capabilities.PresignVisibility, and a URLSigner never does:
	// otherwise it is CodeUnsupported, never a URL that silently stores
	// the object some other way. Non-empty on a GET is
	// CodeInvalidArgument.
	//
	// Who may ask for VisibilityPublic is the caller's decision: an
	// upload ticket handed to a browser lets its holder create a public
	// object, so upload.Presign issues one only under
	// upload.Policy.AllowPublic.
	Visibility Visibility
}

PresignOptions describe a signed URL.

type PresignedPost

type PresignedPost struct {
	URL       string
	Fields    map[string]string
	ExpiresAt time.Time
}

PresignedPost is a form a client posts to URL: every field in Fields, then the file in PostFileField. It expires at ExpiresAt.

type Presigner

type Presigner interface {
	Presign(ctx context.Context, key string, opts PresignOptions) (SignedRequest, error)
}

Presigner issues URLs signed by the backend itself (S3 SigV4, GCS V4). Options arrive validated by the Disk.

type PublicURLer

type PublicURLer interface {
	PublicURL(ctx context.Context, key string) (string, error)
}

PublicURLer returns the backend's own public URL for a key. A Disk configured with WithPublicURL ignores it in favour of the configured base (typically a CDN).

type ReadOptions

type ReadOptions struct {
	// IfNoneMatch is the ETag of the caller's copy; "*" matches any
	// existing object. When it matches, the read answers CodeNotModified.
	// It takes precedence over IfModifiedSince, as in HTTP.
	IfNoneMatch string
	// IfModifiedSince answers CodeNotModified when the object was not
	// modified after it (compared to the second, as HTTP dates are).
	IfModifiedSince time.Time
}

ReadOptions make a read conditional, as HTTP's conditional GET: the body is returned only when the object differs from the caller's copy.

type RestoreOptions

type RestoreOptions struct {
	// Days the restored copy stays readable before it returns to the
	// archive. Required, positive.
	Days int
}

RestoreOptions configure Restore.

type Restorer

type Restorer interface {
	Restore(ctx context.Context, key string, opts RestoreOptions) error
}

Restorer brings an archived object (S3 Glacier, Azure archive tier) back online for a while. The restore is asynchronous in the backend: Restore starts it, and reads succeed once the backend has finished.

type SignedRequest

type SignedRequest struct {
	Method    HTTPMethod
	URL       string
	Header    http.Header
	ExpiresAt time.Time
}

SignedRequest is a pre-authorised HTTP request a client can make without credentials. Header holds headers the client must send unchanged (some backends sign the content type of a PUT).

type StorageClasser

type StorageClasser interface {
	SetStorageClass(ctx context.Context, key, class string) error
}

StorageClasser moves an existing object to another storage class (S3 "STANDARD_IA" or "GLACIER", GCS "NEARLINE", Azure "Cool"), trading read cost and latency for storage cost. Class names are the backend's own and are passed through unchanged.

type SyncAction

type SyncAction string

SyncAction is what Sync does with one object. It is a closed set (SyncActions lists it), so a report consumer can switch on it exhaustively. There is no "skip" action: an unchanged object has no SyncItem and is only counted in SyncReport.Unchanged.

const (
	// SyncCopy copies a source object that is missing on the target or
	// differs from the target's object there.
	SyncCopy SyncAction = "copy"
	// SyncDelete deletes a target object with no source object at the
	// same relative key (SyncOptions.Delete only).
	SyncDelete SyncAction = "delete"
)

The closed set of sync actions.

type SyncItem

type SyncItem struct {
	Action SyncAction
	// Key is the object on the source for a copy, on the target for a
	// delete.
	Key string
	// Target is the key on the target disk: where a copy lands, or the
	// object a delete removes (then equal to Key).
	Target string
	// Size is the object's size: the source's for a copy, the target's
	// for a delete.
	Size int64
	// Err is why the action failed; nil when it succeeded and always nil
	// in a dry run.
	Err error
}

SyncItem is one object Sync acted on, or would act on in a dry run.

type SyncOptions

type SyncOptions struct {
	// Delete also deletes target objects under the target directory that
	// have no source object at the same relative key, making the target a
	// mirror. Off by default: a sync never deletes what it was not asked
	// to. It never deletes outside the target directory.
	Delete bool
	// DryRun reports every action through OnItem and in the report, and
	// changes nothing on either disk.
	DryRun bool
	// Same decides whether a target object already holds the source
	// object's content; nil means SameContent. It is the escape hatch
	// for backends whose ETags are not comparable but that keep a digest
	// of their own in Metadata.
	Same func(src, dst Entry) bool
	// Checksum reads both objects and compares their SHA-256 when Same
	// says they differ but their sizes are equal: the listings of many
	// backends (a local disk, or two backends whose ETags are computed
	// differently) cannot prove equal content, and without it every such
	// object is copied again on every run. It costs a full read of each
	// such pair; a read failure counts as different, so the object is
	// copied.
	Checksum bool
	// OnItem, when set, receives every action as it is done (or, in a dry
	// run, planned), failures included, in source listing order, then
	// the deletes in target listing order.
	OnItem func(SyncItem)
}

SyncOptions configure Sync. The zero value copies new and changed objects and deletes nothing.

type SyncReport

type SyncReport struct {
	Copied    int
	Deleted   int
	Unchanged int
	Failed    int
}

SyncReport counts what Sync did, or would do in a dry run.

func Sync

func Sync(ctx context.Context, src *Disk, srcDir string, dst *Disk, dstDir string, opts SyncOptions) (SyncReport, error)

Sync makes the objects under dstDir on dst match those under srcDir on src: every source object missing on the target, or whose target object differs (SyncOptions.Same), is copied to the same relative key under dstDir; with Delete, target objects under dstDir with no source object are deleted. Both listings are read in full before anything changes, so a listing failure stops Sync with nothing done, and objects it copies are never listed back. Each object is handled on its own: a failed copy or delete is reported (OnItem, SyncReport.Failed) and the rest go on, and the returned error says how many failed with the first cause. A rerun after a failure copies only what is still missing or different.

A delete is limited to keys inside dstDir, checked again before every deletion, so no listing quirk can make Sync delete outside it. When both disks reach the same backend (a scoped disk and the disk it was derived from, a routed or mirrored disk and one of its targets), the two directories as that backend sees them must not overlap in either direction (CodeInvalidArgument): the target would be part of the source, and a sync with Delete would remove the source while copying it. Delete is refused (CodeInvalidArgument) whenever either side is a Route, Overlay, Mirror, CachedRead or Failover disk, or wraps one, since such a disk can reach targets no check can see; sync the disks underneath it instead.

On a target with the CaseInsensitive capability, source objects whose keys differ only in case would overwrite each other there on every run: none of them is copied, and each is reported as a failed copy with CodeConflict naming the other. Matching target objects is then case-insensitive too, so Delete never removes the object a copy wrote.

type Tagger

type Tagger interface {
	Tags(ctx context.Context, key string) (map[string]string, error)
	SetTags(ctx context.Context, key string, tags map[string]string) error
}

Tagger stores key-value tags with an object, separately from its bytes and metadata, so they can change without rewriting it (S3 object tags, Azure blob index tags). Lifecycle rules and cost reports often key off them. SetTags replaces every tag.

type TempFileOptions

type TempFileOptions struct {
	// Dir is the directory the file is created in; "" means os.TempDir.
	Dir string
	// MaxSize, when positive, refuses an object larger than this many bytes
	// with CodeTooLarge, checked from Stat before anything is written and
	// again while copying, so an object that grew in between cannot fill
	// the disk.
	MaxSize int64
}

TempFileOptions configure Disk.TempFile. The zero value writes to the system temporary directory with no size limit.

type URLSigner

type URLSigner interface {
	SignURL(ctx context.Context, disk, key string, opts PresignOptions) (SignedRequest, error)
}

URLSigner signs URLs for disks whose driver cannot (local disk, memory), so that a signed download or upload link works the same in development as on S3. The signedurl sub-package provides an HMAC implementation and the fshttp sub-package serves its URLs.

type Unwrapper

type Unwrapper interface {
	Unwrap() Driver
}

Unwrapper is implemented by wrapper drivers (scoped, mirror) so that the typed escape hatch As can reach the backend driver underneath.

type Version

type Version struct {
	// ID names the version to the backend.
	ID      string
	Size    int64
	ModTime time.Time
	ETag    string
	// Latest is the version a plain Read returns.
	Latest bool
	// DeleteMarker records a deletion rather than content.
	DeleteMarker bool
}

Version is one stored version of an object, newest first in the slice Versions returns.

type VersionLister

type VersionLister interface {
	Versions(ctx context.Context, key string) ([]Version, error)
}

VersionLister lists the versions a versioned bucket keeps of an object.

type VersionReader

type VersionReader interface {
	ReadVersion(ctx context.Context, key, id string) (io.ReadCloser, error)
	RestoreVersion(ctx context.Context, key, id string) error
}

VersionReader reads one stored version of an object and makes a stored version the current content again, by the ID VersionLister reported. RestoreVersion keeps every version: the restored content becomes a new, latest version. A missing version, or a delete marker, is CodeNotFound.

type Visibility

type Visibility string

Visibility is the portable access level of a stored object. Each driver maps it to its own mechanism: file modes on local disk, ACLs or separate buckets on object stores.

const (
	// VisibilityPublic: anyone with the object's public URL may read it.
	VisibilityPublic Visibility = "public"
	// VisibilityPrivate: readable only through the backend's credentials or
	// a signed URL. The default for a Disk unless configured otherwise,
	// because exposing a file by accident is worse than having to opt in.
	VisibilityPrivate Visibility = "private"
)

The closed set of visibilities.

func (Visibility) Valid

func (v Visibility) Valid() bool

Valid reports whether v is one of VisibilityValues.

type VisibilitySetter

type VisibilitySetter interface {
	SetVisibility(ctx context.Context, key string, v Visibility) error
}

VisibilitySetter changes an existing object's visibility.

type WriteOptions

type WriteOptions struct {
	// ContentType is stored by drivers with Capabilities.ContentType. Empty
	// means the Disk detects it from the first bytes and the key's extension.
	ContentType string
	// Visibility overrides the Disk's default visibility. Empty means the
	// default.
	Visibility Visibility
	// Metadata is stored by drivers with Capabilities.Metadata; on any other
	// driver a non-empty map is CodeUnsupported rather than silently lost.
	Metadata map[string]string
	// Headers are stored with the object by drivers with
	// Capabilities.Headers; on any other driver non-empty headers are
	// CodeUnsupported rather than silently lost.
	Headers HTTPHeaders
	// IfNotExists makes the write fail with CodeExists when the key already
	// exists, atomically. Needs Capabilities.ConditionalWrite; without it
	// the write is refused with CodeUnsupported before any byte is read.
	//
	// The condition is checked when the object is committed, not before the
	// body is sent: the local, memory and S3 drivers (and GCS, Azure and
	// SFTP) read or upload the whole body first, so a losing writer streams
	// all of it before getting CodeExists. Stat first when bodies are large
	// and collisions likely; that is a hint, and IfNotExists stays the
	// guarantee.
	IfNotExists bool
	// IfMatch makes the write a compare-and-swap: it replaces the object
	// only while its ETag is still IfMatch (the one the caller read), and
	// fails with CodeConflict, writing nothing, when the object changed or
	// was deleted in between. Two writers that read the same version can
	// then never both win. Needs Capabilities.CompareAndSwap
	// (CodeUnsupported otherwise); not with IfNotExists, and not for
	// multipart uploads.
	IfMatch string
	// SizeHint is the body's length when known in advance. 0 means unknown
	// (an empty body is still written correctly). A positive hint is also a
	// declaration the Disk holds the body to: a body that ends early or
	// runs past it fails the write with CodeInvalidArgument and nothing is
	// stored, so a truncated upload (a dropped connection, a short
	// Content-Length) is never kept as if it were whole. Drivers may use it
	// to choose a single-request upload.
	SizeHint int64
	// SHA256, when set, is the hex SHA-256 the body must have (the client's
	// declared digest, or one computed upstream). The Disk hashes every
	// body anyway and refuses a mismatch with CodeInvalidArgument before
	// the driver commits, on every driver, so a corrupted body never
	// replaces what was stored. Drivers with Capabilities.ChecksumSHA256
	// also send it to the backend, which verifies it on its side of the
	// wire. Not for multipart uploads (CodeUnsupported).
	SHA256 string
	// Checksums are more digests the Disk computes while streaming, beside
	// SHA-256, each optionally verified (Checksum.Want), and reports in
	// Entry.Checksums: MD5, CRC32C, BLAKE3, whatever hash.Hash the caller
	// brings. A malformed list is CodeInvalidArgument. Not for multipart
	// uploads (CodeUnsupported). nil computes SHA-256 only.
	Checksums []Checksum
	// MaxSize, when positive, aborts the write with CodeTooLarge once the
	// body exceeds this many bytes. Enforced by the Disk while streaming, so
	// nothing larger is ever stored.
	MaxSize int64
}

WriteOptions control a write. The zero value writes with the Disk's default visibility and a detected content type.

Directories

Path Synopsis
adapters
s3 module
contrib
imaging module
svg module
Package filesystemtest holds the conformance suite every driver must pass and a Fake disk with assertions for application tests.
Package filesystemtest holds the conformance suite every driver must pass and a Fake disk with assertions for application tests.
Package fshttp serves disks over net/http: signed downloads and uploads (the URLs the signedurl package issues), and public files.
Package fshttp serves disks over net/http: signed downloads and uploads (the URLs the signedurl package issues), and public files.
internal
covmerge command
Command covmerge merges Go coverage profiles of the same packages run in different environments (the records suite on three database engines), counting a block covered when any run covered it.
Command covmerge merges Go coverage profiles of the same packages run in different environments (the records suite on three database engines), counting a block covered when any run covered it.
mdwrap command
Command mdwrap fails when a Markdown file contains hard-wrapped prose: a paragraph or list item broken across physical lines.
Command mdwrap fails when a Markdown file contains hard-wrapped prose: a paragraph or list item broken across physical lines.
Package layer holds the built-in interceptors: cross-cutting behaviour that wraps every Disk operation without wrapping the driver, so no capability is ever hidden (see filesystem.Interceptor and ADR-0003).
Package layer holds the built-in interceptors: cross-cutting behaviour that wraps every Disk operation without wrapping the driver, so no capability is ever hidden (see filesystem.Interceptor and ADR-0003).
Package local is a filesystem.Driver over a directory on local disk.
Package local is a filesystem.Driver over a directory on local disk.
Package memory is an in-process filesystem.Driver that keeps objects in a map.
Package memory is an in-process filesystem.Driver that keeps objects in a map.
records module
Package scan checks uploaded files for malware.
Package scan checks uploaded files for malware.
Package signedurl signs and verifies URLs for disks whose backend cannot sign its own (local disk, memory), so that signed downloads and direct uploads work the same in development as on S3.
Package signedurl signs and verifies URLs for disks whose backend cannot sign its own (local disk, memory), so that signed downloads and direct uploads work the same in development as on S3.
Package sniff detects a file's MIME type from its first bytes, refined by its file name only where the bytes cannot tell.
Package sniff detects a file's MIME type from its first bytes, refined by its file name only where the bytes cannot tell.
Package upload is how bytes arrive from browsers and phones: streaming server uploads, presigned direct uploads confirmed by the server, browser multipart uploads in the protocol Uppy's AWS S3 plugin speaks, and the sweeping of uploads nobody finished.
Package upload is how bytes arrive from browsers and phones: streaming server uploads, presigned direct uploads confirmed by the server, browser multipart uploads in the protocol Uppy's AWS S3 plugin speaks, and the sweeping of uploads nobody finished.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL