Help us shape the future of Go! Take the 2026 Go Developer Survey

storage

package
v0.43.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 24, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package storage provides the abstract Storage Driver interface used to front any byte-stream backend (local FS, S3, SFTP, WebDAV, …).

The base Driver interface is intentionally tiny — additional capabilities are advertised through optional sub-interfaces (Writer, Mover, Copier, Deleter, Mkdirer, Presigner, Watcher), AList-style. The Capabilities helper computes the runtime feature set by checking which sub-interfaces the concrete driver satisfies.

Index

Constants

View Source 
const (
	MetaLinkState = "link_state"
	// MetaLinkTarget carries the driver's own resolved identity for a FOLLOWED
	// directory link — the thing CycleGuard keys on. Only directory links need
	// it, because only a directory link can send a walk round in a circle, and
	// resolving one is the expensive call (measured 1.1 ms on Windows), so it
	// is not paid for file links or for links that were not followed.
	MetaLinkTarget = "link_target"

	// LinkFollowed — the link resolved inside the root and the Object
	// describes its TARGET. Carried so a surface can still badge the row as a
	// link; the Kind, size and mtime are the target's.
	LinkFollowed = "followed"
	// LinkOutsideRoot — the target is outside the storage root and the
	// storage's follow_symlinks option is off.
	LinkOutsideRoot = "outside_root"
	// LinkBroken — the link resolves to nothing at all.
	LinkBroken = "broken"
	// LinkUnresolved — the driver knows this is a link and deliberately did
	// not follow it, because it has no way to tell an in-root target from an
	// out-of-root one. Remote backends say this: filex's boundary there is the
	// account's own permissions, so following a link would leave the
	// configured root with nothing to stop the walk.
	LinkUnresolved = "unresolved"
)

Metadata key and values a driver uses to say WHY an entry came back as KindSymlink instead of as the thing it points at.

A driver that can follow a link reports the TARGET's kind, so an entry left as KindSymlink is always one the caller cannot open. Without a reason attached, the UI has a row it must refuse to act on and nothing to tell anyone — which is exactly how issue #34 was experienced: a 0-byte file that would not open and gave no explanation.

View Source 
const MaxWalkDepth = 64

MaxWalkDepth bounds every recursive walk in filex.

It is the backstop, not the guard: it catches a cyclic tree no driver reported a link for — a plugin backend, or a future driver that resolves links without saying so — and it catches it at a depth no real storage reaches. Deepest directory measured on the storages filex runs against is well under 20; a symlink cycle reached depth 81 on Linux and 127 on Windows before the OS's own link limit stopped it, so 64 sits clear of the first and well inside the second.

View Source 
const MinBackendPartSize = 5 * 1024 * 1024

MinBackendPartSize is the smallest non-final multipart part S3 accepts. Anything smaller is rejected with EntityTooSmall at CompleteMultipartUpload time — that is, after every byte has already been sent.

View Source 
const ScanExcludeKey = "scan_exclude"

ScanExcludeKey is the storage config key holding a storage's scan exclusions: glob patterns, one per line, for paths the scan must not walk. internal/scanrule reads it; the key lives here so the descriptor that offers it and the code that reads it name the same thing.

Variables

View Source 
var ErrKindConflict = errors.New("storage: path exists with a different kind")

ErrKindConflict is returned when a write target already exists as the OTHER kind: a file written onto an existing directory, or a directory created onto an existing file.

On a real filesystem the OS refuses this for us — os.Create on a directory fails, and the error surfaces. An object store has no such rule: keys are flat, so `X` (an object) and `X/y` (an object under a prefix) can both exist and the write succeeds silently. Hetzner Object Storage accepts it; MinIO, which is directory-backed, cannot represent it.

What that cost us (2026-08-06, brkip DR mirror): the mirror could never settle the colliding prefix, so `mc mirror` re-copied it every run — 2760 syncs in 24h, 1016 versions of a single PNG, a 43 MiB folder occupying 45 GB, and the disk at 96%. Worse and quieter: the colliding object made everything under that prefix unlistable, so 314 objects had no DR backup at all and nothing reported it.

View Source 
var ErrNotFound = errors.New("storage: not found")

ErrNotFound is returned by drivers when a path does not exist.

View Source 
var ErrReadOnly = errors.New("storage: read-only")

ErrReadOnly is returned when a write op hits a read-only mount.

View Source 
var ErrRootPathForbidden = errors.New("ROOT_PATH_FORBIDDEN: storage prefix/path cannot be empty or root '/'; use a sub-folder like 'fileman' or 'data/files'")

ErrRootPathForbidden is returned when a storage is configured with an empty or root path/prefix. Operators must always scope a Storage to a sub-folder (e.g. "fileman", "data/files") so that filemanager does not shadow or inadvertently take ownership of pre-existing files at the bucket / FS root.

View Source 
var ErrUnsupported = errors.New("storage: unsupported")

ErrUnsupported is returned when an op is not supported by this driver.

Functions

func ConfigInt 

func ConfigInt(v any) (int, bool)

ConfigInt reads an integer from a driver config value that may arrive as an int (YAML / direct), a float64 (JSON numbers unmarshal to float64), or a numeric string (form input). This is why e.g. an sftp "port" set via a JSON config still resolves correctly.

func ConfigLookup 

func ConfigLookup(cfg map[string]any, key string, aliases ...string) (any, bool)

ConfigLookup returns the first non-empty value for key or any alias. Empty strings count as absent — a blank text input is not a value.

func ConfigString 

func ConfigString(cfg map[string]any, key string, aliases ...string) string

ConfigString is ConfigLookup narrowed to strings.

func EmptyReadCloser 

func EmptyReadCloser() io.ReadCloser

EmptyReadCloser returns a ReadCloser that is immediately at EOF.

func EnsureDirTarget 

func EnsureDirTarget(ctx context.Context, d Driver, p string) error

EnsureDirTarget refuses to create a DIRECTORY at p when p already exists as a file — the same collision approached from the other side.

func EnsureFileTarget 

func EnsureFileTarget(ctx context.Context, d Driver, p string) error

EnsureFileTarget refuses to let a FILE be written at p when p already exists as a directory. Call it before every user-facing write.

This check only became expressible in v0.9.0: before it, the S3 driver's Stat answered ErrNotFound for a bare prefix, so "is this a folder?" had no answer on an object store. Stat now resolves a prefix to KindDirectory.

func Exists 

func Exists(ctx context.Context, d Driver, p string) bool

Exists reports whether anything occupies p on d. Only a definite ErrNotFound reads as free: any other Stat failure (a timeout, a 403, a 503) counts as TAKEN, because every caller asks in order to avoid putting bytes on top of something, and "I could not check" must not read as "nothing is there".

func Landed 

func Landed(ctx context.Context, d Driver, p string, size int64) (int64, string, time.Time)

Landed reads back what a write just left at p — its size, etag and modification time — from the driver's own Stat, for the caller to record on the node row.

⚠ The alternative every overwrite path used to take was to keep the row's etag, i.e. the etag of the file that had just been REPLACED. On a backend that reports etags (S3, WebDAV) that value is load-bearing: content search re-extracts a file's text only when its fingerprint moves, and the fingerprint is the etag (search.ContentFingerprint); clients tell a changed file from an unchanged one by the listing's etag; and a scan that finds the etag it recorded believes nothing happened.

⚠⚠ The mtime is the DRIVER's too, not time.Now(). The row is what every listing reports, and the storage scan later compares it with the driver: a handler-clock mtime that straddles a second boundary reads as drift and is rewritten, so the file's last_modified CHANGES with no byte of it changing (measured on a local storage: a desktop upload listed as ...692000 became ...692587 after one scan). A sync client sees that as a new server version and downloads its own upload back, and its `expect` precondition — the listing signature <size>:<mtime-ms> — no longer matches the file it wrote.

When Stat fails — or answers with something that is not a file — the answer is the caller's size, an EMPTY etag and now. An empty etag never matches the backend's on the next scan, so the row is corrected there, and the search fingerprint falls back to size+mtime at once. It is never the old etag.

func LimitReadCloser 

func LimitReadCloser(rc io.ReadCloser, n int64) io.ReadCloser

LimitReadCloser caps rc at n bytes while still closing the underlying stream — the shape every RangeReader needs when the backend can start at an offset but not stop at one (local seek, SFTP seek, FTP REST). n < 0 returns rc untouched ("to the end of the object").

func Names 

func Names() []string

Names returns the registered driver names sorted alphabetically.

func RefineOfficeMime 

func RefineOfficeMime(detected, filename string) string

RefineOfficeMime upgrades the generic "application/zip" detected by http.DetectContentType to the registered OOXML/ODF MIME when the file has an office extension. Returns the input unchanged otherwise.

Drivers and serving handlers call this BEFORE returning a stored or freshly-sniffed MIME to clients. The motivation: OnlyOffice Document Server refuses pptx/docx/odt sources when the fetch Content-Type is "application/zip" even though the JWT-signed config says fileType=pptx. xlsx happens to be lenient; the rest aren't.

func Register 

func Register(name string, f Factory)

Register adds a driver factory, called from drivers' init().

func RegisterDescriptor 

func RegisterDescriptor(d Descriptor)

RegisterDescriptor records a driver's config contract. Called from the driver package's init(), next to storage.Register.

func Unregister 

func Unregister(name string)

Unregister removes a driver factory. Built-in drivers never call this; it exists for storage PLUGINS (internal/plugin), which come and go while the process runs — an operator disables one, replaces its binary, or the process behind it is gone for good. Unknown names are a no-op.

⚠ Storages already opened on the driver keep their live instance; the server's resolver caches it. Unregistering only refuses NEW instances.

func UnregisterDescriptor 

func UnregisterDescriptor(name string)

UnregisterDescriptor forgets a driver's config contract — the plugin counterpart of Unregister. Unknown names are a no-op.

func ValidateNonRootPath 

func ValidateNonRootPath(driver string, cfg map[string]any) error

ValidateNonRootPath returns ErrRootPathForbidden if the configured driver path/prefix/root is empty or evaluates to the filesystem/bucket root after trimming whitespace and slashes.

The field to look at comes from the driver's Descriptor — the one field flagged Root, plus its legacy aliases. That is deliberate: this function used to carry its own per-driver key list, which is exactly how it drifted away from the admin form (the form collected "base_path", the validator read "root", every submit 400'd). One declaration, both sides.

Drivers without a descriptor keep the historical behaviour: 

s3              → "prefix"
local           → "path" (preferred), then "root"
ftp/sftp/webdav → "root" (preferred), then "remote_path"

Unknown drivers fall back to a sweep of the common keys so future drivers are protected by default. Callers should invoke this from API handlers (Storage create / update) BEFORE persisting the row.

Types

type Capabilities 

type Capabilities struct {
	Read bool `json:"read"`
	// Range reports ranged reads (storage.RangeReader). Callers that serve
	// HTTP bodies use it to decide between http.ServeContent (seekable,
	// 206/Content-Range) and a whole-object io.Copy.
	Range   bool `json:"range"`
	Write   bool `json:"write"`
	Move    bool `json:"move"`
	Copy    bool `json:"copy"`
	Delete  bool `json:"delete"`
	Mkdir   bool `json:"mkdir"`
	Presign bool `json:"presign"`
	Watch   bool `json:"watch"`
}

Capabilities advertises the Storage Driver's operation set.

func ComputeCapabilities 

func ComputeCapabilities(d Driver) Capabilities

ComputeCapabilities introspects a Driver to figure out which optional interfaces it implements. Drivers that lie in their static Capabilities() are still trumped by reality here.

type Collision 

type Collision struct {
	Path string `json:"path"`
	// Dir is the listing the pair was found in ("" = storage root).
	Dir string `json:"dir"`
}

Collision is one path that exists as BOTH a file and a folder.

func ScanKindCollisions 

func ScanKindCollisions(ctx context.Context, d Driver, root string) ([]Collision, error)

ScanKindCollisions walks root and reports every name that a listing returns twice — once as a file, once as a folder.

This finds damage that predates the write guards (EnsureFileTarget / EnsureDirTarget): an object store accepted `X` and `X/…` side by side, and nothing surfaced it. The consequences were only visible downstream — a DR mirror that re-copied the prefix on every run and, worse, a prefix whose contents no longer listed at all, so its objects were silently unbacked.

Detection is cheap because the collision is visible in a single listing: S3 returns `X` in Contents and `X/` in CommonPrefixes, so the driver yields two entries with the same name and different kinds. No full key enumeration is needed.

A directory that fails to list is skipped rather than aborting the scan — a colliding prefix is exactly the thing that may refuse to list, and giving up there would hide the rest of the storage.

type Copier 

type Copier interface {
	Copy(ctx context.Context, src, dst string) error
}

Copier copies objects (server-side if possible).

type CycleGuard 

type CycleGuard struct {
	// contains filtered or unexported fields
}

CycleGuard stops a recursive walk from following a symlinked directory back into a tree it is already inside.

⚠⚠ Why this had to exist BEFORE symlinked directories became walkable. Until v0.43.0 the catalogue walk descended only on Kind == KindDirectory and the local driver called every link KindSymlink, so a linked directory was catalogued as a childless leaf. That accident — not any guard — was the only thing keeping the walk finite. Measured with a wrapper that relabelled links as directories, on a root containing `real/cycle -> root`: 

Linux    depth 81,  123 visits
Windows  depth 127, 192 visits

before the operating system's symlink limit ended it. Every visit re-catalogued the same subtree under a different path, so each pass minted a fresh set of rows with distinct path hashes: the walk does not merely spin, it fills the catalogue with duplicates of real files.

⚠ A `seen` set keyed on the LOGICAL path does not help and looks like it does. The paths in a cycle are all different — /real, /real/cycle/real, /real/cycle/real/cycle/real — which is exactly why kindscan's existing seen[dir] map never stopped one.

The key is the driver's own resolved identity for the link (Object.Metadata under MetaLinkTarget), so two different paths naming one directory collapse to one entry. A driver that cannot resolve targets reports nothing, and such a walk is bounded by MaxWalkDepth alone.

⚠ A link is recorded when the walk DESCENDS into it, not when the root is opened, so a cycle is traversed once before it is recognised. That is deliberate and it is the cheap half of the trade: a link genuinely is a second view of that data, one extra pass is bounded work, and seeding the set would need a real identity for the root that only some drivers can give.

func NewCycleGuard 

func NewCycleGuard() *CycleGuard

NewCycleGuard returns a guard for one walk. It is not safe for concurrent use; each walk owns its own.

func (*CycleGuard) Enter 

func (g *CycleGuard) Enter(o Object, depth int) bool

Enter reports whether a walk at the given depth may descend into o, and records o as visited when it may.

depth is the recursion depth about to be entered (the storage root is 0).

type Deleter 

type Deleter interface {
	Delete(ctx context.Context, path string) error
}

Deleter removes objects (idempotent — no error on missing).

type Descriptor 

type Descriptor struct {
	Driver string `json:"driver"`
	// Label / I18nKey name the driver in a picker.
	Label   string  `json:"label"`
	I18nKey string  `json:"i18n_key"`
	Fields  []Field `json:"fields"`

	// Capabilities is the driver's runtime feature set (ComputeCapabilities
	// on a fresh instance). Surfaces want it in the same payload: a picker
	// can say "no presigned URLs" before anything is saved.
	Capabilities Capabilities `json:"capabilities"`

	// ScanFields are the settings a STORAGE on this driver has for the scan
	// that catalogues it (ScanFields()), as opposed to Fields, which the
	// driver's Init reads. They live in the same config map and are drawn by
	// the same form component, but only where a storage is edited: a
	// replication target is never scanned, so its dialog leaves them out.
	// Filled by Descriptors(); the registry does not hold them, which keeps
	// TestDescriptorMatchesInit about what Init reads.
	ScanFields []Field `json:"scan_fields,omitempty"`
}

Descriptor is a driver's complete, machine-readable config contract.

func DescriptorFor 

func DescriptorFor(name string) (Descriptor, bool)

DescriptorFor returns the descriptor for a driver name.

func Descriptors 

func Descriptors() []Descriptor

Descriptors returns every registered descriptor, driver-name sorted, with Capabilities computed from a fresh driver instance. Descriptors without a registered factory (test doubles) come back with the zero capability set rather than being dropped.

func (Descriptor) Field 

func (d Descriptor) Field(key string) (Field, bool)

Field returns the field with the given key.

func (Descriptor) Keys 

func (d Descriptor) Keys() []string

Keys returns every declared key plus its aliases — the complete set of config keys this driver understands.

func (Descriptor) MissingRequired 

func (d Descriptor) MissingRequired(cfg map[string]any) []string

MissingRequired lists required fields absent (or blank) from cfg, honouring aliases. Fields carrying a Default are never "missing" — the driver fills them in.

func (Descriptor) RootField 

func (d Descriptor) RootField() (Field, bool)

RootField returns the field flagged Root, if the driver has one.

type Driver 

type Driver interface {
	Init(ctx context.Context, cfg map[string]any) error
	Name() string
	List(ctx context.Context, path string) ([]Object, error)
	Stat(ctx context.Context, path string) (Object, error)
	Read(ctx context.Context, path string) (io.ReadCloser, error)
	Capabilities() Capabilities
}

Driver is the minimum surface a backend must implement. All paths use POSIX-style forward slashes and are relative to the storage root.

func Get 

func Get(name string) (Driver, error)

Get returns a freshly-constructed driver instance, error if unknown.

type Event 

type Event struct {
	Op   string // "create", "modify", "delete", "move"
	Path string
	From string // populated for move events only
}

Event is emitted by Watcher-capable drivers.

type EventNotifier 

type EventNotifier interface {
	NotifyReplicaFail(ctx context.Context, path, op string, err error, attempt int)
	NotifyPrimaryReadFail(ctx context.Context, path string, err error)
}

EventNotifier emits replica events to the notify subsystem. We keep this as a tiny interface so the storage package doesn't import notify (and notify can't import storage either — there's already a dep in the other direction via handlers).

type Factory 

type Factory func() Driver

Factory builds a fresh Driver instance — invoked once per Storage row.

type FailureRecorder 

type FailureRecorder interface {
	Record(ctx context.Context, path, op, errCode, errMsg string) error
	Resolve(ctx context.Context, path, op string) error
}

FailureRecorder is the persistence sink for replica errors.

The implementation lives in the replica/recorder.go subpackage and wraps db.Store; storage uses only this thin interface so it can avoid pulling the DB driver in tests.

type Field 

type Field struct {
	Key  string    `json:"key"`
	Type FieldType `json:"type"`

	// Label / Help are the English fallbacks. Surfaces resolve I18nKey /
	// HelpI18nKey first and only fall back to these — never render a
	// hardcoded English string when a translation exists.
	Label       string `json:"label"`
	Help        string `json:"help,omitempty"`
	I18nKey     string `json:"i18n_key"`
	HelpI18nKey string `json:"help_i18n_key,omitempty"`

	Required bool `json:"required"`
	// Secret marks credential material: surfaces must not echo it in
	// logs/URLs and should render a masked input.
	Secret bool `json:"secret"`

	Default     any            `json:"default,omitempty"`
	Placeholder string         `json:"placeholder,omitempty"`
	Options     []SelectOption `json:"options,omitempty"`
	Min         *int           `json:"min,omitempty"`
	Max         *int           `json:"max,omitempty"`

	// Monospace hints that the value is a path / URL / key blob.
	Monospace bool `json:"monospace,omitempty"`
	// Multiline hints at a textarea (PEM blobs).
	Multiline bool `json:"multiline,omitempty"`
	// Advanced fields are collapsed by default — rarely needed knobs.
	Advanced bool `json:"advanced,omitempty"`

	// Root marks THE field that scopes the storage inside the backend
	// (s3 prefix, local path, sftp/ftp/webdav root). ValidateNonRootPath
	// reads this field instead of its own hardcoded per-driver list.
	Root bool `json:"root,omitempty"`

	// Aliases are legacy keys Init still reads for this field.
	Aliases []string `json:"aliases,omitempty"`
}

Field describes exactly one key of a driver's config map.

Key is the wire name — what lands in storages.config_json and what the driver's Init reads. Aliases are historical spellings that Init still honours so rows written before this descriptor existed keep working.

func ScanFields 

func ScanFields() []Field

ScanFields returns the scan settings every storage has, whatever its driver. A fresh slice: callers cannot edit the set.

⚠ The help text says what the setting is NOT, on purpose: an excluded path is not walked, catalogued, indexed, thumbnailed or virus-scanned, but it is still on the storage and still served to whoever asks for it by path — the file protocols and the AI tools read the storage directly. It saves work; it is not an access control.

type FieldType 

type FieldType string

FieldType tells a surface which widget to render for a config field. 

const (
	FieldString   FieldType = "string"
	FieldInt      FieldType = "int"
	FieldBool     FieldType = "bool"
	FieldPassword FieldType = "password"
	FieldSelect   FieldType = "select"
)

Field types. Keep this list small — a surface that meets an unknown type should fall back to a plain text input rather than hide the field.

type Mkdirer 

type Mkdirer interface {
	Mkdir(ctx context.Context, path string) error
}

Mkdirer creates directories — drivers without a directory concept may no-op or persist a placeholder.

type Mover 

type Mover interface {
	Move(ctx context.Context, src, dst string) error
}

Mover renames/moves objects.

type MultipartUploader 

type MultipartUploader interface {
	InitMultipart(ctx context.Context, path string, totalSize int64, partCount int) (uploadID string, partURLs []string, err error)
	CompleteMultipart(ctx context.Context, path string, uploadID string, parts []PartCompletion) error
	AbortMultipart(ctx context.Context, path string, uploadID string) error
}

MultipartUploader is implemented by drivers (S3, GCS, Azure) that support resumable multipart uploads via presigned URLs.

type Object 

type Object struct {
	Path     string            `json:"path"` // logical path within storage (POSIX-style)
	Name     string            `json:"name"` // basename
	Size     int64             `json:"size"`
	Kind     ObjectKind        `json:"kind"`
	Mime     string            `json:"mime,omitempty"`
	Etag     string            `json:"etag,omitempty"`
	Mtime    time.Time         `json:"mtime,omitempty"`
	Metadata map[string]string `json:"metadata,omitempty"`
}

Object is a backend-agnostic representation of a single FS entry.

type ObjectKind 

type ObjectKind string

ObjectKind enumerates Object types. 

const (
	KindFile      ObjectKind = "file"
	KindDirectory ObjectKind = "dir"
	KindSymlink   ObjectKind = "symlink"
)

type PartCompletion 

type PartCompletion struct {
	PartNumber int
	Etag       string
}

PartCompletion is a finished multipart segment (stored in DB during upload).

type PartUploader 

type PartUploader interface {
	MultipartUploader
	UploadPart(ctx context.Context, path, uploadID string, partNumber int, r io.Reader, size int64) (etag string, err error)
}

PartUploader is MultipartUploader plus a SERVER-SIDE part upload.

MultipartUploader on its own is presign-only: it hands part URLs to the browser, which means filex never touches the bytes. The staged upload path (internal/staging, docs/UPLOADS.md) holds the bytes itself and must push them, so it needs a way to send one part from an io.Reader and get its ETag back.

⚠ The part boundaries here are the DRIVER's, not the client's. A client that staged 1 MiB chunks must not break an S3 backend, where every non-final part has to be at least MinBackendPartSize — so the commit step re-chunks.

type PresignedUpload 

type PresignedUpload struct {
	URL       string            `json:"url"`
	Method    string            `json:"method"`
	Headers   map[string]string `json:"headers,omitempty"`
	UploadID  string            `json:"upload_id,omitempty"` // for multipart
	PartSize  int64             `json:"part_size,omitempty"`
	PartCount int               `json:"part_count,omitempty"`
	PartURLs  []string          `json:"part_urls,omitempty"`
	ExpiresAt time.Time         `json:"expires_at"`
}

PresignedUpload holds the URL/method/headers a browser uses for direct uploads.

type Presigner 

type Presigner interface {
	PresignUpload(ctx context.Context, path string, size int64) (PresignedUpload, error)
	PresignDownload(ctx context.Context, path string, ttl time.Duration) (string, error)
}

Presigner returns URLs for browser-direct upload/download.

type RangeReader 

type RangeReader interface {
	ReadRange(ctx context.Context, path string, off, length int64) (io.ReadCloser, error)
}

RangeReader adds ranged (partial) reads, so a caller can fetch a byte window without pulling the whole object. Implemented only by backends that can genuinely start a transfer at an offset — a driver that silently returns the object from byte 0 would hand corrupt data to http.ServeContent, so leaving this unimplemented is always correct and callers must be able to fall back to Read.

Contract:

  • off is an absolute offset from the start of the object and must be >= 0; a negative off is an error (suffix ranges are resolved by the caller, which knows the size).
  • length < 0 means "to the end of the object". length == 0 yields a reader that is immediately at EOF, without touching the backend.
  • off at or past EOF is NOT an error: the returned ReadCloser reports io.EOF on the first Read. (S3 answers 416 for that case and the driver maps it back; local/SFTP get it for free.) The caller cannot always know the current size — it may have shrunk since Stat — so a short read is the honest answer, not a failure.
  • length beyond EOF is clamped by the backend: the reader ends early rather than erroring.
  • A missing path returns ErrNotFound, exactly like Read.

The returned ReadCloser must be closed by the caller.

type Reloader 

type Reloader interface {
	Reload() error
}

Reloader is the optional Reload method exposed by rulesEngine.

type ReplicaMode 

type ReplicaMode string

ReplicaMode is the return type of RuleEngine.Match. 

const (
	ModeMirror     ReplicaMode = "mirror"
	ModeAppendOnly ReplicaMode = "append_only"
	ModeSkip       ReplicaMode = "skip"
)

Mode values, mirroring model.ReplicaMode* but typed for the rules engine to avoid cross-package magic strings.

type ReplicatedDriver 

type ReplicatedDriver struct {
	// contains filtered or unexported fields
}

ReplicatedDriver is the wrapper Driver. Construct via NewReplicated.

func NewReplicated 

func NewReplicated(primary, replica Driver, rules RuleEngine, failures FailureRecorder, notifier EventNotifier) *ReplicatedDriver

NewReplicated wires a wrapper. replica may be nil — the wrapper then degenerates to a pure passthrough on writes (no fan-out).

rules MUST be non-nil; pass DefaultRules() if you don't have a configured engine yet (it returns mirror for everything).

func (*ReplicatedDriver) Capabilities 

func (r *ReplicatedDriver) Capabilities() Capabilities

Capabilities mirrors the primary's capabilities — replica fan-out is invisible to clients above the wrapper.

func (*ReplicatedDriver) Copy 

func (r *ReplicatedDriver) Copy(ctx context.Context, src, dst string) error

Copy clones on primary, then async clones on replica.

func (*ReplicatedDriver) Delete 

func (r *ReplicatedDriver) Delete(ctx context.Context, path string) error

Delete removes from primary. On success, an async goroutine removes from replica unless the path's rule is append_only or skip.

func (*ReplicatedDriver) HasReplica 

func (r *ReplicatedDriver) HasReplica() bool

HasReplica reports whether replication is configured.

func (*ReplicatedDriver) Init 

func (r *ReplicatedDriver) Init(_ context.Context, _ map[string]any) error

Init is a no-op — primary and replica are already initialized by the caller (server bootstrap).

func (*ReplicatedDriver) List 

func (r *ReplicatedDriver) List(ctx context.Context, path string) ([]Object, error)

List always reads from primary (source of truth). On primary error we DO NOT fall back to the replica because a replica list could disagree with primary's state (e.g. mid-replication holes).

func (*ReplicatedDriver) Mkdir 

func (r *ReplicatedDriver) Mkdir(ctx context.Context, path string) error

Mkdir creates the directory on primary, then on replica.

func (*ReplicatedDriver) Move 

func (r *ReplicatedDriver) Move(ctx context.Context, src, dst string) error

Move renames on primary, then re-issues the same move on replica asynchronously. Falls back to delete+write when the replica's driver doesn't support Mover.

func (*ReplicatedDriver) Name 

func (r *ReplicatedDriver) Name() string

Name returns the underlying primary's name (visible to operators).

func (*ReplicatedDriver) Primary 

func (r *ReplicatedDriver) Primary() Driver

Primary returns the underlying primary Driver — used by reconcile + report jobs that need direct access (e.g. counting objects).

func (*ReplicatedDriver) Read 

func (r *ReplicatedDriver) Read(ctx context.Context, path string) (io.ReadCloser, error)

Read tries primary, falls back to replica on error and emits a primary_read_fail notification.

func (*ReplicatedDriver) Replica 

func (r *ReplicatedDriver) Replica() Driver

Replica returns the underlying replica or nil.

func (*ReplicatedDriver) Stat 

func (r *ReplicatedDriver) Stat(ctx context.Context, path string) (Object, error)

Stat reads from primary, with replica fallback on error so the admin UI doesn't 404 if the primary is briefly down.

func (*ReplicatedDriver) Stop 

func (r *ReplicatedDriver) Stop()

Stop waits for in-flight async fan-outs.

func (*ReplicatedDriver) Write 

func (r *ReplicatedDriver) Write(ctx context.Context, path string, body io.Reader, size int64) error

Write writes to primary synchronously. On success, an async goroutine reads back from primary and writes to replica.

We deliberately do not buffer the request body to allow a single fan-out write — replica writes happen via a primary read-back so the streaming primary write doesn't have to fork.

type RuleEngine 

type RuleEngine interface {
	Match(path string) ReplicaMode
}

RuleEngine returns a ReplicaMode for the given path. Concrete impl is in rules.go.

func DefaultRules 

func DefaultRules() RuleEngine

DefaultRules returns a no-op engine (everything mirrors). Useful when callers haven't wired a real engine yet.

func NewRulesEngine 

func NewRulesEngine(reloadFn func() ([]RuleSpec, ReplicaMode)) RuleEngine

NewRulesEngine returns an engine that calls reloadFn lazily. If reloadFn is nil the engine returns ModeMirror for every path.

type RuleSpec 

type RuleSpec struct {
	ID       int64
	Pattern  string
	Mode     ReplicaMode
	Priority int
	Enabled  bool
}

RuleSpec is the declarative shape of one rule. It mirrors model.ReplicaRule but lives here so the storage package doesn't import model.

type SelectOption 

type SelectOption struct {
	Value string `json:"value"`
	// Label is the English fallback shown when I18nKey is missing from
	// the surface's catalogue.
	Label   string `json:"label"`
	I18nKey string `json:"i18n_key,omitempty"`
}

SelectOption is one choice of a FieldSelect field.

type Toucher 

type Toucher interface {
	SetMtime(ctx context.Context, path string, mtime time.Time) error
}

Toucher sets an object's modification time.

This is what carries a file's own timestamp through a transfer instead of stamping it with the moment it arrived. rclone, s3fs and the desktop sync all send the source mtime alongside the bytes (`x-amz-meta-mtime` over S3); a server that drops it reports every synced file as modified-now, and the next run copies the lot again.

⚠ Optional on purpose. An object store has no settable modification time, and a driver that accepted one and quietly ignored it would be worse than one that does not implement this at all: callers can see the difference between "not supported" and "supported and applied", and cannot see the difference between "applied" and "pretended".

type TreeWalker 

type TreeWalker interface {
	WalkTree(ctx context.Context, path string, fn func(Object) error) error
}

TreeWalker lets a backend hand over a whole subtree in ONE pass instead of being asked for every directory separately.

List is one round trip per directory. A poll over an object store with 150,000 objects in a few thousand prefixes therefore made a few thousand ListObjectsV2 calls — each a network round trip — and a scan took longer than its own interval (issue #33). An object store has no directories to descend into: a single un-delimited listing returns every key under a prefix, 1,000 per page, so the same tree costs ~150 calls.

Contract:

  • fn is called once per object below `path`, files AND the directories the driver derives from their paths (every ancestor exactly once, the driver's own markers such as an empty-folder key excluded). Order is the backend's; the caller groups by parent itself.
  • Object.Path is the full logical path (as List would report it); a directory's Kind is KindDirectory.
  • Returning a non-nil error from fn stops the walk and is returned as-is, so a caller can bound how much it is willing to hold.
  • A path that does not exist yields no calls and a nil error (an empty prefix is a listing, not a miss).

Drivers with real directories (local, sftp, …) do not implement this: a recursive readdir is not cheaper than the per-directory walk, and the walk keeps its readdir order guarantees.

type Watcher 

type Watcher interface {
	Subscribe(ctx context.Context) (<-chan Event, error)
}

Watcher streams change events from the backend.

type Writer 

type Writer interface {
	Write(ctx context.Context, path string, r io.Reader, size int64) error
}

Writer adds upload support.

Source Files

View all Source files

Directories

Path Synopsis
drivers
ftp
Package ftp is a Storage Driver fronting a plain FTP (or FTPS) server.
 Package ftp is a Storage Driver fronting a plain FTP (or FTPS) server.
local
Package local is a Storage Driver fronting an on-disk directory.
 Package local is a Storage Driver fronting an on-disk directory.
s3
Package s3 is a Storage Driver fronting any S3-compatible object store.
 Package s3 is a Storage Driver fronting any S3-compatible object store.
sftp
Package sftp is a Storage Driver fronting an SSH/SFTP server.
 Package sftp is a Storage Driver fronting an SSH/SFTP server.
smb
Package smb is a Storage Driver fronting an SMB2/3 share — a NAS, a Windows file server, a Samba box.
 Package smb is a Storage Driver fronting an SMB2/3 share — a NAS, a Windows file server, a Samba box.
webdav
Package webdav is a Storage Driver fronting a WebDAV server.
 Package webdav is a Storage Driver fronting a WebDAV server.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.