fsops

package
v1.34.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 30, 2026 License: GPL-2.0 Imports: 19 Imported by: 0

Documentation

Overview

Package fsops defines the Backend interface that abstracts filesystem operations for qui's services. Current implementations: Local (delegates to os.* on the qui host), Remote (SFTP over the instance's pooled SSH connection; reads only for now) and Noop (returns ErrNoFilesystemAccess for instances without filesystem access). Services resolve their backend per instance through Pool.GetBackend, so the transport is transparent to them.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrNoFilesystemAccess is returned by the NoopBackend for instances that
	// have no filesystem access configured (neither local nor remote).
	ErrNoFilesystemAccess = errors.New("filesystem access is not configured for this instance")

	// ErrNotLocal is returned by Pool.LocalBackend for an instance that no
	// longer has local filesystem access.
	ErrNotLocal = errors.New("instance does not have local filesystem access")

	// ErrNotCapable is returned by Pool.Require for an instance whose
	// filesystem mode does not grant the capability asked for.
	ErrNotCapable = errors.New("instance filesystem mode does not grant this capability")

	// ErrRemoteBackendNotWired is returned by a Pool built without a remote
	// factory for an instance in remote mode: a wiring mistake must fail
	// loudly rather than read as "not configured".
	ErrRemoteBackendNotWired = errors.New("remote filesystem backend is not wired into this pool")

	// ErrConnectionLost marks a remote read that failed because the backend
	// could not be reached (the transport dropped, or the pool would not
	// dial), not because of the path. It ends a walk as its one Err entry, and
	// a consumer that skips per-directory errors checks for it so that a walk
	// cut short is not read as complete. The cause is kept as text only, apart
	// from sshpool's own sentinels, so a lost connection never also matches
	// fs.ErrPermission or fs.ErrNotExist.
	ErrConnectionLost = errors.New("connection to the filesystem backend was lost")

	// ErrUnsupported reports a per-host fact: this server lacks the extension
	// the operation needs, or this transport has no such operation at all.
	// Distinct from ErrNoFilesystemAccess, which means nothing was configured.
	ErrUnsupported = fmt.Errorf("operation is not supported by this filesystem backend: %w", errors.ErrUnsupported)
)

Sentinel errors returned by Backend implementations.

Functions

func CleanPath added in v1.33.0

func CleanPath(d PathDialect, path string) string

CleanPath cleans a path without changing its casing. Uses the backend's path dialect and NFC unicode normalization to avoid mismatches between canonically-equivalent strings (e.g. composed vs decomposed forms on some platforms). Tradeoff: canonically-equivalent names collapse to one key, so byte-distinct NFC/NFD twins on normalization-sensitive filesystems are treated as one path. Use this when the result is shown to the user or stored; use NormalizePath when the result is only compared against another path.

func IsDirNotEmpty added in v1.33.0

func IsDirNotEmpty(err error) bool

IsDirNotEmpty reports whether err comes from removing a folder that is not empty. It reads only error codes, never message text: Windows translates its messages, and a folder named "Not Empty" puts those words in the path. Windows reports ERROR_DIR_NOT_EMPTY, which is not syscall.ENOTEMPTY there, so that code is checked on Windows. Every caller passes an error from a backend's Remove, and both backends report the code: the local one from os.Remove, the remote one as syscall.ENOTEMPTY when a listing shows the folder still holds entries.

func NormalizePath added in v1.33.0

func NormalizePath(d PathDialect, path string) string

NormalizePath cleans a path and case-folds it for consistent comparison. Keys keep d's separators: a slash key would merge a UNC path with a rooted one.

Case-folding is unconditional because runtime.GOOS cannot answer whether the scanned filesystem is case-insensitive: macOS APFS and Windows volumes are case-insensitive by default, and Docker bind-mounts them into a Linux container, where qBittorrent can report two save paths that differ only by case for one physical directory.

Keep the fold stateless. Orphan scan fills its file map before the walk and runs one goroutine per instance, so a per-scan or global fold flag that differs between Add and Has turns every file into an orphan.

Invalid UTF-8 is folded too. strings.ToLower and encoding/json replace a bad byte the same way, so the fold is what lets a name read from disk match the same name after a round trip through the qBittorrent API. Do not add a utf8.ValidString guard here: an owned file whose name carries a non-UTF-8 byte is then reported as an orphan and deleted.

ToLower can turn an NFC string into a non-NFC one (H + U+0331 becomes U+1E96), so the folded result is cleaned again.

ponytail: names that differ only by case, or only in which invalid bytes they carry, collapse to one key. That direction is safe (a missed orphan, never a deleted owned file). Upgrade path if it is ever reported: keep a folded secondary index and confirm hits with os.SameFile.

The result is for comparison only. It must never reach an os.* call or the database, because on a case-sensitive filesystem it can name a different file.

func SameFile added in v1.33.0

func SameFile(a hardlink.FileID, aInstance *models.Instance, b hardlink.FileID, bInstance *models.Instance) bool

SameFile reports whether two IDs name one file. IDs of different kinds never match, and an ID with no key matches nothing.

Types

type Backend

type Backend interface {

	// Stat returns metadata for a single path, following symlinks — FileID
	// and Nlinks describe the target, so symlinked torrent data keeps its
	// identity. Returns a non-nil error wrapping fs.ErrNotExist if the path
	// does not exist. A failure to resolve identity is reported in
	// LstatInfo.FileIDErr, not as a Stat error, so one identity-opaque file
	// cannot fail callers that only need metadata.
	Stat(ctx context.Context, path string) (*LstatInfo, error)

	// Lstat is like Stat but does not follow symlinks.
	Lstat(ctx context.Context, path string) (*LstatInfo, error)

	// ReadDir returns directory entries.
	ReadDir(ctx context.Context, path string) ([]DirEntry, error)

	// WalkDir walks a directory tree and streams entries on the returned channel.
	// Entries within one directory arrive in lexical order and a directory's
	// own entry precedes everything beneath it; nothing more is promised about
	// order, as a backend may list sibling directories concurrently. The
	// channel is closed when the walk completes, is cancelled via ctx, or
	// hits an unrecoverable error. Callers must drain the channel or cancel
	// ctx; abandoning it leaks the walk goroutine. Entries whose metadata
	// cannot be read are skipped, not emitted — WalkEntry.Err carries only
	// enumeration-level walk failures. Setting opts.EmitStatErrors emits
	// those entries instead, with WalkEntry.StatErr set and only Path/RelPath
	// valid, so delete preflights can fail closed on unverifiable paths.
	WalkDir(ctx context.Context, root string, opts WalkOptions) (<-chan WalkEntry, error)

	// Statfs returns free/total bytes for the filesystem containing path.
	Statfs(ctx context.Context, path string) (*StatfsResult, error)

	// SameFilesystem returns true if both paths reside on the same filesystem
	// (same device ID on Unix, same volume serial on Windows).
	SameFilesystem(ctx context.Context, p1, p2 string) (bool, error)

	// MkdirAll creates a directory and all parents. Equivalent to os.MkdirAll.
	// For torrent content and link-tree dirs, pass fsutil.ContentDirMode /
	// fsutil.LinkTreeBaseDirMode rather than a hand-typed mode (#1704, #2086).
	MkdirAll(ctx context.Context, path string, perm fs.FileMode) error

	// Mkdir creates one directory, like os.Mkdir. The error wraps fs.ErrExist
	// when path already exists, so of two calls for one path only one
	// succeeds. Link trees use it to reserve a folder name.
	Mkdir(ctx context.Context, path string, perm fs.FileMode) error

	// Remove removes a file or directory. If opts.Recursive is true, removes
	// the entire tree (like os.RemoveAll).
	Remove(ctx context.Context, path string, opts RemoveOptions) error

	// HardlinkTree creates a hardlink tree from plan. Rolls back what it
	// created on partial failure. The result records the files and dirs this
	// call made, for a later RemoveTree.
	//
	// Every implementation owes the same answer to a target that already
	// exists: skip it when it is provably the same file as its source, and
	// refuse the whole plan with hardlinktree.ErrTargetExists otherwise. A
	// skipped target is counted in SkippedExists and never recorded in Files,
	// because it belongs to whoever made it first; see
	// TestHardlinkTree_ExecTierSkippedLinkSurvivesLaterFailure. A backend that
	// cannot prove identity refuses.
	HardlinkTree(ctx context.Context, plan *hardlinktree.TreePlan) (*TreeCreateResult, error)

	// ReflinkTree creates a reflink (CoW) tree from plan. Rolls back what it
	// created on partial failure. The result records the files and dirs this
	// call made, for a later RemoveTree.
	ReflinkTree(ctx context.Context, plan *hardlinktree.TreePlan) (*TreeCreateResult, error)

	// RemoveTree removes exactly the files and dirs recorded in created —
	// never the whole plan, which could delete links shared with sibling
	// torrents (discussion #2282). A plan root that already existed before
	// the create is therefore NOT removed; callers own pruning it. Safe to
	// call with a nil result.
	RemoveTree(ctx context.Context, created *TreeCreateResult) error

	// Paths is the dialect of this backend's filesystem. Every Join, Dir,
	// IsAbs or Rel on a path that goes to or comes from this backend uses it.
	Paths() PathDialect

	// SupportsReflink returns whether the filesystem at path supports CoW
	// reflinks. The string return is a human-readable reason when unsupported.
	SupportsReflink(ctx context.Context, path string) (bool, string, error)
}

Backend abstracts filesystem operations so services work identically against a local filesystem or an SSH-backed remote. It covers exactly the operations qui's services need: syscall-level primitives (stat, walk, mkdir, remove) plus the high-level tree operations (HardlinkTree, ReflinkTree, RemoveTree) that create-and-rollback as a unit, and the path dialect (Paths) that callers manipulate backend paths with, since a remote backend's paths are not in the host's grammar.

Every method accepts a context.Context and must respect cancellation.

Error semantics are portable across implementations: a missing path is reported compatibly with errors.Is(err, fs.ErrNotExist) and a denied one with errors.Is(err, fs.ErrPermission) — from Stat, Lstat, ReadDir, WalkDir (on entries), Remove, and Statfs alike. Remote backends must map their transport's errors onto the same sentinels.

type BatchStater added in v1.33.0

type BatchStater interface {
	// StatBatch is Stat for each path: infos[i] and errs[i] answer paths[i].
	StatBatch(ctx context.Context, paths []string) (infos []*LstatInfo, errs []error)
	// LstatBatch is Lstat for each path: infos[i] and errs[i] answer paths[i].
	LstatBatch(ctx context.Context, paths []string) (infos []*LstatInfo, errs []error)
}

BatchStater is a Backend that answers many paths in one request, so a remote host costs a few round trips per batch and not a few per path.

type DirEntry

type DirEntry struct {
	Name      string
	IsDir     bool
	IsSymlink bool
	// Mode contains only the entry's type bits, without following links.
	Mode fs.FileMode
}

DirEntry holds one entry from a ReadDir call.

type FileInfo

type FileInfo struct {
	Path      string
	Size      int64
	ModTime   time.Time
	IsDir     bool
	IsSymlink bool
	Mode      fs.FileMode
}

FileInfo holds metadata from a Stat call.

type FileKey added in v1.33.0

type FileKey struct {
	// contains filtered or unexported fields
}

FileKey is a FileID bound to the filesystem it came from. Two keys are equal exactly when SameFile holds, so an index that mixes instances keys on it instead of on the bare FileID.

func FileKeyOf added in v1.33.0

func FileKeyOf(id hardlink.FileID, instance *models.Instance) (key FileKey, ok bool)

FileKeyOf binds id to the instance that reported it. Local kinds all live on the qui host, so they share one scope; a remote kind is scoped to the SSH host and port, the same-filesystem rule orphan scan merges by. A nil instance is the qui host itself. ok is false when id names no file: a zero ID, or a remote ID with a nil instance or an empty SSH host, which has no scope. Such a key matches nothing, so a caller must not index it.

type LstatInfo

type LstatInfo struct {
	FileInfo
	FileID hardlink.FileID
	Nlinks uint64
	// FileIDErr is set when FileID/Nlinks could not be resolved (e.g. a
	// Windows ACL denial on an otherwise statable file). The rest of the
	// struct is still valid; callers that need identity must check this
	// instead of treating the whole stat as failed.
	FileIDErr error
}

LstatInfo holds metadata from an Lstat call, including hardlink identity.

func LstatAll added in v1.33.0

func LstatAll(ctx context.Context, b Backend, paths []string) (infos []*LstatInfo, errs []error)

LstatAll is Lstat for each path: infos[i] and errs[i] answer paths[i]. A BatchStater answers in one call, and any other backend gets one Lstat per path.

func StatAll added in v1.33.0

func StatAll(ctx context.Context, b Backend, paths []string) (infos []*LstatInfo, errs []error)

StatAll is Stat for each path: infos[i] and errs[i] answer paths[i]. A BatchStater answers in one call, and any other backend gets one Stat per path.

type PathDialect added in v1.31.0

type PathDialect interface {
	// Join takes two parts, which is every backend-path join in qui; a
	// variadic slice would escape through the interface call on each file.
	Join(base, name string) string
	Dir(p string) string
	Base(p string) string
	Clean(p string) string
	IsAbs(p string) bool
	Rel(base, target string) (string, error)
	// FromSlash and ToSlash convert between slash-delimited names, which is
	// how qBittorrent reports every file, and the dialect's own form.
	FromSlash(p string) string
	ToSlash(p string) string
	Separator() string
}

PathDialect is the path grammar of a backend's filesystem. A path that goes to or comes from a Backend is manipulated with that backend's dialect, never with the host's filepath: a remote unix path on a Windows host is still slash-delimited and still absolute. Ext is absent on purpose, it reads the same under both grammars.

var HostPaths PathDialect = hostPaths{}

HostPaths is the dialect of the host filesystem: path/filepath.

var SlashPaths PathDialect = slashPaths{}

SlashPaths is the dialect of a slash-delimited POSIX filesystem regardless of the host: path. FromSlash and ToSlash are the identity.

type Pool

type Pool struct {
	// contains filtered or unexported fields
}

Pool resolves an instance ID to the appropriate Backend. For instances with local filesystem access it returns the local backend; for instances pinned for SSH access it returns a remote backend built for that instance; for instances without access it returns a noop backend that errors on every call.

func NewPool

func NewPool(store instanceGetter, local Backend) *Pool

NewPool creates a Backend pool backed by the given instance store and local backend. Without a remote factory a remote-mode instance is an error, not a silent "not configured": NewPoolWithRemote is the constructor that wires one.

func NewPoolWithRemote added in v1.31.0

func NewPoolWithRemote(store instanceGetter, local Backend, remote func(*models.Instance) Backend) *Pool

NewPoolWithRemote is NewPool plus a remote-backend factory, called per resolution: the backend it returns is a thin handle over the shared SSH connection pool, which owns the connections and their lifetime.

func (*Pool) GetBackend

func (p *Pool) GetBackend(ctx context.Context, instanceID int) (Backend, error)

GetBackend returns the appropriate Backend for the given instance ID. Returns ErrNoFilesystemAccess wrapped in a noop backend for instances without filesystem access configured.

func (*Pool) LocalBackend added in v1.31.0

func (p *Pool) LocalBackend(ctx context.Context, instanceID int) (Backend, error)

LocalBackend is Resolve for a caller that admitted the instance on local access from an earlier read of it. Any other mode fails with ErrNotLocal, so a caller whose read went stale never runs local paths against another host.

func (*Pool) Require added in v1.31.0

func (p *Pool) Require(ctx context.Context, instanceID int, capability models.FilesystemCapability, more ...models.FilesystemCapability) (Backend, *models.Instance, error)

Require returns the backend and the instance row for an instance whose mode grants every capability, both from one read, so a caller cannot check one row and act on another. Any other instance fails with ErrNotCapable before a backend is built.

func (*Pool) Resolve added in v1.31.0

func (p *Pool) Resolve(ctx context.Context, instanceID int) (Backend, models.FilesystemMode, error)

Resolve is GetBackend plus the mode the backend was chosen for, both from one instance read, so a caller that records the mode cannot label a walk with a mode the instance changed to in between.

type RemoveOptions

type RemoveOptions struct {
	Recursive bool
}

RemoveOptions controls the behavior of a Remove call.

type StatfsResult

type StatfsResult struct {
	BytesAvailable int64
	BytesTotal     int64
}

StatfsResult holds filesystem space information.

type TreeCreateResult

type TreeCreateResult struct {
	Created       int
	SkippedExists int
	Files         []string
	Dirs          []string
}

TreeCreateResult holds the outcome of a HardlinkTree or ReflinkTree call. Files and Dirs record what the call actually created on disk (pre-existing paths are excluded), so RemoveTree can undo exactly this call's work without touching links shared with sibling torrents (discussion #2282).

type WalkEntry

type WalkEntry struct {
	LstatInfo
	RelPath string
	Err     error
	// StatErr is set when the entry was enumerated but its metadata lookup
	// failed; only Path and RelPath are valid then. Such entries are emitted
	// only when WalkOptions.EmitStatErrors is set and skipped otherwise.
	StatErr error
}

WalkEntry is emitted by WalkDir for each filesystem entry encountered.

type WalkOptions

type WalkOptions struct {
	SkipHidden bool
	// IgnoreDirNames are directory basenames to skip, matched case-insensitively
	// (OS/NAS metadata dirs like $RECYCLE.BIN and @eaDir vary in on-disk case).
	IgnoreDirNames []string
	// IgnoreDirNamePrefixes are matched case-insensitively.
	IgnoreDirNamePrefixes []string
	IgnorePaths           []string
	// WantFileID populates FileID and Nlinks on regular-file entries.
	WantFileID bool
	// EmitStatErrors emits entries whose metadata lookup failed with StatErr
	// set instead of silently skipping them. Consumers that delete based on
	// walk results use this to fail closed on paths they cannot verify.
	EmitStatErrors bool
	// FollowRoot walks a root that is a symlink to a directory as that
	// directory. Entries are reported under the root as given, and symlinks
	// below the root are still not followed.
	FollowRoot bool
}

WalkOptions controls the behavior of a WalkDir call.

func (*WalkOptions) Skip added in v1.31.0

func (o *WalkOptions) Skip(name, fullPath string, isDir, isRoot bool) bool

Skip reports whether a walk leaves out an entry, and its subtree when the entry is a directory. fullPath is in the backend's own path dialect, since IgnorePaths is an exact string compare. The walk root is exempt from every filter except IgnorePaths, so an ignored root gives an empty walk.

Directories

Path Synopsis
Package local implements fsops.Backend by delegating to the local filesystem via os.*, pkg/hardlinktree, pkg/reflinktree, pkg/hardlink, and pkg/fsutil.
Package local implements fsops.Backend by delegating to the local filesystem via os.*, pkg/hardlinktree, pkg/reflinktree, pkg/hardlink, and pkg/fsutil.
Package remote implements fsops.Backend over SFTP, on the pooled SSH connection the instance's stored credentials open.
Package remote implements fsops.Backend over SFTP, on the pooled SSH connection the instance's stored credentials open.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL