Documentation
¶
Overview ¶
Package backend defines where ciphertext lives. The Backend interface is a flat object store keyed by base-relative path: it only moves opaque bytes. The per-namespace blob layout a namespace's secrets live in is a layer up, in internal/secrets. RcloneStorage and local.Storage are the implementations.
Index ¶
- Constants
- Variables
- func CreateRemote(ctx context.Context, name, kind string, params map[string]string) error
- func IsNamespaceBlob(key string) bool
- func IsNotenvObject(key string) bool
- func IsReserved(key string) bool
- func ListRemotes(ctx context.Context) ([]string, error)
- func RcloneInstalled() bool
- func ReadCapped(r io.Reader, limit int64) ([]byte, error)
- func RemoteType(ctx context.Context, name string) (string, error)
- func WithinPrefix(key, prefix string) bool
- type Backend
- type HeaderStore
- type RcloneStorage
- func (s *RcloneStorage) BackupHeader(ctx context.Context) error
- func (s *RcloneStorage) Delete(ctx context.Context, key string) error
- func (s *RcloneStorage) Get(ctx context.Context, key string) ([]byte, error)
- func (s *RcloneStorage) GetHeader(ctx context.Context) ([]byte, error)
- func (s *RcloneStorage) List(ctx context.Context, prefix string) ([]string, error)
- func (s *RcloneStorage) Preflight(ctx context.Context) error
- func (s *RcloneStorage) Probe(ctx context.Context) error
- func (s *RcloneStorage) Put(ctx context.Context, key string, data []byte) error
- func (s *RcloneStorage) PutHeader(ctx context.Context, raw []byte) error
- func (s *RcloneStorage) RestoreHeaderBackup(ctx context.Context) error
- func (s *RcloneStorage) SwapHeader(ctx context.Context, base, updated []byte) error
Constants ¶
const ( MaxHeaderBytes int64 = 8 << 20 // 8 MiB MaxObjectBytes int64 = 64 << 20 // 64 MiB MaxListBytes int64 = 64 << 20 // 64 MiB )
Read caps. Far above any real vault, far below exhausting RAM. A header holds the key slots, the rotation log, and one manifest entry per namespace; a single object is one namespace's blob; MaxListBytes caps the total object-key bytes one List pulls in (a vault stores a couple of objects per namespace, so 64 MiB is ~a million keys). Capping the bytes also bounds the element count JSON parsing can allocate (a header cannot smuggle 10^7 slots in a few MiB).
const ( HeaderName = ".header.json" HeaderBackupName = HeaderName + ".prev" HeaderLockName = ".header.lock" ProbeName = ".notenv-probe" TempPrefix = ".tmp-" )
Reserved object names are storage plumbing, not user blobs: the key-slot header, its backup, the write lock, the connectivity probe, and any temp file a write is staging. They share the flat key space with namespace blobs, so every List MUST exclude them (via IsReserved) and no caller may ever delete one as if it were data. The single source of truth lives here because the two backends once diverged on it: the local store filtered the header out of List and the rclone store did not, which let orphan cleanup mistake the header for a stray and delete it.
Variables ¶
var ErrCommitUncertain = errors.New("the header write may have taken effect but could not be verified")
ErrCommitUncertain reports that a header write may have taken effect but could not be confirmed (the store wrote the bytes but a read-back to verify them failed, e.g. a transient error or read-after-write lag on an eventually consistent remote). It is distinct from ErrHeaderChanged, which means the write definitely did NOT land. A caller that wrote a data object for this header must NOT roll that object back on ErrCommitUncertain: the header may already reference it, so deleting it would strand the committed header. The write is durable; the right response is to surface "written but unverified, recover with `notenv credential restore-backup` if a later read fails".
var ErrHeaderChanged = errors.New("the header changed since this operation started")
ErrHeaderChanged is returned by SwapHeader when the stored header does not match the bytes the caller's operation started from: another writer landed first. The caller re-reads, re-applies its change, and retries.
var ErrNotFound = errors.New("object not found")
ErrNotFound is returned by Get and Delete when no object exists at the key.
var ErrObjectTooLarge = errors.New("stored object exceeds the maximum size notenv will read")
ErrObjectTooLarge reports that a stored object is bigger than notenv will read into memory. Storage is treated as dumb and possibly hostile, so a read is bounded before the bytes are trusted: the header in particular is fetched and JSON-parsed before its master-keyed tag can be checked, on paths that never unlock (vault inspect, the namespace first-use check), so an unbounded read would let a remote OOM the machine pre-auth. Reads fail closed with this error instead.
var ErrRcloneMissing = errors.New("rclone not found in PATH")
ErrRcloneMissing is returned when no rclone binary is on PATH.
Functions ¶
func CreateRemote ¶
CreateRemote drives `rclone config create` into the user's global rclone config. Params pass via argv: briefly visible in /proc to same-user processes, but with no shell nothing lands in history. Acceptable for bucket credentials, which guard only ciphertext; weigh it for SFTP/WebDAV passwords, which may guard a whole server (prefer key-based SFTP auth).
There is no argv-free fix available: as of rclone v1.74 `config create` (and `config update`/`config password`) take parameter values only as argv; only `rclone obscure -` reads from stdin. Writing rclone.conf directly would avoid argv but breaks configs encrypted with RCLONE_CONFIG_PASS (which `config create` handles transparently), so it is rejected. The argv-free path for a user who wants it already exists in `notenv setup`: pick "I'll run rclone config myself" to type secrets at rclone's own stdin prompts, then point notenv at the remote.
func IsNamespaceBlob ¶ added in v0.20.0
IsNamespaceBlob reports whether key names a namespace data blob. This is the delete whitelist for the copy reconcile and vault teardown: a key that is not a namespace blob is never a stray to clean up, it is either plumbing (handled separately) or a foreign file notenv must not touch.
func IsNotenvObject ¶ added in v0.20.0
IsNotenvObject reports whether key is something notenv itself put in a vault: reserved plumbing or a namespace blob. Everything else is a foreign file that does not belong to the vault. Recognizing our own files, rather than trusting that a path notenv was handed is safe to own, is what stops copy from deleting a mispointed destination's contents and delete from removing an unrelated tree.
func IsReserved ¶ added in v0.18.0
IsReserved reports whether key names storage plumbing rather than a user object. List implementations exclude these so no caller can mistake one for a data blob; the copy and delete paths exclude them too.
func ListRemotes ¶
ListRemotes returns the names of the user's configured rclone remotes.
func RcloneInstalled ¶
func RcloneInstalled() bool
RcloneInstalled reports whether an rclone binary is available.
func ReadCapped ¶ added in v0.20.0
ReadCapped reads from r until EOF or limit bytes, whichever comes first, and returns ErrObjectTooLarge if r holds more than limit (it reads one byte past limit to detect the overflow). Memory is bounded to limit+1 regardless of how much r would yield, so a huge or endless object cannot exhaust memory. It never returns a truncated object: on overflow it errors rather than hand back a partial read the crypto layer would then reject anyway.
func RemoteType ¶
RemoteType returns a remote's backend type (for example "b2" or "s3"). Reads the local rclone config only, no network.
func WithinPrefix ¶ added in v0.19.1
WithinPrefix reports whether key falls under the directory named by prefix: everything when prefix is empty, the object exactly at prefix, or anything beneath "prefix/". It matches on the slash boundary, not a raw byte prefix, so a prefix of "ns" never also matches a sibling "ns2/...". This is the single source of truth for what a List prefix means, so the directory backends select the same set rclone's directory-scoped listing does (the same divergence risk IsReserved guards). Surrounding slashes are insignificant.
Types ¶
type Backend ¶
type Backend interface {
// Get returns the object stored at key (or ErrNotFound). It must return the
// exact bytes Put stored: the write path reads a blob back after writing and
// treats a byte difference as corruption.
Get(ctx context.Context, key string) ([]byte, error)
// Put stores data at key. The live write path writes uniquely-named blobs and
// never overwrites one; overwrite semantics are still required for whole-vault
// mirroring (vault copy/fork) and idempotent retries.
Put(ctx context.Context, key string, data []byte) error
// List returns the keys of every object under prefix, base-relative and
// recursive. An absent prefix yields no keys, not an error. The read/write
// path keys off the authenticated header manifest, not List; List is for
// whole-vault operations (copy, delete) and orphan detection (doctor, gc).
List(ctx context.Context, prefix string) ([]string, error)
// Delete removes the object at key. Removing an absent key is not an error.
Delete(ctx context.Context, key string) error
}
Backend is a flat object store. Keys are base-relative paths (for example "myapp/data-9f3a.age"); the store prepends its own base and moves bytes, nothing more.
type HeaderStore ¶
type HeaderStore interface {
// GetHeader returns the raw header object (or ErrNotFound).
GetHeader(ctx context.Context) ([]byte, error)
// PutHeader stores the raw header object unconditionally. Mutations of an
// existing header should go through SwapHeader; PutHeader remains for
// recovery paths that must overwrite no matter what.
PutHeader(ctx context.Context, raw []byte) error
// SwapHeader stores updated iff the current header bytes equal base (nil
// base: no header may exist yet), and returns ErrHeaderChanged otherwise:
// the compare-and-swap every concurrent header mutation serializes on.
// Implementations make this as atomic as their storage allows; see each
// implementation for the guarantee it actually provides.
SwapHeader(ctx context.Context, base, updated []byte) error
// BackupHeader copies the current header to a sibling backup object so a
// clobbered header doesn't lock the user out of every blob (on every backend;
// notenv keeps its own backup rather than relying on a remote's version
// history). The safe-write protocol calls it ONLY when a header exists (it
// skips the backup on virgin storage) and refuses to proceed if it errors, so
// an absent header here is a race or a real failure, never the virgin case: it
// returns an error and the write fails closed rather than overwrite without a
// recoverable copy. (Treating a missing header as a no-op is exactly how an
// ambiguous "not found" could let a write proceed unprotected.)
BackupHeader(ctx context.Context) error
// RestoreHeaderBackup copies the sibling backup object back over the
// header, the recovery counterpart to BackupHeader. It returns ErrNotFound
// when no backup exists yet (a vault's first backup is written on its second
// header write).
RestoreHeaderBackup(ctx context.Context) error
}
HeaderStore is implemented by client-side-crypto backends, which keep the key-slot header next to the ciphertext objects (see internal/crypto: LUKS2-style wrapped master key). Backends where the provider holds plaintext have no key material and won't implement it.
type RcloneStorage ¶
type RcloneStorage struct {
Remote string // rclone remote name, e.g. "b2"
Base string // path within the remote, e.g. "my-bucket/notenv"
}
RcloneStorage implements Backend by shelling out to a system rclone. This keeps the binary small and the dependency explicit; embedding the library is a possible later optimization.
func (*RcloneStorage) BackupHeader ¶ added in v0.2.0
func (s *RcloneStorage) BackupHeader(ctx context.Context) error
BackupHeader copies the current header to its ".prev" sibling so a bad overwrite is recoverable (a server-side copy that moves no bytes through the client; a remote's own version history, if any, is an extra backstop, not a substitute). The safe-write protocol calls this ONLY when a header exists, so every copy failure is returned and the write is refused: a missing source here is a race, not the virgin case, and must not be read as "nothing to back up". Swallowing a "not found" was unsafe because rclone emits that text for non-absent failures too (e.g. "Source doesn't exist or is a directory and destination is a file").
func (*RcloneStorage) Delete ¶ added in v0.3.0
func (s *RcloneStorage) Delete(ctx context.Context, key string) error
func (*RcloneStorage) GetHeader ¶
func (s *RcloneStorage) GetHeader(ctx context.Context) ([]byte, error)
func (*RcloneStorage) List ¶
List returns base-relative keys of every object under prefix, recursively.
func (*RcloneStorage) Preflight ¶
func (s *RcloneStorage) Preflight(ctx context.Context) error
Preflight verifies rclone is installed and the remote exists.
func (*RcloneStorage) Probe ¶
func (s *RcloneStorage) Probe(ctx context.Context) error
Probe round-trips a marker object through the configured base path so a bad credential or bucket fails here, with context, not at the first real `set` days later.
func (*RcloneStorage) PutHeader ¶
func (s *RcloneStorage) PutHeader(ctx context.Context, raw []byte) error
PutHeader writes the header object. It does NOT back up first: the safe-write protocol (internal/keymgmt) calls BackupHeader before this, because a clobbered header locks the user out of every blob under it.
func (*RcloneStorage) RestoreHeaderBackup ¶ added in v0.2.0
func (s *RcloneStorage) RestoreHeaderBackup(ctx context.Context) error
RestoreHeaderBackup restores the ".prev" backup over the header. Returns ErrNotFound when there is no backup to restore (none has been written yet).
func (*RcloneStorage) SwapHeader ¶ added in v0.8.0
func (s *RcloneStorage) SwapHeader(ctx context.Context, base, updated []byte) error
SwapHeader implements the compare-and-swap as read-compare-put-readback, which is the strongest rclone offers: object stores expose no conditional write through it. Two writers that both pass the compare inside the same sub-second window still last-write-wins; the read-back converts the loss into ErrHeaderChanged whenever the winner's bytes have already landed, so the loser re-reads, re-applies, and retries (keymgmt.UpdateHeader), and its superseded blob is reclaimed as an orphan. A read-back that cannot confirm the write surfaces as ErrCommitUncertain (the put may have landed, so the caller must not roll back). A backend with native conditional writes can implement this atomically.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package backendtest holds the shared conformance suites for backend implementations.
|
Package backendtest holds the shared conformance suites for backend implementations. |
|
Package local is a pure-Go backend over a directory: the zero-account, zero-dependency vault.
|
Package local is a pure-Go backend over a directory: the zero-account, zero-dependency vault. |
|
Package memstore is an in-memory backend.HeaderStore (and backend.Backend) for tests.
|
Package memstore is an in-memory backend.HeaderStore (and backend.Backend) for tests. |