Documentation
¶
Overview ¶
Package destinations — destination.go: the Destination interface, Parse and the kind registry.
Purpose: one interface for every place a backup can be written to or read from, so callers stop switching on URI schemes. Inputs: a destination URI and, for host://, the controlplane inventory. Outputs: a Destination of one of three kinds: rclone (s3 r2 minio b2 gcs az and bare rclone remotes written name:path), path (path://<abs dir>) and host (host://<inventory server>/<abs dir>). Each kind also implements the optional Opener, Streamer and Remover interfaces, so backup stream and the heartbeat use the same destinations (and the same refusals) as create and restore. Constraints: Parse never touches the network or starts a process.
host.go — the host:// destination kind (a controlplane inventory server).
Purpose: store backups on a server we own over SSH, without rclone. Inputs: host://<server>/<absolute dir>; the server comes from the inventory. Outputs: objects under the directory on that server. Constraints: every ssh and scp process is started by sdk/go/remote (Run, Start, CopyTo); no argv is built here. Server name and paths are validated before any exec (remote.ValidateCopyPath allowlist, no ".."), remote paths are also single-quoted. Host keys are pinned: the D4 option set sets StrictHostKeyChecking=yes against a known_hosts file holding a key for the alias "nself-ci-<server>"; an unpinned server is refused before ssh runs.
hostio.go — I/O helpers for the host:// destination.
Purpose: stream a remote file through an ssh session and hash local files. Inputs: a started remote.Session; a local path. Outputs: an io.ReadCloser that reports the remote exit status on Close. Constraints: no process is started here; sdk/go/remote owns every exec.
path.go — the path:// destination kind (local or mounted disk).
Purpose: store backups under an absolute directory without rclone. Inputs: path://<absolute dir>, local files and keys. Outputs: objects under the directory, files 0600, directories 0700. Constraints: every access goes through os.Root, so a ".." or a symlink cannot leave the directory; the directory itself and every component below it must not be a symlink; writes go to <key>.tmp, are fsynced, renamed and re-read to compare sha256. A symlink at the final component is refused, never followed.
rclone.go — the rclone destination kind and the pre-interface helpers.
Purpose: drive rclone for s3, r2, minio, b2, gcs and az remotes. IsRemoteKey, FetchRemote and toRclonePath predate the Destination interface and keep their exact behaviour and argv for existing callers (pitr restore). Inputs: remote URIs such as s3://bucket/path/to/object. Outputs: files downloaded or uploaded through rclone. Constraints: rclone argv is frozen (see rclone_golden_test.go).
Remote keys follow rclone's URI scheme:
s3://bucket/path/to/object r2://bucket/path/to/object minio://bucket/path/to/object b2://bucket/path/to/object gcs://bucket/path/to/object az://container/path/to/object
The minio:// scheme is an nSelf convention: it is rewritten to a "minio" rclone remote automatically (rclone remote name "minio"). All other schemes are passed through to rclone verbatim (scheme becomes the rclone remote name).
rclone must be installed and configured (rclone.conf or environment variables) before FetchRemote is called.
stream.go — the Streamer and Remover interfaces and their rclone and path implementations.
Purpose: let backup stream and the heartbeat write an object from a reader (no local copy) and delete a failed one, through the same destination grammar create and restore use. Inputs: a reader and a relative key; for HasInlineCredentials, a destination URI. Outputs: an object under the destination; errors that never echo a URI. Constraints: rclone argv is the old rcat/deletefile argv, byte for byte; the remote string is used verbatim. HasInlineCredentials only reports, it never returns the value it inspected.
stream_host.go — Streamer and Remover for the host:// destination kind.
Purpose: write an object to a server we own from a reader (backup stream, heartbeat) and delete a failed one, without a local copy. Inputs: a reader and a relative key; the pinned-host-key Target of hostDest. Outputs: <dir>/<key> on the server, mode 600, written via <key>.tmp. Constraints: every process is started by sdk/go/remote (Start, Run); remote paths are validated and single-quoted. The object appears only after its sha256 matched the bytes read from the reader (same tail as Put).
Index ¶
- Constants
- func FetchRemote(ctx context.Context, remoteKey, destDir string) (string, error)
- func HasInlineCredentials(uri string) bool
- func IsNativeKey(uri string) bool
- func IsRemoteKey(key string) bool
- func KindOf(uri string) string
- func KnownHostsPath() string
- type Destination
- type Inventory
- type KindInfo
- type Object
- type Opener
- type Remover
- type Streamer
Constants ¶
const ( KindRclone = "rclone" KindPath = "path" KindHost = "host" )
Kind names.
const KnownHostsEnv = "NSELF_BACKUP_KNOWN_HOSTS"
KnownHostsEnv overrides the pinned known_hosts file host:// uses.
Variables ¶
This section is empty.
Functions ¶
func FetchRemote ¶
FetchRemote downloads the object at remoteKey to a local file in destDir. It returns the local file path on success.
remoteKey must be a URI understood by rclone (or a minio:// URI — see package-level documentation for the rewrite rules).
If remoteKey ends in ".age" the caller is responsible for decrypting the returned file; FetchRemote copies the raw (still encrypted) bytes.
func HasInlineCredentials ¶ added in v1.4.14
HasInlineCredentials reports whether uri carries rclone options or credentials in the remote itself: a comma in the remote-name segment (`:s3,access_key_id=X:bucket`, `name,secret_access_key=Y:path`, quoted values included), userinfo in a scheme URI (`s3://KEY:SECRET@bucket/p`), or the same shape in the bare form (`s3:KEY:SECRET@bucket`). Such a value would be written to logs and unit files. path:// and host:// never carry any.
For scheme URIs the bucket segment (text after "://" up to the first "/") is checked, not the authority: a secret may itself contain "/", which would hide the "@". No bucket or container name can hold "@" or ":".
func IsNativeKey ¶ added in v1.4.13
IsNativeKey reports whether uri is a path:// or host:// destination.
func IsRemoteKey ¶
IsRemoteKey reports whether key is a supported remote destination URI (i.e. not a local filesystem path).
func KindOf ¶ added in v1.4.13
KindOf reports the kind a destination URI selects. An empty URI is "".
func KnownHostsPath ¶ added in v1.4.13
func KnownHostsPath() string
KnownHostsPath is the pinned known_hosts file host:// reads.
Types ¶
type Destination ¶ added in v1.4.13
type Destination interface {
Kind() string
Put(ctx context.Context, localPath, key string) error
Get(ctx context.Context, key, localPath string) error
List(ctx context.Context, prefix string) ([]Object, error)
}
Destination is a place backups are stored. key is a relative, slash separated name inside the destination.
func Parse ¶ added in v1.4.13
func Parse(uri string, inv *Inventory, rcloneEnv ...string) (Destination, error)
Parse returns the Destination uri names; the result also implements Opener, Streamer and Remover. rcloneEnv is appended to the environment of every rclone process (nil: none); inv is used by host:// only and may be nil for the other kinds.
func ParseObject ¶ added in v1.4.13
func ParseObject(uri string, inv *Inventory) (Destination, string, error)
ParseObject parses a URI that names one object (restore-remote --from). For path:// and host:// the last path element is the key and the rest is the destination directory. Other URIs are returned whole with an empty key.
type Inventory ¶ added in v1.4.13
type Inventory = controlplane.Inventory
Inventory is the controlplane inventory host:// resolves servers from.
type Opener ¶ added in v1.4.13
Opener is implemented by destinations that can stream an object without a local copy (restore-remote pipes it into age and pg_restore). Streamer and Remover (stream.go) are the write and delete counterparts; all three kinds implement all three.
type Remover ¶ added in v1.4.14
Remover deletes one object. An object that is not there counts as removed.
Source Files