Documentation
¶
Overview ¶
Package ark provides a general-purpose CDC (content-defined chunking) backup library. It offers restic-style deduplicating backups through a clean Source/Sink abstraction that any caller can implement.
A repository is created once with Init and opened with Open. Open never creates one, so a mistyped path fails instead of starting a second repository.
Basic usage:
b, _ := arklocal.New("/path/to/repo")
ark.Init(ctx, b, ark.WithPassword(pw)) // once, per repository
repo, _ := ark.Open(ctx, b, ark.WithPassword(pw))
snapshot, _ := repo.Backup(ctx, mySource)
repo.Restore(ctx, snapshot.ID, mySink)
Passwords ¶
Every repository is encrypted and authenticated. Init creates a random master key and stores it in a key file sealed under the password; the data is encrypted with the master key, never with the password. Open unseals the master key before it reads any other object, and a wrong password fails with ErrWrongPassword. There is no recovery: a lost password is lost data.
A repository without a password is created and opened with WithoutPassword. It has the same format with the master key stored unprotected, so anyone with access to the backend can read and forge it; corruption is still detected. The choice is explicit on every Init and Open: WithoutPassword on a protected repository fails with ErrPasswordRequired, a password on an unprotected one with ErrNotPasswordProtected.
ChangePassword seals the same master key under a new password. It does not revoke a master key that leaked, through the old password or a copy of the old key file; a leaked repository needs a new one. Nor does it revoke the old password against anyone with write access to the backend who kept the old key file: put back, it opens the repository under the old password again.
Init, Open and ChangePassword derive the key with argon2id at 64 MiB of memory, so a container that opens a repository needs that much on top of the restore cache.
What the encryption protects against, what it leaks and which risks are accepted is written down in docs/SECURITY.md (https://github.com/alexjoedt/ark/blob/main/docs/SECURITY.md).
Memory ¶
Restore, Export and Verify stream: they walk the snapshot in order and read the chunks of the file at hand, so their memory is the restore chunk cache (256 MiB by default, see WithRestoreCache) plus one chunk per worker in flight, where the workers are capped by WithConcurrency. Neither the size of the snapshot nor the size of a single file changes that.
Index ¶
- Constants
- Variables
- func Init(ctx context.Context, b Backend, pw Passphrase, opts ...InitOption) error
- type Backend
- type BackupOption
- type BlobInfo
- type CheckResult
- type ConcurrentSource
- type FileInfo
- type FileType
- type ForgetOption
- type ForgetPolicy
- type ForgetResult
- type Handle
- type HandleType
- type InitOption
- type LockInfo
- type Option
- type Passphrase
- type Phase
- type ProgressEvent
- type ProgressFunc
- type PruneOption
- type PruneResult
- type Repository
- func (r *Repository) Backup(ctx context.Context, src Source, opts ...BackupOption) (*Snapshot, error)
- func (r *Repository) ChangePassword(ctx context.Context, pw Passphrase) error
- func (r *Repository) Check(ctx context.Context) (*CheckResult, error)
- func (r *Repository) Close() error
- func (r *Repository) Export(ctx context.Context, id string, w io.Writer) error
- func (r *Repository) Forget(ctx context.Context, policy ForgetPolicy, filter SnapshotFilter, ...) (*ForgetResult, error)
- func (r *Repository) ID() string
- func (r *Repository) Import(ctx context.Context, reader io.Reader) (*Snapshot, error)
- func (r *Repository) Locks(ctx context.Context) ([]LockInfo, error)
- func (r *Repository) Prune(ctx context.Context, opts ...PruneOption) (*PruneResult, error)
- func (r *Repository) Restore(ctx context.Context, id string, sink Sink, opts ...RestoreOption) (*RestoreResult, error)
- func (r *Repository) Snapshot(ctx context.Context, id string) (*Snapshot, error)
- func (r *Repository) Snapshots(ctx context.Context, filter SnapshotFilter) ([]*Snapshot, error)
- func (r *Repository) Unlock(ctx context.Context, opts ...UnlockOption) (int, error)
- func (r *Repository) Verify(ctx context.Context, id string) (*VerifyResult, error)
- func (r *Repository) Walk(ctx context.Context, id string, fn func(FileInfo) error) error
- type RestoreOption
- type RestoreResult
- type Sink
- type Snapshot
- type SnapshotFilter
- type SnapshotStats
- type Source
- type SourceMeta
- type SourceType
- type UnlockOption
- type VerifyResult
- type WalkFunc
Constants ¶
const UnknownNamespace = "unknown"
UnknownNamespace is the namespace name String returns for a handle type this version of ark does not define. Backends compare against it to reject one.
Variables ¶
var ( // ErrNotFound is returned when a requested snapshot does not exist. ErrNotFound = errors.New("ark: snapshot not found") // ErrAmbiguousID is returned when a snapshot id prefix matches more than // one snapshot. The message names the candidates. ErrAmbiguousID = errors.New("ark: snapshot id prefix is ambiguous") // ErrEmptyPolicy is returned by Forget when the policy keeps nothing and // the filter names no ids, which would remove every matching snapshot. ErrEmptyPolicy = errors.New("ark: forget policy keeps nothing and no ids given") // ErrIncompatibleSourceType is returned when restoring a snapshot into a // Sink type that is incompatible with the snapshot's SourceType. ErrIncompatibleSourceType = errors.New("ark: incompatible source type") // ErrCorruptManifest is returned when a snapshot manifest does not // authenticate: it was changed, truncated or moved from another snapshot. // Snapshots, Prune and a Forget policy fail while one exists; remove it // by naming its id to Forget without a policy, then Prune. ErrCorruptManifest = errors.New("ark: corrupt snapshot manifest") // ErrCorruptConfig is returned by Open when a key file opens but the // repository config is not the one its master key was created with: the // config was changed or replaced, or the key file belongs to another // repository. The message says which of the two the config's id points to. ErrCorruptConfig = errors.New("ark: repository config does not match its master key") // ErrCorruptTree is returned by Restore, Export and Verify when a tree // object of the snapshot does not authenticate or does not match its name. ErrCorruptTree = errors.New("ark: corrupt snapshot tree") // ErrCorruptChunk is returned by Restore and Export when a stored chunk // fails its integrity check. Verify lists such a chunk in CorruptChunks. ErrCorruptChunk = errors.New("ark: corrupt chunk") // ErrMissingChunk is returned when a required chunk is absent from the CAS. ErrMissingChunk = errors.New("ark: missing chunk") // ErrRepoLocked is returned when a repo-level lock cannot be acquired. ErrRepoLocked = errors.New("ark: repository is locked by another operation") // ErrNotInitialized is returned by Open when the backend holds no // repository config. Call Init first. ErrNotInitialized = errors.New("ark: repository is not initialized") // ErrAlreadyInitialized is returned by Init when the backend already // holds a repository config. ErrAlreadyInitialized = errors.New("ark: repository is already initialized") // ErrUnsupportedFormat is returned by Open when the repository config // names a format this build does not read. ErrUnsupportedFormat = errors.New("ark: unsupported repository format") // ErrWrongPassword is returned by Open when no key file of the // repository opens with the given password. ErrWrongPassword = errors.New("ark: wrong password or no key file matches") // ErrPasswordRequired is returned by Open when WithoutPassword is given // for a password protected repository. ErrPasswordRequired = errors.New("ark: repository is password protected") // ErrNotPasswordProtected is returned by Open when a password is given // for a repository created with WithoutPassword. ErrNotPasswordProtected = errors.New("ark: repository is not password protected") )
Sentinel errors returned by Repository methods.
Functions ¶
func Init ¶
func Init(ctx context.Context, b Backend, pw Passphrase, opts ...InitOption) error
Init creates a repository in b: a fresh master key sealed under pw in a key file, then the config object. pw is WithPassword or, explicitly, WithoutPassword. The password only unlocks the master key; without it the repository cannot be read, and nothing recovers it.
It returns ErrAlreadyInitialized if b already holds a config, so a typo in a repository path cannot silently create a second repository next to the first. The config is written last, so a failed Init leaves no repository behind.
Types ¶
type Backend ¶
type Backend interface {
// Put stores r under h atomically: a reader never observes a partial
// object. Implementations write to a temporary name in the same
// namespace and rename it into place. Put is idempotent, an existing
// object is replaced. Put need not make the object durable; Sync does.
Put(ctx context.Context, h Handle, r io.Reader) error
// Get retrieves a reader for the object stored under h.
// Callers must close the returned ReadCloser.
Get(ctx context.Context, h Handle) (io.ReadCloser, error)
// Exists returns true if the object identified by h is present.
Exists(ctx context.Context, h Handle) (bool, error)
// Delete removes the object identified by h.
// Removing a non-existent object is not an error.
Delete(ctx context.Context, h Handle) error
// Sync is the durability barrier: every Put and Delete that returned
// before the call has reached stable storage when Sync returns, so a
// power loss cannot undo it. A backend that already makes each Put and
// Delete durable returns nil without doing anything.
Sync(ctx context.Context) error
// List calls fn once for each object of the given type with its name and
// stored size. If fn returns an error, List stops and returns it.
List(ctx context.Context, t HandleType, fn func(BlobInfo) error) error
// Stat returns metadata for the object stored under h.
Stat(ctx context.Context, h Handle) (BlobInfo, error)
// Close releases any resources held by the backend.
Close() error
}
Backend is the storage abstraction for all repository objects.
All methods must be safe for concurrent use. The core issues at most WithConcurrency operations at a time, defaulting to runtime.NumCPU*2, so a remote backend can size its connection pool from that bound.
A backend maps Handle.Type to a namespace by its String() value and applies sharding only to BlobFile. An unknown type maps to the namespace "unknown"; implementations may reject it instead.
type BackupOption ¶
type BackupOption func(*backupOptions)
BackupOption is a functional option for Backup.
func WithBackupProgress ¶
func WithBackupProgress(fn ProgressFunc) BackupOption
WithBackupProgress sets the observer for this backup. One observer per call, so two operations on the same Repository never interleave their events. fn must not block.
func WithComment ¶
func WithComment(s string) BackupOption
WithComment attaches a human-readable description to the snapshot.
func WithHost ¶
func WithHost(h string) BackupOption
WithHost overrides the hostname recorded in the snapshot.
func WithParent ¶
func WithParent(id string) BackupOption
WithParent names the parent snapshot explicitly, as a full id or a unique prefix. Without it the parent is the newest snapshot of the same source. A parent of another source is recorded but no file is taken from it unread.
func WithTags ¶
func WithTags(tags ...string) BackupOption
WithTags attaches one or more tags to the snapshot.
type CheckResult ¶
type CheckResult struct {
SnapshotsChecked int
Errors []string
// Warnings are findings that leave every snapshot intact: a backup that
// did not finish, which Prune clears, or a lock that does not open, which
// Unlock WithForce removes.
Warnings []string
}
CheckResult is returned by Check, which walks all snapshots.
type ConcurrentSource ¶ added in v0.2.0
ConcurrentSource is a Source whose open functions outlive the WalkFunc call that passed them: open may run after fn returned, on any goroutine and concurrently with the walk and with other opens. A backup reads the files of such a source in parallel; it reads any other source inside fn, one file at a time. ConcurrentOpen returning false opts a wrapper out.
type FileInfo ¶
type FileInfo struct {
Path string `json:"path"`
Size int64 `json:"size"`
Mode os.FileMode `json:"mode"`
UID int `json:"uid"`
GID int `json:"gid"`
ModTime time.Time `json:"mtime"`
Type FileType `json:"type"`
LinkDest string `json:"link_dest,omitempty"`
Xattrs map[string][]byte `json:"xattrs,omitempty"`
// ChangeTime and Inode let a backup trust that a regular file is
// unchanged since its parent snapshot: ctime cannot be set from user
// space, so it catches a rewrite that kept the mtime. A source that
// cannot tell (a tar stream) leaves both zero, and only Size and ModTime
// are compared.
ChangeTime time.Time `json:"ctime,omitzero"`
Inode uint64 `json:"inode,omitempty"`
}
FileInfo describes a single entry yielded by a Source or received by a Sink.
The JSON tags are a convenience for callers that log or print entries. They are not the repository wire format; the tree entry type owns that.
type FileType ¶
type FileType string
FileType represents the type of a filesystem entry.
const ( // FileTypeRegular is a normal file. FileTypeRegular FileType = "regular" // FileTypeDirectory is a directory. FileTypeDirectory FileType = "directory" // FileTypeSymlink is a symbolic link. FileTypeSymlink FileType = "symlink" // FileTypeHardlink is a second name for a file already yielded under // another path. LinkDest names that earlier path and the content is // stored once, under the first name. FileTypeHardlink FileType = "hardlink" // FileTypeUnsupported marks an entry the v1 format does not model: // device nodes, fifos, sockets. The core drops it and counts it in // SnapshotStats.Skipped. FileTypeUnsupported FileType = "unsupported" )
type ForgetOption ¶
type ForgetOption func(*forgetConfig)
ForgetOption configures a Forget call.
func WithForgetDryRun ¶
func WithForgetDryRun() ForgetOption
WithForgetDryRun makes Forget report what would be removed without deleting anything.
type ForgetPolicy ¶
type ForgetPolicy struct {
// KeepLast keeps the most recent n snapshots regardless of age.
KeepLast int
// KeepHourly keeps one snapshot per hour for the most recent n hours.
KeepHourly int
// KeepDaily keeps one snapshot per day for the most recent n days.
KeepDaily int
// KeepWeekly keeps one snapshot per ISO week for the most recent n weeks.
KeepWeekly int
// KeepMonthly keeps one snapshot per month for the most recent n months.
KeepMonthly int
// KeepYearly keeps one snapshot per year for the most recent n years.
KeepYearly int
// KeepWithin keeps every snapshot younger than this, measured from the
// newest snapshot of the group rather than from the wall clock, so a
// repository that stopped receiving backups does not lose everything.
KeepWithin time.Duration
}
ForgetPolicy defines the retention rules applied by Repository.Forget. The policy is applied per source name, so a busy source never pushes another source's snapshots out of the window. Zero value keeps everything.
type ForgetResult ¶
ForgetResult is returned by Repository.Forget.
type Handle ¶
type Handle struct {
Type HandleType
Name string
}
Handle identifies a single object in the Backend.
type HandleType ¶
type HandleType uint8
HandleType identifies the category of an object stored in a Backend. Each type owns a namespace named by String(); a backend maps that name to a directory or key prefix. New types may appear in minor versions, so a backend must derive its layout from String() instead of switching on the constants.
const ( // ConfigFile is the repository config object. One per repository. ConfigFile HandleType = 1 + iota // BlobFile is a CAS content chunk. Backends shard these by name prefix. BlobFile // TreeFile is a tree node, named by the hash of its canonical JSON. TreeFile // SnapshotFile is a snapshot manifest, named by the snapshot id. SnapshotFile // LockFile is a repository lock object. LockFile // KeyFile holds the master key sealed under one password. KeyFile // MarkerFile marks a backup in progress, named like its lock. One left // behind by a backup that never finished makes later backups distrust // Exists until Prune. MarkerFile )
func (HandleType) String ¶
func (t HandleType) String() string
String returns the namespace name of the handle type, or UnknownNamespace for a type this version of ark does not define.
type InitOption ¶
type InitOption func(*initConfig)
InitOption configures the repository Init writes. Options are frozen into the config object, so they apply to every later Open of that repository.
func WithChunkerConfig ¶
func WithChunkerConfig(c chunker.Config) InitOption
WithChunkerConfig stores custom FastCDC boundaries in the repository config. Zero fields fall back to the chunker defaults. Seed and IDKey are ignored, the repository's master key supplies them. The parameters cannot be changed afterwards without breaking deduplication against existing chunks.
type LockInfo ¶
type LockInfo struct {
ID string
Time time.Time
Exclusive bool
Hostname string
Username string
PID int
Stale bool
}
LockInfo describes one lock object in the repository.
type Option ¶
type Option func(*openConfig)
Option configures a Repository at Open time.
func WithConcurrency ¶
WithConcurrency caps how many backend operations run at once. Values below one fall back to the default, twice the number of CPUs.
func WithLockWait ¶
WithLockWait makes an operation retry a blocked lock for up to d instead of failing at once with ErrRepoLocked. The default is zero, no waiting.
func WithLogger ¶
WithLogger sets the logger for the repository.
type Passphrase ¶
type Passphrase struct {
// contains filtered or unexported fields
}
Passphrase is the password of a repository, or the explicit statement that it has none. WithPassword and WithoutPassword are its only constructors; the zero value is rejected by Init and Open. The package documentation, section Passwords, describes what the password protects and what it does not.
func WithPassword ¶
func WithPassword(password string) Passphrase
WithPassword returns the passphrase for a password protected repository. An empty password is rejected by Init and Open; use WithoutPassword.
func WithoutPassword ¶
func WithoutPassword() Passphrase
WithoutPassword returns the passphrase for a repository without a password. Its master key is stored unprotected: anyone with access to the backend can read and forge its contents, only corruption is still detected.
func (Passphrase) GoString ¶
func (p Passphrase) GoString() string
GoString keeps %#v from printing the password.
type Phase ¶
type Phase string
Phase names the step an operation is currently in. The values are stable identifiers, safe to compare and to emit as JSON; rendering them for humans is the caller's job.
const ( // PhaseScan opens a backup, PhaseStore and PhaseFinalize follow. PhaseScan Phase = "scan" // walking the source PhaseStore Phase = "store" // chunking and writing PhaseFinalize Phase = "finalize" // tree and manifest // PhaseLoad opens a restore, PhaseFetch follows. PhaseLoad Phase = "load" // manifest and tree PhaseFetch Phase = "fetch" // chunks PhaseWrite Phase = "write" // sink )
type ProgressEvent ¶
type ProgressEvent struct {
Phase Phase
// CurrentPath is the entry the event reports, empty when the phase works
// on no single entry. A backup reports a file once it is stored; files
// read in parallel finish, and are reported, out of walk order.
CurrentPath string
FilesTotal int
FilesDone int
BytesTotal int64
BytesDone int64
ChunksTotal int
ChunksDone int
}
ProgressEvent describes the current progress of a backup or restore.
type ProgressFunc ¶
type ProgressFunc func(ProgressEvent)
ProgressFunc receives progress events from one backup or restore operation. It must not block; implementations should drop events when the consumer is slow. Calls never overlap, so a handler may update its state without a lock. The phases of a restore interleave: chunks of upcoming files are fetched while the current file is written.
Example with a channel:
ch := make(chan ark.ProgressEvent, 32)
snap, err := repo.Backup(ctx, src, ark.WithBackupProgress(func(e ark.ProgressEvent) {
select { case ch <- e: default: }
}))
type PruneOption ¶
type PruneOption func(*pruneConfig)
PruneOption configures a Prune call.
func WithPruneDryRun ¶
func WithPruneDryRun() PruneOption
WithPruneDryRun makes Prune report what would be removed without deleting anything.
type PruneResult ¶
PruneResult is returned by Prune.
type Repository ¶
type Repository struct {
// contains filtered or unexported fields
}
Repository is the primary entry point for all backup operations.
func Open ¶
func Open(ctx context.Context, b Backend, pw Passphrase, opts ...Option) (*Repository, error)
Open opens an existing repository in b.
It reads the config object first and returns ErrNotInitialized when the backend holds none, so Open never creates a repository; use Init for that. A config or key file naming a format this build does not read returns ErrUnsupportedFormat.
Then it unseals the master key with pw before any other object is read. It returns ErrWrongPassword when no key file opens, ErrPasswordRequired for WithoutPassword on a protected repository and ErrNotPasswordProtected for a password on a repository created with WithoutPassword, and ErrCorruptConfig when a key opens but the config is not the one it was sealed with. Unsealing a password protected key file runs argon2id with 64 MiB of memory.
func (*Repository) Backup ¶
func (r *Repository) Backup(ctx context.Context, src Source, opts ...BackupOption) (*Snapshot, error)
Backup walks src and stores its entries as a new snapshot.
A cancelled backup leaves orphan chunks and tree objects behind, no snapshot: the manifest is written last, so the snapshot never becomes visible. The orphans stay until Prune sweeps them, and a later backup of the same data reuses them.
func (*Repository) ChangePassword ¶
func (r *Repository) ChangePassword(ctx context.Context, pw Passphrase) error
ChangePassword replaces the key file the repository was opened with by one sealed under pw, which may also be WithoutPassword. The master key stays, so no data is rewritten; for the same reason the change does not revoke access for anyone who could read the master key before, through the old password or a copy of the old key file.
It takes the exclusive lock. The new key file is written before the old one is deleted: when the write fails the old password still opens the repository, when the delete fails both do and the error says so.
func (*Repository) Check ¶
func (r *Repository) Check(ctx context.Context) (*CheckResult, error)
Check verifies every snapshot in the repository.
func (*Repository) Close ¶
func (r *Repository) Close() error
Close releases any resources held by the repository (and the backend).
func (*Repository) Export ¶
Export writes a portable tar.gz archive of one snapshot to w. id is a full snapshot id or a unique prefix.
func (*Repository) Forget ¶
func (r *Repository) Forget( ctx context.Context, policy ForgetPolicy, filter SnapshotFilter, opts ...ForgetOption, ) (*ForgetResult, error)
Forget removes the snapshots that policy does not keep among those matching filter. Chunks stay until Prune.
A policy without a positive rule removes every match, so it is only accepted together with filter.IDs; otherwise Forget returns ErrEmptyPolicy and deletes nothing.
Removing named ids without a policy reads only those manifests, and one that fails with ErrCorruptManifest is still removed, reported with its ID alone. This is how a corrupt manifest leaves the repository.
func (*Repository) ID ¶
func (r *Repository) ID() string
ID returns the repository id written by Init.
func (*Repository) Locks ¶
func (r *Repository) Locks(ctx context.Context) ([]LockInfo, error)
Locks lists the lock objects in the repository, newest first, stale ones included.
func (*Repository) Prune ¶
func (r *Repository) Prune(ctx context.Context, opts ...PruneOption) (*PruneResult, error)
Prune removes CAS objects that no snapshot references. Pass WithPruneDryRun to report what would be removed without making changes.
func (*Repository) Restore ¶
func (r *Repository) Restore(ctx context.Context, id string, sink Sink, opts ...RestoreOption) (*RestoreResult, error)
Restore streams the snapshot into sink. id is a full snapshot id or a unique prefix.
func (*Repository) Snapshot ¶
Snapshot returns one snapshot. id is a full snapshot id or a unique prefix.
func (*Repository) Snapshots ¶
func (r *Repository) Snapshots(ctx context.Context, filter SnapshotFilter) ([]*Snapshot, error)
Snapshots returns all snapshots matching filter, newest first.
func (*Repository) Unlock ¶
func (r *Repository) Unlock(ctx context.Context, opts ...UnlockOption) (int, error)
Unlock removes stale locks and returns how many it removed. Pass WithForce to remove every lock, including those that do not open.
func (*Repository) Verify ¶
func (r *Repository) Verify(ctx context.Context, id string) (*VerifyResult, error)
Verify checks one snapshot: manifest, tree and every chunk. id is a full snapshot id or a unique prefix.
func (*Repository) Walk ¶
Walk calls fn for every entry of a snapshot in tree order, parents before children, without materialising the list. id is a full snapshot id or a unique prefix. An error from fn ends the walk and is returned.
Directory paths carry a trailing slash, the way tar names them; every other entry carries the plain path a Source yielded.
type RestoreOption ¶
type RestoreOption func(*restoreOptions)
RestoreOption is a functional option for Restore.
func WithRestoreCache ¶
func WithRestoreCache(bytes int64) RestoreOption
WithRestoreCache caps the bytes of chunk content the restore keeps around to serve content that repeats within the snapshot. The default is 256 MiB, and zero disables the cache.
The memory a restore holds is this cap plus one chunk per worker in flight, where the workers are bounded by WithConcurrency. It does not grow with the size of the snapshot or of any single file.
func WithRestorePaths ¶
func WithRestorePaths(paths ...string) RestoreOption
WithRestorePaths restores only the entries under the given paths. Prefixes are matched against the snapshot's own paths, and a path that leads nowhere in the snapshot restores nothing. The walk is pruned before any chunk is read, so the skipped part costs nothing.
func WithRestoreProgress ¶
func WithRestoreProgress(fn ProgressFunc) RestoreOption
WithRestoreProgress sets the observer for this restore. One observer per call, so two operations on the same Repository never interleave their events. fn must not block.
type RestoreResult ¶
RestoreResult is returned by Restore.
type Sink ¶
type Sink interface {
Open(ctx context.Context, meta SourceMeta) error
Receive(ctx context.Context, entry FileInfo, r io.Reader) error
Close() error
}
Sink receives file entries during a restore.
Lifecycle:
- Open — called once before any Receive; Sink may inspect meta and reject incompatible source types by returning an error (wrap ErrIncompatibleSourceType where appropriate).
- Receive — called once per entry, in walk order.
- Close — called once after the last Receive (or on abort).
The core reconstructs each file's byte stream from the CAS and calls Receive once per entry, parents before children. A FileTypeHardlink entry always arrives after the entry it points at.
Implementations must:
- validate source type in Open and return ErrIncompatibleSourceType on mismatch
- handle r = nil for everything but regular files
- fully consume r for regular files before returning
- call Close exactly once after the last Receive
type Snapshot ¶
type Snapshot struct {
ID string
SourceName string
SourceType SourceType
SourceMeta map[string]string // labels the source recorded, see SourceMeta.Labels
Host string
Tags []string
Comment string
CreatedAt time.Time
ParentID string
Stats SnapshotStats
}
Snapshot describes a completed backup, as returned by the public API.
type SnapshotFilter ¶
type SnapshotFilter struct {
// IDs selects exactly these snapshots. Snapshots resolves prefixes to
// full ids before matching, so Match itself compares them verbatim.
IDs []string
// Name filters by source name (volume name, path, etc.); empty = all.
Name string
// Type filters by source type, e.g. SourceTypeFile; empty = all.
Type SourceType
// Tags filters to snapshots carrying ALL the given tags; empty = all.
Tags []string
// Host filters by host; empty = all.
Host string
// Before includes only snapshots created before this time (exclusive).
Before time.Time
// After includes only snapshots created after this time (inclusive).
After time.Time
}
SnapshotFilter narrows which snapshots are returned by Snapshots or Forget. Zero value means no filtering — all snapshots are included.
func (SnapshotFilter) Match ¶
func (f SnapshotFilter) Match(s *Snapshot) bool
Match returns true if the snapshot matches all non-zero fields of f. IDs are compared verbatim; Snapshots resolves prefixes before calling this.
type SnapshotStats ¶
type SnapshotStats struct {
FileCount int `json:"total_files"`
DirCount int `json:"total_dirs"`
Skipped int `json:"skipped"`
FilesUnchanged int `json:"files_unchanged,omitempty"`
TotalSize int64 `json:"total_size"`
StoredSize int64 `json:"stored_size"`
NewChunksRawSize int64 `json:"new_chunks_raw_size"`
ChunksTotal int `json:"chunks_total"`
ChunksNew int `json:"chunks_new"`
ChunksReused int `json:"chunks_reused"`
DeduplicationRatio float64 `json:"deduplication_ratio"`
CompressionRatio float64 `json:"compression_ratio"`
Duration time.Duration `json:"-"`
}
SnapshotStats holds deduplication and compression statistics for a snapshot. Skipped counts entries the source yielded that the v1 format does not model: device nodes, fifos and sockets. FilesUnchanged counts the regular files a backup took from the parent snapshot without reading them.
func (SnapshotStats) MarshalJSON ¶
func (s SnapshotStats) MarshalJSON() ([]byte, error)
MarshalJSON writes the manifest shape: Duration becomes the string field "duration".
func (*SnapshotStats) UnmarshalJSON ¶
func (s *SnapshotStats) UnmarshalJSON(data []byte) error
UnmarshalJSON reads the manifest shape. An unparsable duration leaves the field zero rather than failing the whole manifest.
type Source ¶
type Source interface {
Meta() SourceMeta
Walk(ctx context.Context, fn WalkFunc) error
}
Source drives the backup pipeline entry by entry.
Entries may arrive in any order; the core sorts every tree node by name before hashing it.
Implementations must:
- yield a hardlink target before the FileTypeHardlink entries naming it
- pass open = nil for everything but regular files
- for regular files, pass an open that yields at most entry.Size bytes and does no work until it is called; a walk that never calls open must succeed
- be safe for a single concurrent Walk call
type SourceMeta ¶
type SourceMeta struct {
// Name is the logical name used as the snapshot label (e.g. volume name, path).
Name string
// Type identifies the origin of the data.
Type SourceType
// Labels holds arbitrary key-value metadata (e.g. docker_host, docker_version).
Labels map[string]string
}
SourceMeta is stored verbatim in the snapshot manifest.
type SourceType ¶
type SourceType string
SourceType tags a snapshot with its origin. Core defines only SourceTypeFile. Plugin packages (e.g. pkg/arkdocker) define their own SourceType constants; the string value is stored verbatim in the snapshot manifest and compared at restore time.
const SourceTypeFile SourceType = "file"
SourceTypeFile identifies a local filesystem backup.
type UnlockOption ¶
type UnlockOption func(*unlockConfig)
UnlockOption configures an Unlock call.
func WithForce ¶
func WithForce() UnlockOption
WithForce makes Unlock remove every lock, not only the stale ones. It breaks locks that running operations still hold, so the caller has to know that no such operation exists.
type VerifyResult ¶
type VerifyResult struct {
SnapshotID string
ChunksChecked int
MissingChunks []string
CorruptChunks []string
OK bool
}
VerifyResult is returned by Verify.
type WalkFunc ¶
type WalkFunc func(entry FileInfo, open func() (io.ReadCloser, error)) error
WalkFunc is called for each entry produced by a Source.
open is nil for every entry except regular files; the caller must check entry.Type. For a regular file, open returns a reader for entry.Size bytes. The caller calls open at most once and closes the reader before returning, or never calls it when the parent snapshot already holds the file unchanged. A ConcurrentSource lets the caller open and read the file after returning instead. The next call to fn advances the source past the entry, read or not (archive/tar.Reader.Next skips unread bytes).
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
arkdocker
module
|
|
|
Package arkfs implements ark.Source and ark.Sink for local filesystem directories.
|
Package arkfs implements ark.Source and ark.Sink for local filesystem directories. |
|
Package arklocal implements ark.Backend on the local filesystem.
|
Package arklocal implements ark.Backend on the local filesystem. |
|
Package arkmem implements ark.Backend in memory.
|
Package arkmem implements ark.Backend in memory. |
|
Package arksftp implements ark.Backend over an SFTP connection.
|
Package arksftp implements ark.Backend over an SFTP connection. |
|
Package arktest holds the conformance suites every ark implementation runs.
|
Package arktest holds the conformance suites every ark implementation runs. |
|
Package arkuri opens an ark.Backend from a single configuration string.
|
Package arkuri opens an ark.Backend from a single configuration string. |
|
Package chunker implements content-defined chunking (CDC) using the FastCDC algorithm with gear hashing.
|
Package chunker implements content-defined chunking (CDC) using the FastCDC algorithm with gear hashing. |
|
scripts
|
|
|
bench
command
Command bench builds the seeded benchmark dataset that ark is timed against restic on: gen writes it, mutate applies the incremental change set in place, hash prints a digest of a tree, run times ark against restic.
|
Command bench builds the seeded benchmark dataset that ark is timed against restic on: gen writes it, mutate applies the incremental change set in place, hash prints a digest of a tree, run times ark against restic. |