Documentation
¶
Index ¶
- Constants
- Variables
- func HasArchiveSuffix(name string) bool
- func IsEncrypted(path string) (bool, error)
- type ArchiveEntry
- type Compressor
- func (c *Compressor) CreateTarZstd(ctx context.Context, sourceDir, archivePath string, ...) (Written, error)
- func (c *Compressor) CreateTarZstdOrdered(ctx context.Context, entries []ArchiveEntry, archivePath string, ...) (Written, error)
- func (c *Compressor) DecryptTo(archivePath, outPath string) error
- func (c *Compressor) ExtractTarZstd(ctx context.Context, archivePath, destDir string) error
- func (c *Compressor) ExtractTarZstdWith(ctx context.Context, archivePath, destDir string, opts ExtractOptions) error
- func (c *Compressor) ReadMember(archivePath, name string, limit int64) ([]byte, error)
- func (c *Compressor) VerifyTarZstd(_ context.Context, archivePath string) ([]Member, error)
- func (c *Compressor) VerifyTarZstdDigest(_ context.Context, archivePath string) (*Verified, error)
- type ExtractOptions
- type Member
- type Verified
- type Written
Constants ¶
const ( PlainSuffix = ".tar.zst" EncryptedSuffix = ".tar.zst.age" )
PlainSuffix names archives written before encryption (ODDK <= 0.1.93), and EncryptedSuffix names everything written since. Readers tell the two apart by content, never by name; the name only tells an operator which tool to reach for. See internal-docs/archive-encryption.md.
Variables ¶
var ErrArchiveChanged = errors.New("archive changed during extraction")
ErrArchiveChanged marks an archive file that was modified in place while it was being verified and extracted.
var ErrDigestMismatch = errors.New("archive digest does not match the catalogue")
ErrDigestMismatch marks an archive that verified as intact but whose SHA-256 is not the one its catalogue recorded: a different archive under the expected name.
var ErrMemberNotFound = errors.New("member not found in archive")
ErrMemberNotFound marks an archive that does not hold the member asked for.
var ErrNotDecryptable = errors.New("archive cannot be decrypted with the supplied master key")
ErrNotDecryptable marks an encrypted archive that none of the supplied keys opens: the wrong master key, or none at all. It is refused before anything is created on disk, and is tagged operr.ErrInvalid (HTTP 400) here, at its origin.
Functions ¶
func HasArchiveSuffix ¶ added in v0.1.94
HasArchiveSuffix reports whether name carries either archive suffix.
func IsEncrypted ¶ added in v0.1.94
IsEncrypted reports whether the file at path is an age-encrypted archive.
Types ¶
type ArchiveEntry ¶ added in v0.1.59
ArchiveEntry maps a path on disk to the name it takes inside an archive. A directory source is added recursively under ArchiveName.
type Compressor ¶
type Compressor struct {
// contains filtered or unexported fields
}
Compressor creates, verifies and extracts archives.
Every archive it writes is age-encrypted to keys[0]. Every key is tried when reading, so a restore can supply both its own key and a foreign archive's. With no keys it can still read plaintext archives, and refuses everything else: there is deliberately no way to write a plaintext archive.
func NewCompressor ¶
func NewCompressor(keys ...*crypto.ArchiveKey) *Compressor
NewCompressor returns a Compressor for the given archive keys. The first is the one archives are written to.
func (*Compressor) CreateTarZstd ¶
func (c *Compressor) CreateTarZstd(ctx context.Context, sourceDir, archivePath string, verify func([]Member) error) (Written, error)
CreateTarZstd creates a tar.zst archive from the source directory. The archive contains only the contents of sourceDir, not the directory itself.
verify, if non-nil, receives the members read back out of the finished archive and may reject it — see writeVerifiedArchive.
func (*Compressor) CreateTarZstdOrdered ¶ added in v0.1.59
func (c *Compressor) CreateTarZstdOrdered(ctx context.Context, entries []ArchiveEntry, archivePath string, verify func([]Member) error) (Written, error)
CreateTarZstdOrdered creates a tar.zst archive whose top-level entries appear in the order given, rather than in the map-iteration order CreateTarZstd produces.
Order is not cosmetic here: tar has no index, so reading one member means decompressing everything ahead of it. A snapshot puts its manifest first so a reader can validate version compatibility from the first few kilobytes instead of streaming through gigabytes of dumps to find it.
func (*Compressor) DecryptTo ¶ added in v0.1.94
func (c *Compressor) DecryptTo(archivePath, outPath string) error
DecryptTo writes the plaintext .tar.zst inside an encrypted archive to outPath, for an operator who needs the archive outside ODDK. It goes through verifiedPass like an extraction, so what is written is what was verified. The output appears atomically, 0600, only on success; an existing outPath is refused rather than overwritten.
func (*Compressor) ExtractTarZstd ¶
func (c *Compressor) ExtractTarZstd(ctx context.Context, archivePath, destDir string) error
ExtractTarZstd extracts a tar.zst archive to the destination directory using Go.
It VERIFIES THE ARCHIVE FIRST, before creating anything on disk. That ordering is deliberate: the extractor underneath cannot detect corruption on its own (see VerifyTarZstd for the measurement — 40/40 single-bit flips silently extracted different data with err == nil), and every restore path in ODDK goes through here: `snapshot apply`, `snapshot restore-instance`, `backup restore` and `instance major-upgrade`. Verifying up front turns "restored the wrong bytes and reported success" into "refused before touching the destination".
The cost is one extra decompression pass, with no disk writes. That is cheap next to writing the extracted tree, and cheap next to restoring a corrupt one.
func (*Compressor) ExtractTarZstdWith ¶ added in v0.1.93
func (c *Compressor) ExtractTarZstdWith(ctx context.Context, archivePath, destDir string, opts ExtractOptions) error
ExtractTarZstdWith is ExtractTarZstd with options; see ExtractOptions.
func (*Compressor) ReadMember ¶ added in v0.1.94
func (c *Compressor) ReadMember(archivePath, name string, limit int64) ([]byte, error)
ReadMember returns the content of the member called name, streaming the archive only as far as it — which is why snapshots put their manifest first. At most limit bytes are read.
This is a peek, not a verification: it proves nothing about the rest of the file. What it does read of an encrypted archive is authenticated chunk by chunk, and the wrong key is refused (ErrNotDecryptable) exactly as a full verification would refuse it.
func (*Compressor) VerifyTarZstd ¶ added in v0.1.74
VerifyTarZstd reads an archive back and proves the byte stream is internally consistent, returning the members it found.
WHY THIS IS NOT BUILT ON archives.Extractor: the library's extract path CANNOT DETECT CORRUPTION AT ALL. Measured against this package's own writer, with 40 random single-bit flips in the middle of a 4 MB archive:
ExtractTarZstd (the restore path) missed 40/40, and all 40 silently extracted DIFFERENT data with err == nil. This drain-based check missed 0/40.
The reason is that a tar reader stops at the tar zero-block terminator, which sits before the end of the zstd frame — so the frame's content checksum, the only thing that can detect a flipped bit, is never read. Draining the decompressor to EOF after the tar walk is what forces that check. This is the entire integrity mechanism; nothing else substitutes for it.
Entry bodies are deliberately NOT read. *zstd.Decoder implements neither io.Seeker nor io.ReaderAt, so tar's body skip already pushes every byte through the decompressor — reading them adds code and buys no coverage.
WHAT IT PROVES: the stream decompresses, every tar header parses, and the zstd content checksum matches. That covers truncation, a short final write, a dropped Close error, and bit rot present at read time.
WHAT IT DOES NOT PROVE:
- That the archive holds the RIGHT things. A zero-length file verifies clean with zero members, and a pg_dump of an empty database verifies clean too. Callers must assert the member set they expect — see the verify callback on the writers.
- That the MEDIUM holds it. Immediately after a write the read is served from the page cache; fsync narrows this but does not close it. Detecting later bit rot needs a separate scrub, which is why this is exported.
func (*Compressor) VerifyTarZstdDigest ¶ added in v0.1.93
VerifyTarZstdDigest is VerifyTarZstd plus a SHA-256 of the file, computed from the same read — the verify pass already pulls every byte of the file through the decompressor, so hashing them on the way costs no extra I/O. Returns nil and an error when verification fails: a corrupt archive has no digest worth recording.
The frame checksum and the digest answer different questions. The frame checksum says "this file is internally broken"; it cannot say whether two intact copies (local, S3, a DR host's download) are the SAME archive. A digest recorded when the archive was written can — that is why the catalogue stores it (migration 022).
Only the write and download paths want a digest; the restore paths call VerifyTarZstd and pay nothing for it.
type ExtractOptions ¶ added in v0.1.93
type ExtractOptions struct {
// Keep, when non-nil, selects the members written (by their name in the
// archive). The WHOLE archive is still verified first — integrity is a
// property of the file, not of the members one happens to want — but a
// restore that needs one instance out of a multi-instance snapshot no
// longer needs disk for all of them. Directories are filtered the same
// way, so a caller keeping a subtree must accept the subtree's own
// directory entries (a prefix test does).
Keep func(name string) bool
// WantSHA256, when non-empty, is the digest a catalogue recorded when the
// archive was written. It is checked in the same verification pass, so it
// costs a hash and no extra read. An INTACT archive with a different
// digest is refused: the frame check cannot see a file that was replaced
// by another valid archive, and restoring it would put somebody else's
// data back and report success.
WantSHA256 string
}
ExtractOptions narrows or strengthens an extraction.
type Verified ¶ added in v0.1.93
type Verified struct {
Members []Member
// SHA256 is the lowercase hex digest of the archive FILE (the compressed
// bytes, not the content), in the form `sha256sum` prints, so an operator
// can check a copy by hand.
SHA256 string
}
Verified is what VerifyTarZstdDigest learned about an archive.
type Written ¶ added in v0.1.93
type Written struct {
Size int64
// SHA256 of the published file, from the read-back that verified it — so it
// describes bytes that were proven intact, not bytes the writer believed it
// wrote.
SHA256 string
}
Written describes an archive published by writeVerifiedArchive.
Source Files