Documentation
¶
Overview ¶
Package pipeline orchestrates a backup run: WORM check -> enumerate -> per repo (clone --mirror, bundle, checksum, immutable upload) -> signed run-manifest. A repo failure makes the run fail and the manifest records it.
Index ¶
- Constants
- func LocateForTest(m *Manifest, e RepoEntry) (string, string, string, string, error)
- type ArtifactInfo
- type BackupDeps
- type BackupResult
- type DestInfo
- type DrillDeps
- type DrillRepo
- type DrillReport
- type DrillRequest
- type Manifest
- type RefComparison
- type RefEntry
- type RefMismatch
- type RepoEntry
- type RestoreDeps
- type RestoreRequest
- type RestoreResult
- type SourceInfo
- type ToolInfo
- type VerifyDeps
- type VerifyResult
Constants ¶
const ( StatusSuccess = "success" StatusFailed = "failed" StatusSkipped = "skipped" // nothing to write; see RepoEntry.Reason for which case // The reasons a repository is skipped. Both mean "nothing was written and nothing was // lost", which is why neither fails the run, but they are different facts and an operator // reading a manifest is entitled to know which one applies. ReasonResume = "already backed up for this date" ReasonEmpty = "repository has no commits" // ReasonUnchanged is a PREFIX, not a whole string: the date of the copy being relied on // follows it, because an operator looking at a skipped repository wants to know which copy // they still have. // // The prefix is the stable half and it is part of the contract. A consumer matches on it // and renders the rest; the two reasons above are matched whole, and a dynamic string // where those are constant would have left every consumer falling through to "skipped for // some reason" on what is now the most common outcome of a run. ReasonUnchanged = "unchanged since" )
Status values used in the manifest.
const DrillSchema = "gitdr.drill/v1"
DrillSchema is the versioned identifier of the drill-report contract. Like the manifest, this is a stable public contract: an auditor's tooling reads it, and changing it needs a version bump and a note in SPEC.md.
const ManifestSchema = "gitdr.manifest/v3"
ManifestSchema is the versioned identifier of the run-manifest contract. The manifest schema and the --output json shape are a STABLE PUBLIC CONTRACT: changing them requires a version bump and a note in SPEC.md.
v3 adds RepoEntry.Refs, the ref-to-commit map the source advertised when the copy was made. It is what lets the next run tell whether a repository has changed without cloning it, and it is additive: every v2 field is unchanged and a v2 manifest still verifies.
Variables ¶
This section is empty.
Functions ¶
func LocateForTest ¶ added in v0.1.13
LocateForTest exposes locate to the package's external tests. The parsing it does was wrong in a way only a real repository path revealed, so it is worth testing directly rather than only through a drill.
Types ¶
type ArtifactInfo ¶
type ArtifactInfo struct {
Kind string `json:"kind"` // bundle | meta | sha256
Key string `json:"key"`
Size int64 `json:"size"`
SHA256 string `json:"sha256"`
RetainUntil time.Time `json:"retainUntil"`
}
ArtifactInfo is one stored object with its integrity data.
type BackupDeps ¶
type BackupDeps struct {
Config *config.Config
Source source.Source
Dest dest.Destination
Git *gitexec.Git
SigningKey ed25519.PrivateKey
EncryptionKey []byte // optional client-side envelope key; nil = off
ToolVersion string
Logger *slog.Logger
Now func() time.Time
RequireWORM bool // --require-worm / worm.require: fail closed if not immutable
}
BackupDeps are the inputs to a backup run.
type BackupResult ¶
BackupResult carries the run-manifest and where it was stored. It is returned even when the run fails, so callers can surface the recorded failure.
func Backup ¶
func Backup(ctx context.Context, d BackupDeps) (*BackupResult, error)
Backup runs one backup. It returns a non-nil error on any failure (fail-closed); a BackupResult may still be returned to report what happened.
type DestInfo ¶
type DestInfo struct {
Type string `json:"type"`
Bucket string `json:"bucket"`
WormMode string `json:"wormMode,omitempty"` // configured retention mode
WormImmutable bool `json:"wormImmutable"` // WORM check confirmed immutable
WormDetails string `json:"wormDetails,omitempty"` // observed immutability detail
}
DestInfo identifies where the data went and the immutability observed at write time. wormImmutable records whether the WORM check confirmed the destination immutable, the signed, tamper-evident answer to "was this backup on WORM storage?" (v2).
type DrillDeps ¶ added in v0.1.13
type DrillDeps struct {
Dest dest.Destination
Git *gitexec.Git
EncryptionKey []byte
// PublicKey verifies the manifest being drilled. Without it the drill still runs and the
// report says the manifest's signature was not checked — a drill against an unverified
// manifest proves the artifacts restore, not that they are the ones gitdr wrote.
PublicKey ed25519.PublicKey
// SigningKey signs the report. A drill report is evidence, and unsigned evidence is a
// text file anybody can write.
SigningKey ed25519.PrivateKey
ToolVersion string
Logger *slog.Logger
Now func() time.Time
}
DrillDeps are the inputs to a drill.
type DrillRepo ¶ added in v0.1.13
type DrillRepo struct {
Slug string `json:"slug"`
Status string `json:"status"` // success | failed
Error string `json:"error,omitempty"`
// SourceRefs is how many refs the signed manifest says the source advertised. Zero for a
// copy made by a pre-v3 gitdr, where the drill can still prove the bundle restores and
// cannot prove it matches the source.
SourceRefs int `json:"sourceRefs"`
// BundleRefs is how many the bundle's own header declares.
BundleRefs int `json:"bundleRefs"`
// RestoredRefs is how many of those the restored repository carries at the same object.
RestoredRefs int `json:"restoredRefs"`
// Unreferenced names refs the bundle declares that a clone's refspec does not create.
// Counted apart from RestoredRefs rather than folded into it.
Unreferenced []string `json:"unreferenced,omitempty"`
// SourceMatch says whether the bundle declares exactly what the source advertised. Null
// when the manifest recorded no source refs, which is not the same as false.
SourceMatch *bool `json:"sourceMatch,omitempty"`
// Mismatches name what did not line up, in ref order, so the first one named is stable.
Mismatches []string `json:"mismatches,omitempty"`
}
DrillRepo is one repository's drill outcome.
type DrillReport ¶ added in v0.1.13
type DrillReport struct {
Schema string `json:"schema"`
DrillID string `json:"drillId"`
Tool ToolInfo `json:"tool"`
ManifestKey string `json:"manifestKey"`
// ManifestSigned records whether the manifest's own signature was checked before its
// contents were believed. False is not a failure and is not hidden: it narrows what the
// drill proves, from "the artifacts gitdr wrote restore" to "these artifacts restore".
ManifestSigned bool `json:"manifestSigned"`
StartedAt time.Time `json:"startedAt"`
FinishedAt time.Time `json:"finishedAt"`
Status string `json:"status"`
// Eligible is how many repositories the manifest records a successful copy for; Drilled
// is how many this report actually restored. They differ when Sample is set, and a reader
// must be able to see that without reading the repo list.
Eligible int `json:"eligible"`
Drilled int `json:"drilled"`
Repos []DrillRepo `json:"repos"`
}
DrillReport is the evidence pack. It is signed and written to the destination beside the manifest it drills, so the proof is as immutable as the thing it proves.
func Drill ¶ added in v0.1.13
func Drill(ctx context.Context, d DrillDeps, req DrillRequest) (*DrillReport, error)
Drill restores repositories from a signed manifest and proves what came back.
type DrillRequest ¶ added in v0.1.13
type DrillRequest struct {
// ManifestKey is the run to drill. Empty means the most recent manifest for Host/Owner.
ManifestKey string
Host string
Owner string
// Sample caps how many repositories are restored. Zero means all of them.
//
// A partial drill is honest about being one: the report records how many were eligible and
// how many were tried, so nobody can read a ten-repository sample as a thousand-repository
// guarantee. Repositories are chosen in slug order, so a sample is reproducible rather
// than a different ten every time.
Sample int
// WorkDir is where repositories are restored. Each is removed as soon as it is compared;
// a drill of a large organisation would otherwise need the whole estate on disk at once.
WorkDir string
}
DrillRequest selects what to drill.
type Manifest ¶
type Manifest struct {
Schema string `json:"schema"`
RunID string `json:"runId"`
Tool ToolInfo `json:"tool"`
Source SourceInfo `json:"source"`
Destination DestInfo `json:"destination"`
StartedAt time.Time `json:"startedAt"`
FinishedAt time.Time `json:"finishedAt"`
Status string `json:"status"` // success | failed
Repos []RepoEntry `json:"repos"`
}
Manifest is the signed record of one backup run.
type RefComparison ¶ added in v0.1.13
type RefComparison struct {
Declared int
Matched int
// Unreferenced lists refs the bundle declares that `git clone` does not create.
//
// A clone uses the default refspec, +refs/heads/*:refs/remotes/origin/* plus tags, so a
// bundle's refs/notes/*, refs/merge-requests/*, refs/pull/* and refs/keep-around/*
// entries arrive as objects and get no ref. Verified: a mirror carrying notes, a merge
// request ref and a keep-around ref bundles all five refs, and the clone from that
// bundle has three, while `count-objects -v` shows the same six objects on both sides.
// The data is there; nothing points at it, so a future gc can drop it.
//
// Not a failure, because the cause is `git clone`'s refspec and not the backup: it is
// the same for a perfect bundle and a damaged one, so failing on it would fail every
// restore of every GitLab project, and a check that fires on the happy path gets turned
// off. Not silent either, because these refs genuinely are not in the restored
// repository: they are held out of Matched and named in the summary.
Unreferenced []string
// Mismatches are the failures, in ref-name order, so the first one named is stable.
Mismatches []RefMismatch
}
RefComparison reports a restored repository against the ref map its bundle declares.
Declared counts every line of the bundle header, HEAD included, because every one of them is checked. Matched counts those the restore accounts for at the same object. The two are equal on a healthy restore of a repository that holds only branches and tags; where they differ, Unreferenced and Mismatches say exactly why, so nobody can read a full score off a partial one.
func CompareRestoredRefs ¶ added in v0.1.13
func CompareRestoredRefs(ctx context.Context, g *gitexec.Git, bundlePath, repoDir string) (RefComparison, error)
CompareRestoredRefs reads the ref map the bundle declares and the refs the restored repository holds, and reports whether they agree.
func CompareSourceRefs ¶ added in v0.1.13
func CompareSourceRefs(ctx context.Context, g *gitexec.Git, sourceRefs map[string]string, repoDir string) (RefComparison, error)
CompareSourceRefs answers the second question a drill asks: does the restored repository carry the refs the *source* advertised when the copy was made?
`CompareRestoredRefs` proves the restored repository reproduces what the bundle declares. That is a closed loop — the artifact is consistent with itself — and a bundle written from a half-fetched mirror would pass it perfectly while missing branches the source had. Nothing could check the other half until the manifest started recording the source's own ref map.
The same normalisation as the bundle comparison, and for the same reasons: `git clone` writes a branch as refs/remotes/origin/<name>, and refs a clone's refspec does not create are counted apart rather than folded into the matched total.
One direction only. A restored repository holding a ref the source did not advertise is not missing history — `ls-remote` and a mirror clone can legitimately differ on hidden refs — but a source ref that is not there is exactly the loss a backup exists to prevent.
func (RefComparison) OK ¶ added in v0.1.13
func (c RefComparison) OK() bool
OK reports whether the restored repository carries every ref the bundle declares that a clone can carry, each at the declared object.
func (RefComparison) Summary ¶ added in v0.1.13
func (c RefComparison) Summary(signedBundle bool) string
Summary describes the comparison in the words a restore prints.
signedBundle says whether the bundle these refs came from was itself verified against a signed manifest. It changes the wording rather than the check: without a key the comparison still runs and still proves the restore matches the bundle, but the bundle is then an unauthenticated document, and a sentence that did not say so would let an unverified restore read exactly like a verified one.
type RefEntry ¶ added in v0.1.13
RefEntry is one ref and the object it pointed at, as the source advertised it.
type RefMismatch ¶ added in v0.1.13
type RefMismatch struct {
Ref string // the ref as the bundle declares it
Want string // the object the bundle declares
Got string // what the restore has; empty when the ref is absent altogether
}
RefMismatch is one ref the restored repository does not account for.
func (RefMismatch) String ¶ added in v0.1.13
func (m RefMismatch) String() string
type RepoEntry ¶
type RepoEntry struct {
Slug string `json:"slug"`
Status string `json:"status"` // success | failed | skipped
Error string `json:"error,omitempty"`
// Why a repository was skipped, in plain words. Additive to the manifest schema rather
// than a new status value, because a consumer switching on `status` would break on an
// unknown one and there is more than one reason to skip.
Reason string `json:"reason,omitempty"`
Artifacts []ArtifactInfo `json:"artifacts,omitempty"`
// What the source advertised when this copy was made, from `git ls-remote`.
//
// It is recorded so the next run can ask the same question and compare, and skip a
// repository whose refs have not moved instead of writing a byte-identical copy of its
// entire history. Before this, every run rewrote everything, and on WORM storage the
// customer could not delete any of it.
//
// A slice and not a map: Canonical() relies on struct field order for stable bytes, and
// Go map iteration order is random, so a map here would produce a different signature
// for the same run. Sorted by name for the same reason.
//
// Recorded only on a successful copy. A run that failed halfway has refs that describe a
// repository nothing was written for, and trusting them would skip the retry.
Refs []RefEntry `json:"refs,omitempty"`
// When the artifacts this entry relies on were actually written.
//
// For a copy that was made this run it is this run's finish time. For a repository that
// was skipped as unchanged it is carried forward from the run that made the copy, which
// is the whole point: without it, each skip would reset the age of the copy and the
// refresh bound in unchanged.go would never fire, so a repository that never changes
// would be skipped past its object lock's expiry and end up with nothing.
//
// Found by running the backup three times in a row and watching the third copy in full.
//
// A pointer, because `omitempty` does nothing on a `time.Time`: a struct is never empty to
// encoding/json, so a value field emitted `"copiedAt":"0001-01-01T00:00:00Z"` into every
// manifest including ones re-read from v2 — which changed their canonical bytes and made
// every already-signed manifest fail verification. Caught by the v2 round-trip test.
CopiedAt *time.Time `json:"copiedAt,omitempty"`
}
RepoEntry is the per-repository outcome.
type RestoreDeps ¶
type RestoreDeps struct {
Dest dest.Destination
Git *gitexec.Git
EncryptionKey []byte // optional; must match the backup's key
// PublicKey is optional. When set, restore locates the signed run-manifest for the
// requested date, verifies its signature, and checks every artifact it downloads
// against the checksums the manifest records. The unsigned .sha256 sidecar catches
// corruption but not tampering; the manifest catches both. Without a key restore
// keeps the sidecar-only check and says so in RestoreResult.Verification.
PublicKey ed25519.PublicKey
Logger *slog.Logger
}
RestoreDeps are the inputs to a restore.
type RestoreRequest ¶
type RestoreRequest struct {
Host string // e.g. github.com
Owner string
Name string
Date string // YYYY-MM-DD
OutDir string
}
RestoreRequest selects which dated bundle to restore and where to put it.
type RestoreResult ¶
type RestoreResult struct {
BundleKey string `json:"bundleKey"`
SHA256 string `json:"sha256"`
OutDir string `json:"outDir"`
Verified bool `json:"verified"`
// Verification says in plain words which integrity checks this restore ran, so a
// restore that was not checked against the signed manifest announces itself
// instead of looking identical to one that was. Deliberately kept out of the JSON:
// the --output json shape is a versioned public contract (SPEC §11), and widening
// it is its own change, made on purpose, not as a side effect of a read-side fix.
Verification string `json:"-"`
// Refs is the proof that the restore reproduced the history the bundle declares:
// how many refs the bundle's own header carries and how many of them the restored
// repository has at the same object. Out of the JSON for the same reason as
// Verification — the shape is a versioned contract, and putting this on it is a
// separate, deliberate change.
Refs RefComparison `json:"-"`
}
RestoreResult reports what was restored.
func Restore ¶
func Restore(ctx context.Context, d RestoreDeps, req RestoreRequest) (*RestoreResult, error)
Restore fetches a bundle, verifies its checksum against the stored sidecar (and, when a public key is configured, against the signed run-manifest), checks the bundle, and clones it into OutDir. Read-only against the destination.
type SourceInfo ¶
SourceInfo identifies where the data came from.
type VerifyDeps ¶
VerifyDeps are the inputs to a verify.
type VerifyResult ¶
type VerifyResult struct {
ManifestKey string `json:"manifestKey"`
SignatureValid bool `json:"signatureValid"`
ArtifactsChecked int `json:"artifactsChecked"`
ArtifactsOK int `json:"artifactsOk"`
Failures []string `json:"failures,omitempty"`
}
VerifyResult reports signature and per-artifact checksum results.
func Verify ¶
func Verify(ctx context.Context, d VerifyDeps, manifestKey string) (*VerifyResult, error)
Verify checks the manifest's Ed25519 signature, then re-reads every referenced artifact and recomputes its SHA-256 against the manifest. Read-only.