auditcheckpoint

package
v0.0.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 16, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package auditcheckpoint writes the head of every audit chain somewhere the database cannot reach, and checks a chain against what was written (Task 20404).

The gap it closes

audit_events is hash-chained, so editing or deleting a row anywhere in the middle breaks the chain from that point. Deleting the newest rows does not: the survivors are a shorter chain that verifies, and nothing inside the database can tell it from a quiet one — an attacker who can delete rows can also delete or rewrite anything else the database holds about them. The threat model conceded exactly that, and asked operators to remember to cron an export.

A checkpoint is the cheap version of that export, taken by the hub itself. Every few minutes the cluster leader writes one record per chain: the newest row's id and hash, the row count, the latest prune anchor, the time and the member that wrote it. The record goes to a file outside .cloop/, fsynced, and/or to stderr as one JSON line for a container platform to ship off the box. Later, `cloop hub audit verify --checkpoints <file>` asks each recorded head whether the chain still holds it, and names what it finds: a tail truncated after an id a checkpoint saw, a database restored from a backup older than a checkpoint, a row rewritten under one.

Why the records are sealed

A record anybody could write proves nothing: whoever truncated the chain would write a fresh record agreeing with the truncation. So each record is MACed under a key derived from CLOOP_SECRET_KEY with a label of its own — nothing in state.db can produce it — and stamped with that key's fingerprint, so a record sealed under a rotated key reads as "a different key" rather than as a forgery. A hub with no CLOOP_SECRET_KEY writes unsigned records, which still pin each head against an attacker who cannot write the checkpoint file, and `cloop hub doctor` says they are unsigned.

The seal cannot stop records being deleted from the file. That is what keeping the file off the box, or shipping the stderr copy, is for; the doctor reports a newest checkpoint that has gone stale.

Restores

Restoring a backup does to a chain what deleting its tail while the hub keeps writing does, so `cloop db restore` writes a restore record (Task 20410): a record sealed like any other, with a canonical form of its own, naming the backup, the head the chain was restored to, the newest checkpoint superseded, who and why, and pinning the audit.restored row the restore appended. A verifier holding the key accepts it as the explanation for the checkpoints sealed before it that saw past the restored head — see restore.go — and nothing else does: an unsigned or forged record is refused.

Index

Constants

View Source
const (
	// ReasonInterval: the leader's periodic checkpoint for a window.
	ReasonInterval = "interval"
	// ReasonShutdown: the leader shutting down cleanly.
	ReasonShutdown = "shutdown"
	// ReasonRestore: `cloop db restore` put a backup in place of the
	// database; the record carries a Restore (Task 20410).
	ReasonRestore = "restore"
)

Why a record was written.

View Source
const (
	ChainControlPlane = "control-plane"
	ChainProject      = "project"
)

Chain roles, as the records name them.

View Source
const EnvKey = "CLOOP_SECRET_KEY"

EnvKey is the variable the sealing key is derived from: the same CLOOP_SECRET_KEY the secret broker seals credentials under (secretbroker.EnvPassphraseKey), so a hub has one root secret, not two.

View Source
const Kind = "cloop.audit.checkpoint"

Kind marks a checkpoint record, so one can be picked out of a log stream.

View Source
const Version = 1

Version is the record format.

Variables

View Source
var ErrNoKey = errors.New("auditcheckpoint: " + EnvKey + " is not set")

ErrNoKey is returned when there is no CLOOP_SECRET_KEY to derive from.

Functions

func Age

func Age(recs []Located, now time.Time) (time.Duration, bool)

Age is how long ago the newest of recs was written, relative to now.

func ByPath

func ByPath(recs []Located) map[string][]Located

ByPath groups records by the database path they name, each group in the order the records were written.

func CheckFile

func CheckFile(path string) error

CheckFile reports why path cannot hold checkpoints: the configuration rule (absolute, outside every .cloop/ directory), and the same rule again after resolving symlinks — in its directory and, when it exists, in the file itself — so a link cannot walk the records back in beside the database. An existing path must be a regular file: appending to a device or a FIFO is not keeping a record.

func FormatTime

func FormatTime(t time.Time) string

FormatTime renders t the way Record.Time holds it.

func Paths

func Paths(recs []Located) []string

Paths lists the distinct database paths recs name, sorted.

func ValidPath

func ValidPath(path string) string

ValidPath is path as a record names it: cleaned, and valid UTF-8.

func VerifyAnchor

func VerifyAnchor(anchors []statedb.AuditAnchor, a statedb.AuditAnchor) error

VerifyAnchor reads the archive anchor a sealed and reports why it is not the intact chain a describes — unreadable, not matching its digest, or rows that do not re-hash and link from the boundary before it (anchors is every anchor the chain holds) to a's own. The checkpoint writer calls it once per new anchor and records the answer under its seal.

Types

type ChainReport

type ChainReport struct {
	Path  string
	Chain string

	// Records counts what was checked, by seal and by result.
	Records          int
	Valid            int
	Unsigned         int
	Unchecked        int
	ForeignKey       int
	Refused          int
	Consistent       int
	Pruned           int
	PrunedUnverified int
	// Acknowledged counts records a restore record accounts for.
	Acknowledged int
	// RestoreRecords counts restore records; UntrustedRestores those that
	// account for nothing here, because they are not sealed under the key
	// this verifier holds — or it holds none.
	RestoreRecords    int
	UntrustedRestores int

	// Newest is the newest record used; nil when none was.
	Newest *Checked
	// Failures lists every failing record, oldest first.
	Failures []Checked
	// Restores lists the restores the chain agrees with, oldest first.
	Restores []RestoreFinding
	// Finding names what the earliest failure means for the chain, in one
	// sentence; "" when there is none. Hint adds the reading it cannot rule
	// out, where there is one.
	Finding string
	Hint    string
}

ChainReport is a chain checked against its records.

func VerifyChain

func VerifyChain(db *statedb.DB, path, chain string, recs []Located, key *Key) (ChainReport, error)

VerifyChain checks db's audit chain against recs, the records that name it, sealed under key (nil when this process has no CLOOP_SECRET_KEY).

Each record is judged on its own and the earliest failure is named, because it is where history first stops agreeing with what was seen. A record with a bad seal is refused rather than judged — it would otherwise be a way to make the chain look like something it is not. Records sealed under another key, or that cannot be checked here, are still judged for what they say against the chain: their seal is not what makes a head they saw wrong. But only a trusted record — sealed under the key when there is one — vouches for anything, which matters for prunes (see checkArchive) and for restores (see acknowledgeRestores).

func (ChainReport) OK

func (r ChainReport) OK() bool

OK reports whether no record found anything against the chain.

type Checked

type Checked struct {
	Located
	Seal   SealStatus
	Result Result
	Detail string
}

Checked is one record and its verdict.

type Key

type Key struct {
	// contains filtered or unexported fields
}

Key seals and checks records.

func DeriveKey

func DeriveKey(passphrase string) (*Key, error)

DeriveKey derives the checkpoint key from passphrase. pkg/rootkey caches the derivation, so the cost is paid once per process.

func KeyForRoots added in v0.0.6

func KeyForRoots(current, previous string) (*Key, error)

KeyForRoots derives the key from the current root and, when previous is set, lets it also check records sealed under the previous root: a hub mid-way through a change of root wrote its checkpoints under both, and they all still vouch for the chain. Only the current root seals.

func KeyFromRoots added in v0.0.6

func KeyFromRoots() (*Key, error)

KeyFromRoots derives the key from the hub's roots (pkg/rootkey): the file, the systemd credential or the environment, current and previous. ErrNoKey when no current root is configured; a root source that is configured and refused is an error of its own.

func (*Key) Fingerprint

func (k *Key) Fingerprint() string

Fingerprint names the key: 16 hex characters, stamped on every record it seals.

func (*Key) PreviousFingerprint added in v0.0.6

func (k *Key) PreviousFingerprint() string

PreviousFingerprint names the previous root this key also checks under; "" when it checks under the current one only.

func (*Key) Seal

func (k *Key) Seal(r *Record)

Seal stamps r with k's fingerprint and seal. A nil key leaves r unsigned.

type Located

type Located struct {
	Record
	Line int
}

Located is a record and the line it was read from.

func Newest

func Newest(recs []Located) (Located, bool)

Newest returns the newest record in recs, by time.

func Superseded

func Superseded(recs []Located, key *Key, restoredID int64, now time.Time) (Located, bool)

Superseded returns the record a restore of a chain to restoredID supersedes: the newest of recs that may vouch (see Usable), when it saw an id past restoredID. A newest record within the restored head supersedes nothing, and an older one past it was superseded by an earlier restore, not this one.

func Usable

func Usable(recs []Located, key *Key, now time.Time) []Located

Usable returns the records that may vouch for anything — that checkpoints are being written, that the chain agrees with them: sealed under key when there is one, any that is not refused when there is none, and none dated more than a few minutes after now. Anybody able to append to the file can add a record; under a key, only the hub can add one that counts.

type ParseResult

type ParseResult struct {
	Records []Located
	// Malformed counts lines that name a checkpoint but do not decode — a
	// truncated write, or an edit.
	Malformed []int
	// Oversized lists lines longer than a record can be, which were skipped
	// unread: in a log export they are somebody else's lines; in the
	// checkpoint file itself, an edit.
	Oversized []int
}

ParseResult is what Parse found.

func Parse

func Parse(r io.Reader) (ParseResult, error)

Parse reads records from r: the checkpoint file itself, or a log export with a record somewhere on some of its lines — prefixed by a timestamp, or wrapped in a shipper's JSON envelope. Lines without a record are skipped silently; lines that name one and do not decode are reported, and so are lines too long to be one, which are skipped rather than allowed to end the read — one oversized line appended to the file must not blind verification to every record after it.

func ReadFile

func ReadFile(path string) (ParseResult, error)

ReadFile parses the records in path.

type Record

type Record struct {
	Kind    string `json:"kind"`
	Version int    `json:"v"`

	// Chain is ChainControlPlane or ChainProject; Path the absolute path of
	// the database, which is how a record is matched to a chain.
	Chain string `json:"chain"`
	Path  string `json:"path"`

	// Reason is ReasonInterval or ReasonShutdown. Window is the interval
	// window an interval record belongs to: the Unix time divided by the
	// interval, so every member numbers windows alike.
	Reason string `json:"reason"`
	Window int64  `json:"window,omitempty"`

	// LastID and LastRowHash are the newest row; Rows how many the table
	// held. All zero for an empty chain.
	LastID      int64  `json:"last_id"`
	LastRowHash string `json:"last_row_hash,omitempty"`
	Rows        int64  `json:"rows"`

	// The newest prune anchor, if the chain was ever pruned.
	AnchorID        int64  `json:"anchor_id,omitempty"`
	AnchorThroughID int64  `json:"anchor_through_id,omitempty"`
	AnchorHash      string `json:"anchor_hash,omitempty"`
	// AnchorVerified says the hub read that anchor's archive itself and found
	// it the intact chain the anchor ends in. Seeing an anchor in the database
	// proves only that one is there — whoever can delete rows can write one —
	// so this, under the seal, is what lets a later verifier accept the
	// prune when the archive has since moved out of reach.
	AnchorVerified bool `json:"anchor_verified,omitempty"`

	// Time is when the head was read (RFC 3339, UTC); Member the hub member
	// that read it.
	Time   string `json:"time"`
	Member string `json:"member,omitempty"`

	// Restore says what a restore record records; nil on every other record.
	Restore *Restore `json:"restore,omitempty"`

	// KeyFingerprint names the key Seal was computed under. Both are empty
	// on an unsigned record.
	KeyFingerprint string `json:"key_fingerprint,omitempty"`
	Seal           string `json:"seal,omitempty"`
}

Record is one checkpoint: where one chain ended, at one moment.

Time is a string rather than a time.Time so the sealed bytes are exactly the bytes on disk; a round trip through a time value is one place for a formatting difference to turn a valid seal into a refused one.

func HeadRecord

func HeadRecord(head statedb.AuditHead, chain, path, reason string, window int64, member string, at time.Time) Record

HeadRecord is the unsealed record of head, read at at.

Strings are made valid UTF-8 before they are sealed: JSON replaces invalid bytes on the way out, and a seal over bytes the file no longer holds would refuse every record of a hub whose path is not UTF-8.

func RestoreRecord

func RestoreRecord(head statedb.AuditHead, chain, path, member string, at time.Time, x Restore) Record

RestoreRecord is the unsealed restore record for path's chain, whose head — read after the restore appended its audit.restored row — is head.

func (Record) At

func (r Record) At() time.Time

At parses Time, returning the zero time when it does not parse.

func (Record) IsRestore

func (r Record) IsRestore() bool

IsRestore reports whether r claims to be a restore record. A record that claims it by its reason without carrying a Restore, or carries one under another reason, was not written by `cloop db restore`.

func (Record) Signed

func (r Record) Signed() bool

Signed reports whether the record carries a seal at all.

type Restore

type Restore struct {
	// BackupSHA256 is the restored file's digest; BackupTime when the backup
	// was taken, from its sidecar ("" when it had none). Sidecar says what
	// vouched for the file: "sealed" when its sidecar's seal verified under
	// the hub's key, "unchecked" when it was restored with --allow-unsealed,
	// "skipped" with --skip-checksum.
	BackupSHA256 string `json:"backup_sha256"`
	BackupTime   string `json:"backup_time,omitempty"`
	Sidecar      string `json:"sidecar"`

	// RestoredID, RestoredRowHash and RestoredTime are the newest row the
	// backup holds: where the chain was restored to. All zero for a backup
	// whose chain was empty.
	RestoredID      int64  `json:"restored_id"`
	RestoredRowHash string `json:"restored_row_hash,omitempty"`
	RestoredTime    string `json:"restored_time,omitempty"`

	// SupersededID, SupersededRowHash and SupersededTime are the newest
	// checkpoint the restore found that saw an id past RestoredID: the far end
	// of the history the restore gave up, as far as any checkpoint recorded
	// it. Zero when none did.
	SupersededID      int64  `json:"superseded_id,omitempty"`
	SupersededRowHash string `json:"superseded_row_hash,omitempty"`
	SupersededTime    string `json:"superseded_time,omitempty"`

	// Actor restored it, for Reason, which the command requires.
	Actor  string `json:"actor"`
	Reason string `json:"reason"`

	// SecretLoss names the sealing keys the restore went ahead without,
	// under --accept-secret-loss: whatever the backup holds sealed under them
	// can no longer be opened.
	SecretLoss []string `json:"secret_loss,omitempty"`
}

Restore is what a restore record says happened (Task 20410): which backup replaced the database, the head its chain was restored to, the newest checkpoint that saw history the backup does not hold, and who did it and why.

`cloop db restore` writes it after the restore has appended its own audit.restored row, so the record's head — Record.LastID and LastRowHash — is that row. A chain cut back to the restored head afterwards, or rebuilt from it with other rows, disagrees with the record exactly as it would with any checkpoint.

Only a record sealed under the key the verifier holds accounts for anything: it is what turns "the chain lost the rows these checkpoints saw" from a finding into an acknowledged restore, so whoever can write the database or the checkpoint file must not be able to write one. The seal is computed under the checkpoint key, which nothing in state.db can produce.

func RestoreFrom

func RestoreFrom(head statedb.AuditHead) Restore

RestoreFrom fills in what a restore record says about the backup's chain from its head.

func (Restore) LostEvents

func (x Restore) LostEvents() int64

LostEvents is how many events the superseded checkpoint saw past the restored head — the history the restore gave up, as far as a checkpoint saw it. Events written after that checkpoint and before the restore are not counted: nothing outside the database recorded them.

type RestoreFinding

type RestoreFinding struct {
	// Record is the restore record.
	Record Checked
	// Superseded counts the records sealed before it that it accounts for.
	Superseded int
	// The history lost: ids LostFromID through LostThroughID, LostEvents of
	// them, written between From and To — from the newest row the backup
	// holds to the newest checkpoint that saw what came after. The record
	// names the newest checkpoint the restore found; one the verifier finds
	// later still widens the window.
	LostFromID, LostThroughID int64
	LostEvents                int64
	From, To                  string
}

RestoreFinding is a restore the chain agrees with, and what it gave up.

func (RestoreFinding) Summary

func (f RestoreFinding) Summary() string

Summary is the restore in one line.

func (RestoreFinding) Vouched

func (f RestoreFinding) Vouched() bool

Vouched reports whether the restored backup's sealed sidecar verified: the restore accounts for what it lost either way, but only a vouched-for backup says what was put back.

type Result

type Result string

Result is what one record says about the chain it names.

const (
	// ResultConsistent: the chain still holds the row the record saw.
	ResultConsistent Result = "consistent"
	// ResultPruned: the row was removed by a prune (Task 20218), and the
	// archive its anchor sealed holds it as the record saw it.
	ResultPruned Result = "pruned"
	// ResultPrunedUnverified: a prune anchor covers the row, but its archive
	// cannot be read here (moved to cold storage, say) to confirm it.
	ResultPrunedUnverified Result = "pruned-unverified"
	// ResultTruncated: the chain ends before the row the record saw, and
	// nothing has been appended since — its tail was deleted.
	ResultTruncated Result = "truncated"
	// ResultRestored: the chain no longer has the row the record saw, and
	// has been written to since — the database was restored from a backup
	// taken before the record (or rewound and appended to).
	ResultRestored Result = "restored"
	// ResultRewritten: the chain holds a different row under the id the
	// record saw, timestamped no later than the record — history was
	// rewritten and re-hashed.
	ResultRewritten Result = "rewritten"
	// ResultDeleted: the row is missing from inside the chain's range.
	ResultDeleted Result = "deleted"
	// ResultAnchorMissing: the record saw a prune anchor the database no
	// longer has.
	ResultAnchorMissing Result = "anchor-missing"
	// ResultAnchorRewritten: the anchor the record saw has been altered.
	ResultAnchorRewritten Result = "anchor-rewritten"
	// ResultArchiveMismatch: the archive a prune anchor sealed no longer
	// matches the digest the anchor recorded, or is not the intact chain the
	// anchor's boundary says it ends in.
	ResultArchiveMismatch Result = "archive-mismatch"
	// ResultPruneUnproven: a prune anchor covers the row, no trusted
	// checkpoint ever saw that anchor, and its archive cannot be read here.
	// Rows deleted and an anchor written over them look exactly like this, so
	// it fails until the archive is checked.
	ResultPruneUnproven Result = "prune-unproven"
	// ResultRefused: the record's seal does not verify, or it is a restore
	// record nothing vouches for; it was not used.
	ResultRefused Result = "refused"
	// ResultAcknowledged: the chain no longer holds what the record saw, and
	// a restore record sealed after it accounts for that: the database was
	// restored from a backup that ends before the record's head (Task 20410).
	ResultAcknowledged Result = "acknowledged"
	// ResultRestoreMismatch: a sealed restore record whose head, or the head
	// it says the chain was restored to, the chain no longer holds — it was
	// cut back or rewritten after the restore it records.
	ResultRestoreMismatch Result = "restore-mismatch"
)

func (Result) Failing

func (r Result) Failing() bool

Failing reports whether the result is a finding against the chain (or, for ResultRefused, against the checkpoint file).

type SealStatus

type SealStatus string

SealStatus is what a record's seal says about it.

const (
	// SealValid: sealed under the key checking it, and the seal verifies.
	SealValid SealStatus = "valid"
	// SealUnsigned: written by a hub with no key.
	SealUnsigned SealStatus = "unsigned"
	// SealUnchecked: sealed, but there is no key here to check it with.
	SealUnchecked SealStatus = "unchecked"
	// SealForeignKey: sealed under a different key than the one checking it
	// — a rotated CLOOP_SECRET_KEY, or another hub's record.
	SealForeignKey SealStatus = "foreign-key"
	// SealBad: claims the checking key and does not verify under it, or is
	// half a seal. Edited or forged; such a record is refused.
	SealBad SealStatus = "bad"
)

func Check

func Check(r Record, k *Key) SealStatus

Check reports what r's seal says, checked under k (which may be nil).

type Sink

type Sink struct {
	// File is appended to and fsynced; "" writes no file.
	File string
	// Stderr receives one JSON line per record; nil writes none.
	Stderr io.Writer
}

Sink is where records go: a file, stderr, or both.

func (Sink) Write

func (s Sink) Write(recs []Record) error

Write appends recs to the file with one fsync, then prints them to stderr.

The file comes first and its error is the one returned: it is the copy an operator configured on purpose. A directory that does not exist is an error rather than something to create — a missing mount point would otherwise turn "the records go to the archive volume" into "the records go to the root disk, beside the database".

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL