Documentation
¶
Index ¶
- Constants
- Variables
- func EnsureEventID(ev *Event)
- func HashEvent(prevHash string, ev Event) string
- func LinkEvent(prevHash string, ev *Event)
- func LocalIncarnationID(sandboxID, toolboxToken string) string
- func MintEgressCapability(key, sandboxID, incarnationID string, expiry time.Time) (string, error)
- func NewEventID(nodeID string) string
- func ParseAndVerifyEgressCapability(key, capability string, now time.Time) (sandboxID, incarnationID string, err error)
- func ReadLine(br *bufio.Reader, max int64) (line []byte, consumed int64, terminated, tooLong bool, err error)
- func WithFileLock(lockPath string, fn func() error) error
- type ChainScan
- type ChainVerifier
- type Event
- type ExportSource
- type IndexEntry
- type NodeInput
- type NodeReport
- type PendingHead
- type Result
- type ScanOptions
- type SpillFile
- type SpillRecord
- type WitnessedHead
Constants ¶
const ( // CapabilitySep separates HMAC capability fields. CapabilitySep = "|" // DefaultCapabilityTTL bounds how long a worker may emit egress events // with a given capability without refresh. DefaultCapabilityTTL = 24 * time.Hour )
const ( // KindRetentionCheckpoint is the immutable record retention inserts // where it dropped a prefix: its PrevHash is the last dropped EventHash, // and WitnessedThrough (when set) names the last externally witnessed // head the prefix carried. KindRetentionCheckpoint = "retention_checkpoint" // KindRetentionRedacted is what retention leaves of an expired record // it cannot cut out of the chain: its two link hashes and nothing that // says who opened what. See RedactForRetention. KindRetentionRedacted = "retention_redacted" // MaxLineBytes bounds one JSONL record during chain scans; audit lines // are metadata-only, so anything larger is corruption, not evidence. MaxLineBytes = 1024 * 1024 )
const ( IndexKindOther byte = iota IndexKindSecretOpen IndexKindEgress IndexKindGap IndexKindRetentionCheckpoint )
Index kind classes. Only decisive exclusions may use them: a filter for "egress" skips IndexKindSecretOpen entries but must still read IndexKindOther and IndexKindGap ones and decide on the record itself.
const ( // LockFileName is the stable flock sidecar for the whole audit // directory. It carries no data; every writer, reader, drain and // retention pass takes it, so the file must never be renamed. LockFileName = "secrets.jsonl.lock" // SpillFileName is the overflow buffer the daemon's writer drains. SpillFileName = "secrets.spill.jsonl" )
const GenesisPrevHash = "0"
GenesisPrevHash is the prev_hash of the first event in a chain.
Variables ¶
var ErrIndexEntriesCorrupt = errors.New("audit index entries are corrupt")
ErrIndexEntriesCorrupt reports a posting list that does not decode. The index is derived data; the caller rebuilds it from the JSONL.
var ErrNoSpillPath = errors.New("audit spill path is unset")
ErrNoSpillPath reports a SpillFile with no destination.
Functions ¶
func HashEvent ¶
HashEvent returns SHA-256 hex of prevHash || canonical JSON of ev with EventHash cleared (so the hash does not depend on itself).
func LocalIncarnationID ¶
LocalIncarnationID derives a stable, non-secret lifecycle identifier from the per-create toolbox token. The token is regenerated whenever a sandbox ID is reused, so stale worker capabilities cannot cross into a replacement lifecycle. Only the digest is retained in audit ACL/event records.
func MintEgressCapability ¶
MintEgressCapability returns HMAC(sandboxID|incarnationID|expiry, key) hex-encoded as sandboxID|incarnationID|expiryUnix|mac. key is typically SB_AUDIT_INGEST_TOKEN (or a derived key). The mac binds the sandbox so a stolen capability for one sandbox cannot forge another sandbox's events.
func NewEventID ¶
NewEventID returns a fleet-unique audit event identifier. The normal path is 128 bits from crypto/rand. The fallback includes node, wall clock, process, and a process-local sequence so audit emission remains available even if the host entropy source fails.
func ParseAndVerifyEgressCapability ¶
func ParseAndVerifyEgressCapability(key, capability string, now time.Time) (sandboxID, incarnationID string, err error)
ParseAndVerifyEgressCapability checks the HMAC and expiry. On success it returns the bound sandboxID and incarnationID (never trust client body for those fields when a capability is present).
func ReadLine ¶ added in v0.6.7
func ReadLine(br *bufio.Reader, max int64) (line []byte, consumed int64, terminated, tooLong bool, err error)
ReadLine returns one line (terminator included when present) and the bytes consumed. A line past max is counted but not retained so a NUL-filled or runaway tail cannot balloon the scan. max <= 0 means MaxLineBytes; the daemon's spill drain passes its ingest body cap so a worker cannot plant a line that fails strict boot.
func WithFileLock ¶
WithFileLock runs fn holding an exclusive flock on lockPath. The sidecar is opened read-only: it only needs a stable descriptor, and a read-only open cannot hide a buffered-write failure on close.
Types ¶
type ChainScan ¶ added in v0.6.7
type ChainScan struct {
Head string
EventID string
// LastLineStart is the offset of the record carrying Head.
LastLineStart int64
// Found reports which probed hashes the pass saw.
Found map[string]bool
// WitnessedThrough is the newest retention checkpoint's WitnessedThrough.
WitnessedThrough string
// ValidEnd is the byte offset just past the last complete, verified line.
ValidEnd int64
// TornBytes is the length of an unterminated, unparseable tail after
// ValidEnd — what a crash leaves when it interrupts one append write. The
// writer emits line+'\n' in a single write, so a crash leaves a strict
// byte-prefix, and no proper prefix of a JSON object is itself valid JSON;
// that is what makes this the only defect safe to classify as a tear.
// ScanChain reports a tear rather than failing on it so the daemon's
// boot repair can cut it back; every other reader must treat a non-zero
// value as a chain it cannot verify, never quietly trim evidence.
TornBytes int64
// MissingNewline reports a complete, verified final record without its
// terminator (the write stopped exactly between '}' and '\n').
MissingNewline bool
// Records counts verified records (blank lines excluded).
Records int64
// Redacted counts retention stubs among them: records whose payload
// retention removed in place. The node never produces one inside the
// retention window.
Redacted int64
}
ChainScan is one verified pass over an audit JSONL stream.
func ScanChain ¶ added in v0.6.7
func ScanChain(r io.Reader, opts ScanOptions) (ChainScan, error)
ScanChain walks r once, verifying every record against its predecessor. Blank lines are tolerated but never alter the chain. r should be already bounded (a size snapshot taken under the audit flock) so a verification can run while the writer keeps appending. Memory is O(1) in the number of records.
type ChainVerifier ¶ added in v0.6.7
type ChainVerifier struct {
// contains filtered or unexported fields
}
ChainVerifier checks one chain in stream order with O(1) state: the head so far, whether the next link may break (right after a retention checkpoint), and whether any record has been seen. The zero value is not ready; start from NewChainVerifier, or ResumeChainVerifier to continue a chain someone already verified up to a known head.
func NewChainVerifier ¶ added in v0.6.7
func NewChainVerifier() ChainVerifier
NewChainVerifier starts a verifier at genesis. It is a value so a caller can hand a copy to ScanChain and keep its own state unchanged.
func ResumeChainVerifier ¶ added in v0.6.7
func ResumeChainVerifier(head string, allowBreak bool) ChainVerifier
ResumeChainVerifier primes a verifier mid-chain: head is the EventHash of the last record already verified, and allowBreak is whether that record was a retention checkpoint (whose successor may name a dropped predecessor). Checkpoint boot, the export tailer and the read index all carry exactly this state across passes instead of re-reading the prefix, and a resumed verifier refuses a later retention checkpoint just as one that read the prefix itself would.
func (*ChainVerifier) Add ¶ added in v0.6.7
func (v *ChainVerifier) Add(ev Event) error
Add validates one immutable event in stream order. The only permitted link discontinuity is the first retained event after a retention checkpoint, whose predecessor was deliberately removed. Missing hash fields are never interpreted as a legacy format: this audit format is intentionally one-way.
func (*ChainVerifier) AllowBreak ¶ added in v0.6.7
func (v *ChainVerifier) AllowBreak() bool
AllowBreak reports whether the next record may name a predecessor other than Head — true only right after a retention checkpoint. Callers persist it alongside Head so a resumed pass keeps the same rule.
func (*ChainVerifier) Head ¶ added in v0.6.7
func (v *ChainVerifier) Head() string
Head is the EventHash of the last accepted record, or GenesisPrevHash when none has been accepted yet: what the next record's prev_hash must name.
type Event ¶
type Event struct {
Time time.Time `json:"time"`
Actor string `json:"actor,omitempty"`
SandboxID string `json:"sandbox_id,omitempty"`
Ref string `json:"ref,omitempty"`
Result string `json:"result"`
Reason string `json:"reason,omitempty"`
CorrelationID string `json:"correlation_id,omitempty"`
EventID string `json:"event_id,omitempty"`
NodeID string `json:"node_id,omitempty"`
Kind string `json:"kind,omitempty"`
Destination string `json:"destination,omitempty"`
Network string `json:"network,omitempty"`
BytesIn int64 `json:"bytes_in,omitempty"`
BytesOut int64 `json:"bytes_out,omitempty"`
// IncarnationID scopes the event to a placement incarnation so post-delete
// ACL and recreate history do not cross tenant ownership boundaries.
IncarnationID string `json:"incarnation_id,omitempty"`
// OwnerRef is the tenant account the sandbox belonged to when the event
// was recorded. It makes an exported record self-describing downstream
// and lets a node authorize a post-delete read from the evidence itself,
// without consulting Raft. Empty for operator-owned sandboxes. Being
// omitempty, records written before the field existed hash identically.
OwnerRef string `json:"owner_ref,omitempty"`
// Dropped is set on gap markers to record how many events were coalesced
// into this single overflow marker (lossy buffer evidence).
Dropped int64 `json:"dropped,omitempty"`
// PrevHash / EventHash form the local append-only hash chain used for
// tamper detection once heads are witnessed off-node (E2b).
PrevHash string `json:"prev_hash,omitempty"`
EventHash string `json:"event_hash,omitempty"`
// WitnessedThrough is set on retention_checkpoint records to record the
// last externally witnessed head that prune dropped (optional MVP field).
WitnessedThrough string `json:"witnessed_through,omitempty"`
}
Event is the shared JSONL/wire representation for secret-open and egress audit records. Keeping the daemon, peer transport, and WASM worker on this single shape prevents fields that affect pagination from drifting.
func GapMarker ¶
GapMarker is the record that stands in for dropped events: one marker per flush, carrying the count, so overflow costs the writer one line rather than one per drop and the stream stays honest about the loss.
func RedactForRetention ¶ added in v0.6.7
RedactForRetention is what retention leaves of an expired record that sits behind a newer one and so cannot be cut out of the chain: the two link hashes, the time and id (so a later prune reclaims it with the prefix and an operator can place it), and nothing that says who opened what. EventHash is the original — it is what the successor's prev_hash names — so the chain still verifies through the stub; the verifier links a stub by its stored hashes because the payload that produced them is gone, and refuses one that still carries any other field.
type ExportSource ¶ added in v0.6.7
type ExportSource interface {
// Days lists the UTC dates ("2006-01-02") holding batches for nodeID,
// ascending, starting at from.
Days(ctx context.Context, nodeID, from string) ([]string, error)
// Batches lists one day's batch names.
Batches(ctx context.Context, nodeID, day string) ([]string, error)
// Open reads one batch: one JSON event per line.
Open(ctx context.Context, nodeID, day, batch string) (io.ReadCloser, error)
}
ExportSource reads a node's exported batches in the export layout: node=<id>/YYYY/MM/DD/<batch>.jsonl. Batch names are never used for order; the walk follows the chain itself.
type IndexEntry ¶
type IndexEntry struct {
Offset int64 // first byte of the record
Length int64 // bytes through the trailing newline
TimeNano int64 // Event.Time as UnixNano
Kind byte // IndexKind*; anything unclassified is IndexKindOther
}
IndexEntry locates one record inside the local audit JSONL and carries just enough of it to page and filter without touching the file: the read path decides which lines a page needs from entries alone, then reads only those lines. Offsets are byte positions in the current file generation.
func AppendIndexEntries ¶
func AppendIndexEntries(buf []byte, prev IndexEntry, hasPrev bool, entries []IndexEntry) ([]byte, IndexEntry, error)
AppendIndexEntries delta-encodes entries after prev. Within one posting list offsets are strictly increasing (file order) and times nearly so, so each entry costs a few bytes rather than the 25+ of a fixed layout — the index stays a small fraction of the log it describes. prev is the last entry already in buf; pass hasPrev=false for an empty list.
func DecodeIndexEntries ¶
func DecodeIndexEntries(buf []byte) ([]IndexEntry, error)
DecodeIndexEntries is the inverse of AppendIndexEntries for a whole list.
type NodeInput ¶ added in v0.6.7
type NodeInput struct {
NodeID string
// Heads are the witnessed heads in witness order, oldest first. Heads
// before Since may be included; they are listed, not checked.
Heads []WitnessedHead
// Prior is the newest head witnessed before the earliest of Heads, when
// known: a head equal to it needs no records to link.
Prior *WitnessedHead
// Since is the window start the auditor chose. It must come from the
// auditor, never from the evidence: a node that deletes its oldest
// records must not shrink what gets checked (D22).
Since time.Time
// Now is the verification time; zero means time.Now.
Now time.Time
// MaxLag is how long a head may wait for its records to be exported.
MaxLag time.Duration
// Export supplies exported batches; nil means none.
Export ExportSource
// Local is a copy of the node's secrets.jsonl; nil means none.
Local io.Reader
// HiddenKeys are witness keys that carry a delete marker.
HiddenKeys []string
// NewestWitnessDay is the newest day with any witnessed version.
NewestWitnessDay string
}
NodeInput is everything the verifier knows about one node.
type NodeReport ¶ added in v0.6.7
type NodeReport struct {
NodeID string `json:"node_id"`
Result Result `json:"result"`
WindowStart time.Time `json:"window_start"`
HeadsBeforeWindow int `json:"heads_before_window"`
HeadsInWindow int `json:"heads_in_window"`
HeadsLinked int `json:"heads_linked"`
Pending []PendingHead `json:"pending,omitempty"`
Problems []string `json:"problems,omitempty"`
Warnings []string `json:"warnings,omitempty"`
Silent bool `json:"silent"`
RecordsChecked int64 `json:"records_checked"`
EvidenceFrom *time.Time `json:"evidence_from,omitempty"`
EvidenceTo *time.Time `json:"evidence_to,omitempty"`
ExportBatches int `json:"export_batches"`
// UnreachedBatches were never on the walk: re-sends and overlapping
// re-exports of records already checked, or forks.
UnreachedBatches int `json:"export_batches_unreached"`
// contains filtered or unexported fields
}
NodeReport is the verdict for one node and what it rests on.
func VerifyNode ¶ added in v0.6.7
func VerifyNode(ctx context.Context, in NodeInput) (NodeReport, error)
VerifyNode checks one node. An error means the inputs could not be read; a contradiction in what was read is a FAIL in the report instead.
type PendingHead ¶ added in v0.6.7
type PendingHead struct {
HeadHex string `json:"head"`
At time.Time `json:"witnessed_at"`
Ref string `json:"ref,omitempty"`
// Stale is true once the head has waited longer than MaxLag.
Stale bool `json:"stale"`
}
PendingHead is a head newer than the evidence.
type Result ¶ added in v0.6.7
type Result string
Result is a node's verdict.
const ( // ResultPass means every head in the window links to the next one. ResultPass Result = "PASS" // ResultFail means the evidence contradicts the witness, or is missing // for part of the window. ResultFail Result = "FAIL" // ResultIncomplete means some heads are newer than the evidence and have // been waiting for it longer than the allowed lag. Never a pass: a node // that stopped exporting could otherwise hide a forged newest head. ResultIncomplete Result = "INCOMPLETE" )
type ScanOptions ¶ added in v0.6.7
type ScanOptions struct {
// Base is the byte offset in the underlying file where the reader
// starts; every offset in ChainScan is reported in file coordinates.
Base int64
// Verifier, when set, is copied (never mutated) and continued from.
Verifier *ChainVerifier
// StartEventID and StartLineOffset describe the record carrying the
// primed verifier's head, reported back when nothing follows it.
StartEventID string
StartLineOffset int64
// Probe lists hashes whose presence the scan should report (witness
// ancestry) — O(len(Probe)) memory instead of O(records).
Probe []string
}
ScanOptions parameterize one pass. A nil Verifier starts at genesis; a primed one continues a chain from Base (checkpoint boot).
type SpillFile ¶
SpillFile is one audit directory's overflow buffer.
func SpillFileIn ¶
SpillFileIn names the spill buffer and lock of an audit directory.
func (SpillFile) Append ¶
Append durable-appends events with no capability (gap markers, daemon-side tests). Egress records need AppendRecords: an unauthenticated egress line is drained as a gap.
func (SpillFile) AppendRecords ¶
func (s SpillFile) AppendRecords(records []SpillRecord) error
AppendRecords durable-appends records as one write and one fsync under the audit lock. Events get an id and a time if they lack one. Either every line lands or the error covers the whole batch: a torn spill line is repaired by the daemon's drain (which counts it as a gap), so a crash mid-write loses at most this batch and never corrupts what came before.
type SpillRecord ¶
SpillRecord is one spill line: the event plus the sandbox-scoped capability the worker would have sent on the ingest header. The spill file is written by worker subprocesses and drained by the daemon, so every identity field on the line is unauthenticated until the drain verifies the capability and rebinds them from it — exactly what the HTTP ingest does. The capability is a bearer credential and never reaches secrets.jsonl.