auditlog

package
v0.6.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 16, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// CapabilitySep separates HMAC capability fields.
	CapabilitySep = "|"
	// DefaultCapabilityTTL bounds how long a worker may emit egress events
	// with a given capability without refresh.
	DefaultCapabilityTTL = 24 * time.Hour
)
View Source
const (
	// KindRetentionCheckpoint is the immutable record retention inserts
	// where it dropped a prefix: its PrevHash is the last dropped EventHash,
	// and WitnessedThrough (when set) names the last externally witnessed
	// head the prefix carried.
	KindRetentionCheckpoint = "retention_checkpoint"
	// KindRetentionRedacted is what retention leaves of an expired record
	// it cannot cut out of the chain: its two link hashes and nothing that
	// says who opened what. See RedactForRetention.
	KindRetentionRedacted = "retention_redacted"

	// MaxLineBytes bounds one JSONL record during chain scans; audit lines
	// are metadata-only, so anything larger is corruption, not evidence.
	MaxLineBytes = 1024 * 1024
)
View Source
const (
	IndexKindOther byte = iota
	IndexKindSecretOpen
	IndexKindEgress
	IndexKindGap
	IndexKindRetentionCheckpoint
)

Index kind classes. Only decisive exclusions may use them: a filter for "egress" skips IndexKindSecretOpen entries but must still read IndexKindOther and IndexKindGap ones and decide on the record itself.

View Source
const (
	// LockFileName is the stable flock sidecar for the whole audit
	// directory. It carries no data; every writer, reader, drain and
	// retention pass takes it, so the file must never be renamed.
	LockFileName = "secrets.jsonl.lock"
	// SpillFileName is the overflow buffer the daemon's writer drains.
	SpillFileName = "secrets.spill.jsonl"
)
View Source
const GenesisPrevHash = "0"

GenesisPrevHash is the prev_hash of the first event in a chain.

Variables

View Source
var ErrIndexEntriesCorrupt = errors.New("audit index entries are corrupt")

ErrIndexEntriesCorrupt reports a posting list that does not decode. The index is derived data; the caller rebuilds it from the JSONL.

View Source
var ErrNoSpillPath = errors.New("audit spill path is unset")

ErrNoSpillPath reports a SpillFile with no destination.

Functions

func EnsureEventID

func EnsureEventID(ev *Event)

EnsureEventID assigns an ID exactly once.

func HashEvent

func HashEvent(prevHash string, ev Event) string

HashEvent returns SHA-256 hex of prevHash || canonical JSON of ev with EventHash cleared (so the hash does not depend on itself).

func LinkEvent

func LinkEvent(prevHash string, ev *Event)

LinkEvent sets PrevHash/EventHash on ev given the previous tip hash.

func LocalIncarnationID

func LocalIncarnationID(sandboxID, toolboxToken string) string

LocalIncarnationID derives a stable, non-secret lifecycle identifier from the per-create toolbox token. The token is regenerated whenever a sandbox ID is reused, so stale worker capabilities cannot cross into a replacement lifecycle. Only the digest is retained in audit ACL/event records.

func MintEgressCapability

func MintEgressCapability(key, sandboxID, incarnationID string, expiry time.Time) (string, error)

MintEgressCapability returns HMAC(sandboxID|incarnationID|expiry, key) hex-encoded as sandboxID|incarnationID|expiryUnix|mac. key is typically SB_AUDIT_INGEST_TOKEN (or a derived key). The mac binds the sandbox so a stolen capability for one sandbox cannot forge another sandbox's events.

func NewEventID

func NewEventID(nodeID string) string

NewEventID returns a fleet-unique audit event identifier. The normal path is 128 bits from crypto/rand. The fallback includes node, wall clock, process, and a process-local sequence so audit emission remains available even if the host entropy source fails.

func ParseAndVerifyEgressCapability

func ParseAndVerifyEgressCapability(key, capability string, now time.Time) (sandboxID, incarnationID string, err error)

ParseAndVerifyEgressCapability checks the HMAC and expiry. On success it returns the bound sandboxID and incarnationID (never trust client body for those fields when a capability is present).

func ReadLine added in v0.6.7

func ReadLine(br *bufio.Reader, max int64) (line []byte, consumed int64, terminated, tooLong bool, err error)

ReadLine returns one line (terminator included when present) and the bytes consumed. A line past max is counted but not retained so a NUL-filled or runaway tail cannot balloon the scan. max <= 0 means MaxLineBytes; the daemon's spill drain passes its ingest body cap so a worker cannot plant a line that fails strict boot.

func WithFileLock

func WithFileLock(lockPath string, fn func() error) error

WithFileLock runs fn holding an exclusive flock on lockPath. The sidecar is opened read-only: it only needs a stable descriptor, and a read-only open cannot hide a buffered-write failure on close.

Types

type ChainScan added in v0.6.7

type ChainScan struct {
	Head    string
	EventID string
	// LastLineStart is the offset of the record carrying Head.
	LastLineStart int64
	// Found reports which probed hashes the pass saw.
	Found map[string]bool
	// WitnessedThrough is the newest retention checkpoint's WitnessedThrough.
	WitnessedThrough string
	// ValidEnd is the byte offset just past the last complete, verified line.
	ValidEnd int64
	// TornBytes is the length of an unterminated, unparseable tail after
	// ValidEnd — what a crash leaves when it interrupts one append write. The
	// writer emits line+'\n' in a single write, so a crash leaves a strict
	// byte-prefix, and no proper prefix of a JSON object is itself valid JSON;
	// that is what makes this the only defect safe to classify as a tear.
	// ScanChain reports a tear rather than failing on it so the daemon's
	// boot repair can cut it back; every other reader must treat a non-zero
	// value as a chain it cannot verify, never quietly trim evidence.
	TornBytes int64
	// MissingNewline reports a complete, verified final record without its
	// terminator (the write stopped exactly between '}' and '\n').
	MissingNewline bool
	// Records counts verified records (blank lines excluded).
	Records int64
	// Redacted counts retention stubs among them: records whose payload
	// retention removed in place. The node never produces one inside the
	// retention window.
	Redacted int64
}

ChainScan is one verified pass over an audit JSONL stream.

func ScanChain added in v0.6.7

func ScanChain(r io.Reader, opts ScanOptions) (ChainScan, error)

ScanChain walks r once, verifying every record against its predecessor. Blank lines are tolerated but never alter the chain. r should be already bounded (a size snapshot taken under the audit flock) so a verification can run while the writer keeps appending. Memory is O(1) in the number of records.

type ChainVerifier added in v0.6.7

type ChainVerifier struct {
	// contains filtered or unexported fields
}

ChainVerifier checks one chain in stream order with O(1) state: the head so far, whether the next link may break (right after a retention checkpoint), and whether any record has been seen. The zero value is not ready; start from NewChainVerifier, or ResumeChainVerifier to continue a chain someone already verified up to a known head.

func NewChainVerifier added in v0.6.7

func NewChainVerifier() ChainVerifier

NewChainVerifier starts a verifier at genesis. It is a value so a caller can hand a copy to ScanChain and keep its own state unchanged.

func ResumeChainVerifier added in v0.6.7

func ResumeChainVerifier(head string, allowBreak bool) ChainVerifier

ResumeChainVerifier primes a verifier mid-chain: head is the EventHash of the last record already verified, and allowBreak is whether that record was a retention checkpoint (whose successor may name a dropped predecessor). Checkpoint boot, the export tailer and the read index all carry exactly this state across passes instead of re-reading the prefix, and a resumed verifier refuses a later retention checkpoint just as one that read the prefix itself would.

func (*ChainVerifier) Add added in v0.6.7

func (v *ChainVerifier) Add(ev Event) error

Add validates one immutable event in stream order. The only permitted link discontinuity is the first retained event after a retention checkpoint, whose predecessor was deliberately removed. Missing hash fields are never interpreted as a legacy format: this audit format is intentionally one-way.

func (*ChainVerifier) AllowBreak added in v0.6.7

func (v *ChainVerifier) AllowBreak() bool

AllowBreak reports whether the next record may name a predecessor other than Head — true only right after a retention checkpoint. Callers persist it alongside Head so a resumed pass keeps the same rule.

func (*ChainVerifier) Head added in v0.6.7

func (v *ChainVerifier) Head() string

Head is the EventHash of the last accepted record, or GenesisPrevHash when none has been accepted yet: what the next record's prev_hash must name.

type Event

type Event struct {
	Time          time.Time `json:"time"`
	Actor         string    `json:"actor,omitempty"`
	SandboxID     string    `json:"sandbox_id,omitempty"`
	Ref           string    `json:"ref,omitempty"`
	Result        string    `json:"result"`
	Reason        string    `json:"reason,omitempty"`
	CorrelationID string    `json:"correlation_id,omitempty"`
	EventID       string    `json:"event_id,omitempty"`
	NodeID        string    `json:"node_id,omitempty"`
	Kind          string    `json:"kind,omitempty"`
	Destination   string    `json:"destination,omitempty"`
	Network       string    `json:"network,omitempty"`
	BytesIn       int64     `json:"bytes_in,omitempty"`
	BytesOut      int64     `json:"bytes_out,omitempty"`
	// IncarnationID scopes the event to a placement incarnation so post-delete
	// ACL and recreate history do not cross tenant ownership boundaries.
	IncarnationID string `json:"incarnation_id,omitempty"`
	// OwnerRef is the tenant account the sandbox belonged to when the event
	// was recorded. It makes an exported record self-describing downstream
	// and lets a node authorize a post-delete read from the evidence itself,
	// without consulting Raft. Empty for operator-owned sandboxes. Being
	// omitempty, records written before the field existed hash identically.
	OwnerRef string `json:"owner_ref,omitempty"`
	// Dropped is set on gap markers to record how many events were coalesced
	// into this single overflow marker (lossy buffer evidence).
	Dropped int64 `json:"dropped,omitempty"`
	// PrevHash / EventHash form the local append-only hash chain used for
	// tamper detection once heads are witnessed off-node (E2b).
	PrevHash  string `json:"prev_hash,omitempty"`
	EventHash string `json:"event_hash,omitempty"`
	// WitnessedThrough is set on retention_checkpoint records to record the
	// last externally witnessed head that prune dropped (optional MVP field).
	WitnessedThrough string `json:"witnessed_through,omitempty"`
}

Event is the shared JSONL/wire representation for secret-open and egress audit records. Keeping the daemon, peer transport, and WASM worker on this single shape prevents fields that affect pagination from drifting.

func GapMarker

func GapMarker(node string, dropped int64, now time.Time) Event

GapMarker is the record that stands in for dropped events: one marker per flush, carrying the count, so overflow costs the writer one line rather than one per drop and the stream stays honest about the loss.

func RedactForRetention added in v0.6.7

func RedactForRetention(ev Event) Event

RedactForRetention is what retention leaves of an expired record that sits behind a newer one and so cannot be cut out of the chain: the two link hashes, the time and id (so a later prune reclaims it with the prefix and an operator can place it), and nothing that says who opened what. EventHash is the original — it is what the successor's prev_hash names — so the chain still verifies through the stub; the verifier links a stub by its stored hashes because the payload that produced them is gone, and refuses one that still carries any other field.

type ExportSource added in v0.6.7

type ExportSource interface {
	// Days lists the UTC dates ("2006-01-02") holding batches for nodeID,
	// ascending, starting at from.
	Days(ctx context.Context, nodeID, from string) ([]string, error)
	// Batches lists one day's batch names.
	Batches(ctx context.Context, nodeID, day string) ([]string, error)
	// Open reads one batch: one JSON event per line.
	Open(ctx context.Context, nodeID, day, batch string) (io.ReadCloser, error)
}

ExportSource reads a node's exported batches in the export layout: node=<id>/YYYY/MM/DD/<batch>.jsonl. Batch names are never used for order; the walk follows the chain itself.

type IndexEntry

type IndexEntry struct {
	Offset   int64 // first byte of the record
	Length   int64 // bytes through the trailing newline
	TimeNano int64 // Event.Time as UnixNano
	Kind     byte  // IndexKind*; anything unclassified is IndexKindOther
}

IndexEntry locates one record inside the local audit JSONL and carries just enough of it to page and filter without touching the file: the read path decides which lines a page needs from entries alone, then reads only those lines. Offsets are byte positions in the current file generation.

func AppendIndexEntries

func AppendIndexEntries(buf []byte, prev IndexEntry, hasPrev bool, entries []IndexEntry) ([]byte, IndexEntry, error)

AppendIndexEntries delta-encodes entries after prev. Within one posting list offsets are strictly increasing (file order) and times nearly so, so each entry costs a few bytes rather than the 25+ of a fixed layout — the index stays a small fraction of the log it describes. prev is the last entry already in buf; pass hasPrev=false for an empty list.

func DecodeIndexEntries

func DecodeIndexEntries(buf []byte) ([]IndexEntry, error)

DecodeIndexEntries is the inverse of AppendIndexEntries for a whole list.

type NodeInput added in v0.6.7

type NodeInput struct {
	NodeID string
	// Heads are the witnessed heads in witness order, oldest first. Heads
	// before Since may be included; they are listed, not checked.
	Heads []WitnessedHead
	// Prior is the newest head witnessed before the earliest of Heads, when
	// known: a head equal to it needs no records to link.
	Prior *WitnessedHead
	// Since is the window start the auditor chose. It must come from the
	// auditor, never from the evidence: a node that deletes its oldest
	// records must not shrink what gets checked (D22).
	Since time.Time
	// Now is the verification time; zero means time.Now.
	Now time.Time
	// MaxLag is how long a head may wait for its records to be exported.
	MaxLag time.Duration
	// Export supplies exported batches; nil means none.
	Export ExportSource
	// Local is a copy of the node's secrets.jsonl; nil means none.
	Local io.Reader
	// HiddenKeys are witness keys that carry a delete marker.
	HiddenKeys []string
	// NewestWitnessDay is the newest day with any witnessed version.
	NewestWitnessDay string
}

NodeInput is everything the verifier knows about one node.

type NodeReport added in v0.6.7

type NodeReport struct {
	NodeID            string        `json:"node_id"`
	Result            Result        `json:"result"`
	WindowStart       time.Time     `json:"window_start"`
	HeadsBeforeWindow int           `json:"heads_before_window"`
	HeadsInWindow     int           `json:"heads_in_window"`
	HeadsLinked       int           `json:"heads_linked"`
	Pending           []PendingHead `json:"pending,omitempty"`
	Problems          []string      `json:"problems,omitempty"`
	Warnings          []string      `json:"warnings,omitempty"`
	Silent            bool          `json:"silent"`
	RecordsChecked    int64         `json:"records_checked"`
	EvidenceFrom      *time.Time    `json:"evidence_from,omitempty"`
	EvidenceTo        *time.Time    `json:"evidence_to,omitempty"`
	ExportBatches     int           `json:"export_batches"`
	// UnreachedBatches were never on the walk: re-sends and overlapping
	// re-exports of records already checked, or forks.
	UnreachedBatches int `json:"export_batches_unreached"`
	// contains filtered or unexported fields
}

NodeReport is the verdict for one node and what it rests on.

func VerifyNode added in v0.6.7

func VerifyNode(ctx context.Context, in NodeInput) (NodeReport, error)

VerifyNode checks one node. An error means the inputs could not be read; a contradiction in what was read is a FAIL in the report instead.

type PendingHead added in v0.6.7

type PendingHead struct {
	HeadHex string    `json:"head"`
	At      time.Time `json:"witnessed_at"`
	Ref     string    `json:"ref,omitempty"`
	// Stale is true once the head has waited longer than MaxLag.
	Stale bool `json:"stale"`
}

PendingHead is a head newer than the evidence.

type Result added in v0.6.7

type Result string

Result is a node's verdict.

const (
	// ResultPass means every head in the window links to the next one.
	ResultPass Result = "PASS"
	// ResultFail means the evidence contradicts the witness, or is missing
	// for part of the window.
	ResultFail Result = "FAIL"
	// ResultIncomplete means some heads are newer than the evidence and have
	// been waiting for it longer than the allowed lag. Never a pass: a node
	// that stopped exporting could otherwise hide a forged newest head.
	ResultIncomplete Result = "INCOMPLETE"
)

type ScanOptions added in v0.6.7

type ScanOptions struct {
	// Base is the byte offset in the underlying file where the reader
	// starts; every offset in ChainScan is reported in file coordinates.
	Base int64
	// Verifier, when set, is copied (never mutated) and continued from.
	Verifier *ChainVerifier
	// StartEventID and StartLineOffset describe the record carrying the
	// primed verifier's head, reported back when nothing follows it.
	StartEventID    string
	StartLineOffset int64
	// Probe lists hashes whose presence the scan should report (witness
	// ancestry) — O(len(Probe)) memory instead of O(records).
	Probe []string
}

ScanOptions parameterize one pass. A nil Verifier starts at genesis; a primed one continues a chain from Base (checkpoint boot).

type SpillFile

type SpillFile struct {
	Path     string
	LockPath string
}

SpillFile is one audit directory's overflow buffer.

func SpillFileIn

func SpillFileIn(dir string) SpillFile

SpillFileIn names the spill buffer and lock of an audit directory.

func (SpillFile) Append

func (s SpillFile) Append(events []Event) error

Append durable-appends events with no capability (gap markers, daemon-side tests). Egress records need AppendRecords: an unauthenticated egress line is drained as a gap.

func (SpillFile) AppendRecords

func (s SpillFile) AppendRecords(records []SpillRecord) error

AppendRecords durable-appends records as one write and one fsync under the audit lock. Events get an id and a time if they lack one. Either every line lands or the error covers the whole batch: a torn spill line is repaired by the daemon's drain (which counts it as a gap), so a crash mid-write loses at most this batch and never corrupts what came before.

type SpillRecord

type SpillRecord struct {
	Event
	Capability string `json:"capability,omitempty"`
}

SpillRecord is one spill line: the event plus the sandbox-scoped capability the worker would have sent on the ingest header. The spill file is written by worker subprocesses and drained by the daemon, so every identity field on the line is unauthenticated until the drain verifies the capability and rebinds them from it — exactly what the HTTP ingest does. The capability is a bearer credential and never reaches secrets.jsonl.

type WitnessedHead added in v0.6.7

type WitnessedHead struct {
	HeadHex string
	// At is the witness's own time for the head (S3 LastModified).
	At time.Time
	// Ref says where the head is stored, for messages.
	Ref string
}

WitnessedHead is one head as the witness recorded it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL