Documentation
¶
Overview ¶
Package auditforward ships the audit trail off the hub as it is written (Task 20426).
The gap it closes ¶
The trail is hash-chained, so a row edited in place breaks the chain, and checkpoints (pkg/auditcheckpoint) pin each chain's length off the database, so a deleted tail is caught. Neither keeps the content: whoever can write state.db can delete rows nobody exported, and the verifier can then say only that something is missing. Until this package the content left the hub when an operator remembered to cron `cloop audit-log export | logger`.
A forwarder is the cluster leader's duty (pkg/ui runs it through runLeaderDuty): it tails every chain — the control plane's and each registered project's, the trail pkg/auditmerge reads as one — and sends each row to every sink in audit.forward: syslog over TLS (RFC 5424 messages, RFC 5425 octet-counted framing), HTTPS (an NDJSON POST, or Splunk HEC events), or a file outside every .cloop/ directory.
Delivery ¶
Each sink has a durable cursor per chain in the control plane's database (statedb.AuditForwardCursor): the newest row the sink acknowledged, by id and row_hash. A batch is read from the database after the cursor, sent, and the cursor advanced only once the sink has acknowledged it — by compare-and-swap against the position the batch was read from, so a leader that lost its lease mid-send cannot move a cursor its successor already moved. A leader killed after the sink acknowledged and before the cursor moved leaves the batch to be sent again: delivery is at-least-once, every record carries its row_hash as the idempotency key, and its prev_hash so a receiver can verify the chain it has received.
What counts as acknowledged depends on the sink: an https sink's 2xx (and a HEC body whose code is 0); a file sink's fsync; a syslog sink's receiver having acknowledged every byte at the TCP level (RFC 5425 has no application-level acknowledgement, so a receiver that crashes holding acknowledged bytes loses them — use https where that matters).
What a sink that is down costs ¶
Nothing an audit append waits on. Each sink has its own goroutine, reads the chains through its own handles, and backs off exponentially while it fails. The backlog stays in the database; the forwarder holds at most one batch per sink in memory however far behind a sink falls, and catches up batch by batch. Retention (pkg/auditretention) will not prune a row a live cursor has not passed unless forced, and the forwarder records it if one is forced.
Index ¶
- Constants
- Variables
- func IsSinkRepair(r config.LoadRepair) bool
- func RetentionHold(db *statedb.DB, dbPath, controlPlane string) (hold statedb.AuditForwardHold, held bool, consulted []string, err error)
- func Unhealthy(r Report, maxLag time.Duration, now time.Time) string
- type Chain
- type ChainReport
- type ChainStatus
- type ChainVerdict
- type Credentials
- type Forwarder
- type Hooks
- type IDRange
- type Options
- type Report
- type SinkDescription
- type SinkStatus
- type StreamReport
Constants ¶
const ( SourceControlPlane = "control-plane" SourceProject = "project" )
Chain roles, as records name them: the same words auditmerge and the checkpoints use.
const ( // ReasonPruned: rows after the cursor were pruned before the sink // acknowledged them. ReasonPruned = "pruned" // ReasonTruncated: the chain ends before the cursor. ReasonTruncated = "truncated" // ReasonRewritten: the cursor's row holds another hash. ReasonRewritten = "rewritten" // ReasonBroken: the cursor's row is intact, and what follows it does not // continue it — rows missing inside the chain, or relinked. The rows are // forwarded as they stand. ReasonBroken = "broken" )
Why a sink's cursor stopped lining up with its chain, as the audit.forward.discontinuity row says.
Variables ¶
var ErrHoldUnreadable = errors.New("auditforward: the audit forwarding cursors could not be read")
ErrHoldUnreadable is returned when a control plane that may hold cursors on the chain could not be read: a prune must not guess that nothing holds it.
var ErrMetadataAddress = errors.New("auditforward: refusing to connect to a cloud metadata service")
ErrMetadataAddress is the refusal to connect to a cloud metadata service.
var ErrPlaintextOffHost = errors.New("auditforward: refusing plaintext to a non-loopback address")
ErrPlaintextOffHost is the refusal to send plaintext anywhere but loopback.
Functions ¶
func IsSinkRepair ¶
func IsSinkRepair(r config.LoadRepair) bool
IsSinkRepair reports whether a load repair is a sink the loader refused.
func RetentionHold ¶
func RetentionHold(db *statedb.DB, dbPath, controlPlane string) (hold statedb.AuditForwardHold, held bool, consulted []string, err error)
RetentionHold reports the hold live audit sinks put on the chain held by db at dbPath: the newest row every one of them has acknowledged. held is false when no live cursor names the chain.
The cursors are looked for in db itself, in the control plane the chain's forwarded-by record names, and in controlPlane when it is set — a database the caller names that does not exist is an error, not "nothing holds it". A control plane other than db is opened read-only. consulted lists the databases read.
Types ¶
type Chain ¶
type Chain struct {
// Source is SourceControlPlane or SourceProject.
Source string
// Dir is the directory holding .cloop/state.db.
Dir string
// Path is the database's absolute path: what a cursor is keyed on.
Path string
}
Chain is one audit chain the forwarder tails.
type ChainReport ¶
type ChainReport struct {
Cursor statedb.AuditForwardCursor
// HeadID is the chain's newest row; Backlog how many rows after the
// cursor the sink has not acknowledged; PendingSince the oldest of
// those's timestamp, zero when there are none.
HeadID int64
Backlog int64
PendingSince time.Time
// Err is set when the chain could not be read.
Err error
}
ChainReport is the sink's position in one chain.
type ChainStatus ¶
type ChainStatus struct {
Chain Chain
// AckedID is the newest row the sink acknowledged; HeadID the newest
// row the chain held when last read.
AckedID int64
HeadID int64
// PendingSince is the timestamp of the oldest row the sink has not
// acknowledged; zero when it is caught up.
PendingSince time.Time
}
ChainStatus is one sink's position in one chain.
type ChainVerdict ¶
type ChainVerdict struct {
Source string `json:"source"`
Dir string `json:"dir"`
Hubs []string `json:"hubs,omitempty"`
FirstID int64 `json:"first_id"`
LastID int64 `json:"last_id"`
// Rows counts distinct rows; Duplicates the records that repeated one.
Rows int `json:"rows"`
Duplicates int `json:"duplicates"`
// Gaps are ids between FirstID and LastID the stream does not hold.
Gaps []IDRange `json:"gaps,omitempty"`
// Forks are ids held with more than one row_hash.
Forks []int64 `json:"forks,omitempty"`
// BadHashes are records whose content does not hash to their row_hash.
BadHashes []int64 `json:"bad_hashes,omitempty"`
// BrokenLinks are records whose prev_hash is not the row_hash of the
// record before them — or, for id 1, not genesis.
BrokenLinks []int64 `json:"broken_links,omitempty"`
// Unverified are records sent with their payload left out, whose hash
// cannot be recomputed; their links are still checked.
Unverified []int64 `json:"unverified,omitempty"`
}
ChainVerdict is what a received stream proves about one chain.
func (ChainVerdict) OK ¶
func (v ChainVerdict) OK() bool
OK reports whether the chain arrived whole and unaltered.
type Credentials ¶
Credentials opens a secret-store reference ("name" or "name#KEY") for a sink to present. The caller zeroes what it returns once it is in use.
type Forwarder ¶
type Forwarder struct {
// contains filtered or unexported fields
}
Forwarder ships every chain to every configured sink while Run runs.
func (*Forwarder) Run ¶
Run forwards until ctx ends: it starts a worker per configured sink, keeps the set in step with the configuration, and stops every worker before it returns. Safe to call again after it returns.
func (*Forwarder) Snapshot ¶
func (f *Forwarder) Snapshot() []SinkStatus
Snapshot reports every running sink as its worker last saw it, sorted by name. It is what the leader's metrics are read from.
type Hooks ¶
type Hooks struct {
// AfterAck runs after a sink acknowledged a batch and before its cursor
// moves. A test that never returns from it, or calls runtime.Goexit, is a
// leader killed between the two.
AfterAck func(sink string, chain Chain, firstID, lastID int64)
}
Hooks let a test stand where a process would die.
type Options ¶
type Options struct {
// ControlPlane is the control plane's database path: where the cursors
// live and where the forwarder's own audit rows go.
ControlPlane string
// Config returns the audit.forward section in force. Re-read every
// ConfigPoll, so an edit takes effect without a restart.
Config func() (config.AuditForwardConfig, error)
// ConfigSource names where Config read from, for the audit row recording
// a change ("config.yaml", or the overlay merged over it).
ConfigSource string
// Refused lists the sinks the loader switched off, by repair key, for the
// same audit row. May be nil.
Refused func() []string
// Chains returns the chains to forward, each database once, the control
// plane first. Re-read every ConfigPoll.
Chains func() []Chain
// Credentials opens the secrets sinks name. Nil makes every sink that
// names one fail to start, with that said.
Credentials Credentials
// Version is stamped into CEF records and the HTTP User-Agent.
Version string
// Poll is how often a caught-up sink looks for new rows. Zero means two
// seconds.
Poll time.Duration
// ConfigPoll is how often the configuration and the chain list are
// re-read. Zero means ten seconds.
ConfigPoll time.Duration
// Logf receives one line per noteworthy event: a sink failing, a
// discontinuity. Nil writes to stderr.
Logf func(format string, args ...any)
// Now overrides the clock. Nil means time.Now.
Now func() time.Time
// Hooks are for tests.
Hooks Hooks
}
Options configures a Forwarder.
type Report ¶
type Report struct {
Sink config.AuditForwardSink
Chains []ChainReport
}
Report is one configured sink as the databases say it stands.
func Status ¶
func Status(cpPath string, cfg config.AuditForwardConfig) ([]Report, error)
Status reports every configured sink, from the cursors in the control plane at cpPath and the head of each chain they name. A sink the forwarder has not started has no chains.
func (Report) Lag ¶
Lag is the age of the oldest row the sink has not acknowledged in any chain; zero when it is caught up.
func (Report) LastDelivery ¶
LastDelivery is when the sink last acknowledged a batch; zero when it never has.
type SinkDescription ¶
type SinkDescription struct {
Name string `json:"name"`
Type string `json:"type"`
Format string `json:"format"`
Endpoint string `json:"endpoint"`
Variant string `json:"variant,omitempty"`
Refs []string `json:"refs,omitempty"`
CAFile string `json:"ca_file,omitempty"`
ClientCert string `json:"client_cert,omitempty"`
ServerName string `json:"server_name,omitempty"`
// PlaintextLoopback is true for a sink acknowledged to send plaintext to
// a loopback collector.
PlaintextLoopback bool `json:"plaintext_loopback,omitempty"`
Start string `json:"start"`
Fingerprint string `json:"fingerprint"`
}
SinkDescription is what the audit trail records about a sink: everything that decides where and how its rows go, and no credential — the secret-store references are names.
func Describe ¶
func Describe(s config.AuditForwardSink) SinkDescription
Describe renders a sink for the audit trail and status output.
type SinkStatus ¶
type SinkStatus struct {
Name string
Type string
Endpoint string
// Chains is per chain, keyed by database path.
Chains map[string]ChainStatus
// LastError is the newest send failure, cleared by a delivery.
LastError string
LastErrorAt time.Time
// LastDeliveredAt is when the sink last acknowledged a batch.
LastDeliveredAt time.Time
}
SinkStatus is one sink as its worker last saw it.
func (SinkStatus) Backlog ¶
func (s SinkStatus) Backlog() int64
Backlog is how many rows the sink has not acknowledged, across chains.
type StreamReport ¶
type StreamReport struct {
Chains []ChainVerdict `json:"chains"`
// Lines counts the lines read; Skipped those that held no record, and
// Malformed those that started one and did not decode.
Lines int `json:"lines"`
Skipped int `json:"skipped"`
Malformed []int `json:"malformed,omitempty"`
}
StreamReport is VerifyStream's result.
func VerifyStream ¶
func VerifyStream(r io.Reader) (StreamReport, error)
VerifyStream reads a received stream — one record per line: the JSON a file sink or an NDJSON collector holds, a HEC event, or a syslog line with the record after its header — and verifies every chain in it.
func (StreamReport) OK ¶
func (r StreamReport) OK() bool
OK reports whether every chain arrived whole and unaltered.