auditforward

package
v0.0.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 16, 2026 License: MIT Imports: 34 Imported by: 0

Documentation

Overview

Package auditforward ships the audit trail off the hub as it is written (Task 20426).

The gap it closes

The trail is hash-chained, so a row edited in place breaks the chain, and checkpoints (pkg/auditcheckpoint) pin each chain's length off the database, so a deleted tail is caught. Neither keeps the content: whoever can write state.db can delete rows nobody exported, and the verifier can then say only that something is missing. Until this package the content left the hub when an operator remembered to cron `cloop audit-log export | logger`.

A forwarder is the cluster leader's duty (pkg/ui runs it through runLeaderDuty): it tails every chain — the control plane's and each registered project's, the trail pkg/auditmerge reads as one — and sends each row to every sink in audit.forward: syslog over TLS (RFC 5424 messages, RFC 5425 octet-counted framing), HTTPS (an NDJSON POST, or Splunk HEC events), or a file outside every .cloop/ directory.

Delivery

Each sink has a durable cursor per chain in the control plane's database (statedb.AuditForwardCursor): the newest row the sink acknowledged, by id and row_hash. A batch is read from the database after the cursor, sent, and the cursor advanced only once the sink has acknowledged it — by compare-and-swap against the position the batch was read from, so a leader that lost its lease mid-send cannot move a cursor its successor already moved. A leader killed after the sink acknowledged and before the cursor moved leaves the batch to be sent again: delivery is at-least-once, every record carries its row_hash as the idempotency key, and its prev_hash so a receiver can verify the chain it has received.

What counts as acknowledged depends on the sink: an https sink's 2xx (and a HEC body whose code is 0); a file sink's fsync; a syslog sink's receiver having acknowledged every byte at the TCP level (RFC 5425 has no application-level acknowledgement, so a receiver that crashes holding acknowledged bytes loses them — use https where that matters).

What a sink that is down costs

Nothing an audit append waits on. Each sink has its own goroutine, reads the chains through its own handles, and backs off exponentially while it fails. The backlog stays in the database; the forwarder holds at most one batch per sink in memory however far behind a sink falls, and catches up batch by batch. Retention (pkg/auditretention) will not prune a row a live cursor has not passed unless forced, and the forwarder records it if one is forced.

Index

Constants

View Source
const (
	SourceControlPlane = "control-plane"
	SourceProject      = "project"
)

Chain roles, as records name them: the same words auditmerge and the checkpoints use.

View Source
const (
	// ReasonPruned: rows after the cursor were pruned before the sink
	// acknowledged them.
	ReasonPruned = "pruned"
	// ReasonTruncated: the chain ends before the cursor.
	ReasonTruncated = "truncated"
	// ReasonRewritten: the cursor's row holds another hash.
	ReasonRewritten = "rewritten"
	// ReasonBroken: the cursor's row is intact, and what follows it does not
	// continue it — rows missing inside the chain, or relinked. The rows are
	// forwarded as they stand.
	ReasonBroken = "broken"
)

Why a sink's cursor stopped lining up with its chain, as the audit.forward.discontinuity row says.

Variables

View Source
var ErrHoldUnreadable = errors.New("auditforward: the audit forwarding cursors could not be read")

ErrHoldUnreadable is returned when a control plane that may hold cursors on the chain could not be read: a prune must not guess that nothing holds it.

View Source
var ErrMetadataAddress = errors.New("auditforward: refusing to connect to a cloud metadata service")

ErrMetadataAddress is the refusal to connect to a cloud metadata service.

View Source
var ErrPlaintextOffHost = errors.New("auditforward: refusing plaintext to a non-loopback address")

ErrPlaintextOffHost is the refusal to send plaintext anywhere but loopback.

Functions

func IsSinkRepair

func IsSinkRepair(r config.LoadRepair) bool

IsSinkRepair reports whether a load repair is a sink the loader refused.

func RetentionHold

func RetentionHold(db *statedb.DB, dbPath, controlPlane string) (hold statedb.AuditForwardHold, held bool, consulted []string, err error)

RetentionHold reports the hold live audit sinks put on the chain held by db at dbPath: the newest row every one of them has acknowledged. held is false when no live cursor names the chain.

The cursors are looked for in db itself, in the control plane the chain's forwarded-by record names, and in controlPlane when it is set — a database the caller names that does not exist is an error, not "nothing holds it". A control plane other than db is opened read-only. consulted lists the databases read.

func Unhealthy

func Unhealthy(r Report, maxLag time.Duration, now time.Time) string

Unhealthy says what is wrong with a sink, or "" when nothing is: no cursor yet — the hub has not run with it — rows waiting and nothing ever acknowledged, or the oldest waiting row older than maxLag.

Types

type Chain

type Chain struct {
	// Source is SourceControlPlane or SourceProject.
	Source string
	// Dir is the directory holding .cloop/state.db.
	Dir string
	// Path is the database's absolute path: what a cursor is keyed on.
	Path string
}

Chain is one audit chain the forwarder tails.

type ChainReport

type ChainReport struct {
	Cursor statedb.AuditForwardCursor
	// HeadID is the chain's newest row; Backlog how many rows after the
	// cursor the sink has not acknowledged; PendingSince the oldest of
	// those's timestamp, zero when there are none.
	HeadID       int64
	Backlog      int64
	PendingSince time.Time
	// Err is set when the chain could not be read.
	Err error
}

ChainReport is the sink's position in one chain.

type ChainStatus

type ChainStatus struct {
	Chain Chain
	// AckedID is the newest row the sink acknowledged; HeadID the newest
	// row the chain held when last read.
	AckedID int64
	HeadID  int64
	// PendingSince is the timestamp of the oldest row the sink has not
	// acknowledged; zero when it is caught up.
	PendingSince time.Time
}

ChainStatus is one sink's position in one chain.

type ChainVerdict

type ChainVerdict struct {
	Source string   `json:"source"`
	Dir    string   `json:"dir"`
	Hubs   []string `json:"hubs,omitempty"`

	FirstID int64 `json:"first_id"`
	LastID  int64 `json:"last_id"`
	// Rows counts distinct rows; Duplicates the records that repeated one.
	Rows       int `json:"rows"`
	Duplicates int `json:"duplicates"`

	// Gaps are ids between FirstID and LastID the stream does not hold.
	Gaps []IDRange `json:"gaps,omitempty"`
	// Forks are ids held with more than one row_hash.
	Forks []int64 `json:"forks,omitempty"`
	// BadHashes are records whose content does not hash to their row_hash.
	BadHashes []int64 `json:"bad_hashes,omitempty"`
	// BrokenLinks are records whose prev_hash is not the row_hash of the
	// record before them — or, for id 1, not genesis.
	BrokenLinks []int64 `json:"broken_links,omitempty"`
	// Unverified are records sent with their payload left out, whose hash
	// cannot be recomputed; their links are still checked.
	Unverified []int64 `json:"unverified,omitempty"`
}

ChainVerdict is what a received stream proves about one chain.

func (ChainVerdict) OK

func (v ChainVerdict) OK() bool

OK reports whether the chain arrived whole and unaltered.

type Credentials

type Credentials func(ref string) ([]byte, error)

Credentials opens a secret-store reference ("name" or "name#KEY") for a sink to present. The caller zeroes what it returns once it is in use.

type Forwarder

type Forwarder struct {
	// contains filtered or unexported fields
}

Forwarder ships every chain to every configured sink while Run runs.

func New

func New(opts Options) (*Forwarder, error)

New builds a Forwarder. It opens nothing until Run.

func (*Forwarder) Run

func (f *Forwarder) Run(ctx context.Context)

Run forwards until ctx ends: it starts a worker per configured sink, keeps the set in step with the configuration, and stops every worker before it returns. Safe to call again after it returns.

func (*Forwarder) Snapshot

func (f *Forwarder) Snapshot() []SinkStatus

Snapshot reports every running sink as its worker last saw it, sorted by name. It is what the leader's metrics are read from.

type Hooks

type Hooks struct {
	// AfterAck runs after a sink acknowledged a batch and before its cursor
	// moves. A test that never returns from it, or calls runtime.Goexit, is a
	// leader killed between the two.
	AfterAck func(sink string, chain Chain, firstID, lastID int64)
}

Hooks let a test stand where a process would die.

type IDRange

type IDRange struct {
	From int64 `json:"from"`
	To   int64 `json:"to"`
}

IDRange is an inclusive run of ids.

type Options

type Options struct {
	// ControlPlane is the control plane's database path: where the cursors
	// live and where the forwarder's own audit rows go.
	ControlPlane string

	// Config returns the audit.forward section in force. Re-read every
	// ConfigPoll, so an edit takes effect without a restart.
	Config func() (config.AuditForwardConfig, error)

	// ConfigSource names where Config read from, for the audit row recording
	// a change ("config.yaml", or the overlay merged over it).
	ConfigSource string

	// Refused lists the sinks the loader switched off, by repair key, for the
	// same audit row. May be nil.
	Refused func() []string

	// Chains returns the chains to forward, each database once, the control
	// plane first. Re-read every ConfigPoll.
	Chains func() []Chain

	// Credentials opens the secrets sinks name. Nil makes every sink that
	// names one fail to start, with that said.
	Credentials Credentials

	// Version is stamped into CEF records and the HTTP User-Agent.
	Version string

	// Poll is how often a caught-up sink looks for new rows. Zero means two
	// seconds.
	Poll time.Duration

	// ConfigPoll is how often the configuration and the chain list are
	// re-read. Zero means ten seconds.
	ConfigPoll time.Duration

	// Logf receives one line per noteworthy event: a sink failing, a
	// discontinuity. Nil writes to stderr.
	Logf func(format string, args ...any)

	// Now overrides the clock. Nil means time.Now.
	Now func() time.Time

	// Hooks are for tests.
	Hooks Hooks
}

Options configures a Forwarder.

type Report

type Report struct {
	Sink   config.AuditForwardSink
	Chains []ChainReport
}

Report is one configured sink as the databases say it stands.

func Status

func Status(cpPath string, cfg config.AuditForwardConfig) ([]Report, error)

Status reports every configured sink, from the cursors in the control plane at cpPath and the head of each chain they name. A sink the forwarder has not started has no chains.

func (Report) Backlog

func (r Report) Backlog() int64

Backlog is the rows the sink has not acknowledged, across chains.

func (Report) Delivered

func (r Report) Delivered() int64

Delivered is the rows the sink has acknowledged, across chains.

func (Report) Lag

func (r Report) Lag(now time.Time) time.Duration

Lag is the age of the oldest row the sink has not acknowledged in any chain; zero when it is caught up.

func (Report) LastDelivery

func (r Report) LastDelivery() time.Time

LastDelivery is when the sink last acknowledged a batch; zero when it never has.

func (Report) LastError

func (r Report) LastError() (string, time.Time)

LastError is the newest failure any of the sink's cursors records.

type SinkDescription

type SinkDescription struct {
	Name       string   `json:"name"`
	Type       string   `json:"type"`
	Format     string   `json:"format"`
	Endpoint   string   `json:"endpoint"`
	Variant    string   `json:"variant,omitempty"`
	Refs       []string `json:"refs,omitempty"`
	CAFile     string   `json:"ca_file,omitempty"`
	ClientCert string   `json:"client_cert,omitempty"`
	ServerName string   `json:"server_name,omitempty"`
	// PlaintextLoopback is true for a sink acknowledged to send plaintext to
	// a loopback collector.
	PlaintextLoopback bool   `json:"plaintext_loopback,omitempty"`
	Start             string `json:"start"`
	Fingerprint       string `json:"fingerprint"`
}

SinkDescription is what the audit trail records about a sink: everything that decides where and how its rows go, and no credential — the secret-store references are names.

func Describe

Describe renders a sink for the audit trail and status output.

type SinkStatus

type SinkStatus struct {
	Name     string
	Type     string
	Endpoint string

	// Chains is per chain, keyed by database path.
	Chains map[string]ChainStatus

	// LastError is the newest send failure, cleared by a delivery.
	LastError   string
	LastErrorAt time.Time
	// LastDeliveredAt is when the sink last acknowledged a batch.
	LastDeliveredAt time.Time
}

SinkStatus is one sink as its worker last saw it.

func (SinkStatus) Backlog

func (s SinkStatus) Backlog() int64

Backlog is how many rows the sink has not acknowledged, across chains.

func (SinkStatus) Lag

func (s SinkStatus) Lag(now time.Time) time.Duration

Lag is the age of the oldest row the sink has not acknowledged; zero when it is caught up.

type StreamReport

type StreamReport struct {
	Chains []ChainVerdict `json:"chains"`
	// Lines counts the lines read; Skipped those that held no record, and
	// Malformed those that started one and did not decode.
	Lines     int   `json:"lines"`
	Skipped   int   `json:"skipped"`
	Malformed []int `json:"malformed,omitempty"`
}

StreamReport is VerifyStream's result.

func VerifyStream

func VerifyStream(r io.Reader) (StreamReport, error)

VerifyStream reads a received stream — one record per line: the JSON a file sink or an NDJSON collector holds, a HEC event, or a syslog line with the record after its header — and verifies every chain in it.

func (StreamReport) OK

func (r StreamReport) OK() bool

OK reports whether every chain arrived whole and unaltered.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL