audit

Overview ¶

Package audit provides the pluggable audit persistence interface and concrete implementations for the Pasture epoch workflow audit trail.

The Trail interface is the core abstraction; all persistence is routed through it. Two implementations are provided:

• InMemoryAuditTrail — for testing and local development (non-durable).
• SqliteAuditTrail — for production use (durable, CGO-free via modernc.org/sqlite).

A module-level Trail singleton is also kept (see activities.go). It is injected via InitTrail before the durable runtime starts, for callers that cannot be handed a Trail directly.

Package audit — migrate.go ¶

Migrate is the public entry point for the audit-database schema migration framework introduced by PROPOSAL-2 (§7.10). It is invoked by NewSqliteAuditTrail on every open and (in later slices) by the `pasture migrate` CLI command — both paths share this single implementation, so behaviour is identical.

Versioning model

• Version is stored in the audit_schema_meta table (see schema_meta.go).
• A database with no audit_schema_meta table is treated as version 1 (legacy, pre-PROPOSAL-2).
• MaxKnownSchemaVersion is the highest version this binary knows how to produce. A database whose recorded version exceeds MaxKnownSchemaVersion is rejected with an actionable *StructuredError (Scenario 5).

Transactional guarantees

• Each forward step runs inside a single transaction acquired via BEGIN IMMEDIATE so that a concurrent writer cannot interleave with the migration.
• The audit_schema_meta version bump is the LAST statement in the transaction, so a crash mid-migration leaves the database observably at the previous version (rollback).

Migration table

• v1 → v2: bootstrap audit_schema_meta (S1, landed).
• v2 → v3: new tables context_edges, pasture_agent_categories, pasture_well_known_agents (S2, landed); audit_events.agent_id add + role-backfill + role-drop triple (S3, landed).
• v3 → v4: EpochContext backfill from audit_events.epoch_id into context_edges; audit_events.epoch_id column dropped (S4, landed — migrate_v3_v4.go).
• v4 → v5: additive dedup_key TEXT column on audit_events + a partial unique index over its non-NULL values, for engine exactly-once (migrate_v4_v5.go).
• v5 → v6: lifecycle payload blobs and replay-derived occurrences.
• v6 → v7: normalized byte-exact lifecycle occurrence bindings.
• v7 → v8: written_at stamp on lifecycle payload blobs; every row that existed before the column is stamped with the migration instant (migrate_v7_v8.go).
• v8 → v9: the retired assignment-index triggers and tables are dropped; session claims are kept (migrate_v8_v9.go).
• v9 → v10: the identity-format schema ceiling. A populated store without the slash identity marker is refused before any Provenance activation (migrate_v9_v10.go).

The ceiling is MaxKnownSchemaVersion below, not a number in this comment, and the steps this binary applies are the dispatch table migrationSteps(). The list above describes that table and is maintained by hand; the table and the constant are the authority, and the two cannot disagree silently: TestMigrate_FreshDB_LandsAtMaxKnownVersion runs Migrate on a fresh file and requires the recorded version to equal the constant, which fails when the table ends below it, and Migrate refuses a table with a gap. A future migration appends a step to migrationSteps(), bumps MaxKnownSchemaVersion, and adds its line to the list above.

Package audit — migrate_v1_v2.go

The v1→v2 migration is the bootstrap step that introduces the audit_schema_meta table. It is intentionally a near-no-op: it creates the meta table (if not already created by an earlier partial run) and seeds (version=2, applied_at=<now>) so that subsequent migrations can branch on the recorded version.

All schema work is wrapped in the migrator's enclosing transaction (see migrate.go). v1→v2 itself does NOT touch audit_events or any other existing table — pre-PROPOSAL-2 data is preserved verbatim.

Package audit — migrate_v2_v3.go

The v2→v3 migration adds three new tables that underpin the unified pasture workflow record (PROPOSAL-2 §7.2):

• context_edges — many-to-many event⇄context attachment, BCNF (composite-key, no non-key columns; §7.8).
• pasture_agent_categories — typed categorisation rows for Provenance agents (R8); written by SetAgentCategories.
• pasture_well_known_agents — stable logical-name → AgentId mapping for idempotent automaton registration at daemon startup (BLOCKER A2). UAT-1 schema invariant: (agent_id PK, name UNIQUE).

IMPORTANT scope boundary: this slice (S2) does NOT touch existing audit_events rows. The audit_events.agent_id column add + role-backfill + role-drop triple lives in S3 (migrate_v3 backfill) per BLOCKER A1, which requires the entire (create-column → backfill → drop-role) sequence to run in one BEGIN IMMEDIATE transaction. Workers integrating with this file downstream must not insert any audit_events DDL into migrateV2toV3.

Layer Integration Points exposed by this slice:

• pasture_well_known_agents — DDL here; rows written by S7 (startup registration); cached AgentIDs read by S8 (activity attribution).
• context_edges — DDL here; consumed by S5 (AttachContext writes), S6 (`pasture task contexts` reads), S8 (epoch attachment), S9 (free-floating contexts).
• pasture_agent_categories — DDL here; written by S5 SetAgentCategories; read by S6 (`pasture task agents` listing).

Package audit — migrate_v3_backfill.go

S3 backfill: extends the v2→v3 transition (begun by S2 in migrate_v2_v3.go) with the audit_events.agent_id add + role-backfill + role-drop triple.

All five steps below run inside the SAME BEGIN IMMEDIATE transaction owned by migrate.runStep — including the audit_schema_meta version bump from 2 to 3, which is the LAST statement before tx.Commit per BLOCKER A1 (PROPOSAL-2 §7.10.2). A crash mid-way leaves the file at v2 (the transaction is rolled back); on the next open, Migrate observes MAX(version)=2 and re-runs the v3 step from scratch with no orphan agents_software rows (idempotency guaranteed by the find-or-create branch in step 3).

Pseudocode parity with PROPOSAL-2 §7.10.2:

1. ALTER TABLE audit_events ADD COLUMN agent_id TEXT
2. SELECT DISTINCT role FROM audit_events
3. For each role: find-or-create a SoftwareAgent in agents_software named "pasture/legacy-role/<role>" via raw SQL (no Provenance Go API call). Find query: SELECT a.id FROM agents a JOIN agents_software s ... WHERE a.kind_id = 2 AND s.name = ? LIMIT 1. If absent, INSERT into agents (id, kind_id=2) and agents_software (agent_id, name, version, source). The INSERT shape MUST match Provenance's RegisterSoftwareAgent (provenance/internal/sqlite/agents.go:96-121) byte-for-byte so a future Provenance lookup of the agent succeeds.
4. UPDATE audit_events SET agent_id = ? WHERE role = ? AND agent_id IS NULL
5. SQLite table-rebuild to drop the role column: CREATE NEW → INSERT SELECT → DROP OLD → RENAME → recreate indexes.

C4 compliance: Provenance is NOT modified. The audit migrator opens its own *sql.DB against the same shared SQLite file and writes raw SQL that matches Provenance's INSERT shape exactly. Find-or-create on (kind_id=2, name) is keyed precisely so a partial run that crashed after agent creation but before the version bump reuses the existing row on retry rather than creating a duplicate.

Package audit — migrate_v3_backfill_export.go

Test-only public entry point so the cmd/pasture-migrate-crash binary can run the v3 backfill body (steps 0–5: ensure Provenance tables, add agent_id column, distinct roles, find-or-create agents, backfill agent_id, table-rebuild) WITHOUT the audit_schema_meta version bump or tx.Commit. The crash binary uses this to stage the partial v3 transaction state Scenario 11 needs to assert on.

Why this isn't gated by build tags

The cmd/pasture-migrate-crash binary is built by the standard `make build` target (HANDOFF §7) so it ships in dist/ alongside the production binaries. Build-tag gating would require a parallel build path; given the binary is small, useful only in test contexts, and imports from internal/audit (so accidental production use in any surface besides cmd/pasture-migrate-crash is not a concern), we expose this function unconditionally.

Workers extending the v3 backfill MUST keep the body of migrateV3Backfill in sync with what this function exposes — it is a thin re-export of the same private helper.

Package audit — migrate_v3_v4.go

S4 EpochContext migration: extends the schema chain with the v3→v4 step.

The v4 step does two pieces of work in the SAME BEGIN IMMEDIATE transaction owned by migrate.runStep:

1. Backfill context_edges from audit_events.epoch_id. For every row in audit_events with non-NULL epoch_id, INSERT a context_edges (event_id, 'EpochContext', epoch_id) row. Uses INSERT OR IGNORE so a defensive replay (impossible in practice — the whole step is one transaction — but safe defensively) is a no-op against the composite PK (event_id, context_kind, context_id).
2. SQLite table-rebuild to drop the epoch_id column from audit_events: CREATE NEW (without epoch_id) → INSERT SELECT → DROP OLD → RENAME → recreate post-v3 indexes.

The audit_schema_meta version bump from 3 to 4 is the LAST statement in the transaction (per BLOCKER A1 / PROPOSAL-2 §7.10.1 v4 row). A crash before tx.Commit rolls everything back, leaving the file observably at v3; the next open re-runs the v4 step from scratch (idempotent because step 1 uses INSERT OR IGNORE and step 2 lives entirely inside the rolled-back transaction).

Migration-note guarantee (PROPOSAL-2 §7.12, last paragraph)

Legacy audit_events.epoch_id values are migrated AS-IS into context_edges, regardless of whether the string parses as a Provenance TaskId. Free-string epoch IDs (e.g. "epoch-2026-04-22-mvp") and TaskIDs in the retired two-dash form (e.g. "aura-plugins--01968a3c-1234-7000-8000-...") are both preserved because they are historical records and rejecting them would lose audit data. The §7.12 ParseTaskId validation applies only to NEW workflow starts post-migration; that validation is owned by S8.

Pseudocode parity with PROPOSAL-2 §7.10.1 v4 row:

1. INSERT INTO context_edges (event_id, context_kind, context_id) SELECT id, 'EpochContext', epoch_id FROM audit_events WHERE epoch_id IS NOT NULL
2. CREATE TABLE audit_events_new (no epoch_id column)
3. INSERT INTO audit_events_new SELECT (no epoch_id) FROM audit_events
4. DROP TABLE audit_events
5. ALTER TABLE audit_events_new RENAME TO audit_events
6. CREATE INDEX idx_audit_events_agent ON audit_events (agent_id)
7. CREATE INDEX idx_audit_events_timestamp ON audit_events (timestamp)
8. (Caller — migrate.runStep wrapper — calls writeVersion(4, ...) and Commit.)

Package audit — migrate_v4_v5.go

The v4→v5 step adds a deterministic deduplication key to audit_events so the durable engine can record a forensic row exactly once even when a crashed step replays.

Two additive statements run in the SAME BEGIN IMMEDIATE transaction owned by runStep, then the version bump:

1. ALTER TABLE audit_events ADD COLUMN dedup_key TEXT
2. CREATE UNIQUE INDEX idx_audit_events_dedup ON audit_events(dedup_key) WHERE dedup_key IS NOT NULL

The uniqueness is a PARTIAL unique index, not a column UNIQUE constraint: SQLite's ALTER TABLE ADD COLUMN forbids adding a UNIQUE/PK constraint, and a column constraint would force a full table rebuild. A partial index on the non-NULL keys lets legacy/non-engine rows (which leave dedup_key NULL) coexist freely — multiple NULLs are allowed — while still rejecting a duplicate engine emission. The v4 CREATE TABLE shape is left untouched.

dedup_key is NULL for every existing row and for any writer that does not supply one, so the column is additive and back-compatible: existing databases migrate without data loss.

Package audit — schema_meta.go

The audit_schema_meta table records the on-disk schema version of the pasture audit database. It is the single source of truth read by the migrator on every open to decide which forward migrations (if any) need to run, and to detect a database that is newer than the running binary supports.

Schema (per PROPOSAL-2 §7.10.1):

version    INTEGER PRIMARY KEY  — monotonically increasing schema version
applied_at INTEGER NOT NULL     — Unix nanoseconds UTC when the row landed

The table is created lazily by audit.Migrate (NOT by ensureSchema) so that pre-existing v1 databases — which predate this table — are still openable and detectable as "no audit_schema_meta row yet ⇒ version 1".