Documentation
¶
Overview ¶
Package retention deletes metadata rows past an operator-configured age. It is opt-in per data class and runs on the scheduler leader only: runs and the audit log are kept by default, and the usage events (ADR 0096), which exist only once an operator turns their recording on, keep a 30 day window. Deletes are tenant scoped, bounded per statement, paused between batches and capped per cycle, so the janitor never turns into one long DELETE that holds locks or floods the WAL.
Index ¶
Constants ¶
const ( TableDagRuns = "dag_runs" TableTaskInstances = "task_instances" TableTaskInstanceHistory = "task_instance_history" TableXComIndex = "xcom_index" TableAuditLog = "audit_log" TableUsageEvents = "usage_events" )
Table names used as the `table` metric label.
Variables ¶
var ErrNotLeading = errors.New("retention: not leading")
ErrNotLeading ends a cycle when this instance lost leadership mid-cycle.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
DagRunsDays int
AuditLogDays int
UsageEventsDays int
DryRun bool
BatchSize int
BatchPause time.Duration
MaxRowsPerCycle int
}
Config is the janitor's view of the retention section.
type Counts ¶
type Counts struct {
DagRuns int64
TaskInstances int64
TaskInstanceHistory int64
XComIndex int64
AuditLog int64
UsageEvents int64
// Capped reports that a dry-run count stopped at its per-tenant cap, so
// the counts are a lower bound. Deletes never set it.
Capped bool
}
Counts is the number of rows deleted (or, in dry run, eligible) per table.
type Janitor ¶
type Janitor struct {
// contains filtered or unexported fields
}
Janitor runs retention cycles.
func (*Janitor) RunCycle ¶
RunCycle runs one cycle and logs its outcome. A cycle that fails or loses leadership is retried on the caller's next tick.
func (*Janitor) RunOnce ¶
RunOnce runs one retention cycle and returns what it deleted, or in dry run what it would delete. The lanes (each tenant's runs, then each scope's audit rows) take turns, one batch each, so under the cycle cap no tenant and no class waits behind another's backlog.
func (*Janitor) SetLeading ¶
SetLeading gates every batch on leadership: once leading reports false the cycle stops with ErrNotLeading before its next batch.
type Recorder ¶
type Recorder interface {
RecordRetentionDeleted(table string, n int64)
RecordRetentionEligible(table string, n int64)
ObserveRetentionCycle(d time.Duration)
}
Recorder receives the janitor's metrics.
type Store ¶
type Store interface {
// TenantIDs lists every tenant.
TenantIDs(ctx context.Context) ([]string, error)
// DeleteFinishedRuns removes, in one transaction, at most rowLimit rows of
// up to maxRuns settled runs of tenant that ended before cutoff, run rows
// included, children first. A run with more children than fit is finished
// by later calls, each of which re-checks that the run is still eligible.
DeleteFinishedRuns(ctx context.Context, tenant string, cutoff time.Time, maxRuns, rowLimit int) (Counts, error)
// DeleteAuditLog removes up to limit audit rows of tenant older than cutoff.
// An empty tenant means the tenant-less system rows.
DeleteAuditLog(ctx context.Context, tenant string, cutoff time.Time, limit int) (int64, error)
// DeleteUsageEvents removes up to limit usage events older than cutoff,
// oldest first in the order the service API serves them, and moves the
// pruned mark past them in the same transaction.
DeleteUsageEvents(ctx context.Context, cutoff time.Time, limit int) (int64, error)
// CountEligible counts what a cycle would delete, summed across all tenants;
// a nil cutoff skips that class.
CountEligible(ctx context.Context, runCutoff, auditCutoff, usageCutoff *time.Time) (Counts, error)
// RecordRetentionPurge writes an audit entry for rows of the audit log a
// cycle deleted in the scope of tenant ("" is the system rows).
RecordRetentionPurge(ctx context.Context, tenant string, rows int64, cutoff time.Time) error
}
Store is the storage the janitor deletes through.
Source Files