cleanup

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 2, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package cleanup is the retention center: a registry of cleanup targets, a scheduler that runs them under per-target modes, and a run history.

Index

Constants

View Source
const (
	TargetAuditLog        = "audit.log"
	TargetEmailLog        = "messaging.email_log"
	TargetWebhookDelivery = "messaging.webhook_deliveries"
	TargetEventWebhook    = "messaging.event_webhook_deliveries"
	TargetAlertHistory    = "observability.alert_history"
	TargetAPITokens       = "auth.api_tokens" //nolint:gosec // target id, not a credential

)

Built-in target ids.

View Source
const (
	EnvMaxRows      = "APP_CLEANUP_MAX_ROWS"
	EnvSlice        = "APP_CLEANUP_SLICE"
	EnvStagger      = "APP_CLEANUP_STAGGER"
	EnvStartupDelay = "APP_CLEANUP_STARTUP_DELAY"
	EnvHistoryDays  = "APP_CLEANUP_HISTORY_DAYS"
	EnvUsageTTL     = "APP_CLEANUP_USAGE_TTL"
)

Environment variables that bound a run.

View Source
const (
	TargetAuthExpired     = "auth.expired_credentials"
	TargetAIHistory       = "data.ai_history"
	TargetRestoreHistory  = "data.restore_history"
	TargetProbeAttempts   = "deploys.probe_attempts"
	TargetAppHistory      = "deploys.app_history"
	TargetUsageCost       = "observability.usage_cost"
	TargetStatusIncidents = "observability.status_incidents"
)

Targets for history that had no sweeper before. They start in preview.

View Source
const (
	EventRun    = "cleanup.run"
	EventFailed = "cleanup.failed"
)

Event types emitted through the event webhook mechanism.

View Source
const AuditActionRun = "cleanup_run"

AuditActionRun is the audit action written for every run, dry runs included.

Variables

View Source
var (
	ErrUnknownTarget = errors.New("cleanup: unknown target")
	ErrBusy          = errors.New("cleanup: this target is already running")
	ErrReadOnly      = errors.New("cleanup: this target is managed elsewhere")
	ErrNoRetention   = errors.New("cleanup: retention is set to keep forever, pass days to clean anyway")
	ErrInvalidDays   = errors.New("cleanup: days must be a positive number")
)

Errors the center returns; handlers map them to HTTP statuses.

Default is the process-wide registry whose mode settings join the runtime settings registry.

View Source
var RunsCSVHeader = []string{"id", "target", "started_at", "finished_at", "days", "dry_run", "rows", "bytes", "trigger", "actor", "error"}

RunsCSVHeader is the column order of the CSV export.

Functions

func AuditWindows

func AuditWindows(general, security int) (int, int)

AuditWindows applies the audit floors and the rule that security entries outlive the general window. 0 means keep forever; a general window of 0 keeps everything whatever the security value says.

func ModeDefinition

func ModeDefinition(m Meta) runtimesettings.Definition

ModeDefinition is the runtime setting for m's mode.

func ModeKey

func ModeKey(id string) string

ModeKey is the runtime setting holding a target's mode.

func Register

func Register(t Target)

Register adds t to the default registry.

func RegisterBuiltins

func RegisterBuiltins(r *Registry, d BuiltinDeps)

RegisterBuiltins registers the targets that wrap existing sweepers, the token cleanup and the janitor stages.

func SweepByCutoff

func SweepByCutoff(ctx context.Context, oldest, cutoff time.Time, slice time.Duration, maxRows int64, del func(ctx context.Context, cutoff time.Time) (int64, error)) (rows int64, more bool, err error)

SweepByCutoff removes rows older than cutoff by calling del with a cutoff that advances one slice at a time from the oldest row, so each statement covers a bounded span and the row cap can stop the run between slices. del must delete rows strictly older than the cutoff it gets; nothing at or after cutoff is touched.

func WriteRunsCSV

func WriteRunsCSV(w io.Writer, runs []RunRecord) error

WriteRunsCSV writes runs as CSV. Cells that a spreadsheet would read as a formula are prefixed with a quote.

Types

type AlertHistory

type AlertHistory interface {
	HistoryStats(ctx context.Context) (rows int64, oldest time.Time, bytes int64, err error)
	CountHistoryBefore(ctx context.Context, cutoff time.Time) (int64, error)
	PruneHistory(ctx context.Context, cutoff time.Time) (int64, error)
}

AlertHistory is the alert_history access. *alerting.DB satisfies it.

type AuditSaver

type AuditSaver interface {
	SaveAuditEntry(ctx context.Context, e store.AuditEntry) error
}

AuditSaver writes audit rows. *store.DB satisfies it.

type BuiltinDeps

type BuiltinDeps struct {
	Store   SweepStore
	Alerts  AlertHistory
	Tokens  TokenSource
	Janitor JanitorSource
	// Retention enables the targets for history that had no sweeper before; nil skips them.
	Retention RetentionStore
	Settings  Settings
	Now       func() time.Time
}

BuiltinDeps are the collaborators of the built-in targets. A nil Alerts, Tokens or Janitor skips those targets.

type Category

type Category string

Category groups targets in the dashboard.

const (
	CategoryAuth          Category = "auth"
	CategoryAudit         Category = "audit"
	CategoryDeploys       Category = "deploys"
	CategoryDocker        Category = "docker"
	CategoryData          Category = "data"
	CategoryObservability Category = "observability"
	CategoryMessaging     Category = "messaging"
	CategoryOther         Category = "other"
)

Target categories.

type Center

type Center struct {
	// contains filtered or unexported fields
}

Center runs targets, keeps history and drives the scheduler.

func New

func New(cfg Config) *Center

New builds a Center, filling the defaults a caller left out.

func (*Center) Get

func (c *Center) Get(ctx context.Context, id string) (TargetView, error)

Get returns one target's view.

func (*Center) List

func (c *Center) List(ctx context.Context) (Overview, error)

List returns every target with its usage, mode and last run.

func (*Center) ModeOf

func (c *Center) ModeOf(ctx context.Context, m Meta) Mode

ModeOf resolves the mode of a target from its runtime setting.

func (*Center) Preview

func (c *Center) Preview(ctx context.Context, id string, days *int) (Impact, error)

Preview reports what a run with days (or the saved policy when nil) would remove, writing nothing.

func (*Center) PreviewCount

func (c *Center) PreviewCount(ctx context.Context) int

PreviewCount is how many targets only measure now, for the first-run review. Read-through targets count when the janitor is in dry-run, which is a real preview state.

func (*Center) Run

func (c *Center) Run(ctx context.Context, id string, req Request) (RunRecord, Result, error)

Run executes a run on demand and records it. A returned error is a pre-flight refusal; a failure of the run itself is in Result.Err and in history.

func (*Center) Runs

func (c *Center) Runs(ctx context.Context, f RunsFilter) (RunsPage, error)

Runs lists history. It reads one extra row to know whether another page exists.

func (*Center) Start

func (c *Center) Start(ctx context.Context)

Start runs the scheduler until ctx ends. Each target is due at its last run plus the interval, or at the startup delay plus its stagger slot before it has run; the schedule is re-read every cycle so a changed setting applies without a restart.

func (*Center) Tick

func (c *Center) Tick(ctx context.Context)

Tick runs one scheduled pass: every target whose mode is on (a real run) or preview (a dry run), spaced by the stagger.

type Config

type Config struct {
	Registry *Registry
	Store    RunStore
	Settings Settings
	Audit    AuditSaver
	// Emit hands an event to the event webhook queue; nil drops events.
	Emit func(eventType string, data map[string]any)
	// Notify sends a plain-text notice to the alert channels; nil skips notices.
	Notify func(ctx context.Context, text string) error
	Limits Limits
	Logger *slog.Logger
	Now    func() time.Time
}

Config wires a Center.

type FloorError

type FloorError struct {
	Key   string
	Floor int
}

FloorError means a requested window is shorter than the target's minimum.

func (*FloorError) Error

func (e *FloorError) Error() string

type Impact

type Impact struct {
	Rows  int64 `json:"rows"`
	Bytes int64 `json:"bytes"`
}

Impact is what a run with a given window would remove.

type JanitorSource

type JanitorSource interface {
	janitor.PolicyStore
	ListJanitorRuns(ctx context.Context, limit int) ([]store.JanitorRun, error)
}

JanitorSource reads the janitor policy and its recorded passes. *store.DB satisfies it.

type Limits

type Limits struct {
	// MaxRows caps the rows one run removes; the run reports more_remaining when it stops there.
	MaxRows int64
	// Slice is the time span of one delete batch, so no single DELETE covers an unbounded range.
	Slice time.Duration
	// Stagger spaces targets within one scheduler tick.
	Stagger time.Duration
	// StartupDelay keeps the scheduler quiet right after boot.
	StartupDelay time.Duration
	// HistoryDays is how long run history is kept.
	HistoryDays int
	// UsageTTL is how long a measured usage is reused by the list endpoint.
	UsageTTL time.Duration
}

Limits bound how much one run and the scheduler do.

func LimitsFromEnv

func LimitsFromEnv(lookup func(string) (string, bool)) Limits

LimitsFromEnv reads Limits, falling back to the defaults for unset or invalid values.

type Managed

type Managed interface {
	State(ctx context.Context) (ManagedState, error)
}

Managed is implemented by read-through targets that report their own state.

type ManagedState

type ManagedState struct {
	Days   int
	Mode   Mode
	DryRun bool
}

ManagedState is the policy a read-through target shows.

type Meta

type Meta struct {
	// ID is dotted, for example "audit.log".
	ID       string
	Category Category
	// Title and Description are English fallbacks for the i18n keys.
	TitleKey, Title      string
	DescKey, Description string
	// PolicySetting is the runtime setting holding the retention days.
	PolicySetting string
	// Floor is the shortest allowed retention in days, 0 for none.
	Floor       int
	Destructive bool
	Origin      Origin
	// Presets lists the day counts the dashboard offers; empty means the standard presets.
	Presets []int
	// ReadOnly marks a target whose policy and runs live elsewhere (the janitor); Link says where.
	ReadOnly bool
	Link     string
}

Meta is a target's static description.

type Mode

type Mode string

Mode is what the scheduler does with a target.

const (
	ModeOff     Mode = "off"
	ModePreview Mode = "preview"
	ModeOn      Mode = "on"
)

Target modes.

func DefaultMode

func DefaultMode(o Origin) Mode

DefaultMode is the mode a target starts in: unchanged behavior for existing sweepers, preview for new ones.

type Origin

type Origin string

Origin says whether a target replaces an existing sweeper or covers data that was never bounded.

const (
	OriginExisting Origin = "existing"
	OriginNew      Origin = "new"
)

Target origins.

type Overview

type Overview struct {
	Targets   []TargetView  `json:"targets"`
	Scheduler SchedulerView `json:"scheduler"`
}

Overview is the body of GET /api/v1/cleanup.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry holds the registered targets in registration order.

func NewRegistry

func NewRegistry(registerSetting func(runtimesettings.Definition)) *Registry

NewRegistry builds a registry. registerSetting receives each target's mode setting; nil skips registration, for tests.

func (*Registry) Get

func (r *Registry) Get(id string) (Target, bool)

Get finds a target by id.

func (*Registry) Register

func (r *Registry) Register(t Target)

Register adds t and its mode setting. It panics on a malformed or duplicate target so a bad registration fails at start.

func (*Registry) Targets

func (r *Registry) Targets() []Target

Targets returns a copy of the registered targets.

type Request

type Request struct {
	// Days overrides the saved policy when set.
	Days   *int
	DryRun bool
	Opts   RunOptions
}

Request is one manual or API run.

type Result

type Result struct {
	Rows  int64
	Bytes int64
	More  bool
	Err   error
}

Result is what a run removed. More says the row cap stopped it early.

type RetentionStore

type RetentionStore interface {
	RetentionStats(ctx context.Context, t store.RetentionTable) (store.CleanupTableStats, error)
	RetentionCountOlder(ctx context.Context, t store.RetentionTable, cutoff time.Time) (int64, error)
	RetentionDeleteOlder(ctx context.Context, t store.RetentionTable, cutoff time.Time, batch int) (int64, error)
}

RetentionStore is the table access of the age-bounded history targets. *store.DB satisfies it.

type RunOptions

type RunOptions struct {
	DryRun  bool
	Trigger Trigger
	Actor   string
	// ActorType, ActorID and RemoteAddr fill the audit row; empty means a system actor.
	ActorType  string
	ActorID    string
	RemoteAddr string
	// MaxRows caps one run; Slice is the time span of one delete batch. The center fills both from its limits.
	MaxRows int64
	Slice   time.Duration
}

RunOptions say how one run behaves.

type RunRecord

type RunRecord struct {
	ID         int64     `json:"id"`
	Target     string    `json:"target"`
	StartedAt  time.Time `json:"started_at"`
	FinishedAt time.Time `json:"finished_at"`
	Days       int       `json:"days"`
	DryRun     bool      `json:"dry_run"`
	Rows       int64     `json:"rows"`
	Bytes      int64     `json:"bytes"`
	Trigger    string    `json:"trigger"`
	Actor      string    `json:"actor"`
	Error      string    `json:"error"`
}

RunRecord is one history row on the wire.

type RunStore

type RunStore interface {
	InsertCleanupRun(ctx context.Context, r store.CleanupRun) (int64, error)
	ListCleanupRuns(ctx context.Context, f store.CleanupRunFilter) ([]store.CleanupRun, error)
	LatestCleanupRuns(ctx context.Context) ([]store.CleanupRun, error)
	CleanupFailedSince(ctx context.Context, target string, since time.Time) (bool, error)
	PruneCleanupRuns(ctx context.Context, before time.Time) (int64, error)
}

RunStore persists run history. *store.DB satisfies it.

type RunsFilter

type RunsFilter struct {
	Target string
	Since  time.Time
	Until  time.Time
	DryRun *bool
	Status string
	Limit  int
	Cursor int64
}

RunsFilter narrows Runs.

type RunsPage

type RunsPage struct {
	Runs       []RunRecord
	NextCursor int64
}

RunsPage is one page of history, newest first. NextCursor is 0 on the last page.

type SchedulerView

type SchedulerView struct {
	Enabled  bool   `json:"enabled"`
	Interval string `json:"interval"`
}

SchedulerView is the scheduler state on the wire.

type Settings

type Settings interface {
	Int(ctx context.Context, key string) int
	String(ctx context.Context, key string) string
	Bool(ctx context.Context, key string) bool
	Duration(ctx context.Context, key string) time.Duration
}

Settings resolves runtime settings. *runtimesettings.Service satisfies it, and a nil one yields env then defaults.

type SweepStore

type SweepStore interface {
	CleanupStats(ctx context.Context, t store.CleanupTable) (store.CleanupTableStats, error)
	CleanupCountOlder(ctx context.Context, t store.CleanupTable, cutoff time.Time) (int64, error)
	CountAuditEntriesTiered(ctx context.Context, cutoff, securityCutoff time.Time) (int64, error)
	DeleteAuditEntriesTiered(ctx context.Context, cutoff, securityCutoff time.Time) (int64, error)
	PruneEmailDeliveries(ctx context.Context, before time.Time) (int64, error)
	DeleteWebhookDeliveriesOlderThan(ctx context.Context, cutoff time.Time) (int64, error)
	DeleteEventDeliveriesOlderThan(ctx context.Context, cutoff string) (int64, error)
}

SweepStore is the table access the built-in sweeper targets use. *store.DB satisfies it.

type Target

type Target interface {
	Meta() Meta
	Measure(ctx context.Context) (Usage, error)
	Preview(ctx context.Context, days int) (Impact, error)
	// Run removes data older than days. With opts.DryRun it must not write.
	Run(ctx context.Context, days int, opts RunOptions) Result
}

Target is one kind of data the center can clean.

func Get

func Get(id string) (Target, bool)

Get finds a target in the default registry.

func Targets

func Targets() []Target

Targets lists the default registry's targets.

type TargetView

type TargetView struct {
	ID             string     `json:"id"`
	Category       string     `json:"category"`
	Title          string     `json:"title"`
	TitleKey       string     `json:"title_key,omitempty"`
	Description    string     `json:"description"`
	DescriptionKey string     `json:"description_key,omitempty"`
	PolicyKey      string     `json:"policy_key"`
	Days           int        `json:"days"`
	Source         string     `json:"source"`
	Mode           string     `json:"mode"`
	Origin         string     `json:"origin"`
	Destructive    bool       `json:"destructive"`
	Floor          int        `json:"floor"`
	Presets        []int      `json:"presets"`
	Usage          UsageView  `json:"usage"`
	UsageError     string     `json:"usage_error,omitempty"`
	LastRun        *RunRecord `json:"last_run"`
	NextRun        *time.Time `json:"next_run"`
	ReadThrough    bool       `json:"read_through"`
	Link           string     `json:"link,omitempty"`
	DryRun         *bool      `json:"dry_run,omitempty"`
}

TargetView is one target on the wire.

type TokenSource

type TokenSource = janitor.TokenPurger

TokenSource deletes inactive API tokens. *store.DB satisfies it.

type Trigger

type Trigger string

Trigger records what started a run.

const (
	TriggerScheduled Trigger = "scheduled"
	TriggerManual    Trigger = "manual"
	TriggerCLI       Trigger = "cli"
	TriggerAPI       Trigger = "api"
)

Run triggers.

type Usage

type Usage struct {
	Rows    int64
	Bytes   int64
	Oldest  *time.Time
	NoBytes bool
}

Usage is what a target holds now. NoBytes marks a store that cannot report size, so the wire says null.

type UsageView

type UsageView struct {
	Rows   int64      `json:"rows"`
	Bytes  *int64     `json:"bytes"`
	Oldest *time.Time `json:"oldest"`
}

UsageView is Usage on the wire: bytes is null when size is not measurable.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL