Help us shape the future of Go! Take the 2026 Go Developer Survey

archive

package
v0.40.22 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 2, 2026 License: MIT Imports: 31 Imported by: 0

Documentation

Overview

Package archive holds the telemetry archive's building blocks (N19, telemetry-archive-design.md): per-user data keys and the encrypted stream every archive object is written in.

Stream format (one object):

"RVA1" | 7-byte random nonce prefix | chunk_0 | chunk_1 | ... | chunk_n

Each chunk is AES-256-GCM over up to ChunkSize plaintext bytes with nonce prefix | uint32 big-endian chunk index | 1 byte (1 on the final chunk, else 0), and the header plus the caller's associated data (the object's user, vehicle and month) as AAD. Every chunk but the last carries exactly ChunkSize bytes; the last carries 0..ChunkSize. The final-chunk flag in the nonce makes a truncated object fail to open instead of reading as shorter data, the index makes reordered chunks fail, and the AAD makes an object copied to another user's path fail. This is the STREAM construction (as in age and Tink's streaming AEAD).

Index

Constants

View Source
const ArchiveAfterDays = 90

ArchiveAfterDays is how old a month must be, all of it, before it is archived (and so before it can be thinned). The privacy policy quotes it.

View Source
const ChunkSize = 64 << 10

ChunkSize is the plaintext per chunk.

View Source
const DeleteWithin = 24 * time.Hour

DeleteWithin is the promise for queued archive deletions: the objects are gone within it (they are unreadable at once, the key going first). Pending rows older than it are reported overdue (an alert).

View Source
const KeySize = 32

KeySize is a data key's length (AES-256).

View Source
const RethinAfter = 7 * 24 * time.Hour

RethinAfter is how long a restored month keeps its full detail before thinning reduces it again: time for whatever prompted the restore.

View Source
const Schema = "rivolt-archive/v1"

Schema names the object format; the manifest carries it.

View Source
const TimeLayout = "2006-01-02T15:04:05.000000Z"

TimeLayout is how timestamps appear in the CSV and the manifest (UTC, microseconds, as Postgres stores them).

Variables

View Source
var ErrBusy = errors.New("archive: a maintenance job is running, try again shortly")

ErrBusy is a restore refused because the archive, thinning or retention job is running (they share the maintenance lock); retry shortly.

View Source
var ErrCorrupt = errors.New("archive: object corrupt, truncated or not for this key")

ErrCorrupt is an object that fails authentication or is malformed.

View Source
var ErrInterrupted = errors.New("archive: read interrupted")

ErrInterrupted is a read that failed partway while the object itself reads back whole: a network fault, not a corrupt object.

View Source
var ErrMismatch = errors.New("archive: object doesn't match what was written")

ErrMismatch is an object that decrypts but doesn't hold what was written.

View Source
var ErrNoKEK = errors.New("archive: needs RIVOLT_KEK (data keys can't be stored unsealed)")

ErrNoKEK refuses archiving without a real KEK: the no-op sealer would store data keys in the clear.

View Source
var ErrNoObject = errors.New("archive: no verified object for that vehicle and month")

ErrNoObject is a restore of a (user, vehicle, month) without a verified archive object.

Functions

func Every added in v0.37.7

func Every(ctx context.Context, interval time.Duration, name string, log *slog.Logger, fn func(context.Context) (int64, error))

Every runs fn shortly after start (a random delay under firstRunWithin, or under interval if shorter, so replicas don't start together) and then every interval until ctx is done, logging failures.

func FlagMismatched added in v0.37.8

func FlagMismatched(ctx context.Context, ex Execer, partition string) (int64, error)

FlagMismatched flags for re-archiving every verified, unflagged (user, vehicle) whose rows in partition no longer match its object (count, first or last at): rows that reached the month during its first archive, from a writer the archive's snapshot couldn't see and couldn't flag (there was no archive row yet). Thinning calls it when its exact gate refuses, so the next archive run heals the month instead of it waiting forever (DA on #628). It returns how many it flagged.

func ManifestKey

func ManifestKey(env string, uid, vehicleID uuid.UUID, month time.Time) string

ManifestKey is the object's manifest.

func MarkRearchive added in v0.37.7

func MarkRearchive(ctx context.Context, ex Execer, uid, vehicleID uuid.UUID, month time.Time) error

MarkRearchive flags (uid, vehicleID, month) for re-archiving: rows were added to a month that may already be archived. Writers call it in the transaction that inserted the rows (samples.Store does, for every insert path). It is a no-op for a month without an archive row, which the next archive run picks up anyway. The object stays readable meanwhile: readers take it plus the database rows it lacks.

func MonthArchived

func MonthArchived(ctx context.Context, q Querier, partition string, now time.Time) (bool, error)

MonthArchived is the gate sample thinning and retention pass before they reduce or drop a month: every (user, vehicle) with rows in partition (a monthly vehicle_state partition, attached or not) must have a verified archive object for that month which isn't waiting to be re-archived and isn't held back after an admin restore (rethin_after). A partition with no rows passes.

func MonthArchivedExact

func MonthArchivedExact(ctx context.Context, q Querier, partition string, now time.Time) (bool, error)

MonthArchivedExact is MonthArchived plus a match of every (user, vehicle) in partition against its object: row count, first and last at. Thinning, which happens once per month, checks it under its partition lock, so rows that reached an archived month without setting needs_rearchive (an importer that predates it) refuse the thinning instead of being thinned away.

func MonthString

func MonthString(m time.Time) string

MonthString is the month's name in object keys and in ObjectAAD: "2026-05".

func NewDataKey

func NewDataKey() ([]byte, error)

NewDataKey returns a fresh random data key.

func NewDecryptReader

func NewDecryptReader(r io.Reader, key, aad []byte) (io.Reader, error)

NewDecryptReader returns a reader of the plaintext of an object written by NewEncryptWriter with the same key and aad. Any tampering, truncation, reordering or wrong key/aad is an ErrCorrupt read error; the plaintext read before it must be discarded by the caller.

func NewEncryptWriter

func NewEncryptWriter(w io.Writer, key, aad []byte) (io.WriteCloser, error)

NewEncryptWriter returns a writer that encrypts everything written to it into w. Close must be called to write the final chunk; it does not close w. aad binds the object to its place (user, vehicle, month).

func ObjectAAD

func ObjectAAD(uid, vehicleID uuid.UUID, month string) []byte

ObjectAAD is the associated data binding an object to its place: the archive format version, the user, the vehicle and the month. An object moved to another user's or vehicle's path fails to open.

func ObjectKey

func ObjectKey(env string, uid, vehicleID uuid.UUID, month time.Time) string

ObjectKey is the object for one user, vehicle and month.

func OpenDataKey

func OpenDataKey(ctx context.Context, s crypto.Sealer, uid uuid.UUID, sealed []byte) ([]byte, error)

OpenDataKey opens a sealed data key for uid.

func PairArchivedExact added in v0.37.8

func PairArchivedExact(ctx context.Context, q Querier, partition string, uid, vid uuid.UUID) (ok, mismatch bool, err error)

PairArchivedExact is MonthArchivedExact for one (user, vehicle): its object is verified, not flagged, and holds exactly the rows the partition has for it. mismatch reports the case FlagMismatched exists for (verified and unflagged, but the rows differ).

func SealDataKey

func SealDataKey(ctx context.Context, s crypto.Sealer, uid uuid.UUID, key []byte) ([]byte, error)

SealDataKey seals key for uid with the KEK.

func UserPrefix

func UserPrefix(env string, uid uuid.UUID) string

UserPrefix is everything archived for uid in env; account deletion lists and deletes under it.

Types

type Archiver

type Archiver struct {

	// Now is the clock (tests set it).
	Now func() time.Time
	// MaxObjectBytes caps one object (0: defaultMaxObjectBytes).
	MaxObjectBytes int64
	// Report, when set, is told about every object and every run.
	Report func(Report)
	// contains filtered or unexported fields
}

Archiver writes and verifies monthly archive objects.

func NewArchiver

func NewArchiver(db *sql.DB, cfg Config, sealer crypto.Sealer) (*Archiver, error)

NewArchiver refuses to run without a real KEK.

func (*Archiver) Run

func (a *Archiver) Run(ctx context.Context) (int64, error)

Run archives the oldest month that is wholly older than ArchiveAfterDays and has a (user, vehicle) without a verified object, and returns how many objects it wrote and verified. One month per run; one replica at a time, and never alongside thinning or retention. A failed object is reported and the rest of the month continues; the run then returns an error, and the month is retried the next day. A month where nothing succeeded (only stuck objects left) is passed over so later months still get archived. A store, network or database error stops the run at once.

A (user, vehicle) whose object needs re-archiving (rows added to an archived month: restore, import, replay) is rewritten: from the database if its month wasn't thinned (the database still has every row), otherwise merged with the existing object, whose rows win where both have one.

type Config

type Config struct {
	// Env prefixes every key ("prod", "preview"); each environment's MinIO
	// user can only reach its own prefix.
	Env       string
	Endpoint  string
	Bucket    string
	Region    string
	AccessKey string
	SecretKey string
	// HTTP overrides the S3 client's HTTP client (tests); nil is the default.
	HTTP *http.Client
}

Config is the archive's store and environment.

func ConfigFromEnv

func ConfigFromEnv(getenv func(string) string) (Config, bool, error)

ConfigFromEnv reads RIVOLT_ARCHIVE_*. The archive is on only when RIVOLT_ARCHIVE_ENV is set; then both S3 keys are required.

type DeleteReport added in v0.37.7

type DeleteReport struct {
	Requests int // requests finished this run
	Objects  int // objects deleted
	Failed   int // requests that failed (retried next run)
	Overdue  int // pending requests older than DeleteWithin
}

DeleteReport is one deletion run.

type Deleter added in v0.37.7

type Deleter struct {

	// ClockMargin is added to a request's time when picking the objects
	// written before it: MinIO's clock and Postgres's can differ a little,
	// and the margin errs toward deleting. Nothing new can be archived for
	// the user that soon (the reset removed their samples, and the archive
	// takes months 90 days old). Tests set 0.
	ClockMargin time.Duration
	// Report, when set, is told the run's totals.
	Report func(DeleteReport)
	// contains filtered or unexported fields
}

Deleter removes the objects queued in archive_deletions (account deletion, history reset).

func NewDeleter added in v0.37.7

func NewDeleter(db *sql.DB, cfg Config) (*Deleter, error)

NewDeleter needs no KEK: it deletes ciphertext it never reads.

func (*Deleter) Run added in v0.37.7

func (d *Deleter) Run(ctx context.Context) (int64, error)

Run deletes the objects of pending requests, oldest first, and returns how many requests it finished. Each request deletes only objects written before it was made, so an archive written after a history reset (under the user's new key) stays. Replicas share the work (FOR UPDATE SKIP LOCKED). Finished rows older than 30 days are purged.

type Execer added in v0.37.7

type Execer interface {
	ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error)
}

Execer is a *sql.Tx, *sql.Conn or *sql.DB.

type Manifest

type Manifest struct {
	Schema  string   `json:"schema"`
	Columns []string `json:"columns"`
	Rows    int64    `json:"rows"`
	// FirstAt and LastAt are checked at verify and recorded in archive_months,
	// but not written to the plaintext manifest: after a crypto-shred the
	// manifest lingers until the deletion job runs, and should say nothing
	// about when the user drove (DA).
	FirstAt      string    `json:"-"`
	LastAt       string    `json:"-"`
	SHA256Plain  string    `json:"sha256_plain"`
	SHA256Object string    `json:"sha256_object"`
	Bytes        int64     `json:"bytes"`
	CreatedAt    time.Time `json:"created_at"`
}

Manifest describes one object.

type ObjectRef added in v0.37.7

type ObjectRef struct {
	VehicleID uuid.UUID
	Month     time.Time
	Rows      int64
}

ObjectRef is one readable object: verified. One waiting to be re-archived (needs_rearchive) still holds every row it was written with; the rows that came since are in the database, and readers take the union.

type Querier

type Querier interface {
	QueryRowContext(ctx context.Context, query string, args ...any) *sql.Row
}

Querier is a *sql.DB, *sql.Conn or *sql.Tx.

type Reader added in v0.37.7

type Reader struct {

	// OnCorrupt, when set, is told about an object that exists but fails to
	// read back: corrupt, truncated, or not this user's. For a thinned month
	// that is the full-resolution data lost, so the caller counts and alerts.
	OnCorrupt func(uid, vehicleID uuid.UUID, month time.Time, err error)
	// contains filtered or unexported fields
}

Reader reads a user's archive objects back (the backup download, the admin restore). It never creates keys or writes.

func NewReader added in v0.37.7

func NewReader(db *sql.DB, cfg Config, sealer crypto.Sealer) (*Reader, error)

NewReader refuses to run without a real KEK, like the archiver.

func (*Reader) EachRow added in v0.37.7

func (r *Reader) EachRow(ctx context.Context, uid, vehicleID uuid.UUID, month time.Time, fn func(Row) error) error

EachRow streams the object's rows to fn in `at` order and reads the object to the end. An error returned after rows were passed (a corrupt or truncated object is detected at its end) means what fn got is incomplete: the caller must fail as a whole. A missing object is s3.ErrNotFound before any row.

func (*Reader) Objects added in v0.37.7

func (r *Reader) Objects(ctx context.Context, uid uuid.UUID) ([]ObjectRef, error)

Objects lists uid's readable objects, by month then vehicle.

type Report

type Report struct {
	Month     string
	UserID    uuid.UUID
	VehicleID uuid.UUID
	Rows      int64
	Bytes     int64
	Err       error
	// Done marks the run's summary: Objects archived, Failed objects.
	Done    bool
	Objects int
	Failed  int
}

Report is one object archived (or failed), or with Done set the run's total.

type RestoreResult added in v0.37.8

type RestoreResult struct {
	InObject int64 `json:"in_object"`
	Present  int64 `json:"already_present"`
	Inserted int64 `json:"inserted"`
	// DBOnly is rows the database has that the object lacks; they stay, and
	// the month stays flagged for re-archiving.
	DBOnly      int64      `json:"db_only"`
	RethinAfter *time.Time `json:"rethin_after,omitempty"`
	DryRun      bool       `json:"dry_run"`
}

RestoreResult is what a restore found and did (or would do, on a dry run).

type Restorer added in v0.37.8

type Restorer struct {
	// contains filtered or unexported fields
}

Restorer puts one archived (user, vehicle, month) back into vehicle_state in full detail (the admin site's restore and `rivolt archive restore`).

func NewRestorer added in v0.37.8

func NewRestorer(db *sql.DB, cfg Config, sealer crypto.Sealer) (*Restorer, error)

NewRestorer refuses to run without a real KEK.

func (*Restorer) Restore added in v0.37.8

func (r *Restorer) Restore(ctx context.Context, uid, vid uuid.UUID, month time.Time, dryRun, wait bool) (RestoreResult, error)

Restore reads the object back whole and inserts its rows into the month's partition, keeping any row already there. Afterwards a thinned month is held at full detail for RethinAfter (rethin_after), after which thinning reduces just this vehicle again. The month stays flagged for re-archiving if the database has rows the object lacks (DA on A4). It holds the maintenance lock (never alongside archiving, thinning or retention: with wait false a busy lock is ErrBusy) and the user's archive lock.

type Row added in v0.37.7

type Row struct {
	// contains filtered or unexported fields
}

Row is one archived vehicle_state row, read by column name. Values are the CSV's text: "" for NULL, timestamps in TimeLayout.

func NewRow added in v0.37.7

func NewRow(header, rec []string) Row

NewRow builds a Row from a header and a record (tests, and callers that read CSV themselves).

func (Row) Get added in v0.37.7

func (r Row) Get(name string) (string, bool)

Get returns the named column's value and whether the object has it.

type Stats

type Stats struct {
	Rows        int64
	FirstAt     string
	LastAt      string
	SHA256Plain string
}

Stats is what reading an object back finds.

func ReadStats

func ReadStats(r io.Reader, key, aad []byte, columns []string) (Stats, error)

ReadStats reads a whole object, decrypting and decompressing it, and counts its CSV rows. The header must be columns, and at must be one of them. It reads to the end of the encrypted stream: a corrupt or truncated object is an error even after rows were read (the STREAM construction authenticates the end last), and no Stats are returned with an error.

Directories

Path Synopsis
Package s3 is a minimal S3 client for the telemetry archive: put, get, delete and list objects in one bucket, path-style, signed with AWS Signature Version 4 over the real payload hash (never UNSIGNED-PAYLOAD).
Package s3 is a minimal S3 client for the telemetry archive: put, get, delete and list objects in one bucket, path-style, signed with AWS Signature Version 4 over the real payload hash (never UNSIGNED-PAYLOAD).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL