archive

package
v0.37.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 19, 2026 License: MIT Imports: 26 Imported by: 0

Documentation

Overview

Package archive holds the telemetry archive's building blocks (N19, telemetry-archive-design.md): per-user data keys and the encrypted stream every archive object is written in.

Stream format (one object):

"RVA1" | 7-byte random nonce prefix | chunk_0 | chunk_1 | ... | chunk_n

Each chunk is AES-256-GCM over up to ChunkSize plaintext bytes with nonce prefix | uint32 big-endian chunk index | 1 byte (1 on the final chunk, else 0), and the header plus the caller's associated data (the object's user, vehicle and month) as AAD. Every chunk but the last carries exactly ChunkSize bytes; the last carries 0..ChunkSize. The final-chunk flag in the nonce makes a truncated object fail to open instead of reading as shorter data, the index makes reordered chunks fail, and the AAD makes an object copied to another user's path fail. This is the STREAM construction (as in age and Tink's streaming AEAD).

Index

Constants

View Source
const ArchiveAfterDays = 90

ArchiveAfterDays is how old a month must be, all of it, before it is archived (and so before it can be thinned). The privacy policy quotes it.

View Source
const ChunkSize = 64 << 10

ChunkSize is the plaintext per chunk.

View Source
const KeySize = 32

KeySize is a data key's length (AES-256).

View Source
const Schema = "rivolt-archive/v1"

Schema names the object format; the manifest carries it.

View Source
const TimeLayout = "2006-01-02T15:04:05.000000Z"

TimeLayout is how timestamps appear in the CSV and the manifest (UTC, microseconds, as Postgres stores them).

Variables

View Source
var ErrCorrupt = errors.New("archive: object corrupt, truncated or not for this key")

ErrCorrupt is an object that fails authentication or is malformed.

View Source
var ErrMismatch = errors.New("archive: object doesn't match what was written")

ErrMismatch is an object that decrypts but doesn't hold what was written.

View Source
var ErrNoKEK = errors.New("archive: needs RIVOLT_KEK (data keys can't be stored unsealed)")

ErrNoKEK refuses archiving without a real KEK: the no-op sealer would store data keys in the clear.

Functions

func ManifestKey

func ManifestKey(env string, uid, vehicleID uuid.UUID, month time.Time) string

ManifestKey is the object's manifest.

func MonthArchived

func MonthArchived(ctx context.Context, q Querier, partition string, now time.Time) (bool, error)

MonthArchived is the gate sample thinning and retention pass before they reduce or drop a month: every (user, vehicle) with rows in partition (a monthly vehicle_state partition, attached or not) must have a verified archive object for that month which isn't waiting to be re-archived and isn't held back after an admin restore (rethin_after). A partition with no rows passes.

func MonthArchivedExact

func MonthArchivedExact(ctx context.Context, q Querier, partition string, now time.Time) (bool, error)

MonthArchivedExact is MonthArchived plus a match of every (user, vehicle) in partition against its object: row count, first and last at. Thinning, which happens once per month, checks it under its partition lock, so rows that reached an archived month without setting needs_rearchive (an importer that predates it) refuse the thinning instead of being thinned away.

func MonthString

func MonthString(m time.Time) string

MonthString is the month's name in object keys and in ObjectAAD: "2026-05".

func NewDataKey

func NewDataKey() ([]byte, error)

NewDataKey returns a fresh random data key.

func NewDecryptReader

func NewDecryptReader(r io.Reader, key, aad []byte) (io.Reader, error)

NewDecryptReader returns a reader of the plaintext of an object written by NewEncryptWriter with the same key and aad. Any tampering, truncation, reordering or wrong key/aad is an ErrCorrupt read error; the plaintext read before it must be discarded by the caller.

func NewEncryptWriter

func NewEncryptWriter(w io.Writer, key, aad []byte) (io.WriteCloser, error)

NewEncryptWriter returns a writer that encrypts everything written to it into w. Close must be called to write the final chunk; it does not close w. aad binds the object to its place (user, vehicle, month).

func ObjectAAD

func ObjectAAD(uid, vehicleID uuid.UUID, month string) []byte

ObjectAAD is the associated data binding an object to its place: the archive format version, the user, the vehicle and the month. An object moved to another user's or vehicle's path fails to open.

func ObjectKey

func ObjectKey(env string, uid, vehicleID uuid.UUID, month time.Time) string

ObjectKey is the object for one user, vehicle and month.

func OpenDataKey

func OpenDataKey(ctx context.Context, s crypto.Sealer, uid uuid.UUID, sealed []byte) ([]byte, error)

OpenDataKey opens a sealed data key for uid.

func SealDataKey

func SealDataKey(ctx context.Context, s crypto.Sealer, uid uuid.UUID, key []byte) ([]byte, error)

SealDataKey seals key for uid with the KEK.

func UserPrefix

func UserPrefix(env string, uid uuid.UUID) string

UserPrefix is everything archived for uid in env; account deletion lists and deletes under it.

Types

type Archiver

type Archiver struct {

	// Now is the clock (tests set it).
	Now func() time.Time
	// MaxObjectBytes caps one object (0: defaultMaxObjectBytes).
	MaxObjectBytes int64
	// Report, when set, is told about every object and every run.
	Report func(Report)
	// contains filtered or unexported fields
}

Archiver writes and verifies monthly archive objects.

func NewArchiver

func NewArchiver(db *sql.DB, cfg Config, sealer crypto.Sealer) (*Archiver, error)

NewArchiver refuses to run without a real KEK.

func (*Archiver) Run

func (a *Archiver) Run(ctx context.Context) (int64, error)

Run archives the oldest month that is wholly older than ArchiveAfterDays and has a (user, vehicle) without a verified object, and returns how many objects it wrote and verified. One month per run; one replica at a time, and never alongside thinning or retention. A failed object is reported and the rest of the month continues; the run then returns an error, and the month is retried the next day. A month where nothing succeeded (only stuck objects left) is passed over so later months still get archived.

A (user, vehicle) whose object needs re-archiving (rows restored or imported into an archived month) is not touched here: its month may be thinned already, and only a merge with the existing object (A4) keeps the full detail.

type Config

type Config struct {
	// Env prefixes every key ("prod", "preview"); each environment's MinIO
	// user can only reach its own prefix.
	Env       string
	Endpoint  string
	Bucket    string
	Region    string
	AccessKey string
	SecretKey string
}

Config is the archive's store and environment.

func ConfigFromEnv

func ConfigFromEnv(getenv func(string) string) (Config, bool, error)

ConfigFromEnv reads RIVOLT_ARCHIVE_*. The archive is on only when RIVOLT_ARCHIVE_ENV is set; then both S3 keys are required.

type Manifest

type Manifest struct {
	Schema  string   `json:"schema"`
	Columns []string `json:"columns"`
	Rows    int64    `json:"rows"`
	// FirstAt and LastAt are checked at verify and recorded in archive_months,
	// but not written to the plaintext manifest: after a crypto-shred the
	// manifest lingers until the deletion job runs, and should say nothing
	// about when the user drove (DA).
	FirstAt      string    `json:"-"`
	LastAt       string    `json:"-"`
	SHA256Plain  string    `json:"sha256_plain"`
	SHA256Object string    `json:"sha256_object"`
	Bytes        int64     `json:"bytes"`
	CreatedAt    time.Time `json:"created_at"`
}

Manifest describes one object.

type Querier

type Querier interface {
	QueryRowContext(ctx context.Context, query string, args ...any) *sql.Row
}

Querier is a *sql.DB, *sql.Conn or *sql.Tx.

type Report

type Report struct {
	Month     string
	UserID    uuid.UUID
	VehicleID uuid.UUID
	Rows      int64
	Bytes     int64
	Err       error
	// Done marks the run's summary: Objects archived, Failed objects.
	Done    bool
	Objects int
	Failed  int
}

Report is one object archived (or failed), or with Done set the run's total.

type Stats

type Stats struct {
	Rows        int64
	FirstAt     string
	LastAt      string
	SHA256Plain string
}

Stats is what reading an object back finds.

func ReadStats

func ReadStats(r io.Reader, key, aad []byte, columns []string) (Stats, error)

ReadStats reads a whole object, decrypting and decompressing it, and counts its CSV rows. The header must be columns, and at must be one of them. It reads to the end of the encrypted stream: a corrupt or truncated object is an error even after rows were read (the STREAM construction authenticates the end last), and no Stats are returned with an error.

Directories

Path Synopsis
Package s3 is a minimal S3 client for the telemetry archive: put, get, delete and list objects in one bucket, path-style, signed with AWS Signature Version 4 over the real payload hash (never UNSIGNED-PAYLOAD).
Package s3 is a minimal S3 client for the telemetry archive: put, get, delete and list objects in one bucket, path-style, signed with AWS Signature Version 4 over the real payload hash (never UNSIGNED-PAYLOAD).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL