Documentation
¶
Overview ¶
Package archive holds the telemetry archive's building blocks (N19, telemetry-archive-design.md): per-user data keys and the encrypted stream every archive object is written in.
Stream format (one object):
"RVA1" | 7-byte random nonce prefix | chunk_0 | chunk_1 | ... | chunk_n
Each chunk is AES-256-GCM over up to ChunkSize plaintext bytes with nonce prefix | uint32 big-endian chunk index | 1 byte (1 on the final chunk, else 0), and the header plus the caller's associated data (the object's user, vehicle and month) as AAD. Every chunk but the last carries exactly ChunkSize bytes; the last carries 0..ChunkSize. The final-chunk flag in the nonce makes a truncated object fail to open instead of reading as shorter data, the index makes reordered chunks fail, and the AAD makes an object copied to another user's path fail. This is the STREAM construction (as in age and Tink's streaming AEAD).
Index ¶
- Constants
- Variables
- func Every(ctx context.Context, interval time.Duration, name string, log *slog.Logger, ...)
- func FlagMismatched(ctx context.Context, ex Execer, partition string) (int64, error)
- func ManifestKey(env string, uid, vehicleID uuid.UUID, month time.Time) string
- func MarkRearchive(ctx context.Context, ex Execer, uid, vehicleID uuid.UUID, month time.Time) error
- func MonthArchived(ctx context.Context, q Querier, partition string, now time.Time) (bool, error)
- func MonthArchivedExact(ctx context.Context, q Querier, partition string, now time.Time) (bool, error)
- func MonthString(m time.Time) string
- func NewDataKey() ([]byte, error)
- func NewDecryptReader(r io.Reader, key, aad []byte) (io.Reader, error)
- func NewEncryptWriter(w io.Writer, key, aad []byte) (io.WriteCloser, error)
- func ObjectAAD(uid, vehicleID uuid.UUID, month string) []byte
- func ObjectKey(env string, uid, vehicleID uuid.UUID, month time.Time) string
- func OpenDataKey(ctx context.Context, s crypto.Sealer, uid uuid.UUID, sealed []byte) ([]byte, error)
- func PairArchivedExact(ctx context.Context, q Querier, partition string, uid, vid uuid.UUID) (ok, mismatch bool, err error)
- func SealDataKey(ctx context.Context, s crypto.Sealer, uid uuid.UUID, key []byte) ([]byte, error)
- func UserPrefix(env string, uid uuid.UUID) string
- type Archiver
- type Config
- type DeleteReport
- type Deleter
- type Execer
- type Manifest
- type ObjectRef
- type Querier
- type Reader
- type Report
- type RestoreResult
- type Restorer
- type Row
- type Stats
Constants ¶
const ArchiveAfterDays = 90
ArchiveAfterDays is how old a month must be, all of it, before it is archived (and so before it can be thinned). The privacy policy quotes it.
const ChunkSize = 64 << 10
ChunkSize is the plaintext per chunk.
const DeleteWithin = 24 * time.Hour
DeleteWithin is the promise for queued archive deletions: the objects are gone within it (they are unreadable at once, the key going first). Pending rows older than it are reported overdue (an alert).
const KeySize = 32
KeySize is a data key's length (AES-256).
const RethinAfter = 7 * 24 * time.Hour
RethinAfter is how long a restored month keeps its full detail before thinning reduces it again: time for whatever prompted the restore.
const Schema = "rivolt-archive/v1"
Schema names the object format; the manifest carries it.
const TimeLayout = "2006-01-02T15:04:05.000000Z"
TimeLayout is how timestamps appear in the CSV and the manifest (UTC, microseconds, as Postgres stores them).
Variables ¶
var ErrBusy = errors.New("archive: a maintenance job is running, try again shortly")
ErrBusy is a restore refused because the archive, thinning or retention job is running (they share the maintenance lock); retry shortly.
var ErrCorrupt = errors.New("archive: object corrupt, truncated or not for this key")
ErrCorrupt is an object that fails authentication or is malformed.
var ErrInterrupted = errors.New("archive: read interrupted")
ErrInterrupted is a read that failed partway while the object itself reads back whole: a network fault, not a corrupt object.
var ErrMismatch = errors.New("archive: object doesn't match what was written")
ErrMismatch is an object that decrypts but doesn't hold what was written.
var ErrNoKEK = errors.New("archive: needs RIVOLT_KEK (data keys can't be stored unsealed)")
ErrNoKEK refuses archiving without a real KEK: the no-op sealer would store data keys in the clear.
var ErrNoObject = errors.New("archive: no verified object for that vehicle and month")
ErrNoObject is a restore of a (user, vehicle, month) without a verified archive object.
Functions ¶
func Every ¶ added in v0.37.7
func Every(ctx context.Context, interval time.Duration, name string, log *slog.Logger, fn func(context.Context) (int64, error))
Every runs fn shortly after start (a random delay under firstRunWithin, or under interval if shorter, so replicas don't start together) and then every interval until ctx is done, logging failures.
func FlagMismatched ¶ added in v0.37.8
FlagMismatched flags for re-archiving every verified, unflagged (user, vehicle) whose rows in partition no longer match its object (count, first or last at): rows that reached the month during its first archive, from a writer the archive's snapshot couldn't see and couldn't flag (there was no archive row yet). Thinning calls it when its exact gate refuses, so the next archive run heals the month instead of it waiting forever (DA on #628). It returns how many it flagged.
func ManifestKey ¶
ManifestKey is the object's manifest.
func MarkRearchive ¶ added in v0.37.7
MarkRearchive flags (uid, vehicleID, month) for re-archiving: rows were added to a month that may already be archived. Writers call it in the transaction that inserted the rows (samples.Store does, for every insert path). It is a no-op for a month without an archive row, which the next archive run picks up anyway. The object stays readable meanwhile: readers take it plus the database rows it lacks.
func MonthArchived ¶
MonthArchived is the gate sample thinning and retention pass before they reduce or drop a month: every (user, vehicle) with rows in partition (a monthly vehicle_state partition, attached or not) must have a verified archive object for that month which isn't waiting to be re-archived and isn't held back after an admin restore (rethin_after). A partition with no rows passes.
func MonthArchivedExact ¶
func MonthArchivedExact(ctx context.Context, q Querier, partition string, now time.Time) (bool, error)
MonthArchivedExact is MonthArchived plus a match of every (user, vehicle) in partition against its object: row count, first and last at. Thinning, which happens once per month, checks it under its partition lock, so rows that reached an archived month without setting needs_rearchive (an importer that predates it) refuse the thinning instead of being thinned away.
func MonthString ¶
MonthString is the month's name in object keys and in ObjectAAD: "2026-05".
func NewDecryptReader ¶
NewDecryptReader returns a reader of the plaintext of an object written by NewEncryptWriter with the same key and aad. Any tampering, truncation, reordering or wrong key/aad is an ErrCorrupt read error; the plaintext read before it must be discarded by the caller.
func NewEncryptWriter ¶
NewEncryptWriter returns a writer that encrypts everything written to it into w. Close must be called to write the final chunk; it does not close w. aad binds the object to its place (user, vehicle, month).
func ObjectAAD ¶
ObjectAAD is the associated data binding an object to its place: the archive format version, the user, the vehicle and the month. An object moved to another user's or vehicle's path fails to open.
func OpenDataKey ¶
func OpenDataKey(ctx context.Context, s crypto.Sealer, uid uuid.UUID, sealed []byte) ([]byte, error)
OpenDataKey opens a sealed data key for uid.
func PairArchivedExact ¶ added in v0.37.8
func PairArchivedExact(ctx context.Context, q Querier, partition string, uid, vid uuid.UUID) (ok, mismatch bool, err error)
PairArchivedExact is MonthArchivedExact for one (user, vehicle): its object is verified, not flagged, and holds exactly the rows the partition has for it. mismatch reports the case FlagMismatched exists for (verified and unflagged, but the rows differ).
func SealDataKey ¶
SealDataKey seals key for uid with the KEK.
Types ¶
type Archiver ¶
type Archiver struct {
// Now is the clock (tests set it).
Now func() time.Time
// MaxObjectBytes caps one object (0: defaultMaxObjectBytes).
MaxObjectBytes int64
// Report, when set, is told about every object and every run.
Report func(Report)
// contains filtered or unexported fields
}
Archiver writes and verifies monthly archive objects.
func NewArchiver ¶
NewArchiver refuses to run without a real KEK.
func (*Archiver) Run ¶
Run archives the oldest month that is wholly older than ArchiveAfterDays and has a (user, vehicle) without a verified object, and returns how many objects it wrote and verified. One month per run; one replica at a time, and never alongside thinning or retention. A failed object is reported and the rest of the month continues; the run then returns an error, and the month is retried the next day. A month where nothing succeeded (only stuck objects left) is passed over so later months still get archived. A store, network or database error stops the run at once.
A (user, vehicle) whose object needs re-archiving (rows added to an archived month: restore, import, replay) is rewritten: from the database if its month wasn't thinned (the database still has every row), otherwise merged with the existing object, whose rows win where both have one.
type Config ¶
type Config struct {
// Env prefixes every key ("prod", "preview"); each environment's MinIO
// user can only reach its own prefix.
Env string
Endpoint string
Bucket string
Region string
AccessKey string
SecretKey string
// HTTP overrides the S3 client's HTTP client (tests); nil is the default.
HTTP *http.Client
}
Config is the archive's store and environment.
type DeleteReport ¶ added in v0.37.7
type DeleteReport struct {
Requests int // requests finished this run
Objects int // objects deleted
Failed int // requests that failed (retried next run)
Overdue int // pending requests older than DeleteWithin
}
DeleteReport is one deletion run.
type Deleter ¶ added in v0.37.7
type Deleter struct {
// ClockMargin is added to a request's time when picking the objects
// written before it: MinIO's clock and Postgres's can differ a little,
// and the margin errs toward deleting. Nothing new can be archived for
// the user that soon (the reset removed their samples, and the archive
// takes months 90 days old). Tests set 0.
ClockMargin time.Duration
// Report, when set, is told the run's totals.
Report func(DeleteReport)
// contains filtered or unexported fields
}
Deleter removes the objects queued in archive_deletions (account deletion, history reset).
func NewDeleter ¶ added in v0.37.7
NewDeleter needs no KEK: it deletes ciphertext it never reads.
func (*Deleter) Run ¶ added in v0.37.7
Run deletes the objects of pending requests, oldest first, and returns how many requests it finished. Each request deletes only objects written before it was made, so an archive written after a history reset (under the user's new key) stays. Replicas share the work (FOR UPDATE SKIP LOCKED). Finished rows older than 30 days are purged.
type Execer ¶ added in v0.37.7
type Execer interface {
ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error)
}
Execer is a *sql.Tx, *sql.Conn or *sql.DB.
type Manifest ¶
type Manifest struct {
Schema string `json:"schema"`
Columns []string `json:"columns"`
Rows int64 `json:"rows"`
// FirstAt and LastAt are checked at verify and recorded in archive_months,
// but not written to the plaintext manifest: after a crypto-shred the
// manifest lingers until the deletion job runs, and should say nothing
// about when the user drove (DA).
FirstAt string `json:"-"`
LastAt string `json:"-"`
SHA256Plain string `json:"sha256_plain"`
SHA256Object string `json:"sha256_object"`
Bytes int64 `json:"bytes"`
CreatedAt time.Time `json:"created_at"`
}
Manifest describes one object.
type ObjectRef ¶ added in v0.37.7
ObjectRef is one readable object: verified. One waiting to be re-archived (needs_rearchive) still holds every row it was written with; the rows that came since are in the database, and readers take the union.
type Reader ¶ added in v0.37.7
type Reader struct {
// OnCorrupt, when set, is told about an object that exists but fails to
// read back: corrupt, truncated, or not this user's. For a thinned month
// that is the full-resolution data lost, so the caller counts and alerts.
OnCorrupt func(uid, vehicleID uuid.UUID, month time.Time, err error)
// contains filtered or unexported fields
}
Reader reads a user's archive objects back (the backup download, the admin restore). It never creates keys or writes.
func (*Reader) EachRow ¶ added in v0.37.7
func (r *Reader) EachRow(ctx context.Context, uid, vehicleID uuid.UUID, month time.Time, fn func(Row) error) error
EachRow streams the object's rows to fn in `at` order and reads the object to the end. An error returned after rows were passed (a corrupt or truncated object is detected at its end) means what fn got is incomplete: the caller must fail as a whole. A missing object is s3.ErrNotFound before any row.
type Report ¶
type Report struct {
Month string
UserID uuid.UUID
VehicleID uuid.UUID
Rows int64
Bytes int64
Err error
// Done marks the run's summary: Objects archived, Failed objects.
Done bool
Objects int
Failed int
}
Report is one object archived (or failed), or with Done set the run's total.
type RestoreResult ¶ added in v0.37.8
type RestoreResult struct {
InObject int64 `json:"in_object"`
Present int64 `json:"already_present"`
Inserted int64 `json:"inserted"`
// DBOnly is rows the database has that the object lacks; they stay, and
// the month stays flagged for re-archiving.
DBOnly int64 `json:"db_only"`
RethinAfter *time.Time `json:"rethin_after,omitempty"`
DryRun bool `json:"dry_run"`
}
RestoreResult is what a restore found and did (or would do, on a dry run).
type Restorer ¶ added in v0.37.8
type Restorer struct {
// contains filtered or unexported fields
}
Restorer puts one archived (user, vehicle, month) back into vehicle_state in full detail (the admin site's restore and `rivolt archive restore`).
func NewRestorer ¶ added in v0.37.8
NewRestorer refuses to run without a real KEK.
func (*Restorer) Restore ¶ added in v0.37.8
func (r *Restorer) Restore(ctx context.Context, uid, vid uuid.UUID, month time.Time, dryRun, wait bool) (RestoreResult, error)
Restore reads the object back whole and inserts its rows into the month's partition, keeping any row already there. Afterwards a thinned month is held at full detail for RethinAfter (rethin_after), after which thinning reduces just this vehicle again. The month stays flagged for re-archiving if the database has rows the object lacks (DA on A4). It holds the maintenance lock (never alongside archiving, thinning or retention: with wait false a busy lock is ErrBusy) and the user's archive lock.
type Row ¶ added in v0.37.7
type Row struct {
// contains filtered or unexported fields
}
Row is one archived vehicle_state row, read by column name. Values are the CSV's text: "" for NULL, timestamps in TimeLayout.
type Stats ¶
Stats is what reading an object back finds.
func ReadStats ¶
ReadStats reads a whole object, decrypting and decompressing it, and counts its CSV rows. The header must be columns, and at must be one of them. It reads to the end of the encrypted stream: a corrupt or truncated object is an error even after rows were read (the STREAM construction authenticates the end last), and no Stats are returned with an error.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package s3 is a minimal S3 client for the telemetry archive: put, get, delete and list objects in one bucket, path-style, signed with AWS Signature Version 4 over the real payload hash (never UNSIGNED-PAYLOAD).
|
Package s3 is a minimal S3 client for the telemetry archive: put, get, delete and list objects in one bucket, path-style, signed with AWS Signature Version 4 over the real payload hash (never UNSIGNED-PAYLOAD). |