Documentation
¶
Overview ¶
Package archive holds the telemetry archive's building blocks (N19, telemetry-archive-design.md): per-user data keys and the encrypted stream every archive object is written in.
Stream format (one object):
"RVA1" | 7-byte random nonce prefix | chunk_0 | chunk_1 | ... | chunk_n
Each chunk is AES-256-GCM over up to ChunkSize plaintext bytes with nonce prefix | uint32 big-endian chunk index | 1 byte (1 on the final chunk, else 0), and the header plus the caller's associated data (the object's user, vehicle and month) as AAD. Every chunk but the last carries exactly ChunkSize bytes; the last carries 0..ChunkSize. The final-chunk flag in the nonce makes a truncated object fail to open instead of reading as shorter data, the index makes reordered chunks fail, and the AAD makes an object copied to another user's path fail. This is the STREAM construction (as in age and Tink's streaming AEAD).
Index ¶
- Constants
- Variables
- func ManifestKey(env string, uid, vehicleID uuid.UUID, month time.Time) string
- func MonthArchived(ctx context.Context, q Querier, partition string, now time.Time) (bool, error)
- func MonthArchivedExact(ctx context.Context, q Querier, partition string, now time.Time) (bool, error)
- func MonthString(m time.Time) string
- func NewDataKey() ([]byte, error)
- func NewDecryptReader(r io.Reader, key, aad []byte) (io.Reader, error)
- func NewEncryptWriter(w io.Writer, key, aad []byte) (io.WriteCloser, error)
- func ObjectAAD(uid, vehicleID uuid.UUID, month string) []byte
- func ObjectKey(env string, uid, vehicleID uuid.UUID, month time.Time) string
- func OpenDataKey(ctx context.Context, s crypto.Sealer, uid uuid.UUID, sealed []byte) ([]byte, error)
- func SealDataKey(ctx context.Context, s crypto.Sealer, uid uuid.UUID, key []byte) ([]byte, error)
- func UserPrefix(env string, uid uuid.UUID) string
- type Archiver
- type Config
- type Manifest
- type Querier
- type Report
- type Stats
Constants ¶
const ArchiveAfterDays = 90
ArchiveAfterDays is how old a month must be, all of it, before it is archived (and so before it can be thinned). The privacy policy quotes it.
const ChunkSize = 64 << 10
ChunkSize is the plaintext per chunk.
const KeySize = 32
KeySize is a data key's length (AES-256).
const Schema = "rivolt-archive/v1"
Schema names the object format; the manifest carries it.
const TimeLayout = "2006-01-02T15:04:05.000000Z"
TimeLayout is how timestamps appear in the CSV and the manifest (UTC, microseconds, as Postgres stores them).
Variables ¶
var ErrCorrupt = errors.New("archive: object corrupt, truncated or not for this key")
ErrCorrupt is an object that fails authentication or is malformed.
var ErrMismatch = errors.New("archive: object doesn't match what was written")
ErrMismatch is an object that decrypts but doesn't hold what was written.
var ErrNoKEK = errors.New("archive: needs RIVOLT_KEK (data keys can't be stored unsealed)")
ErrNoKEK refuses archiving without a real KEK: the no-op sealer would store data keys in the clear.
Functions ¶
func ManifestKey ¶
ManifestKey is the object's manifest.
func MonthArchived ¶
MonthArchived is the gate sample thinning and retention pass before they reduce or drop a month: every (user, vehicle) with rows in partition (a monthly vehicle_state partition, attached or not) must have a verified archive object for that month which isn't waiting to be re-archived and isn't held back after an admin restore (rethin_after). A partition with no rows passes.
func MonthArchivedExact ¶
func MonthArchivedExact(ctx context.Context, q Querier, partition string, now time.Time) (bool, error)
MonthArchivedExact is MonthArchived plus a match of every (user, vehicle) in partition against its object: row count, first and last at. Thinning, which happens once per month, checks it under its partition lock, so rows that reached an archived month without setting needs_rearchive (an importer that predates it) refuse the thinning instead of being thinned away.
func MonthString ¶
MonthString is the month's name in object keys and in ObjectAAD: "2026-05".
func NewDecryptReader ¶
NewDecryptReader returns a reader of the plaintext of an object written by NewEncryptWriter with the same key and aad. Any tampering, truncation, reordering or wrong key/aad is an ErrCorrupt read error; the plaintext read before it must be discarded by the caller.
func NewEncryptWriter ¶
NewEncryptWriter returns a writer that encrypts everything written to it into w. Close must be called to write the final chunk; it does not close w. aad binds the object to its place (user, vehicle, month).
func ObjectAAD ¶
ObjectAAD is the associated data binding an object to its place: the archive format version, the user, the vehicle and the month. An object moved to another user's or vehicle's path fails to open.
func OpenDataKey ¶
func OpenDataKey(ctx context.Context, s crypto.Sealer, uid uuid.UUID, sealed []byte) ([]byte, error)
OpenDataKey opens a sealed data key for uid.
func SealDataKey ¶
SealDataKey seals key for uid with the KEK.
Types ¶
type Archiver ¶
type Archiver struct {
// Now is the clock (tests set it).
Now func() time.Time
// MaxObjectBytes caps one object (0: defaultMaxObjectBytes).
MaxObjectBytes int64
// Report, when set, is told about every object and every run.
Report func(Report)
// contains filtered or unexported fields
}
Archiver writes and verifies monthly archive objects.
func NewArchiver ¶
NewArchiver refuses to run without a real KEK.
func (*Archiver) Run ¶
Run archives the oldest month that is wholly older than ArchiveAfterDays and has a (user, vehicle) without a verified object, and returns how many objects it wrote and verified. One month per run; one replica at a time, and never alongside thinning or retention. A failed object is reported and the rest of the month continues; the run then returns an error, and the month is retried the next day. A month where nothing succeeded (only stuck objects left) is passed over so later months still get archived.
A (user, vehicle) whose object needs re-archiving (rows restored or imported into an archived month) is not touched here: its month may be thinned already, and only a merge with the existing object (A4) keeps the full detail.
type Config ¶
type Config struct {
// Env prefixes every key ("prod", "preview"); each environment's MinIO
// user can only reach its own prefix.
Env string
Endpoint string
Bucket string
Region string
AccessKey string
SecretKey string
}
Config is the archive's store and environment.
type Manifest ¶
type Manifest struct {
Schema string `json:"schema"`
Columns []string `json:"columns"`
Rows int64 `json:"rows"`
// FirstAt and LastAt are checked at verify and recorded in archive_months,
// but not written to the plaintext manifest: after a crypto-shred the
// manifest lingers until the deletion job runs, and should say nothing
// about when the user drove (DA).
FirstAt string `json:"-"`
LastAt string `json:"-"`
SHA256Plain string `json:"sha256_plain"`
SHA256Object string `json:"sha256_object"`
Bytes int64 `json:"bytes"`
CreatedAt time.Time `json:"created_at"`
}
Manifest describes one object.
type Report ¶
type Report struct {
Month string
UserID uuid.UUID
VehicleID uuid.UUID
Rows int64
Bytes int64
Err error
// Done marks the run's summary: Objects archived, Failed objects.
Done bool
Objects int
Failed int
}
Report is one object archived (or failed), or with Done set the run's total.
type Stats ¶
Stats is what reading an object back finds.
func ReadStats ¶
ReadStats reads a whole object, decrypting and decompressing it, and counts its CSV rows. The header must be columns, and at must be one of them. It reads to the end of the encrypted stream: a corrupt or truncated object is an error even after rows were read (the STREAM construction authenticates the end last), and no Stats are returned with an error.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package s3 is a minimal S3 client for the telemetry archive: put, get, delete and list objects in one bucket, path-style, signed with AWS Signature Version 4 over the real payload hash (never UNSIGNED-PAYLOAD).
|
Package s3 is a minimal S3 client for the telemetry archive: put, get, delete and list objects in one bucket, path-style, signed with AWS Signature Version 4 over the real payload hash (never UNSIGNED-PAYLOAD). |