Documentation
¶
Overview ¶
Package store keeps the last valid snapshot of each source on disk, shared by the sync and API processes and kept across restarts.
File format, version 1 ¶
Each source has one file, <name>.snapshot, where name is the source lowercased with every run of characters other than a-z and 0-9 turned into a single "-" ("AMF listes blanches" is amf-listes-blanches.snapshot). The file holds two parts, both UTF-8 JSON:
- an envelope on the first line, ended by "\n": {"format":"arnakoff-snapshot","version":1,"sha256":"<hex>"}, where sha256 is the hex SHA-256 digest of the body;
- the body, the rest of the file: {"source":"AMF", "fetched_at":"<RFC 3339>","newest":"<RFC 3339>","header":[...], "rows":[[...],...]}; newest is left out when the source dates no entry.
A file whose envelope, digest or body does not check out, or that holds another source, is rejected with ErrCorrupt; one of another version with ErrVersion. A change to the body or envelope bumps the version.
A file is written to a temporary file in the same directory, synced, then renamed over the previous one, and the directory is synced on Unix: a reader sees the previous file or the new one, never part of one.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( ErrCorrupt = errors.New("corrupt snapshot file") ErrVersion = errors.New("unsupported snapshot format version") ErrSourceName = errors.New("source name gives no file name") )
Store errors, checked with errors.Is.
Functions ¶
func Decode ¶ added in v0.50.0
Decode returns the snapshot of source held in the file content b, or ErrCorrupt or ErrVersion when it does not check out.
func Encode ¶ added in v0.50.0
Encode returns the content of the file of s, in the format of the package doc: the bytes every snapshot store writes, on disk or in a bucket.
func FileName ¶ added in v0.50.0
FileName returns the name of the file of source (see the package doc), also the key of its object in a bucket, or ErrSourceName when source has no letter or digit. The name never holds a path separator or a dot before its extension, so it stays in the store's directory.
Types ¶
type Disk ¶
type Disk struct {
// contains filtered or unexported fields
}
Disk stores one snapshot file per source in a directory. It implements app.SnapshotStore and is safe for concurrent use, also across processes.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package objectstore keeps the last valid snapshot of each source in an S3 bucket, the production store of ADR 0005: Scaleway Object Storage in fr-par, shared by the sync job and the API container, which have no volume in common.
|
Package objectstore keeps the last valid snapshot of each source in an S3 bucket, the production store of ADR 0005: Scaleway Object Storage in fr-par, shared by the sync job and the API container, which have no volume in common. |
|
Package postgres keeps the records of the API keys in PostgreSQL (ADR 0004), through jackc/pgx: their keyed digests, never the keys.
|
Package postgres keeps the records of the API keys in PostgreSQL (ADR 0004), through jackc/pgx: their keyed digests, never the keys. |