Documentation
¶
Overview ¶
Package bundle is the chunked, signed feed bundle (format v2) of api docs/rfcs/RFC-070-chunked-feed-transfer.md: the producer's Writer, the signed Pointer and Manifest, and the Consumer that fetches, verifies and applies a bundle chunk by chunk with a durable checkpoint.
A release is a flat set of files (flat so it fits release-asset hosting):
keyset.dsse.json the feed's key set (verified by the caller's Trust) latest.v2.dsse.json signed Pointer: sequence, manifest names and digests snapshot.v2.manifest.dsse.json signed Manifest of the full record set delta.v2.manifest.dsse.json signed Manifest of the changes since BaseSequence sha256-<hex>.jsonl.gz chunks: gzip JSON Lines, named by their digest
The pointer pins each manifest's sha256; each manifest lists, per record stream, its chunks with sha256, size, uncompressed size, record count and the first and last record id. So one signature covers every byte, and a mirror can serve the files but cannot change them.
Chunk boundaries are content-defined from record ids (records are written in ascending id order), so a change to one record changes one chunk and the others keep their digest across releases: consumers fetch them once (transfer's blob cache) and object-store mirrors store them once.
Stability: Experimental (docs/STABILITY.md).
Index ¶
- Constants
- Variables
- func ChunkFile(sha256Hex string) string
- func ManifestFile(kind string) string
- func ManifestPayloadType(feed string) string
- func PointerPayloadType(feed string) string
- func WritePointer(dir string, p Pointer, signer Signer) error
- type Applier
- type Checkpoint
- type Chunk
- type ChunkRef
- type Config
- type Consumer
- type ConsumerStats
- type Ed25519Signer
- type Ed25519Verifier
- type FileCheckpoint
- type FileRef
- type Limits
- type Located
- type Manifest
- type Pointer
- type Result
- type Signer
- type State
- type Stream
- type Verifier
- type Writer
- type WriterOptions
Constants ¶
const ( Schema = "openctem.bundle/v2" PointerSchema = "openctem.bundle.pointer/v2" KindSnapshot = "snapshot" KindDelta = "delta" PointerFile = "latest.v2.dsse.json" KeySetFile = "keyset.dsse.json" )
Schemas, kinds and file names.
const ( // DefaultTargetRecords is the average records per chunk: a chunk ends // after a record whose id hashes to 0 modulo this (content-defined). DefaultTargetRecords = 2000 // DefaultMaxUncompressed is the hard cap of a chunk's JSON bytes // (about 4-8 MiB compressed for typical feed records). DefaultMaxUncompressed = 48 << 20 )
Writer defaults.
Variables ¶
var ( // ErrRollback: the origin serves an older sequence than the one applied // (a stale mirror or a replay). Nothing is applied. ErrRollback = errors.New("bundle: sequence older than the applied one (rollback refused)") // ErrPinMismatch: a manifest does not match the digest its pointer pins. ErrPinMismatch = errors.New("bundle: manifest does not match the pointer") // ErrChunk: a chunk's content does not match its manifest entry. ErrChunk = errors.New("bundle: chunk does not match the manifest") )
Errors of a consumer run.
var ErrExpired = errors.New("bundle: expired")
ErrExpired: the pointer or manifest is past its expiry.
var ErrInvalid = errors.New("bundle: invalid")
ErrInvalid marks a structurally invalid pointer or manifest.
var ErrSignature = errors.New("bundle: signature not verified")
ErrSignature: no trusted key signed the envelope.
Functions ¶
func ManifestFile ¶
ManifestFile is the manifest file of a kind.
func ManifestPayloadType ¶
ManifestPayloadType is the DSSE payload type of a feed manifest.
func PointerPayloadType ¶
PointerPayloadType and ManifestPayloadType are the DSSE payload types of a feed's pointer and manifests: a signature of one feed never verifies as another feed's, nor a pointer as a manifest.
Types ¶
type Applier ¶
type Applier interface {
// ApplyChunk applies one chunk's records, in one transaction. It must
// return any error of c.Records and then not commit. A database applier
// should also persist next in that transaction (exactly-once); the
// consumer then saves next through the Checkpoint as well.
//
// Upserts are keyed by record id and must not replace a record stored
// by a newer sequence than c.Manifest.Sequence: a chunk may be applied
// twice after a crash.
ApplyChunk(ctx context.Context, c *Chunk, next State) error
// Complete finishes the bundle after its last chunk (for a snapshot:
// remove what the snapshot no longer holds). It may be called again
// after a crash and must be idempotent.
Complete(ctx context.Context, m *Manifest, next State) error
}
Applier applies a verified bundle. Every method runs after the bytes it receives were verified against the signed manifest.
type Checkpoint ¶
type Checkpoint interface {
Load(ctx context.Context) (State, error)
Save(ctx context.Context, s State) error
}
Checkpoint stores a consumer's State. The platform implements it on a database table (one row per feed); FileCheckpoint is the file one.
Save after a chunk may be called again with the same State after a crash: an Applier's upserts must be idempotent (keyed by record id, and never replacing a record of a newer sequence).
type Chunk ¶
Chunk is one verified chunk handed to an Applier.
type ChunkRef ¶
type ChunkRef struct {
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
Uncompressed int64 `json:"uncompressed"`
Records int `json:"records"`
FirstID string `json:"first_id"`
LastID string `json:"last_id"`
}
ChunkRef describes one chunk.
type Config ¶
type Config struct {
Feed string
Fetcher *transfer.Fetcher
// Trust builds the verifier from the release's key set (KeySetFile),
// checking it against the pinned root. Nil: Verifier is used and no key
// set is fetched.
Trust func(ctx context.Context, keySet []byte) (Verifier, error)
Verifier Verifier
Checkpoint Checkpoint
Applier Applier
Limits Limits
// KeepCache keeps chunks of earlier bundles in the fetcher's cache
// (default: only the applied bundle's chunks are kept, for reuse by the
// next release).
KeepCache bool
Logger *slog.Logger
Now func() time.Time
}
Config configures a Consumer.
type Consumer ¶
type Consumer struct {
// contains filtered or unexported fields
}
Consumer fetches, verifies and applies a feed's bundles.
func (*Consumer) Register ¶
func (c *Consumer) Register(reg prometheus.Registerer, namespace string) error
Register exposes the consumer's counters as Prometheus counters named <namespace>_bundle_<counter>_total, with a constant label feed.
func (*Consumer) Run ¶
Run applies the newest bundle if it is newer than the applied one, resuming a bundle a previous run left half-applied. Signatures, pins and caps are checked before any chunk is fetched; each chunk's hash before it is parsed. One Run at a time per feed (the caller holds a lock).
type ConsumerStats ¶
type ConsumerStats struct {
Runs, Completed, Resumes, ChunksApplied, ChunksFailed, Records, Refused uint64
}
ConsumerStats are a consumer's counters.
type Ed25519Signer ¶
type Ed25519Signer struct{ Key ed25519.PrivateKey }
Ed25519Signer signs DSSE v1 envelopes (the envelope, pre-authentication encoding and key id of package jobsig).
type Ed25519Verifier ¶
type Ed25519Verifier struct {
// contains filtered or unexported fields
}
Ed25519Verifier accepts envelopes signed by any of its keys.
func NewEd25519Verifier ¶
func NewEd25519Verifier(pubs ...ed25519.PublicKey) *Ed25519Verifier
NewEd25519Verifier trusts pubs.
type FileCheckpoint ¶
type FileCheckpoint struct{ Path string }
FileCheckpoint stores the State as JSON in one file (0600, written crash-safely).
type FileRef ¶
type FileRef struct {
Name string `json:"name"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
FileRef pins a manifest file.
type Limits ¶
type Limits struct {
MaxPointerBytes int64 // default 64 KiB
MaxKeySetBytes int64 // default 64 KiB
MaxManifestBytes int64 // default 8 MiB
MaxChunks int // per manifest, default 8192
MaxChunkBytes int64 // compressed, default 32 MiB
MaxChunkUncompressed int64 // default 256 MiB
MaxChunkRecords int // default 1 000 000
MaxRecordBytes int // one JSON line, default 4 MiB
MaxTotalBytes int64 // compressed sum of a manifest, default 8 GiB
MaxValidity time.Duration // expires_at - created_at, default 8 days
ClockSkew time.Duration // tolerated future created_at, default 10 min
}
Limits bound what a consumer accepts (decompression bombs, oversized manifests). Zero values take the defaults.
type Manifest ¶
type Manifest struct {
Schema string `json:"schema"`
Feed string `json:"feed"`
Sequence uint64 `json:"sequence"`
Kind string `json:"kind"`
BaseSequence uint64 `json:"base_sequence,omitempty"`
CreatedAt time.Time `json:"created_at"`
ExpiresAt time.Time `json:"expires_at"`
Streams []Stream `json:"streams"`
// Meta is the feed's own signed metadata (sources, licenses, stats,
// collector build). The SDK does not interpret it.
Meta json.RawMessage `json:"meta,omitempty"`
}
Manifest is the signed description of a snapshot or a delta.
type Pointer ¶
type Pointer struct {
Schema string `json:"schema"`
Feed string `json:"feed"`
Sequence uint64 `json:"sequence"`
Snapshot FileRef `json:"snapshot"`
Delta *FileRef `json:"delta,omitempty"`
BaseSequence uint64 `json:"base_sequence,omitempty"`
CreatedAt time.Time `json:"created_at"`
ExpiresAt time.Time `json:"expires_at"`
}
Pointer is the signed pointer to a release's newest bundle.
type Result ¶
type Result struct {
UpToDate bool
Sequence uint64
Kind string
Resumed bool
ChunksApplied int
Records int
}
Result says what a run did.
type State ¶
type State struct {
Feed string `json:"feed"`
// Applied is the last sequence applied completely (0: none).
Applied uint64 `json:"applied"`
// InProgress is the sequence being applied (0: none). It is never
// reported as applied until every chunk and Complete succeeded.
InProgress uint64 `json:"in_progress,omitempty"`
// Kind and Manifest (sha256 of the manifest envelope) identify the
// bundle in progress: a resume continues only the same manifest.
Kind string `json:"kind,omitempty"`
Manifest string `json:"manifest,omitempty"`
// NextChunk is the apply-order index of the next chunk to apply.
NextChunk int `json:"next_chunk,omitempty"`
UpdatedAt time.Time `json:"updated_at"`
}
State is a consumer's durable progress on one feed.
type Stream ¶
type Stream struct {
Name string `json:"name"`
Records int `json:"records"`
Chunks []ChunkRef `json:"chunks"`
}
Stream is one record stream (for example "programs"), applied in manifest order, its chunks in order.
type Verifier ¶
Verifier checks a DSSE envelope of the expected payload type and returns its payload. A feed's verifier checks its key set (offline root, version, expiry) and the online keys it lists; Ed25519Verifier is the plain one.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer writes one bundle's chunks and its signed manifest into a directory. Chunk files are content-addressed: a chunk identical to one already in the directory (an earlier release) is not written twice.
func NewWriter ¶
func NewWriter(dir string, opt WriterOptions) (*Writer, error)
NewWriter returns a Writer into dir (created 0755: bundles are public).
func (*Writer) Add ¶
Add appends a record (one JSON value, no newline) with its id. Ids must strictly ascend within a stream.
type WriterOptions ¶
type WriterOptions struct {
Feed string
Sequence uint64
Kind string // KindSnapshot or KindDelta
BaseSequence uint64 // a delta's base
// TargetRecords is the average records per chunk (content-defined
// boundary); MaxRecords (default 4x TargetRecords) and MaxUncompressed
// are hard caps. A chunk holds at least TargetRecords/4 records unless
// a cap or the stream's end cuts it.
TargetRecords int
MaxRecords int
MaxUncompressed int64
}
WriterOptions configure a Writer.