bundle

package
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 11, 2026 License: GPL-3.0 Imports: 23 Imported by: 0

Documentation

Overview

Package bundle is the chunked, signed feed bundle (format v2) of api docs/rfcs/RFC-070-chunked-feed-transfer.md: the producer's Writer, the signed Pointer and Manifest, and the Consumer that fetches, verifies and applies a bundle chunk by chunk with a durable checkpoint.

A release is a flat set of files (flat so it fits release-asset hosting):

keyset.dsse.json                  the feed's key set (verified by the caller's Trust)
latest.v2.dsse.json               signed Pointer: sequence, manifest names and digests
snapshot.v2.manifest.dsse.json    signed Manifest of the full record set
delta.v2.manifest.dsse.json       signed Manifest of the changes since BaseSequence
sha256-<hex>.jsonl.gz             chunks: gzip JSON Lines, named by their digest

The pointer pins each manifest's sha256; each manifest lists, per record stream, its chunks with sha256, size, uncompressed size, record count and the first and last record id. So one signature covers every byte, and a mirror can serve the files but cannot change them.

Chunk boundaries are content-defined from record ids (records are written in ascending id order), so a change to one record changes one chunk and the others keep their digest across releases: consumers fetch them once (transfer's blob cache) and object-store mirrors store them once.

Stability: Experimental (docs/STABILITY.md).

Index

Constants

View Source
const (
	Schema        = "openctem.bundle/v2"
	PointerSchema = "openctem.bundle.pointer/v2"

	KindSnapshot = "snapshot"
	KindDelta    = "delta"

	PointerFile = "latest.v2.dsse.json"
	KeySetFile  = "keyset.dsse.json"
)

Schemas, kinds and file names.

View Source
const (
	// DefaultTargetRecords is the average records per chunk: a chunk ends
	// after a record whose id hashes to 0 modulo this (content-defined).
	DefaultTargetRecords = 2000
	// DefaultMaxUncompressed is the hard cap of a chunk's JSON bytes
	// (about 4-8 MiB compressed for typical feed records).
	DefaultMaxUncompressed = 48 << 20
)

Writer defaults.

Variables

View Source
var (
	// ErrRollback: the origin serves an older sequence than the one applied
	// (a stale mirror or a replay). Nothing is applied.
	ErrRollback = errors.New("bundle: sequence older than the applied one (rollback refused)")
	// ErrPinMismatch: a manifest does not match the digest its pointer pins.
	ErrPinMismatch = errors.New("bundle: manifest does not match the pointer")
	// ErrChunk: a chunk's content does not match its manifest entry.
	ErrChunk = errors.New("bundle: chunk does not match the manifest")
)

Errors of a consumer run.

View Source
var ErrExpired = errors.New("bundle: expired")

ErrExpired: the pointer or manifest is past its expiry.

View Source
var ErrInvalid = errors.New("bundle: invalid")

ErrInvalid marks a structurally invalid pointer or manifest.

View Source
var ErrSignature = errors.New("bundle: signature not verified")

ErrSignature: no trusted key signed the envelope.

Functions

func ChunkFile

func ChunkFile(sha256Hex string) string

ChunkFile is the file name of a chunk.

func ManifestFile

func ManifestFile(kind string) string

ManifestFile is the manifest file of a kind.

func ManifestPayloadType

func ManifestPayloadType(feed string) string

ManifestPayloadType is the DSSE payload type of a feed manifest.

func PointerPayloadType

func PointerPayloadType(feed string) string

PointerPayloadType and ManifestPayloadType are the DSSE payload types of a feed's pointer and manifests: a signature of one feed never verifies as another feed's, nor a pointer as a manifest.

func WritePointer

func WritePointer(dir string, p Pointer, signer Signer) error

WritePointer writes the signed pointer (PointerFile) into dir.

Types

type Applier

type Applier interface {
	// ApplyChunk applies one chunk's records, in one transaction. It must
	// return any error of c.Records and then not commit. A database applier
	// should also persist next in that transaction (exactly-once); the
	// consumer then saves next through the Checkpoint as well.
	//
	// Upserts are keyed by record id and must not replace a record stored
	// by a newer sequence than c.Manifest.Sequence: a chunk may be applied
	// twice after a crash.
	ApplyChunk(ctx context.Context, c *Chunk, next State) error
	// Complete finishes the bundle after its last chunk (for a snapshot:
	// remove what the snapshot no longer holds). It may be called again
	// after a crash and must be idempotent.
	Complete(ctx context.Context, m *Manifest, next State) error
}

Applier applies a verified bundle. Every method runs after the bytes it receives were verified against the signed manifest.

type Checkpoint

type Checkpoint interface {
	Load(ctx context.Context) (State, error)
	Save(ctx context.Context, s State) error
}

Checkpoint stores a consumer's State. The platform implements it on a database table (one row per feed); FileCheckpoint is the file one.

Save after a chunk may be called again with the same State after a crash: an Applier's upserts must be idempotent (keyed by record id, and never replacing a record of a newer sequence).

type Chunk

type Chunk struct {
	Manifest *Manifest
	Located
	// contains filtered or unexported fields
}

Chunk is one verified chunk handed to an Applier.

func (*Chunk) Records

func (c *Chunk) Records(fn func(line []byte) error) error

Records calls fn for each record line in order, streaming (bounded memory). It refuses a chunk whose decompressed size or record count differs from its manifest entry, or a line longer than MaxRecordBytes. The line is only valid during the call.

type ChunkRef

type ChunkRef struct {
	SHA256       string `json:"sha256"`
	Size         int64  `json:"size"`
	Uncompressed int64  `json:"uncompressed"`
	Records      int    `json:"records"`
	FirstID      string `json:"first_id"`
	LastID       string `json:"last_id"`
}

ChunkRef describes one chunk.

type Config

type Config struct {
	Feed    string
	Fetcher *transfer.Fetcher
	// Trust builds the verifier from the release's key set (KeySetFile),
	// checking it against the pinned root. Nil: Verifier is used and no key
	// set is fetched.
	Trust    func(ctx context.Context, keySet []byte) (Verifier, error)
	Verifier Verifier

	Checkpoint Checkpoint
	Applier    Applier
	Limits     Limits
	// KeepCache keeps chunks of earlier bundles in the fetcher's cache
	// (default: only the applied bundle's chunks are kept, for reuse by the
	// next release).
	KeepCache bool
	Logger    *slog.Logger
	Now       func() time.Time
}

Config configures a Consumer.

type Consumer

type Consumer struct {
	// contains filtered or unexported fields
}

Consumer fetches, verifies and applies a feed's bundles.

func NewConsumer

func NewConsumer(cfg Config) (*Consumer, error)

NewConsumer returns a Consumer.

func (*Consumer) Register

func (c *Consumer) Register(reg prometheus.Registerer, namespace string) error

Register exposes the consumer's counters as Prometheus counters named <namespace>_bundle_<counter>_total, with a constant label feed.

func (*Consumer) Run

func (c *Consumer) Run(ctx context.Context) (Result, error)

Run applies the newest bundle if it is newer than the applied one, resuming a bundle a previous run left half-applied. Signatures, pins and caps are checked before any chunk is fetched; each chunk's hash before it is parsed. One Run at a time per feed (the caller holds a lock).

func (*Consumer) Stats

func (c *Consumer) Stats() ConsumerStats

Stats returns the counters.

type ConsumerStats

type ConsumerStats struct {
	Runs, Completed, Resumes, ChunksApplied, ChunksFailed, Records, Refused uint64
}

ConsumerStats are a consumer's counters.

type Ed25519Signer

type Ed25519Signer struct{ Key ed25519.PrivateKey }

Ed25519Signer signs DSSE v1 envelopes (the envelope, pre-authentication encoding and key id of package jobsig).

func (Ed25519Signer) Sign

func (s Ed25519Signer) Sign(payloadType string, payload []byte) ([]byte, error)

Sign implements Signer.

type Ed25519Verifier

type Ed25519Verifier struct {
	// contains filtered or unexported fields
}

Ed25519Verifier accepts envelopes signed by any of its keys.

func NewEd25519Verifier

func NewEd25519Verifier(pubs ...ed25519.PublicKey) *Ed25519Verifier

NewEd25519Verifier trusts pubs.

func (*Ed25519Verifier) Verify

func (v *Ed25519Verifier) Verify(envelope []byte, payloadType string) ([]byte, error)

Verify implements Verifier.

type FileCheckpoint

type FileCheckpoint struct{ Path string }

FileCheckpoint stores the State as JSON in one file (0600, written crash-safely).

func (FileCheckpoint) Load

Load implements Checkpoint; a missing file is the zero State.

func (FileCheckpoint) Save

func (c FileCheckpoint) Save(_ context.Context, s State) error

Save implements Checkpoint.

type FileRef

type FileRef struct {
	Name   string `json:"name"`
	SHA256 string `json:"sha256"`
	Size   int64  `json:"size"`
}

FileRef pins a manifest file.

type Limits

type Limits struct {
	MaxPointerBytes      int64         // default 64 KiB
	MaxKeySetBytes       int64         // default 64 KiB
	MaxManifestBytes     int64         // default 8 MiB
	MaxChunks            int           // per manifest, default 8192
	MaxChunkBytes        int64         // compressed, default 32 MiB
	MaxChunkUncompressed int64         // default 256 MiB
	MaxChunkRecords      int           // default 1 000 000
	MaxRecordBytes       int           // one JSON line, default 4 MiB
	MaxTotalBytes        int64         // compressed sum of a manifest, default 8 GiB
	MaxValidity          time.Duration // expires_at - created_at, default 8 days
	ClockSkew            time.Duration // tolerated future created_at, default 10 min
}

Limits bound what a consumer accepts (decompression bombs, oversized manifests). Zero values take the defaults.

type Located

type Located struct {
	Stream string
	Index  int
	Ref    ChunkRef
}

Located is a chunk with its stream and its position in apply order.

type Manifest

type Manifest struct {
	Schema       string    `json:"schema"`
	Feed         string    `json:"feed"`
	Sequence     uint64    `json:"sequence"`
	Kind         string    `json:"kind"`
	BaseSequence uint64    `json:"base_sequence,omitempty"`
	CreatedAt    time.Time `json:"created_at"`
	ExpiresAt    time.Time `json:"expires_at"`
	Streams      []Stream  `json:"streams"`
	// Meta is the feed's own signed metadata (sources, licenses, stats,
	// collector build). The SDK does not interpret it.
	Meta json.RawMessage `json:"meta,omitempty"`
}

Manifest is the signed description of a snapshot or a delta.

func (*Manifest) Chunks

func (m *Manifest) Chunks() []Located

Chunks returns the manifest's chunks in apply order with their stream.

func (*Manifest) Validate

func (m *Manifest) Validate(feed string, now time.Time, l Limits) error

Validate checks the manifest's structure, caps and freshness.

type Pointer

type Pointer struct {
	Schema       string    `json:"schema"`
	Feed         string    `json:"feed"`
	Sequence     uint64    `json:"sequence"`
	Snapshot     FileRef   `json:"snapshot"`
	Delta        *FileRef  `json:"delta,omitempty"`
	BaseSequence uint64    `json:"base_sequence,omitempty"`
	CreatedAt    time.Time `json:"created_at"`
	ExpiresAt    time.Time `json:"expires_at"`
}

Pointer is the signed pointer to a release's newest bundle.

func (*Pointer) Validate

func (p *Pointer) Validate(feed string, now time.Time, l Limits) error

Validate checks the pointer's structure and freshness.

type Result

type Result struct {
	UpToDate      bool
	Sequence      uint64
	Kind          string
	Resumed       bool
	ChunksApplied int
	Records       int
}

Result says what a run did.

type Signer

type Signer interface {
	Sign(payloadType string, payload []byte) ([]byte, error)
}

Signer wraps a payload in a signed DSSE envelope.

type State

type State struct {
	Feed string `json:"feed"`
	// Applied is the last sequence applied completely (0: none).
	Applied uint64 `json:"applied"`
	// InProgress is the sequence being applied (0: none). It is never
	// reported as applied until every chunk and Complete succeeded.
	InProgress uint64 `json:"in_progress,omitempty"`
	// Kind and Manifest (sha256 of the manifest envelope) identify the
	// bundle in progress: a resume continues only the same manifest.
	Kind     string `json:"kind,omitempty"`
	Manifest string `json:"manifest,omitempty"`
	// NextChunk is the apply-order index of the next chunk to apply.
	NextChunk int       `json:"next_chunk,omitempty"`
	UpdatedAt time.Time `json:"updated_at"`
}

State is a consumer's durable progress on one feed.

type Stream

type Stream struct {
	Name    string     `json:"name"`
	Records int        `json:"records"`
	Chunks  []ChunkRef `json:"chunks"`
}

Stream is one record stream (for example "programs"), applied in manifest order, its chunks in order.

type Verifier

type Verifier interface {
	Verify(envelope []byte, payloadType string) ([]byte, error)
}

Verifier checks a DSSE envelope of the expected payload type and returns its payload. A feed's verifier checks its key set (offline root, version, expiry) and the online keys it lists; Ed25519Verifier is the plain one.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes one bundle's chunks and its signed manifest into a directory. Chunk files are content-addressed: a chunk identical to one already in the directory (an earlier release) is not written twice.

func NewWriter

func NewWriter(dir string, opt WriterOptions) (*Writer, error)

NewWriter returns a Writer into dir (created 0755: bundles are public).

func (*Writer) Add

func (w *Writer) Add(id string, record []byte) error

Add appends a record (one JSON value, no newline) with its id. Ids must strictly ascend within a stream.

func (*Writer) Finish

func (w *Writer) Finish(meta json.RawMessage, createdAt, expiresAt time.Time, signer Signer) (*Manifest, FileRef, error)

Finish writes the signed manifest (ManifestFile(kind)) and returns it with the reference a Pointer needs.

func (*Writer) Stream

func (w *Writer) Stream(name string) error

Stream starts the next record stream; the previous one is finished.

type WriterOptions

type WriterOptions struct {
	Feed         string
	Sequence     uint64
	Kind         string // KindSnapshot or KindDelta
	BaseSequence uint64 // a delta's base
	// TargetRecords is the average records per chunk (content-defined
	// boundary); MaxRecords (default 4x TargetRecords) and MaxUncompressed
	// are hard caps. A chunk holds at least TargetRecords/4 records unless
	// a cap or the stream's end cuts it.
	TargetRecords   int
	MaxRecords      int
	MaxUncompressed int64
}

WriterOptions configure a Writer.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL