outbox

package
v0.26.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2026 License: GPL-3.0 Imports: 24 Imported by: 0

Documentation

Overview

Package outbox is the sensor's durable, write-ahead queue of results waiting for the platform.

Every result (a CTIS report, a v1 chunk, a command result) is written to the outbox BEFORE the first attempt to send it, and removed only after the platform acknowledged it (2xx). A crash, kill -9 or restart loses nothing: the next process resumes from the files.

Storage is one directory, used by one process at a time (flock):

<dir>/outbox.key        AES-256-GCM key, 0600, generated on first use
<dir>/pending/<id>.item the result (sealed, immutable)
<dir>/pending/<id>.state its delivery state (sealed, rewritten per attempt)
<dir>/dead/             results the platform refused for good, with
                        <id>.reason.json (plain JSON) saying why
<dir>/corrupt/          files that failed authentication (torn writes,
                        foreign files, a lost key), quarantined
<dir>/tmp/              temporary files of in-progress writes

Every file is 0600 and every directory 0700. Writes are crash-safe (write a temporary file, fsync, rename, fsync the directory). Files are sealed with AES-256-GCM, which also detects a torn or corrupted file: it is moved to corrupt/ and never crashes the process or blocks the queue.

The outbox never fills the disk: a byte cap (default 1 GiB, and never more than a fraction of the free space) and an age cap (default 7 days) evict the oldest entries first (dead letters before pending results), with a log line, a counter and a callback the caller turns into a heartbeat warning.

Delivery is the caller's: Run hands each item, oldest first, to a Deliverer, which classifies the outcome by the error it returns (see Permanent, RetryAfter, Unauthorized). Transient failures back off with jitter behind a circuit breaker; Wake (the platform answered a heartbeat again) drains at once; Unauthorized pauses all delivery until Wake or Resume.

Stability: Internal-bound (docs/STABILITY.md): public today because other public packages use it; it moves under internal/ before v1.0.0. Do not import it from a sensor.

Index

Constants

View Source
const (
	// AttrAssetsOnly marks a report pushed with PushAssets (findings dropped).
	AttrAssetsOnly = "assets_only"
	// AttrImportedFrom says where an item came from when it was imported.
	AttrImportedFrom = "imported_from"
)

Attribute keys the SDK sets on Meta.Attrs.

View Source
const (
	DefaultMaxBytes = 1 << 30 // 1 GiB
	// DefaultMaxDiskFraction caps the outbox at this fraction of the space it
	// could use (its own bytes plus the free space), so a small disk is never
	// filled even below MaxBytes.
	DefaultMaxDiskFraction = 0.5
	DefaultMaxAge          = 7 * 24 * time.Hour
	DefaultConcurrency     = 1
	MaxConcurrency         = 2

	// DefaultBackoffBase and DefaultBackoffMax bound the per-item retry delay
	// after a transient failure (exponential, full jitter in [d/2, d]).
	DefaultBackoffBase = 2 * time.Second
	DefaultBackoffMax  = 5 * time.Minute
	// DefaultCircuitThreshold consecutive transient failures open the circuit:
	// no delivery until the cool-down passes or Wake is called.
	DefaultCircuitThreshold = 3
)

Defaults.

Variables

View Source
var (
	// ErrLocked: another process holds the outbox directory.
	ErrLocked = errors.New("outbox: directory is in use by another process")
	// ErrClosed: the outbox was closed.
	ErrClosed = errors.New("outbox: closed")
	// ErrTooLarge: one item is larger than the outbox may ever hold.
	ErrTooLarge = errors.New("outbox: item larger than the outbox byte cap")
	// ErrKeyMissing: the key file is missing while the outbox holds items
	// sealed with a key. Open refuses rather than create a new key, which
	// would make every one of those items unreadable (see CheckKey).
	ErrKeyMissing = errors.New("outbox: encryption key file is missing but sealed items exist")
)

Errors.

Functions

func CheckKey added in v0.15.0

func CheckKey(dir, keyFile string) error

CheckKey reports whether the outbox in dir can be opened with keyFile (default <dir>/outbox.key) without losing items. It returns nil when the key file exists or when the outbox holds no sealed items (a new key is then harmless), and an error wrapping ErrKeyMissing when the key file is missing while sealed items exist: a new key could not read them, so they would all be quarantined. It never creates or changes anything.

func DefaultKeyFile added in v0.15.0

func DefaultKeyFile(dir string) string

DefaultKeyFile is the key file used when Config.KeyFile is empty.

func Permanent

func Permanent(status int, reason string, problem []byte, err error) error

Permanent classifies a refusal as permanent. problem is the server's RFC 9457 document, if any.

func RetryAfter

func RetryAfter(err error, after time.Duration) error

RetryAfter classifies err as transient, to be retried no sooner than after.

func SealedItems added in v0.15.0

func SealedItems(dir string) int

SealedItems counts the items (pending and dead letters) in the outbox in dir; each is sealed with the outbox key. A missing directory counts 0.

func Unauthorized

func Unauthorized(err error) error

Unauthorized classifies err as a credentials rejection.

Types

type AuthError

type AuthError struct{ Err error }

AuthError is a rejection of the credentials.

func (*AuthError) Error

func (e *AuthError) Error() string

func (*AuthError) Unwrap

func (e *AuthError) Unwrap() error

type Collector

type Collector struct {
	// contains filtered or unexported fields
}

Collector exposes an outbox's Stats as Prometheus metrics. Register it on the registry the process exposes:

prometheus.MustRegister(outbox.NewCollector(o))

func NewCollector

func NewCollector(o *Outbox) *Collector

NewCollector returns a collector for o.

func (*Collector) Collect

func (c *Collector) Collect(ch chan<- prometheus.Metric)

Collect implements prometheus.Collector.

func (*Collector) Describe

func (c *Collector) Describe(ch chan<- *prometheus.Desc)

Describe implements prometheus.Collector.

type Config

type Config struct {
	// Dir is the outbox directory. It is created 0700 if missing.
	Dir string
	// KeyFile is the encryption key file. Default: <Dir>/outbox.key. Point it
	// outside Dir (a mounted secret) to keep the key off the data volume.
	KeyFile string
	// MaxBytes caps the bytes on disk (pending + dead letters). Default 1 GiB.
	MaxBytes int64
	// MaxDiskFraction caps the outbox at this fraction of (its own bytes +
	// the free space). Default 0.5. Negative disables the free-space cap.
	MaxDiskFraction float64
	// MaxAge evicts pending items and dead letters older than this. Default
	// 7 days.
	MaxAge time.Duration
	// Concurrency is how many items are delivered at once (1 or 2). Default 1.
	Concurrency int
	// BackoffBase and BackoffMax bound the per-item retry delay.
	BackoffBase time.Duration
	BackoffMax  time.Duration
	// CircuitThreshold is the number of consecutive transient failures that
	// pause all delivery until the back-off passes or Wake is called.
	CircuitThreshold int
	// Logf receives the outbox's log lines (state changes, evictions,
	// quarantines, dead letters). Default: the standard logger.
	Logf func(format string, args ...any)
	// OnEvict is called (without locks held) after items were evicted to
	// respect the caps.
	OnEvict func(evicted []Meta, reason string)
	// OnDeadLetter is called (without locks held) when an item is
	// dead-lettered.
	OnDeadLetter func(DeadLetter)
	// contains filtered or unexported fields
}

Config configures an outbox. Dir is required; zero values use the defaults.

type DeadLetter

type DeadLetter struct {
	Meta   Meta      `json:"meta"`
	DeadAt time.Time `json:"dead_at"`
	// Status is the HTTP status of the refusal (0 when the SDK refused it).
	Status int `json:"status,omitempty"`
	// Reason is a one-line explanation.
	Reason string `json:"reason"`
	// Problem is the server's RFC 9457 problem document, when it sent one.
	Problem json.RawMessage `json:"problem,omitempty"`
	// Attempts made before the refusal.
	Attempts int `json:"attempts"`
}

DeadLetter is an item the platform refused for good.

type Deliverer

type Deliverer interface {
	Deliver(ctx context.Context, d *Delivery) (value any, err error)
}

Deliverer sends one item to the platform. Its returned error classifies the outcome:

  • nil: acknowledged (2xx). The item is deleted; value is passed to the item's Ticket.
  • an error wrapping *PermanentError (see Permanent): refused for good (400, 409, 413 after splitting, 415, 422 ...). The item moves to the dead-letter folder with the reason; it is never retried by itself.
  • an error wrapping *AuthError (see Unauthorized): the platform rejected the credentials (401). All delivery pauses until Wake or Resume; the attempt is not counted against the item.
  • an error wrapping *RetryError (see RetryAfter): transient with a server-advised delay (429/503 Retry-After).
  • any other error: transient (network, 5xx). Exponential back-off with jitter; consecutive ones open the circuit.

type DelivererFunc

type DelivererFunc func(ctx context.Context, d *Delivery) (any, error)

DelivererFunc adapts a function to Deliverer.

func (DelivererFunc) Deliver

func (f DelivererFunc) Deliver(ctx context.Context, d *Delivery) (any, error)

Deliver implements Deliverer.

type Delivery

type Delivery struct {
	Meta    Meta
	Payload []byte
	// State is the item's state before this attempt (Attempts counts earlier
	// attempts).
	State State
	// contains filtered or unexported fields
}

Delivery is one attempt at one item.

func (*Delivery) Outbox

func (d *Delivery) Outbox() *Outbox

Outbox returns the outbox the item belongs to (for DeadLetterForCommand).

func (*Delivery) SaveProgress

func (d *Delivery) SaveProgress(progress json.RawMessage) error

SaveProgress persists deliverer-owned progress (for example which v2 segments were acknowledged) so a restart resumes instead of starting over. It is crash-safe like every outbox write.

type Inspection added in v0.8.1

type Inspection struct {
	Dir             string
	PendingCount    int
	PendingBytes    int64
	OldestPending   time.Time
	DeadLetters     []DeadLetter
	DeadLetterBytes int64
	// Unreadable counts pending entries Inspect could not read (a write in
	// progress, a torn file, another key); the owning process deals with
	// them.
	Unreadable int
}

Inspection is a read-only view of an outbox directory (see Inspect).

func Inspect added in v0.8.1

func Inspect(dir, keyFile string) (*Inspection, error)

Inspect reads an outbox directory without taking its lock and without changing anything, so an operator can look at the outbox of a running sensor. It needs the key file only to read the pending items' creation times; without it the counts and sizes are still reported. Entries changing while it reads are skipped.

type Kind

type Kind string

Kind is what an item carries.

const (
	// KindReport is a CTIS report (findings and/or assets).
	KindReport Kind = "report"
	// KindCommandResult is the completion or failure of a platform command.
	// It is delivered only after every older report of the same command.
	KindCommandResult Kind = "command_result"
	// KindCommandLog is a batch of a command's log lines. It is delivered
	// before the command's result, evicted first under the byte cap, and
	// losing it never marks the command's results as lost.
	KindCommandLog Kind = "command_log"
	// KindChunk was a protocol v1 chunk of a large report. Protocol v1 is
	// retired: such an item left in an outbox is not delivered.
	KindChunk Kind = "chunk"
)

type Meta

type Meta struct {
	// ID is a UUIDv7 and the file name. Enqueue assigns it.
	ID   string `json:"id"`
	Kind Kind   `json:"kind"`
	// ReportID is the stable report id (UUIDv7) the v2 protocol uses as the
	// idempotency key. Enqueue assigns one to a KindReport item that has none.
	ReportID string `json:"report_id,omitempty"`
	// CommandID binds the item to a platform command.
	CommandID string `json:"command_id,omitempty"`
	// CreatedAt orders the items (oldest first). Enqueue sets it.
	CreatedAt time.Time `json:"created_at"`
	// Attrs are free-form small strings for the deliverer.
	Attrs map[string]string `json:"attrs,omitempty"`
}

Meta describes an item. It is fixed when the item is enqueued.

type Outbox

type Outbox struct {
	// contains filtered or unexported fields
}

Outbox is a durable queue. Safe for concurrent use.

func Open

func Open(cfg Config) (*Outbox, error)

Open opens (creating if needed) the outbox directory, takes its lock, loads the key and indexes the items. Items that fail authentication are quarantined; Open fails only when the directory cannot be used at all.

func (*Outbox) Close

func (o *Outbox) Close() error

Close stops accepting items and releases the directory lock. Run returns when its context ends; Close does not wait for it.

func (*Outbox) DeadLetterForCommand

func (o *Outbox) DeadLetterForCommand(commandID string) (DeadLetter, bool)

DeadLetterForCommand returns a dead letter of a report bound to commandID, so a command's result can say its results were refused.

func (*Outbox) DeadLetters

func (o *Outbox) DeadLetters() []DeadLetter

DeadLetters returns the dead letters, oldest first.

func (*Outbox) Dir

func (o *Outbox) Dir() string

Dir returns the outbox directory.

func (*Outbox) Enqueue

func (o *Outbox) Enqueue(m Meta, payload []byte) (*Ticket, error)

Enqueue durably stores an item and returns its ticket. When it returns nil the item is on disk (fsynced) and will be delivered even if the process dies now. Meta.ID and Meta.CreatedAt are assigned; a KindReport without a ReportID gets one.

func (*Outbox) ImportLegacyRetryQueue

func (o *Outbox) ImportLegacyRetryQueue(dir string) (int, error)

ImportLegacyRetryQueue moves the reports a pre-outbox SDK left in its JSON retry queue directory (pkg/retry FileRetryQueue, default ~/.openctem/retry-queue) into the outbox, so an upgrade loses nothing. Each file is removed only after its report is durably enqueued; a file that is not a queue item is left alone. It returns how many reports were imported. A missing directory is not an error.

func (*Outbox) Pending

func (o *Outbox) Pending() []Meta

Pending returns the metadata of the pending items, oldest first.

func (*Outbox) RequeueDead

func (o *Outbox) RequeueDead(id string) error

RequeueDead moves a dead letter back to pending with a fresh attempt count (after the operator fixed the cause).

func (*Outbox) Resume

func (o *Outbox) Resume()

Resume lifts an authentication pause (the API key was replaced).

func (*Outbox) Run

func (o *Outbox) Run(ctx context.Context, d Deliverer) error

Run delivers items with d until ctx ends: oldest first, Concurrency at a time. Call it once per outbox.

func (*Outbox) Stats

func (o *Outbox) Stats() Stats

Stats returns a snapshot.

func (*Outbox) Wait

func (o *Outbox) Wait(ctx context.Context) error

Wait blocks until ctx ends or no pending item is ready to be delivered now (everything left is backing off, paused or there is nothing). It is what a one-shot run calls before exiting, after Wake.

func (*Outbox) Wake

func (o *Outbox) Wake()

Wake says the platform is reachable again (a heartbeat was accepted): the circuit closes, an authentication pause lifts and every item backing off after a network or server error becomes ready now. Items told to wait by a Retry-After keep their delay.

type PermanentError

type PermanentError struct {
	Status  int
	Reason  string
	Problem json.RawMessage
	Err     error
}

PermanentError is a refusal that retrying cannot fix.

func (*PermanentError) Error

func (e *PermanentError) Error() string

func (*PermanentError) Unwrap

func (e *PermanentError) Unwrap() error

type Result

type Result struct {
	// Delivered: the platform acknowledged the item.
	Delivered bool
	// Value is what the Deliverer returned with the acknowledgement.
	Value any
	// Dead is set when the item was dead-lettered.
	Dead *DeadLetter
	// Evicted: the item was dropped by the size or age cap.
	Evicted bool
}

Result is the final outcome of one item, sent on its Ticket.

type RetryError

type RetryError struct {
	After time.Duration
	Err   error
}

RetryError is a transient failure with a server-advised delay.

func (*RetryError) Error

func (e *RetryError) Error() string

func (*RetryError) Unwrap

func (e *RetryError) Unwrap() error

type State

type State struct {
	Attempts    int       `json:"attempts"`
	NextAttempt time.Time `json:"next_attempt,omitzero"`
	LastError   string    `json:"last_error,omitempty"`
	// RateLimited: the last failure carried a Retry-After, which Wake does
	// not override.
	RateLimited bool `json:"rate_limited,omitempty"`
	// Progress is owned by the deliverer (for example the v2 segments the
	// server already acknowledged); see Delivery.SaveProgress.
	Progress json.RawMessage `json:"progress,omitempty"`
	// LostResults is set on a KindCommandResult item when a result of the
	// same command was dropped before delivery (evicted by the caps, or
	// quarantined as unreadable): the command must not be reported as a
	// clean, complete run. It says why.
	LostResults string `json:"lost_results,omitempty"`
}

State is an item's delivery state, persisted after every attempt.

type Stats

type Stats struct {
	PendingCount int
	PendingBytes int64
	// OldestPending is the creation time of the oldest pending item (zero
	// when none).
	OldestPending   time.Time
	DeadLetterCount int
	DeadLetterBytes int64
	// Counters since Open.
	Evicted     int64
	Corrupt     int64
	Delivered   int64
	DeadLetters int64
	Attempts    int64
	LastEvictAt time.Time
	// Delivery control.
	AuthPaused   bool
	CircuitOpen  bool
	CircuitUntil time.Time
	CapBytes     int64
}

Stats is a snapshot of the outbox, for the heartbeat and metrics.

func (Stats) OldestAge

func (s Stats) OldestAge(now time.Time) time.Duration

OldestAge is the age of the oldest pending item at now.

type Ticket

type Ticket struct {
	Meta Meta
	// contains filtered or unexported fields
}

Ticket follows one enqueued item.

func (*Ticket) Done

func (t *Ticket) Done() <-chan Result

Done delivers the item's final Result once (buffered; never blocks the outbox).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL