archive

package
v1.0.0-beta.28 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 17, 2026 License: Apache-2.0 Imports: 33 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// LogQueryContentType is the request body type of log queries and
	// searches.
	LogQueryContentType = "application/json"
	// MaxLogQuerySpans bounds the spans one log query or search names.
	MaxLogQuerySpans = 8192
	// MaxLogQueryBytes bounds a log query or search body.
	MaxLogQueryBytes = 2 << 20
	// MaxLogQueryDigests bounds the digests one span of a query names.
	MaxLogQueryDigests = 64
)
View Source
const (
	DefaultLogSearchLimit = 100
	MaxLogSearchLimit     = 1000
	// MaxLogSearchSnippet bounds a match's text, in bytes.
	MaxLogSearchSnippet = 256
)
View Source
const (
	DefaultTTL   = 7 * 24 * time.Hour
	DefaultQuota = int64(10 << 30)
)
View Source
const (
	LogRecordsAll          = ""
	LogRecordsLogs         = "logs"
	LogRecordsCallPayloads = "call_payloads"
	LogRecordsMetadata     = "metadata"
)
View Source
const AgentBootstrapResource = "agent-bootstrap"

AgentBootstrapResource names the agent-restore bootstrap within an archive. It is agent-specific: final agent/subscription control records plus the recipe closure of their snapshots. The OTLP signal streams alongside it are generic telemetry resources.

View Source
const BootstrapContentType = "application/vnd.dagger.telemetry.bootstrap"
View Source
const CallClosureResource = "calls"

CallClosureResource is the archive endpoint that reads the call payloads of recipe closures (POST a CallClosureQuery). The reply is a framed log stream of call-payload records whose cursors count its frames, not archive rows.

View Source
const ManifestVersion = 5
View Source
const MaxBootstrapPayloadSize = 64 << 20

MaxBootstrapPayloadSize bounds the encoded payload of each bootstrap frame.

View Source
const MaxCallClosureDigests = 256

MaxCallClosureDigests bounds the digests one call-closure query names.

View Source
const MaxListenSpans = 256

MaxListenSpans bounds the subtrees one span selection may listen to. Callers with more split them across requests.

View Source
const MaxMetricNames = 64

MaxMetricNames bounds the names one metric selection may name.

View Source
const MaxTitleRunes = 120

MaxTitleRunes bounds an archive title. Titles come from trace records any session producer can emit, and listings print them to a terminal.

View Source
const SelectionHeader = "X-Dagger-Archive-Selection"

SelectionHeader is the response header in which an engine lists the selections it applied that an older engine would ignore: "metrics" for a MetricSelection.

View Source
const SelectionMetrics = "metrics"

SelectionMetrics is the SelectionHeader value for an applied MetricSelection.

Variables

View Source
var (
	ErrCleanMiss = errors.New("engine archive clean miss")
	ErrState     = errors.New("engine archive state failure")
	ErrCorrupt   = errors.New("engine archive corruption")
	ErrTransient = errors.New("engine archive transient failure")
)
View Source
var ErrBootstrapIncomplete = errors.New("bootstrap stream ended before terminal frame")

ErrBootstrapIncomplete identifies a finite bootstrap response that ended before its terminal frame.

View Source
var ErrCallClosureUnsupported = errors.New("engine archive cannot read call closures")

ErrCallClosureUnsupported fails a call-closure read on an engine that predates the endpoint.

View Source
var ErrSelectionUnsupported = errors.New("engine archive does not support the selection")

ErrSelectionUnsupported fails a read whose selection the engine did not apply: it predates it, and would stream the unselected history instead.

Functions

func BuildBootstrap

func BuildBootstrap(header BootstrapHeader, signals []BootstrapSignal) ([]byte, int64, error)

func DecodeBootstrap

func DecodeBootstrap(r io.Reader, onHeader func(BootstrapHeader) error, consume func([]byte) error) (BootstrapHeader, BootstrapTerminal, error)

DecodeBootstrap requires a header first and a terminal last, rejects trailing bytes, verifies the terminal checksum, calls onHeader before reading any logs frame, and calls consume with each logs frame's payload. EOF before the terminal is an interruption, never a successful finite response.

func DecodeLogBody

func DecodeLogBody(r io.Reader, v any) error

DecodeLogBody reads a bounded JSON request body into v.

func IsCleanMiss

func IsCleanMiss(err error) bool

IsCleanMiss reports whether the connected engine definitively has no usable archive and a caller may try another source.

func ParseSelection

func ParseSelection(signal string, q url.Values) (*SpanSelection, *LogSelection, error)

ParseSelection validates the shared archive client/server query contract.

func SanitizeTitle

func SanitizeTitle(title string) string

SanitizeTitle reduces a trace-derived title to one bounded, printable line: control and format characters (including bidi overrides) are dropped or turned into spaces, whitespace runs collapse, and long titles are cut with an ellipsis.

func VerifyClosure

func VerifyClosure(roots []string, load func(string) (*callpbv1.Call, error)) (map[string]*callpbv1.Call, error)

VerifyClosure loads the recipe closure of the anchors without evaluating any recipe, failing on a missing or cyclic dependency. The returned payloads are exactly the dependency closure of the anchors.

func WriteBootstrapFrame

func WriteBootstrapFrame(w io.Writer, kind BootstrapFrameKind, payload []byte) error

Types

type AgentRevision

type AgentRevision struct {
	Key      agentcontrol.Key
	Revision int64
}

type Bootstrap

type Bootstrap struct {
	File    string `json:"file"`
	Records int64  `json:"records"`
}

type BootstrapBatch

type BootstrapBatch struct {
	Logs *collogspb.ExportLogsServiceRequest
}

BootstrapBatch is one decoded OTLP logs batch from a bootstrap response.

type BootstrapFrameKind

type BootstrapFrameKind byte
const (
	BootstrapFrameHeader   BootstrapFrameKind = 1
	BootstrapFrameLogs     BootstrapFrameKind = 3
	BootstrapFrameTerminal BootstrapFrameKind = 4
)

func ReadBootstrapFrame

func ReadBootstrapFrame(r io.Reader) (BootstrapFrameKind, []byte, error)

type BootstrapHeader

type BootstrapHeader struct {
	Version    int        `json:"version"`
	TraceID    string     `json:"traceID"`
	SealAt     string     `json:"sealAt"`
	HighWater  HighWater  `json:"highWater"`
	Completion Completion `json:"completion"`
}

type BootstrapResult

type BootstrapResult struct {
	Header   BootstrapHeader
	Terminal BootstrapTerminal
}

BootstrapResult is the immutable cut described by a bootstrap.

type BootstrapSignal

type BootstrapSignal struct {
	Payload []byte
	Records int64
}

BootstrapSignal is one encoded OTLP logs batch of Records log records.

type BootstrapTerminal

type BootstrapTerminal struct {
	LogRecords int64  `json:"logRecords"`
	SHA256     string `json:"sha256"`
}

type CallClosureQuery

type CallClosureQuery struct {
	Digests []string `json:"digests"`
	// Known names calls whose recipe closures the caller already holds: the
	// read neither sends them nor walks into them. E.g. a rewind's adopted
	// state, which bounds the abandoned conversation the caller wants from
	// the prefix both share.
	Known []string `json:"known,omitempty"`
}

CallClosureQuery names the calls whose recipe closures to read.

func (CallClosureQuery) Validate

func (q CallClosureQuery) Validate() error

Validate checks q against the shared client/server contract.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client reads telemetry archives through a connected engine's HTTP transport.

func NewClient

func NewClient(httpClient HTTPDoer) *Client

NewClient creates an archive client for the connected engine transport.

func NewClientWithURL

func NewClientWithURL(httpClient HTTPDoer, baseURL string) (*Client, error)

NewClientWithURL creates an archive client with an explicit base URL. It is useful for HTTP proxies and tests; connected engine clients should use NewClient.

func (*Client) Bootstrap

func (c *Client) Bootstrap(ctx context.Context, traceID string, consume func(BootstrapHeader, BootstrapBatch) error) (BootstrapResult, error)

Bootstrap buffers the complete raw bootstrap and checks its header and terminal checksum before invoking consume, so a truncated or corrupt bootstrap never applies part of itself. Consumers may then apply the batches and wait for their frontend barrier, without loading unrelated historical telemetry. The engine verified the roster and recipe closure when it built the bootstrap.

func (*Client) CallClosure

func (c *Client) CallClosure(ctx context.Context, traceID string, unsealed bool, digests, known []string, consume func(*collogspb.ExportLogsServiceRequest) error) error

CallClosure reads the call payloads of digests' recipe closures, up to the archive's cut, as call-payload log records: every frame the archive holds, each once, in no particular order. It is best-effort: a frame the archive does not hold is left out, along with what only it references, so a caller still checks that the rebuild it wanted closes. The closures of known calls are left out (CallClosureQuery.Known). unsealed reads an interrupted archive, as StreamOptions.Unsealed.

func (*Client) Inspect

func (c *Client) Inspect(ctx context.Context, traceID string) (Manifest, error)

Inspect discovers a sealed archive's identity and fixed cut without downloading the bootstrap or historical telemetry. Any other state fails with an ErrState request error; see Unsealed for a best-effort read.

func (*Client) List

func (c *Client) List(ctx context.Context, opts ListOptions) (Page, error)

List returns one page of archives.

func (*Client) ListAll

func (c *Client) ListAll(ctx context.Context, opts ListOptions) ([]Manifest, error)

ListAll follows list cursors until the engine returns the final page.

func (*Client) LogQuery

func (c *Client) LogQuery(ctx context.Context, traceID string, opts StreamOptions, q LogQuery, consume func(int64, *collogspb.ExportLogsServiceRequest) error) (int64, error)

LogQuery reads the display text of the spans q names (see LogQuery) at the archive's fixed cut, opts.HighWater. opts carries no other selection. q.Follow is ignored: archives are finite. It returns the last safe resume cursor.

func (*Client) Logs

func (c *Client) Logs(ctx context.Context, traceID string, opts StreamOptions, consume func(int64, *collogspb.ExportLogsServiceRequest) error) (int64, error)

Logs reads a finite framed log stream. It returns the last safe resume cursor.

func (*Client) Metrics

func (c *Client) Metrics(ctx context.Context, traceID string, opts StreamOptions, consume func(int64, *colmetricspb.ExportMetricsServiceRequest) error) (int64, error)

Metrics reads a finite framed metric stream. It returns the last safe resume cursor.

func (*Client) SearchLogs

func (c *Client) SearchLogs(ctx context.Context, traceID string, unsealed bool, search LogSearch) (LogSearchResult, error)

SearchLogs searches an archive's display text up to its cut. unsealed reads an interrupted archive, as StreamOptions.Unsealed.

func (*Client) Traces

func (c *Client) Traces(ctx context.Context, traceID string, opts StreamOptions, consume func(int64, *coltracepb.ExportTraceServiceRequest) error) (int64, error)

Traces reads a finite framed trace stream. It returns the last safe resume cursor.

func (*Client) Unsealed

func (c *Client) Unsealed(ctx context.Context, traceID string) (UnsealedArchive, error)

Unsealed looks up an interrupted or incomplete archive for a best-effort read of what it recorded; Cut is the current end of its store. It fails with an ErrState request error for any other state, including a sealed (closed) archive.

type Completion

type Completion struct {
	Agents        []AgentRevision        `json:"agents"`
	Subscriptions []SubscriptionRevision `json:"subscriptions"`
}

Completion is the final roster the producer witnessed when the archive was sealed: the agents and subscriptions a restore installs from the bootstrap.

func Witness

func Witness(want agentcontrol.Expectation) Completion

type Config

type Config struct {
	Root        string
	TTL         time.Duration
	QuotaBytes  int64
	Now         func() time.Time
	RemoveStore func(string) (bool, error)
	// StoreSize reports the on-disk size of a main client's telemetry store.
	// It sizes archives that end unsealed, so they count toward the quota.
	StoreSize func(clientID string) (int64, error)
}

type ErrorKind

type ErrorKind string

ErrorKind groups archive failures by the recovery decision a caller should make.

const (
	ErrorCleanMiss ErrorKind = "clean_miss"
	ErrorState     ErrorKind = "state"
	ErrorCorrupt   ErrorKind = "corrupt"
	ErrorTransient ErrorKind = "transient"
)

type Failure

type Failure struct {
	Kind  FailureKind `json:"kind"`
	State State       `json:"state,omitempty"`
	Err   error       `json:"-"`
}

func (*Failure) Error

func (e *Failure) Error() string

func (*Failure) Unwrap

func (e *Failure) Unwrap() error

type FailureKind

type FailureKind string
const (
	FailureNotFound FailureKind = "not_found"
	FailureState    FailureKind = "state"
	FailureCorrupt  FailureKind = "corrupt"
	FailureIO       FailureKind = "io"
)

type FinalizeInput

type FinalizeInput struct {
	HighWater        HighWater
	SealAt           time.Time
	StoreSizeBytes   int64
	BootstrapBytes   []byte
	BootstrapRecords int64
}

FinalizeInput is the sealed cut. The caller built and verified the bootstrap for exactly this cut.

type HTTPDoer

type HTTPDoer interface {
	Do(*http.Request) (*http.Response, error)
}

HTTPDoer is the transport required by the archive client. engine/client's DirectConn implements this interface and carries the connected session's authentication and routing metadata.

type HighWater

type HighWater struct {
	Spans   int64 `json:"spans"`
	Logs    int64 `json:"logs"`
	Metrics int64 `json:"metrics"`
}

type ListOptions

type ListOptions struct {
	After          string
	Limit          int
	ExcludeTraceID string
}

ListOptions controls one archive list page. The engine also excludes the archive currently being captured for this connected session. ExcludeTraceID can omit one additional archive client-side.

type LogMatch

type LogMatch struct {
	TraceID string `json:"trace_id"`
	SpanID  string `json:"span_id"`
	// Digest is the row's dagger.io/dag.digest, for rows dagui routes to
	// the digest's creator span instead of SpanID.
	Digest string `json:"digest,omitempty"`
	// Row is the log row ID (the stream cursor of the row); Line the
	// zero-based line within its body.
	Row  int64 `json:"row"`
	Line int   `json:"line"`
	// Text is the matching line, truncated to MaxLogSearchSnippet bytes.
	Text string `json:"text"`
}

LogMatch locates one matching line.

type LogQuery

type LogQuery struct {
	Spans []LogSpan `json:"spans"`
	// Follow keeps a live read open past the store's current end until the
	// client cancels or the session's telemetry drains. A one-shot read ends
	// at the end the store had when the request arrived. Archives are
	// finite, so it is ignored there.
	Follow bool `json:"follow,omitempty"`
	// Through, when positive, ends a live one-shot read at this log row
	// instead of the store's end, so a reader catching newly selected spans
	// up to a follow's cursor reads exactly the rows that follow did not.
	// Ignored with Follow, and by archives, which end at their cut.
	Through int64 `json:"through,omitempty"`
}

LogQuery selects the display text of a set of spans. The server may return more than the selection (e.g. rows of spans whose position in the tree it does not know yet); the reader applies the frontend's exact routing.

func (LogQuery) Validate

func (q LogQuery) Validate() error

Validate checks q against the shared client/server contract.

type LogSearch

type LogSearch struct {
	Pattern string `json:"pattern"`
	// Regexp interprets Pattern as an RE2 regular expression instead of a
	// literal substring.
	Regexp bool `json:"regexp,omitempty"`
	// IgnoreCase matches case-insensitively.
	IgnoreCase bool `json:"ignore_case,omitempty"`
	// Spans limits the search; empty searches every span.
	Spans []LogSpan `json:"spans,omitempty"`
	// After resumes a search after this log row (LogSearchResult.Next).
	After int64 `json:"after,omitempty"`
	// Limit bounds the matches returned; zero means DefaultLogSearchLimit.
	Limit int `json:"limit,omitempty"`
}

LogSearch searches display text for a pattern, optionally within a set of spans (each optionally with descendants). Matching is per line.

func (LogSearch) Matcher

func (s LogSearch) Matcher() (func(string) bool, error)

Matcher compiles s into a line predicate, after validating it.

type LogSearchResult

type LogSearchResult struct {
	Matches []LogMatch `json:"matches"`
	Next    int64      `json:"next"`
	More    bool       `json:"more,omitempty"`
}

LogSearchResult is one page of search results. Next is the last row the search scanned; More reports that it stopped before the end of the store (at its match limit or its scan budget), so another search After Next continues it.

type LogSelection

type LogSelection struct {
	SpanID      string
	Descendants bool
	After       *time.Time
	Records     string
}

LogSelection selects a span's own or descendant records. Records selects ordinary output (including media), call payloads, or all historical records. Control records remain exclusive to the verified bootstrap in every mode.

type LogSpan

type LogSpan struct {
	SpanID string `json:"span_id"`
	// Descendants also selects the span's subtree, following the same
	// containment (child edges and cause links) as LogSelection.
	Descendants bool `json:"descendants,omitempty"`
	// Digests also selects rows carrying one of these dagger.io/dag.digest
	// values wherever they were emitted: dagui routes such a row (e.g. a
	// service's stdio) to the span that created the digest.
	Digests []string `json:"digests,omitempty"`
}

LogSpan names one span whose display text to read.

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

func NewManager

func NewManager(cfg Config) (*Manager, error)

func (*Manager) BeginFinalizing

func (m *Manager) BeginFinalizing(traceID string) error

func (*Manager) BootstrapPath

func (m *Manager) BootstrapPath(traceID string) string

BootstrapPath is where a sealed archive's agent bootstrap is stored.

func (*Manager) Finalize

func (m *Manager) Finalize(traceID string, in FinalizeInput) (Manifest, error)

func (*Manager) GC

func (m *Manager) GC() (overage int64, err error)

GC deletes expired archives, then the oldest ended archives (closed or unsealed) until the retained ones fit the quota, telemetry-only archives before agent archives. The last ended archive in that order is kept even when it alone exceeds the quota; overage reports by how much. An archive whose store is still open is left in place and retried by the next GC.

func (*Manager) KeepSet

func (m *Manager) KeepSet() map[string]bool

func (*Manager) List

func (m *Manager) List(after, excludeTraceID string, limit int) Page

List pages through agent archives, the sessions `dagger agent -r` can resume. Telemetry-only archives are left out: they have no agent state.

func (*Manager) Manifest

func (m *Manager) Manifest(traceID string) (Manifest, error)

func (*Manager) MarkIncomplete

func (m *Manager) MarkIncomplete(traceID string, cause error) error

func (*Manager) Register

func (m *Manager) Register(traceID, mainClientID string) (Manifest, error)

Register creates the active archive for an agent session's trace. The first session to register a trace owns its archive; a later registration for the same trace (e.g. a nested session that inherited TRACEPARENT) fails. The one exception is an ended telemetry-only archive: agent state outranks a plain session's telemetry, so an agent session sharing its trace takes it over.

func (*Manager) RegisterTelemetryOnly

func (m *Manager) RegisterTelemetryOnly(traceID, mainClientID string, startedAt time.Time) (Manifest, error)

RegisterTelemetryOnly creates the active archive for a session that recorded no agent state, retaining its telemetry by trace ID. Sessions sharing a trace (a nested dagger-in-dagger session inherits its parent's) each hold only their own telemetry, so the earliest-started, outermost one keeps the trace: it supersedes an ended telemetry-only archive of a session that started after it. It never displaces any other archive.

func (*Manager) SetTitle

func (m *Manager) SetTitle(traceID, title string) error

SetTitle records the session title the main client published into an active archive's trace. It is persisted immediately, so an archive recovered after an engine crash keeps it, and the sealed manifest inherits it. The title is sanitized first; an unchanged or empty title writes nothing.

type Manifest

type Manifest struct {
	Version      int    `json:"version"`
	TraceID      string `json:"traceID"`
	MainClientID string `json:"mainClientID"`
	// TelemetryOnly marks the archive of a session that recorded no agent
	// state: it retains the session's telemetry for display, its bootstrap
	// has an empty roster, and it is never offered for agent restore.
	TelemetryOnly bool       `json:"telemetryOnly,omitempty"`
	State         State      `json:"state"`
	Title         string     `json:"title,omitempty"`
	StartedAt     time.Time  `json:"startedAt"`
	ClosedAt      *time.Time `json:"closedAt,omitempty"`
	ExpiresAt     time.Time  `json:"expiresAt"`
	SealAt        *time.Time `json:"sealAt,omitempty"`
	SizeBytes     int64      `json:"sizeBytes"`
	Bootstrap     Bootstrap  `json:"bootstrap,omitempty"`
	HighWater     HighWater  `json:"highWater"`
	Failure       string     `json:"failure,omitempty"`
}

Manifest describes one archive. An archive is identified by its trace ID alone: the first session to register a trace owns its archive, except that an ended telemetry-only archive yields to an agent session, or to a plain session that started earlier (see RegisterTelemetryOnly).

type MetricSelection

type MetricSelection struct {
	Names []string
}

MetricSelection narrows a metric stream to the metrics with these names: e.g. the LLM token counts a cost total sums, without every exec's resource metrics. An engine that predates it ignores it and streams every metric, so the response says whether it applied one (SelectionHeader), and the client fails the read with ErrSelectionUnsupported before consuming any of it.

func ParseMetricSelection

func ParseMetricSelection(signal string, q url.Values) (*MetricSelection, error)

ParseMetricSelection validates a metric stream's selection: nil when the query selects none.

type Page

type Page struct {
	Archives []Manifest `json:"archives"`
	Next     string     `json:"next,omitempty"`
}

type RequestError

type RequestError struct {
	Kind       ErrorKind
	Failure    FailureKind
	State      State
	StatusCode int
	Err        error
}

RequestError is a typed archive transport or protocol failure.

func (*RequestError) Error

func (e *RequestError) Error() string

func (*RequestError) Is

func (e *RequestError) Is(target error) bool

func (*RequestError) Unwrap

func (e *RequestError) Unwrap() error

type SpanSelection

type SpanSelection struct {
	// Full bypasses priority filtering while retaining optional UI annotations.
	Full   bool
	NoRoot bool
	Listen []string
	// Conversation narrows the root view's priority spans to agent
	// conversations (loop and message spans, and each tool call's execution
	// span, which holds its result), leaving out checks, tests, revealed and
	// failed spans. An engine that predates it ignores it and serves the
	// broader priority view.
	Conversation bool
	DagUIView    bool
}

SpanSelection selects the priority view and/or listened subtrees of a fixed archive cut. Nil in StreamOptions preserves the unfiltered history stream. Unlike Cloud, archives are finite: no time-based follow/backfill is needed.

type State

type State string
const (
	StateActive      State = "active"
	StateFinalizing  State = "finalizing"
	StateClosed      State = "closed"
	StateInterrupted State = "interrupted"
	StateIncomplete  State = "incomplete"
)

func (State) Unsealed

func (s State) Unsealed() bool

Unsealed reports whether an archive's session ended without a verified seal: the engine stopped before finalizing it, or finalization failed.

type StreamOptions

type StreamOptions struct {
	Cursor    int64
	HighWater int64
	Unsealed  bool
	// Selections are independent reads; resume a cursor only with the same selection.
	Spans   *SpanSelection
	Logs    *LogSelection
	Metrics *MetricSelection
}

StreamOptions fixes one finite signal read to a high-water cursor. Cursor is the last batch successfully acknowledged by the caller. Unsealed reads an interrupted or incomplete archive at the cut returned by Unsealed; its log stream then includes agent control records.

func (StreamOptions) SelectionQuery

func (s StreamOptions) SelectionQuery(signal string) (url.Values, error)

type SubscriptionRevision

type SubscriptionRevision struct {
	Key      agentcontrol.EdgeKey
	Revision int64
}

type UnsealedArchive

type UnsealedArchive struct {
	Cut HighWater
}

UnsealedArchive is an archive whose session ended without a seal. It has no bootstrap; stream it with StreamOptions.Unsealed at Cut.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL