Documentation
¶
Overview ¶
Package ingest provides an HTTP server that accepts completed LLM conversation turns for storage in the Merkle DAG. This enables "sidecar mode" where an external gateway (e.g., Envoy AI Gateway) handles upstream LLM traffic and tapes only stores and publishes the data. Embeddings are written downstream by the derive worker family (pkg/spanembed), never at ingest time.
Index ¶
Constants ¶
const ( // HeaderPaperAuthOrgID carries the verified org claim. HeaderPaperAuthOrgID = "x-paper-auth-org-id" // HeaderPaperAuthSubject carries the verified `sub` claim. HeaderPaperAuthSubject = "x-paper-auth-subject" )
Server-trusted identity headers, populated by the upstream gateway from validated JWT claims. This is the same contract the wire-capture path consumes (tapes-extproc's internal/headers package reads the identical names into the session envelope): clients are not permitted to send these themselves, and the gateway is responsible for stripping inbound values so only edge-verified identity reaches the handler. When the gateway is not configured to populate them, the headers are absent and the payload envelope's own identity fields stand.
Variables ¶
var ( // ErrEnvelope means the POST body could not be decoded as a TurnPayload. // Returned as 400 Bad Request. ErrEnvelope = errors.New("invalid envelope") // ErrUnprocessable covers validation / parse failures inside a well-formed // envelope: unknown provider, unparseable provider-specific request / // response body, etc. Returned as 422 Unprocessable Entity. ErrUnprocessable = errors.New("unprocessable turn") // ErrDownstream covers failures that originate below the handler: worker // pool saturation, DAG write errors, storage unavailability. Returned as // 502 Bad Gateway. ErrDownstream = errors.New("downstream failure") )
Ingest error classes. Each maps to a distinct HTTP status so operators can tell malformed envelopes from unknown providers from downstream outages without tailing logs.
Functions ¶
This section is empty.
Types ¶
type BatchPayload ¶
type BatchPayload struct {
Turns []TurnPayload `json:"turns"`
}
BatchPayload is the ingest request body for multiple conversation turns.
type BatchResult ¶
type BatchResult struct {
Accepted int `json:"accepted"`
Rejected int `json:"rejected"`
Errors []string `json:"errors,omitempty"`
}
BatchResult reports the outcome of a batch ingest.
type Config ¶
type Config struct {
// ListenAddr is the address to listen on (e.g., ":8082")
ListenAddr string
// Publisher is an optional event publisher for new DAG nodes.
// If nil, publishing is disabled.
Publisher publisher.Publisher
// Project is the git repository or project name to tag on stored nodes.
Project string
}
Config is the ingest server configuration.
type Metrics ¶ added in v0.5.2
type Metrics struct {
// contains filtered or unexported fields
}
Metrics enumerates the Prometheus counters and histograms emitted by the ingest server. Metric names are fixed so dashboards and alerts reference stable identifiers.
func NewMetrics ¶ added in v0.5.2
func NewMetrics() *Metrics
NewMetrics builds a fresh registry and registers the ingest metric set on it. Each Server owns its own registry so tests don't leak counters across suite runs (the default prometheus registry is global state).
func (*Metrics) Handler ¶ added in v0.5.2
Handler returns an http.Handler that serves the Prometheus scrape endpoint backed by this Metrics' registry.
func (*Metrics) ObserveDAGLatency ¶ added in v0.5.2
ObserveDAGLatency records how long it took to enqueue a turn into the worker pool. Latency is a cheap proxy for back-pressure so we graph it even though enqueue is nominally O(1) — a slow enqueue hints at queue saturation.
func (*Metrics) ObserveWrite ¶ added in v0.5.2
ObserveWrite increments the writes counter for a given provider/result. A zero-length provider label becomes "unknown" so scrapes don't drop rows.
func (*Metrics) Registry ¶ added in v0.5.2
func (m *Metrics) Registry() *prometheus.Registry
Registry exposes the backing *prometheus.Registry so callers can mount a scrape handler or assert on the metric state in tests.
func (*Metrics) SetQueueDepth ¶ added in v0.5.2
SetQueueDepth updates the worker queue depth gauge.
type NodeListResponse ¶ added in v0.5.2
type NodeListResponse struct {
Count int `json:"count"`
Nodes []NodeSummary `json:"nodes"`
}
NodeListResponse wraps a slice of NodeSummary in a count+items envelope so the shape is consistent with the rest of the tapes query surface.
type NodeSummary ¶ added in v0.5.2
type NodeSummary struct {
Hash string `json:"hash"`
Role string `json:"role"`
Provider string `json:"provider"`
Model string `json:"model"`
AgentName string `json:"agent_name,omitempty"`
}
NodeSummary is the per-item shape returned by GET /v1/ingest/nodes. Intentionally minimal — operators and canaries only need to confirm a row landed for a given agent; rich querying is served by the tapes-api.
type Result ¶ added in v0.5.2
type Result string
Result enumerates the status-label values emitted on the writes counter. Closed enumeration keeps dashboards safe against label typos.
const ( ResultAccepted Result = "accepted" ResultRejectEnv Result = "reject_envelope" ResultRejectParse Result = "reject_parse" ResultUnknownProv Result = "unknown_provider" ResultQueueFull Result = "queue_full" ResultDownstreamErr Result = "downstream_error" // ResultInternalErr covers a failure inside the handler itself (e.g. a // server-side marshal that should never fail) — a 500, distinct from a bad // payload or a downstream outage — so no handler exit is invisible to the // writes counter. ResultInternalErr Result = "internal_error" )
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server is an HTTP server that accepts completed LLM conversation turns for async storage in the Merkle DAG.
func (*Server) Close ¶
Close gracefully shuts down the server and waits for the worker pool to drain.
func (*Server) Metrics ¶ added in v0.5.2
Metrics exposes the ingest metrics so tests and health checks can scrape the registry programmatically.
type TranscriptPayload ¶ added in v0.16.0
type TranscriptPayload struct {
// Session identifies the harness session the transcript belongs to.
Session *sessions.IngestEnvelope `json:"session"`
// AgentID is empty for the main transcript, or the subagent id for
// subagents/agent-<id>.jsonl files.
AgentID string `json:"agent_id,omitempty"`
// AgentType / Description / ToolUseID mirror the harness's
// subagent meta.json: ToolUseID is the Task tool_use that forked
// this agent — the causal fork edge the deriver attaches.
AgentType string `json:"agent_type,omitempty"`
Description string `json:"description,omitempty"`
ToolUseID string `json:"tool_use_id,omitempty"`
// Records is the transcript's JSONL content as a JSON array,
// verbatim.
Records json.RawMessage `json:"records"`
}
TranscriptPayload is the ingest body for one harness transcript file — the main session transcript or one subagent's. The records land in the immutable raw layer verbatim (source: transcript); the deriver reconciles them against the wire capture to recover the causal/fork skeleton. No node-path processing happens at ingest time.
type TurnMeta ¶ added in v0.16.0
type TurnMeta struct {
RequestID string `json:"request_id,omitempty"`
ContentType string `json:"content_type,omitempty"`
// ThreadID is the harness sub-thread id resolved by the capture
// adapter (extproc headers.ThreadID); "" for main-thread calls.
ThreadID string `json:"thread_id,omitempty"`
Method string `json:"method,omitempty"`
Path string `json:"path,omitempty"`
Endpoint string `json:"endpoint,omitempty"`
Model string `json:"model,omitempty"`
ModelFamily string `json:"model_family,omitempty"`
Stream string `json:"stream,omitempty"`
ContentEncoding string `json:"content_encoding,omitempty"`
UpstreamStatus int `json:"upstream_status,omitempty"`
UpstreamStatusClass string `json:"upstream_status_class,omitempty"`
RequestBytes int `json:"request_bytes,omitempty"`
ResponseBytes int `json:"response_bytes,omitempty"`
ElapsedSeconds float64 `json:"elapsed_seconds,omitempty"`
}
TurnMeta mirrors the capture adapter's meta block (tapes-extproc TurnMeta). Every field is optional; adapters that predate a field simply omit it. Ingest only reads RequestID directly (raw-turn dedup) — the rest ride along verbatim in the raw layer and become queryable post-derive.
type TurnPayload ¶
type TurnPayload struct {
// Provider type: "openai", "anthropic", "ollama"
Provider string `json:"provider"`
// AgentName optionally tags the turn (same as X-Tapes-Agent-Name header)
AgentName string `json:"agent_name,omitempty"`
// RawRequest is the original request body sent to the LLM provider.
RawRequest json.RawMessage `json:"request"`
// Response is the already reduced, provider-agnostic response for the turn.
Response llm.ChatResponse `json:"response"`
// Meta is the capture adapter's metadata block. Parsed for the
// fields ingest promotes (request_id for raw-turn dedup); the
// verbatim JSON is persisted alongside the raw turn so fields
// unknown to this build survive.
Meta TurnMeta `json:"meta"`
// Session is the optional session-tracking envelope. When present,
// ingest UPSERTs a `sessions` row keyed by
// (org_id, harness_id, harness_session_id), resolves the
// parent_session_id FK (placeholder-inserting when needed), and
// rolls up turn counters — all in the same transaction as the
// nodes insert. When absent, ingest treats the turn as
// harness_id="unknown" and derives a synthetic harness_session_id
// from the captured turn's Merkle root prefix.
//
// The type lives in pkg/sessions to avoid an import cycle
// (proxy/worker depends on it too).
Session *sessions.IngestEnvelope `json:"session,omitempty"`
}
TurnPayload is the ingest request body for a single completed conversation turn. It carries the raw provider request plus an already-reduced response. Capture adapters such as tapes-extproc own protocol-specific stream reduction; ingest owns request parsing, validation, and durable storage.