ingest

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 26, 2026 License: AGPL-3.0 Imports: 12 Imported by: 0

Documentation

Overview

Package ingest provides an HTTP server that accepts completed LLM conversation turns for storage in the Merkle DAG. This enables "sidecar mode" where an external gateway (e.g., Envoy AI Gateway) handles upstream LLM traffic and tapes only stores, embeds, and publishes the data.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BatchPayload

type BatchPayload struct {
	Turns []TurnPayload `json:"turns"`
}

BatchPayload is the ingest request body for multiple conversation turns.

type BatchResult

type BatchResult struct {
	Accepted int      `json:"accepted"`
	Rejected int      `json:"rejected"`
	Errors   []string `json:"errors,omitempty"`
}

BatchResult reports the outcome of a batch ingest.

type Config

type Config struct {
	// ListenAddr is the address to listen on (e.g., ":8082")
	ListenAddr string

	// VectorDriver is an optional vector store for storing embeddings.
	// If nil, vector storage is disabled.
	VectorDriver vector.Driver

	// Embedder is an optional embedder for generating embeddings.
	// Required if VectorDriver is set.
	Embedder embeddings.Embedder

	// Publisher is an optional event publisher for new DAG nodes.
	// If nil, publishing is disabled.
	Publisher publisher.Publisher

	// Project is the git repository or project name to tag on stored nodes.
	Project string
}

Config is the ingest server configuration.

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is an HTTP server that accepts completed LLM conversation turns for async storage in the Merkle DAG.

func New

func New(config Config, driver storage.Driver, log *slog.Logger) (*Server, error)

New creates a new ingest Server.

func (*Server) Close

func (s *Server) Close() error

Close gracefully shuts down the server and waits for the worker pool to drain.

func (*Server) Run

func (s *Server) Run() error

Run starts the ingest server on the configured address.

func (*Server) RunWithListener

func (s *Server) RunWithListener(listener net.Listener) error

RunWithListener starts the ingest server using the provided listener.

type TurnPayload

type TurnPayload struct {
	// Provider type: "openai", "anthropic", "ollama"
	Provider string `json:"provider"`

	// AgentName optionally tags the turn (same as X-Tapes-Agent-Name header)
	AgentName string `json:"agent_name,omitempty"`

	// RawRequest is the original request body sent to the LLM provider
	RawRequest json.RawMessage `json:"request"`

	// RawResponse is the complete response body from the LLM provider
	RawResponse json.RawMessage `json:"response"`
}

TurnPayload is the ingest request body for a single completed conversation turn. It carries the raw provider request and response so tapes can parse, store, and embed them exactly as the transparent proxy would.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL