Documentation
¶
Overview ¶
Package spanembed embeds the span projection for semantic search.
Embeddings are a derived-side concern: the derive/worker family is the single writer (the ingest hot path never embeds), and every embedding is keyed by deterministic span identity (org_id, trace_id, span_id) so the pass is idempotent — re-deriving or re-running embeds each span exactly once, skipping identities whose content, model, and dimensions are already current.
Only call_kind="main" llm spans embed. Shadow calls (permission checks, title generation) are the plurality of llm calls in a harness session and poison search relevance; tool and event spans carry payloads better served by structured queries. The embedded text is the span's delta-only content — the fresh input blocks plus the response blocks rendered to text — never the re-sent conversation history, which is what keeps selective embedding cheap.
Index ¶
- Constants
- Variables
- func ContentHash(text string) string
- func RenderSpanText(input, output json.RawMessage) string
- type Candidate
- type Hit
- type Key
- type Pass
- type PassConfig
- type Record
- type Report
- type Sink
- type Source
- type Store
- func (s *Store) EnsureSchema(ctx context.Context) error
- func (s *Store) ListCandidates(ctx context.Context, after Key, limit int) ([]Candidate, error)
- func (s *Store) PruneOrphans(ctx context.Context) (int64, error)
- func (s *Store) Search(ctx context.Context, orgID string, embedding []float32, topK int) ([]Hit, error)
- func (s *Store) Upsert(ctx context.Context, rec Record) error
- type StoreConfig
Constants ¶
const DefaultBatchSize = 100
DefaultBatchSize bounds one candidate page.
const ( // DefaultTableName is the span-embedding table. It lives in the // same database as the versioned spans projection — candidate selection // and search both join against the current physical table family. DefaultTableName = "span_embeddings" )
Variables ¶
var ErrNotInitialized = errors.New("span embeddings not initialized: run the embed pass (tapes dev embed-spans or tapes serve derive-worker --embed-spans)")
ErrNotInitialized is returned by reads when the embedding table does not exist yet — i.e. no embed pass has ever run against this store.
Functions ¶
func ContentHash ¶
ContentHash returns the hex sha256 of the rendered span text — the change detector that makes re-derives cheap: unchanged content under an unchanged model never re-embeds.
func RenderSpanText ¶
func RenderSpanText(input, output json.RawMessage) string
RenderSpanText renders a span's stored delta-only content to the text that gets embedded: the text blocks of the fresh input followed by the text blocks of the output. Tool payloads, thinking, and images are deliberately excluded — they are structured data, not prose, and they drown the signal the search exists for.
Types ¶
type Candidate ¶
type Candidate struct {
OrgID string
TraceID string
SpanID string
SessionID string // "" when the span derived without attribution
// Input and Output are the span's stored delta-only content-block
// arrays (JSONB), nil when empty.
Input json.RawMessage
Output json.RawMessage
// ExistingHash and ExistingModel describe the current embedding
// row; both empty when the span has never been embedded.
ExistingHash string
ExistingModel string
}
Candidate is one main llm span considered for embedding, joined with its existing embedding row (zero values when not yet embedded).
type Hit ¶
type Hit struct {
TraceID string
SpanID string
SessionID string
Score float32
UserPrompt string
Snippet string
Model string
StartedAt time.Time
}
Hit is one similarity-search result with its trace/turn context.
type Pass ¶
type Pass struct {
// contains filtered or unexported fields
}
Pass walks every eligible span and embeds the ones whose embedding is missing or stale. Idempotent by construction: span identity keys the writes and a content hash gates them, so running the pass twice (or concurrently with a re-derive) embeds each span at most once per content+model.
func NewPass ¶
func NewPass(src Source, sink Sink, embedder embeddings.Embedder, cfg PassConfig, log *slog.Logger) (*Pass, error)
NewPass creates an embed pass.
func (*Pass) Run ¶
Run executes one full pass: prune orphaned embeddings, then page through every main llm span and embed the missing/stale ones.
Error discipline: a per-span embed or write failure is counted and logged but never aborts the pass — the span stays un-embedded and the next run retries it. Only infrastructure failures (candidate listing) abort, since they would starve every remaining page.
type PassConfig ¶
type PassConfig struct {
// Model names the embedding model; stored per row so a model
// switch re-embeds existing spans.
Model string
// Dimensions is the expected embedding dimensionality. The first
// vector the model returns is checked against it, so a
// model/dims misconfiguration aborts the pass with one clear
// error instead of failing every row's insert against the sized
// vector column.
Dimensions uint
// BatchSize bounds one candidate page (default DefaultBatchSize).
BatchSize int
}
PassConfig configures one embed pass.
type Record ¶
type Record struct {
OrgID string
TraceID string
SpanID string
SessionID string
Model string
ContentHash string
Embedding []float32
}
Record is one embedding write.
type Report ¶
type Report struct {
// Scanned counts every candidate span considered.
Scanned int `json:"scanned"`
// Embedded counts spans embedded this pass (new or re-embedded
// after a content/model change).
Embedded int `json:"embedded"`
// UpToDate counts spans skipped because their embedding already
// matches the current content and model.
UpToDate int `json:"up_to_date"`
// Empty counts spans skipped because their delta content renders
// to no text at all (e.g. a pure tool-call response).
Empty int `json:"empty"`
// Failed counts spans whose embed or write errored; the pass
// continues past them and the next run retries.
Failed int `json:"failed"`
// Pruned counts orphaned embedding rows removed (their span was
// pruned or reclassified by a re-derive).
Pruned int64 `json:"pruned"`
}
Report summarizes one embed pass.
type Sink ¶
type Sink interface {
Upsert(ctx context.Context, rec Record) error
PruneOrphans(ctx context.Context) (int64, error)
}
Sink persists embeddings. *Store implements it; tests substitute a fake.
type Source ¶
type Source interface {
ListCandidates(ctx context.Context, after Key, limit int) ([]Candidate, error)
}
Source lists embed candidates. *Store implements it; tests substitute a fake.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store reads and writes span embeddings in the tapes Postgres database. The write path (EnsureSchema, Upsert, PruneOrphans) belongs to the derive-side embed pass; the read path (Search) backs the API's span search.
func NewStore ¶
NewStore wraps an existing connection pool. It performs no IO: writers must call EnsureSchema before upserting, readers may query immediately and receive ErrNotInitialized until a writer has run.
func (*Store) EnsureSchema ¶
EnsureSchema creates the embedding table and its HNSW index sized to the configured dimensions, or fail-fasts when an existing table was created with different dimensions. pgvector cannot resize a column in place, so a dims change requires re-embedding into a new (or dropped) table — the error says so instead of letting every subsequent insert fail.
func (*Store) ListCandidates ¶
ListCandidates pages every main llm span (keyset-ordered) joined with its current embedding row. The caller decides per candidate whether an embed is due — content hashing happens in Go, so the query stays a cheap indexable join.
func (*Store) PruneOrphans ¶
PruneOrphans removes embeddings whose span no longer exists as a main llm span — pruned by a re-derive, or reclassified out of the embeddable set.
func (*Store) Search ¶
func (s *Store) Search(ctx context.Context, orgID string, embedding []float32, topK int) ([]Hit, error)
Search returns the topK spans most similar to the query embedding, joined with their trace context (turn prompt and span payloads for the snippet). Scoped to one org — search is a tenant-facing read.
type StoreConfig ¶
type StoreConfig struct {
// TableName defaults to DefaultTableName.
TableName string
// Dimensions is the embedding dimensionality. The table's vector
// column is created with exactly this size and EnsureSchema
// fail-fasts when an existing table disagrees — model and dims are
// a deliberate, explicit pairing (e.g. text-embedding-3-large@1024
// in cloud, embeddinggemma@768 on a local/dev deployment).
Dimensions uint
// OrgID optionally scopes candidate listing and orphan pruning to
// one tenant. Empty embeds every org.
OrgID string
}
StoreConfig configures a span-embedding store.
Source Files