callrecord

package
v1.126.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 26, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package callrecord is the catalog of what the platform's data calls actually did: every SQL query and API invocation as a record with a purpose, a target, and a fate (issue #1321).

A record is not an audit row. The audit log answers "who called what, when", keeps its rows for a fixed retention window, and may drop or redact the arguments a call carried. A record answers "is this query worth running again", and its statement is the whole point of keeping it. The two are joined by event_id, which is the id a call already hands back to its caller as mcp:call:<event_id> (#1320).

Outcome is derived, never stored

A record's outcome is computed on read from three facts that live elsewhere: whether the call itself failed, whether anything later cited it as a source, and whether the same session ran a better version of it afterwards. Storing the outcome would mean recomputing it every time an asset is saved, an insight is captured, or a query is re-run — and being wrong in between. Deriving it means a record's fate is always the current answer, and that the rule can be read in one place (the SQL in postgres.go) rather than reconstructed from the write paths that would have maintained it.

Index

Constants

View Source
const (
	// KindSQL is a query against a query engine (trino_query, trino_execute,
	// trino_export).
	KindSQL = "sql"
	// KindAPI is an HTTP invocation through the API gateway
	// (api_invoke_endpoint, api_export).
	KindAPI = "api"
)

Call kinds. One record shape serves both; each kind fills the columns it has and leaves the other kind's empty.

View Source
const (
	// OutcomeFailed means the call itself returned an error.
	OutcomeFailed = "failed"
	// OutcomeSatisfied means an asset, an export, or a captured insight named
	// this call as a source. It is the only outcome that says the call
	// answered something.
	OutcomeSatisfied = "satisfied"
	// OutcomeSuperseded means a later successful call in the same session
	// addressed the same targets over the same connection, and nothing was
	// ever built from this one: a draft the agent corrected.
	OutcomeSuperseded = "superseded"
	// OutcomeRan means the call succeeded and nothing has come of it yet.
	OutcomeRan = "ran"
)

Outcomes, in the order they are decided. A call that failed is failed whatever else happened; a call something was built from is satisfied even if the session later ran a better one; a call replaced by a later one over the same targets is superseded; anything else simply ran.

View Source
const (
	// SatisfiedByAsset means a saved asset cites the call.
	SatisfiedByAsset = "asset"
	// SatisfiedByExport means an export (trino_export, api_export) cites it.
	SatisfiedByExport = "export"
	// SatisfiedByCapture means a memory_capture insight names the call's
	// mcp:call:<event_id> reference in its sources.
	SatisfiedByCapture = "capture"
)

How a record came to be satisfied. The reviewer sees this at promotion because the routes differ in what they cost the agent: an asset or an export is a by-product of work the agent was doing anyway, while a capture is the agent stating, in its own words and at the price of writing a description, that this query answered the question.

View Source
const (
	// DefaultPerPage is the page size when the caller states none.
	DefaultPerPage = 25
	// MaxPerPage caps a caller-stated page size. Each row derives its outcome
	// from what cites it, so an uncapped page is an uncapped query.
	MaxPerPage = 200
)
View Source
const (
	// DefaultRetentionDays is how long a call that came to nothing is kept.
	// It is deliberately shorter than the year a script run is kept: a run is
	// a scheduled automation's refresh history, which people read, while an
	// unused query is a draft.
	DefaultRetentionDays = 90
)

Variables

View Source
var ErrNoPromotionTarget = errors.New("no promotion target is configured for this kind of record")

ErrNoPromotionTarget is returned when the platform has nowhere to promote a record to: no DataHub connection for a query, no endpoint identity for an API call. It is a configuration answer, not a validation failure, and the caller reports it as such rather than telling the reviewer they did something wrong.

View Source
var ErrNotFound = errors.New("call record not found")

ErrNotFound is returned when no record matches the scope. A record belonging to another caller and a record that never existed produce the same error, so a caller cannot probe for the existence of someone else's query.

View Source
var ErrNotPromotable = errors.New("only a satisfied record that has not already been promoted or rejected can be promoted")

ErrNotPromotable is returned when a record is not in a state a reviewer can act on: it never answered anything, or it has already been promoted or declined.

Outcomes lists every outcome a record can read, for validating a filter.

Functions

func APITarget

func APITarget(connection, method, path, operationID string, pathParams map[string]string) string

APITarget names the resource an API call addressed: its operation id with every path parameter resolved into it, or the request line when the caller addressed the endpoint by path directly. It is scoped by connection for the same reason a dataset URN is scoped by platform: the same operation id against two upstreams is two endpoints.

The path parameters are part of the target, not decoration. A target is what decides whether two calls addressed the same thing, so an operation id alone would make the same mutation against two different resources one target, and every call through a generic dispatch endpoint one target (#1352).

A target that cannot distinguish the call is not returned at all. An operation id holding a placeholder no path parameter resolved names a template rather than a resource, and the empty target it yields is the same answer the SQL side gives when it cannot tell what a statement read: not comparable, so never declared superseded and never credited as reuse.

func ClampPerPage

func ClampPerPage(limit int) int

ClampPerPage bounds a requested page size into [1, MaxPerPage].

func IndexText

func IndexText(rec Record) string

IndexText is what a record is searched and embedded by: the sentence its caller wrote about why, and what the call actually did. Both are needed — a purpose alone does not say which table, and a statement alone does not say what question it answers.

It is deliberately the same corpus the lexical index covers (ftsExpr, and the GIN index migration 000107 builds on it). The two arms of a search must agree about what a record says, or a record found by its words would rank against a vector computed from something else. The targets are left out for that reason and because a statement already names the tables it reads.

func KindForTool

func KindForTool(tool string) string

KindForTool returns the record kind a tool produces, or "" when the tool is not one the catalog records.

func NewRecorder

func NewRecorder(inner audit.Logger, store Store, urn URNBuilder) audit.Logger

NewRecorder wraps an audit store so every data-access call it records is also cataloged. A nil store returns the audit logger unchanged, which is what a deployment without the catalog gets.

func NormalizeStatement

func NormalizeStatement(s string) string

NormalizeStatement collapses a statement to the form reuse matching compares: whitespace runs become single spaces, and the whole is trimmed and lowercased. Two agents that indent the same query differently have run the same query.

func PromotedDescription

func PromotedDescription(rec Record) string

PromotedDescription is what the catalog entry says about itself: the purpose, and where the call came from. The session is named because a promoted query is evidence, and evidence is worth being able to walk back to.

func PromotedName

func PromotedName(rec Record) string

PromotedName is what the promoted record is called in the catalog: the purpose its caller stated, which is the only human sentence a call ever carries. A record with no purpose falls back to what it addressed, so an entry is never nameless.

func RetentionDays

func RetentionDays(configured int) int

RetentionDays resolves the configured retention, applying the default when unset. Zero or negative takes the default, matching every other retention this platform configures.

func ValidOutcome

func ValidOutcome(s string) bool

ValidOutcome reports whether s names an outcome. Used to drop an unknown facet rather than pass it into the query as a value nothing matches.

Types

type Artifact

type Artifact struct {
	// Kind is one of the SatisfiedBy values.
	Kind string `json:"kind" example:"asset"`
	ID   string `json:"id" example:"ast_7c1e"`
	Name string `json:"name" example:"Q3 revenue by region"`
}

Artifact is one thing built from a call: an asset, an export, or a captured insight that named the call as a source.

type Config

type Config struct {
	// RetentionDays bounds how long an unused record is kept. Zero or
	// negative takes the default.
	RetentionDays int
}

Config is what a deployment chooses about the catalog: how long a call that came to nothing is kept. Everything else about a record is derived.

type CuratedQueryWriter

type CuratedQueryWriter interface {
	CreateCuratedQuery(ctx context.Context, datasetURNs []string, name, sql, description string) (string, error)
}

CuratedQueryWriter is the DataHub write path a promoted query takes. It is the platform's existing curated-query write (pkg/toolkits/knowledge's DataHubWriter satisfies it), narrowed to the one method promotion needs, so a promoted record and an apply_knowledge proposal reach DataHub the same way rather than through two write paths that could drift.

type Example

type Example struct {
	Connection  string
	OperationID string
	Method      string
	Path        string
	Name        string
	Description string
	// CallRecordID is the record the example was promoted from, so the
	// endpoint can lead back to the call that produced it.
	CallRecordID string
	CreatedBy    string
}

Example is one endpoint invocation worth keeping.

type ExampleWriter

type ExampleWriter interface {
	SaveExample(ctx context.Context, ex Example) (string, error)
}

ExampleWriter saves a promoted API call as an example on its endpoint, so the next agent reading that endpoint's schema sees a request that is known to have worked. It is the API catalog's counterpart to a DataHub Query entity: an endpoint has no catalog entity of its own to attach a query to.

type Fetcher

type Fetcher struct {
	SessionID string
	UserID    string
}

Fetcher identifies the session dereferencing a record. Reuse is credited to a session, not to a person: the question a reuse count answers is how many separate pieces of work this query was found and used by.

type Filter

type Filter struct {
	// UserID restricts the list to one caller's records. Empty is
	// unrestricted, which only the operator surface may ask for.
	UserID string
	// Kind, Connection and Outcome are exact-match facets.
	Kind       string
	Connection string
	Outcome    string
	// Target keeps records addressing one dataset URN.
	Target string
	// SessionID keeps the calls one session made.
	SessionID string
	// EventIDs keeps the records of named audit events. It is how a caller
	// that already holds a page of events — a session's timeline — reads the
	// records for exactly those events, rather than reading a session's whole
	// history and discarding most of it. Empty states no restriction.
	EventIDs []string
	// Search matches the purpose and the statement.
	Search string
	// PromotableOnly keeps the records a reviewer can act on: satisfied, not
	// yet promoted, not rejected. It is the review queue.
	PromotableOnly bool
	Limit          int
	Offset         int
}

Filter selects records for a list. UserID is deliberately not read from any query string (see FilterFromQuery); each surface assigns it itself.

func FilterFromQuery

func FilterFromQuery(q url.Values) Filter

FilterFromQuery reads the query string into a call filter, leaving UserID for the caller to set. An unparseable or unknown value is treated as absent rather than as an error: the failure mode of a bad filter is an unfiltered or empty page, never a 400 the UI has to model.

type PostgresStore

type PostgresStore struct {
	// contains filtered or unexported fields
}

PostgresStore is the call catalog over PostgreSQL. It also owns the sweep that keeps the catalog from growing without bound; see retention.go.

func NewPostgresStore

func NewPostgresStore(db *sql.DB, cfg Config) *PostgresStore

NewPostgresStore returns a call catalog over db.

func (*PostgresStore) Cleanup

func (s *PostgresStore) Cleanup(ctx context.Context) (int64, error)

Cleanup removes the records older than the retention window that nothing came of, and reports how many it removed.

func (*PostgresStore) Close

func (s *PostgresStore) Close() error

Close stops the sweeper and waits for the tick in flight. It does not close the database handle, which the layer that opened it owns.

func (*PostgresStore) Count

func (s *PostgresStore) Count(ctx context.Context, f Filter) (int, error)

Count returns how many records match the filter, ignoring its paging.

func (*PostgresStore) CreditReuse

func (s *PostgresStore) CreditReuse(ctx context.Context, r Record) (int, error)

CreditReuse credits the records this call re-ran, and reports how many.

func (*PostgresStore) ForTargets

func (s *PostgresStore) ForTargets(ctx context.Context, urns []string, userID string, limit int) ([]Record, error)

ForTargets returns satisfied records addressing any of the given datasets, most reused first.

func (*PostgresStore) Get

func (s *PostgresStore) Get(ctx context.Context, scope Scope) (*Record, error)

Get returns one record with its artifacts, or ErrNotFound when the scope admits none.

func (*PostgresStore) GetByEventID

func (s *PostgresStore) GetByEventID(ctx context.Context, eventID, userID string) (*Record, error)

GetByEventID resolves an mcp:call:<event_id> reference, scoped the same way Get is.

func (*PostgresStore) Insert

func (s *PostgresStore) Insert(ctx context.Context, r Record) error

Insert records one call.

func (*PostgresStore) List

func (s *PostgresStore) List(ctx context.Context, f Filter) ([]Record, error)

List returns records matching the filter, newest first.

func (*PostgresStore) Promote

func (s *PostgresStore) Promote(ctx context.Context, id string, p Promotion) error

Promote stores what the record became.

func (*PostgresStore) RecordFetch

func (s *PostgresStore) RecordFetch(ctx context.Context, recordID string, by Fetcher) error

RecordFetch notes that a session dereferenced this record.

func (*PostgresStore) Reject

func (s *PostgresStore) Reject(ctx context.Context, id string, r Rejection) error

Reject records that the record was reviewed and declined.

func (*PostgresStore) Search

func (s *PostgresStore) Search(ctx context.Context, q SearchQuery) ([]Scored, error)

Search ranks the caller's successful records by relevance.

The outcome is not a filter here but a signal on the hit: an agent looking for a query to reuse should see that one was satisfied and re-run by others while another merely ran, and choose. Filtering to satisfied records only would hide every good query nobody has cited yet.

func (*PostgresStore) StartCleanupRoutine

func (s *PostgresStore) StartCleanupRoutine(interval time.Duration)

StartCleanupRoutine sweeps expired records on an interval until Close. It is started by the layer that assembles the catalog, so a deployment does not have to remember to run it.

type Promoter

type Promoter struct {
	// contains filtered or unexported fields
}

Promoter turns a satisfied record into something the whole platform can see.

func NewPromoter

func NewPromoter(store Store, queries CuratedQueryWriter, examples ExampleWriter) *Promoter

NewPromoter builds the promotion path. Either writer may be nil: a deployment with no DataHub cannot promote a query, and one with no example store cannot promote an API call, and each refuses only its own kind.

func (*Promoter) Promote

func (p *Promoter) Promote(ctx context.Context, scope Scope, actor string) (*Record, error)

Promote publishes one record and records what it became.

The scope is the caller's: an owner promotes their own record from the portal, and a reviewer promotes any record from the operator surface. Both reach the same code, so what promotion means does not depend on which page it was started from.

func (*Promoter) Reject

func (p *Promoter) Reject(ctx context.Context, scope Scope, actor, note string) (*Record, error)

Reject records that a reviewer declined the record, so the queue does not offer it again.

type Promotion

type Promotion struct {
	URN   string
	Actor string
}

Promotion records what a record became.

type Record

type Record struct {
	ID string `json:"id" example:"9b1c0f26-1a3e-4c5f-9d0b-2f7a6e5c4d31"`
	// EventID is the audit event this call was recorded under and the key of
	// its mcp:call:<event_id> reference.
	EventID string `json:"event_id" example:"a1b2c3d4e5f6g7h8"`
	// Reference is EventID in the form an agent cites.
	Reference string `json:"reference" example:"mcp:call:a1b2c3d4e5f6g7h8"`
	Kind      string `json:"kind" example:"sql"`
	ToolName  string `json:"tool_name" example:"trino_query"`
	// Connection is the named connection the call went through. It is part of
	// a record's identity: the same statement against two warehouses is two
	// records, and reuse never crosses connections.
	Connection string `json:"connection,omitempty" example:"acme-warehouse"`

	// Statement is the SQL text, on a sql record.
	Statement string `json:"statement,omitempty"`
	// Method, Path and OperationID are the request line, on an api record.
	Method      string `json:"method,omitempty" example:"GET"`
	Path        string `json:"path,omitempty" example:"/v1/orders"`
	OperationID string `json:"operation_id,omitempty" example:"listOrders"`

	// Targets are what the call addressed: DataHub dataset URNs parsed from
	// the SQL, or the endpoint identity for an API call. Sorted and
	// deduplicated, so two records over the same tables compare equal.
	Targets []string `json:"targets"`

	// Purpose is the reason the caller stated for making the call (#1317).
	Purpose   string `json:"purpose,omitempty" example:"Sizing Q3 revenue by region for the board deck."`
	UserID    string `json:"user_id,omitempty" example:"550e8400-e29b-41d4-a716-446655440000"`
	UserEmail string `json:"user_email,omitempty" example:"marcus.johnson@example.com"`
	SessionID string `json:"session_id,omitempty" example:"dps_9f2c1a4b8e7d6c5a"`
	Persona   string `json:"persona,omitempty" example:"data-engineer"`

	Success       bool   `json:"success" example:"true"`
	ErrorMessage  string `json:"error_message,omitempty"`
	DurationMS    int64  `json:"duration_ms" example:"143"`
	ResponseChars int    `json:"response_chars" example:"2450"`

	// Outcome is derived on every read; see the package comment.
	Outcome string `json:"outcome" example:"satisfied"`
	// SatisfiedBy names the route that satisfied the record (asset, export,
	// capture). Empty on every other outcome.
	SatisfiedBy string `json:"satisfied_by,omitempty" example:"capture"`
	// Artifacts are what was built from this call: the assets, exports and
	// captured insights that cite it.
	Artifacts []Artifact `json:"artifacts,omitempty"`
	// ReuseCount is how many later sessions fetched this record and then ran
	// what it holds. It is the only signal on a record that a stranger, and
	// not its author, found it worth running.
	ReuseCount int `json:"reuse_count" example:"2"`

	PromotedURN   string     `json:"promoted_urn,omitempty" example:"urn:li:query:abc123"`
	PromotedAt    *time.Time `json:"promoted_at,omitempty"`
	PromotedBy    string     `json:"promoted_by,omitempty" example:"marcus.johnson@example.com"`
	RejectedAt    *time.Time `json:"rejected_at,omitempty"`
	RejectedBy    string     `json:"rejected_by,omitempty" example:"marcus.johnson@example.com"`
	RejectionNote string     `json:"rejection_note,omitempty"`

	CreatedAt time.Time `json:"created_at"`
}

Record is one data-access call, cataloged.

func (Record) Promotable

func (r Record) Promotable() bool

Promotable reports whether this record may be promoted: it must have answered something, and it must not already have been promoted or rejected.

type Recorder

type Recorder struct {
	// contains filtered or unexported fields
}

Recorder catalogs data-access calls as they are audited.

It is a decorator over the audit store rather than a middleware of its own, for two reasons. The audit event is the complete record of a call — its id, its purpose, its outcome, its duration, its arguments after the redaction policy has been applied — so a middleware would be reassembling what already exists. And the platform's audit writer is asynchronous, so a decorator here does its work on the writer's drain goroutine: cataloging a query costs the query nothing.

A failure to record is logged and swallowed. The catalog is derived from the audit log; losing an entry costs a query its place in the catalog, and must never cost the audit row it was derived from.

func (*Recorder) Close

func (r *Recorder) Close() error

Close passes through to the audit store, which owns the connection.

func (*Recorder) Log

func (r *Recorder) Log(ctx context.Context, event audit.Event) error

Log writes the audit event, then catalogs it when it is a data-access call. The audit write goes first and its error is returned unchanged: audit is the system of record, and the catalog is a reader of it.

func (*Recorder) Query

func (r *Recorder) Query(ctx context.Context, filter audit.QueryFilter) ([]audit.Event, error)

Query passes through to the audit store.

type Rejection

type Rejection struct {
	Actor string
	Note  string
}

Rejection records that a record was reviewed and declined, so the queue does not offer it again.

type Scope

type Scope struct {
	ID string
	// UserID restricts the read to that caller's own records. Empty is
	// unrestricted.
	UserID string
}

Scope names one record and, optionally, the only caller allowed to read it. The restriction is a predicate inside the query rather than a comparison the handler makes afterwards, so another caller's record id is answered not-found — the same answer an id that was never used gets.

type Scored

type Scored struct {
	Record Record
	Score  float64
}

Scored is one record with its relevance.

type SearchQuery

type SearchQuery struct {
	// Text is the natural-language query, matched lexically.
	Text string
	// Embedding is the query vector. Empty selects lexical-only ranking,
	// which is what a deployment with no embedding provider gets.
	Embedding []float32
	// UserID scopes the search to one caller's records. A search with no
	// caller returns nothing rather than everyone's calls.
	UserID string
	Limit  int
}

SearchQuery is one search over the catalog.

func (SearchQuery) EffectiveLimit

func (q SearchQuery) EffectiveLimit() int

EffectiveLimit returns the bounded limit for a query.

type Store

type Store interface {
	// Insert records one call. It is idempotent on event id: the same call
	// recorded twice yields one record.
	Insert(ctx context.Context, r Record) error
	// List returns records matching the filter, newest first, with their
	// outcomes derived.
	List(ctx context.Context, f Filter) ([]Record, error)
	// Count returns how many records match the filter, ignoring its paging.
	Count(ctx context.Context, f Filter) (int, error)
	// Get returns one record with its outcome, artifacts and reuse count, or
	// ErrNotFound when the scope admits none.
	Get(ctx context.Context, scope Scope) (*Record, error)
	// GetByEventID returns the record for one audit event id, scoped the same
	// way Get is. It is how an mcp:call:<event_id> reference resolves.
	GetByEventID(ctx context.Context, eventID, userID string) (*Record, error)
	// RecordFetch notes that a session dereferenced this record, which is the
	// first half of reuse.
	RecordFetch(ctx context.Context, recordID string, by Fetcher) error
	// CreditReuse credits every earlier record that the given call re-ran:
	// one this session had fetched, produced by a different session, with the
	// same kind, connection and statement (or operation). Returns how many
	// records were credited.
	CreditReuse(ctx context.Context, r Record) (int, error)
	// ForTargets returns satisfied records addressing any of the given
	// dataset URNs, most reused first. It is what the enrichment path shows
	// beside a table.
	ForTargets(ctx context.Context, urns []string, userID string, limit int) ([]Record, error)
	// Promote stores what the record became.
	Promote(ctx context.Context, id string, p Promotion) error
	// Reject records that the record was declined.
	Reject(ctx context.Context, id string, r Rejection) error
}

Store reads and writes call records. Implemented over PostgreSQL by PostgresStore; a deployment with no database keeps no catalog, and every surface that needs one stays unregistered.

type URNBuilder

type URNBuilder func(connectionKind, connection, catalog, schema, table string) string

URNBuilder turns a table reference into the dataset URN the catalog knows it by, applying the connection's platform and catalog mapping. It is the same function the reflexive-capture path takes (middleware.URNBuilder); declared here so this package does not import the middleware that will later read it.

The connection is named by kind and name together. A name alone is ambiguous where a deployment carries it across kinds, and the target it produced then named a platform the statement never ran against (#1384); the audit event this recorder reads carries the kind alongside the name.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL