shadow

package
v0.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package shadow implements the mirror pipeline: halo-kong posts a Job for sampled first-turn requests, halo-shadow replays it against control and candidate (non-streaming) and stores the pair for later LLM-judge grading.

Index

Constants

View Source
const DefaultBudgetUSD = 50

DefaultBudgetUSD caps estimated spend when Config.BudgetUSD is unset.

View Source
const ReservationTTL = 15 * time.Minute

ReservationTTL is how long an unsettled reservation lives before it is reaped and charged in full: longer than queueing plus both upstream calls.

View Source
const TokenHeader = "X-Halo-Shadow-Token" //nolint:gosec // header name, not a credential

TokenHeader authenticates halo-kong to halo-shadow.

Variables

View Source
var ErrPlaintextRow = errors.New("shadow: plaintext pair row in a sealed store")

ErrPlaintextRow is a plaintext (unsealed) pair row read with pair keys configured. Whoever can write the store could otherwise inject unsealed rows into a store that is supposed to be encrypted; reading rows written before sealing was turned on needs the explicit migration switch (DecodePairAllowPlaintext); each front end names its own switch.

View Source
var ReplayHeaders = map[string]bool{"anthropic-version": true, "anthropic-beta": true, "openai-beta": true}

ReplayHeaders is the only client-originated header set halo-shadow forwards.

Functions

func KeyID

func KeyID(key []byte) string

KeyID names a key in sealed rows: first 8 bytes of sha256(key), hex.

func NewHTTPClient

func NewHTTPClient() *http.Client

NewHTTPClient never follows redirects (a 3xx is recorded as the response: credentials must not travel to wherever an upstream points) and bounds every phase of the call.

func ReadKeyFile added in v0.3.0

func ReadKeyFile(path string) ([]byte, error)

ReadKeyFile loads a base64 pair key file, the -pair-key-file format.

Types

type Budget added in v0.3.0

type Budget interface {
	// Reserve books est under reservation id; false = over budget.
	Reserve(ctx context.Context, id string, est float64) (bool, error)
	// Settle swaps reservation id (of est) for charged.
	Settle(ctx context.Context, id string, est, charged float64) error
	// Spent is the settled spend.
	Spent(ctx context.Context) (float64, error)
}

Budget books a job's worst-case cost before anything is sent and settles it to the charged cost afterwards. The default is per process; SharedBudget makes the cap global across replicas.

type BudgetDB added in v0.3.0

type BudgetDB interface {
	Reserve(ctx context.Context, scope, id string, amount, limit float64, ttl time.Duration) (bool, error)
	Settle(ctx context.Context, scope, id string, amount, charged float64) error
	Spent(ctx context.Context, scope string) (float64, error)
}

BudgetDB is the shared budget table (pgstore.DB).

type Config

type Config struct {
	Queue   int // bounded queue length (default 64)
	Workers int // default 4
	Token   string
	// Policy returns the current org (gateway.Snapshot.Get). Both sides of a
	// pair are resolved from it; jobs never name upstreams. Required.
	Policy func() (*policy.Org, error)
	// UpstreamHeaders are halo-shadow's own credentials, keyed by policy upstream
	// name; they are only ever sent to that upstream's URL.
	UpstreamHeaders map[string]map[string]string
	// BudgetUSD caps estimated spend (default DefaultBudgetUSD). Each job
	// reserves a worst-case estimate before anything is sent, so concurrency
	// can't overshoot; the reservation is settled to actual usage afterwards.
	BudgetUSD       float64
	PriceInPerMTok  float64
	PriceOutPerMTok float64
	HTTP            *http.Client // default: NewHTTPClient()
	Log             *slog.Logger
	// Budget, when set, replaces the per-process budget (BudgetUSD) with a
	// shared one (SharedBudget), so the cap holds across replicas.
	Budget Budget
	// Ping, when set, gates /readyz on the shared store being reachable.
	Ping func(context.Context) error
}

Config for the halo-shadow server.

type DBStore added in v0.3.0

type DBStore struct {
	// contains filtered or unexported fields
}

DBStore stores pairs in a shared table, one row per pair in exactly the FileStore row format (sealed with the pair key when one is set), so workers keep no state and `halo-shadow decrypt-pairs` reads an export.

func NewDBStore added in v0.3.0

func NewDBStore(db PairDB, o StoreOptions) (*DBStore, error)

NewDBStore returns a Store on db with FileStore's options.

func (*DBStore) Append added in v0.3.0

func (s *DBStore) Append(ctx context.Context, p Pair) error

Append implements Store; a retried pair (same id) is stored once.

func (*DBStore) Close added in v0.3.0

func (s *DBStore) Close() error

Close implements io.Closer; the database is the caller's to close.

func (*DBStore) Prune added in v0.3.0

func (s *DBStore) Prune(now time.Time) (int, error)

Prune deletes pairs older than Retention (no-op if 0). Every replica may run it; the DELETE is idempotent.

type FileStore

type FileStore struct {
	// contains filtered or unexported fields
}

FileStore appends JSONL to a file (mode 0600).

func NewFileStore

func NewFileStore(path string, o StoreOptions) (*FileStore, error)

func (*FileStore) Append

func (s *FileStore) Append(ctx context.Context, p Pair) error

func (*FileStore) Close

func (s *FileStore) Close() error

func (*FileStore) Prune

func (s *FileStore) Prune(now time.Time) (removed int, err error)

Prune rewrites the file without rows older than Retention (no-op if 0). Rows whose time can't be read are kept: pruning never destroys data it doesn't understand.

type Job

type Job struct {
	Experiment string            `json:"experiment"`
	Variant    string            `json:"variant"`
	Ring       string            `json:"ring,omitempty"`
	SessionID  string            `json:"session_id,omitempty"`
	Protocol   string            `json:"protocol"`
	Method     string            `json:"method"`
	Path       string            `json:"path"`
	Headers    map[string]string `json:"headers,omitempty"`
	Body       json.RawMessage   `json:"body"`
}

Job is what halo-kong / halo-proxy post. It deliberately carries NO upstream URLs or models: halo-shadow resolves both sides from its own policy snapshot (experiment + variant + the alias in Body/Path), so a caller holding the token can't point halo-shadow (and its credentials) anywhere else. Body is the ORIGINAL client body (alias model). Only allowlisted Headers are replayed. Ring/SessionID are recorded metadata only; they never affect routing.

func Jobs

func Jobs(res gateway.Result, h http.Header, path string, body []byte, maxBytes int) []Job

Jobs builds the mirror jobs for a prepared gateway request: one per shadow target, only the ReplayHeaders allowlist (never client credentials) and never an upstream (halo-shadow resolves both sides from its own policy). It returns nil when nothing should be mirrored or body exceeds maxBytes. A body the gateway's DLP redacted is mirrored redacted (ADR-0033).

type Mirrorer

type Mirrorer struct {
	// contains filtered or unexported fields
}

Mirrorer is the halo-kong side: Submit never blocks the request path. Jobs go through a small bounded queue to a single sender; when full they're dropped and counted.

func NewMirrorer

func NewMirrorer(url, token string, queue int) (*Mirrorer, error)

NewMirrorer validates url and starts the single sender goroutine.

func SharedMirrorer

func SharedMirrorer(url, token string) (*Mirrorer, error)

SharedMirrorer returns one Mirrorer per halo-shadow URL (Kong instantiates a plugin config per route/service; they share the queue).

func (*Mirrorer) Dropped

func (m *Mirrorer) Dropped() int64

func (*Mirrorer) Failed

func (m *Mirrorer) Failed() int64

func (*Mirrorer) Submit

func (m *Mirrorer) Submit(j Job) bool

Submit reports whether the job was queued.

type Pair

type Pair struct {
	ID         string          `json:"id"`
	Time       time.Time       `json:"time"`
	Experiment string          `json:"experiment"`
	Ring       string          `json:"ring,omitempty"`
	SessionID  string          `json:"session_id,omitempty"`
	Protocol   string          `json:"protocol"`
	Request    json.RawMessage `json:"request"`
	Control    Side            `json:"control"`
	Candidate  Side            `json:"candidate"`
	CostUSD    float64         `json:"cost_usd_estimate"`
}

Pair is one JSONL row. Request + both responses are everything a judge needs.

func DecodePair

func DecodePair(line []byte, keys ...[]byte) (Pair, error)

DecodePair reads one JSONL row, decrypting it with keys (by key id) when sealed. With no keys it reads plaintext rows; with keys it refuses them (ErrPlaintextRow).

func DecodePairAllowPlaintext added in v0.5.0

func DecodePairAllowPlaintext(line []byte, keys ...[]byte) (Pair, error)

DecodePairAllowPlaintext is DecodePair that also accepts plaintext rows when keys are configured: the migration path for a store sealed part way through its life. Off unless an operator asks for it.

type PairDB added in v0.3.0

type PairDB interface {
	AppendPair(ctx context.Context, id string, at time.Time, row []byte) error
	PrunePairs(ctx context.Context, cutoff time.Time) (int, error)
}

PairDB is a pair table shared by every halo-shadow replica (pgstore.DB).

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is halo-shadow.

func New

func New(cfg Config, store Store) *Server

func (*Server) Handler

func (s *Server) Handler() http.Handler

func (*Server) MetricsHandler

func (s *Server) MetricsHandler() http.Handler

MetricsHandler serves /metrics without the token, for a separate scrape-only listener (Prometheus can't send X-Halo-Shadow-Token). It exposes counters and spend gauges only; keep it off untrusted networks.

func (*Server) Spent

func (s *Server) Spent() float64

Spent is the running estimated spend in USD (settled jobs only); with a shared budget, every replica's. A store error reads as 0 and is logged.

func (*Server) Start

func (s *Server) Start(ctx context.Context)

Start launches workers; they exit when ctx is cancelled after finishing their current job. Wait blocks until they have.

func (*Server) Wait

func (s *Server) Wait()

type SharedBudget added in v0.3.0

type SharedBudget struct {
	DB    BudgetDB
	Scope string  // budget name; change it to start a fresh budget
	Limit float64 // USD
}

SharedBudget is one budget for every replica: a reservation is a conditional UPDATE (spent+reserved+est <= Limit) on the scope's row.

func (SharedBudget) Reserve added in v0.3.0

func (b SharedBudget) Reserve(ctx context.Context, id string, est float64) (bool, error)

Reserve implements Budget.

func (SharedBudget) Settle added in v0.3.0

func (b SharedBudget) Settle(ctx context.Context, id string, est, charged float64) error

Settle implements Budget.

func (SharedBudget) Spent added in v0.3.0

func (b SharedBudget) Spent(ctx context.Context) (float64, error)

Spent implements Budget.

type Side

type Side struct {
	Target       Target          `json:"target"`
	Status       int             `json:"status"`
	LatencyMS    int64           `json:"latency_ms"`
	InputTokens  int64           `json:"input_tokens"`
	OutputTokens int64           `json:"output_tokens"`
	Response     json.RawMessage `json:"response,omitempty"`
	Error        string          `json:"error,omitempty"`
}

Side is one response, flattened for the judge.

type Store

type Store interface {
	Append(ctx context.Context, p Pair) error
}

Store persists pairs.

type StoreOptions

type StoreOptions struct {
	// Key (32 bytes) enables AES-256-GCM at rest. Rows become
	// {"id","time","kid","enc"}; id/time stay clear for retention pruning.
	Key []byte
	// Retention: Prune drops rows older than this (0 = keep forever).
	Retention time.Duration
}

StoreOptions tunes FileStore. Zero value: plaintext, kept forever.

type Target

type Target struct {
	Variant  string `json:"variant,omitempty"`
	Upstream string `json:"upstream"` // policy upstream name
	Model    string `json:"model"`
}

Target is one side of a pair, as resolved by halo-shadow from its own policy.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL