Documentation
¶
Overview ¶
Package shadow implements the mirror pipeline: halo-kong posts a Job for sampled first-turn requests, halo-shadow replays it against control and candidate (non-streaming) and stores the pair for later LLM-judge grading.
Index ¶
- Constants
- Variables
- func KeyID(key []byte) string
- func NewHTTPClient() *http.Client
- func ReadKeyFile(path string) ([]byte, error)
- type Budget
- type BudgetDB
- type Config
- type DBStore
- type FileStore
- type Job
- type Mirrorer
- type Pair
- type PairDB
- type Server
- type SharedBudget
- type Side
- type Store
- type StoreOptions
- type Target
Constants ¶
const DefaultBudgetUSD = 50
DefaultBudgetUSD caps estimated spend when Config.BudgetUSD is unset.
const ReservationTTL = 15 * time.Minute
ReservationTTL is how long an unsettled reservation lives before it is reaped and charged in full: longer than queueing plus both upstream calls.
const TokenHeader = "X-Halo-Shadow-Token" //nolint:gosec // header name, not a credential
TokenHeader authenticates halo-kong to halo-shadow.
Variables ¶
var ErrPlaintextRow = errors.New("shadow: plaintext pair row in a sealed store")
ErrPlaintextRow is a plaintext (unsealed) pair row read with pair keys configured. Whoever can write the store could otherwise inject unsealed rows into a store that is supposed to be encrypted; reading rows written before sealing was turned on needs the explicit migration switch (DecodePairAllowPlaintext); each front end names its own switch.
var ReplayHeaders = map[string]bool{"anthropic-version": true, "anthropic-beta": true, "openai-beta": true}
ReplayHeaders is the only client-originated header set halo-shadow forwards.
Functions ¶
func NewHTTPClient ¶
NewHTTPClient never follows redirects (a 3xx is recorded as the response: credentials must not travel to wherever an upstream points) and bounds every phase of the call.
func ReadKeyFile ¶ added in v0.3.0
ReadKeyFile loads a base64 pair key file, the -pair-key-file format.
Types ¶
type Budget ¶ added in v0.3.0
type Budget interface {
// Reserve books est under reservation id; false = over budget.
Reserve(ctx context.Context, id string, est float64) (bool, error)
// Settle swaps reservation id (of est) for charged.
Settle(ctx context.Context, id string, est, charged float64) error
// Spent is the settled spend.
Spent(ctx context.Context) (float64, error)
}
Budget books a job's worst-case cost before anything is sent and settles it to the charged cost afterwards. The default is per process; SharedBudget makes the cap global across replicas.
type BudgetDB ¶ added in v0.3.0
type BudgetDB interface {
Reserve(ctx context.Context, scope, id string, amount, limit float64, ttl time.Duration) (bool, error)
Settle(ctx context.Context, scope, id string, amount, charged float64) error
Spent(ctx context.Context, scope string) (float64, error)
}
BudgetDB is the shared budget table (pgstore.DB).
type Config ¶
type Config struct {
Queue int // bounded queue length (default 64)
Workers int // default 4
Token string
// Policy returns the current org (gateway.Snapshot.Get). Both sides of a
// pair are resolved from it; jobs never name upstreams. Required.
Policy func() (*policy.Org, error)
// UpstreamHeaders are halo-shadow's own credentials, keyed by policy upstream
// name; they are only ever sent to that upstream's URL.
UpstreamHeaders map[string]map[string]string
// BudgetUSD caps estimated spend (default DefaultBudgetUSD). Each job
// reserves a worst-case estimate before anything is sent, so concurrency
// can't overshoot; the reservation is settled to actual usage afterwards.
BudgetUSD float64
PriceInPerMTok float64
PriceOutPerMTok float64
HTTP *http.Client // default: NewHTTPClient()
Log *slog.Logger
// Budget, when set, replaces the per-process budget (BudgetUSD) with a
// shared one (SharedBudget), so the cap holds across replicas.
Budget Budget
// Ping, when set, gates /readyz on the shared store being reachable.
Ping func(context.Context) error
}
Config for the halo-shadow server.
type DBStore ¶ added in v0.3.0
type DBStore struct {
// contains filtered or unexported fields
}
DBStore stores pairs in a shared table, one row per pair in exactly the FileStore row format (sealed with the pair key when one is set), so workers keep no state and `halo-shadow decrypt-pairs` reads an export.
func NewDBStore ¶ added in v0.3.0
func NewDBStore(db PairDB, o StoreOptions) (*DBStore, error)
NewDBStore returns a Store on db with FileStore's options.
func (*DBStore) Append ¶ added in v0.3.0
Append implements Store; a retried pair (same id) is stored once.
type FileStore ¶
type FileStore struct {
// contains filtered or unexported fields
}
FileStore appends JSONL to a file (mode 0600).
func NewFileStore ¶
func NewFileStore(path string, o StoreOptions) (*FileStore, error)
type Job ¶
type Job struct {
Experiment string `json:"experiment"`
Variant string `json:"variant"`
Ring string `json:"ring,omitempty"`
SessionID string `json:"session_id,omitempty"`
Protocol string `json:"protocol"`
Method string `json:"method"`
Path string `json:"path"`
Headers map[string]string `json:"headers,omitempty"`
Body json.RawMessage `json:"body"`
}
Job is what halo-kong / halo-proxy post. It deliberately carries NO upstream URLs or models: halo-shadow resolves both sides from its own policy snapshot (experiment + variant + the alias in Body/Path), so a caller holding the token can't point halo-shadow (and its credentials) anywhere else. Body is the ORIGINAL client body (alias model). Only allowlisted Headers are replayed. Ring/SessionID are recorded metadata only; they never affect routing.
func Jobs ¶
Jobs builds the mirror jobs for a prepared gateway request: one per shadow target, only the ReplayHeaders allowlist (never client credentials) and never an upstream (halo-shadow resolves both sides from its own policy). It returns nil when nothing should be mirrored or body exceeds maxBytes. A body the gateway's DLP redacted is mirrored redacted (ADR-0033).
type Mirrorer ¶
type Mirrorer struct {
// contains filtered or unexported fields
}
Mirrorer is the halo-kong side: Submit never blocks the request path. Jobs go through a small bounded queue to a single sender; when full they're dropped and counted.
func NewMirrorer ¶
NewMirrorer validates url and starts the single sender goroutine.
func SharedMirrorer ¶
SharedMirrorer returns one Mirrorer per halo-shadow URL (Kong instantiates a plugin config per route/service; they share the queue).
type Pair ¶
type Pair struct {
ID string `json:"id"`
Time time.Time `json:"time"`
Experiment string `json:"experiment"`
Ring string `json:"ring,omitempty"`
SessionID string `json:"session_id,omitempty"`
Protocol string `json:"protocol"`
Request json.RawMessage `json:"request"`
Control Side `json:"control"`
Candidate Side `json:"candidate"`
CostUSD float64 `json:"cost_usd_estimate"`
}
Pair is one JSONL row. Request + both responses are everything a judge needs.
func DecodePair ¶
DecodePair reads one JSONL row, decrypting it with keys (by key id) when sealed. With no keys it reads plaintext rows; with keys it refuses them (ErrPlaintextRow).
func DecodePairAllowPlaintext ¶ added in v0.5.0
DecodePairAllowPlaintext is DecodePair that also accepts plaintext rows when keys are configured: the migration path for a store sealed part way through its life. Off unless an operator asks for it.
type PairDB ¶ added in v0.3.0
type PairDB interface {
AppendPair(ctx context.Context, id string, at time.Time, row []byte) error
PrunePairs(ctx context.Context, cutoff time.Time) (int, error)
}
PairDB is a pair table shared by every halo-shadow replica (pgstore.DB).
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server is halo-shadow.
func (*Server) MetricsHandler ¶
MetricsHandler serves /metrics without the token, for a separate scrape-only listener (Prometheus can't send X-Halo-Shadow-Token). It exposes counters and spend gauges only; keep it off untrusted networks.
func (*Server) Spent ¶
Spent is the running estimated spend in USD (settled jobs only); with a shared budget, every replica's. A store error reads as 0 and is logged.
type SharedBudget ¶ added in v0.3.0
type SharedBudget struct {
}
SharedBudget is one budget for every replica: a reservation is a conditional UPDATE (spent+reserved+est <= Limit) on the scope's row.
type Side ¶
type Side struct {
Target Target `json:"target"`
Status int `json:"status"`
LatencyMS int64 `json:"latency_ms"`
InputTokens int64 `json:"input_tokens"`
OutputTokens int64 `json:"output_tokens"`
Response json.RawMessage `json:"response,omitempty"`
Error string `json:"error,omitempty"`
}
Side is one response, flattened for the judge.
type StoreOptions ¶
type StoreOptions struct {
// Key (32 bytes) enables AES-256-GCM at rest. Rows become
// {"id","time","kid","enc"}; id/time stay clear for retention pruning.
Key []byte
// Retention: Prune drops rows older than this (0 = keep forever).
Retention time.Duration
}
StoreOptions tunes FileStore. Zero value: plaintext, kept forever.