core

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 30, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Package core holds the shared data types of the gaslit engine. It is a leaf package: every other internal package imports it, and it imports nothing internal. The module root re-exports these types under the public `gaslit` package via type aliases.

Index

Constants

View Source
const DefaultJSONInstruction = `` /* 571-byte string literal not displayed */

DefaultJSONInstruction is the output contract agents must follow for multiple-choice answers. Custom AnswerSchema implementations may provide their own instructions instead.

View Source
const OpenEndedInstruction = `` /* 223-byte string literal not displayed */

OpenEndedInstruction is the output contract for free-form (option-less) questions.

Variables

This section is empty.

Functions

func Clamp01

func Clamp01(v float64) float64

Clamp01 bounds a float to [0.0, 1.0].

func DefaultNLIProbe

func DefaultNLIProbe(question, r1, r2 string) string

DefaultNLIProbe is the built-in entailment probe builder. It is used as the LLM scorer's fallback when no PromptTemplates.NLIProbe is wired.

func ExtractJSONObject

func ExtractJSONObject(raw string) (string, error)

ExtractJSONObject finds the outermost balanced JSON object in raw output.

func Tokenize

func Tokenize(s string) map[string]bool

Tokenize lowercases s and splits it into a set of cleaned word tokens.

Types

type AgentConfig

type AgentConfig struct {
	// Instruction, when non-empty, is a per-agent system instruction that is
	// prepended to every prompt sent to this agent.
	Instruction string
	// Weight is the influence weight used by weighted aggregation functions.
	// Zero means the default weight of 1.0.
	Weight float64
	// SkipPhases lists the phases this agent is excluded from.
	SkipPhases map[Phase]bool
}

AgentConfig customizes a single registered panelist, keyed by agent name.

type AggregationFn

type AggregationFn func(after map[string]*Answer, labels []string, weights map[string]float64) string

AggregationFn derives the winning option from the panel's final positions. It returns the winning option label, or "" when there is no clear winner.

type Answer

type Answer struct {
	Answer      string             `json:"answer"`
	Confidences map[string]float64 `json:"confidences,omitempty"`
	Reasoning   string             `json:"reasoning"`
	Challenge   *challengeRequest  `json:"challenge,omitempty"`
}

Answer is a single agent's position on a question: its pick, per-option confidences (absent for open-ended questions), reasoning, and an optional nomination of another panelist for cross-examination.

type AnswerSchema

type AnswerSchema interface {
	// Instruction is the output-contract text appended to prompts.
	Instruction() string
	// Parse extracts an Answer from raw model output. It must tolerate leading
	// prose, markdown fences, and trailing chatter.
	Parse(raw string) (*Answer, error)
}

AnswerSchema defines the JSON contract agents must follow when answering, and how raw model output is parsed into an Answer.

type AttributionFn

type AttributionFn func(res *PromptResult, labels []string, verdict *DriftVerdict)

AttributionFn determines who drove the primary shift. It mutates the verdict with PrimaryDriver and DriverImpact.

type BudgetError

type BudgetError struct {
	MaxCalls  int
	Calls     int64
	MaxTokens int
	Tokens    int64
}

BudgetError is returned when a CostBudget is exhausted.

func (*BudgetError) Error

func (e *BudgetError) Error() string

type CallOption

type CallOption func(*CallOptions)

CallOption customizes a single engine call (Debate, Baseline, Grill, CCI, ...) without mutating the engine's construction-time configuration.

func WithBudgetCall

func WithBudgetCall(b CostBudget) CallOption

func WithClaimCall

func WithClaimCall(s string) CallOption

func WithConfidenceCall

func WithConfidenceCall(c float64) CallOption

func WithConsistencyScorerCall

func WithConsistencyScorerCall(s ConsistencyScorer) CallOption

WithConsistencyScorerCall overrides the consistency scorer for a single call (e.g. Rescore) without rebuilding the engine.

func WithGrillCountCall

func WithGrillCountCall(n int) CallOption

func WithGrillStrategyCall

func WithGrillStrategyCall(s GrillStrategy) CallOption

func WithHooksCall

func WithHooksCall(h Hooks) CallOption

func WithMaxRoundsCall

func WithMaxRoundsCall(n int) CallOption

func WithMetadataCall

func WithMetadataCall(m map[string]any) CallOption

func WithOptionLabelsCall

func WithOptionLabelsCall(l []string) CallOption

func WithProtocolCall

func WithProtocolCall(p Protocol) CallOption

func WithThresholdsCall

func WithThresholdsCall(t Thresholds) CallOption

func WithVerboseCall

func WithVerboseCall(v bool) CallOption

type CallOptions

type CallOptions struct {
	Verbose           bool
	MaxRounds         int
	OptionLabels      []string
	Thresholds        Thresholds
	GrillStrategy     GrillStrategy
	GrillCount        int
	Protocol          Protocol
	Metadata          map[string]any
	Confidence        *float64
	Claim             string
	Hooks             *Hooks
	Budget            CostBudget
	ConsistencyScorer ConsistencyScorer
}

CallOptions is the resolved per-call configuration.

type Classifier

type Classifier func(confScore, consistencyScore float64) (QuadrantState, string)

Classifier maps confidence and consistency scores onto the 2D quadrant grid.

type Config

type Config struct {
	// SupervisorModel is the model that powers the moderator (and, when
	// ConsistencyModel is unset, the default LLM consistency scorer). Required.
	SupervisorModel model.BaseModel[*schema.Message]
	// ConsistencyModel powers the default LLM consistency scorer. When nil it
	// falls back to SupervisorModel. Use a cheap model here and reserve
	// SupervisorModel for the moderator.
	ConsistencyModel model.BaseModel[*schema.Message]
	// Agents is the panel of debaters. At least one is required. Agent names
	// (from Agent.Name) must be unique.
	Agents []adk.Agent
	// Logger receives all verbose/progress output. Default slog.Default().
	Logger *slog.Logger

	// PacingDelay throttles each model invocation by sleeping before the call.
	// Zero disables pacing. Default 0.
	PacingDelay time.Duration
	// RequestTimeout bounds a single model invocation. Zero disables. Default 0.
	RequestTimeout time.Duration
	// MaxRetries is how many corrective re-prompts to attempt when an agent
	// returns unparseable output. Default 2.
	MaxRetries int
	// Concurrency is the number of agents queried in parallel during the
	// baseline and evaluation phases. Default 1 (sequential).
	Concurrency int
	// MaxRounds is the number of (cross-examine → evaluate) rounds run by
	// Debate. Default 1.
	MaxRounds int

	// Thresholds drives drift detection and quadrant classification. Zero
	// Thresholds resolves to the defaults.
	Thresholds Thresholds
	// OptionLabels overrides the auto-generated A, B, C... labels used for
	// multiple-choice questions. Leave nil to auto-generate.
	OptionLabels []string
	// AnswerSchema defines the JSON output contract agents must follow. Default
	// is the MCQ schema. Open-ended questions (no options) always use the
	// open-ended schema regardless of this field.
	AnswerSchema AnswerSchema
	// Protocol selects the cross-examination orchestration strategy.
	Protocol Protocol
	// GrillStrategy selects which agents get grilling diagnostics.
	GrillStrategy GrillStrategy
	// GrillCount is the number of agents grilled by GrillTopN. Default 1.
	GrillCount int
	// AggregationFn derives the winning option from the final positions.
	AggregationFn AggregationFn
	// DriftMetric computes the distribution shift between an agent's before and
	// after positions. Only used for multiple-choice questions.
	DriftMetric DriftMetric
	// AttributionFn determines who drove the primary shift.
	AttributionFn AttributionFn
	// Classifier maps confidence/consistency onto the quadrant grid.
	Classifier Classifier
	// ConsistencyScorer evaluates whether two arguments from the same agent
	// contradict each other. Defaults to an LLM scorer over ConsistencyModel.
	ConsistencyScorer ConsistencyScorer
	// PromptTemplates overrides any or all of the built-in prompts.
	PromptTemplates PromptTemplates

	// AgentOverrides holds per-agent customization keyed by agent name.
	AgentOverrides map[string]AgentConfig
	// ExtraTools are additional tools given to the moderator alongside the
	// built-in cross_examine_agent tool.
	ExtraTools []tool.BaseTool
	// Hooks are observability callbacks fired at pipeline milestones.
	Hooks Hooks
	// Recorder receives a JSON-serializable Event stream for persistence.
	Recorder Recorder
	// Redact, when non-nil, is applied to every outgoing prompt before it is
	// sent to any model.
	Redact func(string) string
	// Budget caps total work per debate.
	Budget CostBudget
	// Verbose enables default progress logging when a call does not override it.
	Verbose bool
}

Config is the construction-time configuration for a Gaslit engine. The zero value is not usable directly; build one via Preset* helpers and/or the Option funcs. The engine applies defaults for every unset field.

func PresetDefault

func PresetDefault() Config

PresetDefault returns the recommended all-purpose configuration.

func PresetLenient

func PresetLenient() Config

PresetLenient returns a configuration that ignores small shifts and grades agents leniently.

func PresetStrict

func PresetStrict() Config

PresetStrict returns a configuration that flags smaller shifts and demands higher confidence/consistency before an agent is graded PEAK.

type ConsistencyScorer

type ConsistencyScorer interface {
	Score(ctx context.Context, question, r1, r2 string) (float64, error)
}

ConsistencyScorer evaluates whether two arguments from the same agent contradict each other. Scores range 0.0 (contradicts itself) to 1.0 (clean).

type CostBudget

type CostBudget struct {
	// MaxCalls limits the total number of model invocations.
	MaxCalls int
	// MaxTokens is an approximate token budget (characters / 4) consumed by
	// all outgoing prompt content.
	MaxTokens int
	// Deadline bounds the total wall-clock time of a debate.
	Deadline time.Duration
}

CostBudget caps how much work a single debate (or debate batch) may perform. A zero value for any field means "unlimited" for that field.

type DiagnosticResult

type DiagnosticResult struct {
	AgentName        string        `json:"agent_name"`
	ConfidenceScore  float64       `json:"confidence_score"`  // X-axis (0.0 -> 1.0)
	ConsistencyScore float64       `json:"consistency_score"` // Y-axis (0.0 -> 1.0)
	Quadrant         QuadrantState `json:"quadrant"`
	InitialReasonR1  string        `json:"r1_initial_reasoning"`
	CounterReasonR2  string        `json:"r2_counter_reasoning"`
	Recommendation   string        `json:"recommendation"`
}

DiagnosticResult is the outcome of grilling a single agent: where it lands on the confidence/consistency grid and what action is recommended.

type DriftMetric

type DriftMetric func(labels []string, before, after *Answer) float64

DriftMetric computes the distribution shift between an agent's before and after confidence distributions over the given option labels.

type DriftVerdict

type DriftVerdict struct {
	// HasDrift reports whether any agent flipped or significantly eroded.
	HasDrift bool `json:"has_drift"`
	// DriftedAgents lists the names of agents that drifted.
	DriftedAgents []string `json:"drifted_agents"`
	// PrimaryDriver is the agent (or the supervisor) responsible for the shift.
	PrimaryDriver string `json:"primary_driver"`
	// DriverImpact is a human-readable description of who swayed whom.
	DriverImpact string `json:"driver_impact"`
	// AgentShifts maps each agent name to its measured shift.
	AgentShifts map[string]float64 `json:"agent_shifts"`
}

DriftVerdict describes whether agents changed their positions over the course of a debate, which agents moved, and who drove the shift.

type Event

type Event struct {
	Time  time.Time       `json:"time"`
	Type  EventType       `json:"type"`
	Agent string          `json:"agent,omitempty"`
	Data  json.RawMessage `json:"data,omitempty"`
}

Event is a JSON-serializable record of a pipeline milestone, suitable for persistence or replay.

type EventType

type EventType string

EventType identifies a recorded pipeline event.

const (
	EventPhaseStart EventType = "phase_start"
	EventPhaseEnd   EventType = "phase_end"
	EventBaseline   EventType = "baseline"
	EventCrossExam  EventType = "cross_exam"
	EventEval       EventType = "evaluation"
	EventDrift      EventType = "drift"
	EventDiagnostic EventType = "diagnostic"
	EventSummary    EventType = "summary"
	EventDone       EventType = "done"
	EventError      EventType = "error"
)

type FullDebateSummary

type FullDebateSummary struct {
	Question       string                       `json:"question"`
	Options        []string                     `json:"options"`
	OptionLabels   []string                     `json:"option_labels"`
	PromptResult   *PromptResult                `json:"prompt_result"`
	DriftVerdict   DriftVerdict                 `json:"drift_verdict"`
	Diagnostics    map[string]*DiagnosticResult `json:"diagnostics,omitempty"` // Targeted grilling results
	Winner         string                       `json:"winner"`                // Aggregated winning option ("" if none)
	OverallOutcome string                       `json:"overall_outcome"`
	Metadata       map[string]any               `json:"metadata,omitempty"`
}

FullDebateSummary is the complete result of a Debate run: the question and options, pre/post positions, drift verdict, targeted grilling diagnostics, and the aggregated winner.

func ReclassifySummary

func ReclassifySummary(s *FullDebateSummary, thr Thresholds) *FullDebateSummary

ReclassifySummary recomputes every diagnostic's quadrant and recommendation from its stored confidence and consistency scores. It makes no model calls, so it is the cheapest way to iterate on thresholds or quadrant-tallying logic against a saved FullDebateSummary.

type GrillStrategy

type GrillStrategy int

GrillStrategy selects which panelists receive a grilling diagnostic.

const (
	// GrillNone skips grilling entirely.
	GrillNone GrillStrategy = iota
	// GrillSingleSuspect grills the single most drifted/influential agent.
	GrillSingleSuspect
	// GrillInfluencer grills the primary driver when it is a panelist,
	// otherwise the most drifted agent.
	GrillInfluencer
	// GrillAllDrifted grills every agent flagged as drifted.
	GrillAllDrifted
	// GrillTopN grills the GrillCount most drifted agents.
	GrillTopN
)

func (GrillStrategy) String

func (s GrillStrategy) String() string

type Hooks

type Hooks struct {
	OnPhaseStart func(ctx context.Context, phase Phase)
	OnPhaseEnd   func(ctx context.Context, phase Phase)
	OnBaseline   func(ctx context.Context, agent string, ans *Answer)
	OnCrossExam  func(ctx context.Context, target, prompt string)
	OnEval       func(ctx context.Context, agent string, ans *Answer)
	OnDrift      func(ctx context.Context, verdict *DriftVerdict)
	OnDiagnostic func(ctx context.Context, diag *DiagnosticResult)
}

Hooks are observability callbacks fired at pipeline milestones. Nil fields are skipped. All hooks receive the debate's context.

func MergeHooks

func MergeHooks(base, override Hooks) Hooks

MergeHooks combines a base set with overrides; non-nil override fields take precedence.

type InstrumentationStats

type InstrumentationStats struct {
	TotalCalls    int64
	Tokens        int64
	AgentCalls    map[string]int64
	PhaseDuration map[string][]time.Duration
}

InstrumentationStats is a point-in-time snapshot of engine usage.

type JSONError

type JSONError struct {
	// contains filtered or unexported fields
}

JSONError is a sentinel for malformed JSON extraction.

func (*JSONError) Error

func (e *JSONError) Error() string

type MCQSchema

type MCQSchema struct{}

MCQSchema is the default schema for multiple-choice questions: it requires an option label and a confidence for every option.

func (*MCQSchema) Instruction

func (s *MCQSchema) Instruction() string

func (*MCQSchema) Parse

func (s *MCQSchema) Parse(raw string) (*Answer, error)

type OpenEndedSchema

type OpenEndedSchema struct{}

OpenEndedSchema is the schema for free-form questions: agents return their answer text with no option labels or confidences.

func (*OpenEndedSchema) Instruction

func (s *OpenEndedSchema) Instruction() string

func (*OpenEndedSchema) Parse

func (s *OpenEndedSchema) Parse(raw string) (*Answer, error)

type Option

type Option func(*Config)

Option mutates a Config before the engine is constructed. All fields left unset still receive their defaults during engine construction.

func WithAgentConfig

func WithAgentConfig(name string, ac AgentConfig) Option

func WithAggregation

func WithAggregation(fn AggregationFn) Option

func WithAnswerSchema

func WithAnswerSchema(s AnswerSchema) Option

func WithAttribution

func WithAttribution(fn AttributionFn) Option

func WithBudget

func WithBudget(b CostBudget) Option

func WithClassifier

func WithClassifier(cl Classifier) Option

func WithConcurrency

func WithConcurrency(n int) Option

func WithConsistencyModel

func WithConsistencyModel(m model.BaseModel[*schema.Message]) Option

WithConsistencyModel points the default LLM consistency scorer at a separate model, decoupling it from the moderator's SupervisorModel.

func WithConsistencyScorer

func WithConsistencyScorer(s ConsistencyScorer) Option

WithConsistencyScorer swaps the consistency check to a custom scorer.

func WithDriftMetric

func WithDriftMetric(m DriftMetric) Option

func WithExtraTools

func WithExtraTools(tools ...tool.BaseTool) Option

func WithGrillCount

func WithGrillCount(n int) Option

func WithGrillStrategy

func WithGrillStrategy(s GrillStrategy) Option

func WithHooks

func WithHooks(h Hooks) Option

func WithLogger

func WithLogger(l *slog.Logger) Option

func WithMaxRetries

func WithMaxRetries(n int) Option

func WithMaxRounds

func WithMaxRounds(n int) Option

func WithOptionLabels

func WithOptionLabels(labels []string) Option

func WithPacingDelay

func WithPacingDelay(d time.Duration) Option

func WithPromptTemplates

func WithPromptTemplates(t PromptTemplates) Option

func WithProtocol

func WithProtocol(p Protocol) Option

func WithRecorder

func WithRecorder(r Recorder) Option

func WithRedact

func WithRedact(fn func(string) string) Option

func WithRequestTimeout

func WithRequestTimeout(d time.Duration) Option

func WithThresholds

func WithThresholds(t Thresholds) Option

func WithVerbose

func WithVerbose(v bool) Option

type Phase

type Phase int

Phase identifies a pipeline stage. It is used by hooks, instrumentation, and per-agent phase skipping.

const (
	PhaseBaseline Phase = iota
	PhaseCrossExamination
	PhaseEvaluation
	PhaseDriftAnalysis
	PhaseGrilling
)

func (Phase) String

func (p Phase) String() string

type PromptResult

type PromptResult struct {
	Before map[string]*Answer
	After  map[string]*Answer
}

PromptResult captures each agent's position before and after the debate.

type PromptTemplates

type PromptTemplates struct {
	// SupervisorSystem is the moderator system prompt. %s is replaced with the
	// comma-joined panelist names.
	SupervisorSystem string
	// CrossExamineLead is formatted with (challenge prompt, schema
	// instruction) when the moderator or a protocol challenges an agent.
	CrossExamineLead string
	// FinalEvalPrompt precedes the schema instruction when re-collecting final
	// positions.
	FinalEvalPrompt string
	// InitialPrompt builds the baseline question prompt.
	InitialPrompt func(question string, options []string, labels []string, instr string) string
	// DebateTrigger builds the moderator's cross-examination briefing.
	DebateTrigger func(question string, names []string, before map[string]*Answer) string
	// GrillProbe builds the adversarial inversion probe.
	GrillProbe func(question, claim string) string
	// NLIProbe builds the consistency-evaluation prompt.
	NLIProbe func(question, r1, r2 string) string
	// ChallengePrompt builds a direct peer challenge used by the non-supervised
	// protocols.
	ChallengePrompt func(from, claim, reasoning string) string
	// ConfidenceProbe asks an agent to self-report confidence in a claim.
	ConfidenceProbe func(claim string) string
}

PromptTemplates lets callers override any or all of the prompts the engine builds. Function fields default to the built-in builders; string fields default to the built-in templates. Zero fields are resolved by New.

func (PromptTemplates) Resolve

func (t PromptTemplates) Resolve() PromptTemplates

Resolve fills zero fields with the built-in defaults.

type Protocol

type Protocol int

Protocol is the cross-examination orchestration strategy.

const (
	// ProtocolSupervised has the moderator agent run the cross-examination.
	ProtocolSupervised Protocol = iota
	// ProtocolAdversarialPairs pairs panelists and runs a direct challenge
	// exchange within each pair.
	ProtocolAdversarialPairs
	// ProtocolRoundRobin has every panelist challenge every other panelist.
	ProtocolRoundRobin
	// ProtocolDevilsAdvocate confronts each panelist with the strongest
	// opposing argument available in the panel.
	ProtocolDevilsAdvocate
)

func (Protocol) String

func (p Protocol) String() string

type QuadrantState

type QuadrantState string

QuadrantState is the 2D classification of an agent based on its confidence (x-axis) and semantic consistency (y-axis) under cross-examination.

const (
	// QuadrantPeak indicates high confidence and high consistency.
	QuadrantPeak QuadrantState = "PEAK (Grounded Knowledge)"
	// QuadrantHallucinating indicates high confidence with fabricated contradictions.
	QuadrantHallucinating QuadrantState = "HALLUCINATING / MALICIOUS"
	// QuadrantKBDeficit indicates consistent but low-confidence answers.
	QuadrantKBDeficit QuadrantState = "KB_DEFICIT (Incomplete Info)"
	// QuadrantWrong indicates low confidence and inconsistent internal logic.
	QuadrantWrong QuadrantState = "WRONG / UNCERTAIN"
)

func ClassifyWith

func ClassifyWith(confScore, consistencyScore float64, thr Thresholds) (QuadrantState, string)

ClassifyWith maps (confidence, consistency) onto the quadrant grid.

func ThresholdClassifier

func ThresholdClassifier(confScore, consistencyScore float64) (QuadrantState, string)

ThresholdClassifier is the default classifier, using the configured confidence/consistency thresholds.

type Recorder

type Recorder interface {
	Record(ctx context.Context, ev Event) error
}

Recorder receives an Event stream for persistence.

type StreamEvent

type StreamEvent struct {
	Type       EventType         `json:"type"`
	Phase      Phase             `json:"phase,omitempty"`
	Agent      string            `json:"agent,omitempty"`
	Prompt     string            `json:"prompt,omitempty"`
	Answer     *Answer           `json:"answer,omitempty"`
	Verdict    *DriftVerdict     `json:"verdict,omitempty"`
	Diagnostic *DiagnosticResult `json:"diagnostic,omitempty"`
	Metadata   map[string]any    `json:"metadata,omitempty"`
	Error      error             `json:"-"`
}

StreamEvent is a single milestone emitted by the streaming API.

type Thresholds

type Thresholds struct {
	// DriftSignificance is the minimum shift (TVD or lexical distance) or
	// confidence erosion required to flag an agent as drifted. Default 0.15.
	DriftSignificance float64
	// Confidence is the x-axis (confidence) threshold for quadrant
	// classification. Default 0.55.
	Confidence float64
	// Consistency is the y-axis (consistency) threshold for quadrant
	// classification. Default 0.60.
	Consistency float64
}

Thresholds are the tunable cut-offs used across drift detection and quadrant classification. All fields are floats in the range [0.0, 1.0].

func (Thresholds) Resolve

func (t Thresholds) Resolve() Thresholds

Resolve fills zero fields with the default values.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL