Documentation
¶
Overview ¶
Package eval provides deterministic contracts and orchestration for Atlas live-agent evaluations.
Index ¶
- Constants
- func EvaluateWorkflow(workflow WorkflowExpectation, events []Event, forjDigest string, ...) (EndpointResult, []EndpointResult)
- func IsResourceExhaustion(err error) bool
- func PromotedEvaluationIDs(suite string) ([]string, error)
- func PromotedEvaluationIDsMatching(filter EvaluationFilter) ([]string, error)
- func SupportedGuidanceProfile(profile string) bool
- type AddHTTPControllerVerifier
- type AgentFailure
- type AgentOutcome
- type AgentPreparation
- type AgentProperties
- type AgentResult
- type AgentSessionIdentity
- type AgentTurn
- type AgentTurnResult
- type ArtifactFile
- type ArtifactManifest
- type ArtifactStore
- type AttemptArtifacts
- func (artifacts *AttemptArtifacts) AppendEvent(event Event) error
- func (artifacts *AttemptArtifacts) Finalize(planDigest, baselineTree, finalTree string) (ArtifactManifest, error)
- func (artifacts *AttemptArtifacts) WriteJSON(name string, value any) error
- func (artifacts *AttemptArtifacts) WriteText(name, content string) error
- type AttemptEnvironment
- type AttemptRequest
- type AttemptResult
- type AttemptScorecard
- type BackendEnvironment
- type BackendRequest
- type BaselineSnapshot
- type Capability
- type CapabilityCoverage
- type CodexAgent
- func (*CodexAgent) Name() string
- func (adapter *CodexAgent) Prepare(_ context.Context, environment RunEnvironment, guidance Guidance) (AgentPreparation, error)
- func (*CodexAgent) Properties(context.Context) (AgentProperties, error)
- func (adapter *CodexAgent) Start(ctx context.Context, agent PreparedAgent) (EvaluationSession, error)
- type CodexCredential
- type CodexOptions
- type CommandRunner
- type CommandSession
- type CoverageCatalog
- type CoverageTier
- type EndpointResult
- type EndpointStatus
- type EvaluationAgent
- type EvaluationDefinition
- type EvaluationFilter
- type EvaluationSession
- type EvaluationStatus
- type EvaluationTaskKind
- type Event
- type EventKind
- type EventSource
- type ExecutionBackend
- type GeneratorRequirement
- type Guidance
- type GuidanceDiagnosticAttempt
- type GuidanceDiagnosticRequest
- type GuidanceDiagnosticResult
- type GuidanceFileIdentity
- type GuidanceResolver
- type Limits
- type LocalDiagnosticTreatmentRequest
- type LocalGuidanceDiagnostic
- type LocalGuidanceDiagnosticOptions
- type LocalGuidanceDiagnosticRequest
- type Milestone
- type PreparationCapabilities
- type PreparationRequest
- type PreparationResult
- type PreparedAgent
- type PreparedProject
- type ProjectChange
- type ProjectGuidanceResolver
- type ProjectPathState
- type ProjectPreparer
- type ProviderTelemetry
- type Redactor
- type Registry
- type RequirementKind
- type ResolvedEvaluation
- type ResolvedPreparationPlan
- type RunEnvironment
- type RunIntent
- type Runner
- type RuntimeIdentity
- type SafeAbstentionVerifier
- type SealedProject
- type SecondaryFailure
- type SoftwareIdentity
- type TriageDisposition
- type TriageRecord
- type TriageReview
- type TriageState
- type TrustedTestFile
- type UnconfinedLocal
- type VerificationInput
- type VerificationResult
- type Verifier
- type VerifierCommands
- type VerifierProject
- type WorkflowExpectation
- type WorkflowRequirement
Constants ¶
const ( // GuidanceProfileNone omits Project-native framework instructions for the control treatment. GuidanceProfileNone = "none" // GuidanceProfileAgents installs only the canonical Project AGENTS.md treatment. GuidanceProfileAgents = "agents" // GuidanceProfileAgentsSkills installs canonical Project instructions and recommended native skills. GuidanceProfileAgentsSkills = "agents-skills" // GuidanceProfileAtlas installs the complete recommended Atlas surface: instructions, skills, and MCP. GuidanceProfileAtlas = "atlas" )
const ( // EventFieldCommandID correlates one trusted command start with its completion. EventFieldCommandID = "command_id" // EventFieldExecutableDigest identifies the resolved executable observed by the supervisor. EventFieldExecutableDigest = "executable_digest" // EventFieldArguments carries a JSON array of arguments excluding the executable. EventFieldArguments = "arguments" // EventFieldExitCode records the decimal process exit code for a completed command. EventFieldExitCode = "exit_code" // EventFieldPath records one Project-relative observed file path. EventFieldPath = "path" )
Variables ¶
This section is empty.
Functions ¶
func EvaluateWorkflow ¶
func EvaluateWorkflow(workflow WorkflowExpectation, events []Event, forjDigest string, capabilities []Capability) (EndpointResult, []EndpointResult)
EvaluateWorkflow checks promoted workflow actions against supervisor-observed command events.
func IsResourceExhaustion ¶
IsResourceExhaustion reports whether an operation failed because its shared storage resource is full.
func PromotedEvaluationIDs ¶
PromotedEvaluationIDs returns promoted evaluation IDs in stable order, optionally limited to one suite.
func PromotedEvaluationIDsMatching ¶
func PromotedEvaluationIDsMatching(filter EvaluationFilter) ([]string, error)
PromotedEvaluationIDsMatching returns promoted evaluation IDs matching every non-empty dimension.
func SupportedGuidanceProfile ¶
SupportedGuidanceProfile reports whether profile names one closed evaluation treatment.
Types ¶
type AddHTTPControllerVerifier ¶
type AddHTTPControllerVerifier struct {
// contains filtered or unexported fields
}
AddHTTPControllerVerifier verifies invoice HTTP behavior independently from GoForj's golden scenario steps.
func NewAddHTTPControllerVerifier ¶
func NewAddHTTPControllerVerifier(runner CommandRunner) *AddHTTPControllerVerifier
NewAddHTTPControllerVerifier creates the promoted verifier with a trusted isolated command runner.
func (*AddHTTPControllerVerifier) Capabilities ¶
func (*AddHTTPControllerVerifier) Capabilities() []Capability
Capabilities returns no agent-observation requirements because outcome checks inspect a sealed tree through supervisor-owned boundaries.
func (*AddHTTPControllerVerifier) ID ¶
func (*AddHTTPControllerVerifier) ID() string
ID returns the promoted verifier contract identity.
func (*AddHTTPControllerVerifier) Verify ¶
func (verifier *AddHTTPControllerVerifier) Verify(ctx context.Context, input VerificationInput) (VerificationResult, error)
Verify checks structure, compilation, registration, and route visibility without comparing candidate source to the golden recipe.
type AgentFailure ¶
type AgentFailure struct {
Outcome AgentOutcome
Err error
}
AgentFailure classifies a failed adapter operation without conflating provider availability with adapter defects.
func (*AgentFailure) Error ¶
func (failure *AgentFailure) Error() string
Error returns the wrapped operation failure.
func (*AgentFailure) Unwrap ¶
func (failure *AgentFailure) Unwrap() error
Unwrap exposes the original operation failure.
type AgentOutcome ¶
type AgentOutcome string
AgentOutcome classifies what happened to the provider-side attempt.
const ( // AgentNotStarted indicates preflight ended before a provider session began. AgentNotStarted AgentOutcome = "not_started" // AgentCompleted indicates the agent reached a terminal response or process exit. AgentCompleted AgentOutcome = "completed" // AgentAbstained indicates an accepted safe clarification ended the attempt. AgentAbstained AgentOutcome = "abstained" // AgentProviderError indicates the provider failed independently from the adapter. AgentProviderError AgentOutcome = "provider_error" // AgentAdapterError indicates the adapter failed independently from the provider. AgentAdapterError AgentOutcome = "adapter_error" // AgentTimeout indicates the attempt exceeded its wall-time budget. AgentTimeout AgentOutcome = "timeout" // AgentCancelled indicates an operator cancelled the attempt. AgentCancelled AgentOutcome = "cancelled" )
type AgentPreparation ¶
type AgentPreparation interface {
Agent() PreparedAgent
Close(context.Context) error
}
AgentPreparation owns private adapter resources acquired before a session starts.
type AgentProperties ¶
type AgentProperties struct {
Properties []Capability
}
AgentProperties declares provider-neutral safety properties enforced by an adapter.
type AgentResult ¶
type AgentResult struct {
Outcome AgentOutcome
Events []Event
Message string
Telemetry *ProviderTelemetry
}
AgentResult records provider completion independently from evaluator validity.
type AgentSessionIdentity ¶
type AgentSessionIdentity struct {
Version string
Model string
ModelProvider string
AuthorityDigest string
SessionDigest string
}
AgentSessionIdentity records effective provider identity established only after a fresh session starts.
type AgentTurnResult ¶
AgentTurnResult records prompt acceptance and any events emitted before it returned.
type ArtifactFile ¶
type ArtifactFile struct {
Path string `json:"path"`
Digest string `json:"digest"`
Size int64 `json:"size"`
Classification string `json:"classification"`
}
ArtifactFile records one retained file's exact identity and classification.
type ArtifactManifest ¶
type ArtifactManifest struct {
SchemaVersion int `json:"schema_version"`
AttemptID string `json:"attempt_id"`
PlanDigest string `json:"plan_digest"`
BaselineTree string `json:"baseline_tree"`
FinalTree string `json:"final_tree"`
Files []ArtifactFile `json:"files"`
Signature string `json:"signature"`
}
ArtifactManifest integrity-checks the complete retained evidence set against later accidental changes.
func ReadVerifiedAttemptSummary ¶
func ReadVerifiedAttemptSummary(directory string, key []byte) (string, ArtifactManifest, error)
ReadVerifiedAttemptSummary authenticates the complete attempt before returning its inert human report.
func VerifyArtifactManifest ¶
func VerifyArtifactManifest(directory string, key []byte) (ArtifactManifest, error)
VerifyArtifactManifest authenticates metadata and every retained file without executing artifact content.
type ArtifactStore ¶
type ArtifactStore struct {
// contains filtered or unexported fields
}
ArtifactStore creates private supervisor-owned attempt directories and manifests with post-run tamper evidence.
func NewArtifactStore ¶
func NewArtifactStore(root string, key []byte, redactor Redactor) (*ArtifactStore, error)
NewArtifactStore requires an integrity key rather than writing unsigned evidence.
func (*ArtifactStore) Begin ¶
func (store *ArtifactStore) Begin(attemptID string) (*AttemptArtifacts, error)
Begin creates one private attempt directory outside any agent-provided path.
type AttemptArtifacts ¶
type AttemptArtifacts struct {
// contains filtered or unexported fields
}
AttemptArtifacts owns append-only evidence for one attempt until finalization.
func (*AttemptArtifacts) AppendEvent ¶
func (artifacts *AttemptArtifacts) AppendEvent(event Event) error
AppendEvent redacts and persists one event before accepting the next sequence item.
func (*AttemptArtifacts) Finalize ¶
func (artifacts *AttemptArtifacts) Finalize(planDigest, baselineTree, finalTree string) (ArtifactManifest, error)
Finalize closes streaming evidence and writes the authenticated manifest last.
func (*AttemptArtifacts) WriteJSON ¶
func (artifacts *AttemptArtifacts) WriteJSON(name string, value any) error
WriteJSON writes one typed artifact after recursively redacting its serialized representation.
func (*AttemptArtifacts) WriteText ¶
func (artifacts *AttemptArtifacts) WriteText(name, content string) error
WriteText writes one bounded inert text artifact selected by the supervisor.
type AttemptEnvironment ¶
type AttemptEnvironment struct {
Intent RunIntent `json:"intent"`
Backend string `json:"backend"`
RequestedShellNetwork string `json:"requested_shell_network"`
EnvironmentKeys []string `json:"environment_keys"`
AgentProperties []Capability `json:"agent_properties,omitempty"`
BackendCapabilities []Capability `json:"backend_capabilities,omitempty"`
Runtime RuntimeIdentity `json:"runtime"`
}
AttemptEnvironment records names and negotiated capabilities without persisting environment values.
type AttemptRequest ¶
type AttemptRequest struct {
AttemptID string
LogicalTrialID string
Intent RunIntent
Definition EvaluationDefinition
Preparation PreparationRequest
GuidanceProfile string
Runtime RuntimeIdentity
}
AttemptRequest contains invocation policy that does not belong in evaluation YAML.
type AttemptResult ¶
type AttemptResult struct {
AttemptID string `json:"attempt_id"`
LogicalTrialID string `json:"logical_trial_id"`
EvaluationID string `json:"evaluation_id"`
PromptDigest string `json:"prompt_digest"`
GuidanceProfile string `json:"guidance_profile"`
GuidanceDigest string `json:"guidance_digest,omitempty"`
GuidanceFiles []GuidanceFileIdentity `json:"guidance_files,omitempty"`
ScenarioID string `json:"scenario_id"`
ScenarioSchema int `json:"scenario_schema,omitempty"`
PlanDigest string `json:"plan_digest,omitempty"`
ScenarioPlanDigest string `json:"scenario_plan_digest,omitempty"`
CatalogDigest string `json:"catalog_digest,omitempty"`
DependencyDigests map[string]string `json:"dependency_digests,omitempty"`
ProjectConfigDigest string `json:"project_config_digest,omitempty"`
EnvironmentDigest string `json:"environment_digest,omitempty"`
PreparedTree string `json:"prepared_tree,omitempty"`
BaselineTree string `json:"baseline_tree,omitempty"`
FinalTree string `json:"final_tree,omitempty"`
ForjExecutable string `json:"forj_executable,omitempty"`
ForjDigest string `json:"forj_digest,omitempty"`
Backend string `json:"backend,omitempty"`
Agent string `json:"agent,omitempty"`
AgentExecutable string `json:"agent_executable,omitempty"`
AgentDigest string `json:"agent_digest,omitempty"`
ProviderAuthorityDigest string `json:"provider_authority_digest,omitempty"`
ProviderSessionDigest string `json:"provider_session_digest,omitempty"`
ProviderTelemetry *ProviderTelemetry `json:"provider_telemetry,omitempty"`
AgentVersion string `json:"agent_version,omitempty"`
Model string `json:"model,omitempty"`
ModelProvider string `json:"model_provider,omitempty"`
Runtime RuntimeIdentity `json:"runtime"`
AgentOutcome AgentOutcome `json:"agent_outcome"`
EvaluationStatus EvaluationStatus `json:"evaluation_status"`
Milestones []Milestone `json:"milestones"`
Verification *VerificationResult `json:"verification,omitempty"`
SecondaryFailures []SecondaryFailure `json:"secondary_failures,omitempty"`
StartedAt time.Time `json:"started_at"`
FinishedAt time.Time `json:"finished_at"`
}
AttemptResult is the complete lifecycle result for one stochastic attempt.
type AttemptScorecard ¶
type AttemptScorecard struct {
EvaluationStatus EvaluationStatus `json:"evaluation_status"`
AgentOutcome AgentOutcome `json:"agent_outcome"`
FrameworkOutcome *EndpointResult `json:"framework_outcome,omitempty"`
WorkflowConformance *EndpointResult `json:"workflow_conformance,omitempty"`
}
AttemptScorecard is the compact machine-readable outcome layer retained beside full run metadata.
type BackendEnvironment ¶
type BackendEnvironment interface {
Environment() RunEnvironment
Baseline(context.Context) (BaselineSnapshot, error)
ObservedEvents(context.Context) ([]Event, error)
Seal(context.Context) (SealedProject, error)
Close(context.Context) error
}
BackendEnvironment owns the isolation resources that enclose one agent attempt.
type BackendRequest ¶
type BackendRequest struct {
Project PreparedProject
ShellNetwork string
Environment []string
CommandLimit int
}
BackendRequest binds one prepared Project to explicit execution policy.
type BaselineSnapshot ¶
BaselineSnapshot is the backend/supervisor-owned baseline captured after treatment setup and before the session starts.
type Capability ¶
type Capability string
Capability identifies one observation or isolation property an evaluation component can prove.
const ( // CapabilityFileReads proves agent file reads through a trusted observation boundary. CapabilityFileReads Capability = "file_reads" // CapabilityFileWrites proves agent file mutations through a trusted observation boundary. CapabilityFileWrites Capability = "file_writes" // CapabilityCommands proves process execution with trusted executable identity and arguments. CapabilityCommands Capability = "commands" // CapabilityMCPToolCalls proves MCP calls through a trusted supervisor-owned interposer. CapabilityMCPToolCalls Capability = "mcp_tool_calls" // CapabilityProcessCleanup proves complete descendant-job termination. CapabilityProcessCleanup Capability = "process_cleanup" // CapabilityCredentialIsolation proves candidate processes cannot reach reusable provider authority. CapabilityCredentialIsolation Capability = "credential_isolation" // CapabilityHostFilesystemIsolation proves candidate processes cannot read or mutate undeclared host paths. CapabilityHostFilesystemIsolation Capability = "host_filesystem_isolation" // CapabilityNetworkEnforcement proves the backend enforces the requested shell network policy. CapabilityNetworkEnforcement Capability = "network_enforcement" // CapabilityVerifierIsolation proves candidate execution cannot mutate verifier code, state, or later phases. CapabilityVerifierIsolation Capability = "verifier_isolation" // CapabilityArtifactIsolation proves candidate processes cannot read signing authority or mutate retained evidence. CapabilityArtifactIsolation Capability = "artifact_isolation" // CapabilityFinalResponseCapture proves the adapter returns the provider's exact terminal response to the verifier. CapabilityFinalResponseCapture Capability = "final_response_capture" )
type CapabilityCoverage ¶
type CapabilityCoverage struct {
ID string `json:"id"`
Area string `json:"area"`
Summary string `json:"summary"`
Tier CoverageTier `json:"tier"`
Evaluations []string `json:"evaluations"`
}
CapabilityCoverage maps one durable framework capability to the evaluations that currently measure it.
func (CapabilityCoverage) Covered ¶
func (coverage CapabilityCoverage) Covered() bool
Covered reports whether at least one promoted evaluation currently measures the capability.
type CodexAgent ¶
type CodexAgent struct {
// contains filtered or unexported fields
}
CodexAgent starts fresh Codex sessions and keeps preparation state private from generic runner contracts.
func NewCodexAgent ¶
func NewCodexAgent(options CodexOptions) (*CodexAgent, error)
NewCodexAgent creates a diagnostic Codex agent without claiming authoritative observation capabilities.
func (*CodexAgent) Name ¶
func (*CodexAgent) Name() string
Name returns the adapter identity recorded by evaluation reports.
func (*CodexAgent) Prepare ¶
func (adapter *CodexAgent) Prepare(_ context.Context, environment RunEnvironment, guidance Guidance) (AgentPreparation, error)
Prepare fingerprints Codex and copies only the credential needed by the private app-server home.
func (*CodexAgent) Properties ¶
func (*CodexAgent) Properties(context.Context) (AgentProperties, error)
Properties reports exact final-response capture while withholding isolation guarantees from the unconfined diagnostic adapter.
func (*CodexAgent) Start ¶
func (adapter *CodexAgent) Start(ctx context.Context, agent PreparedAgent) (EvaluationSession, error)
Start requires the app-server boundary to recheck the prepared launcher path and digest immediately before process startup.
This pre-exec launcher digest check does not make the launcher, its interpreter, or loaded modules immutable, and it leaves a residual check-to-exec race.
type CodexCredential ¶
type CodexCredential struct {
// contains filtered or unexported fields
}
CodexCredential is an immutable credential snapshot shared by every treatment in one logical trial.
func LoadCodexCredential ¶
func LoadCodexCredential(source string) (CodexCredential, error)
LoadCodexCredential freezes one auth document before any treatment can begin.
func NewCodexCredential ¶
func NewCodexCredential(body []byte) (CodexCredential, error)
NewCodexCredential freezes credential bytes supplied by a host-owned read boundary.
func (CodexCredential) Redactor ¶
func (credential CodexCredential) Redactor(redactor Redactor) Redactor
Redactor returns a copy that covers the exact frozen authority used by Codex sessions.
type CodexOptions ¶
type CodexOptions struct {
Executable string
Arguments []string
Model string
ModelProvider string
Credential CodexCredential
CredentialSource string
Environment []string
}
CodexOptions pins the Codex executable, model, provider state, and process environment used by diagnostic sessions.
type CommandRunner ¶
type CommandRunner interface {
// Open creates one disposable verifier session from sealed candidate evidence.
Open(context.Context, VerifierProject) (CommandSession, error)
}
CommandRunner opens verifier-owned black-box checks in an isolated clone of the sealed candidate tree.
type CommandSession ¶
type CommandSession interface {
// WriteFile installs supervisor-owned verifier input inside the disposable Project.
WriteFile(string, []byte) error
// Run executes one allowlisted verifier command.
Run(context.Context, []string) (string, error)
// Close destroys the disposable Project and any supervisor-owned input.
Close(context.Context) error
}
CommandSession executes a bounded sequence against one disposable candidate clone.
type CoverageCatalog ¶
type CoverageCatalog struct {
Capabilities []CapabilityCoverage `json:"capabilities"`
}
CoverageCatalog is the validated framework capability inventory used to expose measured and planned coverage.
func LoadCoverageCatalog ¶
func LoadCoverageCatalog() (CoverageCatalog, error)
LoadCoverageCatalog returns a validated copy of the promoted framework capability inventory.
func (CoverageCatalog) EvaluationIDs ¶
func (catalog CoverageCatalog) EvaluationIDs(tier CoverageTier) ([]string, error)
EvaluationIDs returns the promoted evaluations needed to exercise capabilities through the requested tier.
type CoverageTier ¶
type CoverageTier string
CoverageTier identifies how frequently a capability should participate in live evaluation.
const ( // CoverageTierSmoke identifies release-critical behavior suitable for every evaluation smoke run. CoverageTierSmoke CoverageTier = "smoke" // CoverageTierCore identifies representative framework behavior for the complete core benchmark. CoverageTierCore CoverageTier = "core" // CoverageTierExtended identifies expensive or specialized behavior for scheduled qualification. CoverageTierExtended CoverageTier = "extended" )
type EndpointResult ¶
type EndpointResult struct {
ID string `json:"id"`
Status EndpointStatus `json:"status"`
Details string `json:"details,omitempty"`
Kind RequirementKind `json:"kind,omitempty"`
}
EndpointResult describes one framework, conformance, or contract endpoint.
type EndpointStatus ¶
type EndpointStatus string
EndpointStatus records one independently meaningful verification outcome.
const ( // EndpointPassed indicates that trusted evidence satisfied the endpoint. EndpointPassed EndpointStatus = "passed" // EndpointFailed indicates that complete trusted evidence violated the endpoint. EndpointFailed EndpointStatus = "failed" // EndpointIneligible indicates that required trusted evidence was unavailable. EndpointIneligible EndpointStatus = "ineligible" )
type EvaluationAgent ¶
type EvaluationAgent interface {
Name() string
Properties(context.Context) (AgentProperties, error)
Prepare(context.Context, RunEnvironment, Guidance) (AgentPreparation, error)
Start(context.Context, PreparedAgent) (EvaluationSession, error)
}
EvaluationAgent prepares and starts fresh non-interactive evaluation sessions.
type EvaluationDefinition ¶
type EvaluationDefinition struct {
SchemaVersion int
ID string
Summary string
Suite string
TaskKind EvaluationTaskKind
ProjectScenario string
WorkflowID string
VerifierID string
Limits Limits
Prompt string
PromptDigest string
}
EvaluationDefinition is one resolved manifest and its adjacent natural-language prompt.
func LoadDefinition ¶
func LoadDefinition(directory string) (EvaluationDefinition, error)
LoadDefinition reads one strict evaluation manifest and its adjacent prompt.
func LoadPromotedDefinition ¶
func LoadPromotedDefinition(id string) (EvaluationDefinition, error)
LoadPromotedDefinition reads one reviewed evaluation bundled with Atlas.
type EvaluationFilter ¶
type EvaluationFilter struct {
Suite string
TaskKind EvaluationTaskKind
}
EvaluationFilter selects promoted evaluations by stable measurement dimensions.
type EvaluationSession ¶
type EvaluationSession interface {
Identity() AgentSessionIdentity
Turn(context.Context, AgentTurn) (AgentTurnResult, error)
Wait(context.Context) (AgentResult, error)
Close(context.Context) error
}
EvaluationSession owns one fresh provider session and its complete descendant job.
type EvaluationStatus ¶
type EvaluationStatus string
EvaluationStatus classifies whether trusted verification produced a valid result.
const ( // EvaluationValid indicates complete evidence produced ordinary endpoints. EvaluationValid EvaluationStatus = "valid" // EvaluationValidAbstention indicates the scenario accepted a safe abstention. EvaluationValidAbstention EvaluationStatus = "valid_abstention" // EvaluationNotEvaluated indicates no evidence-valid logical trial began. EvaluationNotEvaluated EvaluationStatus = "not_evaluated" // EvaluationIneligible indicates a required capability was unavailable at preflight. EvaluationIneligible EvaluationStatus = "ineligible" // EvaluationDiagnostic indicates diagnostic artifacts were collected without the evidence required for a valid evaluation. EvaluationDiagnostic EvaluationStatus = "diagnostic" // EvaluationFixtureError indicates Project preparation or starting-state verification failed. EvaluationFixtureError EvaluationStatus = "fixture_error" // EvaluationEvaluatorError indicates capture, verification, or cleanup became unreliable. EvaluationEvaluatorError EvaluationStatus = "evaluator_error" )
type EvaluationTaskKind ¶
type EvaluationTaskKind string
EvaluationTaskKind separates workflow discovery from broader implementation and repair measurements.
const ( // TaskScaffold evaluates whether an agent discovers and uses a framework-owned creation workflow. TaskScaffold EvaluationTaskKind = "scaffold" // TaskFeature evaluates a complete application behavior across one or more framework surfaces. TaskFeature EvaluationTaskKind = "feature" // TaskRepair evaluates diagnosis and correction of an existing broken integration. TaskRepair EvaluationTaskKind = "repair" // TaskAbstention evaluates safe behavior when the available Project evidence is insufficient. TaskAbstention EvaluationTaskKind = "abstention" )
type Event ¶
type Event struct {
Sequence uint64 `json:"sequence"`
Kind EventKind `json:"kind"`
Source EventSource `json:"source"`
Time time.Time `json:"time"`
Fields map[string]string `json:"fields,omitempty"`
}
Event is one ordered provider-neutral observation.
type EventKind ¶
type EventKind string
EventKind is the provider-neutral class of one observed agent action.
const ( // EventFileRead records one observed Project read. EventFileRead EventKind = "file_read" // EventFileWrite records one observed Project mutation. EventFileWrite EventKind = "file_write" // EventCommandStarted records trusted process execution before completion. EventCommandStarted EventKind = "command_started" // EventCommandFinished records trusted process completion. EventCommandFinished EventKind = "command_finished" // EventMCPToolCalled records one trusted MCP request and result classification. EventMCPToolCalled EventKind = "mcp_tool_called" // EventMessage records inert provider text after redaction. EventMessage EventKind = "message" // EventRunFinished records agent session completion. EventRunFinished EventKind = "run_finished" )
type EventSource ¶
type EventSource string
EventSource identifies the observation boundary that produced an event.
const ( // EventSourceSupervisor identifies evidence observed by the execution backend or supervisor. EventSourceSupervisor EventSource = "supervisor" // EventSourceAdapter identifies diagnostic-only telemetry reported by an agent adapter. EventSourceAdapter EventSource = "adapter" )
type ExecutionBackend ¶
type ExecutionBackend interface {
Name() string
Capabilities(context.Context) ([]Capability, error)
Open(context.Context, BackendRequest) (BackendEnvironment, error)
}
ExecutionBackend creates one private execution boundary and reports what it can prove.
type GeneratorRequirement ¶
GeneratorRequirement defines one exact successful GoForj generator action without coupling it to a shell spelling.
type Guidance ¶
Guidance is the exact native instructions, skills, and MCP selection installed for one treatment.
type GuidanceDiagnosticAttempt ¶
type GuidanceDiagnosticAttempt struct {
Profile string `json:"profile"`
Result AttemptResult `json:"result"`
Error string `json:"error,omitempty"`
Cause error `json:"-"`
}
GuidanceDiagnosticAttempt retains one treatment's result and any operational error independently.
type GuidanceDiagnosticRequest ¶
type GuidanceDiagnosticRequest struct {
LogicalTrialID string
Definition EvaluationDefinition
DestinationRoot string
ForjExecutable string
Environments map[string][]string
Runtime RuntimeIdentity
Profiles []string
TreatmentBoundary func(context.Context) error
}
GuidanceDiagnosticRequest identifies one ordered pair of guidance treatments.
type GuidanceDiagnosticResult ¶
type GuidanceDiagnosticResult struct {
LogicalTrialID string `json:"logical_trial_id"`
Attempts []GuidanceDiagnosticAttempt `json:"attempts"`
}
GuidanceDiagnosticResult contains both treatments even when one attempt fails operationally.
type GuidanceFileIdentity ¶
GuidanceFileIdentity records one native instruction projection without retaining its content in run metadata.
type GuidanceResolver ¶
type GuidanceResolver interface {
Resolve(context.Context, string, PreparationResult) (Guidance, error)
}
GuidanceResolver projects one named treatment from the prepared Project's trusted identity.
type LocalDiagnosticTreatmentRequest ¶
type LocalDiagnosticTreatmentRequest struct {
EvaluationID string
GuidanceProfile string
DestinationRoot string
Environment []string
LogicalTrialID string
}
LocalDiagnosticTreatmentRequest identifies one guidance treatment without requiring a paired comparison.
type LocalGuidanceDiagnostic ¶
type LocalGuidanceDiagnostic struct {
// contains filtered or unexported fields
}
LocalGuidanceDiagnostic owns Atlas's standard paired local diagnostic wiring. Hosts retain their Project preparation, private environments, runtime identity, and filesystem lifecycle.
func NewLocalGuidanceDiagnostic ¶
func NewLocalGuidanceDiagnostic(options LocalGuidanceDiagnosticOptions) (*LocalGuidanceDiagnostic, error)
NewLocalGuidanceDiagnostic creates the Atlas-owned registry, promoted verifier, Codex adapter, unconfined backend, artifact store, and runner.
func (*LocalGuidanceDiagnostic) Run ¶
func (diagnostic *LocalGuidanceDiagnostic) Run(ctx context.Context, request LocalGuidanceDiagnosticRequest) (GuidanceDiagnosticResult, error)
Run evaluates the promoted definition with fresh no-guidance and AGENTS.md treatments.
func (*LocalGuidanceDiagnostic) RunTreatment ¶
func (diagnostic *LocalGuidanceDiagnostic) RunTreatment(ctx context.Context, request LocalDiagnosticTreatmentRequest) (GuidanceDiagnosticAttempt, error)
RunTreatment evaluates one promoted definition and guidance profile through the same diagnostic runner used by comparisons.
type LocalGuidanceDiagnosticOptions ¶
type LocalGuidanceDiagnosticOptions struct {
WorkRoot string
ArtifactRoot string
ArtifactKey []byte
Redactor Redactor
Preparer ProjectPreparer
Codex CodexOptions
GoExecutable string
ForjExecutable string
VerifierEnvironment []string
// VerifierModuleProxy is the host-owned read-only Go module proxy for verifier commands.
VerifierModuleProxy string
Runtime RuntimeIdentity
}
LocalGuidanceDiagnosticOptions supplies host-owned boundaries to the standard local diagnostic service.
type LocalGuidanceDiagnosticRequest ¶
type LocalGuidanceDiagnosticRequest struct {
EvaluationID string
DestinationRoot string
Environments map[string][]string
LogicalTrialID string
Profiles []string
TreatmentBoundary func(context.Context) error
}
LocalGuidanceDiagnosticRequest identifies one paired treatment while keeping host-private environments outside Atlas policy wiring.
type Milestone ¶
type Milestone string
Milestone is one monotonic trusted lifecycle boundary.
const ( // MilestonePreflight records successful contract and capability resolution. MilestonePreflight Milestone = "preflight" // MilestoneProviderSessionStarted records acquisition of a provider session. MilestoneProviderSessionStarted Milestone = "provider_session_started" // MilestonePromptDelivered records successful prompt submission. MilestonePromptDelivered Milestone = "prompt_delivered" // MilestoneFirstAgentAction records the first trusted action or terminal response. MilestoneFirstAgentAction Milestone = "first_agent_action" // MilestoneAgentTerminal records provider-side terminal completion. MilestoneAgentTerminal Milestone = "agent_terminal" // MilestoneEvaluationTerminal records verifier and cleanup completion. MilestoneEvaluationTerminal Milestone = "evaluation_terminal" )
type PreparationCapabilities ¶
type PreparationCapabilities struct {
ScenarioSchemaVersions []int
}
PreparationCapabilities describe the schemas and controls a Project preparer supports.
type PreparationRequest ¶
type PreparationRequest struct {
ScenarioID string
DestinationRoot string
ForjExecutable string
OrchestrationID string
Environment []string
}
PreparationRequest identifies trusted scenario inputs without carrying reusable authority.
type PreparationResult ¶
type PreparationResult struct {
ResolutionID string
ProjectRoot string
ScenarioID string
ScenarioSchema int
PlanDigest string
CatalogDigest string
BaselineTree string
ForjExecutable string
ForjDigest string
OwnedPaths []string
}
PreparationResult records the exact Project and tool identities returned by preparation.
type PreparedAgent ¶
type PreparedAgent struct {
Name string
Executable string
ExecutableDigest string
AuthorityDigest string
Model string
Environment RunEnvironment
}
PreparedAgent records attributable agent identity and private configuration.
type PreparedProject ¶
type PreparedProject interface {
Result() PreparationResult
Close(context.Context) error
}
PreparedProject owns one prepared Project until the supervisor closes it.
type ProjectChange ¶
type ProjectChange struct {
Path string `json:"path"`
Before ProjectPathState `json:"before"`
After ProjectPathState `json:"after"`
}
ProjectChange records one supervisor-computed path change between the treatment baseline and sealed Project.
type ProjectGuidanceResolver ¶
type ProjectGuidanceResolver struct{}
ProjectGuidanceResolver composes treatments from the exact prepared Project rather than caller-supplied facts.
func (ProjectGuidanceResolver) Resolve ¶
func (ProjectGuidanceResolver) Resolve(ctx context.Context, profileName string, preparation PreparationResult) (Guidance, error)
Resolve discovers the prepared Project and delegates to the canonical profile composer.
type ProjectPathState ¶
type ProjectPathState struct {
Kind string `json:"kind,omitempty"`
Digest string `json:"digest,omitempty"`
Mode uint32 `json:"mode,omitempty"`
}
ProjectPathState identifies one Project path at a sealed snapshot. A zero value means the path was absent.
type ProjectPreparer ¶
type ProjectPreparer interface {
Capabilities(context.Context) (PreparationCapabilities, error)
Resolve(context.Context, PreparationRequest) (ResolvedPreparationPlan, error)
Prepare(context.Context, PreparationRequest, ResolvedPreparationPlan) (PreparedProject, error)
MaterializeGuidance(context.Context, PreparedProject, Guidance) (Guidance, error)
}
ProjectPreparer is the Atlas-owned boundary implemented by GoForj.
type ProviderTelemetry ¶
type ProviderTelemetry struct {
EventsObserved uint64 `json:"events_observed"`
BytesObserved uint64 `json:"bytes_observed"`
EventsDropped uint64 `json:"events_dropped,omitempty"`
BytesDropped uint64 `json:"bytes_dropped,omitempty"`
NotificationsDropped uint64 `json:"notifications_dropped,omitempty"`
NotificationBytesDropped uint64 `json:"notification_bytes_dropped,omitempty"`
CommandsObserved uint64 `json:"commands_observed"`
}
ProviderTelemetry describes bounded diagnostic retention without promoting adapter observations to trusted evidence.
func (ProviderTelemetry) Truncated ¶
func (telemetry ProviderTelemetry) Truncated() bool
Truncated reports whether any provider-originated evidence was omitted before normalization.
type Redactor ¶
type Redactor struct {
// contains filtered or unexported fields
}
Redactor removes known credentials and common secret-bearing values before persistence.
func NewRedactor ¶
NewRedactor snapshots non-empty secret values in longest-first order.
func (Redactor) Event ¶
Event redacts a copy so callers cannot mutate or later reveal the persisted evidence.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry owns the promoted workflow and verifier contracts available to manifest resolution.
func NewRegistry ¶
func NewRegistry(workflows []WorkflowExpectation, verifiers []Verifier) (*Registry, error)
NewRegistry validates and snapshots promoted contracts so later caller mutation cannot change resolution.
func (*Registry) Resolve ¶
func (registry *Registry) Resolve(definition EvaluationDefinition) (ResolvedEvaluation, error)
Resolve binds exact references and computes capabilities without allowing manifest overrides.
type RequirementKind ¶
type RequirementKind string
RequirementKind separates framework workflow gates from optional quality signals.
const ( // RequirementWorkflow is a declared framework action required for conformance. RequirementWorkflow RequirementKind = "workflow" // RequirementQuality is recorded for calibration but does not fail conformance. RequirementQuality RequirementKind = "quality" )
type ResolvedEvaluation ¶
type ResolvedEvaluation struct {
Definition EvaluationDefinition
Workflow WorkflowExpectation
Verifier Verifier
Capabilities []Capability
}
ResolvedEvaluation binds one manifest to exact promoted contracts and their capability union.
type ResolvedPreparationPlan ¶
type ResolvedPreparationPlan struct {
ResolutionID string
ScenarioID string
ScenarioSchema int
PlanDigest string
ScenarioPlanDigest string
CatalogDigest string
ForjDigest string
EnvironmentDigest string
DependencyDigests map[string]string
ProjectConfiguration []byte
TargetOmitted bool
}
ResolvedPreparationPlan is an immutable data contract authenticated by the trusted caller.
type RunEnvironment ¶
RunEnvironment is the backend-owned namespace presented to an agent adapter.
type RunIntent ¶
type RunIntent string
RunIntent determines whether unavailable evidence blocks execution or remains an explicit diagnostic limitation.
const ( // IntentAuthoritative requires every imported observation capability before Project mutation. IntentAuthoritative RunIntent = "authoritative" // IntentDiagnostic permits useful outcome evaluation while marking unsupported evidence endpoints ineligible. IntentDiagnostic RunIntent = "diagnostic" )
type Runner ¶
type Runner struct {
Registry *Registry
Preparer ProjectPreparer
Backend ExecutionBackend
Agent EvaluationAgent
Guidance GuidanceResolver
Artifacts *ArtifactStore
Now func() time.Time
}
Runner coordinates trusted preparation, one agent session, verification, and cleanup.
func (Runner) Run ¶
func (runner Runner) Run(ctx context.Context, request AttemptRequest) (result AttemptResult, runErr error)
Run executes one logical attempt while preserving agent outcome and evaluator failures separately.
func (Runner) RunGuidanceDiagnostic ¶
func (runner Runner) RunGuidanceDiagnostic(ctx context.Context, request GuidanceDiagnosticRequest) (GuidanceDiagnosticResult, error)
RunGuidanceDiagnostic runs two isolated treatments against the same promoted definition.
type RuntimeIdentity ¶
type RuntimeIdentity struct {
Supervisor SoftwareIdentity `json:"supervisor"`
Framework SoftwareIdentity `json:"framework"`
GoVersion string `json:"go_version"`
GoExecutableDigest string `json:"go_executable_digest,omitempty"`
GoRootDigest string `json:"go_root_digest,omitempty"`
GOOS string `json:"goos"`
GOARCH string `json:"goarch"`
}
RuntimeIdentity records the supervisor, framework, and selected Go launcher provenance needed to reconstruct a diagnostic environment.
type SafeAbstentionVerifier ¶
type SafeAbstentionVerifier struct{}
SafeAbstentionVerifier accepts a precise execution-mode clarification without allowing speculative Project changes.
func NewSafeAbstentionVerifier ¶
func NewSafeAbstentionVerifier() *SafeAbstentionVerifier
NewSafeAbstentionVerifier creates the promoted unknown-framework-shape verifier.
func (*SafeAbstentionVerifier) Capabilities ¶
func (*SafeAbstentionVerifier) Capabilities() []Capability
Capabilities requires an adapter that captures the provider's exact terminal response.
func (*SafeAbstentionVerifier) ID ¶
func (*SafeAbstentionVerifier) ID() string
ID returns the immutable verifier contract identifier.
func (*SafeAbstentionVerifier) Verify ¶
func (*SafeAbstentionVerifier) Verify(_ context.Context, input VerificationInput) (VerificationResult, error)
Verify accepts only a closed, decision-relevant clarification without authored Project changes.
type SealedProject ¶
SealedProject is the immutable verifier input captured after every agent descendant has stopped.
type SecondaryFailure ¶
type SecondaryFailure struct {
Phase string `json:"phase"`
Message string `json:"message"`
// Cause retains operational identity for callers without serializing host error details.
Cause error `json:"-"`
}
SecondaryFailure preserves evaluator failures without overwriting the agent outcome.
type SoftwareIdentity ¶
type SoftwareIdentity struct {
Module string `json:"module"`
Version string `json:"version"`
Commit string `json:"commit,omitempty"`
Dirty bool `json:"dirty,omitempty"`
}
SoftwareIdentity records a retrievable release or development revision without relying on a temporary executable path.
type TriageDisposition ¶
type TriageDisposition string
TriageDisposition is a human-confirmed product cause kept outside immutable attempt artifacts.
type TriageRecord ¶
type TriageRecord struct {
State TriageState `json:"state"`
SuspectedCause string `json:"suspected_cause,omitempty"`
Confidence string `json:"confidence,omitempty"`
EvidenceNeeded []string `json:"evidence_needed,omitempty"`
}
TriageRecord keeps automated suspicion distinct from a confirmed product cause.
type TriageReview ¶
type TriageReview struct {
AttemptID string `json:"attempt_id"`
Disposition TriageDisposition `json:"disposition"`
Reviewer string `json:"reviewer"`
ReviewedAt time.Time `json:"reviewed_at"`
}
TriageReview associates a later human disposition with an authenticated attempt without rewriting its artifacts.
func NewTriageReview ¶
func NewTriageReview(attemptID string, disposition TriageDisposition, reviewer string, reviewedAt time.Time) (TriageReview, error)
NewTriageReview creates an external review record; callers must store it in their review system rather than mutating signed attempt artifacts.
type TriageState ¶
type TriageState string
TriageState records whether a human has established the cause of a failed attempt.
const ( // TriageUnreviewed indicates that no product disposition has been confirmed. TriageUnreviewed TriageState = "unreviewed" // TriageNeedsEvidence indicates that retained evidence cannot yet support a disposition. TriageNeedsEvidence TriageState = "needs-evidence" )
type TrustedTestFile ¶
TrustedTestFile retains one immutable pre-agent test for isolated verifier clones.
type UnconfinedLocal ¶
type UnconfinedLocal struct {
WorkRoot string
}
UnconfinedLocal provides disposable process state without claiming host, command, filesystem, or network isolation.
func (UnconfinedLocal) Capabilities ¶
func (UnconfinedLocal) Capabilities(context.Context) ([]Capability, error)
Capabilities intentionally returns none because host-local process execution is not authoritative evidence.
func (UnconfinedLocal) Name ¶
func (UnconfinedLocal) Name() string
Name returns the backend identity recorded in diagnostic artifacts.
func (UnconfinedLocal) Open ¶
func (backend UnconfinedLocal) Open(_ context.Context, request BackendRequest) (BackendEnvironment, error)
Open creates private agent state outside the candidate-writable Project tree.
type VerificationInput ¶
type VerificationInput struct {
ProjectRoot string
BaselineTree string
FinalTree string
// BaselineTests contains supervisor-captured tests from before the agent received the Project.
BaselineTests []TrustedTestFile
// Changes is the supervisor-computed, path-level projection of the sealed Project delta.
Changes []ProjectChange
Events []Event
// FinalResponse is the exact terminal provider response captured by the adapter.
FinalResponse string
}
VerificationInput contains sealed Project and evidence identities, never candidate verifier code.
type VerificationResult ¶
type VerificationResult struct {
FrameworkOutcome EndpointResult `json:"framework_outcome"`
WorkflowConformance EndpointResult `json:"workflow_conformance"`
Contract *EndpointResult `json:"contract,omitempty"`
Abstention *EndpointResult `json:"abstention,omitempty"`
Checks []EndpointResult `json:"checks,omitempty"`
}
VerificationResult separates framework behavior from workflow conformance.
type Verifier ¶
type Verifier interface {
ID() string
Capabilities() []Capability
Verify(context.Context, VerificationInput) (VerificationResult, error)
}
Verifier is one promoted deterministic outcome contract.
func PromotedVerifiers ¶
func PromotedVerifiers(runner CommandRunner) []Verifier
PromotedVerifiers returns every live verifier that can run through the selected isolated command boundary.
type VerifierCommands ¶
type VerifierCommands struct {
WorkRoot string
GoExecutable string
ForjExecutable string
Environment []string
// ModuleProxy is the host-owned read-only Go module proxy exposed to verifier commands.
ModuleProxy string
}
VerifierCommands executes an allowlisted toolchain in a disposable copy of the candidate Project.
func (VerifierCommands) Open ¶
func (runner VerifierCommands) Open(ctx context.Context, project VerifierProject) (CommandSession, error)
Open clones the sealed Project for exactly one verifier phase, so no phase can observe another phase's candidate-controlled mutation.
type VerifierProject ¶
type VerifierProject struct {
Root string
BaselineTests []TrustedTestFile
BaselineTestExclusions []string
}
VerifierProject combines the sealed candidate with immutable pre-agent tests restored into its private clone.
type WorkflowExpectation ¶
type WorkflowExpectation struct {
ID string
Requirements []WorkflowRequirement
Generators []GeneratorRequirement
}
WorkflowExpectation is a promoted and versioned framework workflow contract.
func PromotedWorkflows ¶
func PromotedWorkflows() []WorkflowExpectation
PromotedWorkflows returns the versioned workflow contracts available to live evaluations.
type WorkflowRequirement ¶
type WorkflowRequirement struct {
ID string
Kind RequirementKind
Capability Capability
Description string
Paths []string
}
WorkflowRequirement defines one typed, observation-backed workflow expectation.
Source Files
¶
- add_http_controller.go
- application_behavior_probes.go
- artifact_repair.go
- artifacts.go
- artifacts_open_unix.go
- cache_behavior.go
- codex.go
- contracts.go
- coverage.go
- diagnostic.go
- diff.go
- guidance.go
- invoice_behavior.go
- local_guidance_diagnostic.go
- manifest.go
- promoted.go
- redact.go
- registry.go
- report.go
- resource_exhaustion.go
- resource_exhaustion_unix.go
- runner.go
- safe_abstention.go
- surface_behavior_probes.go
- surface_contracts.go
- surface_verifier.go
- unconfined.go
- verifier_commands.go
- workflow.go