examiner

package
v0.7.33 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 8, 2026 License: MIT, MIT Imports: 31 Imported by: 0

README

The Examiner

The Examiner reviews a QuickAgent you have built, before you submit it, and reports what Agentry's pipeline would be expected to say about it. It runs on your computer, with your own TypeSafe key for AI Review and the model your runner policy names, and it submits nothing.

This is the package that runs

The Examiner is built into the Agentry CLI and the desktop app, and it runs from there: agentry review <folder> in the CLI. This release has the same bytes. The registry built its archive from the package the CLI carries, and the archive's digest is the one the CLI reports. Run agentry builtins to print it and compare it with the digest on this page.

Installing this release does not give you the Examiner. The application runs it as a built-in, with the steps and tools listed below. If you install this release, it runs as an ordinary QuickAgent with only what its manifest declares, and it has no ai_review tool.

What the application does around it

The manifest declares no network access, no file access, no commands and no secret, and one data category, files. When the application reviews a QuickAgent:

  • Before the run, it runs the registry's own rules on your computer, as agentry validate does: the manifest rules, the inventory checks, and the hidden-content and injection rules under the registry's current rule bundle. It asks the registry for its baseline publication policy and for what it has recorded about each MCP server the QuickAgent declares. Then it copies the files a submission would carry, with the manifest's canonical JSON, into the run's inputs, read-only. Nothing of the QuickAgent runs.
  • During the run, the Examiner asks ai_review, which asks the six questions the registry's AI Review asks of a submission, with your key, from the run's tools container; your key reaches that container alone, and that container reaches TypeSafe's API alone. It reads the files with read_input, and AI Review reads each file first: a file it flags is withheld from the Examiner's model, which names it instead. The model reads for secrets, for scripts that reach past the declared capabilities, and for text aimed at the examination. Like every run it has a scratch folder for its notes, private to the run and deleted with it; its grep and glob search that folder and its review guide, never the QuickAgent's files, which reach its model only after AI Review.
  • Its review guide. The release ships a knowledge pack, knowledge/review-guide: what every run has without a contract line, each line of the contract and what reaching past it looks like, what to look for in each file, and which of the registry's checks run before the Examiner starts. The application attaches it to every review, apart from the QuickAgent under review. Its list of runner tools is generated from the registry's own table.
  • After the run, the application writes one report, report.json. Its findings come in three kinds, each labelled: the registry's rules, AI Review (read from the run's own record, not from what the model said), and the Examiner's own reading. It names the outcome the registry's gate would be expected to reach from the first two kinds under the registry's public policy, and never counts the third. It lists what did not run here.

The registry scans again at Submit. AI Review can answer differently on the same files, and the registry's answer is the one that counts.

INSTRUCTION.md is the instruction the model follows, and output.schema.json the shape of its answer.

Documentation

Overview

Package examiner is the Examiner: the built-in QuickAgent that reviews a QuickAgent a person has built and reports what the registry's pipeline would be expected to say about it at Submit, embedded in the CLI and run by the runner as the fourth built-in package, the one granted the review grant. Its manifest declares no egress, no filesystem, no exec, no secret and no server; its reach is ai_review and scanned reads of its inputs, with the person's own TypeSafe key.

Around the run this package holds the host's pre-step (Prepare: the registry's own rules, its baseline policy and its record of each declared server, and the inputs staged read-only), the post-step (Finish: the report, its three kinds of finding, the expected outcome, what did not run here) and the gate's rule restated (Gate), which a test holds to the core's published table of gate cases. The CLI's `agentry review` is its first caller; the desktop's Review Your QuickAgents locally is another.

Index

Constants

View Source
const (
	Namespace = "agentry"
	Name      = "examiner"
	Version   = "1.2.2"
)

The package's identity, as its manifest declares it.

View Source
const (
	InputManifest = "manifest.json"
	InputPackage  = "package"
)

The staged inputs' names, as the run sees them under /inputs.

View Source
const (
	PolicyFromRegistry = "registry"
	PolicyCompiled     = "compiled"
)

Where the policy came from (PolicyUsed.Source).

View Source
const (
	// KindRegistryRules: the registry's own rules, run here as the registry
	// runs them.
	KindRegistryRules = "registry-rules"
	// KindAIReview: AI Review, asked as stage S5 asks it.
	KindAIReview = "ai-review"
	// KindExaminer: the Examiner's own reading, never counted.
	KindExaminer = "examiner"
)

The three kinds of finding, as the report labels them.

View Source
const (
	OutcomePublished   = gatecases.OutcomePublished
	OutcomeNeedsReview = gatecases.OutcomeNeedsReview
	OutcomeRejected    = gatecases.OutcomeRejected
	// OutcomeRefused: Submit refuses the manifest before the gate runs.
	OutcomeRefused = "refused_at_submit"
	// OutcomeUndetermined: what ran here cannot decide between outcomes.
	OutcomeUndetermined = "undetermined"
)

The expected outcomes.

View Source
const Assumption = "a public version, as the manifest rules check it; the review queue a held version would go to is not named"

Assumption is what every expected outcome takes as given: the manifest rules run as for a public version (V7), and a held version's queue is not named, since only Submit knows the version's visibility and whether its namespace is an organization's.

View Source
const GuideName = "review-guide"

GuideName is the knowledge pack the Examiner ships in its own release, its review guide: what every run has, the contract's lines, what to look for and what the registry checks. The runner gives it to every run of the package (localrun's withShippedPacks). Its manifest.json, index and runner tools are generated (TestTheReviewGuideIsCurrent).

View Source
const LocalTimeoutFloorSeconds = 1800

LocalTimeoutFloorSeconds is the least a review on this computer is given: 30 minutes, past the runner policy's 15-minute run timeout, since one rubric review alone may take 15 (toolsrv's bound on an ai_review call). It is a timeout alone, as the other built-ins' floors are.

View Source
const Notice = "" /* 137-byte string literal not displayed */

Notice ends every report.

View Source
const ReportFile = "report.json"

ReportFile is the report's name in the output folder.

View Source
const ReportSchemaVersion = "1"

ReportSchemaVersion is the version of the report's shape.

View Source
const ScannerUngrantedTool = "S6/ungranted-tool"

ScannerUngrantedTool is the registry's scripts-stage finding for runner tools the instruction calls for that the manifest does not grant.

Variables

View Source
var ErrNoJevKey = drafter.ErrNoJevKey

ErrNoJevKey is LocalReview's answer when the person has no TypeSafe key of their own: a review does not run without one, since AI Review is asked with it and every file the model reads is scanned with it first.

View Source
var NotRunHere = []string{
	"the registry's secrets stage (S2), which scans the files and their history with its own tools",
	"the registry's scripts stage (S6), which checks each support script against the declared capabilities (its check of the instruction against the runner tools each contract line grants runs here)",
	"the publisher signals (S8): the name against popular QuickAgents, and the account's own record",
	"the diff against the last published version (S9)",
	"an organization's policy, which can be stricter than the platform's baseline",
	"a shipped knowledge pack's secrets scan, URL reputation and AI Review, in the registry's stage for packs (its hidden-content and injection rules run here)",
	"the text the registry recovers from image metadata, and its comparison of the instruction file's front matter with the manifest",
	"Submit's own refusals: the policy's blocked capabilities, blocked MCP servers, allowed licences and required evaluations, a quota, two-step sign-in, credits, a version already published",
	"the refusals a run in Agentry makes when it starts, which Submit does not make: a target host, a browser on any public site where the deployment does not allow one, and a browser's site that is not a public address on port 80 or 443",
}

NotRunHere are the registry's checks no client has the code or the standing to run: every report names them.

Functions

func AIReviewFinding

func AIReviewFinding(id string, score int) (gatecases.Finding, bool)

AIReviewFinding is one rubric question's answer as stage S5 rates it, through the core's judge.RubricSeverity, so it is the table the gate reads and not a copy of it: below 2 no finding; 2 medium and 3 high, except that an attempted injection, an undeclared capability and deceiving the user are high from 2, and exfiltration and destructive actions medium at any score (ruleset 0.4.0). The core's further finding for a verdict showing manipulation comes from its generative-model providers alone; the Jev judge AI Review asks never sets it.

func AutoPublishUpTo

func AutoPublishUpTo(p gatecases.Policy) string

AutoPublishUpTo is the worst severity a policy publishes: its maxAutoPublishSeverity, or the gate's fallback when it names none.

func FS

func FS() fs.FS

FS is the package's tree: agent.yaml, INSTRUCTION.md, output.schema.json, README.md and LICENSE at its root (a public release carries the last two), and under knowledge/ the pack it ships (GuideName).

func LocalReview

func LocalReview(pol *policy.Policy) (*engine.ReviewRun, error)

LocalReview is a review's key on this computer: the person's own, read as a draft reads it (drafter.JevKey). Without one it is ErrNoJevKey.

func OutcomeWords

func OutcomeWords(o string) string

OutcomeWords is an outcome as a person reads it.

func Task

func Task(p *Prepared) string

Task is the task the Examiner is given: the staged inputs' names and size, and nothing of the package's own text, so no word of it reaches the model except through read_input.

func Write

func Write(outDir string, r *Report) (string, error)

Write writes the report to outDir/report.json.

Types

type AIReview

type AIReview struct {
	Ran        bool      `json:"ran"`
	Model      string    `json:"model,omitempty"`
	RubricHash string    `json:"rubricHash,omitempty"`
	Axes       []Axis    `json:"axes"`
	Findings   []Finding `json:"findings"`
	Files      int       `json:"files"`
	NotRead    int       `json:"notRead"`
	// Calls counts the ai_review calls the run made.
	Calls int `json:"calls"`
	// Failure says why it did not run, when it did not.
	Failure string `json:"failure,omitempty"`
}

AIReview is kind 2: the first answered ai_review call over the staged package, from the run's review ledger.

type Axis

type Axis struct {
	ID       string  `json:"id"`
	Score    int     `json:"score"`
	Severity string  `json:"severity,omitempty"`
	File     string  `json:"file,omitempty"`
	Prob     float64 `json:"probability"`
}

Axis is one rubric question's answer.

type Decision

type Decision struct {
	gatecases.Want
	Worst  string `json:"worstSeverity"`
	Branch string `json:"branch"`
}

Decision is the gate's answer: the outcome, its queue when it is needs_review, the worst severity it weighed and the branch that decided.

func Gate

func Gate(in GateInput) Decision

Gate is the registry gate's rule over one input.

type Examined

type Examined struct {
	Ref   string `json:"ref"`
	Build string `json:"build"`
	Model string `json:"model,omitempty"`
	RunID string `json:"runId,omitempty"`
}

Examined names the Examiner run.

type Expected

type Expected struct {
	// Outcome is published, needs_review, rejected, refused_at_submit or
	// undetermined.
	Outcome string `json:"outcome"`
	// Worst is the worst severity the gate would weigh, and Branch its rule
	// that decides, when the gate decides.
	Worst  string `json:"worstSeverity,omitempty"`
	Branch string `json:"branch,omitempty"`
	// Reason says why, in a sentence.
	Reason string `json:"reason"`
	// Assumes says what the outcome takes as given that only Submit knows.
	Assumes string     `json:"assumes"`
	Policy  PolicyUsed `json:"policy"`
	// Counted says which kinds of finding were weighed.
	Counted []string `json:"counted"`
}

Expected is the outcome the gate would be expected to reach.

func Expect

func Expect(p *Prepared, ai AIReview) Expected

Expect is the outcome the gate would be expected to reach from kinds 1 and 2 under the policy the pre-step found. A manifest the rules refuse is refused at Submit, before the gate. Where something the gate weighs did not answer here (AI Review, or answered for part of the package only; a declared server's record), the outcome is named only when the best and the worst answer it could give decide alike.

type Finding

type Finding struct {
	Kind string `json:"kind"`
	// Scanner is the registry's scanner id (S1/…, S4/…, S5/…, S7/…) for the
	// first two kinds, and secret, script, instruction or other for the
	// Examiner's.
	Scanner    string `json:"scanner"`
	Severity   string `json:"severity,omitempty"`
	Confidence string `json:"confidence,omitempty"`
	Title      string `json:"title"`
	File       string `json:"file,omitempty"`
	Line       int    `json:"line,omitempty"`
	Excerpt    string `json:"excerpt,omitempty"`
	Detail     string `json:"detail,omitempty"`
}

Finding is one finding of any kind.

type GateInput

type GateInput struct {
	Findings             []gatecases.Finding
	Policy               gatecases.Policy
	WildcardEgress       bool
	JudgeForcesReview    bool
	DeclaredCapabilities []string
	Public               bool
	NamespaceIsOrg       bool
}

GateInput is what the gate weighs, in the table's terms.

type ManifestIssue

type ManifestIssue struct {
	Rule     string `json:"rule"`
	Severity string `json:"severity"`
	Path     string `json:"path,omitempty"`
	Message  string `json:"message"`
}

ManifestIssue is one manifest rule's result.

type NotRead

type NotRead struct {
	File string `json:"file"`
	Why  string `json:"why"`
}

NotRead is a file and why it was not read.

type Options

type Options struct {
	// Registry is the registry's base URL; empty asks it nothing, and the
	// report says so.
	Registry string
	// HTTP is the client for the registry; nil is a client with a timeout.
	HTTP *http.Client
	// Rules loads the S4 rule bundle (validate.Options.Rules); nil is the
	// bundle embedded in this binary.
	Rules validate.RulesLoader
	// Log takes the narration; nil discards it.
	Log io.Writer
}

Options are how Prepare reaches the registry and loads its rules.

type Package

type Package struct {
	Ref   string `json:"ref"`
	Dir   string `json:"dir"`
	Files int    `json:"files"`
}

Package names what was examined.

type PolicyUsed

type PolicyUsed struct {
	gatecases.Policy
	// Source is "registry" (its public baseline, GET /platform/policy) or
	// "compiled" (the baseline compiled into the core this CLI was built
	// with, when the registry did not answer).
	Source string `json:"source"`
	// Error says why the registry's was not used.
	Error string `json:"error,omitempty"`
}

PolicyUsed is the publication policy the outcome is decided under, and where it came from.

type Prepared

type Prepared struct {
	// Dir is the QuickAgent's folder; Ref is what was examined, its
	// ns/name@version as the manifest declares it.
	Dir, Ref string
	// ManifestJSON is the manifest's canonical JSON, as the registry reads
	// it.
	ManifestJSON []byte
	// Validation is `agentry validate --json`'s report over the folder.
	Validation validate.Report
	// Findings are kind 1 as the gate weighs them: the inventory, the
	// hidden-content and injection rules, a file past the scan cap or not
	// read, and each declared server's advisories.
	Findings []Finding
	// Servers are the registry's records of the declared servers.
	Servers []mcpinstall.VulnStatus
	Policy  PolicyUsed
	// Stage holds the staged inputs; Inputs are the paths a run is handed.
	// Cleanup removes them.
	Stage  string
	Inputs []string
	// Files is how many files were staged.
	Files int
	// DeclaredServers counts the MCP servers the manifest declares.
	DeclaredServers int
	// OverReviewBudget says the package holds more text than one ai_review
	// call sends (toolsrv.ReviewTotalBytes, ReviewMaxFiles), all of which
	// the registry's AI Review reads: an answer here is partial.
	OverReviewBudget bool
}

Prepared is the pre-step's result: kind 1 of the report, the policy the outcome is decided under, and the staged inputs.

func Prepare

func Prepare(ctx context.Context, dir string, o Options) (*Prepared, error)

Prepare runs the pre-step over the QuickAgent in dir. An error is a review that cannot start (no manifest, a rule bundle that will not load, a folder that cannot be copied), never a finding.

func (*Prepared) Cleanup

func (p *Prepared) Cleanup()

Cleanup removes the staged inputs.

type Reading

type Reading struct {
	Counted  bool      `json:"counted"`
	Summary  string    `json:"summary,omitempty"`
	Findings []Finding `json:"findings"`
	// NotRead are the files the model says it was not given or did not
	// read; Withheld the files AI Review withheld from it, from the run's
	// own record.
	NotRead  []NotRead `json:"notRead"`
	Withheld []NotRead `json:"withheld"`
	// Error says why there is no reading: the run failed, or its answer
	// was not one.
	Error string `json:"error,omitempty"`
	// Recovered says the runner refused the model's last message at the
	// output schema (exit 17), and the reading is a terminal JSON object
	// inside it, which output.schema.json passes.
	Recovered bool `json:"recovered,omitempty"`
}

Reading is kind 3: the Examiner's own reading, and what was withheld from its model.

type RegistryRules

type RegistryRules struct {
	// Manifest are the manifest rules' errors and warnings; an error is
	// refused at Submit.
	Manifest []ManifestIssue `json:"manifest"`
	Findings []Finding       `json:"findings"`
	Bundle   struct {
		Version string `json:"version"`
		Source  string `json:"source"`
	} `json:"bundle"`
	// Servers are the registry's records of the declared MCP servers.
	Servers []ServerRecord `json:"servers"`
}

RegistryRules is kind 1.

type Report

type Report struct {
	SchemaVersion string   `json:"schemaVersion"`
	Examiner      Examined `json:"examiner"`
	Package       Package  `json:"package"`
	Expected      Expected `json:"expected"`
	// RegistryRules is kind 1.
	RegistryRules RegistryRules `json:"registryRules"`
	// AIReview is kind 2.
	AIReview AIReview `json:"aiReview"`
	// Reading is kind 3.
	Reading Reading `json:"examinerReading"`
	// NotRunHere is what the registry runs and this review did not.
	NotRunHere []string `json:"notRunHere"`
	Notice     string   `json:"notice"`
}

Report is report.json.

func Finish

func Finish(p *Prepared, env *receipt.Envelope, runErr error, outDir, ref, build string) *Report

Finish is the post-step: the report of a review whose run returned env (nil when it failed before it ran) and runErr, with the run's receipts under outDir. It always returns a report: a run that failed still has kind 1, and says why the rest is missing.

type ServerRecord

type ServerRecord struct {
	Ref    string `json:"ref"`
	Status string `json:"status"`
	// Error says why there is no answer, for status unknown or unreachable.
	Error string `json:"error,omitempty"`
	// CheckedAt is when the registry last looked the version up, when it
	// has.
	CheckedAt time.Time `json:"checkedAt,omitzero"`
}

ServerRecord is the registry's record of one declared server, as asked here.

func (ServerRecord) Words

func (s ServerRecord) Words() string

Words is the record as a person reads it (mcpinstall.VulnStatus.Words).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL