Documentation
¶
Overview ¶
Package examiner is the Examiner: the built-in QuickAgent that reviews a QuickAgent a person has built and reports what the registry's pipeline would be expected to say about it at Submit, embedded in the CLI and run by the runner as the fourth built-in package, the one granted the review grant. Its manifest declares no egress, no filesystem, no exec, no secret and no server; its reach is ai_review and scanned reads of its inputs, with the person's own TypeSafe key, or through the registry on an operation paid with their organization's credits.
Around the run this package holds the host's pre-step (Prepare: the registry's own rules, its baseline policy and its record of each declared server, and the inputs staged read-only), the post-step (Finish: the report, its three kinds of finding, the expected outcome, what did not run here) and the gate's rule restated (Gate), which a test holds to the core's published table of gate cases. The CLI's `agentry review` is its first caller; the desktop's Review Your QuickAgents locally is another.
Index ¶
- Constants
- Variables
- func AIReviewFinding(id string, score int) (gatecases.Finding, bool)
- func AutoPublishUpTo(p gatecases.Policy) string
- func FS() fs.FS
- func LocalReview(pol *policy.Policy) (*engine.ReviewRun, error)
- func LocalReviewPaid(ctx context.Context, pol *policy.Policy, pay drafter.Pay) (*engine.ReviewRun, error)
- func OutcomeWords(o string) string
- func Task(p *Prepared) string
- func Write(outDir string, r *Report) (string, error)
- type AIReview
- type Axis
- type Decision
- type Examined
- type Expected
- type Finding
- type GateInput
- type ManifestIssue
- type NotRead
- type Options
- type Package
- type PolicyUsed
- type Prepared
- type Reading
- type RegistryRules
- type Report
- type ServerRecord
Constants ¶
const ( Namespace = "agentry" Name = "examiner" Version = "1.2.4" )
The package's identity, as its manifest declares it.
const ( InputManifest = "manifest.json" InputPackage = "package" )
The staged inputs' names, as the run sees them under /inputs.
const ( PolicyFromRegistry = "registry" PolicyCompiled = "compiled" )
Where the policy came from (PolicyUsed.Source).
const ( // KindRegistryRules: the registry's own rules, run here as the registry // runs them. KindRegistryRules = "registry-rules" // KindAIReview: AI Review, asked as stage S5 asks it. KindAIReview = "ai-review" // KindExaminer: the Examiner's own reading, never counted. KindExaminer = "examiner" )
The three kinds of finding, as the report labels them.
const ( OutcomePublished = gatecases.OutcomePublished OutcomeNeedsReview = gatecases.OutcomeNeedsReview OutcomeRejected = gatecases.OutcomeRejected // OutcomeRefused: Submit refuses the manifest before the gate runs. OutcomeRefused = "refused_at_submit" // OutcomeUndetermined: what ran here cannot decide between outcomes. OutcomeUndetermined = "undetermined" )
The expected outcomes.
const Assumption = "a public version, as the manifest rules check it; the review queue a held version would go to is not named"
Assumption is what every expected outcome takes as given: the manifest rules run as for a public version (V7), and a held version's queue is not named, since only Submit knows the version's visibility and whether its namespace is an organization's.
const GuideName = "review-guide"
GuideName is the knowledge pack the Examiner ships in its own release, its review guide: what every run has, the contract's lines, what to look for and what the registry checks. The runner gives it to every run of the package (localrun's withShippedPacks). Its manifest.json, index and runner tools are generated (TestTheReviewGuideIsCurrent).
const LocalTimeoutFloorSeconds = 1800
LocalTimeoutFloorSeconds is the least a review on this computer is given: 30 minutes, past the runner policy's 15-minute run timeout, since one rubric review alone may take 15 (toolsrv's bound on an ai_review call). It is a timeout alone, as the other built-ins' floors are.
const Notice = "" /* 137-byte string literal not displayed */
Notice ends every report.
const ReportFile = "report.json"
ReportFile is the report's name in the output folder.
const ReportSchemaVersion = "1"
ReportSchemaVersion is the version of the report's shape.
const ScannerUngrantedTool = "S6/ungranted-tool"
ScannerUngrantedTool is the registry's scripts-stage finding for runner tools the instruction calls for that the manifest does not grant.
Variables ¶
var ErrNoJevKey = drafter.ErrNoJevKey
ErrNoJevKey is LocalReview's answer when the person has no TypeSafe key of their own: a review does not run without one, since AI Review is asked with it and every file the model reads is scanned with it first.
var NotRunHere = []string{
"the registry's secrets stage (S2), which scans the files and their history with its own tools",
"the registry's scripts stage (S6), which checks each support script against the declared capabilities (its check of the instruction against the runner tools each contract line grants runs here)",
"the publisher signals (S8): the name against popular QuickAgents, and the account's own record",
"the diff against the last published version (S9)",
"an organization's policy, which can be stricter than the platform's baseline",
"a shipped knowledge pack's secrets scan, URL reputation and AI Review, in the registry's stage for packs (its hidden-content and injection rules run here)",
"the text the registry recovers from image metadata, and its comparison of the instruction file's front matter with the manifest",
"Submit's own refusals: the policy's blocked capabilities, blocked MCP servers, allowed licences and required evaluations, a quota, two-step sign-in, credits, a version already published",
"the refusals a run in Agentry makes when it starts, which Submit does not make: a target host, a browser on any public site where the deployment does not allow one, and a browser's site that is not a public address on port 80 or 443",
}
NotRunHere are the registry's checks no client has the code or the standing to run: every report names them.
Functions ¶
func AIReviewFinding ¶
AIReviewFinding is one rubric question's answer as stage S5 rates it, through the core's judge.RubricSeverity, so it is the table the gate reads and not a copy of it: below 2 no finding; 2 medium and 3 high, except that an attempted injection, an undeclared capability and deceiving the user are high from 2, and exfiltration and destructive actions medium at any score (ruleset 0.4.0). The core's further finding for a verdict showing manipulation comes from its generative-model providers alone; the Jev judge AI Review asks never sets it.
func AutoPublishUpTo ¶
AutoPublishUpTo is the worst severity a policy publishes: its maxAutoPublishSeverity, or the gate's fallback when it names none.
func FS ¶
FS is the package's tree: agent.yaml, INSTRUCTION.md, output.schema.json, README.md and LICENSE at its root (a public release carries the last two), and under knowledge/ the pack it ships (GuideName).
func LocalReview ¶
LocalReview is a review's key on this computer: the person's own, read as a draft reads it (drafter.JevKey). Without one it is ErrNoJevKey. Where an organization's credits pay for AI Review, it refuses with the organization's sentence: work that can pay with them calls LocalReviewPaid.
func LocalReviewPaid ¶ added in v0.7.46
func LocalReviewPaid(ctx context.Context, pol *policy.Policy, pay drafter.Pay) (*engine.ReviewRun, error)
LocalReviewPaid is LocalReview for work that can pay with an organization's credits: where the managed settings say they pay for AI Review, it opens an operation with pay (managed.PurposeReview) and the review asks Jev through the registry with the operation's token. A nil pay refuses such work with the organization's sentence.
func OutcomeWords ¶
OutcomeWords is an outcome as a person reads it.
Types ¶
type AIReview ¶
type AIReview struct {
Ran bool `json:"ran"`
Model string `json:"model,omitempty"`
RubricHash string `json:"rubricHash,omitempty"`
Axes []Axis `json:"axes"`
Findings []Finding `json:"findings"`
Files int `json:"files"`
NotRead int `json:"notRead"`
// Calls counts the ai_review calls the run made.
Calls int `json:"calls"`
// Failure says why it did not run, when it did not.
Failure string `json:"failure,omitempty"`
}
AIReview is kind 2: the first answered ai_review call over the staged package, from the run's review ledger.
type Axis ¶
type Axis struct {
ID string `json:"id"`
Score int `json:"score"`
Severity string `json:"severity,omitempty"`
File string `json:"file,omitempty"`
Prob float64 `json:"probability"`
}
Axis is one rubric question's answer.
type Decision ¶
type Decision struct {
gatecases.Want
Worst string `json:"worstSeverity"`
Branch string `json:"branch"`
}
Decision is the gate's answer: the outcome, its queue when it is needs_review, the worst severity it weighed and the branch that decided.
type Examined ¶
type Examined struct {
Ref string `json:"ref"`
Build string `json:"build"`
Model string `json:"model,omitempty"`
RunID string `json:"runId,omitempty"`
}
Examined names the Examiner run.
type Expected ¶
type Expected struct {
// Outcome is published, needs_review, rejected, refused_at_submit or
// undetermined.
Outcome string `json:"outcome"`
// Worst is the worst severity the gate would weigh, and Branch its rule
// that decides, when the gate decides.
Worst string `json:"worstSeverity,omitempty"`
Branch string `json:"branch,omitempty"`
// Reason says why, in a sentence.
Reason string `json:"reason"`
// Assumes says what the outcome takes as given that only Submit knows.
Assumes string `json:"assumes"`
Policy PolicyUsed `json:"policy"`
// Counted says which kinds of finding were weighed.
Counted []string `json:"counted"`
}
Expected is the outcome the gate would be expected to reach.
func Expect ¶
Expect is the outcome the gate would be expected to reach from kinds 1 and 2 under the policy the pre-step found. A manifest the rules refuse is refused at Submit, before the gate. Where something the gate weighs did not answer here (AI Review, or answered for part of the package only; a declared server's record), the outcome is named only when the best and the worst answer it could give decide alike.
type Finding ¶
type Finding struct {
Kind string `json:"kind"`
// Scanner is the registry's scanner id (S1/…, S4/…, S5/…, S7/…) for the
// first two kinds, and secret, script, instruction or other for the
// Examiner's.
Scanner string `json:"scanner"`
Severity string `json:"severity,omitempty"`
Confidence string `json:"confidence,omitempty"`
Title string `json:"title"`
File string `json:"file,omitempty"`
Line int `json:"line,omitempty"`
Excerpt string `json:"excerpt,omitempty"`
Detail string `json:"detail,omitempty"`
}
Finding is one finding of any kind.
type GateInput ¶
type GateInput struct {
Findings []gatecases.Finding
Policy gatecases.Policy
WildcardEgress bool
JudgeForcesReview bool
DeclaredCapabilities []string
Public bool
NamespaceIsOrg bool
}
GateInput is what the gate weighs, in the table's terms.
type ManifestIssue ¶
type ManifestIssue struct {
Rule string `json:"rule"`
Severity string `json:"severity"`
Path string `json:"path,omitempty"`
Message string `json:"message"`
}
ManifestIssue is one manifest rule's result.
type Options ¶
type Options struct {
// Registry is the registry's base URL; empty asks it nothing, and the
// report says so.
Registry string
// HTTP is the client for the registry; nil is a client with a timeout.
HTTP *http.Client
// Rules loads the S4 rule bundle (validate.Options.Rules); nil is the
// bundle embedded in this binary.
Rules validate.RulesLoader
// Log takes the narration; nil discards it.
Log io.Writer
}
Options are how Prepare reaches the registry and loads its rules.
type PolicyUsed ¶
type PolicyUsed struct {
gatecases.Policy
// Source is "registry" (its public baseline, GET /platform/policy) or
// "compiled" (the baseline compiled into the core this CLI was built
// with, when the registry did not answer).
Source string `json:"source"`
// Error says why the registry's was not used.
Error string `json:"error,omitempty"`
}
PolicyUsed is the publication policy the outcome is decided under, and where it came from.
type Prepared ¶
type Prepared struct {
// Dir is the QuickAgent's folder; Ref is what was examined, its
// ns/name@version as the manifest declares it.
Dir, Ref string
// ManifestJSON is the manifest's canonical JSON, as the registry reads
// it.
ManifestJSON []byte
// Validation is `agentry validate --json`'s report over the folder.
Validation validate.Report
// Findings are kind 1 as the gate weighs them: the inventory, the
// hidden-content and injection rules, a file past the scan cap or not
// read, and each declared server's advisories.
Findings []Finding
// Servers are the registry's records of the declared servers.
Servers []mcpinstall.VulnStatus
Policy PolicyUsed
// Stage holds the staged inputs; Inputs are the paths a run is handed.
// Cleanup removes them.
Stage string
Inputs []string
// Files is how many files were staged.
Files int
// DeclaredServers counts the MCP servers the manifest declares.
DeclaredServers int
// OverReviewBudget says the package holds more text than one ai_review
// call sends (toolsrv.ReviewTotalBytes, ReviewMaxFiles), all of which
// the registry's AI Review reads: an answer here is partial.
OverReviewBudget bool
}
Prepared is the pre-step's result: kind 1 of the report, the policy the outcome is decided under, and the staged inputs.
type Reading ¶
type Reading struct {
Counted bool `json:"counted"`
Summary string `json:"summary,omitempty"`
Findings []Finding `json:"findings"`
// NotRead are the files the model says it was not given or did not
// read; Withheld the files AI Review withheld from it, from the run's
// own record.
NotRead []NotRead `json:"notRead"`
Withheld []NotRead `json:"withheld"`
// Error says why there is no reading: the run failed, or its answer
// was not one.
Error string `json:"error,omitempty"`
// Recovered says the runner refused the model's last message at the
// output schema (exit 17), and the reading is a terminal JSON object
// inside it, which output.schema.json passes.
Recovered bool `json:"recovered,omitempty"`
}
Reading is kind 3: the Examiner's own reading, and what was withheld from its model.
type RegistryRules ¶
type RegistryRules struct {
// Manifest are the manifest rules' errors and warnings; an error is
// refused at Submit.
Manifest []ManifestIssue `json:"manifest"`
Findings []Finding `json:"findings"`
Bundle struct {
Version string `json:"version"`
Source string `json:"source"`
} `json:"bundle"`
// Servers are the registry's records of the declared MCP servers.
Servers []ServerRecord `json:"servers"`
}
RegistryRules is kind 1.
type Report ¶
type Report struct {
SchemaVersion string `json:"schemaVersion"`
Examiner Examined `json:"examiner"`
Package Package `json:"package"`
Expected Expected `json:"expected"`
// RegistryRules is kind 1.
RegistryRules RegistryRules `json:"registryRules"`
// AIReview is kind 2.
AIReview AIReview `json:"aiReview"`
// Reading is kind 3.
Reading Reading `json:"examinerReading"`
// NotRunHere is what the registry runs and this review did not.
NotRunHere []string `json:"notRunHere"`
Notice string `json:"notice"`
}
Report is report.json.
type ServerRecord ¶
type ServerRecord struct {
Ref string `json:"ref"`
Status string `json:"status"`
// Error says why there is no answer, for status unknown or unreachable.
Error string `json:"error,omitempty"`
// CheckedAt is when the registry last looked the version up, when it
// has.
CheckedAt time.Time `json:"checkedAt,omitzero"`
}
ServerRecord is the registry's record of one declared server, as asked here.
func (ServerRecord) Words ¶
func (s ServerRecord) Words() string
Words is the record as a person reads it (mcpinstall.VulnStatus.Words).