Documentation
¶
Overview ¶
Package detect implements the deterministic, offline MCP tool-scanner v2 (Spec 076).
Contract (see specs/076-deterministic-tool-scanner/contracts/detect-engine.md):
Offline: this package performs NO I/O. It imports no networking (net, net/http), no process execution (os/exec), no filesystem access (os), and no HTTP/Docker client. Detection runs purely over in-memory tool definitions supplied by the caller. The offline guarantee is enforced by the standing import-guard test (imports_test.go) and backs FR-001.
Deterministic: identical input (a RegistryView) yields byte-identical output, including finding and signal ordering. No maps are iterated for output ordering; no clocks or randomness are consulted.
Total: every registered Check.Inspect call is run under recover(). A check that panics or errors is isolated, counted in Coverage, and never aborts the scan. A degraded scan still returns its other findings, the same way the existing scanner surfaces scanners_failed.
The engine aggregates per-tool Signals into the existing internal/security/scanner.ScanFinding type (now additively carrying Confidence and Signals), so all CLI/REST/MCP entry points keep their shapes.
Index ¶
- Constants
- func CapEvidence(s string) string
- func CapSpanSnippet(s string) (snippet string, coveredBytes int, truncated bool)
- func ClampConfidence(c float64) float64
- func FoldHomoglyphsIn(s string, latinMajority bool) string
- func LatinMajority(s string) bool
- func Normalize(s string) string
- func NormalizeForPhrase(s string) string
- func NormalizeForPhraseIn(s string, latinMajority bool) string
- func NormalizeMixedScript(s string) (normalized string, changed bool)
- func Squash(s string) string
- func UTF16Offsets(s string, byteStart, byteEnd int) (start, end int, ok bool)
- type Check
- type Coverage
- type Engine
- type Finding
- type Options
- type Position
- type RegistryView
- type Result
- type Signal
- type Span
- type SpanField
- type Tier
- type ToolView
Constants ¶
const ( SeverityCritical = "critical" SeverityHigh = "high" SeverityMedium = "medium" SeverityLow = "low" SeverityInfo = "info" )
Severity levels — string values mirror internal/security/scanner so a Finding maps onto scanner.ScanFinding without translation (the scanner wiring copies these strings verbatim). detect cannot import scanner (import cycle), so the vocabulary is mirrored here, not aliased.
const ( ThreatLevelDangerous = "dangerous" // any hard signal → auto-quarantine ThreatLevelWarning = "warning" // soft-only → review ThreatLevelInfo = "info" )
Threat levels — user-facing severity, mirrors scanner.ThreatLevel*.
const ( ThreatToolPoisoning = "tool_poisoning" ThreatPromptInjection = "prompt_injection" ThreatRugPull = "rug_pull" ThreatExfiltration = "exfiltration" ThreatMaliciousCode = "malicious_code" ThreatUncategorized = "uncategorized" )
Threat types — the report vocabulary, mirrors scanner.Threat* plus the exfiltration category from the Spec-076 data model.
const MaxEvidenceLen = 200
MaxEvidenceLen caps the rune length of rendered evidence before an ellipsis is appended, keeping reports bounded regardless of input size.
const MaxSpansPerFinding = 32
MaxSpansPerFinding bounds the payload for pathological descriptions. A 4,500-character description tripping a dozen rules must not turn one finding into an unbounded array in every scan-report response.
Variables ¶
This section is empty.
Functions ¶
func CapEvidence ¶
CapEvidence makes a string safe to render in CLI/HTML/JSON reports: control runes and zero-width/bidi format runes are escaped to a visible \uXXXX form (revealed, never silently dropped — the whole point is to expose smuggling), and the result is truncated to MaxEvidenceLen runes with an ellipsis.
func CapSpanSnippet ¶ added in v0.64.0
CapSpanSnippet encodes s for use as Span.Snippet, and reports how much of s the result covers and whether it stopped short.
It is NOT CapEvidence, and the difference is the whole point. CapEvidence escapes a control or Cf rune to a six-character `\uXXXX` form but does NOT escape a backslash the text already contained, so an escaped rune and the six literal characters spelling that same escape, typed into a description by hand, encode identically. That ambiguity is harmless for Signal.Evidence, which a human reads in a report, and fatal here, because Snippet is the ONLY staleness check the frontend has: a description edited from one form to the other keeps its checksum, the span still "verifies", and the UI confidently marks characters no rule matched — the "a mark on the wrong words is worse than no mark" failure this package exists to avoid. CapEvidence itself is deliberately untouched: it feeds user-facing evidence throughout the package and its exact output is pinned by existing tests.
The encoding here is injective because it is PREFIX-FREE per rune: a literal backslash always doubles, so a `\u` or `\U` in the output can only have come from an escape, and the fixed digit count (4 below U+10000, 8 above it) says where that escape ends. Equal snippets therefore imply equal inputs — no argument about which code points happen to be escapable is needed, which matters because that set grows with each Unicode revision. CapEvidence's minimum-width `%04x` is not fixed-width and cannot make that promise.
The render-safety property CapEvidence was originally chosen for is kept intact: control, zero-width and bidi runes still never land verbatim in the JSON payload. Doubling a backslash costs nothing there — it is printable ASCII either way.
coveredBytes exists because a snippet is a checksum, not a quote. The cap is MaxEvidenceLen runes, and the bundle's dot-all `<important>…` rules routinely match far more, so a snippet that verified only a 200-rune PREFIX would say nothing whatsoever about the rest of the range: edit only the tail of such a description after the scan and the span still verifies. Producers therefore CLAMP the span to coveredBytes, so a mark can never extend past text the snippet actually pins down.
The truncation point is rune-aligned, unlike CapEvidence's, which slices the escaped string and can cut through a `\uXXXX` escape. Alignment is required here and not there: the frontend re-escapes the LIVE slice of the clamped range and compares, so a snippet ending in half an escape sequence would never match and would silently lose the highlight.
truncated is RETURNED rather than written into the snippet as a trailing ellipsis. See Span.Truncated for why sniffing for one is unsound.
func ClampConfidence ¶
ClampConfidence constrains a confidence to the valid [0,1] range.
func FoldHomoglyphsIn ¶ added in v0.71.0
FoldHomoglyphsIn applies NFKC, format-rune stripping and homoglyph folding (the first steps of NormalizeForPhraseIn) to a WHOLE text before a caller cuts it into fragments. Folding needs a word's neighbours: an entirely look-alike chunk ("тем" in "sys. тем. pro. mpt.") folds to Latin only beside a Latin word, and once the text is cut into clauses those neighbours sit in other fragments (SEC-01 review, Sol round 8). Genuine Cyrillic prose keeps its spelling because its neighbours are Cyrillic too.
func LatinMajority ¶ added in v0.71.0
LatinMajority reports whether Latin letters outnumber Cyrillic, Greek and Armenian letters together in s (after NFKC). It is the document-level script census homoglyph folding uses to decide which way an ambiguous mixed token ("тeм": Cyrillic т/м, Latin e) folds.
func Normalize ¶
Normalize canonicalizes free text for semantic matching by the soft checks (FR-009). It applies, in order: NFKC, removal of zero-width / bidi / BOM format runes (Unicode category Cf), lowercasing, contraction expansion (so "don't disclose" and "do not tell" both yield a "do not …" form), whitespace collapse + trim, and light stemming.
It is pure and deterministic. The HARD unicode check (FR-007) deliberately runs on RAW text instead, so stripping format runes here never hides an attack — it only stabilizes phrase matching.
func NormalizeForPhrase ¶ added in v0.71.0
NormalizeForPhrase is Normalize preceded by a de-obfuscation pass that undoes the cheap, reversible tricks an attacker uses to keep an injection directive readable to a model while hiding it from literal phrase matchers (SEC-01):
- homoglyphs: Cyrillic/Greek/Armenian look-alike letters mixed into a Latin word ("іgnоrе") are folded to Latin; Latin look-alikes mixed into a Cyrillic word ("игнoрируй") are folded to Cyrillic. A token is folded only when it MIXES scripts, or when it is made entirely of look-alike letters inside predominantly-Latin text, so ordinary Russian/Greek/Ukrainian prose keeps its spelling and stays matchable by the non-English families.
- leet digits inside a letter word ("1gn0r3", "pr3v10u5"): only tokens that mix letters with the digits 0 1 3 4 5 7 are touched, so version strings, hashes and identifiers like "sha256", "utf8" or "oauth2" are left alone.
- separator interleaving ("i.g.n.o.r.e", "ig-no-re", "i_g_n_o_r_e"): a token made of three or more 1–2 letter segments joined by one repeated separator is joined. Two-segment forms ("e.g.", "i.e.", "co-op") are untouched, which keeps the example cues the position classifier relies on.
- letter spacing ("i g n o r e"): a run of three or more single letters separated by single spaces is joined; a wider gap ends the run, so "I G N O R E A L L" yields "ignore all".
The result is consumed ONLY by the phrase matchers (phrase.injection, directive.imperative) through ToolView.PhraseText. NormalizedText is unchanged, so capability/shadowing checks and every existing consumer see the same text as before. Pure and deterministic, like Normalize.
func NormalizeForPhraseIn ¶ added in v0.71.0
NormalizeForPhraseIn is NormalizeForPhrase for one fragment (clause) of a larger text, with the script majority taken from the WHOLE text (LatinMajority of it). A short fragment's own census is not evidence: the chunk "тeм." of "sys. тeм. pro. mpt." has a Cyrillic majority on its own, which would fold it to "тем" and hide "system prompt" from the squashed pass (SEC-01 review, Sol round 7).
func NormalizeMixedScript ¶ added in v0.71.0
NormalizeMixedScript is Normalize preceded by homoglyph folding of MIXED-script words only (RT-04): a word that combines Latin letters with Cyrillic, Greek or Armenian look-alikes ("іgnоrе", "рrеvіоuѕ") is folded to one script, while a word written entirely in one script keeps its spelling, whatever its neighbours. Unlike NormalizeForPhrase it does not join spaced or separated letters and does not rewrite leet digits, so word boundaries, punctuation, negations and quotes are exactly those of the plain text and position classification stays truthful. A single word that mixes scripts is not ordinary writing; it is the homoglyph trick itself. changed reports whether any word was folded.
func Squash ¶ added in v0.71.0
Squash reduces text to its letters only. Phrase checks use it as a last, soft-tier pass that is indifferent to ANY spacing or punctuation scheme the earlier steps did not anticipate ("s y s t e m p r o m p t").
func UTF16Offsets ¶ added in v0.64.0
UTF16Offsets converts a byte range in s to UTF-16 code-unit offsets.
ok=false when the range is inverted, out of bounds, or does not start and end on rune boundaries — all three would otherwise yield offsets pointing at characters the check never matched. Callers MUST drop the span on !ok; there is deliberately no best-effort fallback.
An empty in-bounds range (byteStart == byteEnd) converts successfully; it is a legal conversion, and producers simply do not emit zero-width spans.
Types ¶
type Check ¶
type Check interface {
ID() string
Inspect(tool ToolView, reg RegistryView) []Signal
}
Check inspects one tool against the whole registry snapshot and emits zero or more signals. Implementations MUST be pure, total, and deterministic — the engine wraps every Inspect in recover(), but a well-behaved check never panics and returns its signals in a stable order.
type Coverage ¶
Coverage records how complete a scan was: a check whose Inspect panicked or errored is recovered, counted here, and never aborts the scan — mirroring the existing scanners_failed degradation path.
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine runs all registered checks over a registry snapshot and aggregates per-tool signals into findings. Pure aside from the recover() isolation that keeps a misbehaving check from aborting the scan.
func (*Engine) Scan ¶
func (e *Engine) Scan(reg RegistryView) Result
Scan inspects every tool in the snapshot. The RegistryView is built once per scan (indexes + NormalizedText) if the caller passed an unindexed view, then shared with every check. A check that panics is isolated; the scan still returns its other findings. Output (findings and ordering) is deterministic for identical input.
type Finding ¶
type Finding struct {
RuleID string
Scanner string
ThreatType string
ThreatLevel string
Severity string
Category string
Title string
Description string
Location string
Evidence string
Confidence float64
Signals []string
// Spans is the UNION of every contributing signal's spans (deduped, sorted,
// capped), not just the primary's. See unionSpans.
Spans []Span
}
Finding is the per-tool aggregation output. It is self-contained (no scanner import) and converts 1:1 to scanner.ScanFinding in the scanner wiring (T012); the additive Confidence/Signals fields already exist on ScanFinding (T004).
type Options ¶
type Options struct {
// Checks are run, in this order, against every tool. Order is part of the
// determinism contract.
Checks []Check
// ScannerID is stamped onto every finding's Scanner field. Defaults to
// "tpa-descriptions" when empty.
ScannerID string
}
Options configures an Engine.
type Position ¶
type Position int
Position classifies where a phrase match sits in the surrounding text, so the checks can tell a genuine embedded instruction ("ignore previous instructions") apart from a tool that merely DESCRIBES such phrases ("analyzes prompts that ignore previous instructions"). This is the core false-positive control for legitimate security tooling (FR-009, US2), and the deciding factor between the hard/soft tiers for a matched directive.
It is a three-way scale, not a binary one, because the two false-positive pressures pull in opposite directions (Spec 077 US1, Codex round-2):
- A bare label prefix ("Prompt:", "Message:") must NOT neutralize a clear imperative — that framing is exactly what an attacker uses to smuggle a directive (finding A, recall bypass). Such matches stay PositionInstruction and hard-block.
- A tool that DESCRIBES an injection — a relative clause ("prompts that ignore…") or an analytical verb governing the phrase ("returns text: ignore…") — must not hard-block a benign tool (finding B). Such matches are PositionDescriptive: the hard tier is discounted below its floor (HARD→SOFT), but the soft tier still surfaces the match for review, so a real injection wearing descriptive clothing never fully vanishes.
- Genuine quotation/illustration ("such as", "e.g.", surrounding quotes) is PositionExample and most heavily discounted — that framing is the weakest injection signal. It is NOT, however, silently dropped: a check built for recall (phrase.injection) re-floors an already-MATCHED phrase to the soft emit floor so a quoted/illustrated injection surfaces as review-only rather than vanishing entirely (Spec 077 US1, Codex round-3 — the "never fully suppress a matched injection" invariant that closes the recurring silent-bypass class). "Quiet" for a legitimate quoting scanner now means soft/review, never invisible.
Position framing is decided per SENTENCE, not per description: a cue in a PRIOR sentence must not discount an imperative that begins a LATER one. Both the analytical-verb heuristic and the "example"/"such as" word cues are sentence-scoped (only the inline abbreviations "e.g."/"i.e." and quote runs, which carry their own internal periods, are matched across the whole window). Without this, a benign lead ("Example output format. Ignore all previous instructions…") would misclassify the following injection as an example and bypass the hard tier (Codex round-3 finding #1).
const ( // PositionInstruction is an imperative/instruction-position match; full // confidence is kept. This is also where a bare "label:" prefix lands — a // label alone does not discount a clear imperative (finding A). PositionInstruction Position = iota // PositionDescriptive is a match the tool DESCRIBES rather than issues: a // relative clause ("… that ignore …") or an analytical verb governing the // phrase. Discounted enough to drop a hard match below its emit floor // (HARD→SOFT) while a soft match still surfaces for review — no total // suppression, so a genuine injection cannot bypass detection by adopting // descriptive phrasing (finding B without reopening finding A). PositionDescriptive // PositionExample is a quotation/illustration-position match (after "such // as"/"e.g." or inside quotes); most heavily discounted. On its own it clears // neither per-check floor, so a SOFT-only check (directive.imperative) stays // quiet — its US2 false-positive control. A recall-oriented HARD check // (phrase.injection) instead re-floors an already-matched phrase to the soft // emit floor, so the match surfaces for review rather than vanishing (the // "never fully suppress" invariant). PositionExample )
func ClassifyPosition ¶
ClassifyPosition decides whether the match starting at byte offset matchStart in text is in instruction-, descriptive-, or example-position. text may be raw or normalized; matching is case-insensitive on the preceding window.
This is the entry point for SOFT-only checks (directive.imperative): an example-position match falls below their emit floor and is SILENTLY dropped, which is their designed US2 false-positive control. It deliberately does NOT honor the example-ADJECTIVE label cue (issue #795) — see ClassifyPositionForRecall for why that cue is unsafe here.
func ClassifyPositionForRecall ¶ added in v0.57.0
ClassifyPositionForRecall is ClassifyPosition plus the issue-#795 example-adjective label cue ("Sample response:", "Expected output:"). It is ONLY for recall-oriented checks (phrase.injection) that re-floor an example-position match up to the SOFT review floor (never-fully-suppress): there a labeled example downgrades HARD→SOFT and stays visible. The cue is intentionally withheld from ClassifyPosition because a SOFT-only check drops example-position entirely, so honoring the label there would let "Test response: <directive>" silently clear to a clean verdict — a real widening, not a friction change (Codex PR-#977 round 2).
type RegistryView ¶
type RegistryView struct {
Tools []ToolView
ToolsByName map[string][]ToolView // name → tools with that name (collision detection)
ToolNames map[string]struct{} // fast membership for cross-tool references
}
RegistryView is a read-only snapshot of every server's current tools, built once per scan and passed to every check so cross-tool checks (shadowing) can reason about collisions and references.
func NewRegistryView ¶
func NewRegistryView(tools []ToolView) RegistryView
NewRegistryView builds the cross-tool indexes and precomputes each tool's NormalizedText (once). It preserves input ordering in Tools and in each ToolsByName slice, which keeps downstream findings deterministic.
type Signal ¶
type Signal struct {
CheckID string // stable identifier, e.g. "unicode.hidden"
Tier Tier // hard (auto-quarantine) vs soft (review)
ThreatType string // maps to ScanFinding.ThreatType
Confidence float64 // 0.0–1.0; already position-discounted for soft signals
Evidence string // render-safe; for payload.decoded this is the decoded content
Detail string // short human explanation
// Spans locate this signal's match inside the RAW tool text, for inline
// highlighting in the Web UI. Empty (and legitimately so) when the check
// matched NormalizedText, which is lossy and non-invertible, or when the
// evidence is a synthesized statement about the whole description rather
// than a located substring. Never guessed: a span that points at the wrong
// characters is worse than no span.
Spans []Span
}
Signal is the output of a single Check for a single tool.
type Span ¶ added in v0.64.0
type Span struct {
Field SpanField `json:"field"`
Start int `json:"start"`
End int `json:"end"`
CheckID string `json:"check_id"`
Tier string `json:"tier"` // "hard" | "soft" — per-span severity
Snippet string `json:"snippet,omitempty"`
Truncated bool `json:"truncated,omitempty"`
}
Span locates one check's match inside one RAW (un-normalized) tool text field.
Start/End are half-open [Start, End) offsets in UTF-16 CODE UNITS — i.e. JavaScript string indices — because the only consumer that renders them is the Web UI, where description.slice(Start, End) is then exactly the matched text including across surrogate pairs. Producers hold Go BYTE offsets and convert them with UTF16Offsets; a span that survives that conversion is provably the location of the text the check matched, and one that does not is dropped rather than approximated (a mark on the wrong words is worse than no mark).
Snippet is CapSpanSnippet(raw[byteStart:byteEnd]): render-SAFE (control, Cf and bidi runes escaped to a visible \uXXXX form, capped at MaxEvidenceLen). It is NOT the render source — the UI renders the tool's own LIVE description sliced by the offsets, so a description edited since the scan cannot smuggle stale text back onto the page. Snippet exists solely as a staleness checksum, and it is escaped for the reason tpa_bundle.go already documents: a dot-all bundle regex can match bidi/zero-width runes from an attacker-controlled description, and the raw span must never land verbatim in a JSON payload.
Being a checksum is exactly why Snippet is NOT CapEvidence: see CapSpanSnippet for why that function's escaping is ambiguous and what an ambiguous checksum costs here.
Truncated says, explicitly, that Snippet stopped short of the matched text — it is never inferred from the snippet's last character. An ordinary tool description can legitimately end a matched passage with an ellipsis, so a consumer sniffing for one would prefix-compare a snippet that was never truncated and accept any later description sharing the prefix. The flag is the declaration; the snippet carries no marker at all.
CheckID and Tier are carried PER SPAN because aggregate() emits exactly one Finding per tool: the finding-level RuleID/ThreatLevel describe the primary signal only, so per-mark labelling has to come from the span itself.
func DescriptionSpan ¶ added in v0.64.0
DescriptionSpan builds the highlight span for a check that matched description[byteStart:byteEnd], or nil when no honest span can be produced.
Two ways it returns nil, both deliberate: a byte range that will not convert exactly to UTF-16 (see UTF16Offsets), and a range the snippet cannot cover at all. Everything else is clamped to the snippet's extent rather than trimmed by the caller, so every producer gets the same guarantee — the marked range is exactly the range the checksum verifies.
type SpanField ¶ added in v0.64.0
type SpanField string
SpanField names the raw tool text field a Span is measured against. A span is always relative to ONE field: checks that match over a concatenation of description + schemas must attribute each hit to the field it fell in and subtract that field's origin before emitting.
const ( // SpanFieldDescription is ToolView.Description, verbatim. SpanFieldDescription SpanField = "description" // SpanFieldInputSchema is the raw ToolView.InputSchema JSON text. SpanFieldInputSchema SpanField = "input_schema" // SpanFieldOutputSchema is the raw ToolView.OutputSchema JSON text. SpanFieldOutputSchema SpanField = "output_schema" )
type Tier ¶
type Tier int
Tier ranks a signal's contribution to the risk decision.
const ( // TierSoft signals raise a finding for review but never auto-quarantine on // their own. Zero value, so an uninitialized Signal is conservatively soft. TierSoft Tier = iota // TierHard signals contribute to auto-quarantine; checks emitting them are // built for near-zero false positives. TierHard )
type ToolView ¶
type ToolView struct {
Server string
Name string
Description string // raw, un-normalized
InputSchema json.RawMessage
OutputSchema json.RawMessage
NormalizedText string // precomputed Normalize(description + schema text)
// PhraseText is NormalizeForPhrase(description + schema text): the
// normalized text after homoglyph, leet, separator and letter-spacing
// de-obfuscation (SEC-01). Only the phrase matchers read it; when it is
// empty (a hand-built ToolView) they fall back to NormalizedText.
PhraseText string
}
ToolView is a read-only projection of one tool supplied to checks. An empty description/schema is valid input and yields zero signals (no error).