Documentation
¶
Overview ¶
Package langguard is the detection foundation of the outbound language guard (roadmap/SP-152-language-guard.md): it extracts prose from model messages, identifies the dominant writing script of that prose, and decides whether a message is long enough to judge at all.
The guard's purpose is to keep model-authored text from being displayed when it is not written in the user's language. This package provides the deterministic, in-process half of that check: no external service and no model call, pure Go with the standard library only (no cgo), so the same code runs in the CLI process, in the sprout process of a hosted workspace, and compiled into the WASM browser build — and can be called by other Go programs embedding sprout.
The intended pipeline, which later items build on this package:
prose := langguard.ExtractProse(modelReply)
if langguard.Judgable(prose) {
if v := langguard.CheckScript(prose, userScript); v == langguard.VerdictMismatch {
// same-script detector or regeneration
}
}
The same-script trigram detector (language.go) and user-language resolution (resolve.go) build on this package's prose and script primitives. Deliberately not included here: reply-path wiring and streaming hold-back, and mismatch metrics. They layer on top of this package's API; the API is shaped for them (Check and CheckLanguage take the user's script and language as parameters, ResolveUserLanguage returns the resolved user language, String methods exist for logging, and the thresholds are tunable constants).
The same-script trigram pass: identify the natural language of extracted prose with a small, deterministic trigram detector. The script pass (script.go) catches the "wrong script entirely" failure mode, but it cannot tell Spanish from English — two languages that share the Latin script. This pass discriminates within a script.
The detector is github.com/abadojack/whatlanggo (v1.0.1): a pure-Go, stdlib-only trigram-frequency library (no cgo, no network, no model download), so it runs in the CLI process, in the sprout process of a hosted workspace, and compiled into the WASM browser build — the same three places the pass requires, and it stays callable from other Go programs embedding sprout.
The intended pipeline (layered items build on this file):
prose := langguard.ExtractProse(modelReply)
lang, conf, reliable := langguard.DetectLanguage(prose)
if reliable && lang.Code != userLang.Code {
// mismatch: regenerate once or fall back to the
// script pass; log with model ID and role.
}
user-language resolution derives the user's own language with the same DetectLanguage call over their recent messages; the final-message guard uses CheckLanguage to compare the reply against it.
Prose extraction for the language guard. Code blocks, inline code, URLs, file paths and quoted user text are "not prose" and must be stripped before language detection: code is written in whatever language the codebase uses regardless of the conversation language, and URLs and paths carry no language signal at all.
User-language resolution: work out the language the user is writing in, from their own recent messages, with a configured fallback for very short or mixed input. There is no fixed list of supported languages — any language the detector recognizes is a valid target, so the resolver has no hardcoded language list of its own; it simply votes over what the trigram detector reports.
The resolver is deterministic, in-process, and pure Go (no cgo): it runs the same DetectLanguage call over each recent message, takes the majority language over the reliable per-message detections, and falls back to a configured language when the input is too short or too mixed to yield a unique winner. It is shaped so the final-message guard can consume it directly, and can be called by other Go programs embedding sprout.
The Unicode script pass: identify the dominant writing script of a prose string using the standard library's unicode range tables. This is the cheap, exact first half of the guard: it catches the common failure mode — a reply that is entirely in the wrong script (e.g. Chinese in a Spanish conversation). Languages that share a script (Spanish vs English) are not distinguishable here by design; that is the same-script detector's job.
Index ¶
Constants ¶
const MinDetectConfidence = 0.8
MinDetectConfidence is the minimum detector confidence for a DetectLanguage result to count as reliable. It matches whatlanggo's own reliable-confidence threshold (0.8) and its strict-greater semantics (its IsReliable: the rating "has to be succeeded"): at or below it, the trigram distance between the two closest candidate languages is too small to call the winner reliable.
const MinJudgedProseRunes = 32
MinJudgedProseRunes is the minimum length, in runes, of the extracted prose below which a message is not judged at all ("Messages below a length threshold are not judged (detection is unreliable on a few words)"). The threshold applies to the extracted prose — code, URLs and paths removed — not to the raw message, so a reply that is long only because it contains code is still not judged.
32 runes is roughly five to seven words of Latin prose or one full sentence of CJK. It will be tuned from the detector tests; change it here.
const MinScriptRunes = 8
MinScriptRunes is the minimum number of runes of the dominant script for DominantScript to call its result reliable.
Variables ¶
This section is empty.
Functions ¶
func ExtractProse ¶
ExtractProse returns the prose-only portion of text: fenced code blocks, inline code, URLs, file paths and quoted spans are removed, and the remaining whitespace is collapsed to single spaces. The result is what language detection (Judgable, DominantScript) should see.
Extraction is total: it never fails, never mutates its input, and returns "" for empty and garbage input. What is removed is deliberately conservative — ordinary prose that merely contains a slash ("3/4", "and/or") or an apostrophe ("don't") survives, because leaving real prose in is a harmless false negative for the detector, while stripping it is a false positive.
Types ¶
type Language ¶
type Language struct {
// Code is the language's ISO 639-1 code ("en", "es", "ru", ...), or
// — for the few languages without a 639-1 code — the ISO 639-3
// code ("cmn", "pes", ...). It is the stable identity used to
// compare two detection results (the majority vote over the
// user's recent messages).
Code string
// Name is the detector's human-readable language name ("English",
// "Spanish", ...). It feeds diagnostics and the explicit
// regeneration instruction.
Name string
}
Language identifies the natural language the trigram pass reports. The zero value (empty Code) means "no known language".
func DetectLanguage ¶
DetectLanguage identifies the natural language of already-extracted prose (for same-script languages, "a small trigram detector with no network or model download"). It wraps whatlanggo's trigram detector and returns:
- the detected language (ISO code + human name);
- the detector's confidence in [0, 1];
- whether the result is reliable enough to judge the reply: the prose must be Judgable (too-short and code-only prose are never judged) AND the confidence must reach MinDetectConfidence.
Prose that is not judgable, carries no recognized script, or yields an unreliable confidence returns the zero Language and reliable=false — the caller treats the reply as undetermined (as the script pass does) instead of guessing.
func ParseLanguage ¶
ParseLanguage turns a configured language code into a Language. It accepts an ISO 639-1 code ("es", "en", ...) or, for the few languages without a 639-1 code, the 639-3 code ("pes", "ceb", ...), case-insensitively. The Name is filled from the detector's catalog when the code is recognized, so a parsed language renders usefully in diagnostics.
The Code is canonicalized to the same form DetectLanguage reports — the 639-1 code when the language has one, otherwise the 639-3 code — so a configured value stays comparable to a detection result by code equality. A configured 639-3 code for a language that also has a 639-1 code (e.g. "spa" for Spanish) canonicalizes to the 639-1 code ("es"), the form the trigram pass emits. This matters because the final-message guard's CheckLanguage compares the reply's detected code against the user's code.
An empty code, or a code the detector does not recognize, returns the zero Language (Code == ""): "no fixed language list" means the detector's own catalog is the universe of valid targets, and a code outside it is undetermined rather than guessed. This is the read side of the configured language setting: read the config field, pass it here, and feed the result to ResolveUserLanguage.
func ResolveUserLanguage ¶
ResolveUserLanguage resolves the user's language from their own recent messages (raw text — prose is extracted internally) with a configured fallback. It returns the resolved Language and whether it was determined (a non-zero language the caller can judge a reply against).
The resolution is a majority vote over the reliable per-message detections. Each message is run through DetectLanguage over its extracted prose, and only the reliable results (reliable == true) count; they are grouped by language Code case-insensitively. The caller decides how many "recent" messages to pass (a few turns); the resolver votes over exactly the messages it is given.
- A unique winner — the single most-frequent language, with no tie at the top — wins, whether or not it is an absolute majority of the recent messages. It is the user's most consistent writing language.
- If there are no reliable detections at all (every message is too short, code-only, or not judgable), or the messages are genuinely mixed (a tie for the top, i.e. no unique winner), the configured fallback is returned.
- If there is no configured fallback either (its Code is empty), the zero Language (Code == "") is returned with determined == false. The caller treats the user's language as undetermined; the guard never guesses.
A clear majority always beats a configured value: the configured language is only a fallback for short or mixed input, never a tie-breaker.
type Script ¶
type Script int
Script is a writing script in the Unicode sense. The script pass compares scripts, not languages.
const ( // ScriptUnknown means "no script" or "no reliable script". ScriptUnknown Script = iota // ScriptLatin covers the Latin script, including the Latin-1 and // extended blocks (accented letters and ligatures) — the script of // English, Spanish, French, Portuguese, German, Polish, and so on. ScriptLatin // ScriptCyrillic — Russian, Ukrainian, Bulgarian, and so on. ScriptCyrillic // ScriptGreek. ScriptGreek // ScriptArabic. ScriptArabic // ScriptHebrew. ScriptHebrew // ScriptDevanagari — Hindi, Marathi, and so on. ScriptDevanagari // ScriptThai. ScriptThai // ScriptHangul — Korean. ScriptHangul // ScriptHiragana and ScriptKatakana are the two Japanese kana // scripts; together they cover the kana of Japanese text (Japanese // also uses Han, see ScriptHan). ScriptHiragana ScriptKatakana // ScriptHan — CJK Unified Ideographs (Chinese characters). Japanese // and Korean text contains Han as well, but a text dominated by Han // alone is overwhelmingly Chinese. ScriptHan )
The scripts the guard recognizes: the writing scripts of the major world languages. Text written in any other script (Bengali, Tamil, Georgian, …) is counted as an untracked letter by DominantScript and therefore suppresses reliability instead of being misassigned.
func DominantScript ¶
DominantScript identifies the dominant writing script of prose. It returns the dominant script, the number of runes of that script, and whether the result is reliable.
A result is reliable only when the dominant script accounts for at least MinScriptRunes runes AND a strict majority of all classified letters — letters of the recognized scripts plus letters of untracked ones. A text split between two scripts (or mostly an untracked script) is therefore not reliable, so the caller should treat the message as undetermined rather than guess.
func (Script) IsJapanese ¶
IsJapanese reports whether s is one of the Japanese kana scripts (hiragana or katakana).
type Verdict ¶
type Verdict int
Verdict is the outcome of the script pass over one model message.
const ( // VerdictPass: the reply's dominant script is compatible with the // user's script (equal scripts, or the two Japanese kana scripts). VerdictPass Verdict = iota // VerdictMismatch: the reply is reliably written in a different // script than the user's — the "wrong script entirely" failure mode // the script pass exists to catch. VerdictMismatch // VerdictUndetermined: the message is too short to judge, has no // reliable dominant script, or the user's script is unknown. The // script pass neither passes nor fails it; a same-script detector // may still judge it. VerdictUndetermined )
The script-pass verdicts.
func Check ¶
Check is the intended single entry point for the reply paths: it extracts the prose from a raw model reply and runs the script pass against the user's script.
func CheckLanguage ¶
CheckLanguage is the trigram pass's raw-message entry point for the reply paths: it extracts the prose from a raw model reply and reports the combined verdict of the reply's language against the user's language (from the user-language resolution, or the configured language setting). It parallels the script pass's Check:
Check: text + user's script -> Verdict (wrong script) CheckLanguage: text + user's language -> Verdict (wrong language, same script)
A mismatch can only be asserted from a reliable detection: when the reply is too short, code-heavy, or the detector is unsure, the verdict is VerdictUndetermined and the script pass remains the fallback judgment. Language codes are compared case-insensitively, so a configured "ES" matches the detector's "es".
func CheckScript ¶
CheckScript runs the script pass on the extracted prose of a model reply. userScript is the script the user writes in; user-language resolution will derive it — it can be built directly on this package's DominantScript. Pass ScriptUnknown when it could not be determined: the pass then reports VerdictUndetermined instead of guessing.
The check only judges messages whose extracted prose is Judgable and whose dominant script is reliable; everything else is VerdictUndetermined.
Source Files