Documentation
¶
Overview ¶
Package aitext estimates whether text-detection tools would flag prose as written by a language model, and says why. It needs no model and no network: it measures what detectors measure in the open (burstiness, the punctuation profile, lexical diversity, compressibility against model-written text, how sentences open, repeated phrasing, participle clauses, abstract nouns) plus the habits of current models and the fillers of people writing, and combines them with weights fitted on labelled human and generated text in English, German, French, Spanish, Italian, Portuguese, Dutch and Polish. Other languages are read with the language-agnostic measurements only.
A score is an estimate, not proof. Thresholds are set for about 1% false alarms on human writing from sources the model never saw, so a quiet result means little and a loud one means the text shares the measurable habits of generated prose.
Index ¶
Constants ¶
const MinWords = 150
MinWords is the shortest text the estimate is made for; below it every feature is noise.
const Window = 500
Window is the length the models were fitted on: longer texts are scored window by window, because repetition and compression grow with length.
Variables ¶
var FeatureNames = []string{
"sentence-burstiness", "paragraph-burstiness", "mean-sentence", "commas", "semicolons", "parentheses",
"colons", "questions", "dashes", "lexical-diversity", "word-length-spread", "compression",
"model-likeness", "opener-diversity", "repeated-trigrams", "top-word-share", "habits", "asides",
"fillers", "mid-length-sentences", "even-paragraphs", "claude-likeness",
"participles", "nominals", "person", "short-sentences", "long-sentences", "punctuation-entropy", "one-sentence-paragraphs",
}
FeatureNames name the features in the order they are computed; the weights follow it.
Functions ¶
func CountWords ¶
CountWords counts the matches of a Words pattern, including adjacent ones that share a boundary character.
Types ¶
type Pattern ¶
type Pattern struct {
// contains filtered or unexported fields
}
Pattern is a Words pattern with a literal prefilter: every match contains one of its keys, so the regular expression only runs on text holding one. Go's regexp cannot skip ahead through an alternation framed by word boundaries; the prefilter makes the common case (no key in the paragraph) a few substring searches.
func Compile ¶
Compile is a regular expression with a prefilter of the literals every match contains, each at least that many runes long (3 suits single words, 4 running text).
func Words ¶
Words compiles a case-insensitive alternation of words and phrases that matches only at word boundaries in any script. Go's \b knows ASCII only, so "\bübung\b" never matches; this boundary is any character that is not a letter or digit. Matches include the boundary characters; Trim removes them.
func (*Pattern) FindAllString ¶
FindAllString returns up to n matches (all for n < 0).
func (*Pattern) FindString ¶
FindString returns the leftmost match, or "".
func (*Pattern) FindStringIndex ¶
FindStringIndex returns the location of the leftmost match, or nil.
func (*Pattern) MatchString ¶
MatchString reports whether s contains a match.
type Result ¶
type Result struct {
Words int `json:"words"`
Score float64 `json:"score"` // 0 to 1: how strongly the text matches generated prose
Likely bool `json:"likely"`
Signals []Signal `json:"signals,omitempty"`
Features []float64 `json:"-"`
// Strongest is the index of the first paragraph of the window that
// matches generated prose most: where a revision helps most.
Strongest int `json:"strongestParagraph"`
// contains filtered or unexported fields
}
Result is the estimate for one text.
Source Files