Documentation
¶
Index ¶
- func DetectJSShell(rawHTML string) string
- func DetectJSShellFromDoc(doc *goquery.Document, rawHTML string) string
- func ExtractSelector(rawHTML, selector string) (string, error)
- func PostProcess(s string, trim bool, maxChars int) string
- func StripMarkdown(s string) string
- func Title(html string) string
- func Truncate(s string, maxChars int) string
- type Detector
- type Extractor
- type Result
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DetectJSShell ¶
DetectJSShell analyzes raw HTML and returns whether the page appears to be a JavaScript shell that needs browser rendering for content extraction. Returns: "static" (has content), "likely_shell" (needs browser), or "ambiguous".
It uses only the built-in markers. Callers with operator-configured markers should build a Detector via NewDetector and call its method instead.
func DetectJSShellFromDoc ¶
DetectJSShellFromDoc is the core detector given an already-parsed document and its raw source. Useful when callers have already paid for the parse. It uses only the built-in markers; see NewDetector for operator markers.
func ExtractSelector ¶
ExtractSelector runs a CSS selector against raw HTML and returns the matched elements converted to markdown. If no elements match, returns an empty string and no error.
func PostProcess ¶
PostProcess applies trim (StripMarkdown) then Truncate to markdown content. It is the shared output post-processing step for the CLI's --trim/--max-chars flags and the MCP tools' trim/max_chars params.
func StripMarkdown ¶
StripMarkdown removes markdown formatting syntax while preserving content text. Images are removed entirely; links keep their text; headings keep their text. Fenced code blocks (``` ... ```) are left intact — their content is code, not prose.
Types ¶
type Detector ¶
type Detector struct {
// contains filtered or unexported fields
}
Detector classifies raw HTML as "static" (has content), "likely_shell" (needs browser rendering), or "ambiguous". The zero value is ready to use and matches only the built-in framework markers; NewDetector folds in operator-configured markers (config spa_markers) on top of the built-ins.
func NewDetector ¶
NewDetector returns a Detector that also matches the given operator-supplied SPA markers (substrings of the lowercased HTML) in addition to the built-in ones. Markers are lowercased and trimmed; blank entries are dropped so a stray "" can never match every page.
func (*Detector) DetectJSShell ¶
DetectJSShell parses rawHTML and classifies it. See DetectJSShell.
Source Files