Documentation
¶
Index ¶
- Variables
- func DetectJSShell(rawHTML string) string
- func DetectJSShellFromDoc(doc *goquery.Document, rawHTML string) string
- func ExtractSelector(rawHTML, selector string) (string, error)
- func PostProcess(s string, trim bool, maxChars int) string
- func StripMarkdown(s string) string
- func Title(html string) string
- func Truncate(s string, maxChars int) string
- type Detector
- type Extractor
- type PDFExtractor
- type Result
Constants ¶
This section is empty.
Variables ¶
var ErrPDFNoText = errors.New("PDF contains no extractable text")
ErrPDFNoText reports a valid PDF with no extractable text layer.
Functions ¶
func DetectJSShell ¶
DetectJSShell analyzes raw HTML and returns whether the page appears to be a JavaScript shell that needs browser rendering for content extraction. Returns: "static" (has content), "likely_shell" (needs browser), or "ambiguous".
It uses only the built-in markers. Callers with operator-configured markers should build a Detector via NewDetector and call its method instead.
func DetectJSShellFromDoc ¶
DetectJSShellFromDoc is the core detector given an already-parsed document and its raw source. Useful when callers have already paid for the parse. It uses only the built-in markers; see NewDetector for operator markers.
func ExtractSelector ¶
ExtractSelector runs a CSS selector against raw HTML and returns the matched elements converted to markdown. If no elements match, returns an empty string and no error.
func PostProcess ¶
PostProcess applies trim (StripMarkdown) then Truncate to markdown content. It is the shared output post-processing step for the CLI's --trim/--max-chars flags and the MCP tools' trim/max_chars params.
func StripMarkdown ¶
StripMarkdown removes markdown formatting syntax while preserving content text. Images are removed entirely; links keep their text; headings keep their text. Fenced code blocks (``` ... ```) are left intact — their content is code, not prose.
Types ¶
type Detector ¶
type Detector struct {
// contains filtered or unexported fields
}
Detector classifies raw HTML as "static" (has content), "likely_shell" (needs browser rendering), or "ambiguous". The zero value is ready to use and matches only the built-in framework markers; NewDetector folds in operator-configured markers (config spa_markers) on top of the built-ins.
func NewDetector ¶
NewDetector returns a Detector that also matches the given operator-supplied SPA markers (substrings of the lowercased HTML) in addition to the built-in ones. Markers are lowercased and trimmed; blank entries are dropped so a stray "" can never match every page.
func (*Detector) DetectJSShell ¶
DetectJSShell parses rawHTML and classifies it. See DetectJSShell.
type PDFExtractor ¶ added in v0.12.0
type PDFExtractor interface {
Extract(ctx context.Context, src []byte) (markdown string, err error)
}
PDFExtractor converts PDF bytes into markdown-compatible text.
func NewExternalPDFExtractor ¶ added in v0.12.0
func NewExternalPDFExtractor(command string, timeout time.Duration) (PDFExtractor, error)
NewExternalPDFExtractor creates a PDF extractor that invokes command directly (without a shell). The shlex-parsed template must contain exactly one {input} placeholder, which is replaced with a temporary PDF path.
func NewPDFExtractor ¶ added in v0.12.0
func NewPDFExtractor() PDFExtractor
NewPDFExtractor returns the built-in pure-Go PDF text extractor.
Source Files