analyzer

package
v0.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 24, 2026 License: AGPL-3.0 Imports: 12 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BuildHTMLExtractionPrompt

func BuildHTMLExtractionPrompt(rawHTML string, originalURL string) string

BuildHTMLExtractionPrompt constructs the user prompt with truncated HTML.

func BuildNextDataPathsPrompt

func BuildNextDataPathsPrompt(pageProps map[string]any, originalURL string) string

BuildNextDataPathsPrompt constructs the user prompt with a key-tree skeleton of pageProps. Shows the structure (keys, types, array lengths) without the bulky values, so the LLM can identify paths accurately.

func BuildPrompt

func BuildPrompt(har *browser.HARLog, dom string, originalURL string) string

BuildPrompt constructs the user prompt from HAR traffic, DOM snapshot, and the original page URL for the LLM to analyze.

func BuildSuggestPrompt

func BuildSuggestPrompt(originalURL string, failureReason string) string

BuildSuggestPrompt constructs the user prompt for knowledge-based discovery.

func HTMLExtractionSystemPrompt

func HTMLExtractionSystemPrompt() string

HTMLExtractionSystemPrompt returns the system prompt for identifying CSS selectors and entity fields that extract structured content from an HTML page. It asks for page type classification and typed entity extraction, producing compact agent-friendly output.

func NextDataPathsSystemPrompt

func NextDataPathsSystemPrompt() string

NextDataPathsSystemPrompt returns the system prompt for analyzing __NEXT_DATA__ pageProps and identifying named extraction paths.

func SuggestSystemPrompt

func SuggestSystemPrompt() string

SuggestSystemPrompt returns the system prompt for knowledge-based API discovery. Used when browser capture fails — the LLM uses its training knowledge to suggest public API endpoints for the site.

func SystemPrompt

func SystemPrompt() string

SystemPrompt returns the LLM system prompt that instructs the model to analyze captured HAR traffic and identify API endpoints.

func TruncateJSONValues

func TruncateJSONValues(v any, maxLen int) any

TruncateJSONValues preserves the full key structure of a JSON-like value but truncates string values to maxLen characters (appending "..."), caps arrays to the first 3 items, and recurses into nested maps and arrays. Numbers, bools, and nil are preserved unchanged. Returns a new value; the input is never mutated.

Types

type OpenAIAnalyzer

type OpenAIAnalyzer struct {
	// contains filtered or unexported fields
}

OpenAIAnalyzer implements the Service interface using an OpenAI-compatible API.

func NewOpenAIAnalyzer

func NewOpenAIAnalyzer(config OpenAIConfig) *OpenAIAnalyzer

NewOpenAIAnalyzer creates a new OpenAIAnalyzer with the given configuration.

func (*OpenAIAnalyzer) Analyze

func (a *OpenAIAnalyzer) Analyze(ctx context.Context, har *browser.HARLog, dom string, originalURL string) (*schema.Schema, error)

Analyze implements the Service interface. It sends the HAR log, DOM snapshot, and original URL to the configured LLM and parses the response into a Schema. Uses the fast ClassifyModel (if configured) since HAR classification is a simple NOISE/CANDIDATE task that doesn't need an expensive model.

func (*OpenAIAnalyzer) AnalyzeHTML

func (a *OpenAIAnalyzer) AnalyzeHTML(ctx context.Context, rawHTML string, originalURL string) (*schema.ExtractionRules, error)

AnalyzeHTML examines raw HTML to identify CSS selectors for content extraction.

func (*OpenAIAnalyzer) AnalyzeNextDataPaths

func (a *OpenAIAnalyzer) AnalyzeNextDataPaths(ctx context.Context, pageProps map[string]any, originalURL string) (map[string]string, error)

AnalyzeNextDataPaths examines __NEXT_DATA__ pageProps and identifies named extraction paths so subsequent fetches return targeted sub-trees.

func (*OpenAIAnalyzer) Ask

func (a *OpenAIAnalyzer) Ask(ctx context.Context, data string, prompt string) (string, error)

Ask sends data + a natural language prompt to the fast classify model and returns a plain text answer. The data is truncated to fit context.

func (*OpenAIAnalyzer) Suggest

func (a *OpenAIAnalyzer) Suggest(ctx context.Context, originalURL string, failureReason string) (*schema.Schema, error)

Suggest uses LLM knowledge to suggest public API endpoints when browser capture fails.

type OpenAIConfig

type OpenAIConfig struct {
	BaseURL       string
	APIKey        string
	Model         string
	ClassifyModel string // fast/cheap model for HAR classification (optional, defaults to Model)
}

OpenAIConfig holds configuration for the OpenAI-compatible LLM client.

type Service

type Service interface {
	// Analyze examines HAR traffic + DOM to identify API endpoints.
	Analyze(ctx context.Context, har *browser.HARLog, dom string, originalURL string) (*schema.Schema, error)

	// Suggest uses LLM knowledge to suggest public API endpoints when
	// browser capture fails (auth wall, Cloudflare, empty HAR).
	Suggest(ctx context.Context, originalURL string, failureReason string) (*schema.Schema, error)

	// AnalyzeHTML examines raw HTML to identify CSS selectors for extracting
	// the main content. Returns extraction rules that can be cached and
	// replayed without LLM on subsequent visits.
	AnalyzeHTML(ctx context.Context, rawHTML string, originalURL string) (*schema.ExtractionRules, error)

	// AnalyzeNextDataPaths examines __NEXT_DATA__ pageProps and identifies
	// named extraction paths (e.g. "products" -> ".ssrQuery.hits") so that
	// subsequent fetches return targeted sub-trees instead of the full blob.
	AnalyzeNextDataPaths(ctx context.Context, pageProps map[string]any, originalURL string) (map[string]string, error)

	// Ask sends fetched data + a natural language prompt to the LLM and
	// returns a plain text answer. Uses the fast classify model.
	Ask(ctx context.Context, data string, prompt string) (string, error)
}

Service defines the interface for analyzing captured browser traffic.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL