Documentation
¶
Index ¶
- func BuildHTMLExtractionPrompt(rawHTML string, originalURL string) string
- func BuildNextDataPathsPrompt(pageProps map[string]any, originalURL string) string
- func BuildPrompt(har *browser.HARLog, dom string, originalURL string) string
- func BuildSuggestPrompt(originalURL string, failureReason string) string
- func HTMLExtractionSystemPrompt() string
- func NextDataPathsSystemPrompt() string
- func SuggestSystemPrompt() string
- func SystemPrompt() string
- func TruncateJSONValues(v any, maxLen int) any
- type OpenAIAnalyzer
- func (a *OpenAIAnalyzer) Analyze(ctx context.Context, har *browser.HARLog, dom string, originalURL string) (*schema.Schema, error)
- func (a *OpenAIAnalyzer) AnalyzeHTML(ctx context.Context, rawHTML string, originalURL string) (*schema.ExtractionRules, error)
- func (a *OpenAIAnalyzer) AnalyzeNextDataPaths(ctx context.Context, pageProps map[string]any, originalURL string) (map[string]string, error)
- func (a *OpenAIAnalyzer) Ask(ctx context.Context, data string, prompt string) (string, error)
- func (a *OpenAIAnalyzer) Suggest(ctx context.Context, originalURL string, failureReason string) (*schema.Schema, error)
- type OpenAIConfig
- type Service
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BuildHTMLExtractionPrompt ¶
BuildHTMLExtractionPrompt constructs the user prompt with truncated HTML.
func BuildNextDataPathsPrompt ¶
BuildNextDataPathsPrompt constructs the user prompt with a key-tree skeleton of pageProps. Shows the structure (keys, types, array lengths) without the bulky values, so the LLM can identify paths accurately.
func BuildPrompt ¶
BuildPrompt constructs the user prompt from HAR traffic, DOM snapshot, and the original page URL for the LLM to analyze.
func BuildSuggestPrompt ¶
BuildSuggestPrompt constructs the user prompt for knowledge-based discovery.
func HTMLExtractionSystemPrompt ¶
func HTMLExtractionSystemPrompt() string
HTMLExtractionSystemPrompt returns the system prompt for identifying CSS selectors and entity fields that extract structured content from an HTML page. It asks for page type classification and typed entity extraction, producing compact agent-friendly output.
func NextDataPathsSystemPrompt ¶
func NextDataPathsSystemPrompt() string
NextDataPathsSystemPrompt returns the system prompt for analyzing __NEXT_DATA__ pageProps and identifying named extraction paths.
func SuggestSystemPrompt ¶
func SuggestSystemPrompt() string
SuggestSystemPrompt returns the system prompt for knowledge-based API discovery. Used when browser capture fails — the LLM uses its training knowledge to suggest public API endpoints for the site.
func SystemPrompt ¶
func SystemPrompt() string
SystemPrompt returns the LLM system prompt that instructs the model to analyze captured HAR traffic and identify API endpoints.
func TruncateJSONValues ¶
TruncateJSONValues preserves the full key structure of a JSON-like value but truncates string values to maxLen characters (appending "..."), caps arrays to the first 3 items, and recurses into nested maps and arrays. Numbers, bools, and nil are preserved unchanged. Returns a new value; the input is never mutated.
Types ¶
type OpenAIAnalyzer ¶
type OpenAIAnalyzer struct {
// contains filtered or unexported fields
}
OpenAIAnalyzer implements the Service interface using an OpenAI-compatible API.
func NewOpenAIAnalyzer ¶
func NewOpenAIAnalyzer(config OpenAIConfig) *OpenAIAnalyzer
NewOpenAIAnalyzer creates a new OpenAIAnalyzer with the given configuration.
func (*OpenAIAnalyzer) Analyze ¶
func (a *OpenAIAnalyzer) Analyze(ctx context.Context, har *browser.HARLog, dom string, originalURL string) (*schema.Schema, error)
Analyze implements the Service interface. It sends the HAR log, DOM snapshot, and original URL to the configured LLM and parses the response into a Schema. Uses the fast ClassifyModel (if configured) since HAR classification is a simple NOISE/CANDIDATE task that doesn't need an expensive model.
func (*OpenAIAnalyzer) AnalyzeHTML ¶
func (a *OpenAIAnalyzer) AnalyzeHTML(ctx context.Context, rawHTML string, originalURL string) (*schema.ExtractionRules, error)
AnalyzeHTML examines raw HTML to identify CSS selectors for content extraction.
func (*OpenAIAnalyzer) AnalyzeNextDataPaths ¶
func (a *OpenAIAnalyzer) AnalyzeNextDataPaths(ctx context.Context, pageProps map[string]any, originalURL string) (map[string]string, error)
AnalyzeNextDataPaths examines __NEXT_DATA__ pageProps and identifies named extraction paths so subsequent fetches return targeted sub-trees.
type OpenAIConfig ¶
type OpenAIConfig struct {
BaseURL string
APIKey string
Model string
ClassifyModel string // fast/cheap model for HAR classification (optional, defaults to Model)
}
OpenAIConfig holds configuration for the OpenAI-compatible LLM client.
type Service ¶
type Service interface {
// Analyze examines HAR traffic + DOM to identify API endpoints.
Analyze(ctx context.Context, har *browser.HARLog, dom string, originalURL string) (*schema.Schema, error)
// Suggest uses LLM knowledge to suggest public API endpoints when
// browser capture fails (auth wall, Cloudflare, empty HAR).
Suggest(ctx context.Context, originalURL string, failureReason string) (*schema.Schema, error)
// AnalyzeHTML examines raw HTML to identify CSS selectors for extracting
// the main content. Returns extraction rules that can be cached and
// replayed without LLM on subsequent visits.
AnalyzeHTML(ctx context.Context, rawHTML string, originalURL string) (*schema.ExtractionRules, error)
// AnalyzeNextDataPaths examines __NEXT_DATA__ pageProps and identifies
// named extraction paths (e.g. "products" -> ".ssrQuery.hits") so that
// subsequent fetches return targeted sub-trees instead of the full blob.
AnalyzeNextDataPaths(ctx context.Context, pageProps map[string]any, originalURL string) (map[string]string, error)
// Ask sends fetched data + a natural language prompt to the LLM and
// returns a plain text answer. Uses the fast classify model.
Ask(ctx context.Context, data string, prompt string) (string, error)
}
Service defines the interface for analyzing captured browser traffic.