Documentation
¶
Overview ¶
Package parse extracts versions from upstream content: JSON paths, regular expressions and HTML/XPath, version transforms and series selection, version-history extraction, and the validation of an extracted version against a registry record. It imports registry and ebuilds.
The headless-browser ("script") parser stays in internal/autoupdate: only the checker drives it, and its build-tag-selected backend is pinned there.
Index ¶
- Constants
- Variables
- func ApplySuffix(log *slog.Logger, v string, cfg *registry.PackageConfig) string
- func ApplyTransforms(log *slog.Logger, v string, rules [][]string) string
- func ExtractVersionHistory(content []byte, cfg *registry.PackageConfig) ([]string, error)
- func HasVersionHistoryConfig(cfg *registry.PackageConfig) bool
- func NavigateJSONPath(data interface{}, path string) (interface{}, error)
- func NormalizeUpstreamVersion(version string) string
- func ParseVersion(content []byte, cfg *registry.PackageConfig) (string, error)
- func SelectVersion(log *slog.Logger, cands []string, cfg *registry.PackageConfig) string
- func StripVersionPrefix(version string) string
- func TestExtraction(content []byte, schema *registry.PackageConfig) (string, error)
- type HTMLParser
- type HTMLVersionHistoryExtractor
- type JSONParser
- type JSONVersionHistoryExtractor
- type Parser
- type RegexParser
- type RegexVersionHistoryExtractor
- type ValidationResult
- type VersionHistoryExtractor
- type XPathVersionHistoryExtractor
Constants ¶
const MaxVersionHistoryLimit = 10
MaxVersionHistoryLimit is the maximum number of versions to extract from history. Per Requirement 9.3, version history is limited to 10 versions.
Variables ¶
var ( // ErrInvalidSelector is returned when the CSS selector syntax is invalid ErrInvalidSelector = errors.New("invalid CSS selector") // ErrInvalidXPath is returned when the XPath expression syntax is invalid ErrInvalidXPath = errors.New("invalid XPath expression") // ErrNoElementFound is returned when no element matches the selector/xpath ErrNoElementFound = errors.New("no element found matching selector") // ErrNoSelectorOrXPath is returned when neither selector nor xpath is provided ErrNoSelectorOrXPath = errors.New("either selector or xpath must be provided") )
Error variables for HTML parser errors
var ( // ErrJSONPathNotFound is returned when the JSON path does not exist in the document ErrJSONPathNotFound = errors.New("JSON path not found in response") // ErrRegexNoMatch is returned when the regex pattern does not match the content ErrRegexNoMatch = errors.New("regex pattern did not match") // ErrNoVersionFound is returned when no version could be extracted from upstream ErrNoVersionFound = errors.New("could not extract version from upstream") // ErrInvalidJSONPath is returned when the JSON path syntax is invalid. It // is the jsonpath package's sentinel, so errors.Is matches either name. ErrInvalidJSONPath = jsonpath.ErrInvalidJSONPath // ErrInvalidRegexPattern is returned when the regex pattern is invalid ErrInvalidRegexPattern = errors.New("invalid regex pattern") // ErrNoCaptureGroup is returned when the regex pattern has no capture group ErrNoCaptureGroup = errors.New("regex pattern must contain at least one capture group") )
Error variables for parser errors
var ( // ErrValidationFailed is returned when schema validation fails ErrValidationFailed = errors.New("schema validation failed") // ErrExtractionFailed is returned when version extraction fails during validation ErrExtractionFailed = errors.New("version extraction failed") // ErrVersionMismatch is returned when extracted version doesn't match ebuild version ErrVersionMismatch = errors.New("extracted version does not match ebuild version") )
Error variables for validation errors
Functions ¶
func ApplySuffix ¶
ApplySuffix appends cfg.Suffix to v when the record declares a pre-release channel, i.e. when the version upstream publishes is a development release its own numbering does not mark (LibreOffice's testing line ships a plain "26.8.0.1"). With cfg.SuffixWhen set the suffix applies only to a version matching that regex, which is how one endpoint listing several release lines marks the development one alone.
It is idempotent in two ways, because the pipeline applies it both per candidate (inside selectVersion, so "max" orders the values that will actually become the PV) and once on the final version (in fetchUpstreamVersion, which also covers the script/fallback/LLM paths): a version that already ends in a suffix is returned untouched, so neither a second pass nor an upstream string that is already marked (2.0.0_rc1) can stack into "2.0.0_rc1_pre".
A malformed suffix_when is warned and ignored rather than fatal: the record is rejected up front by ValidatePackageConfig, and a check that got this far must not die on the annotation. The warning goes to log; nil discards it.
func ApplyTransforms ¶
ApplyTransforms applies ordered regex substitutions to an extracted version. Each rule is [regex, repl]; repl follows regexp.ReplaceAllString semantics. A malformed rule (wrong arity or uncompilable regex) is warned and skipped, so a single bad rule never aborts a check (ValidatePackageConfig warns too). The warning goes to log; nil discards it.
func ExtractVersionHistory ¶
func ExtractVersionHistory(content []byte, cfg *registry.PackageConfig) ([]string, error)
ExtractVersionHistory extracts version history from content using the configured extractor. Returns nil if no version history is configured.
func HasVersionHistoryConfig ¶
func HasVersionHistoryConfig(cfg *registry.PackageConfig) bool
HasVersionHistoryConfig checks if a PackageConfig has version history configuration.
func NavigateJSONPath ¶
NavigateJSONPath navigates through JSON data following the given path. Supports dot notation (field.subfield) and array indexing (field[0]).
func NormalizeUpstreamVersion ¶
NormalizeUpstreamVersion trims whitespace and removes a common tag prefix ("v", "release-", …) from an upstream version value. It is the ONE normalization every consumer of an upstream version must agree on: the checker applies it before a value reaches CheckResult and the pending list, and the validation plan applies it before classifying a pending entry that may have been written by an older binary with the prefix still on. Plan and run classifying different values is exactly the bug this exists to prevent ("v3.2.3" charged as major in the plan, run as patch by Validate).
func ParseVersion ¶
func ParseVersion(content []byte, cfg *registry.PackageConfig) (string, error)
ParseVersion attempts to extract version using configured parsers with fallback logic. It tries the primary parser first, then fallback parser if configured, and returns the first successful result.
func SelectVersion ¶
SelectVersion picks one version from a candidate list according to cfg.Select. Each candidate is transformed BEFORE validation/comparison, because a raw candidate such as "7.1.2-24" is not a valid Gentoo version until the transform runs — selecting before transforming would discard every candidate. The pre-release suffix is applied next, so candidates are ordered as the versions they will become: a record marking one release line as _pre must compare that line's _pre value, not the bare one.
- "max": the highest comparable Gentoo version.
- "last": the last candidate that is a comparable version (document order).
Non-comparable candidates (per ebuild.IsValidVersion, after transform and prefix stripping) are skipped. Returns "" when no candidate is comparable. A malformed rule is warned about to log; nil discards the warning.
func StripVersionPrefix ¶
StripVersionPrefix removes common version prefixes like 'v', 'V', 'version-', etc.
func TestExtraction ¶
func TestExtraction(content []byte, schema *registry.PackageConfig) (string, error)
TestExtraction attempts to extract version using the schema configuration. It creates the appropriate parser based on the schema and extracts the version from the provided content.
Parameters:
- content: The raw content to extract version from
- schema: The PackageConfig defining how to extract the version
Returns the extracted version string or an error if extraction fails.
Types ¶
type HTMLParser ¶
type HTMLParser struct {
// Selector is the CSS selector for extracting version
Selector string
// XPath is the XPath expression for extracting version (alternative to Selector)
XPath string
// Regex is an optional regex pattern to apply to the extracted text
Regex string
// contains filtered or unexported fields
}
HTMLParser extracts version using CSS selector or XPath expression. It supports optional regex post-processing to extract version from matched text.
func NewHTMLParser ¶
func NewHTMLParser(selector, xpath, regex string) (*HTMLParser, error)
NewHTMLParser creates a new HTMLParser with the given configuration. At least one of selector or xpath must be provided.
type HTMLVersionHistoryExtractor ¶
type HTMLVersionHistoryExtractor struct {
// VersionsSelector is the CSS selector for version elements
VersionsSelector string
// Regex is an optional regex pattern to apply to each extracted text
Regex string
// Limit caps the number of versions returned (see effectiveLimit).
Limit int
}
HTMLVersionHistoryExtractor extracts version history using CSS selector. The selector should match multiple elements containing version strings.
func (*HTMLVersionHistoryExtractor) ExtractVersions ¶
func (e *HTMLVersionHistoryExtractor) ExtractVersions(content []byte) ([]string, error)
ExtractVersions extracts version history from HTML content using the configured selector. Returns at most MaxVersionHistoryLimit versions.
type JSONParser ¶
type JSONParser struct {
// Path is the JSON path to the version field (e.g., "notes[0].version", "tag_name")
Path string
}
JSONParser extracts version using a JSON path. The path supports dot notation and array indexing (e.g., "notes[0].version").
type JSONVersionHistoryExtractor ¶
type JSONVersionHistoryExtractor struct {
// VersionsPath is the JSON path to the version array (e.g., "[*].tag_name")
VersionsPath string
// Limit caps the number of versions returned. See effectiveLimit for the
// 0/<0/>0 convention. Zero value keeps the historic 10-version cap.
Limit int
}
JSONVersionHistoryExtractor extracts version history using a JSON path. The path should point to an array of version strings or objects with version fields.
func (*JSONVersionHistoryExtractor) ExtractVersions ¶
func (e *JSONVersionHistoryExtractor) ExtractVersions(content []byte) ([]string, error)
ExtractVersions extracts version history from JSON content using the configured path. Returns at most MaxVersionHistoryLimit versions.
type Parser ¶
type Parser interface {
// Parse extracts a version string from the given content.
// Returns the extracted version or an error if extraction fails.
Parse(content []byte) (string, error)
}
Parser defines the interface for version extraction from content. Implementations extract version strings from different content formats.
func NewParser ¶
NewParser creates a parser based on the specified type. parserType must be "json", "regex", or "html". pathOrPattern is the JSON path for json parser or regex pattern for regex parser. For HTML parser, use NewParserFromConfig instead.
func NewParserFromConfig ¶
func NewParserFromConfig(cfg *registry.PackageConfig) (Parser, error)
NewParserFromConfig creates a parser from a PackageConfig. This supports all parser types including HTML which requires additional fields.
type RegexParser ¶
type RegexParser struct {
// Pattern is the regex pattern with at least one capture group
Pattern string
// contains filtered or unexported fields
}
RegexParser extracts version using a regular expression with capture group. The first capture group in the pattern is used as the version.
type RegexVersionHistoryExtractor ¶
type RegexVersionHistoryExtractor struct {
// Pattern is the regex pattern with at least one capture group.
Pattern string
// Limit caps the number of versions returned (see effectiveLimit).
Limit int
// contains filtered or unexported fields
}
RegexVersionHistoryExtractor extracts version history using a regex pattern. Every match of the first capture group becomes a candidate version. This is the list-extraction counterpart of RegexParser, used by the "select" path so a pattern like `gn-([0-9.]+)\.tar\.xz` yields every version on the page.
func (*RegexVersionHistoryExtractor) ExtractVersions ¶
func (e *RegexVersionHistoryExtractor) ExtractVersions(content []byte) ([]string, error)
ExtractVersions extracts every first-capture-group match from content. It compiles the pattern lazily so the extractor is safe to use as the entry point (RegexParser.Parse compiles on its own path).
type ValidationResult ¶
type ValidationResult struct {
// Valid indicates if the schema validation was successful
Valid bool
// ExtractedVersion is the version extracted using the schema
ExtractedVersion string
// EbuildVersion is the current version from the ebuild
EbuildVersion string
// VersionsMatch indicates if extracted version matches ebuild version
VersionsMatch bool
// Error contains any error that occurred during validation
Error error
}
ValidationResult represents the result of schema validation. It contains information about whether the schema successfully extracted a version and whether it matches the expected ebuild version.
func ValidateSchema ¶
func ValidateSchema(content []byte, schema *registry.PackageConfig, ebuildVersion string) *ValidationResult
ValidateSchema tests a schema by extracting version from content and comparing it with the ebuild version. This implements the schema validation flow: 1. Test version extraction using the schema 2. Compare the extracted version with the ebuild version 3. Mark as validated if versions match
Parameters:
- content: The raw content fetched from the URL
- schema: The PackageConfig schema to validate
- ebuildVersion: The current version from the ebuild file
Returns a ValidationResult with the validation outcome.
func ValidateSchemaWithFallback ¶
func ValidateSchemaWithFallback(primaryContent, fallbackContent []byte, schema *registry.PackageConfig, ebuildVersion string) *ValidationResult
ValidateSchemaWithFallback validates a schema and tries fallback URL if primary fails. This is useful when the primary URL might be temporarily unavailable.
Parameters:
- primaryContent: Content from the primary URL
- fallbackContent: Content from the fallback URL (can be nil)
- schema: The PackageConfig schema to validate
- ebuildVersion: The current version from the ebuild file
Returns a ValidationResult with the validation outcome.
type VersionHistoryExtractor ¶
type VersionHistoryExtractor interface {
// ExtractVersions extracts a list of versions from content.
// Returns at most MaxVersionHistoryLimit versions.
ExtractVersions(content []byte) ([]string, error)
}
VersionHistoryExtractor defines the interface for extracting version history.
func NewSelectExtractor ¶
func NewSelectExtractor(cfg *registry.PackageConfig) (VersionHistoryExtractor, error)
NewSelectExtractor builds a list extractor for the "select" path, dispatching on cfg.Parser and reusing the primary path/pattern/selector fields. The cap is disabled (Limit=-1) so select="max" sees the whole list and is not defeated by truncation of an ascending list. Returns (nil, nil) when the parser cannot produce a list (e.g. "script"); callers then fall back to first-match behavior.
func NewVersionHistoryExtractor ¶
func NewVersionHistoryExtractor(cfg *registry.PackageConfig) (VersionHistoryExtractor, error)
NewVersionHistoryExtractor creates a version history extractor from a PackageConfig. It uses VersionsPath for JSON parser or VersionsSelector for HTML parser.
type XPathVersionHistoryExtractor ¶
type XPathVersionHistoryExtractor struct {
// VersionsXPath is the XPath expression for version nodes
VersionsXPath string
// Regex is an optional regex pattern to apply to each extracted text
Regex string
// Limit caps the number of versions returned (see effectiveLimit).
Limit int
}
XPathVersionHistoryExtractor extracts version history using XPath expression. The xpath should match multiple nodes containing version strings.
func (*XPathVersionHistoryExtractor) ExtractVersions ¶
func (e *XPathVersionHistoryExtractor) ExtractVersions(content []byte) ([]string, error)
ExtractVersions extracts version history from HTML content using the configured XPath. Returns at most MaxVersionHistoryLimit versions.
Source Files