Help us shape the future of Go! Take the 2026 Go Developer Survey

specoracle

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 19, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package specoracle keeps a local, gitignored cache of the OpenAPI blocks JFrog embeds on each docs.jfrog.com endpoint page and compares them with the specs this project authors under api/. JFrog's definitions are proprietary, so the cache is a test oracle only: it is fetched on demand, never committed, and the committed artefact is the parity baseline, which lists only paths, field names and status codes (decision record 0001).

Index

Constants

View Source
const DefaultUserAgent = "artifactweave-specoracle (https://github.com/weaveplatform/artifactweave)"

DefaultUserAgent identifies the fetcher to docs.jfrog.com so a polite operator can find us if the crawl misbehaves.

View Source
const ManifestFile = "manifest.json"

ManifestFile is the name of the manifest written into the cache directory. Its presence is how TestSpecParity decides whether a cache exists.

View Source
const Placeholder = "TODO(author)"

Placeholder is written into every description a scaffolded fragment carries. TestOpenAPIFieldDescriptions rejects it once the fragment loses its x-draft marker, so an author cannot ship a scaffold by accident.

Variables

View Source
var (

	// ErrBadStatus is wrapped around unexpected HTTP statuses.
	ErrBadStatus = errors.New("unexpected HTTP status")
)

Sentinel errors, wrapped with context where they are returned.

Functions

func ExtractBlock

func ExtractBlock(page string) ([]byte, bool)

ExtractBlock finds the OpenAPI document embedded in an endpoint page. The pages carry it under an "OpenAPI definition" heading as a fenced json block; older renderings inline the JSON. Both are handled, and HTML entities are decoded because some pages escape angle brackets inside descriptions. The returned bytes are compacted JSON so cache hashes are stable across whitespace changes on the site.

func FamilySlug

func FamilySlug(index string) string

FamilySlug turns an index URL into a directory name, dropping the host, the "reference" segment and the llms.txt file name: https://docs.jfrog.com/artifactory/reference/repositories-api/llms.txt becomes artifactory-repositories-api.

func NormalisePath

func NormalisePath(p string) string

NormalisePath replaces every {name} with {} so differently named path parameters still match.

func PageSlug

func PageSlug(pageURL string) string

PageSlug is the last path segment of a page URL without its .md suffix.

func ParseIndex

func ParseIndex(body string) [][2]string

ParseIndex extracts (title, url) pairs from an llms.txt body.

Types

type Baseline

type Baseline struct {
	Missing   []string            `json:"missing"`
	Divergent map[string][]string `json:"divergent"`
}

Baseline is the committed list of accepted gaps. It names only operation keys and finding strings (paths, field names, status codes), never JFrog's text, so it can live in the repository while the cache cannot.

func ReadBaseline

func ReadBaseline(path string) (Baseline, error)

ReadBaseline loads a baseline file. A missing file is an empty baseline, which makes every documented operation a regression: that is the right failure mode for a repository that forgot to commit the file.

type ContentRecord

type ContentRecord struct {
	Media  string `json:"media"`
	Schema string `json:"schema,omitempty"`
}

ContentRecord names the schema behind one media type.

type Diff

type Diff struct {
	Regressions  []string
	Improvements []string
}

Diff splits the report against a baseline into regressions (new gaps) and improvements (baseline entries that no longer apply and can be removed).

type FamilySummary

type FamilySummary struct {
	Family    string
	Total     int
	Missing   int
	Divergent int
	Matched   int
}

FamilySummary is one row of Summary.

type FetchOptions

type FetchOptions struct {
	// Indexes are llms.txt URLs; each lists endpoint pages as "- [Title](url): text".
	Indexes []string
	// CacheDir receives <index-slug>/<page-slug>.json files and the manifest.
	CacheDir string
	// Client is used for every request; nil means http.DefaultClient with a timeout.
	Client *http.Client
	// Concurrency bounds parallel page fetches (default 4).
	Concurrency int
	// Delay is the pause before each page request (default 200 ms).
	Delay time.Duration
	// UserAgent overrides DefaultUserAgent.
	UserAgent string
	// Retries is the number of attempts on 429 and 5xx responses (default 3).
	Retries int
}

FetchOptions configures Fetch. Zero values pick polite defaults.

type Inventory

type Inventory struct {
	GeneratedAt time.Time      `json:"generated_at"`
	Pages       int            `json:"pages"`
	Schemas     []SchemaRecord `json:"schemas"`
	Operations  []OpRecord     `json:"operations"`
	Errors      []string       `json:"errors,omitempty"`
}

Inventory is what `specoracle inventory` writes.

func BuildInventory

func BuildInventory(cacheDir string) (Inventory, error)

BuildInventory extracts schemas and operations from every cached block.

func (Inventory) WriteJSON

func (inv Inventory) WriteJSON(path string) error

WriteJSON writes the inventory as indented JSON.

func (Inventory) WriteMarkdown

func (inv Inventory) WriteMarkdown(path string) error

WriteMarkdown writes the human-readable inventory: one operations table and one fields table per family, no prose from the source.

type Manifest

type Manifest struct {
	FetchedAt time.Time       `json:"fetched_at"`
	UserAgent string          `json:"user_agent"`
	Entries   []ManifestEntry `json:"entries"`
}

Manifest is written as manifest.json and lists what the cache holds.

func Fetch

func Fetch(ctx context.Context, opts FetchOptions) (Manifest, error)

Fetch downloads every endpoint page listed by the indexes, extracts the embedded OpenAPI block and writes it into CacheDir. Pages without a block and transport failures are recorded in the manifest rather than aborting, because the docs site changes and a single bad page must not hide the rest.

func (Manifest) Blocks

func (m Manifest) Blocks() int

Blocks returns the number of entries with a cached OpenAPI block.

type ManifestEntry

type ManifestEntry struct {
	Index   string `json:"index"`
	Family  string `json:"family"`
	Title   string `json:"title"`
	URL     string `json:"url"`
	File    string `json:"file,omitempty"`
	Status  int    `json:"status,omitempty"`
	SHA256  string `json:"sha256,omitempty"`
	Skipped string `json:"skipped,omitempty"`
	Error   string `json:"error,omitempty"`
}

ManifestEntry records one page of one index.

type OpRecord

type OpRecord struct {
	Key        string                     `json:"key"`
	Family     string                     `json:"family"`
	Tag        string                     `json:"tag,omitempty"`
	ID         string                     `json:"id,omitempty"`
	Method     string                     `json:"method"`
	Path       string                     `json:"path"`
	Source     string                     `json:"source"`
	Params     []ParamRecord              `json:"params,omitempty"`
	Request    []ContentRecord            `json:"request,omitempty"`
	Responses  map[string][]ContentRecord `json:"responses"`
	Deprecated bool                       `json:"deprecated,omitempty"`
}

OpRecord is one documented operation.

type Operation

type Operation struct {
	// Key is "METHOD /full/path" with every path parameter name replaced by
	// {}, because clients never see parameter names.
	Key          string
	Family       string
	Source       string
	ID           string
	Method       string
	Path         string
	Params       []Param
	RequestMedia []string
	RequestProps []string
	// Responses maps a status code to the property paths of its JSON body.
	Responses  map[string][]string
	Deprecated bool
}

Operation is one method on one full path, flattened to what matters for wire compatibility. Property paths use "a.b" for nested objects and "a[].b" for array items.

type Oracle

type Oracle struct {
	Ops    map[string]Operation
	Errors []string
}

Oracle is a set of operations plus the parse problems met while loading.

func Load

func Load(cacheDir string) (Oracle, error)

Load reads every cached block under cacheDir. A block that fails to parse is recorded in Errors and skipped: one bad page must not hide the rest. The cache is exactly two levels deep (family/page.json), so a plain directory listing is enough and avoids walking symlinks.

func LoadOurs

func LoadOurs(paths ...string) (Oracle, error)

LoadOurs reads the project's own YAML specs the same way, so both sides go through one normaliser and the comparison cannot be skewed by a second parser.

type Param

type Param struct {
	Name     string
	In       string
	Required bool
	Type     string
	Enum     []string
}

Param is a normalised parameter.

type ParamRecord

type ParamRecord struct {
	Name     string   `json:"name"`
	In       string   `json:"in"`
	Required bool     `json:"required,omitempty"`
	Type     string   `json:"type,omitempty"`
	Enum     []string `json:"enum,omitempty"`
	Default  any      `json:"default,omitempty"`
}

ParamRecord is an operation parameter with the facts clients depend on.

type Report

type Report struct {
	// Missing lists operation keys JFrog documents that we do not define.
	Missing []string
	// Divergent maps an operation key to the findings against our definition.
	Divergent map[string][]string
	// Matched lists keys we define with no findings.
	Matched []string
	// Extra lists keys we define that JFrog does not document (probes, stubs).
	Extra []string
	// Families records the family of every JFrog key, for the summary.
	Families map[string]string
	// Errors carries parse problems from either side.
	Errors []string
}

Report is the outcome of comparing JFrog's operations with ours.

func Compare

func Compare(theirs, ours Oracle) Report

Compare checks every documented operation against ours. Descriptions are never compared: only what a client can observe on the wire.

func (Report) Diff

func (r Report) Diff(b Baseline) Diff

Diff compares the report with the accepted baseline.

func (Report) String

func (r Report) String() string

String renders the report in a stable order so it can be diffed.

func (Report) Summary

func (r Report) Summary() []FamilySummary

Summary counts outcomes per documentation family, sorted by family.

func (Report) WriteBaseline

func (r Report) WriteBaseline(path string) error

WriteBaseline records the report's current gaps as the accepted baseline.

type ScaffoldOptions

type ScaffoldOptions struct {
	// CacheDir is the oracle cache.
	CacheDir string
	// Family is the cache directory (index slug) to scaffold.
	Family string
	// ServiceBase is our service prefix, for example "/artifactory". Only
	// operations whose full path starts with it are emitted; the rest are
	// listed in Result.Skipped so the author can scaffold them elsewhere.
	ServiceBase string
	// Tag is written on every operation; defaults to the family's last word.
	Tag string
	// SkipSchemas are component names the service already defines (the shared
	// error envelope, for instance); references are kept but the schema is not
	// emitted again.
	SkipSchemas []string
}

ScaffoldOptions selects what to scaffold and how to name it.

type ScaffoldResult

type ScaffoldResult struct {
	YAML       []byte
	Operations int
	Schemas    int
	// Skipped lists operation keys outside ServiceBase.
	Skipped []string
	// Renamed maps original schema names to the family-prefixed names used
	// because another family defines the same name with a different shape.
	Renamed map[string]string
}

ScaffoldResult reports what was written and what the author must resolve.

func Scaffold

func Scaffold(opts ScaffoldOptions) (ScaffoldResult, error)

Scaffold converts every block of one family into an OpenAPI 3.0.3 fragment (paths plus components) with placeholder descriptions. Structure is a fact about the wire protocol; prose is not, so none of JFrog's is carried over.

type SchemaField

type SchemaField struct {
	Name     string   `json:"name"`
	Type     string   `json:"type,omitempty"`
	Format   string   `json:"format,omitempty"`
	Ref      string   `json:"ref,omitempty"`
	Items    string   `json:"items,omitempty"`
	Enum     []string `json:"enum,omitempty"`
	Required bool     `json:"required,omitempty"`
	ReadOnly bool     `json:"read_only,omitempty"`
	// WriteOnly marks fields JFrog accepts but never returns, such as passwords.
	WriteOnly bool `json:"write_only,omitempty"`
	Nullable  bool `json:"nullable,omitempty"`
	Default   any  `json:"default,omitempty"`
}

SchemaField is one property of a named schema.

type SchemaRecord

type SchemaRecord struct {
	Family   string        `json:"family"`
	Name     string        `json:"name"`
	Type     string        `json:"type,omitempty"`
	Composed []string      `json:"composed,omitempty"`
	Fields   []SchemaField `json:"fields,omitempty"`
	Variants int           `json:"variants"`
	UsedBy   int           `json:"used_by"`
}

SchemaRecord is a named component schema within one family. Variants counts the distinct shapes the same name has across that family's blocks; anything above one means authors must look at both pages before trusting the fields.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL