corpuscheck

package
v0.48.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 4, 2026 License: MIT Imports: 8 Imported by: 0

Documentation

Overview

Package corpuscheck parses the upstream tree-sitter corpus test format. It compares gotreesitter output with the expected syntax expressions.

A survey of approximately 190 upstream grammars found this format:

====================
test name
:attribute1
:attribute2
====================
<source text, verbatim, may be multiple lines>
---
<expected S-expression, may be multiple lines, may be empty>

Repeated for every test in the file. Variations this parser handles:

  • Header and divider lengths can vary in one file. Headers use three or more '=' characters. Result dividers use three or more '-' characters.
  • A marker can have a non-whitespace suffix. The parser reads the suffix from the opening header. The closing header and result divider must use the same suffix. This rule distinguishes markers from similar source lines.
  • Attribute lines can occur inside the header. A ':' at the start of source or expected output is ordinary content.
  • The parser accepts carriage return and line feed endings. It removes the carriage return before marker matching.
  • An ":error" test can omit the expected output.

ParseFile documents the unsupported format variants.

Index

Constants

View Source
const (
	CategoryShape                 = "shape"                       // nil-ness / child count differs
	CategoryType                  = "type"                        // node type name differs
	CategoryField                 = "field"                       // field name differs
	CategoryMissing               = "missing"                     // MISSING-ness differs
	CategoryMissingValue          = "missing_value"               // both MISSING, different token/type
	CategoryUnexpectedUnsupported = "unexpected_node_unsupported" // expected side uses UNEXPECTED; gotreesitter has no equivalent node kind
)

Mismatch categories. Keep these values stable because callers aggregate them.

Variables

This section is empty.

Functions

func DiscoverCorpusDirs

func DiscoverCorpusDirs(root string) ([]string, error)

DiscoverCorpusDirs recursively finds every directory literally named "corpus" (or, to also catch variants such as tree-sitter-foam's "corpus_windows" CRLF suite, named with a "corpus" prefix) under root. Multi-grammar repos (typescript+tsx, markdown+markdown-inline, apex's three sub-grammars, ...) surface as multiple directories; callers that only want the single dominant grammar for a language name should use the first (shortest-path) entry.

Types

type CaseResult

type CaseResult struct {
	File   string
	Name   string
	Line   int
	Reason string // skip reason, or mismatch category on failure
	Detail string
	Path   string // divergence path, if applicable
}

CaseResult records one test case's outcome.

type CompareOptions

type CompareOptions struct {
	// IgnoreMissingExpectedFields accepts a missing expected field when the
	// actual tree has one. It does not accept the reverse case.
	// It also rejects two different field names.
	// Some upstream repositories added fields without updating their fixtures.
	// Repository histories confirm this state for JSON and Go.
	// Use this option only as a labeled secondary measure.
	IgnoreMissingExpectedFields bool
}

CompareOptions controls how much latitude Compare gives the comparison. The zero value is the fully strict comparison Compare uses.

type Divergence

type Divergence struct {
	// Path identifies the mismatching node.
	// Example: "/source_file/function_declaration[0]/block[2]".
	Path string
	// Category classifies the mismatch for aggregate reporting.
	Category string
	Expected string
	Actual   string
}

Divergence describes the first place two SNode trees stop matching.

func Compare

func Compare(expected, actual *SNode) *Divergence

Compare walks both trees in lockstep. It returns the first difference. It returns nil when every type, field, missing state, and child matches.

func CompareWithOptions

func CompareWithOptions(expected, actual *SNode, opts CompareOptions) *Divergence

CompareWithOptions is Compare with configurable leniency; see CompareOptions.

func (*Divergence) String

func (d *Divergence) String() string

type FileResult

type FileResult struct {
	Path             string
	Cases            int
	Pass             int
	PassFieldLenient int
	Fail             []CaseResult
	Skipped          []CaseResult
	// FormatErrors holds test-framing errors ParseFile recovered from
	// (see TestCase.ParseError); each one means at least one test in the
	// file could not be extracted at all.
	FormatErrors []error
}

FileResult is the outcome for one corpus file.

type LanguageReport

type LanguageReport struct {
	Language   string
	CorpusRoot string
	Files      int
	Cases      int
	Pass       int
	// PassFieldLenient counts cases that pass under the strict
	// comparison OR whose only divergences are "fixture has no field
	// name, gotreesitter's actual tree does" (see
	// CompareOptions.IgnoreMissingExpectedFields). This is a secondary,
	// clearly-separate lens: several upstream grammar repos' own corpus
	// fixtures are stale with respect to fields their grammar.js added
	// years ago (verified via each repo's git history for json and go;
	// see the corpuscheck report). It is never folded into Pass.
	PassFieldLenient int
	Skipped          []CaseResult
	Failed           []CaseResult
	FormatErrors     []error
	// SkipReason, if non-empty, means the whole language was not run at
	// all (e.g. unsupported parse backend, corpus directory not found).
	SkipReason string
}

LanguageReport aggregates every file's results for one language.

func RunLanguage

func RunLanguage(language, corpusDir string) *LanguageReport

RunLanguage parses every corpus file under corpusDir with language's registered grammar loader and compares each test case against its upstream expected S-expression.

func RunLanguageLimited

func RunLanguageLimited(language, corpusDir string, maxCases int) *LanguageReport

RunLanguageLimited behaves like RunLanguage but stops issuing new parses once report.Cases reaches maxCases (0 means unlimited).

func RunLanguageWithOptions

func RunLanguageWithOptions(language, corpusDir string, opts RunOptions) *LanguageReport

RunLanguageWithOptions is RunLanguage with full control over RunOptions. See RunOptions for the -- language() tag filtering and per-language case cap -- knobs it exposes.

func (*LanguageReport) FailCategoryCounts

func (r *LanguageReport) FailCategoryCounts() map[string]int

FailCategoryCounts tallies Failed by Reason.

func (*LanguageReport) SkipReasonCounts

func (r *LanguageReport) SkipReasonCounts() map[string]int

SkipReasonCounts tallies Skipped by Reason.

type RunOptions

type RunOptions struct {
	// MaxCases stops issuing new parses once the report's Cases count
	// reaches this many (0 means unlimited).
	MaxCases int
	// RequireLanguageTag is for languages whose fixtures live inside
	// another language's shared corpus directory, distinguished only by
	// a `:language(name)` attribute (e.g. tsx and dtd both live inside
	// typescript's and xml's corpus directories respectively, alongside
	// untagged typescript/xml tests). When true, a test case with no
	// `:language()` attribute at all is skipped rather than treated as
	// belonging to this language.
	RequireLanguageTag bool
}

RunOptions controls RunLanguageWithOptions.

type SNode

type SNode struct {
	// Field is the field name this node is attached under in its parent,
	// or "" if none.
	Field string
	// Type is the node's type name: a grammar rule name for ordinary
	// nodes, "ERROR" for error nodes, or (for a Missing anonymous token)
	// the token's literal text.
	Type string
	// Missing marks a MISSING node (always a leaf).
	Missing bool
	// MissingIsAnon marks that Type is a literal anonymous token's text
	// (was double-quoted in the fixture) rather than a named node type.
	MissingIsAnon bool
	// Unexpected marks an UNEXPECTED node -- the C tree-sitter runtime's
	// marker for a single lexically-unrecognized byte inside an ERROR
	// node. gotreesitter has no equivalent concept (see compare.go); this
	// flag exists purely so the comparator can name that gap explicitly
	// instead of reporting a generic shape mismatch.
	Unexpected bool
	// Children holds this node's named children, in order.
	Children []*SNode
}

SNode is a canonical, whitespace-insensitive representation of one node in an S-expression tree -- either the one a grammar author wrote by hand in a corpus fixture, or the one gotreesitter's own tree renders to. Both sides are converted to this shape so that comparison never has to worry about indentation, line wrapping, or trailing newlines.

func ParseExpected

func ParseExpected(text string) (*SNode, error)

ParseExpected parses the expected-output text from a corpus test case (everything after the "---" divider) into a canonical SNode tree.

Supported: node types, field names ("field: (...)"), ERROR nodes, and MISSING nodes (both quoted-anonymous-token and bare-named-type forms, e.g. `(MISSING ".")` and `(MISSING identifier)`).

NOT supported: UNEXPECTED nodes are recognized syntactically (so a fixture that contains one doesn't fail to parse) but are flagged via SNode.Unexpected rather than modeled with real content, since gotreesitter has no equivalent node kind to compare them against.

func RenderTree

func RenderTree(root *gotreesitter.Node, lang *gotreesitter.Language) *SNode

RenderTree converts a live gotreesitter parse tree into the same canonical SNode shape ParseExpected produces from a corpus fixture's text, so the two can be compared structurally without caring about whitespace or indentation.

A node is included if it is named, or if it is a MISSING node (including an anonymous MISSING token, which tree-sitter's own s-expression dump shows even though the corresponding present token would be invisible). Every other unnamed node (ordinary anonymous punctuation/keyword tokens) is skipped entirely, matching upstream tree-sitter's own S-expression dump.

type TestCase

type TestCase struct {
	// Name is the test title. An anonymous test can have an empty name.
	Name string
	// Attrs contains the header attributes without the leading ':'.
	Attrs []string
	// Input is the verbatim source text for this test.
	Input []byte
	// Expected contains the raw text after the result divider.
	// An ":error" test can omit this text.
	Expected string
	// Line is the one-based line number of the opening header marker.
	Line int
	// ParseError contains a test framing error.
	// Input and Expected are not reliable when ParseError is not nil.
	ParseError error
}

TestCase is one corpus test extracted from a corpus file.

func ParseFile

func ParseFile(data []byte) ([]TestCase, error)

ParseFile parses one corpus file's contents into its test cases.

Known format variants this parser does NOT support (found during the survey but rare enough, or upstream-tool-specific enough, that implementing them was not worth the complexity for a comparison harness):

  • Nothing observed forced an outright parse failure across the surveyed languages; the recovery strategy below (re-synchronize on the next header marker) is believed sufficient for any remaining oddities. If a file produces zero test cases where it plainly has content, that is reported as a parse error by the caller, not silently swallowed.

func (*TestCase) AttrValue

func (tc *TestCase) AttrValue(name string) (string, bool)

AttrValue returns the parenthesized value of the named attribute. It also reports whether the attribute is present.

func (*TestCase) HasAttr

func (tc *TestCase) HasAttr(name string) bool

HasAttr reports whether the test has the named attribute. A valued attribute also matches its bare name.

Directories

Path Synopsis
cmd
corpuscheck command
Command corpuscheck runs gotreesitter against upstream tree-sitter corpora.
Command corpuscheck runs gotreesitter against upstream tree-sitter corpora.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL