Help us shape the future of Go! Take the 2026 Go Developer Survey

compile

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 7, 2026 License: BSD-3-Clause Imports: 2 Imported by: 0

Documentation

Overview

Package compile lowers a syntax AST into the flat instruction program that the backtracking VM executes.

Index

Constants

View Source
const (
	// UTF8 is the default encoding: the dot and byte-oriented classes advance by
	// a whole UTF-8 code point.
	UTF8 = syntax.UTF8
	// ASCII8BIT is Ruby's binary (/n) encoding: every atom advances one byte.
	ASCII8BIT = syntax.ASCII8BIT
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Encoding

type Encoding = syntax.Encoding

Encoding selects how the byte-oriented input-advancing atoms — the dot (OpAny) and a byte-oriented character class (OpClass without a \p{…} member, code-point range, or /i fold) — traverse the input. It is the same type the parser uses (it governs lookbehind byte-width validation there).

In UTF8 mode (the default, matching Ruby's behaviour on a UTF-8 string) these atoms decode one full UTF-8 code point and advance by its byte length, so the dot matches a whole multi-byte character (`/./` on "é" consumes "é", as MRI does). In ASCII8BIT mode (Ruby's /n binary encoding) every atom advances a single byte, so the dot consumes one byte of a multi-byte sequence — the engine's original byte-oriented behaviour. The rune-aware atoms (OpFoldChar, OpUniProp, and a rune-aware OpClass) always decode a code point in UTF8 mode and are forced to a single byte in ASCII8BIT mode (where Unicode folding and properties operate per byte, ASCII-only). Match offsets are byte offsets in both modes.

type Inst

type Inst struct {
	Op         Op
	B          byte                 // OpChar
	Rune       rune                 // OpFoldChar
	X, Y       int                  // OpSplit, OpJmp
	GuardTo    int                  // OpSplit: where the empty-loop guard jumps on revisit
	Quant      bool                 // OpSplit: this split is a quantifier (*, +, ?, {m,n}) decision, not an alternation fork. Its GuardTo is the deterministic continuation past the whole quantifier, which the prefilter's mandatory-spine walk follows; an alternation split (Quant=false) is a true branch the spine must stop at.
	Slot       int                  // OpSave
	Ranges     []ast.ClassRange     // OpClass
	RuneRanges []ast.RuneClassRange // OpClass (rune-aware code-point ranges: literal multi-byte members in UTF8, /i-folded members, \R)
	Props      []ast.PropRef        // OpClass (rune-aware members)
	Prop       ast.PropRef          // OpUniProp
	Negate     bool                 // OpClass, OpLook
	Behind     bool                 // OpLook
	Fold       bool                 // OpClass (case-insensitive, /i: rune-aware folding)
	DotAll     bool                 // OpAny (Ruby /m: the dot also matches '\n')
	Min        int                  // OpLook (lookbehind width lower bound), OpLoop (min count)
	Max        int                  // OpLook (lookbehind width upper bound), OpLoop (max count, -1 unbounded)
	Sub        Op                   // OpLoop: the opcode of the single atom the loop repeats
	Greedy     bool                 // OpLoop: greedy (longest-first) vs lazy (shortest-first)
	// ByteSet is a 256-bit membership bitset over the byte-oriented Ranges of an
	// OpClass (bit b set iff byte b is in some range), precomputed at compile time
	// so the hot match loop tests class membership with a single indexed bit read
	// instead of a linear scan over Ranges. It is set on every OpClass (nil on all
	// other ops) and covers exactly Ranges; the rune-aware members (RuneRanges,
	// Props, folding) are unaffected and still decided by the range/property walk,
	// so results are identical — the bitset only accelerates the ASCII/byte portion.
	ByteSet *[4]uint64
}

Inst is a single VM instruction. Only the fields relevant to its Op are used.

func (*Inst) ClassHasByte

func (in *Inst) ClassHasByte(b byte) bool

ClassHasByte reports whether byte b is in the class's byte ranges, using the precomputed ByteSet when present (the common OpClass case) and falling back to a linear range scan otherwise. It does NOT apply Negate — callers apply that — so it is a drop-in for a raw range-membership test.

func (*Inst) ClassHasRune

func (in *Inst) ClassHasRune(r rune) bool

ClassHasRune reports whether code point r is in the class's byte ranges. Those ranges are byte-valued (0–255), so a code point above 255 is never a member; a code point in range is decided by the same 256-bit ByteSet. It backs the UTF8 non-rune-aware class test, where a multi-byte code point can only match through the byte ranges (RuneRanges/Props being empty). Negate is applied by the caller.

type Op

type Op int

Op is the opcode of a VM instruction.

const (
	// OpChar matches the single byte B and advances.
	OpChar Op = iota
	// OpFoldChar matches one UTF-8 code point case-insensitively (/i): it decodes
	// the code point at the cursor and accepts it when it is in the same simple
	// -case-folding orbit as Rune, advancing by that code point's byte length. It
	// is rune-aware, like OpUniProp.
	OpFoldChar
	// OpAny matches any byte and advances; unless DotAll is set it excludes
	// '\n' (Ruby's /m option makes the dot match a newline too).
	OpAny
	// OpClass matches a byte in (or, if Negate, not in) Ranges and advances. When
	// Props or RuneRanges is non-empty, or Fold is set, the class is rune-aware: it
	// decodes one UTF-8 code point and tests it against Ranges, RuneRanges and Props
	// before Negate is applied, advancing by the code point's byte length.
	OpClass
	// OpUniProp matches one UTF-8 code point that is a member of (or, if Negate,
	// not a member of) the Unicode property Prop, advancing by its byte length.
	OpUniProp
	// OpSplit forks: try X first, then Y on backtrack (greedy ordering).
	OpSplit
	// OpJmp jumps to X.
	OpJmp
	// OpSave records the current position into capture slot Slot.
	OpSave
	// OpAssertBeginText asserts the start of the input (\A).
	OpAssertBeginText
	// OpAssertEndText asserts the end of the input (\z).
	OpAssertEndText
	// OpAssertEndTextOptNL asserts end of input, allowing one trailing '\n' (\Z).
	OpAssertEndTextOptNL
	// OpAssertBeginLine asserts the start of a line (^).
	OpAssertBeginLine
	// OpAssertEndLine asserts the end of a line ($).
	OpAssertEndLine
	// OpBackref matches the text previously captured by group Slot.
	OpBackref
	// OpCall invokes a subexpression call (\g<…>): it pushes the return address
	// (the pc just past this instruction) onto the VM's call stack and jumps to X,
	// the entry pc of the referenced group's sub-program (group 0's entry is the
	// whole pattern). The called sub-program re-runs and re-captures, and on
	// reaching its closing OpReturn control returns to the saved address.
	OpCall
	// OpReturn ends the callable sub-program of the group whose index it carries in
	// Slot. It completes a \g<…> call only when the active call frame is a call to
	// *this* group (frame.group == Slot): it then pops that frame and jumps to the
	// saved return address. Otherwise — there is no active call, or the active call
	// targets an enclosing group and execution is merely passing linearly through a
	// nested group's terminator — it falls through to the next instruction. Tagging
	// the return with its group index is what lets a nested group's OpReturn be
	// skipped during an outer group's recursive call instead of stealing its frame.
	OpReturn
	// OpAssertPrevMatch asserts the position equals the scan/previous-match start
	// (\G).
	OpAssertPrevMatch
	// OpAssertWordBoundary asserts a word boundary (\b): the position lies between
	// a word character and a non-word character or a string edge. The word-char
	// notion is encoding-dependent and mirrors Onigmo/MRI's \b — a Unicode word
	// code point (\p{Word}) in UTF8 mode, an ASCII word byte ([0-9A-Za-z_]) in
	// ASCII8BIT mode — which the VM evaluates from the bytes surrounding the cursor.
	OpAssertWordBoundary
	// OpAssertNonWordBoundary asserts the complement of OpAssertWordBoundary (\B):
	// the position is NOT a word boundary.
	OpAssertNonWordBoundary
	// OpLook begins a lookaround assertion. Its sub-program is emitted inline
	// immediately after it and is terminated by OpLookEnd; X is the continuation
	// pc just past that OpLookEnd. Negate selects the negative form, Behind the
	// lookbehind form, and Min/Max bound the lookbehind width.
	OpLook
	// OpLookEnd marks a successful run of a lookaround sub-program.
	OpLookEnd
	// OpAtomicBegin opens an atomic (possessive) group (?>…): it records the
	// current backtrack-stack depth so the matching OpAtomicEnd can discard every
	// alternative created while the group's body matched. It is a no-op on input
	// position; only the backtrack stack is touched.
	OpAtomicBegin
	// OpAtomicEnd closes an atomic group: it truncates the backtrack stack back to
	// the depth its OpAtomicBegin recorded, dropping every backtrack point created
	// inside the group. After this the group's sub-match is committed — the engine
	// can never re-enter the body to try a shorter repetition or an alternate
	// sub-match — which is exactly the possessive/atomic barrier.
	OpAtomicEnd
	// OpMatch reports a successful match.
	OpMatch
	// OpLoop is a fused quantifier over a single input-consuming atom (OpChar,
	// OpClass, OpAny, OpUniProp, or OpFoldChar) with no captures or nested
	// branching inside it. It replaces the generic split/atom/jmp loop the compiler
	// would otherwise emit for X{Min,Max} when X is exactly one such atom, so a run
	// like [a-z]+ or .x's dot or \p{L}+ advances in one tight inner loop instead of
	// re-dispatching the outer switch (and touching the memo) per character. The
	// repeated atom is carried inline: Sub names which atom opcode the loop runs,
	// and the atom's own fields (B, Ranges, RuneRanges, Props, Prop, Negate, Fold,
	// DotAll, Rune) are reused on this same Inst to describe it. Min/Max bound the
	// repetition count (Max == -1 is unbounded) and Quant-style Greedy selects the
	// matching preference. X is the continuation pc just past the loop. It produces
	// exactly the same set of matches, in the same leftmost-first order, as the
	// unfused form: greedy consumes the maximal run then gives back one atom at a
	// time on backtrack; lazy consumes the minimum then takes one more atom at a
	// time when forced.
	OpLoop
)

type Program

type Program struct {
	Insts      []Inst
	NumCapture int
	Names      map[string]int
	HasBackref bool
	HasCall    bool
	// HasSplit records whether any instruction is an OpSplit (a quantifier or
	// alternation decision point). The VM consults the (pc, sp) memo only at
	// OpSplit, so a split-free program (a plain literal/class/dot run) never needs
	// the memo allocated at all; the VM uses this to skip that allocation and its
	// per-position reset, which on a long no-split scan would otherwise be O(n²).
	HasSplit bool
	// Enc is the input encoding (UTF8 by default, ASCII8BIT for binary /n). It
	// governs how the dot and byte-oriented classes advance: by a whole code
	// point in UTF8 mode, by one byte in ASCII8BIT mode.
	Enc Encoding
}

Program is a compiled regular expression: the instruction list, the number of capture groups (group 0 being the whole match), and the named-group map.

HasBackref records whether any instruction reads a capture (OpBackref). The VM uses it to decide whether (instruction, position) memoization is sound: with no backreference, captures are write-only and never influence whether a match can succeed, so two arrivals at the same (pc, sp) have identical futures and the later one can be pruned. A backreference makes the future depend on captured text, so memoization is disabled for such programs.

HasCall records whether any instruction is a subexpression call (OpCall). A call re-runs and re-captures a group, so like a backreference it makes the future depend on captured/recursive state; the VM therefore disables the persistent (pc, sp) memo for such programs and relies on the recursion-depth and step budgets to bound pathological recursion.

func Compile

func Compile(r syntax.Result) *Program

Compile turns a parse result into an executable program in the default UTF-8 encoding. It wraps the whole pattern in save slots 0/1 (the overall match span) and terminates with OpMatch.

func CompileEnc

func CompileEnc(r syntax.Result, enc Encoding) *Program

CompileEnc is Compile with an explicit input encoding (see Encoding). UTF8 makes the dot and byte-oriented classes advance by a whole code point; ASCII8BIT makes every atom advance one byte.

func (*Program) NumSlots

func (p *Program) NumSlots() int

NumSlots returns the number of save slots the VM must allocate (two per capture group, including group 0).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL