compile

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 7, 2026 License: BSD-3-Clause Imports: 2 Imported by: 0

Documentation

Overview

Package compile lowers a syntax AST into the flat instruction program that the backtracking VM executes.

Index

Constants

View Source
const (
	// UTF8 is the default encoding: the dot and byte-oriented classes advance by
	// a whole UTF-8 code point.
	UTF8 = syntax.UTF8
	// ASCII8BIT is Ruby's binary (/n) encoding: every atom advances one byte.
	ASCII8BIT = syntax.ASCII8BIT
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Encoding

type Encoding = syntax.Encoding

Encoding selects how the byte-oriented input-advancing atoms — the dot (OpAny) and a byte-oriented character class (OpClass without a \p{…} member, code-point range, or /i fold) — traverse the input. It is the same type the parser uses (it governs lookbehind byte-width validation there).

In UTF8 mode (the default, matching Ruby's behaviour on a UTF-8 string) these atoms decode one full UTF-8 code point and advance by its byte length, so the dot matches a whole multi-byte character (`/./` on "é" consumes "é", as MRI does). In ASCII8BIT mode (Ruby's /n binary encoding) every atom advances a single byte, so the dot consumes one byte of a multi-byte sequence — the engine's original byte-oriented behaviour. The rune-aware atoms (OpFoldChar, OpUniProp, and a rune-aware OpClass) always decode a code point in UTF8 mode and are forced to a single byte in ASCII8BIT mode (where Unicode folding and properties operate per byte, ASCII-only). Match offsets are byte offsets in both modes.

type Inst

type Inst struct {
	Op         Op
	B          byte                 // OpChar
	Rune       rune                 // OpFoldChar
	X, Y       int                  // OpSplit, OpJmp
	GuardTo    int                  // OpSplit: where the empty-loop guard jumps on revisit
	Quant      bool                 // OpSplit: this split is a quantifier (*, +, ?, {m,n}) decision, not an alternation fork. Its GuardTo is the deterministic continuation past the whole quantifier, which the prefilter's mandatory-spine walk follows; an alternation split (Quant=false) is a true branch the spine must stop at.
	Slot       int                  // OpSave
	Ranges     []ast.ClassRange     // OpClass
	RuneRanges []ast.RuneClassRange // OpClass (rune-aware code-point ranges: literal multi-byte members in UTF8, /i-folded members, \R)
	Props      []ast.PropRef        // OpClass (rune-aware members)
	Prop       ast.PropRef          // OpUniProp
	Negate     bool                 // OpClass, OpLook
	Behind     bool                 // OpLook
	Fold       bool                 // OpClass (case-insensitive, /i: rune-aware folding)
	DotAll     bool                 // OpAny (Ruby /m: the dot also matches '\n')
	Min        int                  // OpLook (lookbehind width lower bound), OpLoop (min count)
	Max        int                  // OpLook (lookbehind width upper bound), OpLoop (max count, -1 unbounded)
	Sub        Op                   // OpLoop: the opcode of the single atom the loop repeats
	Greedy     bool                 // OpLoop: greedy (longest-first) vs lazy (shortest-first)
	// ByteSet is a 256-bit membership bitset over the byte-oriented Ranges of an
	// OpClass (bit b set iff byte b is in some range), precomputed at compile time
	// so the hot match loop tests class membership with a single indexed bit read
	// instead of a linear scan over Ranges. It is set on every OpClass (nil on all
	// other ops) and covers exactly Ranges; the rune-aware members (RuneRanges,
	// Props, folding) are unaffected and still decided by the range/property walk,
	// so results are identical — the bitset only accelerates the ASCII/byte portion.
	ByteSet *[4]uint64
}

Inst is a single VM instruction. Only the fields relevant to its Op are used.

func (*Inst) ClassHasByte

func (in *Inst) ClassHasByte(b byte) bool

ClassHasByte reports whether byte b is in the class's byte ranges, using the precomputed ByteSet when present (the common OpClass case) and falling back to a linear range scan otherwise. It does NOT apply Negate — callers apply that — so it is a drop-in for a raw range-membership test.

func (*Inst) ClassHasRune

func (in *Inst) ClassHasRune(r rune) bool

ClassHasRune reports whether code point r is in the class's byte ranges. Those ranges are byte-valued (0–255), so a code point above 255 is never a member; a code point in range is decided by the same 256-bit ByteSet. It backs the UTF8 non-rune-aware class test, where a multi-byte code point can only match through the byte ranges (RuneRanges/Props being empty). Negate is applied by the caller.

type Op

type Op int

Op is the opcode of a VM instruction.

const (
	// OpChar matches the single byte B and advances.
	OpChar Op = iota
	// OpFoldChar matches one UTF-8 code point case-insensitively (/i): it decodes
	// the code point at the cursor and accepts it when it is in the same simple
	// -case-folding orbit as Rune, advancing by that code point's byte length. It
	// is rune-aware, like OpUniProp.
	OpFoldChar
	// OpAny matches any byte and advances; unless DotAll is set it excludes
	// '\n' (Ruby's /m option makes the dot match a newline too).
	OpAny
	// OpClass matches a byte in (or, if Negate, not in) Ranges and advances. When
	// Props or RuneRanges is non-empty, or Fold is set, the class is rune-aware: it
	// decodes one UTF-8 code point and tests it against Ranges, RuneRanges and Props
	// before Negate is applied, advancing by the code point's byte length.
	OpClass
	// OpUniProp matches one UTF-8 code point that is a member of (or, if Negate,
	// not a member of) the Unicode property Prop, advancing by its byte length.
	OpUniProp
	// OpSplit forks: try X first, then Y on backtrack (greedy ordering).
	OpSplit
	// OpJmp jumps to X.
	OpJmp
	// OpSave records the current position into capture slot Slot.
	OpSave
	// OpAssertBeginText asserts the start of the input (\A).
	OpAssertBeginText
	// OpAssertEndText asserts the end of the input (\z).
	OpAssertEndText
	// OpAssertEndTextOptNL asserts end of input, allowing one trailing '\n' (\Z).
	OpAssertEndTextOptNL
	// OpAssertBeginLine asserts the start of a line (^).
	OpAssertBeginLine
	// OpAssertEndLine asserts the end of a line ($).
	OpAssertEndLine
	// OpBackref matches the text previously captured by group Slot.
	OpBackref
	// OpCall invokes a subexpression call (\g<…>): it pushes the return address
	// (the pc just past this instruction) onto the VM's call stack and jumps to X,
	// the entry pc of the referenced group's sub-program (group 0's entry is the
	// whole pattern). The called sub-program re-runs and re-captures, and on
	// reaching its closing OpReturn control returns to the saved address.
	OpCall
	// OpReturn ends the callable sub-program of the group whose index it carries in
	// Slot. It completes a \g<…> call only when the active call frame is a call to
	// *this* group (frame.group == Slot): it then pops that frame and jumps to the
	// saved return address. Otherwise — there is no active call, or the active call
	// targets an enclosing group and execution is merely passing linearly through a
	// nested group's terminator — it falls through to the next instruction. Tagging
	// the return with its group index is what lets a nested group's OpReturn be
	// skipped during an outer group's recursive call instead of stealing its frame.
	OpReturn
	// OpAssertPrevMatch asserts the position equals the scan/previous-match start
	// (\G).
	OpAssertPrevMatch
	// OpAssertWordBoundary asserts a word boundary (\b): the position lies between
	// a word character and a non-word character or a string edge. The word-char
	// notion is encoding-dependent and mirrors Onigmo/MRI's \b — a Unicode word
	// code point (\p{Word}) in UTF8 mode, an ASCII word byte ([0-9A-Za-z_]) in
	// ASCII8BIT mode — which the VM evaluates from the bytes surrounding the cursor.
	OpAssertWordBoundary
	// OpAssertNonWordBoundary asserts the complement of OpAssertWordBoundary (\B):
	// the position is NOT a word boundary.
	OpAssertNonWordBoundary
	// OpLook begins a lookaround assertion. Its sub-program is emitted inline
	// immediately after it and is terminated by OpLookEnd; X is the continuation
	// pc just past that OpLookEnd. Negate selects the negative form, Behind the
	// lookbehind form, and Min/Max bound the lookbehind width.
	OpLook
	// OpLookEnd marks a successful run of a lookaround sub-program.
	OpLookEnd
	// OpAtomicBegin opens an atomic (possessive) group (?>…): it records the
	// current backtrack-stack depth so the matching OpAtomicEnd can discard every
	// alternative created while the group's body matched. It is a no-op on input
	// position; only the backtrack stack is touched.
	OpAtomicBegin
	// OpAtomicEnd closes an atomic group: it truncates the backtrack stack back to
	// the depth its OpAtomicBegin recorded, dropping every backtrack point created
	// inside the group. After this the group's sub-match is committed — the engine
	// can never re-enter the body to try a shorter repetition or an alternate
	// sub-match — which is exactly the possessive/atomic barrier.
	OpAtomicEnd
	// OpMatch reports a successful match.
	OpMatch
	// OpLoop is a fused quantifier over a single input-consuming atom (OpChar,
	// OpClass, OpAny, OpUniProp, or OpFoldChar) with no captures or nested
	// branching inside it. It replaces the generic split/atom/jmp loop the compiler
	// would otherwise emit for X{Min,Max} when X is exactly one such atom, so a run
	// like [a-z]+ or .x's dot or \p{L}+ advances in one tight inner loop instead of
	// re-dispatching the outer switch (and touching the memo) per character. The
	// repeated atom is carried inline: Sub names which atom opcode the loop runs,
	// and the atom's own fields (B, Ranges, RuneRanges, Props, Prop, Negate, Fold,
	// DotAll, Rune) are reused on this same Inst to describe it. Min/Max bound the
	// repetition count (Max == -1 is unbounded) and Quant-style Greedy selects the
	// matching preference. X is the continuation pc just past the loop. It produces
	// exactly the same set of matches, in the same leftmost-first order, as the
	// unfused form: greedy consumes the maximal run then gives back one atom at a
	// time on backtrack; lazy consumes the minimum then takes one more atom at a
	// time when forced.
	OpLoop
)

type Program

type Program struct {
	Insts      []Inst
	NumCapture int
	Names      map[string]int
	HasBackref bool
	HasCall    bool
	// HasSplit records whether any instruction is an OpSplit (a quantifier or
	// alternation decision point). The VM consults the (pc, sp) memo only at
	// OpSplit, so a split-free program (a plain literal/class/dot run) never needs
	// the memo allocated at all; the VM uses this to skip that allocation and its
	// per-position reset, which on a long no-split scan would otherwise be O(n²).
	HasSplit bool
	// Enc is the input encoding (UTF8 by default, ASCII8BIT for binary /n). It
	// governs how the dot and byte-oriented classes advance: by a whole code
	// point in UTF8 mode, by one byte in ASCII8BIT mode.
	Enc Encoding
}

Program is a compiled regular expression: the instruction list, the number of capture groups (group 0 being the whole match), and the named-group map.

HasBackref records whether any instruction reads a capture (OpBackref). The VM uses it to decide whether (instruction, position) memoization is sound: with no backreference, captures are write-only and never influence whether a match can succeed, so two arrivals at the same (pc, sp) have identical futures and the later one can be pruned. A backreference makes the future depend on captured text, so memoization is disabled for such programs.

HasCall records whether any instruction is a subexpression call (OpCall). A call re-runs and re-captures a group, so like a backreference it makes the future depend on captured/recursive state; the VM therefore disables the persistent (pc, sp) memo for such programs and relies on the recursion-depth and step budgets to bound pathological recursion.

func Compile

func Compile(r syntax.Result) *Program

Compile turns a parse result into an executable program in the default UTF-8 encoding. It wraps the whole pattern in save slots 0/1 (the overall match span) and terminates with OpMatch.

func CompileEnc

func CompileEnc(r syntax.Result, enc Encoding) *Program

CompileEnc is Compile with an explicit input encoding (see Encoding). UTF8 makes the dot and byte-oriented classes advance by a whole code point; ASCII8BIT makes every atom advance one byte.

func (*Program) NumSlots

func (p *Program) NumSlots() int

NumSlots returns the number of save slots the VM must allocate (two per capture group, including group 0).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL