Documentation
¶
Overview ¶
Package compile lowers a syntax AST into the flat instruction program that the backtracking VM executes.
Index ¶
Constants ¶
const ( // UTF8 is the default encoding: the dot and byte-oriented classes advance by // a whole UTF-8 code point. UTF8 = syntax.UTF8 // ASCII8BIT is Ruby's binary (/n) encoding: every atom advances one byte. ASCII8BIT = syntax.ASCII8BIT )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Encoding ¶
Encoding selects how the byte-oriented input-advancing atoms — the dot (OpAny) and a byte-oriented character class (OpClass without a \p{…} member, code-point range, or /i fold) — traverse the input. It is the same type the parser uses (it governs lookbehind byte-width validation there).
In UTF8 mode (the default, matching Ruby's behaviour on a UTF-8 string) these atoms decode one full UTF-8 code point and advance by its byte length, so the dot matches a whole multi-byte character (`/./` on "é" consumes "é", as MRI does). In ASCII8BIT mode (Ruby's /n binary encoding) every atom advances a single byte, so the dot consumes one byte of a multi-byte sequence — the engine's original byte-oriented behaviour. The rune-aware atoms (OpFoldChar, OpUniProp, and a rune-aware OpClass) always decode a code point in UTF8 mode and are forced to a single byte in ASCII8BIT mode (where Unicode folding and properties operate per byte, ASCII-only). Match offsets are byte offsets in both modes.
type Inst ¶
type Inst struct {
Op Op
B byte // OpChar
Rune rune // OpFoldChar
X, Y int // OpSplit, OpJmp
GuardTo int // OpSplit: where the empty-loop guard jumps on revisit
Quant bool // OpSplit: this split is a quantifier (*, +, ?, {m,n}) decision, not an alternation fork. Its GuardTo is the deterministic continuation past the whole quantifier, which the prefilter's mandatory-spine walk follows; an alternation split (Quant=false) is a true branch the spine must stop at.
Slot int // OpSave
Ranges []ast.ClassRange // OpClass
RuneRanges []ast.RuneClassRange // OpClass (rune-aware code-point ranges: literal multi-byte members in UTF8, /i-folded members, \R)
Props []ast.PropRef // OpClass (rune-aware members)
Prop ast.PropRef // OpUniProp
Negate bool // OpClass, OpLook
Behind bool // OpLook
Fold bool // OpClass (case-insensitive, /i: rune-aware folding)
DotAll bool // OpAny (Ruby /m: the dot also matches '\n')
Min int // OpLook (lookbehind width lower bound), OpLoop (min count)
Max int // OpLook (lookbehind width upper bound), OpLoop (max count, -1 unbounded)
Sub Op // OpLoop: the opcode of the single atom the loop repeats
Greedy bool // OpLoop: greedy (longest-first) vs lazy (shortest-first)
// ByteSet is a 256-bit membership bitset over the byte-oriented Ranges of an
// OpClass (bit b set iff byte b is in some range), precomputed at compile time
// so the hot match loop tests class membership with a single indexed bit read
// instead of a linear scan over Ranges. It is set on every OpClass (nil on all
// other ops) and covers exactly Ranges; the rune-aware members (RuneRanges,
// Props, folding) are unaffected and still decided by the range/property walk,
// so results are identical — the bitset only accelerates the ASCII/byte portion.
ByteSet *[4]uint64
}
Inst is a single VM instruction. Only the fields relevant to its Op are used.
func (*Inst) ClassHasByte ¶
ClassHasByte reports whether byte b is in the class's byte ranges, using the precomputed ByteSet when present (the common OpClass case) and falling back to a linear range scan otherwise. It does NOT apply Negate — callers apply that — so it is a drop-in for a raw range-membership test.
func (*Inst) ClassHasRune ¶
ClassHasRune reports whether code point r is in the class's byte ranges. Those ranges are byte-valued (0–255), so a code point above 255 is never a member; a code point in range is decided by the same 256-bit ByteSet. It backs the UTF8 non-rune-aware class test, where a multi-byte code point can only match through the byte ranges (RuneRanges/Props being empty). Negate is applied by the caller.
type Op ¶
type Op int
Op is the opcode of a VM instruction.
const ( // OpChar matches the single byte B and advances. OpChar Op = iota // OpFoldChar matches one UTF-8 code point case-insensitively (/i): it decodes // the code point at the cursor and accepts it when it is in the same simple // -case-folding orbit as Rune, advancing by that code point's byte length. It // is rune-aware, like OpUniProp. OpFoldChar // OpAny matches any byte and advances; unless DotAll is set it excludes // '\n' (Ruby's /m option makes the dot match a newline too). OpAny // OpClass matches a byte in (or, if Negate, not in) Ranges and advances. When // Props or RuneRanges is non-empty, or Fold is set, the class is rune-aware: it // decodes one UTF-8 code point and tests it against Ranges, RuneRanges and Props // before Negate is applied, advancing by the code point's byte length. OpClass // OpUniProp matches one UTF-8 code point that is a member of (or, if Negate, // not a member of) the Unicode property Prop, advancing by its byte length. OpUniProp // OpSplit forks: try X first, then Y on backtrack (greedy ordering). OpSplit // OpJmp jumps to X. OpJmp // OpSave records the current position into capture slot Slot. OpSave // OpAssertBeginText asserts the start of the input (\A). OpAssertBeginText // OpAssertEndText asserts the end of the input (\z). OpAssertEndText // OpAssertEndTextOptNL asserts end of input, allowing one trailing '\n' (\Z). OpAssertEndTextOptNL // OpAssertBeginLine asserts the start of a line (^). OpAssertBeginLine // OpAssertEndLine asserts the end of a line ($). OpAssertEndLine // OpBackref matches the text previously captured by group Slot. OpBackref // OpCall invokes a subexpression call (\g<…>): it pushes the return address // (the pc just past this instruction) onto the VM's call stack and jumps to X, // the entry pc of the referenced group's sub-program (group 0's entry is the // whole pattern). The called sub-program re-runs and re-captures, and on // reaching its closing OpReturn control returns to the saved address. OpCall // OpReturn ends the callable sub-program of the group whose index it carries in // Slot. It completes a \g<…> call only when the active call frame is a call to // *this* group (frame.group == Slot): it then pops that frame and jumps to the // saved return address. Otherwise — there is no active call, or the active call // targets an enclosing group and execution is merely passing linearly through a // nested group's terminator — it falls through to the next instruction. Tagging // the return with its group index is what lets a nested group's OpReturn be // skipped during an outer group's recursive call instead of stealing its frame. OpReturn // OpAssertPrevMatch asserts the position equals the scan/previous-match start // (\G). OpAssertPrevMatch // OpAssertWordBoundary asserts a word boundary (\b): the position lies between // a word character and a non-word character or a string edge. The word-char // notion is encoding-dependent and mirrors Onigmo/MRI's \b — a Unicode word // code point (\p{Word}) in UTF8 mode, an ASCII word byte ([0-9A-Za-z_]) in // ASCII8BIT mode — which the VM evaluates from the bytes surrounding the cursor. OpAssertWordBoundary // OpAssertNonWordBoundary asserts the complement of OpAssertWordBoundary (\B): // the position is NOT a word boundary. OpAssertNonWordBoundary // OpLook begins a lookaround assertion. Its sub-program is emitted inline // immediately after it and is terminated by OpLookEnd; X is the continuation // pc just past that OpLookEnd. Negate selects the negative form, Behind the // lookbehind form, and Min/Max bound the lookbehind width. OpLook // OpLookEnd marks a successful run of a lookaround sub-program. OpLookEnd // OpAtomicBegin opens an atomic (possessive) group (?>…): it records the // current backtrack-stack depth so the matching OpAtomicEnd can discard every // alternative created while the group's body matched. It is a no-op on input // position; only the backtrack stack is touched. OpAtomicBegin // OpAtomicEnd closes an atomic group: it truncates the backtrack stack back to // the depth its OpAtomicBegin recorded, dropping every backtrack point created // inside the group. After this the group's sub-match is committed — the engine // can never re-enter the body to try a shorter repetition or an alternate // sub-match — which is exactly the possessive/atomic barrier. OpAtomicEnd // OpMatch reports a successful match. OpMatch // OpLoop is a fused quantifier over a single input-consuming atom (OpChar, // OpClass, OpAny, OpUniProp, or OpFoldChar) with no captures or nested // branching inside it. It replaces the generic split/atom/jmp loop the compiler // would otherwise emit for X{Min,Max} when X is exactly one such atom, so a run // like [a-z]+ or .x's dot or \p{L}+ advances in one tight inner loop instead of // re-dispatching the outer switch (and touching the memo) per character. The // repeated atom is carried inline: Sub names which atom opcode the loop runs, // and the atom's own fields (B, Ranges, RuneRanges, Props, Prop, Negate, Fold, // DotAll, Rune) are reused on this same Inst to describe it. Min/Max bound the // repetition count (Max == -1 is unbounded) and Quant-style Greedy selects the // matching preference. X is the continuation pc just past the loop. It produces // exactly the same set of matches, in the same leftmost-first order, as the // unfused form: greedy consumes the maximal run then gives back one atom at a // time on backtrack; lazy consumes the minimum then takes one more atom at a // time when forced. OpLoop )
type Program ¶
type Program struct {
Insts []Inst
NumCapture int
Names map[string]int
HasBackref bool
HasCall bool
// HasSplit records whether any instruction is an OpSplit (a quantifier or
// alternation decision point). The VM consults the (pc, sp) memo only at
// OpSplit, so a split-free program (a plain literal/class/dot run) never needs
// the memo allocated at all; the VM uses this to skip that allocation and its
// per-position reset, which on a long no-split scan would otherwise be O(n²).
HasSplit bool
// Enc is the input encoding (UTF8 by default, ASCII8BIT for binary /n). It
// governs how the dot and byte-oriented classes advance: by a whole code
// point in UTF8 mode, by one byte in ASCII8BIT mode.
Enc Encoding
}
Program is a compiled regular expression: the instruction list, the number of capture groups (group 0 being the whole match), and the named-group map.
HasBackref records whether any instruction reads a capture (OpBackref). The VM uses it to decide whether (instruction, position) memoization is sound: with no backreference, captures are write-only and never influence whether a match can succeed, so two arrivals at the same (pc, sp) have identical futures and the later one can be pruned. A backreference makes the future depend on captured text, so memoization is disabled for such programs.
HasCall records whether any instruction is a subexpression call (OpCall). A call re-runs and re-captures a group, so like a backreference it makes the future depend on captured/recursive state; the VM therefore disables the persistent (pc, sp) memo for such programs and relies on the recursion-depth and step budgets to bound pathological recursion.
func Compile ¶
Compile turns a parse result into an executable program in the default UTF-8 encoding. It wraps the whole pattern in save slots 0/1 (the overall match span) and terminates with OpMatch.
func CompileEnc ¶
CompileEnc is Compile with an explicit input encoding (see Encoding). UTF8 makes the dot and byte-oriented classes advance by a whole code point; ASCII8BIT makes every atom advance one byte.