README
¶
Scanner
A scanner takes a string in input and returns an array of tokens.
graph LR
s[ ] --> |source| a(scanner)
--> |tokens| b(parser)
--> |AST| c[ ]
subgraph scanner
a
end
style s height:0px;
style c height:0px;
Tokens can be of the following kinds:
- identifier
- number
- operator
- separator
- string
- block
Resolving nested blocks in the scanner is making the parser simple and generic, without having to resort to parse tables.
The lexical rules are provided by a language specification at language level which includes the following:
- a set of composable properties (1 per bit, on an integer) for each character in the ASCII range (where all separator, operators and reserved keywords must be defined).
- for each block or string, the specification of starting and ending delimiter.
Development status
A successful test must be provided to check the status.
- numbers starting with a digit
- numbers starting otherwise
- unescaped strings (including multiline)
- escaped string (including multiline)
- separators (in UTF-8 range)
- single line string (\n not allowed)
- identifiers (in UTF-8 range)
- operators, concatenated or not
- single character block/string delimiters
- arbitrarly nested blocks and strings
- multiple characters block/string delimiters
- semi-colon automatic insertion after newline
- blocks delimited by operator characters
- blocks delimited by identifiers
- blocks with delimiter inclusion/exclusion rules
- blocks delimited by indentation level (python, yaml, ...)
Documentation
¶
Overview ¶
Package scan provide a language independent scanner.
Index ¶
- Variables
- type Scanner
- type Source
- type Sources
- func (ss *Sources) Add(name, src string) int
- func (ss Sources) ByName(name string) *Source
- func (ss Sources) FormatPos(pos int) string
- func (ss Sources) LineText(pos int) string
- func (ss Sources) Resolve(pos int) (name string, line, col int)
- func (ss Sources) Snippet(pos int) string
- func (ss Sources) SourceIndex(pos int) int
- type Token
Constants ¶
This section is empty.
Variables ¶
var ( ErrBlock = errors.New("block not terminated") ErrIllegal = errors.New("illegal token") )
Error definitions.
Functions ¶
This section is empty.
Types ¶
type Scanner ¶
type Scanner struct {
*lang.Spec
Sources Sources // source position registry (multi-file / REPL)
PosBase int // base offset for current source
// contains filtered or unexported fields
}
Scanner contains the scanner rules for a language.
func NewScanner ¶
NewScanner returns a new scanner for a given language specification.
type Source ¶
type Source struct {
Name string
Base int // base byte offset in the unified position space
Len int // length in bytes
// contains filtered or unexported fields
}
Source describes a source text.
type Sources ¶
type Sources []Source
Sources is an ordered list of Source entries.
func (Sources) ByName ¶ added in v0.2.0
ByName returns the source with the given name, or nil if not found. When multiple sources share a name (e.g. the REPL's anonymous chunks), the first registered match is returned.
func (Sources) LineText ¶ added in v0.2.0
LineText returns the source line containing pos, without the trailing newline. Returns "" if pos is out of range.
func (Sources) Resolve ¶
Resolve converts a global byte offset to (source name, line, col). Returns ("", 0, 0) if pos is out of range.
func (Sources) Snippet ¶ added in v0.3.0
Snippet renders the source line containing pos plus a caret pointing at pos, as
\n <line> | <text>\n <pad>^\n
Returns "" when pos has no resolvable source line. Shared by the runtime PanicError renderer (vm) and compile-time diagnostics (interp.Eval).
func (Sources) SourceIndex ¶ added in v0.4.0
SourceIndex returns the index of the source containing pos, or -1 if out of range. The index is the per-file tag used to scope import aliases.
type Token ¶
type Token struct {
Tok lang.Token // token identificator
Pos int // position in source
Str string // string in source
Beg int // length of begin delimiter (block, string)
End int // length of end delimiter (block, string)
}
Token defines a scanner token.
func (*Token) Describe ¶ added in v0.3.0
Describe returns a short human-readable form of t for diagnostics: the token kind plus a quoted, whitespace-collapsed, truncated snippet of its source.
Source Files