Documentation
¶
Index ¶
- Constants
- func TokenEnd(tok *Token) (endLine, endCol, endPos int)
- type Location
- type LocationError
- type Rune
- type Scanner
- func (s *Scanner) Accept(fn func(rune) bool) bool
- func (s *Scanner) AcceptAny(charset string) bool
- func (s *Scanner) AcceptDigit() bool
- func (s *Scanner) AcceptRune(c rune) bool
- func (s *Scanner) AcceptSeq(fn func(rune) bool) int
- func (s *Scanner) AcceptSeqAny(charset string) int
- func (s *Scanner) AcceptSeqDigit() int
- func (s *Scanner) AcceptSeqRune(c rune) int
- func (s *Scanner) AcceptSeqSpace() int
- func (s *Scanner) AcceptSpace() bool
- func (s *Scanner) AcceptString(literal string) (int, bool)
- func (s *Scanner) EOF() bool
- func (s *Scanner) EmitToken(typ Type) *Token
- func (s *Scanner) Err() error
- func (s *Scanner) Ignore()
- func (s *Scanner) Loc() *Location
- func (s *Scanner) LocStart() *Location
- func (s *Scanner) Peek() (rune, bool)
- func (s *Scanner) Rune() rune
- func (s *Scanner) ScanRune() error
- func (s *Scanner) SetPath(path string)
- func (s *Scanner) Text() string
- type Source
- type Token
- type Type
Constants ¶
const DefaultBufSize = 128 << 10
DefaultBufSize is the size of the sliding window NewScanner allocates. It is also the largest single token NewScanner can scan: the window never grows, so a token that fills it fails with "token exceeds maximum allowable size".
const NativeFile = "<native code>"
NativeFile is the File reported by a Location that does not come from a source stream -- a value constructed by Go code rather than read from a file. Paired with Pos == -1, which is what Location.String and the "is this a real position?" checks throughout the tree test for.
Variables ¶
This section is empty.
Functions ¶
func TokenEnd ¶ added in v1.17.0
TokenEnd computes the end position of a token from its start position and text. All three results are in the units Location documents: endCol is an exclusive BYTE column, endPos an absolute BYTE offset. For a multi-line token (a raw string) the line and column are adjusted accordingly.
THE DEFECT (elps#463). This used to advance col by ONE PER RUNE:
for _, ch := range tok.Text {
if ch == '\n' { line++; col = 1 } else { col++ }
}
`range` over a string iterates runes, so that produced Col + runeCount -- a rune width added to the byte base Scanner.LocStart computes as `startPos - s.startLinePos + 1`. On a pure-ASCII token the two units coincide and it was right by accident; on a token holding any multi-byte rune it was short by len(text)-runeCount(text) and was in neither unit, while endPos beside it was already byte-exact. A field that is correct for most inputs and quietly wrong for the rest is the shape of bug that reaches users, and this one reached them through textDocument/rename: the edit range was narrower than the identifier, so the rename replaced a prefix and left the tail, turning "éx" into "zzx" rather than "zz" with no diagnostic. TestRenameNonASCIIIdentifierRewritesWholeName is the end-to-end pin.
Counting BYTES rather than runes here is the whole fix, and it is a fix under either answer to elps#464 (whether the SERVER should be emitting UTF-16 code units on the wire): the invariant this restores is that endCol agrees with the Col it is measured from, which any wire encoding needs before it can convert anything.
The loop scans bytes rather than runes deliberately. A byte scan is exact on invalid UTF-8, where `range` yields RuneError with a width that does not describe the input; '\n' cannot occur as a UTF-8 continuation byte, so looking for it bytewise finds exactly the newlines a rune scan would.
Types ¶
type Location ¶
type Location struct {
File string // a name representing the source stream
Path string // a physical location which may differ from File
Pos int // BYTE offset of the first byte of the token/expr
Line int // line number (starting at 1 when tracked)
Col int // BYTE column within the line (1-based; 0 = not tracked)
EndPos int // BYTE offset one past the last byte (0 = not tracked)
EndLine int // end line (1-based, 0 = not tracked)
EndCol int // BYTE column one past the last byte (1-based, exclusive; 0 = not tracked)
}
Location is a span in a source stream.
UNITS. Every offset and column in this struct is counted in BYTES, and says so on its own line below. The unit is not a detail: Col and EndCol are subtracted from and added to each other by consumers all over the tree (analysis.scopeContainingAnalysis, lsp.locContainsCol, lsp.elpsToLSPRange, lint.endPosFromNode), and a value in one unit compared against a value in another is wrong without being obviously wrong. Whatever unit is chosen, the four position fields have to agree on it.
They did not. TokenEnd derived EndCol by counting RUNES onto Scanner's byte-valued Col, so on any token containing a multi-byte rune EndCol was short by len(text)-utf8.RuneCountInString(text) and was in neither unit. Every LSP range built from EndCol was correspondingly short, and textDocument/rename -- which builds its TextEdit ranges from the same helper -- therefore replaced fewer bytes than the name occupied and left the tail behind: renaming "éx" to "zz" produced "zzx", a different program, silently (elps#463). The absence of a stated unit is what allowed that, so the unit is stated here.
BYTES is an INTERNAL convention, not what LSP asks for: LSP 3.16 counts a position in UTF-16 code units unless client and server negotiate otherwise, and this server neither offers positionEncoding nor converts anything. That server-wide gap is elps#464 and is deliberately not what this comment settles. #464 is about which unit crosses the wire; the requirement here is only that these four fields agree with EACH OTHER, which they must under any choice #464 goes on to make.
func NativeLocation ¶ added in v1.50.0
func NativeLocation() Location
NativeLocation returns the Location describing code that has no source stream: values constructed by Go rather than read by the parser.
It returns a VALUE, deliberately, and it is the only definition of that location in the tree. A function handing out a *Location here would hand every caller a pointer into shared state that any one of them could write through, which is issue #362 -- and lisp.nativeSource is exactly that function. Callers that need a *Location for a node they own take the address of their own copy:
loc := token.NativeLocation() v.Source = &loc
func (*Location) Copy ¶ added in v1.50.0
Copy returns a pointer to an independent copy of loc, or nil if loc is nil.
Location holds no reference-typed fields, so the shallow copy is fully independent: a later write through either pointer is invisible to the other. Nil is preserved rather than materialised into a zero Location because a nil Source is meaningful throughout the tree ("no position recorded") and distinct from a zero one ("position 0").
Use it at every point where a Location owned by one object is stored into another that outlives, or is mutated independently of, the first -- see issues #362 and #366.
type LocationError ¶
type LocationError struct {
Err error
Source *Location
Code string // error classification (empty = unclassified)
}
func (*LocationError) Error ¶
func (err *LocationError) Error() string
func (*LocationError) Unwrap ¶ added in v1.17.0
func (err *LocationError) Unwrap() error
Unwrap returns the underlying error for use with errors.Is/errors.As.
type Rune ¶
Rune contains a rune that read by Scanner during peeking operations.
func (Rune) IsRuneError ¶
IsRuneError returns true if Rune represents an invalid utf-8 sequence read by utf8.DecodeRune.
type Scanner ¶
type Scanner struct {
// contains filtered or unexported fields
}
Scanner facilitates construction of tokens from a byte stream (io.Reader).
func NewScanner ¶
NewScanner initializes and returns a new Scanner reading through a DefaultBufSize sliding window.
func NewScannerString ¶ added in v1.49.0
NewScannerString initializes and returns a new Scanner reading src, sizing the sliding window to src rather than allocating the DefaultBufSize window NewScanner uses. src is already in memory, so the window costs nothing extra and no token can overrun it.
This exists because the fixed window is charged per SCANNER, not per byte scanned, and the parser re-reads short strings: readsBackAsSymbol scans each #' operand and each half of every package-qualified symbol to check that it reads back as a symbol (issue #319). Through NewScanner that is 128KiB per check -- 2.6GB and 1.7s to parse 10k qualified symbols, which is both a pointless cost on ordinary source and an allocation amplification an attacker controls, in a parser whose job is to survive untrusted phylum source.
func (*Scanner) AcceptDigit ¶
func (*Scanner) AcceptRune ¶
func (*Scanner) AcceptSeqAny ¶
func (*Scanner) AcceptSeqDigit ¶
func (*Scanner) AcceptSeqRune ¶
func (*Scanner) AcceptSeqSpace ¶
func (*Scanner) AcceptSpace ¶
func (*Scanner) EmitToken ¶
EmitToken returns a token containing the text scanned since the last call to either EmitToken or Ignore.
func (*Scanner) Err ¶
Err returns an error encountered during the last read on the input stream. Err will always return false while there are still buffered runes that need to be accepted.
func (*Scanner) Ignore ¶
func (s *Scanner) Ignore()
Ignore causes the scanner to skip all text scanned since the last call to either EmitToken or Ignore.
func (*Scanner) Loc ¶
Loc returns a Location referencing the current scanner position, the last position of the current token.
func (*Scanner) LocStart ¶
LocStart returns a Location referencing the beginning of the current token, just beyond the end of the previous token.
func (*Scanner) Peek ¶
Peek returns the next rune to be scanned, if there are any. If an invalid utf-8 sequence or EOF prevents futher runes from being scanned Peek returns a false second value. If Peek returns a false value the next call to s.ScanRune will return an error that reflects of the cause.
func (*Scanner) Rune ¶
Rune returns the current unicode rune that is being scanned. The rune returned by Rune is the last rune in a token returned by EmitToken.
func (*Scanner) ScanRune ¶
ScanRune attempts to scan a utf-8 rune from the input for inclusion in the current token. If an error prevents a valid unicode rune from being scanned then an error will be returned.
type Source ¶
type Source interface {
// Token returns the current token. Token returns nil if Scan has not been
// called.
Token() *Token
// Peek returns the next token in the stream. At the end of the stream
// Peek should return a value to indicate the lack of a token (EOF).
Peek() *Token
// Scan advances the token stream if possible. If there are no tokens
// remaining Scan returns false.
Scan() bool
}
Source is an abstract stream of tokens which allows one token lookahead.
type Token ¶
type Token struct {
Source *Location
Text string
Type Type
PrecedingNewlines int // newlines in whitespace before this token
PrecedingSpaces int // spaces in whitespace before this token (same-line only)
}
func (*Token) Copy ¶ added in v1.50.0
Copy returns a pointer to an independent copy of tok, or nil if tok is nil.
Source is copied rather than carried across, for the reason Location.Copy documents: a Location stored into an object that outlives -- or is mutated independently of -- the object it came from is issue #362's shape, and a copied Token is by definition a second owner. Text is a string and needs no copy.
Nil is preserved because a nil *Token is meaningful where these are held: SourceMeta.TrailingComment nil means "no inline comment on this node", not "an empty comment".
type Type ¶
type Type uint
const ( INVALID Type = iota ERROR EOF HASH_BANG // Atomic expressions & literals SYMBOL INT INT_OCTAL_MACRO INT_OCTAL INT_HEX_MACRO INT_HEX FLOAT STRING STRING_RAW COMMENT // Operators NEGATIVE // arithmetic negation is parsed specially QUOTE UNBOUND FUN_REF // Delimiters PAREN_L PAREN_R BRACE_L BRACE_R )
Type constants used for the elps lexer/parser. These constants aren't necessary to use the package.
Source Files