Documentation
¶
Index ¶
- Constants
- func NewReader() lisp.Reader
- type Interactive
- type ParseResult
- type Parser
- func (p *Parser) Accept(typ ...token.Type) bool
- func (p *Parser) Float(x float64) *lisp.LVal
- func (p *Parser) Int(x int) *lisp.LVal
- func (p *Parser) Location() *token.Location
- func (p *Parser) Parse() (*lisp.LVal, error)
- func (p *Parser) ParseConsExpression() *lisp.LVal
- func (p *Parser) ParseExpression() *lisp.LVal
- func (p *Parser) ParseFunRef() *lisp.LVal
- func (p *Parser) ParseList() *lisp.LVal
- func (p *Parser) ParseLiteralFloat() *lisp.LVal
- func (p *Parser) ParseLiteralInt() *lisp.LVal
- func (p *Parser) ParseLiteralIntHex() *lisp.LVal
- func (p *Parser) ParseLiteralIntOctal() *lisp.LVal
- func (p *Parser) ParseLiteralString() *lisp.LVal
- func (p *Parser) ParseLiteralStringRaw() *lisp.LVal
- func (p *Parser) ParseNegative() *lisp.LVal
- func (p *Parser) ParseProgram() ([]*lisp.LVal, error)
- func (p *Parser) ParseProgramFaultTolerant() ParseResult
- func (p *Parser) ParseQuote() *lisp.LVal
- func (p *Parser) ParseSymbol() *lisp.LVal
- func (p *Parser) ParseUnbound() *lisp.LVal
- func (p *Parser) PeekLocation() *token.Location
- func (p *Parser) PeekType() token.Type
- func (p *Parser) PendingComments() []*token.Token
- func (p *Parser) QExpr(cells []*lisp.LVal) *lisp.LVal
- func (p *Parser) Quote(v *lisp.LVal) *lisp.LVal
- func (p *Parser) ReadToken() *token.Token
- func (p *Parser) SExpr(cells []*lisp.LVal) *lisp.LVal
- func (p *Parser) String(s string) *lisp.LVal
- func (p *Parser) Symbol(sym string) *lisp.LVal
- func (p *Parser) TokenText() string
- func (p *Parser) TokenType() token.Type
- type TokenGenerator
- type TokenSource
- type TokenStream
Constants ¶
const DefaultMaxParseDepth = 10000
DefaultMaxParseDepth is the maximum nesting depth allowed by the parser. Deeply nested input (e.g. 50K+ unmatched parens) can overflow the Go goroutine stack with a fatal crash that recover() cannot catch. This limit converts the fatal crash into a parse error.
It bounds the PARSER's recursion only. It used to be, incidentally, the only thing bounding the EVALUATOR's recursion as well -- nesting had to be parsed before it could be evaluated -- but that composition was never sound: a recursive macro generates nesting at expansion time and never passes through here, as does any embedder building an AST through the Go API. lisp.DefaultMaxEvalNesting is the limit that bounds evaluation depth (issue #316); this one is not load-bearing for it.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Interactive ¶
type Interactive struct {
Read TokenGenerator
// contains filtered or unexported fields
}
Interactive implements a parser that parses a single expression at a time and defers to a TokenGenerator function when it is necessary to read more tokens.
func NewInteractive ¶
func NewInteractive(read TokenGenerator) *Interactive
NewInteractive initializes and returns a new Interactive parser.
func (*Interactive) IsParsing ¶
func (p *Interactive) IsParsing() bool
IsParsing returns true if p is in the middle of parsing an expression. IsParsing can be called at any time, potentially by concurrent goroutines or when p is nil.
func (*Interactive) Parse ¶
func (p *Interactive) Parse() (*lisp.LVal, error)
Parse parses one expression from the interactive token stream and returns it, or any error encountered. A REPL would call this function in its main runloop. If a parse error is encountered, any buffered tokens (presumably from the current tty line) are discarded so corrected source can be re-read.
func (*Interactive) ParseExpression ¶
func (p *Interactive) ParseExpression() (*lisp.LVal, error)
ParseExpression parses one expression from the interactive token stream and returns it, or any error encountered.
NOTE: ParseExpression is deprecated and Parse should be used instead.
func (*Interactive) Prompt ¶
func (p *Interactive) Prompt() string
Prompt returns a simple prompt that can be used by a REPL token generator.
func (*Interactive) SetPrompts ¶
func (p *Interactive) SetPrompts(prompt, cont string)
SetPrompts configures the string prompts reutrned by p.Prompt(). The cont string is used to prompt the user when the parser is in the middle of parsing an expression at the start of a line.
type ParseResult ¶ added in v1.26.0
ParseResult holds the result of a fault-tolerant parse: a partial AST and any errors encountered during parsing.
type Parser ¶
type Parser struct {
// contains filtered or unexported fields
}
Parser is a lisp parser.
func NewFormatting ¶ added in v1.16.10
NewFormatting initializes a Parser in format-preserving mode. Comments, bracket types, original literal text, and blank lines are attached as formatting metadata (internal/fmtmeta) on the returned AST.
func NewFromSource ¶
func NewFromSource(src *TokenSource) *Parser
NewFromSource initializes and returns a Parser that reads tokens from src.
func (*Parser) Location ¶
Location returns the position of the token under the cursor, as a Location the caller OWNS. No two callers are ever given the same one.
THE DEFECT (elps#426). This used to return p.src.Token.Source -- the scanner's own object -- so every caller asking about one token took joint ownership of it with every other, and the parser has callers that WRITE through what Location returns. tokenLVal stores it on the LVal it is building and then writes EndLine/EndCol/EndPos into it; a prefix form builds TWO LVals from one token, because the operand is parsed first and nothing is consumed after it, so the operand's token is still under the cursor when the enclosing s-expression is built. Both nodes held that one object, and applyPrefixLocation then rewrote it to the prefix's column -- moving the operand's reported position along with the form's. "#'car" reported the symbol "car" as spanning "#'car"; "#^a" gave all three of its nodes one position. Longhand "(quote x)" consumes a ")" after its operand, so its form is built from a different token, and it reported "x" at "x" all along: #426 is the two spellings disagreeing about where the operand is.
THE MODEL. A Location is MOVED, not shared. Scanner.LocStart allocates one per token; the parser hands that object to the FIRST caller to ask about the token, which is the node the token produces, and gives every later caller an independent copy. The scanner's per-token allocation therefore becomes the AST node's, rather than being duplicated by it.
Ownership is tracked by comparing against the last object given away, which is sufficient because the parser never returns to an earlier token: a pointer equal to locGivenAway is necessarily the current token's, asked for a second time. A token whose Location has been given away is not thereby corrupted -- the new owner writes only the End fields (tokenLVal) or is a private copy (every prefix form's enclosing node, which is by construction a second ask), and the start fields the parser reads back off a token, in the "unclosed %s opened at %s" errors, are never written.
Copying HERE rather than only in tokenLVal is the choice among the three #426 lists. It is the only one that also covers the next caller: errorf and scanError put the result on an error value that outlives the parse, and locateSynthesized writes end positions into it.
The alternative -- copying unconditionally -- was implemented and measured first. It costs one extra Location per parsed node: +7.4% to +8.5% allocs/op and +8.7% to +16.1% B/op across the six BenchmarkParser corpora, deterministic (all rows p=0.000, +-0%). That is a real cost on the parse path for no additional safety, since a Location given away exactly once cannot be shared.
A nil result means "no position": either no token has been scanned yet (see the token-under-the-cursor note above, elps#430) or the token carries no Location. Callers store it on an LVal or an error, where nil Source already means the same thing.
func (*Parser) Parse ¶
Parse is a generic entry point that is similar to ParseExpression but is capable of handling EOF before reading an expression.
func (*Parser) ParseConsExpression ¶
func (*Parser) ParseExpression ¶
ParseExpression parses a single expression. Unlike Parse, ParseExpression requires an expression to be present in the input stream and will report unexpected EOF tokens encountered.
func (*Parser) ParseFunRef ¶ added in v1.2.0
func (*Parser) ParseLiteralFloat ¶
func (*Parser) ParseLiteralInt ¶
func (*Parser) ParseLiteralIntHex ¶
func (*Parser) ParseLiteralIntOctal ¶
func (*Parser) ParseLiteralString ¶
func (*Parser) ParseLiteralStringRaw ¶
func (*Parser) ParseNegative ¶
func (*Parser) ParseProgram ¶
ParseProgram parses a series of expressions potentially preceded by a hash-bang, `#!`.
func (*Parser) ParseProgramFaultTolerant ¶ added in v1.26.0
func (p *Parser) ParseProgramFaultTolerant() ParseResult
ParseProgramFaultTolerant parses a series of expressions like ParseProgram, but recovers from errors and continues parsing. It returns all successfully parsed expressions alongside all collected errors. This is useful for IDE/LSP scenarios where partial ASTs are valuable.
func (*Parser) ParseQuote ¶
func (*Parser) ParseSymbol ¶
func (*Parser) ParseUnbound ¶
func (*Parser) PeekLocation ¶
PeekLocation returns the position of the NEXT token, as a Location the caller owns.
It copies unconditionally rather than taking ownership the way Location does (elps#426). Taking it would give away the Location of a token the parser has not reached yet, so the node eventually built from that token would be the one paying for a copy -- charging the parse path for a call it did not make. PeekLocation has no caller in this repository and is exported, which is the whole reason it is closed here: an embedder's peek must not be able to move a position the parser later reports.
func (*Parser) PendingComments ¶ added in v1.16.10
PendingComments returns any comments collected but not yet attached to an LVal (e.g., trailing comments at end of file). Only useful in formatting mode.
type TokenGenerator ¶
TokenGenerator implements TokenStream. The function will be called any time a TokenSource wants a token.
func (TokenGenerator) ReadToken ¶
func (fn TokenGenerator) ReadToken() []*token.Token
ReadToken implements TokenStream.
type TokenSource ¶
TokenSource abstracts a TokenStream by adding "memory" and providing methods to process and branch off the stream's tokens.
func NewTokenSource ¶
func NewTokenSource(scanner *token.Scanner) *TokenSource
TokenSource initializes and returns a new token.Source that scans tokens from scanner.
func NewTokenStreamSource ¶
func NewTokenStreamSource(stream TokenStream) *TokenSource
func (*TokenSource) AcceptType ¶
func (s *TokenSource) AcceptType(typ ...token.Type) bool
func (*TokenSource) IsEOF ¶
func (s *TokenSource) IsEOF() bool
func (*TokenSource) Peek ¶
func (s *TokenSource) Peek() *token.Token
func (*TokenSource) Scan ¶
func (s *TokenSource) Scan() bool
type TokenStream ¶
type TokenStream interface {
// ReadToken returns a set of token from an input source. When no more
// tokens can be generated ReadToken returns a token with type token.EOF.
// ReadToken never returns an empty slice. In the presence of io errors a
// TokenStream must return a token with type token.ERROR whenever called.
ReadToken() []*token.Token
}
TokenStream is an arbitrary sequence of tokens. Typically, a TokenStream will be a *lexer.Lexer but other implementations may be desirable for implementation a REPL or other dynamic environments.
func TokenChannel ¶
func TokenChannel(c <-chan []*token.Token) TokenStream
TokenChannel returns a TokenStream that returns tokens receieved from c.
Source Files