README
¶
Buffer 
This package contains several buffer types used in https://github.com/tdewolff/parse for example.
Installation
Run the following command
go get github.com/tdewolff/buffer
or add the following import and run the project with go get
import "github.com/tdewolff/buffer"
Reader
Reader is a wrapper around a []byte that implements the io.Reader interface. It is a much thinner layer than bytes.Buffer provides and is therefore faster.
Writer
Writer is a buffer that implements the io.Writer interface. It is a much thinner layer than bytes.Buffer provides and is therefore faster. It will expand the buffer when needed.
The reset functionality allows for better memory reuse. After calling Reset, it will overwrite the current buffer and thus reduce allocations.
Shifter
Shifter is a read buffer specifically for building lexers. It reads in chunks from an io.Reader and allows to keep track two positions: the start and end position. The start position is the beginning of the current token being parsed, the end position is being moved forward until a valid token is found. Calling Shift will collapse the positions to the end and return the parsed []byte.
Moving the end position can go through Move(int) which also accepts negative integers or MoveTo(int) where the integer will be the new length of the selected bytes. MoveTo(int) is useful when you saved a previous position through Pos() int and want to return to that position.
Peek(int) byte will peek forward (relative to the end position, ie. the position set with Move/MoveTo) and return the byte at that location. PeekRune(int) (rune, int) returns UTF-8 runes and its length at the given byte position. Consecutive calls to Peek may invalidate previously returned byte slices. So if you need to use the content of a byte slice after the next call to Peek(int) byte, it needs to be copied in principal (see exception below).
Bytes() []byte will return the currently selected bytes, Skip() will collapse the selection. Shift() []byte is a combination of Bytes() []byte and Skip().
When the internal io.Reader returned an error, Err() error will return that error (even if subsequent peeks are still possible). If Peek(int) byte returns 0 when an error occurred. IsEOF() bool is a faster alternative than Err() == io.EOF, if it returns true it means the internal buffer will not be reallocated/overwritten. So returned byte slices need not be copied for use after subsequent Peek(int) byte calls. When the io.Reader provides the Bytes() []byte function (which Reader does in this package), it will use that buffer instead and thus IsEOF() returns always true (ie. copying returned slices is not needed).
Lexer
Lexer is an improvement over Shifter in that it does not need the returned byte slices to be copied. Instead you can call ShiftLen() int, which returns the number of bytes that have been shifted since the previous call to ShiftLen, and use that to specify how many bytes need to be freed up from the buffer. Calling Free(n int) frees up n bytes from the internal buffer(s). It holds an array of buffers to accommodate for keeping everything in-memory. If you don't need to keep returned byte slices around, call Free(ShiftLen()) after every Shift call.
The MoveTo(int) function has been renamed to Rewind(int) to fit its meaning better. Also Bytes() []byte has been renamed to Lexeme() []byte for the same reason.
License
Released under the MIT license.
Documentation
¶
Overview ¶
Package buffer contains buffer and wrapper types for byte slices. It is useful for writing lexers or other high-performance byte slice handling.
The `Reader` and `Writer` types implement the `io.Reader` and `io.Writer` respectively and provide a thinner and faster interface than `bytes.Buffer`. The `Shifter` type is useful for building lexers because it keeps track of the start and end position of a byte selection, and shifts the bytes whenever a valid token is found. The `Lexer` is however an improved version of `Shifter`, allowing zero-copy for the parser by using a (kind of) ring buffer underneath.
Index ¶
- Variables
- type Lexer
- func (z *Lexer) Err() error
- func (z *Lexer) Free(n int)
- func (z *Lexer) Lexeme() []byte
- func (z *Lexer) Move(n int)
- func (z *Lexer) Peek(pos int) byte
- func (z *Lexer) PeekRune(pos int) (rune, int)
- func (z *Lexer) Pos() int
- func (z *Lexer) Rewind(pos int)
- func (z *Lexer) Shift() []byte
- func (z *Lexer) ShiftLen() int
- func (z *Lexer) Skip()
- type Reader
- type Shifter
- func (z *Shifter) Bytes() []byte
- func (z *Shifter) Err() error
- func (z *Shifter) IsEOF() bool
- func (z *Shifter) Move(n int)
- func (z *Shifter) MoveTo(n int)
- func (z *Shifter) Peek(end int) byte
- func (z *Shifter) PeekRune(i int) (rune, int)
- func (z *Shifter) Pos() int
- func (z *Shifter) Shift() []byte
- func (z *Shifter) Skip()
- type Writer
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var MinBuf = defaultBufSize
MinBuf specifies the default initial length of internal buffers. Solely here to support old versions of parse.
Functions ¶
This section is empty.
Types ¶
type Lexer ¶
type Lexer struct {
// contains filtered or unexported fields
}
Lexer is a buffered reader that allows peeking forward and shifting, taking an io.Reader. It keeps data in-memory until Free, taking a byte length, is called to move beyond the data.
func NewLexer ¶
NewLexer returns a new Lexer for a given io.Reader with a 4kB estimated buffer size. If the io.Reader implements Bytes, that buffer is used instead.
func NewLexerSize ¶
NewLexerSize returns a new Lexer for a given io.Reader and estimated required buffer size. If the io.Reader implements Bytes, that buffer is used instead.
func (*Lexer) Err ¶
Err returns the error returned from io.Reader. It may still return valid bytes for a while though.
func (*Lexer) Free ¶
Free frees up bytes of length n from previously shifted tokens. Each call to Shift should at one point be followed by a call to Free with a length returned by ShiftLen.
func (*Lexer) Peek ¶
Peek returns the ith byte relative to the end position and possibly does an allocation. Peek returns zero when an error has occurred, Err returns the error. TODO: inline function
func (*Lexer) PeekRune ¶
PeekRune returns the rune and rune length of the ith byte relative to the end position.
func (*Lexer) Shift ¶
Shift returns the bytes of the current selection and collapses the position to the end of the selection. It also returns the number of bytes we moved since the last call to Shift. This can be used in calls to Free.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader implements an io.Reader over a byte slice.
func NewReader ¶
NewReader returns a new Reader for a given byte slice.
Example ¶
r := NewReader([]byte("Lorem ipsum"))
w := &bytes.Buffer{}
io.Copy(w, r)
fmt.Println(w.String())
Output: Lorem ipsum
type Shifter ¶
type Shifter struct {
// contains filtered or unexported fields
}
Shifter is a buffered reader that allows peeking forward and shifting, taking an io.Reader.
func NewShifter ¶
NewShifter returns a new Shifter for a given io.Reader with a 4kB estimated buffer size. If the io.Reader implements Bytes, that buffer is used instead.
Example ¶
b := bytes.NewBufferString("Lorem ipsum")
z := NewShifter(b)
for {
c := z.Peek(0)
if c == ' ' {
break
}
z.Move(1)
}
fmt.Println(string(z.Shift()))
Output: Lorem
func NewShifterSize ¶
NewShifterSize returns a new Shifter for a given io.Reader and estimated required buffer size. If the io.Reader implements Bytes, that buffer is used instead.
func (*Shifter) Err ¶
Err returns the error returned from io.Reader. It may still return valid bytes for a while though.
func (*Shifter) IsEOF ¶
IsEOF returns true when it has encountered EOF meaning that it has loaded the last data in memory (ie. previously returned byte slice will not be overwritten by Peek). Calling IsEOF is faster than checking Err() == io.EOF.
Example ¶
b := bytes.NewBufferString("Lorem ipsum") // bytes.Buffer provides a Bytes function, NewShifter uses that and r.IsEOF() always returns true
z := NewShifter(b)
z.Move(5)
lorem := z.Shift()
if !z.IsEOF() { // required when io.Reader doesn't provide a Bytes function
buf := make([]byte, len(lorem))
copy(buf, lorem)
lorem = buf
}
z.Peek(0) // might reallocate the internal buffer
fmt.Println(string(lorem))
Output: Lorem
func (*Shifter) Peek ¶
Peek returns the ith byte relative to the end position and possibly does an allocation. Calling Peek may invalidate previous returned byte slices by Bytes or Shift, unless IsEOF returns true. Peek returns zero when an error has occurred, Err returns the error.
func (*Shifter) PeekRune ¶
PeekRune returns the rune and rune length of the ith byte relative to the end position.
Example ¶
b := bytes.NewBufferString("† dagger") // † has a byte length of 3
z := NewShifter(b)
c, n := z.PeekRune(0)
fmt.Println(string(c), n)
Output: † 3
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer implements an io.Writer over a byte slice.
func NewWriter ¶
NewWriter returns a new Writer for a given byte slice.
Example ¶
w := NewWriter(make([]byte, 0, 11)) // initial buffer length is 11
w.Write([]byte("Lorem ipsum"))
fmt.Println(string(w.Bytes()))
Output: Lorem ipsum
func (*Writer) Reset ¶
func (w *Writer) Reset()
Reset empties and reuses the current buffer. Subsequent writes will overwrite the buffer, so any reference to the underlying slice is invalidated after this call.
Example ¶
w := NewWriter(make([]byte, 0, 11)) // initial buffer length is 10
w.Write([]byte("garbage that will be overwritten")) // does reallocation
w.Reset()
w.Write([]byte("Lorem ipsum"))
fmt.Println(string(w.Bytes()))
Output: Lorem ipsum
Source Files