Documentation
¶
Overview ¶
Package consoleline splits the output of a Firehose instrumented node into lines as it is read. "FIRE BLOCK" lines can have their base64 payload decoded while they are read, so the encoded text of a block is never held in memory as a whole.
Index ¶
Constants ¶
const ( // MaxLineLength is the largest line in bytes the Firehose stack can handle. Its base64 // payload decodes to at most 3/4 of it, which keeps the block and 1 MiB for the message // around it within the 2 GiB responses of dgrpc.MaxResponseSize. MaxLineLength = (2*1024*1024*1024 - 1024*1024) / 3 * 4 // DefaultBufferSize is the normal size in bytes of a Splitter buffer when none is given. DefaultBufferSize = 100 * 1024 * 1024 // MinBufferSize is the smallest normal size in bytes of a Splitter buffer. MinBufferSize = 32 * 1024 )
Variables ¶
var ErrLineTooLong = errors.New("line too long")
ErrLineTooLong is returned by Splitter.Write when a line is longer than the maximum line length of the Splitter.
Functions ¶
This section is empty.
Types ¶
type Block ¶
type Block struct {
// Header is the text between "FIRE BLOCK " and the payload, without the space
// separating it from the payload.
Header string
// Payload is the decoded base64 payload, it is nil when Err is set.
Payload []byte
// Err is set when the payload is not valid base64.
Err error
// LineLength is the length in bytes of the line as it was read, newline excluded.
LineLength int
}
Block is a "FIRE BLOCK" line whose payload was decoded while it was read.
type Splitter ¶
type Splitter struct {
// contains filtered or unexported fields
}
Splitter is an io.Writer that splits what is written to it into lines, stripping the trailing "\n" or "\r\n", and hands each line to onLine.
When onBlock is set, a "FIRE BLOCK" line is handed to onBlock instead, with its payload decoded while it is written. Finding the payload requires the number of header fields, which depends on the protocol version announced by the "FIRE INIT" line. Until that line is seen, "FIRE BLOCK" lines are handed to onLine as text.
A Splitter is not safe for concurrent use.
func NewSplitter ¶
NewSplitter returns a Splitter handing lines to onLine and, when onBlock is non-nil, decoded "FIRE BLOCK" lines to onBlock. Its buffer has a normal size of bufferSize bytes (DefaultBufferSize when 0, at least MinBufferSize), grows past it for longer lines and shrinks by half, down to it, each time enough blocks in a row used less than half of it. Write fails with ErrLineTooLong on a line longer than MaxLineLength bytes.
func (*Splitter) Close ¶
Close hands the last line to its handler when it did not end with a newline.
Source Files