Documentation
¶
Overview ¶
Package parse turns raw log bytes into events: it reads bounded lines, assembles multi-line events (stack traces) under the event caps, and decodes each event's format into a structured tree with exact numbers.
Everything produced here is raw, unmasked log content. It is consumed only by internal/source, internal/mask and internal/engine; nothing in this package may be returned to an agent.
Index ¶
Constants ¶
const ( // MaxEventBytes is the largest assembled event, in bytes of text. MaxEventBytes = 256 << 10 // MaxEventLines is the largest number of lines in one assembled event. MaxEventLines = 500 // MaxLineBytes is the longest line kept; the rest of a longer line is // consumed (it counts towards offsets and budgets) but dropped. MaxLineBytes = MaxEventBytes // MaxDepth is the deepest JSON nesting decoded as structure. MaxDepth = 64 // MaxKeyBytes is the longest JSON or logfmt key decoded as structure. MaxKeyBytes = 256 )
Caps (design §2.3, §4.5). They bound memory and per-event masking cost and are inputs of the engine's worst-case cost model, so they are constants.
const ( // MultilineIndent (default) joins a line that starts with a space or a // tab, or with "Caused by:" / "Suppressed:", to the free-text event // before it (stack traces). Structured events are never joined, and a // line that starts with a space or a tab is always free text (so that // a reader resumed on it agrees with one that read through). MultilineIndent = "indent" // MultilineNone makes every line an event. MultilineNone = "none" // MultilinePattern starts an event at each line matching Start; other // lines join the event before them. MultilinePattern = "pattern" )
Multiline modes.
const MaxPatternBytes = 256
MaxPatternBytes bounds a multiline start pattern (RE2).
Variables ¶
var FrameKeys = []string{"stream", "attrs"}
FrameKeys are the attributes a framer sets (cri, docker-json). They are labels (Event.Attrs), a namespace apart from the payload's fields, which may hold the same keys: a source of a framed format names them as labels (index.LabelKey, source.Term.Label), never as line fields.
Functions ¶
func Formats ¶
func Formats() []string
Formats returns the registered format names plus "auto", sorted.
Types ¶
type Event ¶
type Event struct {
// Format is the format that decoded the event ("json", "logfmt",
// "plain", ...). When a structured format fails on a line, the event
// falls back to "plain" and Fallback is set.
Format string
Fallback bool
// Time is the event's timestamp. HasTime is false when the event
// carries none; Inherited is set when the reader filled Time from the
// previous timed event of the same stream.
Time time.Time
HasTime bool
Inherited bool
// Fields is the structured part (top-level members), without the keys
// that became Time and Message. For free text, only parser-known
// fields (a plain line's level), else nil.
Fields []Member
// TimeKey and MessageKey name the keys Time and Message came from.
TimeKey string
MessageKey string
// Message is the free-text part, masked in text mode.
Message string
// Raw is the assembled text of the event (lines joined by "\n", line
// terminators removed, invalid UTF-8 replaced). Callers must not
// modify it.
Raw string
// Offset and Length locate the event in the stream: the byte offset of
// its first line and the number of stream bytes it consumed, including
// line terminators and any dropped tail of a truncated line.
Offset int64
Length int64
// Lines is the number of lines in the event.
Lines int
// Truncated is set when a line was cut at MaxLineBytes.
Truncated bool
// Split is set when this event continues a previous one that hit
// MaxEventBytes or MaxEventLines.
Split bool
// Unterminated is set when the event's last line had no terminator
// (the end of a stream that may still be written).
Unterminated bool
// Attrs are the container frame's fields (stream, ...) for framed
// formats, after the source's static attributes; nil otherwise.
Attrs []Member
// Framed is set when the event came out of a container frame (its
// FrameKeys are in Attrs).
Framed bool
// contains filtered or unexported fields
}
Event is one assembled, decoded log event.
func (*Event) SetTime ¶
SetTime gives the event an authoritative time from outside its text (a container runtime's frame, a remote backend's entry timestamp). A timestamp the payload carried stays in the payload: a lifted time field goes back to its place in Fields, and a plain line keeps it in Message.
type Format ¶
type Format interface {
// Name is the configuration name of the format.
Name() string
// Detect reports whether a line is in this format; "auto" tries the
// registered detectable formats in priority order.
Detect(line string) bool
// Decode decodes an event's text. It fills Format, Time, Fields,
// Message and the key names; the reader fills the position. It
// returns false when the text is not in this format.
Decode(text string, o *Options) (Event, bool)
}
Format decodes the text of one event.
type Frame ¶
type Frame struct {
Text string
Time time.Time
HasTime bool
Partial bool
// Attrs are the frame's own fields (stream, ...), trusted metadata of
// the runtime, kept apart from the payload's fields.
Attrs []Member
}
Frame is the content of one framed line.
type Framer ¶
Framer is implemented by formats that wrap each application line in a container frame (CRI, docker json-file): the frame carries the runtime's timestamp, the stream, and whether the line was split by the runtime ("partial"). The reader joins partial frames into one logical line, then decodes the payload with "auto".
type Kind ¶
type Kind uint8
Kind is the kind of a structured value.
Value kinds. Numbers keep their exact literal text.
type Line ¶
type Line struct {
// Text is the line without its terminator ("\n" or "\r\n"), cut at
// MaxLineBytes on a UTF-8 boundary, invalid UTF-8 replaced by U+FFFD.
Text string
// Offset is the stream offset of the line's first byte.
Offset int64
// Length is the number of stream bytes the line consumed, terminator
// and dropped tail included.
Length int64
// Truncated is set when the line was longer than MaxLineBytes.
Truncated bool
// Unterminated is set for a final line without a terminator: on a
// stream that is still being written, the rest of it may follow.
Unterminated bool
}
Line is one physical line of a stream.
type LineReader ¶
type LineReader struct {
// contains filtered or unexported fields
}
LineReader reads bounded lines. Its memory use is bounded by MaxLineBytes plus the bufio buffer, whatever the input.
func NewLineReader ¶
func NewLineReader(r io.Reader, base int64) *LineReader
NewLineReader reads lines from r; base is the stream offset of r's first byte (non-zero when a scan starts inside a file).
func (*LineReader) Next ¶
func (lr *LineReader) Next() (Line, error)
Next returns the next line, or io.EOF when the stream is exhausted. A final line without a terminator is returned as a line.
func (*LineReader) Offset ¶
func (lr *LineReader) Offset() int64
Offset returns the stream offset of the next line Next returns.
func (*LineReader) Unread ¶
func (lr *LineReader) Unread(l Line)
Unread pushes l back: the next call to Next returns it. At most one line can be pushed back, and it must be the line Next returned last.
type Options ¶
type Options struct {
// Format is a registered format name or "auto" (default).
Format string
// Multiline is one of the Multiline* modes (default indent).
Multiline string
// Start is the RE2 pattern of MultilinePattern.
Start string
// Location reads zone-less timestamps; nil means UTC.
Location *time.Location
// FrameAttrs are the keys of a docker-json frame's "attrs" object (the
// log driver's labels and env) kept in the event's "attrs" attribute;
// the others are dropped, and the attribute too when none is left.
// Default: none.
FrameAttrs []string
// StaticAttrs are attributes the source puts on every event, framed or
// not, before the frame's own (docker: the trusted container name).
StaticAttrs []Member
}
Options configure a Parser.
type Parser ¶
type Parser struct {
// contains filtered or unexported fields
}
Parser is a validated, immutable parsing configuration, safe for concurrent use.
func (*Parser) Line ¶
Line decodes one already-delimited record (a Loki entry, a journald MESSAGE): no line splitting, no multiline assembly; invalid UTF-8 is replaced, then the text is cut at MaxEventLines lines and MaxEventBytes bytes, like an assembled event.
func (*Parser) NewReader ¶
NewReader reads events from r; base is the stream offset of r's first byte. A scan that starts inside a file must start at a line boundary.
func (*Parser) NewSectionReader ¶
func (p *Parser) NewSectionReader(lr *LineReader, end int64) *Reader
NewSectionReader reads the events of the lines of lr that start before end. The first line starting at or after end is left in lr, so the next section reads on from it. Events never cross a section boundary: a section read alone yields the same events as when it is read after the previous one, which makes fixed byte sections independent plan granules.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader assembles and decodes the events of one stream. Memory is bounded by MaxEventBytes plus one line. It is not safe for concurrent use.
Every byte of the stream is accounted for: events are returned in stream order, never overlap, and the bytes between two events (blank lines) are only those the Reader skipped. Offset reports the stream position.
func (*Reader) Offset ¶
Offset returns the stream offset up to which events have been returned or skipped: resuming a stream at Offset (with a new Reader) yields the events that follow, except that a Split continuation resumed alone is not flagged Split. When the last event returned ends with an unterminated line (the stream may still be written), Offset is that event's offset.
type Value ¶
type Value struct {
Kind Kind
Str string
Members []Member // KindObject
Items []Value // KindArray
}
Value is one node of a structured event. Scalars carry their text in Str: "true"/"false" for bools, the exact literal for numbers (never rounded through float64), the decoded string for strings. Objects keep member order and duplicate keys, so nothing a source wrote is hidden from the masker.
func DecodeJSON ¶
DecodeJSON decodes text as exactly one JSON value (surrounding whitespace allowed). Numbers keep their literal; object member order and duplicate keys are kept. Nesting deeper than MaxDepth or a key longer than MaxKeyBytes is an error: the caller treats the text as free text, which is masked in text mode.