parse

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 6, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package parse turns raw log bytes into events: it reads bounded lines, assembles multi-line events (stack traces) under the event caps, and decodes each event's format into a structured tree with exact numbers.

Everything produced here is raw, unmasked log content. It is consumed only by internal/source, internal/mask and internal/engine; nothing in this package may be returned to an agent.

Index

Constants

View Source
const (
	// MaxEventBytes is the largest assembled event, in bytes of text.
	MaxEventBytes = 256 << 10
	// MaxEventLines is the largest number of lines in one assembled event.
	MaxEventLines = 500
	// MaxLineBytes is the longest line kept; the rest of a longer line is
	// consumed (it counts towards offsets and budgets) but dropped.
	MaxLineBytes = MaxEventBytes
	// MaxDepth is the deepest JSON nesting decoded as structure.
	MaxDepth = 64
	// MaxKeyBytes is the longest JSON or logfmt key decoded as structure.
	MaxKeyBytes = 256
)

Caps (design §2.3, §4.5). They bound memory and per-event masking cost and are inputs of the engine's worst-case cost model, so they are constants.

View Source
const (
	// MultilineIndent (default) joins a line that starts with a space or a
	// tab, or with "Caused by:" / "Suppressed:", to the free-text event
	// before it (stack traces). Structured events are never joined, and a
	// line that starts with a space or a tab is always free text (so that
	// a reader resumed on it agrees with one that read through).
	MultilineIndent = "indent"
	// MultilineNone makes every line an event.
	MultilineNone = "none"
	// MultilinePattern starts an event at each line matching Start; other
	// lines join the event before them.
	MultilinePattern = "pattern"
)

Multiline modes.

View Source
const MaxPatternBytes = 256

MaxPatternBytes bounds a multiline start pattern (RE2).

Variables

View Source
var FrameKeys = []string{"stream", "attrs"}

FrameKeys are the attributes a framer sets (cri, docker-json). They are labels (Event.Attrs), a namespace apart from the payload's fields, which may hold the same keys: a source of a framed format names them as labels (index.LabelKey, source.Term.Label), never as line fields.

Functions

func Formats

func Formats() []string

Formats returns the registered format names plus "auto", sorted.

func ParseTime

func ParseTime(s string, loc *time.Location) (time.Time, bool)

ParseTime parses a textual timestamp (RFC 3339 and common ISO-8601 forms, or a decimal Unix epoch in s, ms, µs or ns). loc is used for zone-less layouts; nil means UTC.

func Register

func Register(f Format, prio int)

Register adds a format. prio orders it for "auto" (lower first); a negative prio keeps it out of "auto". Registering a name twice panics.

Types

type Event

type Event struct {
	// Format is the format that decoded the event ("json", "logfmt",
	// "plain", ...). When a structured format fails on a line, the event
	// falls back to "plain" and Fallback is set.
	Format   string
	Fallback bool

	// Time is the event's timestamp. HasTime is false when the event
	// carries none; Inherited is set when the reader filled Time from the
	// previous timed event of the same stream.
	Time      time.Time
	HasTime   bool
	Inherited bool

	// Fields is the structured part (top-level members), without the keys
	// that became Time and Message. For free text, only parser-known
	// fields (a plain line's level), else nil.
	Fields []Member
	// TimeKey and MessageKey name the keys Time and Message came from.
	TimeKey    string
	MessageKey string
	// Message is the free-text part, masked in text mode.
	Message string

	// Raw is the assembled text of the event (lines joined by "\n", line
	// terminators removed, invalid UTF-8 replaced). Callers must not
	// modify it.
	Raw string

	// Offset and Length locate the event in the stream: the byte offset of
	// its first line and the number of stream bytes it consumed, including
	// line terminators and any dropped tail of a truncated line.
	Offset int64
	Length int64
	// Lines is the number of lines in the event.
	Lines int
	// Truncated is set when a line was cut at MaxLineBytes.
	Truncated bool
	// Split is set when this event continues a previous one that hit
	// MaxEventBytes or MaxEventLines.
	Split bool

	// Unterminated is set when the event's last line had no terminator
	// (the end of a stream that may still be written).
	Unterminated bool

	// Attrs are the container frame's fields (stream, ...) for framed
	// formats, after the source's static attributes; nil otherwise.
	Attrs []Member
	// Framed is set when the event came out of a container frame (its
	// FrameKeys are in Attrs).
	Framed bool
	// contains filtered or unexported fields
}

Event is one assembled, decoded log event.

func (*Event) SetTime

func (ev *Event) SetTime(t time.Time)

SetTime gives the event an authoritative time from outside its text (a container runtime's frame, a remote backend's entry timestamp). A timestamp the payload carried stays in the payload: a lifted time field goes back to its place in Fields, and a plain line keeps it in Message.

type Format

type Format interface {
	// Name is the configuration name of the format.
	Name() string
	// Detect reports whether a line is in this format; "auto" tries the
	// registered detectable formats in priority order.
	Detect(line string) bool
	// Decode decodes an event's text. It fills Format, Time, Fields,
	// Message and the key names; the reader fills the position. It
	// returns false when the text is not in this format.
	Decode(text string, o *Options) (Event, bool)
}

Format decodes the text of one event.

type Frame

type Frame struct {
	Text    string
	Time    time.Time
	HasTime bool
	Partial bool
	// Attrs are the frame's own fields (stream, ...), trusted metadata of
	// the runtime, kept apart from the payload's fields.
	Attrs []Member
}

Frame is the content of one framed line.

type Framer

type Framer interface {
	Format
	Unwrap(line string, o *Options) (Frame, bool)
}

Framer is implemented by formats that wrap each application line in a container frame (CRI, docker json-file): the frame carries the runtime's timestamp, the stream, and whether the line was split by the runtime ("partial"). The reader joins partial frames into one logical line, then decodes the payload with "auto".

type Kind

type Kind uint8

Kind is the kind of a structured value.

const (
	KindNull Kind = iota
	KindBool
	KindNumber
	KindString
	KindObject
	KindArray
)

Value kinds. Numbers keep their exact literal text.

func (Kind) String

func (k Kind) String() string

type Line

type Line struct {
	// Text is the line without its terminator ("\n" or "\r\n"), cut at
	// MaxLineBytes on a UTF-8 boundary, invalid UTF-8 replaced by U+FFFD.
	Text string
	// Offset is the stream offset of the line's first byte.
	Offset int64
	// Length is the number of stream bytes the line consumed, terminator
	// and dropped tail included.
	Length int64
	// Truncated is set when the line was longer than MaxLineBytes.
	Truncated bool
	// Unterminated is set for a final line without a terminator: on a
	// stream that is still being written, the rest of it may follow.
	Unterminated bool
}

Line is one physical line of a stream.

type LineReader

type LineReader struct {
	// contains filtered or unexported fields
}

LineReader reads bounded lines. Its memory use is bounded by MaxLineBytes plus the bufio buffer, whatever the input.

func NewLineReader

func NewLineReader(r io.Reader, base int64) *LineReader

NewLineReader reads lines from r; base is the stream offset of r's first byte (non-zero when a scan starts inside a file).

func (*LineReader) Next

func (lr *LineReader) Next() (Line, error)

Next returns the next line, or io.EOF when the stream is exhausted. A final line without a terminator is returned as a line.

func (*LineReader) Offset

func (lr *LineReader) Offset() int64

Offset returns the stream offset of the next line Next returns.

func (*LineReader) Unread

func (lr *LineReader) Unread(l Line)

Unread pushes l back: the next call to Next returns it. At most one line can be pushed back, and it must be the line Next returned last.

type Member

type Member struct {
	Key   string
	Value Value
}

Member is one key of an object.

type Options

type Options struct {
	// Format is a registered format name or "auto" (default).
	Format string
	// Multiline is one of the Multiline* modes (default indent).
	Multiline string
	// Start is the RE2 pattern of MultilinePattern.
	Start string
	// Location reads zone-less timestamps; nil means UTC.
	Location *time.Location
	// FrameAttrs are the keys of a docker-json frame's "attrs" object (the
	// log driver's labels and env) kept in the event's "attrs" attribute;
	// the others are dropped, and the attribute too when none is left.
	// Default: none.
	FrameAttrs []string
	// StaticAttrs are attributes the source puts on every event, framed or
	// not, before the frame's own (docker: the trusted container name).
	StaticAttrs []Member
}

Options configure a Parser.

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

Parser is a validated, immutable parsing configuration, safe for concurrent use.

func New

func New(opts Options) (*Parser, error)

New validates opts.

func (*Parser) Line

func (p *Parser) Line(text string) Event

Line decodes one already-delimited record (a Loki entry, a journald MESSAGE): no line splitting, no multiline assembly; invalid UTF-8 is replaced, then the text is cut at MaxEventLines lines and MaxEventBytes bytes, like an assembled event.

func (*Parser) NewReader

func (p *Parser) NewReader(r io.Reader, base int64) *Reader

NewReader reads events from r; base is the stream offset of r's first byte. A scan that starts inside a file must start at a line boundary.

func (*Parser) NewSectionReader

func (p *Parser) NewSectionReader(lr *LineReader, end int64) *Reader

NewSectionReader reads the events of the lines of lr that start before end. The first line starting at or after end is left in lr, so the next section reads on from it. Events never cross a section boundary: a section read alone yields the same events as when it is read after the previous one, which makes fixed byte sections independent plan granules.

func (*Parser) Options

func (p *Parser) Options() Options

Options returns the normalised options.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader assembles and decodes the events of one stream. Memory is bounded by MaxEventBytes plus one line. It is not safe for concurrent use.

Every byte of the stream is accounted for: events are returned in stream order, never overlap, and the bytes between two events (blank lines) are only those the Reader skipped. Offset reports the stream position.

func (*Reader) Next

func (r *Reader) Next() (Event, error)

Next returns the next event, or io.EOF.

func (*Reader) Offset

func (r *Reader) Offset() int64

Offset returns the stream offset up to which events have been returned or skipped: resuming a stream at Offset (with a new Reader) yields the events that follow, except that a Split continuation resumed alone is not flagged Split. When the last event returned ends with an unterminated line (the stream may still be written), Offset is that event's offset.

type Value

type Value struct {
	Kind    Kind
	Str     string
	Members []Member // KindObject
	Items   []Value  // KindArray
}

Value is one node of a structured event. Scalars carry their text in Str: "true"/"false" for bools, the exact literal for numbers (never rounded through float64), the decoded string for strings. Objects keep member order and duplicate keys, so nothing a source wrote is hidden from the masker.

func DecodeJSON

func DecodeJSON(text string) (Value, error)

DecodeJSON decodes text as exactly one JSON value (surrounding whitespace allowed). Numbers keep their literal; object member order and duplicate keys are kept. Nesting deeper than MaxDepth or a key longer than MaxKeyBytes is an error: the caller treats the text as free text, which is masked in text mode.

func Get

func Get(members []Member, key string) (Value, bool)

Get returns the first member named key.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL