Documentation
¶
Overview ¶
Package file is the file log source: globs of plain, gzip and zstd files under the console's read-only mount roots, read beneath their root without following symbolic links, planned in fixed granules, and indexed locally.
Index ¶
- Constants
- Variables
- type CountQuery
- type CountResult
- type CountRow
- type Options
- type Source
- func (s *Source) Around(ctx context.Context, ref source.RawRef, before, after int) ([]source.RawRecord, error)
- func (s *Source) AroundMeasured(ctx context.Context, ref source.RawRef, before, after int) ([]source.RawRecord, int64, error)
- func (s *Source) Describe(ctx context.Context) (source.Schema, error)
- func (s *Source) IndexCounts(ctx context.Context, q CountQuery) (CountResult, error)
- func (s *Source) IndexStep(ctx context.Context, maxBytes int64, upkeep bool) (int64, bool, error)
- func (s *Source) Kind() string
- func (s *Source) Name() string
- func (s *Source) NumericTerms()
- func (s *Source) Plan(ctx context.Context, r source.Range, sel source.Selector) (source.Byteplan, error)
- func (s *Source) Reread(ctx context.Context, ref source.RawRef) (source.RawRecord, error)
- func (s *Source) Scan(ctx context.Context, c source.Chunk) (source.Iterator, error)
Constants ¶
const ( // MaxRecursion is the deepest "**" expansion (directories below the // point where "**" appears). MaxRecursion = 4 // MaxFiles bounds the files one glob may match. MaxFiles = 10000 )
const ( // PrefixBytes is the length of the prefix hashed into a file's // identity. PrefixBytes = 64 << 10 // MaxRatio bounds decompressed bytes per compressed byte. MaxRatio = 100 )
const ClaimMargin = time.Minute
ClaimMargin is the timestamp disorder the coverage claim allows for. The claim also assumes no record is stamped later than its file's mtime plus five minutes (zone-less local times read as UTC on a host ahead of UTC, or a clock running ahead, break that assumption: accepted limitation).
indexed_from is that much after (and indexed_until that much before) the nearest record the index may lack, rounded to whole minutes.
const Kind = "file"
Kind is the source kind.
const MaxAround = 50
MaxAround bounds before and after of Around.
Variables ¶
var ErrNotIndexable = source.ErrNotIndexable
ErrNotIndexable, CountQuery, CountRow and CountResult: see package source.
Functions ¶
This section is empty.
Types ¶
type CountQuery ¶
type CountQuery = source.CountQuery
type CountResult ¶
type CountResult = source.CountResult
type Options ¶
type Options struct {
Name string
Paths []string // absolute globs
Recursive bool
// Roots are the read-only mount roots; every path must be under one.
Roots []string
Parser *parse.Parser
// Trusted are the trusted low-cardinality line fields the index
// counts, and labels as index.LabelKey (a framer's attributes: the
// docker source's container and stream).
Trusted []string
// Index is the local index store (nil: no index).
Index *index.Store
// Sensitive reports a trusted value the masker would change as a
// trusted leaf (mask.Masker.TrustedValue): it is indexed as
// index.Redacted, never held clear.
Sensitive func(field, value string) bool
// RulesGeneration changes when the rules behind Sensitive change
// (mask.Masker.RulesGeneration); nil: they never do.
RulesGeneration func() uint64
// Now is the clock (tests).
Now func() time.Time
// NoXDev refuses to open a file across a mount point under its root
// (RESOLVE_NO_XDEV; the fallback walk compares mount ids, else
// devices): the docker source, whose root may receive propagated
// mounts after the console started. On a kernel without openat2 and
// before 5.8 (no statx mount id), only the device is compared: a bind
// mount of the same filesystem is not detected there.
NoXDev bool
// Tag is stored with each stream's index and checked like the trusted
// fields: an index built under another tag is rebuilt (the docker
// source's container name, which every record carries).
Tag string
// Alternates maps a literal path to the path read instead while the
// first is absent (docker: <id>-json.log.1, then .1.gz once Docker
// compressed it): never both.
Alternates map[string]string
// Guard pins the roots (nil: the source's own). A root gone or
// replaced since it was first read fails every call with
// source.ErrMountLost. The docker source shares one across its
// containers' sources.
Guard *source.RootGuard
}
Options configure a file source.
type Source ¶
type Source struct {
// contains filtered or unexported fields
}
Source is a file source. It is safe for concurrent use.
func (*Source) Around ¶
func (s *Source) Around(ctx context.Context, ref source.RawRef, before, after int) ([]source.RawRecord, error)
Around returns up to before records before ref, ref's record and up to after records after it, reading back at most 16 granules.
func (*Source) AroundMeasured ¶
func (s *Source) AroundMeasured(ctx context.Context, ref source.RawRef, before, after int) ([]source.RawRecord, int64, error)
AroundMeasured is Around, also returning the bytes it processed (read, decompressed and discarded, over every read-back attempt): see source.AroundMeter.
func (*Source) Describe ¶
Describe returns the source's schema: streams, sizes, the index coverage and the trusted fields with their indexed values.
func (*Source) IndexCounts ¶
func (s *Source) IndexCounts(ctx context.Context, q CountQuery) (CountResult, error)
IndexCounts answers a trusted-field count from the local index, at minute resolution: a minute counts when its start is in the range. Granules the index does not hold come back as a residual plan.
func (*Source) IndexStep ¶
IndexStep indexes granules not yet in the local index, newest first: files by mtime, and within a plain file from its last granule backwards, so that "when did it start" over recent data is answered from the index as early as possible. A compressed file is indexed in one forward pass, which may exceed maxBytes (by up to the decompression ratio cap): the scheduler's token bucket then goes negative and waits it out. It processes about maxBytes (a compressed file may exceed it) and reports the bytes processed and whether nothing is left to index.
With upkeep false, only backfill runs: streams and granules never indexed. Upkeep is re-indexing a grown tail granule; the scheduler pauses it while agents are busy.
func (*Source) NumericTerms ¶ added in v0.2.0
func (s *Source) NumericTerms()
NumericTerms implements source.NumericSelector: the index treats an ambiguous value of a numeric term as unknown.
func (*Source) Plan ¶
func (s *Source) Plan(ctx context.Context, r source.Range, sel source.Selector) (source.Byteplan, error)
Plan returns the chunks of the files that may hold records of r matching sel. Granules of the local index are filtered by their time span and, when every selector field is indexed, by their counts; other granules are all included.
Source Files