Documentation
¶
Overview ¶
Package loki is the Loki log source. It sends only allowlisted read requests through the egress client, selects server-side by time and by trusted stream labels only (trusted structured metadata and text are matched locally, after masking), drops every untrusted label and metadata field from what it returns, and plans fixed time slices whose cost is the bytes actually fetched.
Index ¶
- Constants
- Variables
- type Client
- type Matcher
- type Options
- type Selector
- type Source
- func (s *Source) Around(ctx context.Context, ref source.RawRef, before, after int) ([]source.RawRecord, error)
- func (s *Source) Describe(ctx context.Context) (source.Schema, error)
- func (s *Source) IndexCounts(ctx context.Context, q source.CountQuery) (source.CountResult, error)
- func (s *Source) Kind() string
- func (s *Source) Name() string
- func (s *Source) Plan(ctx context.Context, r source.Range, sel source.Selector) (source.Byteplan, error)
- func (s *Source) Reread(ctx context.Context, ref source.RawRef) (source.RawRecord, error)
- func (s *Source) Scan(ctx context.Context, c source.Chunk) (source.Iterator, error)
- func (s *Source) Split(g source.Granule) []source.Granule
Constants ¶
const ( // Limit is the entries per query_range page (Loki's default max). Limit = 5000 // InFlight is the requests in flight per source (the knee measured). InFlight = 4 // MaxSlices bounds the time slices of one plan. MaxSlices = 512 )
Defaults from the S3 spike.
const AroundWindow = 15 * time.Minute
AroundWindow bounds how far Around looks on each side.
const DescribeLookback = 24 * time.Hour
DescribeLookback is how far back Describe looks by default: a label or a value seen only before it is not listed (it is still queryable).
const Kind = "loki"
Kind is the source kind.
const MaxBoundaryFetches = 64
MaxBoundaryFetches bounds the boundaries whose entries one count fetches (each a log request under the call's limiter); past it, the count is not served and the engine scans.
const MinSlice = time.Second
MinSlice is the narrowest slice: one holding more than a call's budget is refused.
Variables ¶
var MaxAroundBytes int64 = 64 << 20
MaxAroundBytes bounds the response bytes of one Around when its context carries no byte limit (source.WithByteLimit). A var for tests.
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client interface {
GetWith(ctx context.Context, path string, q url.Values, hdr map[string]string) (egress.Response, error)
}
Client is the egress client of the source (egress.Client).
type Matcher ¶
type Matcher struct {
Name string
Op string // "=", "!=", "=~", "!~"
// Values: one for = and !=; the alternatives of =~ and !~ (literal
// values, quoted into an anchored alternation).
Values []string
}
Matcher is one label matcher or structured-metadata filter of a query.
type Options ¶
type Options struct {
Name string
Client Client
Parser *parse.Parser
// Selector is the profile's selector, ANDed into every query.
Selector map[string]string
// TrustedLabels and TrustedMetadata are shown (after the detectors)
// and selectable; every other label and metadata field is dropped.
TrustedLabels []string
TrustedMetadata []string
// InFlight overrides InFlight (tests).
InFlight int
// Limit overrides Limit (tests).
Limit int
// DescribeLookback is how far back Describe lists labels and values
// (0: DescribeLookback).
DescribeLookback time.Duration
Now func() time.Time
}
Options configure a Loki source.
type Selector ¶
Selector is what locklog sends Loki: stream matchers (the profile's selector ANDed with trusted-label terms) and, in the grammar, | filters on structured metadata. The source sends no Meta filters (a | filter also matches stream labels: see Source.pushable). Text never goes to Loki.
type Source ¶
type Source struct {
// contains filtered or unexported fields
}
Source is a Loki source. It is safe for concurrent use.
func (*Source) Around ¶
func (s *Source) Around(ctx context.Context, ref source.RawRef, before, after int) ([]source.RawRecord, error)
Around returns up to before entries before ref, ref's entry and up to after entries after it, from the same query. The window is fetched under the byte limit of ctx (else MaxAroundBytes): past it, Around fails with source.ErrByteLimit rather than buffer the window whole.
func (*Source) Describe ¶
Describe lists the trusted labels and metadata fields with the trusted labels' values (asked only for trusted labels), the names of other labels (masked by the caller), and the source kind. Every label request carries the profile's selector as its query: only the profile's streams are described. Labels and values are those of the last DescribeLookback (Options.DescribeLookback): older ones are not listed, though queryable.
func (*Source) IndexCounts ¶
func (s *Source) IndexCounts(ctx context.Context, q source.CountQuery) (source.CountResult, error)
IndexCounts answers a trusted-label count with server-side metric queries: no line leaves the source.
The engine's buckets are [b, b+bucket); Loki's count_over_time at t counts (t-bucket, t], to the nanosecond. Loki cannot be asked for the engine's windows directly: its query frontend rounds a range query's start down to the step (pkg/querier/queryrange/splitters.go, alignStartEnd), and grafana/loki:3.5.7 returns no point at all for an offset under 1 ms (an offset of 1 ms or more would move whole milliseconds across buckets). So the counts are Loki's windows, corrected at the boundaries:
- sum by (labels) (count_over_time({selector}[bucket])) at the points From+bucket .. To, step bucket (one bucket for the range: an instant query at To): the point at t is bucket t-bucket;
- the entries exactly on a boundary b belong to the bucket starting at b, not to the one Loki ends at b. A guard query, sum(count_over_time({selector}[1ms])) at From .. To, finds the boundaries with an entry in (b-1ms, b] (an instant count: From and To); the entries of those are fetched at exactly b and moved (Loki's window misses those at From and counts those at To, outside the range).
Grouping: Loki adds each entry's structured metadata to the labels a metric groups by, under the label's name unless the stream has that label (pkg/logql/log/metrics_extraction.go, LabelsBuilder.Add). An untrusted metadata field could then pass for a by-label's value. So each by-label is counted present (L!="" in the stream selector: the metadata becomes L_extracted) and absent (L="", not grouped on, value ""), one metric query per combination.
Ingestion during the count: the guard and the boundary fetches run before and after the metric queries, and must agree, else the count is not served.
Cost: one metric query per by-label combination (up to 4), two guard queries, and twice one log request per boundary to correct (at most MaxBoundaryFetches, at most 8 MiB of responses in all), every request under the call's limiter; the boundary fetches are marked as log fetches (source.AsLogFetch): requests, not metric queries. The boundary lines never leave the source.
Not served (source.ErrNotIndexable): a Loki limit refusal (too many series, too long a range, too many outstanding requests, a timeout); a point off the grid asked for; a count by anything but a trusted stream label (structured metadata included), or by more than two; a term on anything but a trusted stream label, or with no value or an empty one; an empty range; a bucket under a second, not whole seconds, not dividing the range, or cutting it into more than 11000 points; a bucketed range whose ends are not multiples of the bucket since the epoch; more than MaxBoundaryFetches boundaries to fetch or more than 8 MiB of them, or one with a page or more of entries at its nanosecond.
func (*Source) Plan ¶
func (s *Source) Plan(ctx context.Context, r source.Range, sel source.Selector) (source.Byteplan, error)
Plan returns one chunk of epoch-aligned time slices covering r, for the selector built from the profile and the trusted terms. The slices' byte cost is unknown until fetched: the engine charges the bytes fetched.
func (*Source) Reread ¶
Reread fetches the entry at ref again: ref.Offset is its timestamp and ref.Length its ordinal (from 1) among the entries of that nanosecond.
func (*Source) Scan ¶
Scan yields the records of the chunk's slices in time order. Slices are fetched lazily, at most InFlight of them ahead of the one Next is reading (each buffered whole, to be sorted), and every page's bytes count towards Processed as it arrives. The byte limit of ctx (source.WithByteLimit) bounds the bytes all of them fetch: past it, the fetches stop and Next fails with source.ErrByteLimit.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package lokitest is a fake Loki 3 for the tests of the packages that open a Loki source (the session, the engine through it): an http.Handler over in-memory entries that answers the requests locklog sends (labels, label values, query_range for streams and for the metric form, the instant metric query), with Loki's paging (limit over all streams, forward, end excluded) and categorized labels, behind basic authentication unless Open.
|
Package lokitest is a fake Loki 3 for the tests of the packages that open a Loki source (the session, the engine through it): an http.Handler over in-memory entries that answers the requests locklog sends (labels, label values, query_range for streams and for the metric form, the instant metric query), with Loki's paging (limit over all streams, forward, end excluded) and categorized labels, behind basic authentication unless Open. |