accesslog

package
v1.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 15, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package accesslog parses and aggregates explicitly configured proxy logs. Collection is pull based, so it adds no work to an application's HTTP path.

Index

Constants

View Source 
const (
	// DefaultMaxKeys bounds distinct method/path rows in one generation.
	DefaultMaxKeys = 10000
	// OverflowURI is used when the bounded key budget is exhausted.
	OverflowURI = "(other)"
)
View Source 
const (
	StatusOK      = "ok"
	StatusPartial = "partial"
	StatusError   = "error"
)
View Source 
const (
	MaxPathRuleSpecBytes = 8192
	MaxPathRules         = 64
	MaxPathPatternBytes  = 256
	MaxPathOutputBytes   = 256
	MaxPathInputBytes    = 4096
	UnmatchedURI         = "(unmatched)"
)
View Source 
const SectionName = "accesslog"

SectionName is the snapshot section this collector fills. It is the key the existing snapshot and health output already use for the access log.

View Source 
const SkippedLateRegistration = "" /* 144-byte string literal not displayed */

SkippedLateRegistration is the health reason recorded when a run is refused a section because this collector joined after the run had opened. It names the operator-fixable cause, which "no accesslog section" on its own does not.

Variables

View Source 
var (
	// ErrForeignHandle rejects a handle minted by another collector or by
	// another instance of this one.
	ErrForeignHandle = errors.New("accesslog: generation handle belongs to another collector")
	// ErrHandleReleased reports a handle whose data was already freed.
	ErrHandleReleased = errors.New("accesslog: generation handle was released")
	// ErrNotDrained reports a generation that was never drained, so no fixed
	// value exists to collect.
	ErrNotDrained = errors.New("accesslog: generation was not drained")
	// ErrFreezeWithoutBoundary rejects a closing boundary that would mint this
	// collector's first generation, which happens when the adapter was
	// registered after the run had already opened. See the ordering
	// requirement on GenerationCollector.
	ErrFreezeWithoutBoundary = errors.New("accesslog: freeze without a preceding boundary for this epoch; the section is skipped")
)

Handle errors. They are sentinels because the run controller maps a Collect failure onto a stable machine-readable code and must be able to tell "this handle is not mine" from "this generation has no data yet".

View Source 
var ErrCollectLimit = errors.New("accesslog: per-collect byte limit reached")

ErrCollectLimit means one collection pass reached its configured byte cap. A later pass resumes from the retained offset.

View Source 
var ErrSkipLine = errors.New("accesslog: non-record line")

ErrSkipLine is returned for comments and format headers that are not request records. Collector treats it as neither malformed nor dropped.

Functions

This section is empty.

Types

type Aggregator 

type Aggregator struct {
	// contains filtered or unexported fields
}

Aggregator is a concurrency-safe bounded access-log aggregation table.

func NewAggregator 

func NewAggregator(maxKeys int) *Aggregator

NewAggregator returns an empty table. A non-positive maxKeys selects the default bound.

func NewAggregatorWithFlowVisualization added in v1.7.0 

func NewAggregatorWithFlowVisualization(maxKeys int, options flowviz.Options) (*Aggregator, error)

NewAggregatorWithFlowVisualization enables bounded funnel and graph output without changing the legacy constructor's zero-overhead behavior.

func (*Aggregator) Observe 

func (a *Aggregator) Observe(rec Record)

Observe adds a parsed record.

func (*Aggregator) Reset 

func (a *Aggregator) Reset()

Reset clears the table and restores its key budget.

func (*Aggregator) Snapshot 

func (a *Aggregator) Snapshot() Snapshot

Snapshot returns rows sorted by total request time descending.

type Collector 

type Collector struct {
	// contains filtered or unexported fields
}

Collector tails a log from a generation baseline. It retains an open file descriptor so an inode-rotated file can be drained before the new path.

func New 

func New(path string, opts ...Option) *Collector

New creates a collector and sets its generation baseline to the current EOF. Startup failures are recorded in Health and do not panic or stop the caller.

func (*Collector) Close 

func (c *Collector) Close()

Close releases the retained log descriptor. It is idempotent.

func (*Collector) Collect 

func (c *Collector) Collect() error

Collect consumes all currently available complete lines. It returns only I/O failures; malformed records are dropped and reported through Health.

func (*Collector) CollectContext added in v1.1.0 

func (c *Collector) CollectContext(ctx context.Context) error

CollectContext is Collect with cancellation between bounded file reads.

func (*Collector) CollectUntilStable 

func (c *Collector) CollectUntilStable(ctx context.Context, quietFor, pollEvery time.Duration) error

CollectUntilStable polls until no newly flushed bytes are observed for quietFor. It is intended for explicitly buffered nginx logs at snapshot time; callers bound the wait with ctx. A non-positive quietFor performs one ordinary Collect.

func (*Collector) Health 

func (c *Collector) Health() Health

Health returns a copy of the current generation's collector health.

func (*Collector) Peek added in v1.2.0 

func (c *Collector) Peek() Snapshot

Peek returns an immutable copy of the current generation's aggregate for a reader that only wants to display it.

It is Snapshot without the one thing a display read must never do: while a generation's freeze point is fixed and not yet drained, Peek does not read the log at all. Reading there would carry the offset past the freeze point, and the drain that follows would seal the bytes beyond it — traffic the boundary was placed to exclude — into the generation it is cutting. With no boundary outstanding there is nothing to cross, so Peek refreshes first and the report stays live.

The test and the read happen under one hold of the collector's lock, so a boundary racing a peek either wins, and the peek reads nothing, or loses, and the freeze point it records already accounts for the bytes just read.

func (*Collector) Reset 

func (c *Collector) Reset() error

Reset clears aggregates and baselines the next generation at current EOF.

func (*Collector) Snapshot 

func (c *Collector) Snapshot() Snapshot

Snapshot collects available lines and returns an immutable aggregate copy. Any collection error is represented in the embedded Health value.

It reads the log to end of file, so it must not be called between a generation boundary and that generation's drain: the bytes it would pull in belong to the generation after the boundary, and the drain would seal them into the one it is cutting. A reader that only wants to display what has already been consumed must use Peek instead. See GenerationCollector.

type DimensionCount 

type DimensionCount struct {
	Value string `json:"value"`
	Count int64  `json:"count"`
}

DimensionCount is a deterministic value/count pair used for cache status and content type breakdowns.

type Entry 

type Entry struct {
	Method string `json:"method"`
	URI    string `json:"uri"`

	Count        int64         `json:"count"`
	LatencyCount int64         `json:"latency_count"`
	RequestTotal time.Duration `json:"request_total_ns"`
	RequestAvg   time.Duration `json:"request_avg_ns"`
	RequestP95   time.Duration `json:"request_p95_ns"`
	RequestMax   time.Duration `json:"request_max_ns"`

	BytesTotal int64 `json:"bytes_total"`
	BytesAvg   int64 `json:"bytes_avg"`

	UpstreamTotal    time.Duration `json:"upstream_total_ns"`
	UpstreamAttempts int64         `json:"upstream_attempts"`
	ResidualTotal    time.Duration `json:"residual_total_ns"`
	ResidualCount    int64         `json:"residual_count"`
	NoUpstreamCount  int64         `json:"no_upstream_timing_count"`

	Status304 int64 `json:"status_304"`
	Status499 int64 `json:"status_499"`
	Status5xx int64 `json:"status_5xx"`

	CacheStatuses []DimensionCount `json:"cache_statuses"`
	ContentTypes  []DimensionCount `json:"content_types"`
}

Entry is one method/path aggregate. Status 101 contributes to count and bytes, but is deliberately excluded from latency and residual fields.

type File 

type File interface {
	io.Reader
	io.Seeker
	io.Closer
	Stat() (fs.FileInfo, error)
}

File is the seekable file boundary used by Collector.

type FileSystem 

type FileSystem interface {
	Open(name string) (File, error)
	Stat(name string) (fs.FileInfo, error)
}

FileSystem permits deterministic collector tests and non-OS-backed log sources. Implementations should return stable FileInfo values suitable for SameFile comparisons.

type FlowEntry added in v1.0.0 

type FlowEntry struct {
	From  string `json:"from"`
	To    string `json:"to"`
	Count int64  `json:"count"`
}

FlowEntry is one observed session transition (previous request -> next). Aggregated from the pseudonymous sess: log field, it shows how users actually move through the application.

type Format added in v1.6.0 

type Format string

Format names an access-log wire contract. Explicit formats are preferred: they make product-specific duration units impossible to confuse. 

const (
	FormatAuto         Format = "auto"
	FormatIsutoolsLTSV Format = "isutools-ltsv"
	FormatIsutoolsJSON Format = "isutools-json-v1"
	FormatCaddyJSON    Format = "caddy-json"
	FormatTraefikJSON  Format = "traefik-json"
	FormatIISW3C       Format = "iis-w3c"
)

func ParseFormat added in v1.6.0 

func ParseFormat(value string) (Format, error)

ParseFormat validates a configured decoder name. A few compatibility aliases are accepted, and ParseFormat returns the canonical name.

func (Format) String added in v1.6.0 

func (f Format) String() string

type GenerationCollector added in v1.2.0 

type GenerationCollector struct {
	// contains filtered or unexported fields
}

GenerationCollector adapts a log Collector to runctl.GenerationCollector.

An access log is pulled, not pushed, so a generation cannot be closed by a pointer swap: the lines it owns are still sitting in the file. The boundary therefore fixes the *freeze point* — the offset the log had ended at — and Drain reads up to exactly that offset. That split is what keeps a Finish from swallowing requests that were logged after it, and it keeps the boundary free of file I/O beyond a single stat.

The consequence of the pull model is that this adapter has to be the only reader: a bare Collector.Collect between a boundary and its drain reads to end of file, which pulls the next generation's lines into the aggregate the closing generation is about to be cut from. Collector.Snapshot flushes too, so it carries the same restriction. A reader that only wants to display the aggregate — the live report, which refreshes whenever somebody has the page open — uses Collector.Peek, which declines to read while this adapter has a freeze point outstanding.

Ordering requirement

Every run must reach BeginBoundary on this collector before it reaches Freeze, and therefore the adapter must be registered before the first run opens. A generation's lower edge is its predecessor's freeze point, so the very first generation of a collector begins at the collector's start of life. If the first generation were minted by a Freeze — the shape of an adapter registered in the middle of a run — the run's section would contain every line logged since the process started and would be reported as that run's interval. Such a Freeze is refused with ErrFreezeWithoutBoundary and the section is skipped: a missing interval is recoverable, a silently inflated one is not. The refusal mints nothing, so the next run that takes both boundaries measures correctly.

func NewGenerationCollector added in v1.2.0 

func NewGenerationCollector(collector *Collector) *GenerationCollector

NewGenerationCollector wraps a log collector. A nil collector yields an adapter whose boundaries fail instead of one that panics on first use: an unconfigured access log must not be able to break a run.

func (*GenerationCollector) BeginBoundary added in v1.2.0 

func (c *GenerationCollector) BeginBoundary(ctx context.Context, runID string, ep runctl.Epoch) (runctl.BoundaryResult, error)

BeginBoundary fixes the freeze point of the generation it closes and starts the next one at that same offset.

func (*GenerationCollector) Collect added in v1.2.0 

func (c *GenerationCollector) Collect(h runctl.GenerationHandle) (any, error)

Collect returns the frozen access-log aggregate. It reads only what Drain fixed and never triggers a read of its own.

func (*GenerationCollector) Drain added in v1.2.0 

func (c *GenerationCollector) Drain(ctx context.Context, h runctl.GenerationHandle) error

Drain reads the log up to the handle's freeze point and not one byte further, then cuts the aggregate so the next generation starts empty.

A cancelled context stops the catch-up between bounded reads; whatever was read stays in the generation and a later Drain resumes from there.

func (*GenerationCollector) Freeze added in v1.2.0 

func (c *GenerationCollector) Freeze(ctx context.Context, runID string, ep runctl.Epoch) (runctl.BoundaryResult, error)

Freeze fixes the run's closing offset. Lines appended after it belong to the next generation, outside the run, even though they are read later.

It fails with ErrFreezeWithoutBoundary when it would mint this collector's first generation, because that generation would start at the collector's start of life rather than at the run's opening boundary. See the ordering requirement on GenerationCollector.

func (*GenerationCollector) Name added in v1.2.0 

func (c *GenerationCollector) Name() string

Name identifies the snapshot section this collector fills.

func (*GenerationCollector) Release added in v1.2.0 

func (c *GenerationCollector) Release(h runctl.GenerationHandle)

Release drops the frozen aggregate. It is idempotent, and a handle this collector never minted is ignored rather than reported: Release has no error channel and must not panic into the caller.

type Health 

type Health struct {
	Status        string `json:"status"`
	Message       string `json:"message,omitempty"`
	LastError     string `json:"last_error,omitempty"`
	Errors        int64  `json:"errors"`
	Dropped       int64  `json:"dropped"`
	Partial       int64  `json:"partial"`
	Rotations     int64  `json:"rotations"`
	CopyTruncates int64  `json:"copy_truncates"`
	Offset        int64  `json:"offset"`
}

Health exposes collection failures without making them fatal to the host application. Counters apply to the current generation.

type Option 

type Option func(*Collector)

Option customizes a Collector.

func WithFileSystem 

func WithFileSystem(fsys FileSystem) Option

WithFileSystem replaces the OS filesystem boundary.

func WithFlowVisualization added in v1.7.0 

func WithFlowVisualization(options flowviz.Options) Option

WithFlowVisualization enables bounded funnel and transition graph output. It is order-independent with WithMaxKeys.

func WithFormat added in v1.6.0 

func WithFormat(format Format) Option

WithFormat selects one validated decoder contract.

func WithFormatSpec added in v1.6.0 

func WithFormatSpec(spec string) Option

WithFormatSpec parses ISUTOOLS_ACCESS_LOG_FORMAT without retaining its raw value in health output.

func WithMaxCollectBytes added in v1.1.0 

func WithMaxCollectBytes(n int64) Option

WithMaxCollectBytes bounds bytes read in one Collect/CollectContext call.

func WithMaxKeys 

func WithMaxKeys(n int) Option

WithMaxKeys bounds distinct method/path aggregates.

func WithMaxLineBytes 

func WithMaxLineBytes(n int) Option

WithMaxLineBytes bounds one pending log line. Oversized lines are discarded and reported through Health.

func WithPathRules added in v1.5.0 

func WithPathRules(rules *PathRules) Option

WithPathRules applies one immutable grouping policy after query stripping and before aggregate, flow, and story observation.

func WithPathRulesSpec added in v1.5.0 

func WithPathRulesSpec(spec string, unmatched UnmatchedPolicy) Option

WithPathRulesSpec parses a bounded configuration without ever putting the raw specification into health output.

func WithSameFile 

func WithSameFile(same func(fs.FileInfo, fs.FileInfo) bool) Option

WithSameFile replaces inode-equivalence detection for an injected filesystem.

type PathRuleError added in v1.5.0 

type PathRuleError struct{ Code string }

func (*PathRuleError) Error added in v1.5.0 

func (e *PathRuleError) Error() string

type PathRules added in v1.5.0 

type PathRules struct {
	// contains filtered or unexported fields
}

PathRules is an immutable, ordered, full-match URI grouping policy.

func ParsePathRules added in v1.5.0 

func ParsePathRules(spec string, unmatched UnmatchedPolicy) (*PathRules, error)

func (*PathRules) Normalize added in v1.5.0 

func (r *PathRules) Normalize(uri string) string

type ProtocolEntry added in v1.1.0 

type ProtocolEntry struct {
	Protocol   string        `json:"protocol"`
	Count      int64         `json:"count"`
	Status5xx  int64         `json:"status_5xx"`
	RequestP95 time.Duration `json:"request_p95_ns"`
}

ProtocolEntry summarizes client-facing protocol traffic observed at the reverse proxy. It must not be confused with the proxy-to-application upstream protocol.

type Record 

type Record struct {
	Method string `json:"method"`
	URI    string `json:"uri"`
	// Protocol is the client-facing HTTP protocol reported by the proxy.
	// It is optional so existing access-log formats remain compatible.
	Protocol string `json:"protocol,omitempty"`
	Status   int    `json:"status"`

	RequestTime time.Duration `json:"request_time_ns"`
	Bytes       int64         `json:"bytes"`
	CacheStatus string        `json:"cache_status"`
	ContentType string        `json:"content_type"`

	UpstreamRaw      string        `json:"upstream_raw"`
	UpstreamTotal    time.Duration `json:"upstream_total_ns"`
	UpstreamAttempts int           `json:"upstream_attempts"`
	UpstreamValid    bool          `json:"upstream_valid"`
	UpstreamComplete bool          `json:"upstream_complete"`
	NoUpstreamTiming bool          `json:"no_upstream_timing"`

	// Session is a pseudonymous session fragment (sess: field) used for
	// user-flow aggregation. Never a full cookie value.
	Session string `json:"session,omitempty"`
	// Scenario is a caller-supplied, non-secret story label such as
	// "login_and_browse". It is never inferred from authentication material.
	Scenario string `json:"scenario,omitempty"`
	// ObservedAt is populated by in-process middleware. Existing proxy formats
	// omit it, which is reported explicitly when a configured funnel uses a
	// cross-request time window.
	ObservedAt time.Time `json:"observed_at,omitempty"`

	QueryStripped bool   `json:"query_stripped,omitempty"`
	Partial       bool   `json:"partial,omitempty"`
	Issue         string `json:"issue,omitempty"`
}

Record is one normalized access-log record.

func ParseLine added in v0.5.0 

func ParseLine(line string) (Record, error)

ParseLine auto-detects the record syntax: JSON object lines (nginx log_format with JSON output, isutools or alp-style keys) or isutools LTSV.

func ParseLineFormat added in v1.6.0 

func ParseLineFormat(format Format, line string) (Record, error)

ParseLineFormat decodes one record using an explicit wire contract.

func ParseNginxJSON added in v0.5.0 

func ParseNginxJSON(line string) (Record, error)

ParseNginxJSON parses one JSON-object access-log line. The name is retained for compatibility; flat isutools/alp JSON and Caddy native JSON are accepted. Missing optional fields default to bytes 0, upstime "-", empty cache/ctype.

func ParseNginxLTSV 

func ParseNginxLTSV(line string) (Record, error)

ParseNginxLTSV parses the explicit nginx "isutools" LTSV format documented in DESIGN.md. Invalid optional upstream timing preserves the useful request fields and marks the record partial; invalid required fields return an error.

type Snapshot 

type Snapshot struct {
	Entries       []Entry           `json:"entries"`
	Protocols     []ProtocolEntry   `json:"protocols,omitempty"`
	Stories       []StoryEntry      `json:"stories,omitempty"`
	StoryDropped  int64             `json:"story_dropped,omitempty"`
	Flows         []FlowEntry       `json:"flows,omitempty"`
	FlowDropped   int64             `json:"flow_dropped,omitempty"`
	Visualization *flowviz.Snapshot `json:"visualization,omitempty"`
	Lines         int64             `json:"lines"`
	PartialLines  int64             `json:"partial_lines"`
	Health        Health            `json:"health"`
}

Snapshot is an immutable copy of one access-log generation.

type StoryEntry added in v1.1.0 

type StoryEntry struct {
	Scenario string   `json:"scenario"`
	Journey  []string `json:"journey"`
	Sessions int64    `json:"sessions"`
	Requests int64    `json:"requests"`
}

StoryEntry is one observed request sequence grouped by an explicit scenario label. Sessions is the number of pseudonymous sessions with this sequence.

type UnmatchedPolicy added in v1.5.0 

type UnmatchedPolicy string
const (
	UnmatchedKeep     UnmatchedPolicy = "keep"
	UnmatchedCollapse UnmatchedPolicy = "collapse"
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.