accesslog

package
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 4, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package accesslog parses and aggregates explicitly configured nginx LTSV, flat JSON, Caddy native JSON, and explicit Apache JSON access logs. Collection is pull based, so it adds no work to an application's HTTP path.

Index

Constants

View Source 
const (
	// DefaultMaxKeys bounds distinct method/path rows in one generation.
	DefaultMaxKeys = 10000
	// OverflowURI is used when the bounded key budget is exhausted.
	OverflowURI = "(other)"
)
View Source 
const (
	StatusOK      = "ok"
	StatusPartial = "partial"
	StatusError   = "error"
)

Variables

View Source 
var ErrCollectLimit = errors.New("accesslog: per-collect byte limit reached")

ErrCollectLimit means one collection pass reached its configured byte cap. A later pass resumes from the retained offset.

Functions

This section is empty.

Types

type Aggregator 

type Aggregator struct {
	// contains filtered or unexported fields
}

Aggregator is a concurrency-safe bounded access-log aggregation table.

func NewAggregator 

func NewAggregator(maxKeys int) *Aggregator

NewAggregator returns an empty table. A non-positive maxKeys selects the default bound.

func (*Aggregator) Observe 

func (a *Aggregator) Observe(rec Record)

Observe adds a parsed record.

func (*Aggregator) Reset 

func (a *Aggregator) Reset()

Reset clears the table and restores its key budget.

func (*Aggregator) Snapshot 

func (a *Aggregator) Snapshot() Snapshot

Snapshot returns rows sorted by total request time descending.

type Collector 

type Collector struct {
	// contains filtered or unexported fields
}

Collector tails a log from a generation baseline. It retains an open file descriptor so an inode-rotated file can be drained before the new path.

func New 

func New(path string, opts ...Option) *Collector

New creates a collector and sets its generation baseline to the current EOF. Startup failures are recorded in Health and do not panic or stop the caller.

func (*Collector) Close 

func (c *Collector) Close()

Close releases the retained log descriptor. It is idempotent.

func (*Collector) Collect 

func (c *Collector) Collect() error

Collect consumes all currently available complete lines. It returns only I/O failures; malformed records are dropped and reported through Health.

func (*Collector) CollectContext added in v1.1.0 

func (c *Collector) CollectContext(ctx context.Context) error

CollectContext is Collect with cancellation between bounded file reads.

func (*Collector) CollectUntilStable 

func (c *Collector) CollectUntilStable(ctx context.Context, quietFor, pollEvery time.Duration) error

CollectUntilStable polls until no newly flushed bytes are observed for quietFor. It is intended for explicitly buffered nginx logs at snapshot time; callers bound the wait with ctx. A non-positive quietFor performs one ordinary Collect.

func (*Collector) Health 

func (c *Collector) Health() Health

Health returns a copy of the current generation's collector health.

func (*Collector) Reset 

func (c *Collector) Reset() error

Reset clears aggregates and baselines the next generation at current EOF.

func (*Collector) Snapshot 

func (c *Collector) Snapshot() Snapshot

Snapshot collects available lines and returns an immutable aggregate copy. Any collection error is represented in the embedded Health value.

type DimensionCount 

type DimensionCount struct {
	Value string `json:"value"`
	Count int64  `json:"count"`
}

DimensionCount is a deterministic value/count pair used for cache status and content type breakdowns.

type Entry 

type Entry struct {
	Method string `json:"method"`
	URI    string `json:"uri"`

	Count        int64         `json:"count"`
	LatencyCount int64         `json:"latency_count"`
	RequestTotal time.Duration `json:"request_total_ns"`
	RequestAvg   time.Duration `json:"request_avg_ns"`
	RequestP95   time.Duration `json:"request_p95_ns"`
	RequestMax   time.Duration `json:"request_max_ns"`

	BytesTotal int64 `json:"bytes_total"`
	BytesAvg   int64 `json:"bytes_avg"`

	UpstreamTotal    time.Duration `json:"upstream_total_ns"`
	UpstreamAttempts int64         `json:"upstream_attempts"`
	ResidualTotal    time.Duration `json:"residual_total_ns"`
	ResidualCount    int64         `json:"residual_count"`
	NoUpstreamCount  int64         `json:"no_upstream_timing_count"`

	Status304 int64 `json:"status_304"`
	Status499 int64 `json:"status_499"`
	Status5xx int64 `json:"status_5xx"`

	CacheStatuses []DimensionCount `json:"cache_statuses"`
	ContentTypes  []DimensionCount `json:"content_types"`
}

Entry is one method/path aggregate. Status 101 contributes to count and bytes, but is deliberately excluded from latency and residual fields.

type File 

type File interface {
	io.Reader
	io.Seeker
	io.Closer
	Stat() (fs.FileInfo, error)
}

File is the seekable file boundary used by Collector.

type FileSystem 

type FileSystem interface {
	Open(name string) (File, error)
	Stat(name string) (fs.FileInfo, error)
}

FileSystem permits deterministic collector tests and non-OS-backed log sources. Implementations should return stable FileInfo values suitable for SameFile comparisons.

type FlowEntry added in v1.0.0 

type FlowEntry struct {
	From  string `json:"from"`
	To    string `json:"to"`
	Count int64  `json:"count"`
}

FlowEntry is one observed session transition (previous request -> next). Aggregated from the pseudonymous sess: log field, it shows how users actually move through the application.

type Health 

type Health struct {
	Status        string `json:"status"`
	Message       string `json:"message,omitempty"`
	LastError     string `json:"last_error,omitempty"`
	Errors        int64  `json:"errors"`
	Dropped       int64  `json:"dropped"`
	Partial       int64  `json:"partial"`
	Rotations     int64  `json:"rotations"`
	CopyTruncates int64  `json:"copy_truncates"`
	Offset        int64  `json:"offset"`
}

Health exposes collection failures without making them fatal to the host application. Counters apply to the current generation.

type Option 

type Option func(*Collector)

Option customizes a Collector.

func WithFileSystem 

func WithFileSystem(fsys FileSystem) Option

WithFileSystem replaces the OS filesystem boundary.

func WithMaxCollectBytes added in v1.1.0 

func WithMaxCollectBytes(n int64) Option

WithMaxCollectBytes bounds bytes read in one Collect/CollectContext call.

func WithMaxKeys 

func WithMaxKeys(n int) Option

WithMaxKeys bounds distinct method/path aggregates.

func WithMaxLineBytes 

func WithMaxLineBytes(n int) Option

WithMaxLineBytes bounds one pending log line. Oversized lines are discarded and reported through Health.

func WithSameFile 

func WithSameFile(same func(fs.FileInfo, fs.FileInfo) bool) Option

WithSameFile replaces inode-equivalence detection for an injected filesystem.

type ProtocolEntry added in v1.1.0 

type ProtocolEntry struct {
	Protocol   string        `json:"protocol"`
	Count      int64         `json:"count"`
	Status5xx  int64         `json:"status_5xx"`
	RequestP95 time.Duration `json:"request_p95_ns"`
}

ProtocolEntry summarizes client-facing protocol traffic observed at the reverse proxy. It must not be confused with the proxy-to-application upstream protocol.

type Record 

type Record struct {
	Method string `json:"method"`
	URI    string `json:"uri"`
	// Protocol is the client-facing HTTP protocol reported by the proxy.
	// It is optional so existing access-log formats remain compatible.
	Protocol string `json:"protocol,omitempty"`
	Status   int    `json:"status"`

	RequestTime time.Duration `json:"request_time_ns"`
	Bytes       int64         `json:"bytes"`
	CacheStatus string        `json:"cache_status"`
	ContentType string        `json:"content_type"`

	UpstreamRaw      string        `json:"upstream_raw"`
	UpstreamTotal    time.Duration `json:"upstream_total_ns"`
	UpstreamAttempts int           `json:"upstream_attempts"`
	UpstreamValid    bool          `json:"upstream_valid"`
	UpstreamComplete bool          `json:"upstream_complete"`
	NoUpstreamTiming bool          `json:"no_upstream_timing"`

	// Session is a pseudonymous session fragment (sess: field) used for
	// user-flow aggregation. Never a full cookie value.
	Session string `json:"session,omitempty"`
	// Scenario is a caller-supplied, non-secret story label such as
	// "login_and_browse". It is never inferred from authentication material.
	Scenario string `json:"scenario,omitempty"`

	QueryStripped bool   `json:"query_stripped,omitempty"`
	Partial       bool   `json:"partial,omitempty"`
	Issue         string `json:"issue,omitempty"`
}

Record is one normalized access-log record.

func ParseLine added in v0.5.0 

func ParseLine(line string) (Record, error)

ParseLine auto-detects the record syntax: JSON object lines (nginx log_format with JSON output, isutools or alp-style keys) or isutools LTSV.

func ParseNginxJSON added in v0.5.0 

func ParseNginxJSON(line string) (Record, error)

ParseNginxJSON parses one JSON-object access-log line. The name is retained for compatibility; flat isutools/alp JSON and Caddy native JSON are accepted. Missing optional fields default to bytes 0, upstime "-", empty cache/ctype.

func ParseNginxLTSV 

func ParseNginxLTSV(line string) (Record, error)

ParseNginxLTSV parses the explicit nginx "isutools" LTSV format documented in DESIGN.md. Invalid optional upstream timing preserves the useful request fields and marks the record partial; invalid required fields return an error.

type Snapshot 

type Snapshot struct {
	Entries      []Entry         `json:"entries"`
	Protocols    []ProtocolEntry `json:"protocols,omitempty"`
	Stories      []StoryEntry    `json:"stories,omitempty"`
	StoryDropped int64           `json:"story_dropped,omitempty"`
	Flows        []FlowEntry     `json:"flows,omitempty"`
	FlowDropped  int64           `json:"flow_dropped,omitempty"`
	Lines        int64           `json:"lines"`
	PartialLines int64           `json:"partial_lines"`
	Health       Health          `json:"health"`
}

Snapshot is an immutable copy of one access-log generation.

type StoryEntry added in v1.1.0 

type StoryEntry struct {
	Scenario string   `json:"scenario"`
	Journey  []string `json:"journey"`
	Sessions int64    `json:"sessions"`
	Requests int64    `json:"requests"`
}

StoryEntry is one observed request sequence grouped by an explicit scenario label. Sessions is the number of pseudonymous sessions with this sequence.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.