Help us shape the future of Go! Take the 2026 Go Developer Survey

observability

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 3, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package observability exposes ScrapeGoat's operational metrics.

Metrics are backed by prometheus/client_golang rather than a hand-written text exposition. The previous implementation emitted `# TYPE <name> counter` for every metric — including gauges like queue depth and active workers, where `rate()` produces nonsense — carried no labels, and had no histograms, so p95 fetch latency (the number a crawler is actually judged on) could not be expressed at all. Hand-rolling the format also forfeited the Go runtime collectors and guaranteed drift from the spec.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AddEvent

func AddEvent(span *Span, name string, attrs map[string]any)

AddEvent records an event within a span.

func AddTag

func AddTag(span *Span, key string, value any)

AddTag adds a key-value tag to a span.

func SetError

func SetError(span *Span, err error)

SetError marks a span as failed.

Types

type LogExporter

type LogExporter struct {
	// contains filtered or unexported fields
}

LogExporter exports spans to slog.

func NewLogExporter

func NewLogExporter(logger *slog.Logger) *LogExporter

NewLogExporter creates a log-based span exporter.

func (*LogExporter) Close

func (e *LogExporter) Close() error

Close is a no-op for log exporter.

func (*LogExporter) Export

func (e *LogExporter) Export(spans []*Span) error

Export logs each span.

type Metrics

type Metrics struct {

	// RequestsTotal counts fetch attempts by domain and outcome. `outcome` is
	// "ok", "error", or "retry" — a status code alone cannot distinguish a
	// transport failure from an HTTP error.
	RequestsTotal *prometheus.CounterVec

	// ResponsesTotal counts responses by domain and status code. The code is a
	// label rather than a metric per class, so 404-vs-410 stays visible.
	ResponsesTotal *prometheus.CounterVec

	// FetchDuration is the histogram that makes p50/p95/p99 latency answerable.
	// Buckets span 10 ms to ~60 s: the range between a warm CDN and a timeout.
	FetchDuration *prometheus.HistogramVec

	// ResponseSize tracks downloaded body sizes, which is how you notice a site
	// that has started serving an interstitial instead of content.
	ResponseSize *prometheus.HistogramVec

	// ItemsTotal counts pipeline outcomes: scraped, dropped, stored.
	ItemsTotal *prometheus.CounterVec

	// BytesDownloaded is the cumulative transfer counter.
	BytesDownloaded prometheus.Counter

	// FrontierDepth and ActiveWorkers are gauges — they go down as well as up,
	// which is exactly what the old exposition could not express.
	FrontierDepth prometheus.Gauge
	ActiveWorkers prometheus.Gauge

	// ProxyRotations and ProxyErrors track proxy pool health.
	ProxyRotations prometheus.Counter
	ProxyErrors    prometheus.Counter
	// contains filtered or unexported fields
}

Metrics holds the crawler's Prometheus collectors.

Construct one with NewMetrics and hand it to engine.New. An engine given a nil Metrics records nothing, which keeps metrics genuinely optional rather than requiring every caller and test to build one.

func NewMetrics

func NewMetrics(logger *slog.Logger) *Metrics

NewMetrics creates and registers the collector set.

func (*Metrics) Handler

func (m *Metrics) Handler() http.Handler

Handler returns the HTTP handler serving the exposition endpoint.

func (*Metrics) RecordItem

func (m *Metrics) RecordItem(outcome string)

RecordItem records a pipeline outcome: "scraped", "dropped", or "stored".

func (*Metrics) RecordProxyError

func (m *Metrics) RecordProxyError()

RecordProxyError records a proxy failure.

func (*Metrics) RecordProxyRotation

func (m *Metrics) RecordProxyRotation()

RecordProxyRotation records a proxy switch.

func (*Metrics) RecordRequest

func (m *Metrics) RecordRequest(domain, outcome string)

RecordRequest records a dispatched request.

func (*Metrics) RecordResponse

func (m *Metrics) RecordResponse(domain, fetcher string, status int, d time.Duration, size int64)

RecordResponse records a completed fetch: status, duration, and size.

func (*Metrics) Registry

func (m *Metrics) Registry() *prometheus.Registry

Registry exposes the underlying registry, for tests and for callers that want to register collectors of their own.

func (*Metrics) ServeHTTP

func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP serves metrics in the Prometheus exposition format.

func (*Metrics) SetActiveWorkers

func (m *Metrics) SetActiveWorkers(n int)

SetActiveWorkers publishes the current busy-worker count.

func (*Metrics) SetFrontierDepth

func (m *Metrics) SetFrontierDepth(n int)

SetFrontierDepth publishes the current queue depth.

func (*Metrics) StartServer

func (m *Metrics) StartServer(port int, path string) error

StartServer starts the metrics HTTP server.

type Span

type Span struct {
	TraceID   string         `json:"trace_id"`
	SpanID    string         `json:"span_id"`
	ParentID  string         `json:"parent_id,omitempty"`
	Operation string         `json:"operation"`
	Service   string         `json:"service"`
	StartTime time.Time      `json:"start_time"`
	EndTime   time.Time      `json:"end_time,omitempty"`
	Duration  time.Duration  `json:"duration,omitempty"`
	Status    SpanStatus     `json:"status"`
	Tags      map[string]any `json:"tags,omitempty"`
	Events    []SpanEvent    `json:"events,omitempty"`
	// contains filtered or unexported fields
}

Span represents a unit of work in the request lifecycle.

type SpanEvent

type SpanEvent struct {
	Name      string         `json:"name"`
	Timestamp time.Time      `json:"timestamp"`
	Attrs     map[string]any `json:"attrs,omitempty"`
}

SpanEvent represents a notable event within a span.

type SpanExporter

type SpanExporter interface {
	Export(spans []*Span) error
	Close() error
}

SpanExporter exports completed spans to an external system.

type SpanStatus

type SpanStatus int

SpanStatus indicates the outcome of a span.

const (
	SpanStatusUnset SpanStatus = iota
	SpanStatusOK
	SpanStatusError
)

type Tracer

type Tracer struct {
	// contains filtered or unexported fields
}

Tracer provides distributed tracing for request lifecycle tracking. Implements a lightweight tracing API compatible with OpenTelemetry concepts.

Each crawl request creates a span: Fetch → Parse → Pipeline → Storage. Spans can be exported to Jaeger, Zipkin, or logged to stdout.

func NewTracer

func NewTracer(cfg *TracerConfig, logger *slog.Logger) *Tracer

NewTracer creates a new tracer.

func (*Tracer) Close

func (t *Tracer) Close() error

Close flushes remaining spans and closes the exporter.

func (*Tracer) EndSpan

func (t *Tracer) EndSpan(span *Span)

EndSpan completes a span and records it.

func (*Tracer) Flush

func (t *Tracer) Flush() error

Flush exports all collected spans.

func (*Tracer) SetExporter

func (t *Tracer) SetExporter(e SpanExporter)

SetExporter sets the span exporter.

func (*Tracer) StartSpan

func (t *Tracer) StartSpan(ctx context.Context, operation string) (*Span, context.Context)

StartSpan begins a new span for the given operation.

func (*Tracer) Stats

func (t *Tracer) Stats() map[string]any

Stats returns tracing statistics.

type TracerConfig

type TracerConfig struct {
	Enabled     bool
	ServiceName string
	SampleRate  float64
	MaxSpans    int
}

TracerConfig configures the tracer.

func DefaultTracerConfig

func DefaultTracerConfig() *TracerConfig

DefaultTracerConfig returns sensible defaults.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL