Documentation
¶
Overview ¶
Package observability exposes ScrapeGoat's operational metrics.
Metrics are backed by prometheus/client_golang rather than a hand-written text exposition. The previous implementation emitted `# TYPE <name> counter` for every metric — including gauges like queue depth and active workers, where `rate()` produces nonsense — carried no labels, and had no histograms, so p95 fetch latency (the number a crawler is actually judged on) could not be expressed at all. Hand-rolling the format also forfeited the Go runtime collectors and guaranteed drift from the spec.
Index ¶
- func AddEvent(span *Span, name string, attrs map[string]any)
- func AddTag(span *Span, key string, value any)
- func SetError(span *Span, err error)
- type LogExporter
- type Metrics
- func (m *Metrics) Handler() http.Handler
- func (m *Metrics) RecordItem(outcome string)
- func (m *Metrics) RecordProxyError()
- func (m *Metrics) RecordProxyRotation()
- func (m *Metrics) RecordRequest(domain, outcome string)
- func (m *Metrics) RecordResponse(domain, fetcher string, status int, d time.Duration, size int64)
- func (m *Metrics) Registry() *prometheus.Registry
- func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request)
- func (m *Metrics) SetActiveWorkers(n int)
- func (m *Metrics) SetFrontierDepth(n int)
- func (m *Metrics) StartServer(port int, path string) error
- type Span
- type SpanEvent
- type SpanExporter
- type SpanStatus
- type Tracer
- type TracerConfig
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type LogExporter ¶
type LogExporter struct {
// contains filtered or unexported fields
}
LogExporter exports spans to slog.
func NewLogExporter ¶
func NewLogExporter(logger *slog.Logger) *LogExporter
NewLogExporter creates a log-based span exporter.
func (*LogExporter) Export ¶
func (e *LogExporter) Export(spans []*Span) error
Export logs each span.
type Metrics ¶
type Metrics struct {
// RequestsTotal counts fetch attempts by domain and outcome. `outcome` is
// "ok", "error", or "retry" — a status code alone cannot distinguish a
// transport failure from an HTTP error.
RequestsTotal *prometheus.CounterVec
// ResponsesTotal counts responses by domain and status code. The code is a
// label rather than a metric per class, so 404-vs-410 stays visible.
ResponsesTotal *prometheus.CounterVec
// FetchDuration is the histogram that makes p50/p95/p99 latency answerable.
// Buckets span 10 ms to ~60 s: the range between a warm CDN and a timeout.
FetchDuration *prometheus.HistogramVec
// ResponseSize tracks downloaded body sizes, which is how you notice a site
// that has started serving an interstitial instead of content.
ResponseSize *prometheus.HistogramVec
// ItemsTotal counts pipeline outcomes: scraped, dropped, stored.
ItemsTotal *prometheus.CounterVec
// BytesDownloaded is the cumulative transfer counter.
BytesDownloaded prometheus.Counter
// FrontierDepth and ActiveWorkers are gauges — they go down as well as up,
// which is exactly what the old exposition could not express.
FrontierDepth prometheus.Gauge
ActiveWorkers prometheus.Gauge
// ProxyRotations and ProxyErrors track proxy pool health.
ProxyRotations prometheus.Counter
ProxyErrors prometheus.Counter
// contains filtered or unexported fields
}
Metrics holds the crawler's Prometheus collectors.
Construct one with NewMetrics and hand it to engine.New. An engine given a nil Metrics records nothing, which keeps metrics genuinely optional rather than requiring every caller and test to build one.
func NewMetrics ¶
NewMetrics creates and registers the collector set.
func (*Metrics) RecordItem ¶
RecordItem records a pipeline outcome: "scraped", "dropped", or "stored".
func (*Metrics) RecordProxyError ¶
func (m *Metrics) RecordProxyError()
RecordProxyError records a proxy failure.
func (*Metrics) RecordProxyRotation ¶
func (m *Metrics) RecordProxyRotation()
RecordProxyRotation records a proxy switch.
func (*Metrics) RecordRequest ¶
RecordRequest records a dispatched request.
func (*Metrics) RecordResponse ¶
RecordResponse records a completed fetch: status, duration, and size.
func (*Metrics) Registry ¶
func (m *Metrics) Registry() *prometheus.Registry
Registry exposes the underlying registry, for tests and for callers that want to register collectors of their own.
func (*Metrics) ServeHTTP ¶
func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request)
ServeHTTP serves metrics in the Prometheus exposition format.
func (*Metrics) SetActiveWorkers ¶
SetActiveWorkers publishes the current busy-worker count.
func (*Metrics) SetFrontierDepth ¶
SetFrontierDepth publishes the current queue depth.
type Span ¶
type Span struct {
TraceID string `json:"trace_id"`
SpanID string `json:"span_id"`
ParentID string `json:"parent_id,omitempty"`
Operation string `json:"operation"`
Service string `json:"service"`
StartTime time.Time `json:"start_time"`
EndTime time.Time `json:"end_time,omitempty"`
Duration time.Duration `json:"duration,omitempty"`
Status SpanStatus `json:"status"`
Tags map[string]any `json:"tags,omitempty"`
Events []SpanEvent `json:"events,omitempty"`
// contains filtered or unexported fields
}
Span represents a unit of work in the request lifecycle.
type SpanEvent ¶
type SpanEvent struct {
Name string `json:"name"`
Timestamp time.Time `json:"timestamp"`
Attrs map[string]any `json:"attrs,omitempty"`
}
SpanEvent represents a notable event within a span.
type SpanExporter ¶
SpanExporter exports completed spans to an external system.
type SpanStatus ¶
type SpanStatus int
SpanStatus indicates the outcome of a span.
const ( SpanStatusUnset SpanStatus = iota SpanStatusOK SpanStatusError )
type Tracer ¶
type Tracer struct {
// contains filtered or unexported fields
}
Tracer provides distributed tracing for request lifecycle tracking. Implements a lightweight tracing API compatible with OpenTelemetry concepts.
Each crawl request creates a span: Fetch → Parse → Pipeline → Storage. Spans can be exported to Jaeger, Zipkin, or logged to stdout.
func NewTracer ¶
func NewTracer(cfg *TracerConfig, logger *slog.Logger) *Tracer
NewTracer creates a new tracer.
func (*Tracer) SetExporter ¶
func (t *Tracer) SetExporter(e SpanExporter)
SetExporter sets the span exporter.
type TracerConfig ¶
TracerConfig configures the tracer.
func DefaultTracerConfig ¶
func DefaultTracerConfig() *TracerConfig
DefaultTracerConfig returns sensible defaults.