trafficexport

package
v1.3.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Overview

Package trafficexport delivers finished edge requests to analytics vendors through bounded, lossy queues so a slow vendor never slows a real request.

Index

Constants

View Source
const (
	EnvTimeout       = "APP_TRAFFIC_EXPORT_TIMEOUT"        // HTTP timeout per delivery, default 10s
	EnvAllowPrivate  = "APP_TRAFFIC_EXPORT_ALLOW_PRIVATE"  // 1 allows loopback and link-local targets
	EnvMaxBodyBytes  = "APP_TRAFFIC_EXPORT_MAX_BODY_BYTES" // largest request body, default 20 MiB
	EnvQueueSize     = "APP_TRAFFIC_EXPORT_QUEUE_SIZE"     // events buffered per sink, default 10000
	EnvBatchSize     = "APP_TRAFFIC_EXPORT_BATCH_SIZE"     // events per delivery, default 500
	EnvFlushInterval = "APP_TRAFFIC_EXPORT_FLUSH_INTERVAL" // longest wait for a full batch, default 5s
	EnvMaxAttempts   = "APP_TRAFFIC_EXPORT_MAX_ATTEMPTS"   // deliveries tried per batch, default 5
	EnvBackoffBase   = "APP_TRAFFIC_EXPORT_BACKOFF_BASE"   // first retry delay, doubles each try, default 500ms
	EnvBackoffMax    = "APP_TRAFFIC_EXPORT_BACKOFF_MAX"    // retry delay ceiling, default 30s
)

Environment variable names read by this package; invalid values fall back to defaults.

View Source
const (
	KindAuth     = "auth"
	KindRejected = "rejected"
	KindRate     = "rate_limited"
	KindServer   = "server"
	KindBlocked  = "blocked_target"
	KindTimeout  = "timeout"
	KindNetwork  = "network"
	KindConfig   = "config"
	KindOther    = "other"
)

Error kinds reported in status; they never carry a message, URL or secret.

View Source
const (
	FieldReferrerHost = "referrer_host"
	FieldClientIP     = "client_ip"
	FieldUserAgent    = "user_agent"
	FieldVisitorID    = "visitor_id"
)

Optional record fields a sink may opt into. Everything else in Event is always exported; cookies, auth headers, query strings and bodies never exist.

View Source
const (
	PostHogHostUS = "https://us.i.posthog.com"
	PostHogHostEU = "https://eu.i.posthog.com"
)

PostHog cloud ingestion hosts.

View Source
const (
	SinkPostHog = "posthog"
	SinkOTLP    = "otlp"
	SinkWebhook = "webhook"
)

Sink kinds.

View Source
const (
	SecretURL        = "url"
	SecretToken      = "token"
	SecretAuthHeader = "auth_header"
)

Secret slots stored per sink. They are write-only through the API.

Variables

This section is empty.

Functions

func ErrorKind

func ErrorKind(err error) string

ErrorKind classifies a delivery error into a short stable label.

func HasIdentity

func HasIdentity(fields []string) bool

HasIdentity reports whether any of fields identifies a visitor.

func IsIdentityField

func IsIdentityField(f string) bool

IsIdentityField reports whether f can single out a visitor.

func OptionalFields

func OptionalFields() []string

OptionalFields lists every field a sink can opt into, sorted.

func SecretSlots

func SecretSlots(kind string) []string

SecretSlots returns the secret slots a kind uses.

func SinkKinds

func SinkKinds() []string

SinkKinds lists the supported sink kinds.

func TruncateIP

func TruncateIP(ip string) string

TruncateIP zeroes the last octet of an IPv4 address and the last 80 bits of an IPv6 address. An unparsable value yields "" so a raw string never leaks.

func ValidateFields

func ValidateFields(fields []string) error

ValidateFields rejects an unknown field name.

func ValidateSpec

func ValidateSpec(spec SinkSpec) error

ValidateSpec checks spec against the sink rules and the environment's transport policy.

func VisitorID

func VisitorID(secret []byte, day time.Time, ip, ua string) string

VisitorID returns an anonymous id for a client that is stable within one UTC day and unlinkable across days. secret is the server secret; an empty secret yields "" so an unsalted hash is never exported.

Types

type Event

type Event struct {
	Time          time.Time `json:"time"`
	App           string    `json:"app"`
	Host          string    `json:"host"`
	Method        string    `json:"method"`
	Path          string    `json:"path"`
	Status        int       `json:"status"`
	DurationMS    float64   `json:"duration_ms"`
	BytesIn       int64     `json:"bytes_in"`
	BytesOut      int64     `json:"bytes_out"`
	RequestID     string    `json:"request_id"`
	TraceID       string    `json:"trace_id,omitempty"`
	UpstreamError bool      `json:"upstream_error"`
	ColdStart     bool      `json:"cold_start"`
	VisitorID     string    `json:"visitor_id,omitempty"`
	ReferrerHost  string    `json:"referrer_host,omitempty"`
	UAFamily      string    `json:"ua_family,omitempty"`
	ClientIP      string    `json:"client_ip,omitempty"`
	UserAgent     string    `json:"user_agent,omitempty"`
}

Event is the vendor-neutral record of one finished edge request. It never carries a query string, cookie, auth header or body. ClientIP, UserAgent and VisitorID are set only for sinks that opt in (see Policy).

func NewEvent

func NewEvent(app string, rec ingress.AccessRecord) Event

NewEvent builds an Event from an access record and the app name the caller resolved for its route key.

type HTTPOptions

type HTTPOptions struct {
	Timeout      time.Duration
	AllowPrivate bool
	MaxBodyBytes int
	// Namespace prefixes vendor attribute names and the service name; the
	// caller passes the brand short name. Empty means "edge".
	Namespace string
}

HTTPOptions is the transport policy shared by every HTTP sink.

func HTTPOptionsFromEnv

func HTTPOptionsFromEnv() HTTPOptions

HTTPOptionsFromEnv reads the sink transport policy from the environment.

type OTLPConfig

type OTLPConfig struct {
	Endpoint string
	URL      string
	Headers  map[string]string
	HTTP     HTTPOptions
}

OTLPConfig configures the OTLP/HTTP logs sink. Endpoint is a base URL that gets /v1/logs appended; URL, when set, is used as is (for example https://us.i.posthog.com/i/v1/logs).

type OTLPSink

type OTLPSink struct {
	// contains filtered or unexported fields
}

OTLPSink posts one LogRecord per event as OTLP/HTTP protobuf.

func NewOTLPSink

func NewOTLPSink(cfg OTLPConfig) (*OTLPSink, error)

NewOTLPSink validates the target and returns the sink.

func (*OTLPSink) Name

func (s *OTLPSink) Name() string

Name identifies the sink in stats and logs.

func (*OTLPSink) Send

func (s *OTLPSink) Send(ctx context.Context, events []Event) error

Send posts one ExportLogsServiceRequest.

type Options

type Options struct {
	QueueSize     int
	BatchSize     int
	FlushInterval time.Duration
	MaxAttempts   int
	BackoffBase   time.Duration
	BackoffMax    time.Duration
}

Options tunes the per-sink queue, batching and retry behavior.

func OptionsFromEnv

func OptionsFromEnv() Options

OptionsFromEnv reads the pipeline tuning from the environment.

type Pipeline

type Pipeline struct {
	// contains filtered or unexported fields
}

Pipeline fans events out to per-sink bounded queues. Enqueue never blocks.

func NewPipeline

func NewPipeline(opts Options, sinks ...SinkConfig) *Pipeline

NewPipeline starts one delivery goroutine per sink.

func (*Pipeline) Close

func (p *Pipeline) Close(ctx context.Context) error

Close stops accepting events, flushes what is queued and waits for the workers. If ctx ends first, in-flight deliveries are cancelled.

func (*Pipeline) Enqueue

func (p *Pipeline) Enqueue(e Event)

Enqueue offers e to every sink and drops it for any sink whose queue is full or whose sampler rejects it.

func (*Pipeline) Stats

func (p *Pipeline) Stats() []SinkStats

Stats returns one entry per sink in the order they were added.

type Policy

type Policy struct {
	Fields     []string
	IPTruncate bool
}

Policy decides which optional fields one sink receives. The zero value exports no visitor identity and no referrer.

func (Policy) Apply

func (p Policy) Apply(e Event) Event

Apply returns a copy of e holding only what the policy allows. It scrubs first, so a field added to Event later is private by default.

type PostHogConfig

type PostHogConfig struct {
	Host  string
	Token string
	HTTP  HTTPOptions
}

PostHogConfig configures the PostHog events sink. Host is PostHogHostUS, PostHogHostEU or a self-hosted base URL.

type PostHogSink

type PostHogSink struct {
	// contains filtered or unexported fields
}

PostHogSink posts http_request events to PostHog's /batch endpoint.

func NewPostHogSink

func NewPostHogSink(cfg PostHogConfig) (*PostHogSink, error)

NewPostHogSink validates the host and returns the sink.

func (*PostHogSink) Name

func (s *PostHogSink) Name() string

Name identifies the sink in stats and logs.

func (*PostHogSink) Send

func (s *PostHogSink) Send(ctx context.Context, events []Event) error

Send posts the events, splitting into several requests when one would exceed the body cap.

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service owns one pipeline per enabled sink and taps the access log. Every failure stays inside it: a request is never slowed or failed by export.

func NewService

func NewService(src Source, httpOpts HTTPOptions, opts Options, resolve func(string) string, salt func(context.Context) ([]byte, error), logger *slog.Logger) *Service

NewService returns a Service. resolve maps a route key to an app name; salt returns the server secret used for visitor ids (nil disables them).

func (*Service) Close

func (s *Service) Close(ctx context.Context)

Close flushes and stops every pipeline.

func (*Service) IdentityWanted

func (s *Service) IdentityWanted() bool

IdentityWanted reports whether any active sink opted into identity or referrer data.

func (*Service) Install

func (s *Service) Install(log *ingress.AccessLog) func()

Install subscribes the service to log and returns the unsubscribe func.

func (*Service) Reload

func (s *Service) Reload(ctx context.Context) error

Reload rebuilds pipelines for changed sinks, keeps unchanged ones running and retires removed ones.

func (*Service) Status

func (s *Service) Status() []SinkStatus

Status returns one entry per stored sink, ordered by name then id.

func (*Service) Tap

func (s *Service) Tap(key string, rec ingress.AccessRecord)

Tap is the access-log subscriber. It never blocks and never panics out.

func (*Service) Test

func (s *Service) Test(ctx context.Context, spec SinkSpec) TestResult

Test sends one synthetic record through a fresh sink built from spec, shaped by the spec's own field policy. It does not touch the live pipelines.

type Sink

type Sink interface {
	Send(ctx context.Context, events []Event) error
	Name() string
}

Sink delivers a batch of events to one destination.

func BuildSink

func BuildSink(spec SinkSpec, o HTTPOptions) (Sink, error)

BuildSink validates spec and returns its sink.

type SinkConfig

type SinkConfig struct {
	Sink Sink
	// Sample is the fraction of traces exported, 0 to 1. Zero means none, so
	// callers wanting everything pass 1.
	Sample float64
	// Secrets are scrubbed from the sink's last_error.
	Secrets []string
}

SinkConfig attaches a sink to the pipeline.

type SinkSpec

type SinkSpec struct {
	ID         string
	Name       string
	Kind       string
	Enabled    bool
	Host       string
	Fields     []string
	IPTruncate bool
	Sample     float64
	Secrets    map[string]string
}

SinkSpec is one configured sink with its resolved secrets. Secrets must never be logged or returned to a client.

func (SinkSpec) Policy

func (s SinkSpec) Policy() Policy

Policy is the field policy of the spec.

type SinkStats

type SinkStats struct {
	Name        string    `json:"name"`
	Sent        uint64    `json:"sent"`
	Dropped     uint64    `json:"dropped"`
	Failed      uint64    `json:"failed"`
	SampledOut  uint64    `json:"sampled_out"`
	Queued      int       `json:"queued"`
	LastError   string    `json:"last_error,omitempty"`
	LastKind    string    `json:"last_error_kind,omitempty"`
	LastSuccess time.Time `json:"last_success"`
}

SinkStats is a point-in-time view of one sink.

type SinkStatus

type SinkStatus struct {
	ID            string    `json:"id"`
	Name          string    `json:"name"`
	Kind          string    `json:"kind"`
	Enabled       bool      `json:"enabled"`
	Active        bool      `json:"active"`
	LastSuccess   time.Time `json:"last_success"`
	LastErrorKind string    `json:"last_error_kind,omitempty"`
	Queued        int       `json:"queued"`
	Sent          uint64    `json:"sent"`
	Dropped       uint64    `json:"dropped"`
	Failed        uint64    `json:"failed"`
	SampledOut    uint64    `json:"sampled_out"`
}

SinkStatus is the live view of one stored sink.

type Source

type Source interface {
	LoadSpecs(ctx context.Context) ([]SinkSpec, error)
}

Source loads the stored sinks with their secrets resolved.

type StatusError

type StatusError struct {
	Sink string
	Code int
}

StatusError is a non-2xx vendor response.

func (*StatusError) Error

func (e *StatusError) Error() string

func (*StatusError) Retryable

func (e *StatusError) Retryable() bool

Retryable reports whether trying again could succeed.

type TestResult

type TestResult struct {
	OK         bool    `json:"ok"`
	ErrorKind  string  `json:"error_kind,omitempty"`
	DurationMS float64 `json:"duration_ms"`
}

TestResult is the outcome of one synthetic delivery.

type WebhookConfig

type WebhookConfig struct {
	URL     string
	Headers map[string]string
	HTTP    HTTPOptions
}

WebhookConfig configures a generic newline-delimited JSON sink.

type WebhookSink

type WebhookSink struct {
	// contains filtered or unexported fields
}

WebhookSink posts events as application/x-ndjson.

func NewWebhookSink

func NewWebhookSink(cfg WebhookConfig) (*WebhookSink, error)

NewWebhookSink validates the target and returns the sink.

func (*WebhookSink) Name

func (s *WebhookSink) Name() string

Name identifies the sink in stats and logs.

func (*WebhookSink) Send

func (s *WebhookSink) Send(ctx context.Context, events []Event) error

Send posts one ndjson body.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL