Documentation
¶
Overview ¶
Package trafficexport delivers finished edge requests to analytics vendors through bounded, lossy queues so a slow vendor never slows a real request.
Index ¶
- Constants
- func ErrorKind(err error) string
- func HasIdentity(fields []string) bool
- func IsIdentityField(f string) bool
- func OptionalFields() []string
- func SecretSlots(kind string) []string
- func SinkKinds() []string
- func TruncateIP(ip string) string
- func ValidateFields(fields []string) error
- func ValidateSpec(spec SinkSpec) error
- func VisitorID(secret []byte, day time.Time, ip, ua string) string
- type Event
- type HTTPOptions
- type OTLPConfig
- type OTLPSink
- type Options
- type Pipeline
- type Policy
- type PostHogConfig
- type PostHogSink
- type Service
- func (s *Service) Close(ctx context.Context)
- func (s *Service) IdentityWanted() bool
- func (s *Service) Install(log *ingress.AccessLog) func()
- func (s *Service) Reload(ctx context.Context) error
- func (s *Service) Status() []SinkStatus
- func (s *Service) Tap(key string, rec ingress.AccessRecord)
- func (s *Service) Test(ctx context.Context, spec SinkSpec) TestResult
- type Sink
- type SinkConfig
- type SinkSpec
- type SinkStats
- type SinkStatus
- type Source
- type StatusError
- type TestResult
- type WebhookConfig
- type WebhookSink
Constants ¶
const ( EnvTimeout = "APP_TRAFFIC_EXPORT_TIMEOUT" // HTTP timeout per delivery, default 10s EnvAllowPrivate = "APP_TRAFFIC_EXPORT_ALLOW_PRIVATE" // 1 allows loopback and link-local targets EnvMaxBodyBytes = "APP_TRAFFIC_EXPORT_MAX_BODY_BYTES" // largest request body, default 20 MiB EnvQueueSize = "APP_TRAFFIC_EXPORT_QUEUE_SIZE" // events buffered per sink, default 10000 EnvBatchSize = "APP_TRAFFIC_EXPORT_BATCH_SIZE" // events per delivery, default 500 EnvFlushInterval = "APP_TRAFFIC_EXPORT_FLUSH_INTERVAL" // longest wait for a full batch, default 5s EnvMaxAttempts = "APP_TRAFFIC_EXPORT_MAX_ATTEMPTS" // deliveries tried per batch, default 5 EnvBackoffBase = "APP_TRAFFIC_EXPORT_BACKOFF_BASE" // first retry delay, doubles each try, default 500ms EnvBackoffMax = "APP_TRAFFIC_EXPORT_BACKOFF_MAX" // retry delay ceiling, default 30s )
Environment variable names read by this package; invalid values fall back to defaults.
const ( KindAuth = "auth" KindRejected = "rejected" KindRate = "rate_limited" KindServer = "server" KindBlocked = "blocked_target" KindTimeout = "timeout" KindNetwork = "network" KindConfig = "config" KindOther = "other" )
Error kinds reported in status; they never carry a message, URL or secret.
const ( FieldReferrerHost = "referrer_host" FieldClientIP = "client_ip" FieldUserAgent = "user_agent" FieldVisitorID = "visitor_id" )
Optional record fields a sink may opt into. Everything else in Event is always exported; cookies, auth headers, query strings and bodies never exist.
const ( PostHogHostUS = "https://us.i.posthog.com" PostHogHostEU = "https://eu.i.posthog.com" )
PostHog cloud ingestion hosts.
const ( SinkPostHog = "posthog" SinkOTLP = "otlp" SinkWebhook = "webhook" )
Sink kinds.
const ( SecretURL = "url" SecretToken = "token" SecretAuthHeader = "auth_header" )
Secret slots stored per sink. They are write-only through the API.
Variables ¶
This section is empty.
Functions ¶
func HasIdentity ¶
HasIdentity reports whether any of fields identifies a visitor.
func IsIdentityField ¶
IsIdentityField reports whether f can single out a visitor.
func OptionalFields ¶
func OptionalFields() []string
OptionalFields lists every field a sink can opt into, sorted.
func SecretSlots ¶
SecretSlots returns the secret slots a kind uses.
func TruncateIP ¶
TruncateIP zeroes the last octet of an IPv4 address and the last 80 bits of an IPv6 address. An unparsable value yields "" so a raw string never leaks.
func ValidateFields ¶
ValidateFields rejects an unknown field name.
func ValidateSpec ¶
ValidateSpec checks spec against the sink rules and the environment's transport policy.
Types ¶
type Event ¶
type Event struct {
Time time.Time `json:"time"`
App string `json:"app"`
Host string `json:"host"`
Method string `json:"method"`
Path string `json:"path"`
Status int `json:"status"`
DurationMS float64 `json:"duration_ms"`
BytesIn int64 `json:"bytes_in"`
BytesOut int64 `json:"bytes_out"`
RequestID string `json:"request_id"`
TraceID string `json:"trace_id,omitempty"`
UpstreamError bool `json:"upstream_error"`
ColdStart bool `json:"cold_start"`
VisitorID string `json:"visitor_id,omitempty"`
ReferrerHost string `json:"referrer_host,omitempty"`
UAFamily string `json:"ua_family,omitempty"`
ClientIP string `json:"client_ip,omitempty"`
UserAgent string `json:"user_agent,omitempty"`
}
Event is the vendor-neutral record of one finished edge request. It never carries a query string, cookie, auth header or body. ClientIP, UserAgent and VisitorID are set only for sinks that opt in (see Policy).
type HTTPOptions ¶
type HTTPOptions struct {
Timeout time.Duration
AllowPrivate bool
MaxBodyBytes int
// Namespace prefixes vendor attribute names and the service name; the
// caller passes the brand short name. Empty means "edge".
Namespace string
}
HTTPOptions is the transport policy shared by every HTTP sink.
func HTTPOptionsFromEnv ¶
func HTTPOptionsFromEnv() HTTPOptions
HTTPOptionsFromEnv reads the sink transport policy from the environment.
type OTLPConfig ¶
type OTLPConfig struct {
Endpoint string
URL string
Headers map[string]string
HTTP HTTPOptions
}
OTLPConfig configures the OTLP/HTTP logs sink. Endpoint is a base URL that gets /v1/logs appended; URL, when set, is used as is (for example https://us.i.posthog.com/i/v1/logs).
type OTLPSink ¶
type OTLPSink struct {
// contains filtered or unexported fields
}
OTLPSink posts one LogRecord per event as OTLP/HTTP protobuf.
func NewOTLPSink ¶
func NewOTLPSink(cfg OTLPConfig) (*OTLPSink, error)
NewOTLPSink validates the target and returns the sink.
type Options ¶
type Options struct {
QueueSize int
BatchSize int
FlushInterval time.Duration
MaxAttempts int
BackoffBase time.Duration
BackoffMax time.Duration
}
Options tunes the per-sink queue, batching and retry behavior.
func OptionsFromEnv ¶
func OptionsFromEnv() Options
OptionsFromEnv reads the pipeline tuning from the environment.
type Pipeline ¶
type Pipeline struct {
// contains filtered or unexported fields
}
Pipeline fans events out to per-sink bounded queues. Enqueue never blocks.
func NewPipeline ¶
func NewPipeline(opts Options, sinks ...SinkConfig) *Pipeline
NewPipeline starts one delivery goroutine per sink.
func (*Pipeline) Close ¶
Close stops accepting events, flushes what is queued and waits for the workers. If ctx ends first, in-flight deliveries are cancelled.
type Policy ¶
Policy decides which optional fields one sink receives. The zero value exports no visitor identity and no referrer.
type PostHogConfig ¶
type PostHogConfig struct {
Host string
Token string
HTTP HTTPOptions
}
PostHogConfig configures the PostHog events sink. Host is PostHogHostUS, PostHogHostEU or a self-hosted base URL.
type PostHogSink ¶
type PostHogSink struct {
// contains filtered or unexported fields
}
PostHogSink posts http_request events to PostHog's /batch endpoint.
func NewPostHogSink ¶
func NewPostHogSink(cfg PostHogConfig) (*PostHogSink, error)
NewPostHogSink validates the host and returns the sink.
func (*PostHogSink) Name ¶
func (s *PostHogSink) Name() string
Name identifies the sink in stats and logs.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service owns one pipeline per enabled sink and taps the access log. Every failure stays inside it: a request is never slowed or failed by export.
func NewService ¶
func NewService(src Source, httpOpts HTTPOptions, opts Options, resolve func(string) string, salt func(context.Context) ([]byte, error), logger *slog.Logger) *Service
NewService returns a Service. resolve maps a route key to an app name; salt returns the server secret used for visitor ids (nil disables them).
func (*Service) IdentityWanted ¶
IdentityWanted reports whether any active sink opted into identity or referrer data.
func (*Service) Reload ¶
Reload rebuilds pipelines for changed sinks, keeps unchanged ones running and retires removed ones.
func (*Service) Status ¶
func (s *Service) Status() []SinkStatus
Status returns one entry per stored sink, ordered by name then id.
type SinkConfig ¶
type SinkConfig struct {
Sink Sink
// Sample is the fraction of traces exported, 0 to 1. Zero means none, so
// callers wanting everything pass 1.
Sample float64
// Secrets are scrubbed from the sink's last_error.
Secrets []string
}
SinkConfig attaches a sink to the pipeline.
type SinkSpec ¶
type SinkSpec struct {
ID string
Name string
Kind string
Enabled bool
Host string
Fields []string
IPTruncate bool
Sample float64
Secrets map[string]string
}
SinkSpec is one configured sink with its resolved secrets. Secrets must never be logged or returned to a client.
type SinkStats ¶
type SinkStats struct {
Name string `json:"name"`
Sent uint64 `json:"sent"`
Dropped uint64 `json:"dropped"`
Failed uint64 `json:"failed"`
SampledOut uint64 `json:"sampled_out"`
Queued int `json:"queued"`
LastError string `json:"last_error,omitempty"`
LastKind string `json:"last_error_kind,omitempty"`
LastSuccess time.Time `json:"last_success"`
}
SinkStats is a point-in-time view of one sink.
type SinkStatus ¶
type SinkStatus struct {
ID string `json:"id"`
Name string `json:"name"`
Kind string `json:"kind"`
Enabled bool `json:"enabled"`
Active bool `json:"active"`
LastSuccess time.Time `json:"last_success"`
LastErrorKind string `json:"last_error_kind,omitempty"`
Queued int `json:"queued"`
Sent uint64 `json:"sent"`
Dropped uint64 `json:"dropped"`
Failed uint64 `json:"failed"`
SampledOut uint64 `json:"sampled_out"`
}
SinkStatus is the live view of one stored sink.
type StatusError ¶
StatusError is a non-2xx vendor response.
func (*StatusError) Error ¶
func (e *StatusError) Error() string
func (*StatusError) Retryable ¶
func (e *StatusError) Retryable() bool
Retryable reports whether trying again could succeed.
type TestResult ¶
type TestResult struct {
OK bool `json:"ok"`
ErrorKind string `json:"error_kind,omitempty"`
DurationMS float64 `json:"duration_ms"`
}
TestResult is the outcome of one synthetic delivery.
type WebhookConfig ¶
type WebhookConfig struct {
URL string
Headers map[string]string
HTTP HTTPOptions
}
WebhookConfig configures a generic newline-delimited JSON sink.
type WebhookSink ¶
type WebhookSink struct {
// contains filtered or unexported fields
}
WebhookSink posts events as application/x-ndjson.
func NewWebhookSink ¶
func NewWebhookSink(cfg WebhookConfig) (*WebhookSink, error)
NewWebhookSink validates the target and returns the sink.
func (*WebhookSink) Name ¶
func (s *WebhookSink) Name() string
Name identifies the sink in stats and logs.