receiver

package
v1.147.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 17, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package receiver serves POST /hooks/{source} (#1870): it authenticates each request, turns its body into events, buffers them per source, writes them to object storage as gzipped JSON-lines segments, and answers 202 only once the segment holding them is written.

A 2xx therefore means the events are in object storage. A replica that dies holding a buffer answered nobody for those events, so their senders retry, and backpressure is the sender's own retry queue: a full buffer or a failed write is answered 503 with Retry-After, and the platform keeps no backlog of its own beyond each source's buffer limit.

A receiver given a spool (#2121) answers 202 once the events are synced to its own disk instead, so a sender's wait is local disk latency rather than object storage's, and drains them to object storage behind the answer; see spool.go.

Index

Constants

View Source
const (
	OutcomeAccepted      = "accepted"
	OutcomeUnauthorized  = "unauthorized"
	OutcomeTooLarge      = "too_large"
	OutcomeRateLimited   = "rate_limited"
	OutcomeBufferFull    = "buffer_full"
	OutcomeWriteFailed   = "write_failed"
	OutcomeUnknownSource = "unknown_source"
	OutcomeInvalidBody   = "invalid_body"
	// OutcomeSenderGone is a delivery whose sender disconnected before its
	// segment was written. Unlike write_failed, its write goes on and may
	// well store the events; a log line says how it ended.
	OutcomeSenderGone = "sender_gone"
)

Outcomes a request is counted under.

View Source
const (
	DefaultWriteTimeout = 30 * time.Second
	DefaultRefresh      = 5 * time.Second
	DefaultStatsFlush   = 10 * time.Second
)

Defaults for the receiver's own timing.

View Source
const DefaultMissRefresh = time.Second

DefaultMissRefresh is the least time between two re-reads of the sources made for requests naming a source this replica does not serve. A caller inventing names costs the database at most one read per interval.

View Source
const DefaultRegisterRetry = 15 * time.Second

DefaultRegisterRetry is how often a window whose raw partition could not be registered is tried again.

View Source
const OutcomeSpoolFull = "spool_full"

OutcomeSpoolFull is a request refused because the spool is at its bound.

View Source
const PathPrefix = "/hooks/"

PathPrefix is where the receiver is mounted.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config

type Config struct {
	Sources    SourceLister
	Objects    ObjectWriter
	Bucket     string
	Recorder   Recorder
	RawWindows RawWindows
	Metrics    Metrics
	// Replica is this process's identity, the value of X-Platform-Instance.
	Replica string
	Logger  *slog.Logger
	// WriteTimeout bounds one segment write, including its retries by the
	// S3 client. A write that has not finished by then is answered 503.
	WriteTimeout time.Duration
	// Refresh is how often the source list is re-read, which is how a
	// change made on another replica reaches this one.
	Refresh    time.Duration
	StatsFlush time.Duration
	// RegisterRetry is how often a window whose raw partition could not be
	// registered is tried again.
	RegisterRetry time.Duration
	// MissRefresh is the least time between two re-reads of the sources made
	// for requests naming a source this replica does not serve.
	MissRefresh time.Duration
	// Spool, when set, is the receiver's local spool (#2121): a request is
	// answered once its events are synced to it, and drained to object
	// storage behind the answer. Nil answers once they are in object storage.
	Spool       Spool
	SpoolLimits SpoolLimits
	Now         func() time.Time
}

Config is what a Receiver is built from.

type Metrics

type Metrics interface {
	WebhookRequest(ctx context.Context, source, outcome string)
	WebhookEvents(ctx context.Context, source string, n int)
	WebhookSegmentWritten(ctx context.Context, source string)
	WebhookAck(ctx context.Context, source string, d time.Duration)
	// WebhookBuffer reports how many events a source holds in memory: those
	// waiting to be written and those in a write under way.
	WebhookBuffer(ctx context.Context, source string, events int)
	// WebhookSpool reports what the receiver's spool holds and how old its
	// oldest event waiting is (#2121).
	WebhookSpool(ctx context.Context, bytes, events int64, files int, oldestAge time.Duration)
	// WebhookSpoolDrained counts events drained from the spool to object
	// storage.
	WebhookSpoolDrained(ctx context.Context, source string, events int)
	// WebhookSpoolDrainFailed counts a drain of the spool that failed.
	WebhookSpoolDrainFailed(ctx context.Context, source string)
	// WebhookResponse counts a request by the status it was answered with,
	// WebhookWritten a segment's events and bytes written to object
	// storage, and WebhookAckStages an acknowledged delivery's wait in each
	// stage (#2150).
	WebhookResponse(ctx context.Context, source string, status int)
	WebhookWritten(ctx context.Context, source string, events, bytes int)
	WebhookAckStages(ctx context.Context, source string, buffer, mark, put time.Duration)
}

Metrics is what the receiver reports to the platform's metrics registry. A nil Metrics reports nothing.

type ObjectWriter

type ObjectWriter interface {
	PutObject(ctx context.Context, bucket, key string, data []byte, contentType string) error
}

ObjectWriter writes one object. The managed-resources S3 client satisfies it.

type RawWindows

type RawWindows interface {
	EnsureRawWindow(ctx context.Context, src whsource.Source, start time.Time) error
}

RawWindows registers a new window's raw partition, so an event is queryable moments after it is acknowledged. The receiver calls it from its own loop, never while a sender waits. A nil RawWindows leaves new windows to the compactor's sync.

type Receiver

type Receiver struct {
	// contains filtered or unexported fields
}

Receiver serves the webhook routes.

func New

func New(cfg Config) *Receiver

New builds a receiver. Start begins serving.

func (*Receiver) Reload

func (r *Receiver) Reload()

Reload re-reads the sources now rather than at the next refresh: it is what an administrator's change on this replica calls.

func (*Receiver) ServeHTTP

func (r *Receiver) ServeHTTP(w http.ResponseWriter, req *http.Request)

ServeHTTP handles /hooks/{source} and /hooks/{source}/{token}.

func (*Receiver) Start

func (r *Receiver) Start(ctx context.Context)

Start loads the sources and begins refreshing them and writing counts.

func (*Receiver) Stop

func (r *Receiver) Stop()

Stop writes every pending buffer, answering the requests waiting on them, then the counts, and releases the limiters.

type Recorder

type Recorder interface {
	MarkSegment(ctx context.Context, source string, start time.Time, length time.Duration, bytes int64) error
	RecordCounts(ctx context.Context, counts []whstore.Count) error
	RecordRejections(ctx context.Context, rejections []whstore.Rejection) error
	RecordLatency(ctx context.Context, latencies []whstore.Latency) error
	// RecordSpool writes the receiver's spool state; called only when the
	// receiver has a spool.
	RecordSpool(ctx context.Context, st whstore.SpoolState) error
	// RecordReplicaMinutes adds the receiver's minutes: events accepted and
	// written, and what its spool held (#2150).
	RecordReplicaMinutes(ctx context.Context, minutes []whstore.ReplicaMinute) error
}

Recorder is the control data the receiver writes. whstore.Store satisfies it.

type SourceLister

type SourceLister interface {
	List(ctx context.Context) ([]whsource.Source, error)
}

SourceLister lists every source. whsource.Store satisfies it.

type Spool added in v1.146.0

type Spool interface {
	Dir() string
	Append(source string, rec whspool.Record) (took time.Duration, err error)
	Active(source string) (whspool.Stats, bool)
	Seal(source string) error
	Sealed(source string) []whspool.Segment
	Sources() []string
	Read(seg whspool.Segment) ([]whspool.Record, error)
	Remove(seg whspool.Segment) error
	Stats() whspool.Stats
}

Spool is the receiver's local spool. whspool.Spool satisfies it.

type SpoolLimits added in v1.146.0

type SpoolLimits struct {
	MaxBytes int64
	MaxAge   time.Duration
}

SpoolLimits are the bounds a spool was opened with, for its reported state.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL