Help us shape the future of Go! Take the 2026 Go Developer Survey

receiver

package
v1.150.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 23, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package receiver serves POST /hooks/{source} (#1870): it authenticates each request, turns its body into events, buffers them per source, writes them to object storage as gzipped JSON-lines segments, and answers 202 only once the segment holding them is written.

A 2xx therefore means the events are in object storage. A replica that dies holding a buffer answered nobody for those events, so their senders retry, and backpressure is the sender's own retry queue: a full buffer or a failed write is answered 503 with Retry-After, and the platform keeps no backlog of its own beyond each source's buffer limit.

A receiver given a spool (#2121) answers 202 once the events are synced to its own disk instead, so a sender's wait is local disk latency rather than object storage's, and drains them to object storage behind the answer; see spool.go.

Index

Constants

View Source
const (
	OutcomeAccepted      = "accepted"
	OutcomeUnauthorized  = "unauthorized"
	OutcomeTooLarge      = "too_large"
	OutcomeRateLimited   = "rate_limited"
	OutcomeBufferFull    = "buffer_full"
	OutcomeWriteFailed   = "write_failed"
	OutcomeUnknownSource = "unknown_source"
	OutcomeInvalidBody   = "invalid_body"
	// OutcomeSenderGone is a delivery whose sender disconnected before its
	// segment was written. Unlike write_failed, its write goes on and may
	// well store the events; a log line says how it ended.
	OutcomeSenderGone = "sender_gone"
)

Outcomes a request is counted under.

View Source
const (
	DefaultWriteTimeout = 30 * time.Second
	DefaultRefresh      = 5 * time.Second
	DefaultStatsFlush   = 10 * time.Second
)

Defaults for the receiver's own timing.

View Source
const DefaultMissRefresh = time.Second

DefaultMissRefresh is the least time between two re-reads of the sources made for requests naming a source this replica does not serve. A caller inventing names costs the database at most one read per interval.

View Source
const DefaultRegisterRetry = 15 * time.Second

DefaultRegisterRetry is how often a window whose raw partition could not be registered is tried again.

View Source
const OutcomeSpoolFull = "spool_full"

OutcomeSpoolFull is a request refused because the spool is at its bound.

View Source
const PathPrefix = "/hooks/"

PathPrefix is where the receiver is mounted.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config

type Config struct {
	Sources    SourceLister
	Objects    ObjectWriter
	Bucket     string
	Recorder   Recorder
	RawWindows RawWindows
	Metrics    Metrics
	// Replica is this process's identity, the value of X-Platform-Instance.
	Replica string
	Logger  *slog.Logger
	// WriteTimeout bounds one segment write, including its retries by the
	// S3 client. A write that has not finished by then is answered 503.
	WriteTimeout time.Duration
	// Refresh is how often the source list is re-read, which is how a
	// change made on another replica reaches this one.
	Refresh    time.Duration
	StatsFlush time.Duration
	// RegisterRetry is how often a window whose raw partition could not be
	// registered is tried again.
	RegisterRetry time.Duration
	// MissRefresh is the least time between two re-reads of the sources made
	// for requests naming a source this replica does not serve.
	MissRefresh time.Duration
	// Spool, when set, is the receiver's local spool (#2121): a request is
	// answered once its events are synced to it, and drained to object
	// storage behind the answer. Nil answers once they are in object storage.
	Spool       Spool
	SpoolLimits SpoolLimits
	Now         func() time.Time
}

Config is what a Receiver is built from.

type Metrics

type Metrics interface {
	WebhookRequest(ctx context.Context, source, outcome string)
	WebhookEvents(ctx context.Context, source string, n int)
	WebhookSegmentWritten(ctx context.Context, source string)
	WebhookAck(ctx context.Context, source string, d time.Duration)
	// WebhookBuffer reports how many events a source holds in memory: those
	// waiting to be written and those in a write under way.
	WebhookBuffer(ctx context.Context, source string, events int)
	// WebhookSpool reports what the receiver's spool holds and how old its
	// oldest event waiting is (#2121).
	WebhookSpool(ctx context.Context, bytes, events int64, files int, oldestAge time.Duration)
	// WebhookSpoolDrained counts events drained from the spool to object
	// storage.
	WebhookSpoolDrained(ctx context.Context, source string, events int)
	// WebhookSpoolDrainFailed counts a drain of the spool that failed.
	WebhookSpoolDrainFailed(ctx context.Context, source string)
	// WebhookResponse counts a request by the status it was answered with,
	// WebhookWritten a segment's events and bytes written to object
	// storage, and WebhookAckStages an acknowledged delivery's wait in each
	// stage (#2150).
	WebhookResponse(ctx context.Context, source string, status int)
	WebhookWritten(ctx context.Context, source string, events, bytes int)
	WebhookAckStages(ctx context.Context, source string, buffer, mark, put time.Duration)
}

Metrics is what the receiver reports to the platform's metrics registry. A nil Metrics reports nothing.

type ObjectWriter

type ObjectWriter interface {
	PutObject(ctx context.Context, bucket, key string, data []byte, contentType string) error
}

ObjectWriter writes one object. The managed-resources S3 client satisfies it.

type RawWindows

type RawWindows interface {
	EnsureRawWindow(ctx context.Context, src whsource.Source, start time.Time) error
}

RawWindows registers a new window's raw partition, so an event is queryable moments after it is acknowledged. The receiver calls it from its own loop, never while a sender waits. A nil RawWindows leaves new windows to the compactor's sync.

type Receiver

type Receiver struct {
	// contains filtered or unexported fields
}

Receiver serves the webhook routes.

func New

func New(cfg Config) *Receiver

New builds a receiver. Start begins serving.

func (*Receiver) Reload

func (r *Receiver) Reload()

Reload re-reads the sources now rather than at the next refresh: it is what an administrator's change on this replica calls.

func (*Receiver) ServeHTTP

func (r *Receiver) ServeHTTP(w http.ResponseWriter, req *http.Request)

ServeHTTP handles /hooks/{source} and /hooks/{source}/{token}.

func (*Receiver) Start

func (r *Receiver) Start(ctx context.Context)

Start loads the sources and begins refreshing them and writing counts.

func (*Receiver) Stop

func (r *Receiver) Stop()

Stop writes every pending buffer, answering the requests waiting on them, then the counts, and releases the limiters.

type Recorder

type Recorder interface {
	MarkSegment(ctx context.Context, source string, start time.Time, length time.Duration, bytes int64) error
	RecordCounts(ctx context.Context, counts []whstore.Count) error
	RecordRejections(ctx context.Context, rejections []whstore.Rejection) error
	RecordLatency(ctx context.Context, latencies []whstore.Latency) error
	// RecordSpool writes the receiver's spool state; called only when the
	// receiver has a spool.
	RecordSpool(ctx context.Context, st whstore.SpoolState) error
	// RecordReplicaMinutes adds the receiver's minutes: events accepted and
	// written, and what its spool held (#2150).
	RecordReplicaMinutes(ctx context.Context, minutes []whstore.ReplicaMinute) error
}

Recorder is the control data the receiver writes. whstore.Store satisfies it.

type SourceLister

type SourceLister interface {
	List(ctx context.Context) ([]whsource.Source, error)
}

SourceLister lists every source. whsource.Store satisfies it.

type Spool added in v1.146.0

type Spool interface {
	Dir() string
	Append(source string, rec whspool.Record) (took time.Duration, err error)
	Active(source string) (whspool.Stats, bool)
	Seal(source string) error
	Sealed(source string) []whspool.Segment
	Sources() []string
	Read(seg whspool.Segment) ([]whspool.Record, error)
	Remove(seg whspool.Segment) error
	Stats() whspool.Stats
}

Spool is the receiver's local spool. whspool.Spool satisfies it.

type SpoolLimits added in v1.146.0

type SpoolLimits struct {
	MaxBytes int64
	MaxAge   time.Duration
}

SpoolLimits are the bounds a spool was opened with, for its reported state.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL