Documentation
¶
Overview ¶
Package receiver serves POST /hooks/{source} (#1870): it authenticates each request, turns its body into events, buffers them per source, writes them to object storage as gzipped JSON-lines segments, and answers 202 only once the segment holding them is written.
A 2xx therefore means the events are in object storage. A replica that dies holding a buffer answered nobody for those events, so their senders retry, and backpressure is the sender's own retry queue: a full buffer or a failed write is answered 503 with Retry-After, and the platform keeps no backlog of its own beyond each source's buffer limit.
A receiver given a spool (#2121) answers 202 once the events are synced to its own disk instead, so a sender's wait is local disk latency rather than object storage's, and drains them to object storage behind the answer; see spool.go.
Index ¶
Constants ¶
const ( OutcomeAccepted = "accepted" OutcomeTooLarge = "too_large" OutcomeRateLimited = "rate_limited" OutcomeBufferFull = "buffer_full" OutcomeWriteFailed = "write_failed" OutcomeUnknownSource = "unknown_source" OutcomeInvalidBody = "invalid_body" // OutcomeSenderGone is a delivery whose sender disconnected before its // segment was written. Unlike write_failed, its write goes on and may // well store the events; a log line says how it ended. OutcomeSenderGone = "sender_gone" )
Outcomes a request is counted under.
const ( DefaultWriteTimeout = 30 * time.Second DefaultRefresh = 5 * time.Second DefaultStatsFlush = 10 * time.Second )
Defaults for the receiver's own timing.
const DefaultMissRefresh = time.Second
DefaultMissRefresh is the least time between two re-reads of the sources made for requests naming a source this replica does not serve. A caller inventing names costs the database at most one read per interval.
const DefaultRegisterRetry = 15 * time.Second
DefaultRegisterRetry is how often a window whose raw partition could not be registered is tried again.
const OutcomeSpoolFull = "spool_full"
OutcomeSpoolFull is a request refused because the spool is at its bound.
const PathPrefix = "/hooks/"
PathPrefix is where the receiver is mounted.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
Sources SourceLister
Objects ObjectWriter
Bucket string
Recorder Recorder
RawWindows RawWindows
Metrics Metrics
// Replica is this process's identity, the value of X-Platform-Instance.
Replica string
Logger *slog.Logger
// WriteTimeout bounds one segment write, including its retries by the
// S3 client. A write that has not finished by then is answered 503.
WriteTimeout time.Duration
// Refresh is how often the source list is re-read, which is how a
// change made on another replica reaches this one.
Refresh time.Duration
StatsFlush time.Duration
// RegisterRetry is how often a window whose raw partition could not be
// registered is tried again.
RegisterRetry time.Duration
// MissRefresh is the least time between two re-reads of the sources made
// for requests naming a source this replica does not serve.
MissRefresh time.Duration
// Spool, when set, is the receiver's local spool (#2121): a request is
// answered once its events are synced to it, and drained to object
// storage behind the answer. Nil answers once they are in object storage.
Spool Spool
SpoolLimits SpoolLimits
Now func() time.Time
}
Config is what a Receiver is built from.
type Metrics ¶
type Metrics interface {
WebhookRequest(ctx context.Context, source, outcome string)
WebhookEvents(ctx context.Context, source string, n int)
WebhookSegmentWritten(ctx context.Context, source string)
WebhookAck(ctx context.Context, source string, d time.Duration)
// WebhookBuffer reports how many events a source holds in memory: those
// waiting to be written and those in a write under way.
WebhookBuffer(ctx context.Context, source string, events int)
// WebhookSpool reports what the receiver's spool holds and how old its
// oldest event waiting is (#2121).
WebhookSpool(ctx context.Context, bytes, events int64, files int, oldestAge time.Duration)
// WebhookSpoolDrained counts events drained from the spool to object
// storage.
WebhookSpoolDrained(ctx context.Context, source string, events int)
// WebhookSpoolDrainFailed counts a drain of the spool that failed.
WebhookSpoolDrainFailed(ctx context.Context, source string)
// WebhookResponse counts a request by the status it was answered with,
// WebhookWritten a segment's events and bytes written to object
// storage, and WebhookAckStages an acknowledged delivery's wait in each
// stage (#2150).
WebhookResponse(ctx context.Context, source string, status int)
WebhookWritten(ctx context.Context, source string, events, bytes int)
WebhookAckStages(ctx context.Context, source string, buffer, mark, put time.Duration)
}
Metrics is what the receiver reports to the platform's metrics registry. A nil Metrics reports nothing.
type ObjectWriter ¶
type ObjectWriter interface {
PutObject(ctx context.Context, bucket, key string, data []byte, contentType string) error
}
ObjectWriter writes one object. The managed-resources S3 client satisfies it.
type RawWindows ¶
type RawWindows interface {
EnsureRawWindow(ctx context.Context, src whsource.Source, start time.Time) error
}
RawWindows registers a new window's raw partition, so an event is queryable moments after it is acknowledged. The receiver calls it from its own loop, never while a sender waits. A nil RawWindows leaves new windows to the compactor's sync.
type Receiver ¶
type Receiver struct {
// contains filtered or unexported fields
}
Receiver serves the webhook routes.
func (*Receiver) Reload ¶
func (r *Receiver) Reload()
Reload re-reads the sources now rather than at the next refresh: it is what an administrator's change on this replica calls.
func (*Receiver) ServeHTTP ¶
func (r *Receiver) ServeHTTP(w http.ResponseWriter, req *http.Request)
ServeHTTP handles /hooks/{source} and /hooks/{source}/{token}.
type Recorder ¶
type Recorder interface {
MarkSegment(ctx context.Context, source string, start time.Time, length time.Duration, bytes int64) error
RecordCounts(ctx context.Context, counts []whstore.Count) error
RecordRejections(ctx context.Context, rejections []whstore.Rejection) error
RecordLatency(ctx context.Context, latencies []whstore.Latency) error
// RecordSpool writes the receiver's spool state; called only when the
// receiver has a spool.
RecordSpool(ctx context.Context, st whstore.SpoolState) error
// RecordReplicaMinutes adds the receiver's minutes: events accepted and
// written, and what its spool held (#2150).
RecordReplicaMinutes(ctx context.Context, minutes []whstore.ReplicaMinute) error
}
Recorder is the control data the receiver writes. whstore.Store satisfies it.
type SourceLister ¶
SourceLister lists every source. whsource.Store satisfies it.
type Spool ¶ added in v1.146.0
type Spool interface {
Dir() string
Append(source string, rec whspool.Record) (took time.Duration, err error)
Active(source string) (whspool.Stats, bool)
Seal(source string) error
Sealed(source string) []whspool.Segment
Sources() []string
Read(seg whspool.Segment) ([]whspool.Record, error)
Remove(seg whspool.Segment) error
Stats() whspool.Stats
}
Spool is the receiver's local spool. whspool.Spool satisfies it.
type SpoolLimits ¶ added in v1.146.0
SpoolLimits are the bounds a spool was opened with, for its reported state.