ingest

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 24, 2026 License: AGPL-3.0 Imports: 34 Imported by: 0

Documentation

Overview

Package ingest receives provider events: the webhook endpoint (POST /hooks/{adapter}/{token}) verifies the signature, stores the raw payload and answers immediately; a River job normalizes it later. SMTP bounces arrive through the bounce mailbox poller and take the same path.

Index

Constants

View Source
const MaxBody = 5 << 20

MaxBody is the largest webhook payload accepted.

View Source
const RetentionDays = 90

RetentionDays is how long raw payloads and the event log are kept.

Variables

This section is empty.

Functions

func RegisterLog

func RegisterLog(h huma.API, e *EventLog)

RegisterLog adds the event log operations.

Types

type BackfillWorker

type BackfillWorker struct {
	river.WorkerDefaults[manage.BackfillArgs]
	N *Normalizer
}

BackfillWorker pulls message status from a provider's API for messages that got no webhook event (manage.BackfillArgs).

func (*BackfillWorker) Timeout

Timeout bounds one backfill.

func (*BackfillWorker) Work

Work implements river.Worker.

type EventLog

type EventLog struct {
	Pool  *pgxpool.Pool
	Q     *store.Queries
	Audit *audit.Logger
	Now   func() time.Time
}

EventLog serves the provider event log.

func (*EventLog) Export

func (e *EventLog) Export(ctx context.Context, ws *store.Workspace, f LogFilter) ([]byte, error)

Export returns the filtered log as CSV (up to 50,000 rows). Recipients are shown in full only to members who may export contacts.

func (*EventLog) List

List returns one page of provider events, newest first.

type Handler

type Handler struct {
	Pool    *pgxpool.Pool
	Q       *store.Queries
	Factory *manage.Factory
	Jobs    func() *jobs.Client
	Log     *slog.Logger
	Now     func() time.Time
}

Handler serves the webhook endpoint.

func (*Handler) Routes

func (h *Handler) Routes(r chi.Router)

Routes mounts the endpoint.

func (*Handler) ServeHTTP

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP verifies, stores and acknowledges one webhook request.

func (*Handler) Validate

func (h *Handler) Validate(w http.ResponseWriter, r *http.Request)

Validate answers the CloudEvents webhook validation handshake (an OPTIONS request with WebHook-Request-Origin, as Azure Event Grid sends before it delivers CloudEvents): only a known webhook URL is confirmed.

type LogFilter

type LogFilter struct {
	Provider string
	Type     string
	Query    string
	Page     int
	PageSize int
}

LogFilter filters the event log.

type MailboxArgs

type MailboxArgs struct{}

MailboxArgs polls the bounce mailboxes of SMTP providers.

func (MailboxArgs) InsertOpts

func (MailboxArgs) InsertOpts() river.InsertOpts

InsertOpts implements river.JobArgsWithInsertOpts.

func (MailboxArgs) Kind

func (MailboxArgs) Kind() string

Kind implements river.JobArgs.

type MailboxWorker

type MailboxWorker struct {
	river.WorkerDefaults[MailboxArgs]
	N    *Normalizer
	Jobs func() *jobs.Client
	// Open overrides bouncebox.Open (tests).
	Open func(ctx context.Context, c bouncebox.Config) (bouncebox.Box, error)
}

MailboxWorker reads bounce reports from every SMTP provider's mailbox.

func (*MailboxWorker) Poll

func (w *MailboxWorker) Poll(ctx context.Context, p store.Provider) (int, error)

Poll reads up to 200 new reports from a provider's bounce mailbox, stores each as a raw event and queues its normalization.

func (*MailboxWorker) Timeout

Timeout bounds one poll.

func (*MailboxWorker) Work

Work implements river.Worker.

type NormalizeArgs

type NormalizeArgs struct {
	RawEventID uuid.UUID `json:"raw_event_id"`
}

NormalizeArgs normalizes one stored webhook payload.

func (NormalizeArgs) InsertOpts

func (NormalizeArgs) InsertOpts() river.InsertOpts

InsertOpts puts normalization on the events queue.

func (NormalizeArgs) Kind

func (NormalizeArgs) Kind() string

Kind implements river.JobArgs.

type NormalizeWorker

type NormalizeWorker struct {
	river.WorkerDefaults[NormalizeArgs]
	N *Normalizer
}

NormalizeWorker is the River worker for NormalizeArgs.

func (*NormalizeWorker) Work

Work implements river.Worker.

type Normalizer

type Normalizer struct {
	Pool    *pgxpool.Pool
	Q       *store.Queries
	Factory *manage.Factory
	Log     *slog.Logger
	Now     func() time.Time
	// OnSuppressed is called after new addresses joined the shared
	// suppression list, so other providers get them soon (sync job).
	OnSuppressed func(ctx context.Context, workspaceID uuid.UUID)
}

Normalizer turns raw provider events into ReputeCast's model.

func (*Normalizer) Apply

func (n *Normalizer) Apply(ctx context.Context, tx pgx.Tx, p store.Provider, rawID *uuid.UUID, source string, receivedAt time.Time, ev providers.Event) (int, error)

Apply records one normalized event: the delivery event, the message's status, campaign counters, provider outcome stats and, for hard bounces, complaints and provider unsubscribes, the shared suppression list. It returns how many addresses were newly suppressed.

func (*Normalizer) Backfill

func (n *Normalizer) Backfill(ctx context.Context, providerID uuid.UUID) (int, error)

Backfill looks up messages accepted since the webhooks went silent (or in the last week) and records what the provider reports about them.

func (*Normalizer) Normalize

func (n *Normalizer) Normalize(ctx context.Context, rawID uuid.UUID) error

Normalize processes a stored payload once; running it again is a no-op.

type ProviderEventLog

type ProviderEventLog struct {
	Items         []ProviderLogEvent `json:"items"`
	Total         int64              `json:"total"`
	Page          int                `json:"page"`
	PageSize      int                `json:"pageSize"`
	RetentionDays int                `json:"retentionDays"`
}

ProviderEventLog is one page of the event log.

type ProviderLogEvent

type ProviderLogEvent struct {
	ID         string    `json:"id"`
	ReceivedAt time.Time `json:"receivedAt"`
	OccurredAt time.Time `json:"occurredAt"`
	ProviderID string    `json:"providerId,omitempty"`
	Provider   string    `json:"provider"`
	// Channel is how the event arrived: webhook, bounce_mailbox or backfill.
	Channel       string `json:"channel" enum:"webhook,bounce_mailbox,backfill,test,internal"`
	RawType       string `json:"rawType"`
	Type          string `json:"type" enum:"delivered,deferred,hard_bounce,soft_bounce,complaint,rejected"`
	Code          string `json:"code"`
	Diagnostic    string `json:"diagnostic"`
	BounceClass   string `json:"bounceClass"`
	Recipient     string `json:"recipient" doc:"Masked"`
	ProviderMsgID string `json:"providerMessageId"`
	Source        string `json:"source"`
	CampaignID    string `json:"campaignId,omitempty"`
}

ProviderLogEvent is one row of the event log: the raw provider event beside the normalized ReputeCast event and the original code.

type SweepArgs

type SweepArgs struct{}

SweepArgs re-queues payloads whose normalization never finished and purges raw payloads past retention.

func (SweepArgs) InsertOpts

func (SweepArgs) InsertOpts() river.InsertOpts

InsertOpts implements river.JobArgsWithInsertOpts.

func (SweepArgs) Kind

func (SweepArgs) Kind() string

Kind implements river.JobArgs.

type SweepWorker

type SweepWorker struct {
	river.WorkerDefaults[SweepArgs]
	N    *Normalizer
	Jobs func() *jobs.Client
}

SweepWorker runs SweepArgs.

func (*SweepWorker) Work

func (w *SweepWorker) Work(ctx context.Context, _ *river.Job[SweepArgs]) error

Work implements river.Worker.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL