whimap

package
v1.160.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2026 License: Apache-2.0 Imports: 25 Imported by: 0

Documentation

Overview

Package whimap is the imap input of a webhook source (#2247): the platform polls a mailbox and files each new message from an allowed, authenticated sender as one event, written exactly as the HTTP receiver writes a request's events. Compaction into Parquet, the source's table, retention and alerts then treat it like any other source. Attachments are written as managed resources the event names.

Nothing here is a tool. The mail reaches a model or a script only through the source's table and the resources, like every webhook's data.

Index

Constants

View Source
const EventType = "mail"

EventType is the event_type of every message an imap source files.

View Source
const LockKeyspace int32 = 4713211

LockKeyspace is the advisory-lock space an imap source's poll is held in; the key within it is a hash of the source's name.

View Source
const OutcomeUnreadable = "unreadable"

OutcomeUnreadable is a message skipped because the mail client could not read the server's answer about it (#2459), as the source's rejections list and webhook_requests_total name it.

View Source
const Tick = 15 * time.Second

Tick is how often the poller looks for a source whose poll is due.

Variables

This section is empty.

Functions

func AttachmentsFolder

func AttachmentsFolder(source string) string

AttachmentsFolder is the library folder every attachment a source saved sits under, removed with the source.

func LockKey

func LockKey(source string) int32

LockKey is the key a source's poll is locked under.

Types

type Attachment

type Attachment struct {
	Name        string `json:"name"`
	ContentType string `json:"content_type"`
	Size        int64  `json:"size"`
	URI         string `json:"uri,omitempty"`
	ResourceID  string `json:"resource_id,omitempty"`
	Skipped     string `json:"skipped,omitempty"`
}

Attachment is one file a message carried: the resource it was saved as, or why it was not.

type Config

type Config struct {
	Sources    Sources
	Objects    Objects
	Bucket     string
	Recorder   Recorder
	RawWindows RawWindows
	Marks      MarkStore
	Files      Files
	// Rejections and Metrics record a message skipped as unreadable; either
	// may be nil.
	Rejections Rejections
	Metrics    Metrics
	Lock       Lock
	Replica    string
	Logger     *slog.Logger
	// Open opens a session; nil is imapmail.Open.
	Open func(ctx context.Context, a imapmail.Account) (*imapmail.Session, error)
	Now  func() time.Time
}

Config assembles a poller.

type External

type External struct {
	From    string   `json:"from"`
	To      []string `json:"to"`
	Cc      []string `json:"cc,omitempty"`
	Subject string   `json:"subject"`
	Body    string   `json:"body"`
}

External is what the message's sender wrote, cleaned of invisible and reordering characters, hidden HTML removed from the body.

type File

type File struct {
	Folder      string
	Name        string
	ContentType string
	Data        []byte
}

File is one attachment to save: the library folder it goes in, its name, type and bytes.

type Files

type Files interface {
	PutFile(ctx context.Context, src whsource.Source, f File) (uri, id string, err error)
}

Files saves an attachment as a managed resource in the source's library.

type Lock

type Lock func(ctx context.Context, source string) (release func(), ok bool, err error)

Lock takes the one-replica lock on a source's poll, reporting whether it was free.

type Mailbox

type Mailbox struct {
	// contains filtered or unexported fields
}

Mailbox is what the admin service asks of an imap source: whether its mailbox can be read as configured, and how each folder's polls are going.

func NewMailbox

func NewMailbox(marks *Marks) *Mailbox

NewMailbox returns the admin side of the imap input.

func (*Mailbox) Check

func (m *Mailbox) Check(ctx context.Context, src whsource.Source) error

Check logs in and opens each folder read-only, so a source whose login or folders are wrong is refused when it is saved rather than failing every poll.

func (*Mailbox) Marks

func (m *Mailbox) Marks(ctx context.Context, source string) ([]Mark, error)

Marks lists a source's folders as its polls left them.

type Mark

type Mark struct {
	Source      string `json:"-"`
	Folder      string `json:"folder"`
	UIDValidity uint32 `json:"uid_validity"`
	LastUID     uint32 `json:"last_uid"`
	Stored      int64  `json:"stored"`
	// HiddenNotAllowed and HiddenUnauthenticated count the messages the
	// sender rules withheld. Nothing else of them is kept.
	HiddenNotAllowed      int64      `json:"hidden_not_allowed"`
	HiddenUnauthenticated int64      `json:"hidden_unauthenticated"`
	LastPolledAt          *time.Time `json:"last_polled_at,omitempty"`
	LastError             string     `json:"last_error,omitempty"`
}

Mark is where one folder of a source has been read to, and what its polls found.

type MarkStore

type MarkStore interface {
	Get(ctx context.Context, source, folder string) (Mark, error)
	Put(ctx context.Context, mk Mark) error
}

MarkStore keeps each folder's watermark.

type Marks

type Marks struct {
	// contains filtered or unexported fields
}

Marks stores the marks in webhook_imap_marks.

func NewMarks

func NewMarks(db *sql.DB) *Marks

NewMarks returns the store, nil without a database.

func (*Marks) Get

func (m *Marks) Get(ctx context.Context, source, folder string) (Mark, error)

Get returns a folder's mark, zero when it has never been read.

func (*Marks) List

func (m *Marks) List(ctx context.Context, source string) ([]Mark, error)

List returns a source's marks, by folder.

func (*Marks) Put

func (m *Marks) Put(ctx context.Context, mk Mark) error

Put writes a folder's mark.

type Metrics added in v1.153.0

type Metrics interface {
	WebhookRequest(ctx context.Context, source, outcome string)
}

Metrics counts what a poll skipped, under webhook_requests_total.

type Objects

type Objects = whwrite.Objects

Objects writes a segment to the managed-resources bucket.

type Payload

type Payload struct {
	Folder      string `json:"folder"`
	UID         uint32 `json:"uid"`
	UIDValidity uint32 `json:"uid_validity"`
	MessageID   string `json:"message_id,omitempty"`
	InReplyTo   string `json:"in_reply_to,omitempty"`
	// References are the thread's earlier Message-IDs.
	References []string `json:"references,omitempty"`
	Date       string   `json:"date,omitempty"`
	Arrived    string   `json:"arrived"`
	// FromAddress is the sender's address as the From header names it, and
	// SenderAuth what the receiving server recorded of it.
	FromAddress string `json:"from_address"`
	SenderAuth  string `json:"sender_auth"`
	Size        int64  `json:"size"`
	BodyFormat  string `json:"body_format"`
	// BodySkipped says why the body was not read, empty when it was.
	BodySkipped string       `json:"body_skipped,omitempty"`
	Attachments []Attachment `json:"attachments"`
	External    External     `json:"external"`
	Notice      string       `json:"notice"`
}

Payload is one message as its event's payload: what the platform read of it and, under External, everything its sender wrote.

type Poller

type Poller struct {
	// contains filtered or unexported fields
}

Poller polls every imap source on its interval.

func New

func New(cfg Config) *Poller

New returns a poller, nil when a dependency is missing.

func (*Poller) Pass

func (p *Poller) Pass(ctx context.Context) error

Pass polls every imap source whose interval has passed.

func (*Poller) Poll

func (p *Poller) Poll(ctx context.Context, src whsource.Source) error

Poll reads every folder of one source once.

func (*Poller) SetStored added in v1.152.0

func (p *Poller) SetStored(fn func(ctx context.Context, source string, events []whevent.Event))

SetStored is what is told each written segment's events from here on: the hook the runs of an automation naming the source are enqueued through (#2332). It is set before the poller starts.

func (*Poller) Start

func (p *Poller) Start(ctx context.Context)

Start runs the poll loop until Stop.

func (*Poller) Stop

func (p *Poller) Stop()

Stop ends the loop and waits for a poll in progress.

type RawWindows

type RawWindows = whwrite.RawWindows

RawWindows registers a window's raw partition so its events are queryable before compaction.

type Recorder

type Recorder = whwrite.Recorder

Recorder records a window a segment was written to.

type Rejections added in v1.153.0

type Rejections interface {
	RecordRejections(ctx context.Context, rejections []whstore.Rejection) error
}

Rejections keeps what a poll skipped, in the source's rejections list beside the receiver's refusals (#2459).

type Sources

type Sources interface {
	List(ctx context.Context) ([]whsource.Source, error)
}

Sources lists the stored sources, secrets decrypted.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL