Documentation
¶
Overview ¶
Package pipeline takes one new email from arrival to outcome: ingest (once), fetch and parse, evaluate the rules, ask the decision model when rules with an intent are in play, hand the winning rule's actions to the executor or park the email in Needs review, record it all, and learn. It never changes a mailbox itself.
Index ¶
Constants ¶
const ( // RetryEvery is how long a failed message waits before the retry job runs it again. RetryEvery = 5 * time.Minute // MaxRetries is how many times the retry job tries before the message goes to Needs review. MaxRetries = 6 // ReasonNoModel is the Needs review reason while no decision model is configured. ReasonNoModel = "No decision model is set" // ReasonUnreadable is the decision's reason when the model's answer could not be read // after every try, and the email was kept. ReasonUnreadable = "The AI's answer could not be read after 3 tries, so the email was left where it is." // ReasonOwnMail is the decision's reason for an email sent from the mailbox's own // address while the leave_own_mail setting is on (Decider.Own). ReasonOwnMail = "Sent from this mailbox's own address, so MailRules left it alone." // ReasonAllowance is the Needs review reason of an email a model was needed for while // the tenant's monthly allowance of AI-sorted emails is used up (Decider.Allowance). ReasonAllowance = "This month's AI-sorted emails are used up, so MailRules left this one for you." )
Variables ¶
var ErrHalted = errors.New("the plan has ended, so MailRules sorts nothing")
ErrHalted is what the pipeline answers, sorting nothing, while the tenant's plan has ended (Pipeline.Halted).
var ErrModel = errors.New("the decision model failed")
ErrModel marks a failure of the decision model, as opposed to the mailbox or the database.
Functions ¶
func Corrections ¶
func Corrections(st *store.Store, v store.Viewer) func(ctx context.Context, email message.Summary, candidates []int64) []learn.Example
Corrections is the Examples of a Decider that learns from the corrections made to mail of the mailboxes v sees. A failure to read them is logged and costs the examples, never the decision.
Types ¶
type Decider ¶
type Decider struct {
Router *models.Router // nil = no decision model is set
// Override returns the router for a rule's own model, or nil when it cannot be used.
Override func(ctx context.Context, spec string) *models.Router
MinConfidence float64 // act threshold for rules that set none
Now time.Time // for age_days
// Examples returns the user's corrections to show the fallback model for this email,
// most similar first; candidates are the rule ids on offer. nil = show none.
Examples func(ctx context.Context, email message.Summary, candidates []int64) []learn.Example
// Own are the mailbox's own addresses, whose mail is left alone: no sender rule, rule or
// model is consulted for it. Empty = none (the leave_own_mail setting is off). See OwnMail.
Own []string
// RouteOnly are rules a sender rule may route to although they are not in the rules
// given to Settle (see rules.Options.RouteOnly). nil = none.
RouteOnly []rules.Rule
// Allowance, when set, says whether the tenant's monthly allowance of emails a model
// decides on keys the operator holds is used up. While it is, no model on those keys is
// asked: a model on the tenant's own keys decides (models.Router.OwnKeysOnly), or else
// the email waits in Needs review (ReasonAllowance). Sender rules and rules without an
// intent decide as ever. nil = no allowance (self-host).
Allowance func(ctx context.Context) (reached bool, err error)
// Spend, when set, counts each email a model decides on keys the operator holds as Settle
// decides it: a cleanup check's, which records no decision. Live sorting leaves it nil and
// counts with the decision's row (store.Decision.Metered).
Spend func(ctx context.Context) error
}
Decider settles which rule applies to one email: the rules' evaluation order first and, when rules with an intent are in play, the decision model. It changes nothing but what Spend counts, so the live pipeline, cleanup and the rule tester all decide through it and cannot disagree.
func (Decider) Settle ¶
func (d Decider) Settle(ctx context.Context, sum message.Summary, rs []rules.Rule, senders []rules.SenderRule) (Outcome, error)
Settle runs the evaluation order for one email and asks the model when it has to. The email is decided by the model of the highest-priority candidate rule that names one (and whose model can be used), otherwise by the default router.
type Executor ¶
type Executor interface {
Apply(ctx context.Context, d actions.DecisionRecord, acts []rules.Action, batchID int64) ([]actions.ActionRecord, error)
ApplyMany(ctx context.Context, ms []actions.Many, a rules.Action, batchID int64) (dry bool, errs []error)
}
Executor is the part of actions.Executor the pipeline needs.
type Outcome ¶
type Outcome struct {
rules.Result
RuleName string // the name of Result.RuleID; empty when there is none
Reason string // one sentence saying why
Asked bool // a model was asked
// NoModel: rules with an intent were in play and no model could be asked, so they were
// passed over. With a model set, this is an email the model would have decided.
NoModel bool
// Metered: a model on keys the operator holds decided the email, so it counts against the
// tenant's monthly allowance.
Metered bool
Calls int // how many model calls that took, counting retries and the fallback
Usage models.Usage // what asking cost, over every call made
// Probabilities is what the decision model gave each candidate, by rule id (0 = none
// of them); nil when it gives none.
Probabilities map[int64]float64
}
Outcome is what was settled for one email.
type Pipeline ¶
type Pipeline struct {
Store *store.Store
Mailbox mail.Mailbox
Account store.Account
Router *models.Router // nil = no decision model is set
Exec Executor // nil = record decisions only
Hub *events.Hub // nil = publish nothing
MinConfidence float64 // act threshold for rules that set none
BodyChars int // plain-text characters shown to models
// Live, when set, supplies Router and MinConfidence afresh for every message from the
// settings of the account's tenant, so a change made in Settings needs no restart
// (settings.Settings.Live).
Live func(ctx context.Context, tenantID int64) (router *models.Router, minConfidence float64)
// Override returns the tenant's router for a rule's own model (rules.Rule.Model), or nil
// when that model cannot be used (settings.Settings.RouterFor). nil = rules' models are ignored.
Override func(ctx context.Context, tenantID int64, spec string) *models.Router
// Batch is the batch the actions belong to. 0 = the day's live batch; a cleanup run
// sets its own, so the whole run can be undone as one.
Batch int64
// Spent, when set, is told what each model decision cost: a cleanup run shows the total.
Spent func(tokens int, costUSD float64)
// Allowance, when set, says whether a tenant's monthly allowance of emails a model
// decides on keys the operator holds is used up at now (Decider.Allowance). An email
// counted already (a message tried again) is decided as before. nil = no allowance.
Allowance func(ctx context.Context, tenantID int64, now time.Time) (reached bool, err error)
// Halted, when set, says whether the tenant's mail must not be sorted at now: its plan
// ended (a build that sells plans, ext.Module.Billing). While it is, Process, Sort,
// SortSaved and RetryDue leave every message as it is, the folder's watch position
// included, and answer ErrHalted; the supervisor then stops. Mail that arrived meanwhile
// is sorted when the mailbox starts again with a plan, as after a pause: the watcher
// resumes from the stored position. nil = never halted.
Halted func(ctx context.Context, tenantID int64, now time.Time) (bool, error)
Now func() time.Time // nil = time.Now
// Rediscover re-runs folder discovery. It is called once when a step fails because a
// folder is unknown, and the step is then tried again. nil = never.
Rediscover func(ctx context.Context) error
}
Pipeline processes the mail of one account. It is not safe for concurrent use: the account's supervisor calls it from one goroutine, which keeps mail in UID order.
func (*Pipeline) Process ¶
Process handles one message the watcher reported. It is idempotent: a message seen before is not processed again, unless an earlier run was cut off before it finished, and neither is a known email that came back into the folder under a new UID (cameBack). The folder's last processed UID advances only once the message's row is committed; on an error (the daemon is stopping, or the database failed) it does not, so the watcher sends the message again after a restart.
func (*Pipeline) RetryDue ¶
RetryDue is the retry job: it re-runs every failed message of the account whose wait is over, and returns how many it ran.
func (*Pipeline) Sort ¶
Sort runs one message of existing mail through the same steps as new mail: cleanup's way in. Unlike Process it does not look at whether the message was seen before (the rules may have changed since, or dry-run been switched off), and it leaves the folder's watch position alone.
func (*Pipeline) SortSaved ¶
SortSaved applies decisions a cleanup check already settled to existing emails of one folder (one UIDVALIDITY), without asking a model: it verifies they are still where the check found them with one fetch for all of them (BODY.PEEK, never marking them read), ingests their rows, records each decision carrying the check's stage, rule, model, confidence and reason — but no fresh cost, which is on the ledger once from the check — and hands the actions to the executor in the run's batch: the emails whose one action is the same move go to the server together (Executor.ApplyMany), any other email on its own. applied[i] is false when email i has moved or gone since the check: it is passed over. An error is the first email whose actions failed or whose outcome could not be recorded; what the others did is on record.