integration

package
v1.162.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: Apache-2.0 Imports: 25 Imported by: 0

Documentation

Overview

Package integration hands operations only a deployment's environment knows how to do to a script an administrator chooses (#2659): routing a hostname through the ingress, and later operations of the same shape. Each operation is a contract: the function the script defines, the items it is handed, and the status it returns for each. With no script chosen the platform does nothing more than it did before.

The mechanism is one copy for every operation. An operation's items are computed, never described by the call sites that change them: a pass computes the set now, compares it with the set the script was last handed, and the difference is the change. A pass runs when a change is announced (the tables an operation reads notify integration_changed), when an integration's run ends, and on a timer, on one replica at a time. One run of an operation is in flight at most; a change that arrives meanwhile is in the next run's difference.

Index

Constants

View Source
const (
	StatusPending = "pending"
	StatusReady   = "ready"
	StatusFailed  = "failed"
)

The statuses an item reads.

View Source
const (
	ReasonChange = "change"
	ReasonTimer  = "timer"
)

The reasons a run is made for.

View Source
const (
	DefaultInterval = 15 * 60
	MinInterval     = 60
	MaxInterval     = 24 * 60 * 60
)

The interval bounds an operation's timer may be set within, in seconds.

View Source
const Channel = "integration_changed"

Channel is where a change to an operation's items is announced.

View Source
const LockKey int64 = 4713210405

LockKey is the advisory lock one replica's pass holds.

View Source
const (
	// OpSiteRouting is every hostname the platform serves for sites and
	// applications, handed to a script that routes them (#2659).
	OpSiteRouting = "site_routing"
)

The operations the platform registers.

View Source
const RetryInterval = 15 * time.Second

RetryInterval is how long after a run starts a reader's retry is refused.

Variables

View Source
var (
	// ErrUnknown is an operation the platform does not register.
	ErrUnknown = errors.New("unknown integration")
	// ErrNotAllowed is a caller who may not change an integration.
	ErrNotAllowed = errors.New("not allowed")
	// ErrInvalid is a request that cannot be carried out as it is.
	ErrInvalid = errors.New("invalid integration request")
	// ErrNothingFailed is a reader's retry when nothing they can see failed.
	ErrNothingFailed = errors.New("nothing it shows failed")
	// ErrTooSoon is a reader's retry within RetryInterval of the last run.
	ErrTooSoon = errors.New("asked too soon")
)

Errors a caller reads by identity.

View Source
var ErrNotBound = errors.New("no script is chosen for this operation")

ErrNotBound is an operation with no script chosen.

Functions

func Check

func Check(source string) error

Check refuses a source that could serve no operation: it defines none of the operations' functions. It is the handler kind's check, which the save gate asks of every script an operation is bound to.

func CleanMessage

func CleanMessage(s string) string

CleanMessage is a script's message as it is kept and shown: control and invisible runes removed, one line, bounded.

func Contracts

func Contracts() string

Contracts is every operation's function, as the handler kind names it.

func Entry

func Entry(call map[string]any) (function string, req map[string]any, err error)

Entry is the function and request an integration run calls, read from its call.

func Handler

func Handler(call map[string]any, into *json.RawMessage) (*scriptdialect.Handler, error)

Handler is the call an integration's run makes: the operation's function with the request its call carries, what it returns checked against that request and kept in into. A value of the wrong shape fails the run.

func Prepare

func Prepare(trigger string, opts *scriptrun.Options) (finish func(*script.RunResult), err error)

Prepare readies an integration's run (trigger integration): it calls the operation's function with the request its call carries in place of main(), and finish keeps what the function returned, checked, as the run's result. Any other run is left as it is.

func ReadResult

func ReadResult(value any, keys []string) (map[string]ItemResult, error)

ReadResult reads the value the function returned against the keys of the request it was handed: a dict keyed by an item the request named, each value a dict of status ready or failed and an optional message. A key the request did not name is ignored, so a script cannot set the status of an item it was not asked about. The message is untrusted text: cleaned and bounded.

func RequestKeys

func RequestKeys(s Spec, req map[string]any) (items, removed []string)

RequestKeys is every key a run's request names: its items now and the ones it removes.

func Returned

func Returned(call map[string]any, value any) (json.RawMessage, error)

Returned checks what an integration run's function returned against the request in the run's call, and is the result the run keeps: one entry per item it was asked about, message cleaned. A value of the wrong shape fails the run, naming what was wrong, which only an administrator reads.

func RunCall

func RunCall(s Spec, req Request) map[string]any

RunCall is what an integration's run carries in its call column.

Types

type Actor

type Actor struct {
	Email string
	Roles []string
	Admin bool
}

Actor is who changes an integration.

type Applied

type Applied struct {
	RunID    string
	Results  map[string]ItemResult
	Failed   []string
	FailWith string
	Removed  []string
	Status   string
	Error    string
}

Applied is what a finished run left: each item's status, and the items it removed.

type Approvals

type Approvals interface {
	HandlerRunAs(ctx context.Context, scriptID string) (approval.RunAs, bool)
	ApproveForHandlers(ctx context.Context, actor approval.Actor, sc *script.Script, version int) error
}

Approvals is what a bound script's version is approved and run through (approval.Service): the same approval every handler kind runs on.

type BindRequest

type BindRequest struct {
	Operation string
	ScriptID  string
	// IntervalSeconds is the timer; 0 keeps the current one or the default.
	IntervalSeconds int
	// Approve approves the script's latest version for handlers as the actor.
	Approve bool
}

BindRequest chooses an operation's script.

type Binding

type Binding struct {
	ID              string     `json:"id"`
	Operation       string     `json:"operation"`
	ScriptID        string     `json:"script_id"`
	IntervalSeconds int        `json:"interval_seconds"`
	CurrentRunID    string     `json:"current_run_id,omitempty"`
	RetryRequested  bool       `json:"retry_requested"`
	LastStartedAt   *time.Time `json:"last_started_at,omitempty"`
	LastRunID       string     `json:"last_run_id,omitempty"`
	LastStatus      string     `json:"last_status,omitempty"`
	LastReason      string     `json:"last_reason,omitempty"`
	LastError       string     `json:"last_error,omitempty"`
	LastFinishedAt  *time.Time `json:"last_finished_at,omitempty"`
	CreatedBy       string     `json:"created_by"`
	UpdatedBy       string     `json:"updated_by"`
	CreatedAt       time.Time  `json:"created_at"`
	UpdatedAt       time.Time  `json:"updated_at"`
}

Binding is the script chosen for an operation and how its runs stand.

type Bindings

type Bindings interface {
	Binding(ctx context.Context, op string) (*Binding, error)
	Bind(ctx context.Context, op, scriptID string, intervalSeconds int, by string) (*Binding, error)
	Clear(ctx context.Context, op string) (bool, error)
	RequestRetry(ctx context.Context, op string) error
	Items(ctx context.Context, op string) ([]StoredItem, error)
	ItemsByKey(ctx context.Context, op string, keys []string) (map[string]StoredItem, error)
	StartRun(ctx context.Context, op string, r NewRun) error
	RecordRefusal(ctx context.Context, op, reason string) error
	RunOutcome(ctx context.Context, runID string) (RunOutcome, error)
	Apply(ctx context.Context, op string, a Applied) error
	// Lock takes the lock one replica's pass holds; ok is false when another
	// replica holds it.
	Lock(ctx context.Context) (release func(), ok bool, err error)
}

Bindings is where bindings, items and an operation's runs are kept (*Store).

type Candidate

type Candidate struct {
	ID            string `json:"id"`
	Name          string `json:"name"`
	OwnerEmail    string `json:"owner_email"`
	Version       int    `json:"version"`
	Running       int    `json:"running_version"`
	ApprovedBy    string `json:"approved_by,omitempty"`
	RunningServes bool   `json:"running_serves"`
}

Candidate is a script the picker offers for an operation: its latest version defines the operation's function. Running is the version approved for handlers (0 for none) and RunningServes whether that one defines it too.

type Item

type Item struct {
	Key   string
	Entry map[string]any
}

Item is one thing an operation hands the script: its key and the entry the request carries for it, which holds the key under the operation's key field.

type ItemResult

type ItemResult struct {
	Status  string `json:"status"`
	Message string `json:"message"`
}

ItemResult is the status the script returned for one item.

type ItemStatus

type ItemStatus struct {
	Status    string     `json:"status"`
	Message   string     `json:"message"`
	CheckedAt *time.Time `json:"checked_at,omitempty"`
	// RunID and RunError are the failing run's, filled for an administrator
	// only, when the script itself failed.
	RunID       string `json:"run_id,omitempty"`
	RunScriptID string `json:"run_script_id,omitempty"`
	RunError    string `json:"run_error,omitempty"`
	// ScriptFailed says the script itself failed, so Retry is offered.
	ScriptFailed bool `json:"script_failed,omitempty"`
}

ItemStatus is one item's status as the surfaces that show the item report it: under routing, with no run error, which only an administrator reads.

type NewRun

type NewRun struct {
	ID        string
	ScriptID  string
	VersionID string
	Version   int
	Approver  string
	Roles     []string
	Call      map[string]any
	Reason    string
	Added     []Item
	Removed   []string
}

NewRun is one integration run to queue.

type Overview

type Overview struct {
	Spec
	Binding    *Binding       `json:"binding,omitempty"`
	ScriptName string         `json:"script_name,omitempty"`
	Running    int            `json:"running_version,omitempty"`
	ApprovedBy string         `json:"approved_by,omitempty"`
	Counts     map[string]int `json:"counts"`
	Items      []StoredItem   `json:"items"`
	// Total is how many items the operation has now, whether or not a script
	// is chosen.
	Total int `json:"total"`
}

Overview is one operation as the settings page shows it.

type Request

type Request struct {
	Reason  string           `json:"reason"`
	Added   []map[string]any `json:"added"`
	Removed []map[string]any `json:"removed"`
	Items   []map[string]any `json:"-"`
}

Request is what the script's function is handed.

func (Request) Map

func (r Request) Map(s Spec) map[string]any

Map is the request as the script reads it, its items under the operation's field. Empty lists are [], never null.

type RunOutcome

type RunOutcome struct {
	Finished bool
	Status   string
	Error    string
	Result   []byte
	Call     map[string]any
}

RunOutcome is how an integration's run ended.

type Runtime

type Runtime struct {
	Service *Service
	// DSN opens the LISTEN connection; empty polls only.
	DSN string
	// contains filtered or unexported fields
}

Runtime runs the passes on this replica: on every announcement on Channel and every passEvery.

func (*Runtime) Notify

func (r *Runtime) Notify()

Notify wakes the next pass; the listener calls it on every announcement.

func (*Runtime) Start

func (r *Runtime) Start(ctx context.Context) error

Start listens for announcements and starts the passes.

func (*Runtime) Stop

func (r *Runtime) Stop(context.Context) error

Stop ends the listener and the passes.

type Scripts

type Scripts interface {
	GetByID(ctx context.Context, id string) (*script.Script, error)
	GetVersion(ctx context.Context, scriptID string, version int) (*script.Version, error)
}

Scripts reads a bound script and its versions.

type Service

type Service struct {
	Store     Bindings
	Scripts   Scripts
	Approvals Approvals
	// Sources computes each operation's items; an operation with none is
	// never run.
	Sources map[string]Source
	// contains filtered or unexported fields
}

Service binds operations to scripts and runs them.

func (*Service) Bind

func (s *Service) Bind(ctx context.Context, a Actor, req BindRequest) (*Binding, error)

Bind chooses the script an operation calls. Only an administrator binds, and only a script whose version that will run defines the operation's function.

func (*Service) Candidates

func (s *Service) Candidates(ctx context.Context, a Actor, op, search string) ([]Candidate, error)

Candidates is the scripts an administrator may choose for an operation, matching search.

func (*Service) Clear

func (s *Service) Clear(ctx context.Context, a Actor, op string) error

Clear stops an operation's calls and drops every status it kept: the deployment is back to behaving as it did with no script.

func (*Service) Overview

func (s *Service) Overview(ctx context.Context) ([]Overview, error)

Overview is every operation with its binding and items. Only an administrator reads it: it carries the run error.

func (*Service) Pass

func (s *Service) Pass(ctx context.Context) error

Pass runs every bound operation once, on one replica at a time: it applies a finished run, then starts the next when the items changed, a retry was asked for, or the timer is due.

func (*Service) Retry

func (s *Service) Retry(ctx context.Context, op string) error

Retry asks for a run with every item at once. The caller has been allowed it by its surface: an administrator, or someone who can see a failed item.

func (*Service) RetryFailed

func (s *Service) RetryFailed(ctx context.Context, op string, keys []string) error

RetryFailed is a retry asked for by someone who is not an administrator and can see keys (#2659): honored only when one of keys failed, and not within RetryInterval of the last run's start, so a reader can ask again for what failed but cannot keep the script running.

func (*Service) Statuses

func (s *Service) Statuses(ctx context.Context, op string, keys []string, admin bool) (map[string]ItemStatus, error)

Statuses is each key's status for an operation, nil when no script is chosen (the field is then omitted everywhere). A key the script has not answered for yet is pending. admin adds the run error to an item the script itself failed.

type Source

type Source interface {
	Items(ctx context.Context) ([]Item, error)
}

Source computes an operation's items now.

type Spec

type Spec struct {
	// Name is the operation's name in the API and the view's target.
	Name string `json:"operation"`
	// Label is how the portal names it.
	Label string `json:"label"`
	// Function is the top-level function the script defines, taking req.
	Function string `json:"function"`
	// ItemsField is the request field holding every item now, and the name
	// an item is counted by.
	ItemsField string `json:"items_field"`
	// KeyField is the entry field an item is keyed by.
	KeyField string `json:"key_field"`
	// FailedMessage is what someone other than an administrator reads on an
	// item when the script itself failed.
	FailedMessage string `json:"-"`
}

Spec is one operation's contract.

func FromRunCall

func FromRunCall(call map[string]any) (Spec, map[string]any, error)

FromRunCall reads an integration run's call: its operation and request.

func SpecOf

func SpecOf(name string) (Spec, bool)

SpecOf is the operation with the name.

func Specs

func Specs() []Spec

Specs is every registered operation.

func (Spec) CheckFor

func (s Spec) CheckFor(source string) error

CheckFor refuses a source that could not serve the operation.

func (Spec) Contract

func (s Spec) Contract() string

Contract is the function a script defines to serve the operation, as a refusal names it.

type Store

type Store struct{ DB *sql.DB }

Store keeps bindings and items in PostgreSQL.

func (*Store) Apply

func (s *Store) Apply(ctx context.Context, op string, a Applied) error

Apply writes a finished run's statuses, deletes the items it removed, and clears the run in flight, in one transaction.

func (*Store) Bind

func (s *Store) Bind(ctx context.Context, op, scriptID string, intervalSeconds int, by string) (*Binding, error)

Bind chooses the script for an operation, or changes it or its interval. The next pass calls the script with every item.

func (*Store) Binding

func (s *Store) Binding(ctx context.Context, op string) (*Binding, error)

Binding is the operation's binding, nil when no script is chosen.

func (*Store) Clear

func (s *Store) Clear(ctx context.Context, op string) (bool, error)

Clear removes the operation's binding and every status it kept.

func (*Store) Items

func (s *Store) Items(ctx context.Context, op string) ([]StoredItem, error)

Items is every item the operation's script was last handed.

func (*Store) ItemsByKey

func (s *Store) ItemsByKey(ctx context.Context, op string, keys []string) (map[string]StoredItem, error)

ItemsByKey is the stored items among keys, by key.

func (*Store) Lock

func (s *Store) Lock(ctx context.Context) (release func(), ok bool, err error)

Lock takes the advisory lock one replica's pass holds.

func (*Store) RecordRefusal

func (s *Store) RecordRefusal(ctx context.Context, op, reason string) error

RecordRefusal records why a pass could not call the script, and holds the next try to the operation's interval.

func (*Store) RequestRetry

func (s *Store) RequestRetry(ctx context.Context, op string) error

RequestRetry asks the next pass to call the operation's script with every item, and wakes the passes.

func (*Store) RunOutcome

func (s *Store) RunOutcome(ctx context.Context, runID string) (RunOutcome, error)

RunOutcome reads a run; Finished is false while it is pending or running.

func (*Store) StartRun

func (s *Store) StartRun(ctx context.Context, op string, r NewRun) error

StartRun records the items a run adds (pending) and removes (removing), queues the run and marks it the operation's run in flight, in one transaction, then tells the run workers.

type StoredItem

type StoredItem struct {
	Key       string         `json:"key"`
	Entry     map[string]any `json:"entry"`
	Status    string         `json:"status"`
	Message   string         `json:"message"`
	Removing  bool           `json:"-"`
	FailedRun string         `json:"failed_run,omitempty"`
	CheckedAt *time.Time     `json:"checked_at,omitempty"`
}

StoredItem is one item as the script was last handed it, and the status it last returned.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL