Documentation
¶
Overview ¶
Package admin is the operations side of ubgo/workflow (list failed runs, retry, forget, clear a backlog, pause, list schedules, and the data a queue dashboard shows): a Go API, a JSON HTTP handler that is closed by default, and a threshold Monitor. It is its own module, needs nothing beyond the standard library and the core, and holds no state: every call goes to the client.
Operations over any client ¶
New(client) returns an Admin over any workflow.Client. Each operation needs a capability and an optional core interface (Requirements):
runs, run, failures, queues CapRunQuery workflow.RunQuerier discarded CapRunQuery and CapRunStates workflow.RunQuerier retry CapRetryRun workflow.RunRetrier delete CapDeleteRun workflow.RunDeleter cancel CapCancel Client.Cancel cancel_all CapCancelAll workflow.BulkCanceller pause CapPauseWorkflow workflow.Pauser (CapPauseQueue for a queue target) schedules CapScheduleList workflow.ScheduleLister schedule_pause CapScheduleList and CapSchedulePause workflow.ScheduleLister and workflow.SchedulePauser schedule_trigger CapScheduleList and CapScheduleTrigger workflow.ScheduleLister and workflow.ScheduleTriggerer force CapForceRun workflow.RunForcer retry_all CapRunQuery and CapRetryRun workflow.RunQuerier and workflow.RunRetrier delete_all CapRunQuery and CapDeleteRun workflow.RunQuerier and workflow.RunDeleter nodes none workflow.NodeLister health none workflow.CheckHealth (every client) reschedule none workflow.RunRescheduler cancel_step none workflow.StepCanceller
Support reports which ones the client has and what it lacks, so a screen hides what cannot work; an unsupported call returns a *workflow.UnsupportedError naming "admin.<operation>" and the missing capabilities. QueueStats pages RunQuerier over queued runs to count each queue's depth and find its oldest run, reading at most WithMaxScan runs (DefaultMaxScan) and saying Truncated when there were more. On a client that implements workflow.WorkerScaler (CapWorkerScale) it adds each queue's worker count and running attempts on that client (QueueReport.Workers), the state a workflow.BalanceLoop acts on. Failures lists failed runs and Discarded, beside it, the runs a step dropped with workflow.Discard; a driver without CapRunStates reports a discarded run as failed or cancelled, so Discarded is refused there. CountRuns counts the runs a query matches, within the same max scan, and returns the cursor that starts each page, so a screen can show "1–50 of N" and numbered pages over a driver's opaque cursors. Runs, Failures, Discarded and CountRuns search by label: RunQuery.Labels (every key=value pair must match) and RunQuery.LabelPrefixes (a key's value starts with a prefix; the HTTP filter label=key=prefix*), which is refused with an UnsupportedError on a client without CapLabelPrefix.
RetryAll and DeleteAll act on every run a filter matches: they list the matches first (at most WithMaxBulk, DefaultMaxBulk, per call; Truncated says to call again) and then retry or delete each, counting runs that changed in between as Skipped and stopping on any other error. A BulkOptions.DryRun lists and changes nothing, returning the count and a sample of refs; PreviewCancelAll is the same for CancelAll. DeleteAll refuses an empty filter, so forgetting every finished run is said explicitly with its statuses.
Metrics reads run-end history for the charts from a MetricsSource wired with WithMetrics: counts per step by terminal status, zero-filled. MemoryMetrics is the in-process default, fed by registering its Observe with the run-finished hook; it keeps one-minute buckets for a day, drops repeated deliveries, and starts empty on a restart (Since says when its data begins). An app with a metrics store adapts it with MetricsSourceFunc.
Nodes lists the online nodes (any client implementing workflow.NodeLister) and Health returns the client's health report, which an admin screen shows; a probe should call workflow.CheckHealth on the app's own unauthenticated route. Nodes, Reschedule (move a delayed run) and CancelStep (withdraw one step) have no capability bit: whether they are offered is workflow.Supports, which a wrapper (workflow.Forward, failover) answers for the clients it wraps, and a client without them is refused at the call.
Schedule returns one schedule with its overlap policy and, on a client with CapScheduleHistory, its last fire and the run it started (ScheduleInfo.Last). TriggerSchedule runs a schedule now with its payload and labels; the overlap policy still applies, so with OverlapSkip and the previous run unfinished it returns an error matching workflow.ErrOverlap and enqueues nothing. ForceComplete and ForceFail end a run from outside (workflow.RunForcer); ForceFail's reason is bounded by MaxForceReason. A run's per-attempt errors (RunInfo.Errors, CapErrorHistory) come with it. Captured step output is read through an OutputSource the app wires with WithOutputs; the library keeps no output itself, and MemoryOutputs is an in-process, bounded store that is both the middleware.CaptureOutput sink and the source.
The HTTP handler is closed by default ¶
NewHandler(admin, Auth[P]{Identify, Authorize}) serves the JSON API. P is the app's own principal type. With either function nil, every request, on every path, is refused with 403. Otherwise each request is identified first (401 before routing on any error or panic), then parsed, then authorized with a Request[P] saying who, which Action and which Resource (the run, the parsed filter or the pause target); any error or panic from Authorize refuses it with 403, so a policy that fails, fails closed. Writes need Content-Type: application/json, which a cross-site form cannot send. Payloads are left out of responses unless WithPayloads is given, and an error without a known core sentinel is logged (WithLogger) and answered with a bare 500, so a database error never reaches the browser. Run lists are paginated with the driver's opaque cursor (count=1 adds the total and every page's cursor); schedules with an offset cursor. Attempt errors are sent with each run; their stack traces only with WithStacks. Force-fail sends its reason to Authorize in Resource.Reason, so a policy can require one or log it. Bulk retry, delete and cancel send the dry-run flag to Authorize in Resource.DryRun, so a policy can let more people preview than act. A run's tenant, due time, progress, nodes and result are sent with it; its progress data and its result only with WithPayloads, since both are app data like a payload.
Web UI ¶
The same handler serves a web UI at ui/ under its mount point (GET / and GET /ui redirect there): queues with depth, oldest wait and worker counts, and pause and resume of queues, workflows and everything; runs filtered by status, workflow, queue and label, with numbered pages and a page size; one run with its attempts, error and, only with WithPayloads, its payload; its timing (queue wait, duration), the error of each failed attempt (stacks with WithStacks), its captured output per attempt when WithOutputs is wired, and force complete and force fail; failed and discarded runs with retry and delete; schedules with their last fire and last run, run now, pause and resume, and a page per schedule with its recent runs; "all matching" retry, delete and cancel on a run list, each showing the server's dry-run count before the confirming click; a run's progress, tenant, due time, nodes and result, with reschedule for a delayed run and cancel step for a running one; charts of run ends per step by status over one, six or 24 hours, with totals and a table view, when WithMetrics is wired; and a system page with the health report and the online nodes. It is three files embedded in the package, written for it, with no third-party code and nothing fetched at run time. They are served only after Identify and Authorize (ActionUI), like every route, and each call the page makes is a request to the JSON API, authorized on its own action, so a read-only role sees the screens and is refused every button. Writes go through the API's JSON-only rule. A browser sends no Authorization header when it opens a page, so Identify must accept the app's session cookie for the UI to load. Pages carry a Content-Security-Policy that allows only those files and same-origin requests, and forbids framing. Buttons for an operation the client does not support are hidden (the capabilities route).
Alerts ¶
NewMonitor polls QueueStats on the Admin's clock (WithClock) and calls an Alerter when a queue's depth or oldest wait goes over a Threshold, once on crossing and once when it resolves (WithRepeat re-sends while it stays over). Nothing runs until Monitor.Run; Check polls once. Which alerts are open is kept in memory, so run one Monitor per deployment.
The OpenTelemetry queue-depth gauge lives in the otel module; Admin.QueueDepths has the otel.QueueDepthFunc signature, so neither module imports the other.
Index ¶
- Constants
- Variables
- func NewHandler[P any](a *Admin, auth Auth[P], opts ...HandlerOption) http.Handler
- func Requirements(op Operation) workflow.Caps
- func ValidateMetricsQuery(q MetricsQuery) error
- type Action
- type Admin
- func (a *Admin) Cancel(ctx context.Context, ref workflow.RunRef) error
- func (a *Admin) CancelAll(ctx context.Context, q workflow.RunQuery) (int, error)
- func (a *Admin) CancelStep(ctx context.Context, ref workflow.RunRef, step string) error
- func (a *Admin) Client() workflow.Client
- func (a *Admin) CountRuns(ctx context.Context, q workflow.RunQuery) (RunCount, error)
- func (a *Admin) Delete(ctx context.Context, ref workflow.RunRef) error
- func (a *Admin) DeleteAll(ctx context.Context, q workflow.RunQuery, opts BulkOptions) (BulkResult, error)
- func (a *Admin) Discarded(ctx context.Context, q workflow.RunQuery) (workflow.RunPage, error)
- func (a *Admin) Failures(ctx context.Context, q workflow.RunQuery) (workflow.RunPage, error)
- func (a *Admin) ForceComplete(ctx context.Context, ref workflow.RunRef) error
- func (a *Admin) ForceFail(ctx context.Context, ref workflow.RunRef, reason string) error
- func (a *Admin) HasMetrics() bool
- func (a *Admin) HasOutputs() bool
- func (a *Admin) Health(ctx context.Context) workflow.HealthReport
- func (a *Admin) Metrics(ctx context.Context, q MetricsQuery) (MetricsSeries, error)
- func (a *Admin) Nodes(ctx context.Context) ([]workflow.NodeInfo, error)
- func (a *Admin) Pause(ctx context.Context, t workflow.PauseTarget) error
- func (a *Admin) PauseSchedule(ctx context.Context, id workflow.ScheduleID) error
- func (a *Admin) Paused(ctx context.Context) ([]workflow.PauseTarget, error)
- func (a *Admin) PreviewCancelAll(ctx context.Context, q workflow.RunQuery) (BulkResult, error)
- func (a *Admin) QueueDepths(ctx context.Context) (map[string]int64, error)
- func (a *Admin) QueueStats(ctx context.Context) (QueueReport, error)
- func (a *Admin) Reschedule(ctx context.Context, ref workflow.RunRef, at time.Time) (workflow.RunRef, error)
- func (a *Admin) Resume(ctx context.Context, t workflow.PauseTarget) error
- func (a *Admin) ResumeSchedule(ctx context.Context, id workflow.ScheduleID) error
- func (a *Admin) Retry(ctx context.Context, ref workflow.RunRef) (workflow.RunRef, error)
- func (a *Admin) RetryAll(ctx context.Context, q workflow.RunQuery, opts BulkOptions) (BulkResult, error)
- func (a *Admin) Run(ctx context.Context, ref workflow.RunRef) (workflow.RunInfo, error)
- func (a *Admin) RunOutput(ctx context.Context, ref workflow.RunRef) ([]AttemptOutput, error)
- func (a *Admin) Runs(ctx context.Context, q workflow.RunQuery) (workflow.RunPage, error)
- func (a *Admin) Schedule(ctx context.Context, id workflow.ScheduleID) (workflow.ScheduleInfo, error)
- func (a *Admin) Schedules(ctx context.Context) ([]workflow.ScheduleInfo, error)
- func (a *Admin) Support() []OperationSupport
- func (a *Admin) Supports(op Operation) bool
- func (a *Admin) TriggerSchedule(ctx context.Context, id workflow.ScheduleID) (workflow.RunRef, error)
- type Alert
- type AlertKind
- type Alerter
- type AlerterFunc
- type AttemptOutput
- type Auth
- type BulkOptions
- type BulkResult
- type HandlerOption
- type MemoryMetrics
- type MemoryMetricsConfig
- type MemoryOutputLimits
- type MemoryOutputs
- type MetricsPoint
- type MetricsQuery
- type MetricsSeries
- type MetricsSource
- type MetricsSourceFunc
- type Monitor
- type MonitorOption
- type Operation
- type OperationSupport
- type Option
- type OutputSource
- type OutputSourceFunc
- type QueueReport
- type QueueStat
- type Request
- type Resource
- type RunCount
- type Threshold
Constants ¶
const ( CodeForbidden = "forbidden" // 403: no Auth configured, or Authorize refused CodeInvalid = "invalid" // 400: a malformed request or workflow.ErrInvalid CodeNotFound = "not_found" // 404: workflow.ErrNotFound, or no such route CodeUnsupported = "unsupported" // 501: workflow.ErrUnsupported CodeConflict = "conflict" // 409: workflow.ErrWrongState, workflow.ErrDuplicate or workflow.ErrOverlap CodeMediaType = "unsupported_media_type" // 415: a write without Content-Type: application/json CodeInternal = "internal" // 500: anything else; the detail is logged, not sent )
The error codes in an error response's "code" field.
const ( DefaultSchedulesLimit = 100 MaxSchedulesLimit = 1000 )
DefaultSchedulesLimit and MaxSchedulesLimit bound a page of GET /schedules.
const ( DefaultMetricsWindow = time.Hour DefaultMetricsPoints = 60 )
DefaultMetricsWindow is the span GET /metrics covers without from or window, and DefaultMetricsPoints how many steps it splits a span into without step.
const ( DefaultMetricsBucket = time.Minute DefaultMetricsKeep = 24 * time.Hour // DefaultMetricsDedupe is how many recent run ends MemoryMetrics // remembers to drop a repeated delivery (delivery is at least once). DefaultMetricsDedupe = 4096 )
The MemoryMetrics defaults.
const ( DefaultOutputRuns = 1000 DefaultOutputPerRun = 20 DefaultOutputBytes = 64 << 10 )
The MemoryOutputs defaults.
const BulkSample = 20
BulkSample is how many matched refs a BulkResult carries, so a dry run shows which runs it would touch without sending thousands of refs.
const DefaultMaxBody = 64 << 10
DefaultMaxBody is the largest request body the handler reads.
const DefaultMaxBulk = 1000
DefaultMaxBulk bounds how many runs one RetryAll or DeleteAll call acts on, so one HTTP request does a bounded amount of work; a caller with more matches repeats the call (BulkResult.Truncated says so). It is a floor on request cost, not on what an operator can clear: each repeat takes the next runs, because the ones acted on no longer match.
const DefaultMaxScan = 10_000
DefaultMaxScan bounds how many queued runs QueueStats reads per call, so a queue of a million runs costs a bounded number of RunQuerier pages.
const DefaultMonitorInterval = 30 * time.Second
DefaultMonitorInterval is how often a Monitor polls when WithInterval is not given.
const MaxForceReason = 1024
MaxForceReason bounds the reason ForceFail stores on a run, so an operator's text cannot grow a run row without limit.
const MaxMetricsPoints = 1440
MaxMetricsPoints bounds how many points one Metrics call returns, so a chart request over a year at one-minute steps is refused rather than answered with half a million points.
Variables ¶
var Actions = []Action{ ActionCapabilities, ActionListRuns, ActionShowRun, ActionListFailures, ActionListDiscarded, ActionRetryRun, ActionCancelRun, ActionDeleteRun, ActionCancelRuns, ActionPause, ActionResume, ActionListPauses, ActionListSchedules, ActionQueueStats, ActionPauseSchedule, ActionResumeSchedule, ActionShowSchedule, ActionTriggerSchedule, ActionForceComplete, ActionForceFail, ActionShowOutput, ActionUI, ActionRetryRuns, ActionDeleteRuns, ActionMetrics, ActionListNodes, ActionHealth, ActionRescheduleRun, ActionCancelStep, }
Actions lists every action, so a test or a permission screen can walk them.
var Operations = []Operation{OpRuns, OpRun, OpFailures, OpDiscarded, OpQueues, OpRetry, OpDelete, OpCancel, OpCancelAll, OpPause, OpSchedules, OpSchedulePause, OpScheduleTrigger, OpForce, OpRetryAll, OpDeleteAll, OpNodes, OpHealth, OpReschedule, OpCancelStep}
Operations lists every operation in a fixed order, the order Support reports them in.
Functions ¶
func NewHandler ¶
NewHandler returns the admin JSON API over a. It is closed by default: with a nil Auth.Identify or Auth.Authorize every request, on every path, is refused with 403 before anything else happens. With both set, each request is identified first (401 on error, also before routing, so an anonymous caller learns nothing about the routes), then parsed, then authorized with its action and resource (403 on any error or panic), and only then run.
Writes (POST, DELETE) must send Content-Type: application/json, which a browser cannot send cross-site without a CORS preflight the handler never answers, so a forged form post from another site is refused (415). Mount it under a prefix with http.StripPrefix.
Routes (JSON in and out; times are RFC 3339):
GET /capabilities the operations the client supports, and what it lacks
GET /runs list and search runs; filters below; paginated
GET /runs/{id} one run (?scheduled=1 when id is a ScheduledID)
GET /failures failed runs; same filters but status
GET /discarded runs a step discarded; same filters but status
POST /runs/{id}/retry retry a failed run from its failed step
POST /runs/{id}/cancel cancel one run
DELETE /runs/{id} forget a finished run
POST /runs/cancel cancel every run a filter body matches; returns the count; {"dry_run":true} counts only
POST /pause body {"workflow":..} | {"queue":..} | {"all":true}
POST /resume same body
GET /pauses the active pauses
GET /schedules schedules with next fire times; ?limit= and ?cursor=
GET /queues queue depth and oldest wait (workers and running when the driver scales workers)
POST /schedules/{id}/pause stop one schedule firing
POST /schedules/{id}/resume let it fire again
GET /schedules/{id} one schedule: next fire, overlap policy, last fire and its run
POST /schedules/{id}/trigger run it now; returns the run's ref (409 when its overlap policy skips)
POST /runs/{id}/complete force-complete a run that is not terminal, or is failed
POST /runs/{id}/fail force-fail a run that is not terminal; body {"reason":".."}
GET /runs/{id}/output captured step output, oldest attempt first (needs WithOutputs, else 501)
POST /runs/retry retry every failed run a filter body matches; {"dry_run":true} previews
POST /runs/delete forget every finished run a filter body matches; same body
GET /metrics run ends per step by status (needs WithMetrics, else 501); ?from=&to= or ?window=, ?step=, ?workflow=
GET /nodes the online nodes (workflow.NodeLister)
GET /health the client's health report: live, ready and each check
POST /runs/{id}/reschedule move a delayed run; body {"at":"RFC 3339"}
POST /runs/{id}/steps/{step}/cancel withdraw one step of a run
GET /ui/ the web UI (see below); GET / and GET /ui redirect to it
Run filters: workflow, queue, tenant (CapTenant), due_from and due_to (RunInfo.RunAt, CapDueQuery), status (repeat or comma-separated), label=key=value (repeat; every pair must match; a value ending in "*" is a prefix, needing CapLabelPrefix, else 501), from and to (enqueue time), limit and cursor (from the previous page's "next"). On /runs, /failures and /discarded, count=1 adds "total", "truncated" and "cursors" (the cursor that starts each page at this limit), read by Admin.CountRuns within the max scan, so a screen can show numbered pages over the driver's opaque cursors.
The web UI is three files embedded in this package (a page, a style sheet and a script; no third-party code, nothing fetched at run time), served behind the same gate as the API: Identify, then Authorize with ActionUI. Every call the UI makes is a request to the routes above and is authorized on its own. A browser sends no Authorization header when it loads a page, so Identify must accept the app's session cookie for the UI to open. Pages are sent with a Content-Security-Policy that allows only these files and same-origin requests.
func Requirements ¶
Requirements returns the capabilities op needs. OpPause reports both pause capabilities; a client with either one supports pausing the targets that capability covers.
func ValidateMetricsQuery ¶
func ValidateMetricsQuery(q MetricsQuery) error
ValidateMetricsQuery checks q's shape: From before To, a positive Step, and at most MaxMetricsPoints points. Errors wrap workflow.ErrInvalid.
Types ¶
type Action ¶
type Action string
Action is what an HTTP request asks to do, as the Authorizer sees it. The names are stable, so an app can store them in its own permission tables.
const ( ActionCapabilities Action = "capabilities.read" // GET /capabilities ActionListRuns Action = "runs.list" // GET /runs ActionShowRun Action = "runs.show" // GET /runs/{id} ActionListFailures Action = "failures.list" // GET /failures ActionListDiscarded Action = "discarded.list" // GET /discarded ActionRetryRun Action = "runs.retry" // POST /runs/{id}/retry ActionCancelRun Action = "runs.cancel" // POST /runs/{id}/cancel ActionDeleteRun Action = "runs.delete" // DELETE /runs/{id} ActionCancelRuns Action = "runs.cancel_all" // POST /runs/cancel ActionPause Action = "pauses.create" // POST /pause ActionResume Action = "pauses.delete" // POST /resume ActionListPauses Action = "pauses.list" // GET /pauses ActionListSchedules Action = "schedules.list" // GET /schedules ActionQueueStats Action = "queues.list" // GET /queues // ActionPauseSchedule and ActionResumeSchedule pause and resume one // schedule by ID. ActionPauseSchedule Action = "schedules.pause" // POST /schedules/{id}/pause ActionResumeSchedule Action = "schedules.resume" // POST /schedules/{id}/resume // ActionShowSchedule reads one schedule with its last and next fire; // ActionTriggerSchedule runs it now. ActionShowSchedule Action = "schedules.show" // GET /schedules/{id} ActionTriggerSchedule Action = "schedules.trigger" // POST /schedules/{id}/trigger // ActionForceComplete and ActionForceFail end a run from outside // (workflow.RunForcer). ActionForceComplete Action = "runs.force_complete" // POST /runs/{id}/complete ActionForceFail Action = "runs.force_fail" // POST /runs/{id}/fail // ActionShowOutput reads a run's captured step output (WithOutputs). ActionShowOutput Action = "runs.output" // GET /runs/{id}/output // ActionRetryRuns and ActionDeleteRuns retry or forget every run a // filter matches (Admin.RetryAll, Admin.DeleteAll), dry runs included. ActionRetryRuns Action = "runs.retry_all" // POST /runs/retry ActionDeleteRuns Action = "runs.delete_all" // POST /runs/delete // ActionMetrics reads run-end history for the charts (WithMetrics). ActionMetrics Action = "metrics.read" // GET /metrics // ActionListNodes lists the online nodes; ActionHealth reads the // client's health report. ActionListNodes Action = "nodes.list" // GET /nodes ActionHealth Action = "health.read" // GET /health // ActionRescheduleRun moves a delayed run; ActionCancelStep withdraws // one step of a run. ActionRescheduleRun Action = "runs.reschedule" // POST /runs/{id}/reschedule ActionCancelStep Action = "runs.cancel_step" // POST /runs/{id}/steps/{step}/cancel // ActionUI is loading the web UI's page and its files. The UI's data // comes from the routes above, each authorized with its own action, // so a caller allowed only ActionUI sees an empty screen. ActionUI Action = "ui.read" // GET /, GET /ui, GET /ui/{file} )
The actions, one per route.
type Admin ¶
type Admin struct {
// contains filtered or unexported fields
}
Admin is the operations an admin screen or a CLI needs, over any workflow.Client. It holds no state of its own: every call goes to the client, so several Admins over one store agree.
Each operation checks the client's capabilities first and then its optional interface, and refuses with a *workflow.UnsupportedError (which matches workflow.ErrUnsupported) naming the operation and the missing capabilities when either is absent. The capability is checked because a wrapper implements every optional interface and refuses at the call when the client it wraps lacks it.
func New ¶
New returns an Admin over c. It never fails: what c cannot do is reported by Support and refused per call.
func (*Admin) CancelAll ¶
CancelAll cancels every run matching q and returns how many; see workflow.BulkCanceller for which statuses an empty q.Statuses means.
func (*Admin) CancelStep ¶
CancelStep withdraws one step of a run; see workflow.StepCanceller for what the run does next.
func (*Admin) CountRuns ¶
CountRuns pages through the runs q matches, at q's page size, and returns their count and each page's cursor. q.Cursor must be empty. It reads at most the max scan (WithMaxScan) of runs, so a screen pays a bounded cost; a driver that could count in one query is still asked page by page, because RunQuerier is the only portable way to ask.
func (*Admin) DeleteAll ¶
func (a *Admin) DeleteAll(ctx context.Context, q workflow.RunQuery, opts BulkOptions) (BulkResult, error)
DeleteAll forgets every terminal run q matches, as Delete does one. The filter must name at least one field (workflow, queue, statuses, labels, label prefixes, from or to), so "delete everything" is never an empty request: an operator who means every finished run says statuses=succeeded,failed,cancelled. q.Statuses may list terminal statuses only; empty means every terminal status the client reports (discarded only with CapRunStates). It needs OpDeleteAll: RunQuerier and RunDeleter. Runs that changed between listing and deleting are Skipped; any other error stops the call.
func (*Admin) Discarded ¶
Discarded lists runs a step ended with workflow.Discard, matching q, beside Failures: the run's Error keeps the reason. q.Statuses must be empty. It needs CapRunStates: on a driver without it a discarded run reads failed or cancelled and shows up there instead.
func (*Admin) Failures ¶
Failures lists failed runs matching q. q.Statuses must be empty: the operation sets it to StatusFailed, and a caller that asks for other statuses wants Runs. Discarded runs are not failures (a step chose to drop them) and are listed by Discarded.
func (*Admin) ForceComplete ¶
ForceComplete ends a run as succeeded without running its remaining steps; see workflow.RunForcer. A run that is not terminal, or is failed, is accepted; another terminal run returns ErrWrongState.
func (*Admin) ForceFail ¶
ForceFail ends a run that is not terminal as failed, with reason in its error and no retry; see workflow.RunForcer. A reason longer than MaxForceReason bytes is refused with ErrInvalid.
func (*Admin) HasMetrics ¶
HasMetrics reports whether a MetricsSource is wired (WithMetrics).
func (*Admin) HasOutputs ¶
HasOutputs reports whether an OutputSource is wired (WithOutputs).
func (*Admin) Health ¶
func (a *Admin) Health(ctx context.Context) workflow.HealthReport
Health is the client's health report (workflow.CheckHealth). It never fails: a client without HealthChecker is reported live and ready with a check saying it cannot tell. An admin screen shows it; a liveness or readiness probe should call workflow.CheckHealth on the app's own unauthenticated probe route instead of going through the admin API.
func (*Admin) Metrics ¶
func (a *Admin) Metrics(ctx context.Context, q MetricsQuery) (MetricsSeries, error)
Metrics returns run ends over q from the wired MetricsSource. Without WithMetrics it returns an error matching workflow.ErrUnsupported.
func (*Admin) Nodes ¶
Nodes lists the nodes that are online, sorted by ID; see workflow.NodeLister for when a node drops out.
func (*Admin) PauseSchedule ¶
PauseSchedule stops the schedule id firing without removing it; see workflow.SchedulePauser. An empty id is refused with ErrInvalid.
func (*Admin) PreviewCancelAll ¶
PreviewCancelAll is CancelAll's dry run: it counts the runs CancelAll would cancel (workflow.CancelStatuses) and samples them, changing nothing. It needs OpCancelAll and OpRuns. The count is a moment's snapshot: runs that start before the real call are no longer cancelled by its default statuses.
func (*Admin) QueueDepths ¶
QueueDepths returns QueueStats as queue name to depth. Its signature matches otel.QueueDepthFunc, so a method value plugs straight into the OpenTelemetry queue-depth gauge without either module importing the other.
func (*Admin) QueueStats ¶
func (a *Admin) QueueStats(ctx context.Context) (QueueReport, error)
QueueStats counts the queued runs per queue and finds the oldest one in each, by paging RunQuerier over StatusQueued. It reads at most the max scan (WithMaxScan) of runs, so its cost is bounded; a driver that can count with one query is still asked page by page, because RunQuerier is the only portable way to ask.
Depth and the oldest run come from that scan on every driver, so they count the queue across every process sharing the store. On a client that implements workflow.WorkerScaler and declares CapWorkerScale, QueueLoads adds this client's own worker count and running attempts per queue (the state a workflow.BalanceLoop acts on); a QueueLoads error fails the call, since a report that silently drops the worker counts would read as "no workers".
func (*Admin) Reschedule ¶
func (a *Admin) Reschedule(ctx context.Context, ref workflow.RunRef, at time.Time) (workflow.RunRef, error)
Reschedule moves a delayed run that has not started to at; see workflow.RunRescheduler. A zero at is refused here (the interface reads it as now, which an operator should say on purpose with the current time).
func (*Admin) Resume ¶
Resume removes the pause on t; Resume(workflow.PauseAll()) removes every pause.
func (*Admin) ResumeSchedule ¶
ResumeSchedule lets a paused schedule fire again.
func (*Admin) Retry ¶
Retry re-runs a failed run from the step that failed; see workflow.RunRetrier for what is kept.
func (*Admin) RetryAll ¶
func (a *Admin) RetryAll(ctx context.Context, q workflow.RunQuery, opts BulkOptions) (BulkResult, error)
RetryAll retries every failed run q matches, from the step that failed, as Retry does one; see workflow.RunRetrier. q.Statuses must be empty or exactly [failed], since Retry refuses every other status; an empty filter retries every failed run, up to the bound. It needs OpRetryAll: RunQuerier to list and RunRetrier to act. The runs are listed first and then retried one by one, so a run that changes in between is counted in Skipped; any other error stops the call and is returned with the result so far.
func (*Admin) Run ¶
Run returns one run. An unknown ref returns workflow.ErrNotFound. With WithRunQuerier, a run the querier does not hold is looked up on the client when the client can list runs.
func (*Admin) RunOutput ¶
RunOutput returns the captured output of the run ref's attempts, oldest first. Without WithOutputs it returns an error matching workflow.ErrUnsupported. A ref with only a ScheduledID has not run yet and has no output.
func (*Admin) Runs ¶
Runs lists runs matching q, one page at a time, newest enqueue first. Pass the returned RunPage.Next as q.Cursor for the next page.
func (*Admin) Schedule ¶
func (a *Admin) Schedule(ctx context.Context, id workflow.ScheduleID) (workflow.ScheduleInfo, error)
Schedule returns one live schedule by ID, from the same list Schedules reads, with its next fire time and, on a client with CapScheduleHistory, its last fire (ScheduleInfo.Last). An empty id is refused with ErrInvalid and an unknown one returns ErrNotFound.
func (*Admin) Schedules ¶
Schedules lists the live schedules with their next fire time, sorted by workflow, then name.
func (*Admin) Support ¶
func (a *Admin) Support() []OperationSupport
Support reports, for every operation in Operations order, whether the client supports it. A UI hides what is unsupported instead of showing a button that always fails.
func (*Admin) TriggerSchedule ¶
func (a *Admin) TriggerSchedule(ctx context.Context, id workflow.ScheduleID) (workflow.RunRef, error)
TriggerSchedule runs the schedule id now, with its payload and labels, and returns the run it enqueued; see workflow.ScheduleTriggerer. The schedule's overlap policy still applies: with OverlapSkip and its previous run unfinished it enqueues nothing and returns an error matching workflow.ErrOverlap. An empty id is refused with ErrInvalid.
type Alert ¶
type Alert struct {
Kind AlertKind
Queue string
// Depth and Wait are the queue's values at the poll that raised the
// alert (Wait is its OldestWait).
Depth int
Wait time.Duration
// Threshold is the one crossed.
Threshold Threshold
// Resolved is true on the alert that reports the queue back under the
// threshold.
Resolved bool
// Truncated is copied from the QueueReport: Depth is a lower bound.
Truncated bool
// At is the poll time on the Admin's clock.
At time.Time
}
Alert is one threshold crossing, or the end of one.
type Alerter ¶
Alerter receives alerts. Send them to a chat channel, a pager or a log. Alert runs on the Monitor's goroutine, one call at a time; a slow Alerter delays the next poll, and a panic is recovered and reported to the error handler.
type AlerterFunc ¶
AlerterFunc adapts a function to Alerter.
type AttemptOutput ¶
type AttemptOutput struct {
Step string
Attempt int
// Started and Finished bound the attempt, on the capturing clock.
Started, Finished time.Time
// Text is the captured output (middleware.FormatOutput's format).
Text string
// Truncated is true when Text was cut to the source's size bound.
Truncated bool
// Error is the attempt's error message; empty when it succeeded.
Error string
}
AttemptOutput is what one step attempt wrote, as an admin screen shows it: the text middleware.CaptureOutput captured, how the attempt ended and when.
type Auth ¶
type Auth[P any] struct { // Identify returns the caller of r, from a session cookie or a bearer // token. An error means anonymous: the request is refused with 401 // before any of it is parsed. Identify func(r *http.Request) (P, error) // Authorize allows req by returning nil. Any error, or a panic, refuses // it with 403, so an Authorizer that fails, fails closed. Authorize func(ctx context.Context, req Request[P]) error }
Auth is how the handler decides who may do what. Both functions are required: a handler with either one nil refuses every request, so the zero Auth is closed.
type BulkOptions ¶
type BulkOptions struct {
// DryRun lists what the call would act on and changes nothing.
DryRun bool
// Max lowers the Admin's bound (WithMaxBulk) for this call; zero, or a
// value above that bound, keeps the bound.
Max int
}
BulkOptions shape one bulk call.
type BulkResult ¶
type BulkResult struct {
// Matched is how many runs the filter matched, up to the bound.
Matched int
// Done is how many runs were retried or deleted; zero on a dry run.
Done int
// Skipped is how many matched runs were left alone because they
// changed between the listing and the action: no longer in a state
// the action accepts (workflow.ErrWrongState), gone
// (workflow.ErrNotFound), or, for a retry, their idempotency key held
// by another unfinished run (workflow.ErrDuplicate).
Skipped int
// Truncated is true when more runs match than the bound: repeat the
// call for the rest.
Truncated bool
// Sample is the first BulkSample matched refs, newest first.
Sample []workflow.RunRef
}
BulkResult is what a bulk call matched and did.
type HandlerOption ¶
type HandlerOption func(*handlerConfig)
HandlerOption configures NewHandler.
func WithLogger ¶
func WithLogger(l *slog.Logger) HandlerOption
WithLogger logs internal errors (the ones answered with a bare 500) and authorization panics. Nil, the default, discards them.
func WithPayloads ¶
func WithPayloads() HandlerOption
WithPayloads includes each run's payload (base64) in run responses. By default it is left out, because payloads can carry personal data and an admin screen is seen by more people than the code that wrote them.
func WithStacks ¶
func WithStacks() HandlerOption
WithStacks includes the stack trace of each attempt error (workflow.AttemptError.Stack) in run responses. By default it is left out: a stack names source files and functions, which an admin screen does not need to show to everyone who can read run errors.
type MemoryMetrics ¶
type MemoryMetrics struct {
// contains filtered or unexported fields
}
MemoryMetrics is an in-process, bounded MetricsSource: it counts run ends by status, per workflow, in fixed buckets, fed by the run-finished hook (register Observe with workflow.RunNotifier.OnRunFinished before Start). Each process counts the runs it was told about, so on a durable driver where every process receives every end it shows the fleet, and where ends are delivered to one process it shows that process; it starts empty on every restart (Series.Since says when). An app that needs history across restarts wires its own MetricsSource.
func NewMemoryMetrics ¶
func NewMemoryMetrics(cfg MemoryMetricsConfig) *MemoryMetrics
NewMemoryMetrics returns an empty MemoryMetrics.
func (*MemoryMetrics) Observe ¶
func (m *MemoryMetrics) Observe(_ context.Context, ev workflow.RunEvent)
Observe counts one run end. Its signature is workflow.RunFinishedFunc, so it registers with OnRunFinished directly. A non-terminal status is ignored; a repeat of a recent (run, status, finish time) is counted once.
func (*MemoryMetrics) Series ¶
func (m *MemoryMetrics) Series(_ context.Context, q MetricsQuery) (MetricsSeries, error)
Series sums the buckets into q's steps. q.Step is rounded up to a multiple of the bucket, and From down to a bucket boundary, so every bucket falls in exactly one point.
type MemoryMetricsConfig ¶
type MemoryMetricsConfig struct {
// Bucket is the resolution counts are kept at (DefaultMetricsBucket).
Bucket time.Duration
// Keep is how long counts are kept (DefaultMetricsKeep).
Keep time.Duration
// Dedupe is how many recent (run, status) pairs are remembered so a
// redelivered end is counted once (DefaultMetricsDedupe). A repeat
// arriving after that many other ends is counted again.
Dedupe int
// Clock prunes old buckets and stamps an event without a FinishedAt;
// nil means clock.Real.
Clock clock.Clock
}
MemoryMetricsConfig configures NewMemoryMetrics. Zero or less in a duration or count keeps its default.
type MemoryOutputLimits ¶
type MemoryOutputLimits struct {
// Runs is how many runs' output is kept; past it the run whose output
// arrived first is forgotten. Default DefaultOutputRuns.
Runs int
// PerRun is how many attempts are kept per run, the newest. Default
// DefaultOutputPerRun.
PerRun int
// Bytes bounds one attempt's Text; past it the text is cut and marked
// Truncated. Default DefaultOutputBytes.
Bytes int
}
MemoryOutputLimits bounds a MemoryOutputs. Zero or less in a field keeps its default.
type MemoryOutputs ¶
type MemoryOutputs struct {
// contains filtered or unexported fields
}
MemoryOutputs keeps captured output in process memory, bounded by MemoryOutputLimits. It is both ends: a middleware.OutputSink for CaptureOutput and an OutputSource for WithOutputs. It sees only the attempts that ran in this process, so it suits one process or a development setup; a fleet stores output somewhere shared and writes its own OutputSource over it.
func NewMemoryOutputs ¶
func NewMemoryOutputs(lim MemoryOutputLimits) *MemoryOutputs
NewMemoryOutputs returns an empty MemoryOutputs.
func (*MemoryOutputs) Deliver ¶
func (m *MemoryOutputs) Deliver(_ context.Context, out middleware.Output) error
Deliver stores one attempt's output under its run. Output without StepInfo names no run and is refused.
func (*MemoryOutputs) RunOutput ¶
func (m *MemoryOutputs) RunOutput(_ context.Context, run workflow.RunID) ([]AttemptOutput, error)
RunOutput returns a copy of the run's stored attempts, oldest first.
type MetricsPoint ¶
type MetricsPoint struct {
At time.Time
Succeeded int64
Failed int64
Cancelled int64
// Discarded is counted only from a driver with CapRunStates; on
// others a discarded run arrives as failed or cancelled.
Discarded int64
}
MetricsPoint is the run ends in [At, At+Step), by terminal status.
func (MetricsPoint) Total ¶
func (p MetricsPoint) Total() int64
Total is every run end in the point.
type MetricsQuery ¶
type MetricsQuery struct {
From, To time.Time
// Step is the width of one point. A source with a coarser resolution
// rounds it up to a multiple of its own (MetricsSeries.Step says what
// was used).
Step time.Duration
// Workflow narrows the counts to one workflow; empty counts all.
Workflow string
}
MetricsQuery asks for run ends over [From, To) in steps of Step.
type MetricsSeries ¶
type MetricsSeries struct {
Step time.Duration
Points []MetricsPoint
// Since is the earliest time the source has data for; points before
// it read zero because nothing was recorded, not because nothing ran.
// Zero when the source cannot say.
Since time.Time
}
MetricsSeries is the answer to a MetricsQuery: one point per step from From to To, zero-filled where nothing ended, oldest first.
type MetricsSource ¶
type MetricsSource interface {
Series(ctx context.Context, q MetricsQuery) (MetricsSeries, error)
}
MetricsSource answers historical run-end counts for the charts screen. The library keeps no history of its own: an app with a metrics store (Prometheus, a warehouse table, an OpenTelemetry backend) adapts it with MetricsSourceFunc, and MemoryMetrics is the in-process default for an app that has none. Implementations must return one point per step over the query, zero-filled.
type MetricsSourceFunc ¶
type MetricsSourceFunc func(ctx context.Context, q MetricsQuery) (MetricsSeries, error)
MetricsSourceFunc adapts a function to MetricsSource.
func (MetricsSourceFunc) Series ¶
func (f MetricsSourceFunc) Series(ctx context.Context, q MetricsQuery) (MetricsSeries, error)
Series calls f.
type Monitor ¶
type Monitor struct {
// contains filtered or unexported fields
}
Monitor polls Admin.QueueStats on the Admin's clock and calls an Alerter when a queue crosses a Threshold, and again when it is back under. It keeps which alerts are open in memory only, so two Monitors on two nodes each alert; run one per deployment.
func NewMonitor ¶
func NewMonitor(a *Admin, alerter Alerter, thresholds []Threshold, opts ...MonitorOption) (*Monitor, error)
NewMonitor returns a Monitor over a. It refuses, with workflow.ErrInvalid, a nil Alerter, no thresholds, a negative field, a threshold that checks nothing, an invalid queue name or two thresholds for one queue; and, with an UnsupportedError, a client QueueStats cannot work on.
type MonitorOption ¶
type MonitorOption func(*Monitor)
MonitorOption configures NewMonitor.
func WithErrorHandler ¶
func WithErrorHandler(fn func(error)) MonitorOption
WithErrorHandler receives poll errors and recovered Alerter panics. Nil, the default, drops them; Run keeps polling either way.
func WithInterval ¶
func WithInterval(d time.Duration) MonitorOption
WithInterval sets how often Run polls. Zero or less keeps DefaultMonitorInterval.
func WithRepeat ¶
func WithRepeat(d time.Duration) MonitorOption
WithRepeat re-sends an alert every d while its queue stays over the threshold. Zero, the default, alerts once on crossing and once on resolving.
type Operation ¶
type Operation string
Operation names one thing Admin can do. The names are part of the JSON the capabilities route returns, so they never change once released.
const ( OpRuns Operation = "runs" // list and search runs (RunQuerier) OpRun Operation = "run" // show one run (RunQuerier) OpFailures Operation = "failures" // list failed runs (RunQuerier) OpDiscarded Operation = "discarded" // list discarded runs (RunQuerier, CapRunStates) OpQueues Operation = "queues" // queue depth and oldest wait (RunQuerier) OpRetry Operation = "retry" // retry a failed run from its failed step (RunRetrier) OpDelete Operation = "delete" // forget a finished run (RunDeleter) OpCancel Operation = "cancel" // cancel one run (Client.Cancel) OpCancelAll Operation = "cancel_all" // cancel every run matching a filter (BulkCanceller) OpPause Operation = "pause" // pause, resume and list pauses (Pauser) OpSchedules Operation = "schedules" // list schedules with their next fire time (ScheduleLister) // OpSchedulePause pauses and resumes one schedule (SchedulePauser). OpSchedulePause Operation = "schedule_pause" // OpScheduleTrigger runs one schedule now, outside its cron // (ScheduleTriggerer). OpScheduleTrigger Operation = "schedule_trigger" // OpForce ends a run from outside: force-complete or force-fail // (RunForcer). OpForce Operation = "force" // OpRetryAll retries every failed run a filter matches (RunQuerier // and RunRetrier). OpRetryAll Operation = "retry_all" // OpDeleteAll forgets every finished run a filter matches (RunQuerier // and RunDeleter). OpDeleteAll Operation = "delete_all" // OpNodes lists the nodes that are online (NodeLister). OpNodes Operation = "nodes" // OpHealth reads the client's health report (workflow.CheckHealth); // every client supports it, one without HealthChecker saying it // cannot tell. OpHealth Operation = "health" // OpReschedule moves a delayed run to a new time (RunRescheduler). OpReschedule Operation = "reschedule" // OpCancelStep withdraws one step of a run (StepCanceller). OpCancelStep Operation = "cancel_step" )
The operations. Each needs the capabilities and optional interface listed in Requirements.
type OperationSupport ¶
type OperationSupport struct {
Operation Operation
Supported bool
// Missing is the capabilities the client lacks for the operation;
// zero when Supported. For OpPause it lists both pause capabilities
// when the client has neither. It is zero for an unsupported
// operation that needs no capability, only an interface; Reason then
// says so.
Missing workflow.Caps
// Reason says why an operation that needs no capability is
// unsupported (its interface is absent); empty otherwise.
Reason string
}
OperationSupport says whether the client supports one operation, and which capabilities it lacks when it does not.
type Option ¶
type Option func(*Admin)
Option configures New.
func WithClock ¶
WithClock sets the clock QueueStats measures wait times with and the Monitor polls on. Nil means clock.Real.
func WithMaxBulk ¶
WithMaxBulk bounds how many runs one RetryAll or DeleteAll call acts on. Zero or less keeps DefaultMaxBulk.
func WithMaxScan ¶
WithMaxScan bounds how many queued runs QueueStats reads per call; past it the stats are marked Truncated. Zero or less keeps DefaultMaxScan.
func WithMetrics ¶
func WithMetrics(src MetricsSource) Option
WithMetrics lets Admin.Metrics, the metrics route and the UI's charts read run-end history from src. Without it they report that no metrics source is wired.
func WithOutputs ¶
func WithOutputs(src OutputSource) Option
WithOutputs lets Admin.RunOutput, the run output route and the UI's run page read captured output from src. Without it they report that no output store is wired. Output can hold whatever a step logged, so wiring a source is the opt-in that shows it to admin users.
func WithRunQuerier ¶ added in v0.1.1
func WithRunQuerier(rq workflow.RunQuerier, caps workflow.Caps) Option
WithRunQuerier makes the read operations (runs, run, failures, discarded, queue stats, counts) read rq instead of the client, typically a run recorder's store (recorder.Store), whose history outlives the driver's own and covers a driver without a run query. caps is what rq's queries honour (recorder.QueryCaps for a recorder store); a query filter rq lacks is refused, as it would be by the client. Actions (retry, cancel, delete, force, the bulk ones and the listings they act on) still go to the client, which owns the runs. Run falls back to the client's own RunQuerier for a run rq does not hold, so a run the recorder never saw still opens. A nil rq keeps the client.
type OutputSource ¶
type OutputSource interface {
RunOutput(ctx context.Context, run workflow.RunID) ([]AttemptOutput, error)
}
OutputSource returns the captured output of one run's attempts, oldest first. The library keeps no output itself: CaptureOutput hands each attempt's output to a sink the app chooses, so where it is kept (a table, an object store, a log service) is the app's decision, and an OutputSource is how the admin reads it back. A run with no captured output returns an empty slice, not an error.
type OutputSourceFunc ¶
OutputSourceFunc adapts a function to OutputSource: the glue that maps another module's output store (one that does not import admin) to AttemptOutput.
func (OutputSourceFunc) RunOutput ¶
func (f OutputSourceFunc) RunOutput(ctx context.Context, run workflow.RunID) ([]AttemptOutput, error)
RunOutput calls f.
type QueueReport ¶
type QueueReport struct {
Queues []QueueStat
// Truncated is true when more than the max scan of queued runs exist:
// every Depth is then a lower bound and the oldest run may be missed.
Truncated bool
// At is when the report was taken, on the Admin's clock.
At time.Time
// Workers is true when the client implements workflow.WorkerScaler
// and declares CapWorkerScale: each QueueStat then carries Workers
// and Running, and every queue the client serves is listed, even
// with nothing queued.
Workers bool
}
QueueReport is every queue with at least one queued run, sorted by name.
type QueueStat ¶
type QueueStat struct {
// Queue is the queue name.
Queue string
// Depth is how many runs wait for a worker (StatusQueued), including
// runs held by a pause. Delayed runs (StatusScheduled) are not counted
// until they are due.
Depth int
// OldestEnqueuedAt is the enqueue time of the oldest queued run.
OldestEnqueuedAt time.Time
// OldestWait is how long that run has waited, on the Admin's clock.
OldestWait time.Duration
// Workers and Running are this client's worker count for the queue
// and how many attempts those workers run now. They are filled only
// when QueueReport.Workers is true; see QueueStats.
Workers int
Running int
}
QueueStat is one queue's backlog at the moment QueueStats read it.
type Request ¶
type Request[P any] struct { Who P Action Action Resource Resource // HTTP is the request, for headers an Authorizer needs (a tenant // header, the remote address). Its body has been read; do not read it. HTTP *http.Request }
Request is one HTTP call as the Authorizer sees it: who, which action, which resource. P is the app's own principal type, whatever Identify returns.
type Resource ¶
type Resource struct {
// Run is the run for ActionShowRun, ActionRetryRun, ActionCancelRun,
// ActionDeleteRun, ActionForceComplete, ActionForceFail,
// ActionShowOutput, ActionRescheduleRun and ActionCancelStep.
Run workflow.RunRef
// Query is the filter for ActionListRuns, ActionListFailures,
// ActionCancelRuns, ActionRetryRuns and ActionDeleteRuns, so an
// Authorizer can scope a caller to its own workflows, queues or labels
// (a tenant label, for example).
Query *workflow.RunQuery
// DryRun is true when a bulk action (ActionCancelRuns,
// ActionRetryRuns, ActionDeleteRuns) only previews, so an Authorizer
// can let more people preview than act.
DryRun bool
// Metrics is the query for ActionMetrics.
Metrics *MetricsQuery
// Step is the step for ActionCancelStep.
Step string
// At is the new time for ActionRescheduleRun.
At time.Time
// Pause is the target for ActionPause and ActionResume.
Pause *workflow.PauseTarget
// Schedule is the schedule for ActionPauseSchedule,
// ActionResumeSchedule, ActionShowSchedule and ActionTriggerSchedule.
Schedule workflow.ScheduleID
// Reason is the operator's reason for ActionForceFail, so an
// Authorizer can require one or write it to an audit log.
Reason string
// File is the UI file for ActionUI ("index.html", "app.js", "app.css").
File string
}
Resource is what an action is about. The field the action uses is set; the others are zero.
type RunCount ¶
type RunCount struct {
// Total is how many runs match, up to the max scan.
Total int
// Truncated is true when more runs match than the max scan read:
// Total is then a lower bound and the last pages are missing.
Truncated bool
// Cursors holds the cursor that starts each page of the query's
// page size: Cursors[0] is "" (the first page), Cursors[i] starts
// page i+1. Pass one as RunQuery.Cursor with the same filters and
// Limit to open that page.
Cursors []string
}
RunCount is how many runs match a query and where each page of it starts, for a screen that shows "1–50 of N" and numbered pages over a driver whose cursors are opaque.
type Threshold ¶
type Threshold struct {
// Queue is the queue it applies to. Empty is the default for every
// queue that has no Threshold of its own.
Queue string
// MaxDepth alerts when Depth is above it.
MaxDepth int
// MaxWait alerts when OldestWait is above it.
MaxWait time.Duration
}
Threshold is when a queue is in trouble: too deep, or its oldest run waiting too long. A zero field is not checked.