Documentation
¶
Index ¶
- Constants
- func CheckPassword(plaintext, hashed string) bool
- func FetchWorkersForWatchdog(ctx context.Context, db *sql.DB) ([]watchdog.WorkerInfo, error)
- func GateExit() func(int)
- func GateStdout() io.Writer
- func HttpServer(cfg config.Config) (tls.Certificate, string, http.Handler, error)
- func IsAdmin(ctx context.Context) bool
- func LoadEmbeddedCertificates() (tls.Certificate, string, error)
- func LogoutHandler(w http.ResponseWriter, r *http.Request)
- func NewLogin(s *taskQueueServer) http.HandlerFunc
- func Serve(cfg config.Config, ctx context.Context, cancel context.CancelFunc) error
- func SetGateExit(f func(int))
- func SetGateStdout(w io.Writer)
- func StatsDBForTest() *sql.DB
- func TriggerStuckDeleteCleanup() error
- func WorkerUpgradeStatus(workerArch, workerCommit, serverCommit string) string
- type Accumulator
- type AuthenticatedUser
- type Job
- type ObjectiveDefinition
- type ParamSpec
- type QualityDefinition
- type QualityResult
- type StepAgg
- type StepAggSnapshot
- type StepCounterSnapshot
- type StepStatsAgg
- func (a *StepStatsAgg) Adjust(workflowID, stepID int32, fn func(*StepAgg))
- func (a *StepStatsAgg) EnsureStep(workflowID, stepID int32)
- func (a *StepStatsAgg) FlushLoop(interval time.Duration, stop <-chan struct{})
- func (a *StepStatsAgg) FlushOnce()
- func (a *StepStatsAgg) MarkDirty(workflowID, stepID int32)
- func (a *StepStatsAgg) MarkDirtyLocked(workflowID, stepID int32)
- func (a *StepStatsAgg) Reconcile(db *sql.DB)
- func (a *StepStatsAgg) RemoveStep(stepID int32) int32
- func (a *StepStatsAgg) RemoveWorkflow(workflowID int32)
- func (a *StepStatsAgg) SnapshotCounters() map[int32]map[int32]StepAggSnapshot
- type TaskUpdateBroadcast
- type TransitionExtras
- type TransitionRequest
- type TransitionResult
Constants ¶
const ( ErrClassAuth = "auth" ErrClassQuota = "quota" ErrClassCapacity = "capacity" ErrClassUnsupportedFlavor = "unsupported_flavor" ErrClassTransient = "transient" ErrClassUnknown = "unknown" )
Job error classes surfaced on the job queue and used by the UI banner / CLI. See migrations/000028_job_error_class.up.sql for the schema and 2026-05-05 incident for the motivating story.
const ( UpgradeStatusUpToDate = "up_to_date" UpgradeStatusNeedsUpgrade = "needs_upgrade" UpgradeStatusUnsupportedArch = "unsupported_arch" UpgradeStatusUnknown = "unknown" )
const DefaultAssignTrigger int32 = 500 // 5 sec
const SupportedUpgradeArch = "linux/amd64"
SupportedUpgradeArch is the only arch Phase II auto-apply will target. Phase I (visibility) reports status for every arch but distinguishes `unsupported_arch` from `needs_upgrade` so the UI/CLI can hide the "redeploy this one" call-to-action where redeployment is gated on multi-arch binary distribution we haven't built yet.
Variables ¶
This section is empty.
Functions ¶
func CheckPassword ¶
func FetchWorkersForWatchdog ¶
FetchWorkersForWatchdog reads workers and builds WorkerInfo list
func GateExit ¶ added in v0.8.1
func GateExit() func(int)
GateExit returns the current exit hook used by the SIGUSR1 graceful drain. Tests use this to swap in a recorder, then restore on cleanup.
func GateStdout ¶ added in v0.8.1
GateStdout / SetGateStdout: same pattern for the stdout writer the gate uses to emit its drained-line JSON. Tests substitute a buffer.
func HttpServer ¶
func LoadEmbeddedCertificates ¶
func LoadEmbeddedCertificates() (tls.Certificate, string, error)
func LogoutHandler ¶
func LogoutHandler(w http.ResponseWriter, r *http.Request)
LogoutHandler clears the session_token cookie to log out the user.
func NewLogin ¶
func NewLogin(s *taskQueueServer) http.HandlerFunc
NewLogin returns an HTTP handler function to process login requests.
func SetGateExit ¶ added in v0.8.1
func SetGateExit(f func(int))
SetGateExit replaces the SIGUSR1 graceful-drain exit hook. Default is `os.Exit`. Tests must restore the previous value via t.Cleanup().
func SetGateStdout ¶ added in v0.8.1
func StatsDBForTest ¶ added in v0.9.1
StatsDBForTest returns the DB handle the live server is using.
func TriggerStuckDeleteCleanup ¶ added in v0.8.1
func TriggerStuckDeleteCleanup() error
TriggerStuckDeleteCleanup runs one synchronous tick of the stuck-delete janitor. The janitor's regular schedule (~5 min) is too slow for tests; this seam lets the integration test fire it on demand. Returns nil if the server hasn't started its janitor yet (the trigger is only armed inside startStuckDeleteCleanup).
func WorkerUpgradeStatus ¶ added in v0.8.1
WorkerUpgradeStatus computes the derived upgrade status from the worker's reported build identity and the server's own commit. Inputs:
- workerArch: GOOS/GOARCH the worker reported, e.g. "linux/amd64". Empty string = pre-Phase-I worker (didn't report).
- workerCommit: the worker's git SHA (full or short — compared as-is). Empty = pre-Phase-I worker.
- serverCommit: the server's own git SHA. If empty (e.g. a `dev` build that lost its VCS stamp), every worker reports `unknown` — we can't make a comparison either side.
Returns one of the UpgradeStatus* constants.
Types ¶
type Accumulator ¶
In-memory aggregation types for step/workflow stats Accumulator holds aggregation stats for durations/counts.
type AuthenticatedUser ¶
func GetUserFromContext ¶
func GetUserFromContext(ctx context.Context) *AuthenticatedUser
type Job ¶
type Job struct {
JobID int32
WorkerID int32
WorkerName string
ProviderID int32
ProviderName string
Region string
Flavor string
// HasGPU is the recruited flavor's has_gpu signal (looked up from
// the flavor catalog at job-creation time). The provider's Create
// uses it to swap to the operator's `gpu_image` config when set.
// false for CPU-only flavors and when the catalog row is missing.
HasGPU bool
// Per-recruiter cloud image overrides (NULL when the deploy came
// from CreateWorker without a recruiter hint, or when the
// recruiter row left both columns NULL). Format is
// provider-specific. Precedence at provider Create:
// gpu_image (if HasGPU) > image > scitq.yaml default.
Image *string
GPUImage *string
// Per-recruiter swapfile sizing override (see Recruiter.swap_proportion).
// NULL = provider.Create falls back to cfg.Scitq.SwapProportion.
// Carried through so the provider's cloud-init can bake the right
// `-swap` value into the worker's install command.
SwapProportion *float32
// Per-recruiter extra block storage (see Recruiter.extra_storage_gb /
// extra_storage_type). NULL / 0 = no extra volume. OVH only for now;
// the Azure provider returns an error when ExtraStorageGB > 0.
ExtraStorageGB *int32
ExtraStorageType *string
Action rune // "C", "D", "R"
Retry int
Timeout time.Duration
FromRecruiter bool
}
Job represents a task to be executed.
type ObjectiveDefinition ¶ added in v0.7.5
type ObjectiveDefinition struct {
Formula string `json:"formula"`
Direction string `json:"direction"` // "maximize" or "minimize"
}
ObjectiveDefinition defines one objective in a multi-objective quality definition.
type ParamSpec ¶
type ParamSpec struct {
Name string `json:"name"`
Type string `json:"type"`
Required bool `json:"required"`
// Default and Choices accept whatever JSON type the schema carries —
// boolean, integer, float, string, etc. The Python side
// (ParamSpec.schema in python/src/scitq2/param.py) emits native
// types, not stringified defaults, so a YAML `default: true` round-
// trips as the JSON literal `true`. Trying to unmarshal that into
// `string` aborted the whole UploadTemplate with "json: cannot
// unmarshal bool into Go struct field". Neither field is consumed
// downstream (UploadTemplate only validates Name/Type and then
// stores the schema verbatim), so the loose `interface{}` typing
// has no downside.
Default interface{} `json:"default"`
Help string `json:"help"` // Or `json:"description"` if your field is named so
Choices []interface{} `json:"choices,omitempty"`
}
type QualityDefinition ¶ added in v0.7.2
type QualityDefinition struct {
Variables map[string]string `json:"variables"` // name -> regex pattern (one capture group)
Formula string `json:"formula,omitempty"` // single-objective formula (backward compat)
Objectives []ObjectiveDefinition `json:"objectives,omitempty"` // multi-objective: list of {formula, direction}
}
QualityDefinition is the JSON structure stored in step.quality_definition.
func ParseQualityDefinition ¶ added in v0.7.2
func ParseQualityDefinition(jsonStr string) (*QualityDefinition, error)
ParseQualityDefinition parses a JSON quality definition string.
type QualityResult ¶ added in v0.7.2
type QualityResult struct {
Vars map[string]float64 `json:"vars"`
Score float64 `json:"score"` // primary score (first objective, or single formula)
Scores []float64 `json:"scores,omitempty"` // all objective scores (multi-objective only)
}
QualityResult holds extracted variables and computed scores.
func ExtractQuality ¶ added in v0.7.2
func ExtractQuality(def *QualityDefinition, stdout, stderr string) (*QualityResult, error)
ExtractQuality runs quality variable regexes against log text and evaluates the formula. Each regex captures the LAST match (for iterative programs that output metrics repeatedly).
type StepAgg ¶
type StepAgg struct {
// Task status counters
Waiting int32
Pending int32
Accepted int32
Running int32
Uploading int32
Succeeded int32
Failed int32
ReallyFailed int32 // tasks that have exhausted all retries and are failed
Retrying int32 // tasks currently being retried
RetryingSet map[int32]bool // set of task IDs currently retrying
Total int32
// Accumulators for durations
Download Accumulator
Upload Accumulator
SuccessRun Accumulator
FailRun Accumulator
// RunningTasks maps taskID to start time
RunningTasks map[int32]time.Time
// Time bounds (epoch seconds)
StartTime *int64
EndTime *int64
}
StepAgg aggregates statistics for a step within a workflow.
func NewStepAgg ¶
func NewStepAgg() *StepAgg
NewStepAgg creates a new StepAgg with sensible defaults.
type StepAggSnapshot ¶ added in v0.9.1
type StepAggSnapshot struct {
Waiting int32
Pending int32
Accepted int32
Running int32
Uploading int32
Succeeded int32
Failed int32
ReallyFailed int32
Retrying int32
}
StepAggSnapshot is a value-type copy of a StepAgg's status counters, exported so integration tests can compare the live aggregator state with a fresh Reconcile without holding the aggregator mutex. Excludes RunningTasks (timestamps drift at ns precision between delta path and SQL) and duration Accumulators (compared separately when the test cares) — this is the "primary counters" projection the property test's invariant is built on.
type StepCounterSnapshot ¶ added in v0.9.1
type StepCounterSnapshot struct {
StepId int32 `json:"stepId"`
TotalTasks int32 `json:"totalTasks"`
WaitingTasks int32 `json:"waitingTasks"`
PendingTasks int32 `json:"pendingTasks"`
AcceptedTasks int32 `json:"acceptedTasks"`
RunningTasks int32 `json:"runningTasks"`
UploadingTasks int32 `json:"uploadingTasks"`
SuccessfulTasks int32 `json:"successfulTasks"`
FailedTasks int32 `json:"failedTasks"` // hidden retried parents
ReallyFailedTasks int32 `json:"reallyFailedTasks"` // visible F only
RetryingTasks int32 `json:"retryingTasks"`
// Accumulators — sent as flat min/max/avg for the UI's duration
// column. UI already renders these; format matches today's
// StepStats proto response.
SuccessRun stepRunStats `json:"successRun"`
FailedRun stepRunStats `json:"failedRun"`
RunningRun stepRunStats `json:"runningRun"`
Download stepRunStats `json:"download"`
Upload stepRunStats `json:"upload"`
// RunningTaskStarts is the raw set of running tasks (task_id +
// run_started_at as fractional epoch seconds). The client rebuilds
// its local runningByStep map from this list on every snapshot so
// its 1 s timer computes min/avg/max from an up-to-date set — R
// transitions after mount are no longer stale.
RunningTaskStarts []stepRunningTask `json:"runningTaskStarts"`
StartTime *int64 `json:"startTime,omitempty"`
EndTime *int64 `json:"endTime,omitempty"`
}
StepCounterSnapshot is the wire shape shipped over WS. Field names match the UI's step-row fields so the client can `Object.assign` and be done. Kept flat (not nested under a `counters:` map) to keep the payload small and the client-side reactivity trivial.
type StepStatsAgg ¶
type StepStatsAgg struct {
// contains filtered or unexported fields
}
StepStatsAgg holds in-memory aggregated statistics for steps in workflows.
func NewStepStatsAgg ¶
func NewStepStatsAgg(db *sql.DB) (*StepStatsAgg, error)
NewStepStatsAgg creates a new StepStatsAgg with initialized internal maps.
func StatsAggForTest ¶ added in v0.9.1
func StatsAggForTest() *StepStatsAgg
StatsAggForTest returns the live server's aggregator or nil if no server has been started this process lifetime.
func (*StepStatsAgg) Adjust ¶ added in v0.7.5
func (a *StepStatsAgg) Adjust(workflowID, stepID int32, fn func(*StepAgg))
Adjust runs fn with the StepAgg for (workflowID, stepID) while holding the aggregator mutex. It lazily creates any missing map entries so call sites don't need to care about initialisation order. This is the canonical way to mutate counters from anywhere outside UpdateTaskStatus; using it eliminates the two classes of drift caused by unlocked direct map access:
- races with concurrent writers (int32 increments are not atomic)
- updates to a StepAgg that Reconcile has just orphaned by swapping data
func (*StepStatsAgg) EnsureStep ¶
func (a *StepStatsAgg) EnsureStep(workflowID, stepID int32)
EnsureStep guarantees an entry exists for (workflowID, stepID).
func (*StepStatsAgg) FlushLoop ¶ added in v0.9.1
func (a *StepStatsAgg) FlushLoop(interval time.Duration, stop <-chan struct{})
FlushLoop drives FlushOnce on a fixed cadence. Caller wires stop. Also runs the "running heartbeat" every runningHeartbeatInterval so steps with active tasks re-emit their RunningRun stats and the UI sees a live wall-clock counter rather than a frozen last-transition value.
func (*StepStatsAgg) FlushOnce ¶ added in v0.9.1
func (a *StepStatsAgg) FlushOnce()
FlushOnce collects the current dirty set under the mutex, snapshots each dirty step's counters, clears the set, then emits one WS message per workflow OUTSIDE the mutex. Zero traffic when nothing is dirty.
func (*StepStatsAgg) MarkDirty ¶ added in v0.9.1
func (a *StepStatsAgg) MarkDirty(workflowID, stepID int32)
MarkDirty is the entry point for callers that mutated the aggregator under a self-held lock (e.g. UpdateTaskStatus, which acquires a.mu.Lock() itself and manipulates a.data[wid][sid].Waiting-- etc. directly). Once every such site is migrated to Adjust or TransitionTask, this can go private again.
func (*StepStatsAgg) MarkDirtyLocked ¶ added in v0.9.1
func (a *StepStatsAgg) MarkDirtyLocked(workflowID, stepID int32)
MarkDirtyLocked is the same as MarkDirty but assumes the caller already holds a.mu — used inside UpdateTaskStatus which holds the lock across its aggregator work.
func (*StepStatsAgg) Reconcile ¶ added in v0.7.2
func (a *StepStatsAgg) Reconcile(db *sql.DB)
Reconcile rebuilds the in-memory stats from the database, fixing any drift between cached counters and actual task states.
func (*StepStatsAgg) RemoveStep ¶
func (a *StepStatsAgg) RemoveStep(stepID int32) int32
RemoveStep deletes a step entry by its stepID, scanning all workflows to locate it.
func (*StepStatsAgg) RemoveWorkflow ¶
func (a *StepStatsAgg) RemoveWorkflow(workflowID int32)
func (*StepStatsAgg) SnapshotCounters ¶ added in v0.9.1
func (a *StepStatsAgg) SnapshotCounters() map[int32]map[int32]StepAggSnapshot
SnapshotCounters returns a deep-copied snapshot of every step's primary counters. Test uses this on both the live aggregator and a fresh NewStepStatsAgg(db) and compares them.
type TaskUpdateBroadcast ¶
type TransitionExtras ¶ added in v0.9.1
TransitionExtras carries additional columns the transition writes. Populated as call sites migrate; add fields as more transitions need them (e.g. failure_class on F, run_started_at on R).
type TransitionRequest ¶ added in v0.9.1
type TransitionRequest struct {
TaskID int32
// New status letter: 'P'|'W'|'A'|'C'|'D'|'O'|'R'|'U'|'V'|'S'|'F'.
NewStatus string
// Optional guard: only transition if the CURRENT status equals
// this value. Empty = unconditional. Mirrors the "WHERE status='W'"
// inline guards several existing sites already use.
ExpectStatus string
// Extras applied in the same UPDATE. Nil pointers = leave alone.
// The struct is intentionally typed rather than a map so a new
// column touched by a transition forces an API extension rather
// than sneaking in via a stringly-typed side channel.
Extras TransitionExtras
}
TransitionRequest is the input to TransitionTask.
type TransitionResult ¶ added in v0.9.1
type TransitionResult struct {
// OldStatus is the row's status before the UPDATE. Empty when the
// row didn't exist or the guard (ExpectStatus) didn't match.
OldStatus string
// WorkflowID / StepID scraped from the row via the RETURNING
// clause. Used internally to bill the transition against the
// right aggregator bucket.
WorkflowID int32
StepID int32
// Applied is true iff the UPDATE affected a row (guard matched,
// row existed, row wasn't hidden).
Applied bool
}
TransitionResult reports what happened.
Source Files
¶
- assigntask.go
- auth.go
- error_class.go
- flavor_stats.go
- http.go
- jobqueue.go
- lifetime_sweep.go
- mcp.go
- mcp_docs.go
- metrics_refresh.go
- orphan_cleanup.go
- orphan_volumes.go
- quality.go
- server.go
- stepstats.go
- stuck_delete_cleanup.go
- syncproviders.go
- task_curves.go
- task_transitions.go
- test_seams.go
- version_compare.go
- worker_event.go
- worker_stats_history.go
- worker_stats_history_handlers.go
- workflow_chain.go
- workflow_chain_fire.go
- workflow_templates.go
Directories
¶
| Path | Synopsis |
|---|---|
|
Package metrics exposes scitq server state as Prometheus-format gauges + counters.
|
Package metrics exposes scitq server state as Prometheus-format gauges + counters. |
|
Package notifications sends user-facing convenience notifications (workflow completion, ...) to configured channels.
|
Package notifications sends user-facing convenience notifications (workflow completion, ...) to configured channels. |