controller

package
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 12, 2026 License: Apache-2.0 Imports: 25 Imported by: 0

Documentation

Overview

Package controller is the automated experiment loop: every tick it evaluates each running experiment (promote.Evaluate), persists the verdict for halo-server, and acts on it.

  • rollback: trip the traffic-plane kill switch immediately, open a PR setting status: paused with the evidence, notify.
  • promote: open a PR concluding the experiment (never merged), notify.
  • expired: open a PR concluding the experiment, notify.

Every action is taken at most once per experiment run (StateLog), so repeated ticks never duplicate PRs or notifications. Nothing here merges.

Index

Constants

View Source
const (
	SignatureHeader = "X-Halo-Signature"
	TimestampHeader = "X-Halo-Timestamp"
	// WebhookTolerance is the recommended receiver clock-skew/replay window.
	WebhookTolerance = 5 * time.Minute
)

Webhook authentication. The signature covers the timestamp so a captured request cannot be replayed later:

X-Halo-Timestamp: <unix seconds>
X-Halo-Signature: sha256=<hex HMAC-SHA256(secret, timestamp + "." + body)>

Receivers must recompute the HMAC over the raw body with the timestamp from X-Halo-Timestamp, compare in constant time, and reject timestamps more than WebhookTolerance from their clock (Verify does all of this).

View Source
const (
	ActionKill = "kill" // detail: "" = enforced, else why it is not known to be
	ActionPR   = "pr"
	// ActionNotify is recorded per channel as ActionNotify+":"+name (see Named);
	// a bare ActionNotify (older logs) means every channel was notified.
	ActionNotify = "notify"
	// ActionRollback marks that this run rolled back (detail: the report JSON);
	// the controller stops evaluating the experiment until it leaves "running".
	ActionRollback = "rollback"
	// ActionHold marks the once-per-run warning that an experiment is killed
	// and therefore not evaluated.
	ActionHold = "hold"
)

Actions the controller takes at most once per experiment run.

View Source
const Actor = "halo-controller"

Actor is the identity the controller records on kills and audit entries.

View Source
const DefaultTickTimeout = 2 * time.Minute

DefaultTickTimeout bounds one tick (all experiments).

View Source
const VerdictKilled promote.Verdict = "killed"

VerdictKilled is the pseudo-verdict of hold records and of the Event sent when a running experiment is found killed (it is not evaluated).

Variables

This section is empty.

Functions

func Evidence

func Evidence(rep promote.Report) string

Evidence renders a verdict report as Markdown for PR bodies.

func Sign

func Sign(secret []byte, ts int64, body []byte) string

Sign returns the SignatureHeader value for body sent at unix time ts.

func Verify

func Verify(secret []byte, h http.Header, body []byte, now time.Time, tolerance time.Duration) error

Verify is the receiver side: it checks the timestamp header is within tolerance of now and the signature header matches body.

Types

type Controller

type Controller struct {
	Org     func() (*policy.Org, error) // e.g. policy.Load of the served policy dir
	Metrics promote.MetricSource
	Writer  Writer     // nil: no PRs (logged)
	Kill    KillSwitch // nil: rollbacks are not enforced until the PR merges (logged, notified)
	// KillCaveat, when set, is why a successful Kill is not known to be
	// enforced by gateways (e.g. `halo controller run` without
	// --killswitch-served); messages report it instead of claiming enforcement.
	KillCaveat string
	// Kills is read each tick: a killed running experiment is not evaluated
	// (its evidence is all-control) and humans are told once. nil = no check.
	Kills        *KillStore
	Notifier     Notifier // nil: none
	State        *StateLog
	VerdictsPath string // "" = do not persist verdicts
	Clock        func() time.Time
	Log          *slog.Logger
	TickTimeout  time.Duration // default 2m; bounds one whole tick

	// RolloutStateDir holds one hash-chained state file per Rollout ("" =
	// rollouts are not driven); PolicyDir is where step scorecards are read.
	RolloutStateDir, PolicyDir string
	// contains filtered or unexported fields
}

Controller wires the loop. Org, Metrics and State are required.

func (*Controller) Run

func (c *Controller) Run(ctx context.Context, interval time.Duration)

Run ticks every interval until ctx ends. After a failed tick it retries sooner (15s, 30s, ... capped at interval) so a transient ClickHouse or git error doesn't delay a rollback by a whole interval.

func (*Controller) Tick

func (c *Controller) Tick(ctx context.Context) error

Tick evaluates every running experiment once and acts on the verdicts, then drives active rollouts (see tickRollouts). Per-experiment failures don't stop the others; all are joined in the result.

func (*Controller) WriteMetrics

func (c *Controller) WriteMetrics(w io.Writer)

WriteMetrics writes Prometheus text-format counters.

type Event

type Event struct {
	// Rollout is set for rollout events; Experiment is then its backing experiment.
	Rollout     string          `json:"rollout,omitempty"`
	Experiment  string          `json:"experiment"`
	Axis        string          `json:"axis,omitempty"`
	Verdict     promote.Verdict `json:"verdict"`
	Reason      string          `json:"reason"`
	Killed      bool            `json:"killed"` // traffic-plane kill switch is active for the experiment
	KillOutcome KillOutcome     `json:"killOutcome,omitempty"`
	KillError   string          `json:"killError,omitempty"` // detail for every non-enforced outcome
	PRURL       string          `json:"prURL,omitempty"`
	PRError     string          `json:"prError,omitempty"`
	Report      promote.Report  `json:"report"`
	At          time.Time       `json:"at"`
}

Event is what the controller tells humans about a verdict it acted on.

func (Event) Text

func (e Event) Text() string

Text is a one-paragraph human summary.

type GitWriter

type GitWriter struct {
	RepoDir string // git checkout (or any dir inside one)
	Subdir  string // policy root inside RepoDir
	Base    string // PR base branch; "" = repo default
	Opener  promote.PROpener
}

GitWriter implements Writer with promote.PlanStatus and a promote.PROpener (GHOpener: branch in a temp worktree, git push, gh pr create). It is the one "set experiment status + open PR" implementation: `halo controller run` uses it on the CI checkout; halo-server's server.GitPolicyWriter uses it on its dedicated clone after syncing that clone to origin (under its own lock). It never touches RepoDir's branch, index or working tree.

func (*GitWriter) ProposeRollout

func (w *GitWriter) ProposeRollout(ctx context.Context, plan func(string) (*promote.Change, error), branch, title string, body func(string) string) (string, error)

ProposeRollout implements RolloutWriter on this checkout.

func (*GitWriter) ProposeStatus

func (w *GitWriter) ProposeStatus(ctx context.Context, experiment, status, title, body string) (string, error)

type KillFunc

type KillFunc func(ctx context.Context, experiment, by, reason string) (bool, error)

KillFunc adapts a function (e.g. server.KillExperiment, which also audit-logs) to KillSwitch.

func (KillFunc) Kill

func (f KillFunc) Kill(ctx context.Context, experiment, by, reason string) (bool, error)

type KillOutcome

type KillOutcome string

KillOutcome is the typed result of the kill-switch step of a rollback.

const (
	KillEnforced      KillOutcome = "enforced"             // tripped and enforced by gateways
	KillNotGateway    KillOutcome = "not_gateway_evidence" // evidence not gateway-sourced; human decides
	KillNotConfigured KillOutcome = "not_configured"       // no kill switch wired
	KillRecorded      KillOutcome = "recorded_only"        // written, enforcement not guaranteed (KillCaveat)
	KillFailed        KillOutcome = "failed"               // the kill call errored
)

type KillRecord

type KillRecord struct {
	Experiment string    `json:"experiment"`
	Killed     bool      `json:"killed"`
	By         string    `json:"by"`
	Reason     string    `json:"reason,omitempty"`
	At         time.Time `json:"at"`
}

KillRecord is one line of killswitch.jsonl; the last line per experiment wins.

type KillStore

type KillStore struct {
	Now func() time.Time
	// contains filtered or unexported fields
}

KillStore is the persisted kill state halo-server serves to gateways: an append-only JSONL log (killswitch.jsonl in the data dir) replayed into memory. Appends from another process sharing the file (`halo controller run` against halo-server's data dir) are picked up on the next read. ponytail: whole-file re-read when the size changes; kills are rare. Cross-process appends rely on O_APPEND single-line writes (local filesystem).

func OpenKillStore

func OpenKillStore(dir string) (*KillStore, error)

OpenKillStore opens dir/killswitch.jsonl ("" = memory only).

func (*KillStore) Close

func (s *KillStore) Close() error

Close closes the log file.

func (*KillStore) Kill

func (s *KillStore) Kill(ctx context.Context, experiment, by, reason string) (bool, error)

Kill implements KillSwitch.

func (*KillStore) Killed

func (s *KillStore) Killed() ([]KillRecord, uint64, error)

Killed returns the active kills sorted by experiment, and the state version.

func (*KillStore) Unkill

func (s *KillStore) Unkill(ctx context.Context, experiment, by, reason string) (bool, error)

Unkill lifts a kill; changed is false when experiment was not killed.

type KillSwitch

type KillSwitch interface {
	// Kill marks experiment killed; changed is false when it already was.
	Kill(ctx context.Context, experiment, by, reason string) (changed bool, err error)
}

KillSwitch trips the traffic-plane kill switch for an experiment.

type Named

type Named interface {
	Name() string
}

Named is implemented by notifiers with a stable channel name. The controller records each channel's delivery separately (ActionNotify + ":" + name), so a failing webhook is retried without re-posting to Slack.

type Notifier

type Notifier interface {
	Notify(ctx context.Context, e Event) error
}

Notifier delivers events. Implementations must bound their own latency.

type Notifiers

type Notifiers []Notifier

Notifiers fans out to every notifier and joins their errors.

func NotifiersFromFiles

func NotifiersFromFiles(slackURLFile, webhookURLFile, webhookSecretFile string) (Notifiers, error)

NotifiersFromFiles builds notifiers from secret files (any path may be ""): a Slack incoming-webhook URL, a generic webhook URL and its HMAC secret.

func (Notifiers) Notify

func (ns Notifiers) Notify(ctx context.Context, e Event) error

type RolloutWriter

type RolloutWriter interface {
	ProposeRollout(ctx context.Context, plan func(dir string) (*promote.Change, error), branch, title string, body func(patch string) string) (prURL string, err error)
}

RolloutWriter opens a rollout PR: plan builds the change in the writer's checkout dir, body renders the PR body around its patch. GitWriter and server.GitPolicyWriter implement it. It must never merge.

type Slack

type Slack struct {
	URL    string
	Client *http.Client
}

Slack posts to a Slack(-compatible) incoming webhook.

func (Slack) Name

func (Slack) Name() string

Name implements Named.

func (Slack) Notify

func (s Slack) Notify(ctx context.Context, e Event) error

type StateLog

type StateLog struct {
	// contains filtered or unexported fields
}

StateLog remembers which (experiment, verdict, action) the controller has already done, so repeated ticks never open a second PR or send a second notification. An experiment's entries are forgotten once it is seen not running (a human merged the pause/conclude), so a later restart starts clean. ponytail: no compaction; one line per action per experiment run.

func OpenStateLog

func OpenStateLog(dir string) (*StateLog, error)

OpenStateLog opens dir/controller-state.jsonl ("" = memory only).

func (*StateLog) Close

func (l *StateLog) Close() error

Close closes the log file.

func (*StateLog) Done

func (l *StateLog) Done(exp, verdict, action string) (string, bool)

Done reports whether the action was already taken, and its detail.

func (*StateLog) Record

func (l *StateLog) Record(exp, verdict, action, detail string, at time.Time) error

Record marks an action as taken.

func (*StateLog) Reset

func (l *StateLog) Reset(exp string, at time.Time) error

Reset forgets exp's actions; a no-op (nothing written) when there are none.

type Webhook

type Webhook struct {
	URL    string
	Secret []byte
	Client *http.Client
	Now    func() time.Time // default time.Now
}

Webhook posts the Event JSON, signed with Secret when set.

func (Webhook) Name

func (Webhook) Name() string

Name implements Named.

func (Webhook) Notify

func (w Webhook) Notify(ctx context.Context, e Event) error

type Writer

type Writer interface {
	ProposeStatus(ctx context.Context, experiment, status, title, body string) (prURL string, err error)
}

Writer opens a policy-repo PR setting an experiment's status. It must never merge.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL