Documentation
¶
Overview ¶
Package controller is the automated experiment loop: every tick it evaluates each running experiment (promote.Evaluate), persists the verdict for halo-server, and acts on it.
- rollback: trip the traffic-plane kill switch immediately, open a PR setting status: paused with the evidence, notify.
- promote: open a PR concluding the experiment (never merged), notify.
- expired: open a PR concluding the experiment, notify.
Every action is taken at most once per experiment run (StateLog), so repeated ticks never duplicate PRs or notifications. Nothing here merges.
Index ¶
- Constants
- func Evidence(rep promote.Report) string
- func Sign(secret []byte, ts int64, body []byte) string
- func Verify(secret []byte, h http.Header, body []byte, now time.Time, ...) error
- type Controller
- type Event
- type GitWriter
- type KillFunc
- type KillOutcome
- type KillRecord
- type KillStore
- type KillSwitch
- type Named
- type Notifier
- type Notifiers
- type RolloutWriter
- type Slack
- type StateLog
- type Webhook
- type Writer
Constants ¶
const ( SignatureHeader = "X-Halo-Signature" TimestampHeader = "X-Halo-Timestamp" // WebhookTolerance is the recommended receiver clock-skew/replay window. WebhookTolerance = 5 * time.Minute )
Webhook authentication. The signature covers the timestamp so a captured request cannot be replayed later:
X-Halo-Timestamp: <unix seconds> X-Halo-Signature: sha256=<hex HMAC-SHA256(secret, timestamp + "." + body)>
Receivers must recompute the HMAC over the raw body with the timestamp from X-Halo-Timestamp, compare in constant time, and reject timestamps more than WebhookTolerance from their clock (Verify does all of this).
const ( ActionKill = "kill" // detail: "" = enforced, else why it is not known to be ActionPR = "pr" // ActionNotify is recorded per channel as ActionNotify+":"+name (see Named); // a bare ActionNotify (older logs) means every channel was notified. ActionNotify = "notify" // ActionRollback marks that this run rolled back (detail: the report JSON); // the controller stops evaluating the experiment until it leaves "running". ActionRollback = "rollback" // ActionHold marks the once-per-run warning that an experiment is killed // and therefore not evaluated. ActionHold = "hold" )
Actions the controller takes at most once per experiment run.
const Actor = "halo-controller"
Actor is the identity the controller records on kills and audit entries.
const DefaultTickTimeout = 2 * time.Minute
DefaultTickTimeout bounds one tick (all experiments).
const VerdictKilled promote.Verdict = "killed"
VerdictKilled is the pseudo-verdict of hold records and of the Event sent when a running experiment is found killed (it is not evaluated).
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Controller ¶
type Controller struct {
Org func() (*policy.Org, error) // e.g. policy.Load of the served policy dir
Metrics promote.MetricSource
Writer Writer // nil: no PRs (logged)
Kill KillSwitch // nil: rollbacks are not enforced until the PR merges (logged, notified)
// KillCaveat, when set, is why a successful Kill is not known to be
// enforced by gateways (e.g. `halo controller run` without
// --killswitch-served); messages report it instead of claiming enforcement.
KillCaveat string
// Kills is read each tick: a killed running experiment is not evaluated
// (its evidence is all-control) and humans are told once. nil = no check.
Kills *KillStore
Notifier Notifier // nil: none
State *StateLog
VerdictsPath string // "" = do not persist verdicts
Clock func() time.Time
Log *slog.Logger
TickTimeout time.Duration // default 2m; bounds one whole tick
// RolloutStateDir holds one hash-chained state file per Rollout ("" =
// rollouts are not driven); PolicyDir is where step scorecards are read.
RolloutStateDir, PolicyDir string
// contains filtered or unexported fields
}
Controller wires the loop. Org, Metrics and State are required.
func (*Controller) Run ¶
func (c *Controller) Run(ctx context.Context, interval time.Duration)
Run ticks every interval until ctx ends. After a failed tick it retries sooner (15s, 30s, ... capped at interval) so a transient ClickHouse or git error doesn't delay a rollback by a whole interval.
func (*Controller) Tick ¶
func (c *Controller) Tick(ctx context.Context) error
Tick evaluates every running experiment once and acts on the verdicts, then drives active rollouts (see tickRollouts). Per-experiment failures don't stop the others; all are joined in the result.
func (*Controller) WriteMetrics ¶
func (c *Controller) WriteMetrics(w io.Writer)
WriteMetrics writes Prometheus text-format counters.
type Event ¶
type Event struct {
// Rollout is set for rollout events; Experiment is then its backing experiment.
Rollout string `json:"rollout,omitempty"`
Experiment string `json:"experiment"`
Axis string `json:"axis,omitempty"`
Verdict promote.Verdict `json:"verdict"`
Reason string `json:"reason"`
Killed bool `json:"killed"` // traffic-plane kill switch is active for the experiment
KillOutcome KillOutcome `json:"killOutcome,omitempty"`
KillError string `json:"killError,omitempty"` // detail for every non-enforced outcome
PRURL string `json:"prURL,omitempty"`
PRError string `json:"prError,omitempty"`
Report promote.Report `json:"report"`
At time.Time `json:"at"`
}
Event is what the controller tells humans about a verdict it acted on.
type GitWriter ¶
type GitWriter struct {
RepoDir string // git checkout (or any dir inside one)
Subdir string // policy root inside RepoDir
Base string // PR base branch; "" = repo default
Opener promote.PROpener
}
GitWriter implements Writer with promote.PlanStatus and a promote.PROpener (GHOpener: branch in a temp worktree, git push, gh pr create). It is the one "set experiment status + open PR" implementation: `halo controller run` uses it on the CI checkout; halo-server's server.GitPolicyWriter uses it on its dedicated clone after syncing that clone to origin (under its own lock). It never touches RepoDir's branch, index or working tree.
type KillFunc ¶
KillFunc adapts a function (e.g. server.KillExperiment, which also audit-logs) to KillSwitch.
type KillOutcome ¶
type KillOutcome string
KillOutcome is the typed result of the kill-switch step of a rollback.
const ( KillEnforced KillOutcome = "enforced" // tripped and enforced by gateways KillNotGateway KillOutcome = "not_gateway_evidence" // evidence not gateway-sourced; human decides KillNotConfigured KillOutcome = "not_configured" // no kill switch wired KillRecorded KillOutcome = "recorded_only" // written, enforcement not guaranteed (KillCaveat) KillFailed KillOutcome = "failed" // the kill call errored )
type KillRecord ¶
type KillRecord struct {
Experiment string `json:"experiment"`
Killed bool `json:"killed"`
By string `json:"by"`
Reason string `json:"reason,omitempty"`
At time.Time `json:"at"`
}
KillRecord is one line of killswitch.jsonl; the last line per experiment wins.
type KillStore ¶
KillStore is the persisted kill state halo-server serves to gateways: an append-only JSONL log (killswitch.jsonl in the data dir) replayed into memory. Appends from another process sharing the file (`halo controller run` against halo-server's data dir) are picked up on the next read. ponytail: whole-file re-read when the size changes; kills are rare. Cross-process appends rely on O_APPEND single-line writes (local filesystem).
func OpenKillStore ¶
OpenKillStore opens dir/killswitch.jsonl ("" = memory only).
type KillSwitch ¶
type KillSwitch interface {
// Kill marks experiment killed; changed is false when it already was.
Kill(ctx context.Context, experiment, by, reason string) (changed bool, err error)
}
KillSwitch trips the traffic-plane kill switch for an experiment.
type Named ¶
type Named interface {
Name() string
}
Named is implemented by notifiers with a stable channel name. The controller records each channel's delivery separately (ActionNotify + ":" + name), so a failing webhook is retried without re-posting to Slack.
type Notifiers ¶
type Notifiers []Notifier
Notifiers fans out to every notifier and joins their errors.
func NotifiersFromFiles ¶
NotifiersFromFiles builds notifiers from secret files (any path may be ""): a Slack incoming-webhook URL, a generic webhook URL and its HMAC secret.
type RolloutWriter ¶
type RolloutWriter interface {
ProposeRollout(ctx context.Context, plan func(dir string) (*promote.Change, error), branch, title string, body func(patch string) string) (prURL string, err error)
}
RolloutWriter opens a rollout PR: plan builds the change in the writer's checkout dir, body renders the PR body around its patch. GitWriter and server.GitPolicyWriter implement it. It must never merge.
type StateLog ¶
type StateLog struct {
// contains filtered or unexported fields
}
StateLog remembers which (experiment, verdict, action) the controller has already done, so repeated ticks never open a second PR or send a second notification. An experiment's entries are forgotten once it is seen not running (a human merged the pause/conclude), so a later restart starts clean. ponytail: no compaction; one line per action per experiment run.
func OpenStateLog ¶
OpenStateLog opens dir/controller-state.jsonl ("" = memory only).