Documentation
¶
Overview ¶
Package engine is ynf's outer loop (ADR-001): one idempotent step per event — correlate, claim, probe, decide, act, record, release — plus the sweep that turns searches and due timers into events. It holds nothing in memory that matters between steps (NFR-2), so any host can run it.
Index ¶
- Constants
- func AdhocTracker(s store.Store) tracker.Tracker
- func ShadowQuery(q string, since time.Time) string
- func VerifyGitHubSignature(secret, header string, body []byte) error
- type ActionRecord
- type Checker
- type Connection
- type DecisionRecord
- type Engine
- func (e *Engine) CloseTrackers()
- func (e *Engine) Connections(ctx context.Context) ([]Connection, error)
- func (e *Engine) Enrolled(ctx context.Context) ([]string, error)
- func (e *Engine) Factory(ctx context.Context) (*FactoryPolicy, error)
- func (e *Engine) FlushMemory(ctx context.Context)
- func (e *Engine) Handle(ctx context.Context, key string, ev event.Event) (err error)
- func (e *Engine) HandleGitHubEvent(ctx context.Context, name string, body []byte) (Touched, error)
- func (e *Engine) HostYnh(ctx context.Context) runner.Detection
- func (e *Engine) ItemRepo(it item.Item) string
- func (e *Engine) Items(ctx context.Context) ([]item.Item, error)
- func (e *Engine) LaneID(rp *RepoPolicy, lane policy.Lane) string
- func (e *Engine) LaneRuns(ctx context.Context, repo string) ([]LaneRun, error)
- func (e *Engine) LaneState(ctx context.Context, repo, lane string) (LaneState, string, error)
- func (e *Engine) MemoryQueued(ctx context.Context) (int, time.Time, error)
- func (e *Engine) MirrorDelivery(ctx context.Context, name, id, subject, result string)
- func (e *Engine) MirrorIntake(parent context.Context, ev event.Event, itemKey, result string) context.Context
- func (e *Engine) Policy(ctx context.Context, repo string) (*RepoPolicy, error)
- func (e *Engine) PolicyWithLayer(ctx context.Context, repo string, own []byte) (*RepoPolicy, error)
- func (e *Engine) RequiredChecks(ctx context.Context) ([]RequiredReport, error)
- func (e *Engine) ResetPolicies()
- func (e *Engine) RunDue(ctx context.Context) (int, error)
- func (e *Engine) SetPaused(ctx context.Context, repo, lane string, paused bool, reason, by string) error
- func (e *Engine) Settled(ctx context.Context) (bool, error)
- func (e *Engine) ShadowRun(ctx context.Context, req ShadowRequest) (shadow.Run, error)
- func (e *Engine) Start(ctx context.Context, req StartRequest) (item.Item, error)
- func (e *Engine) Stats(ctx context.Context) ([]Stats, error)
- func (e *Engine) Sweep(ctx context.Context) error
- func (e *Engine) SweepRepos(ctx context.Context, repos []string) error
- func (e *Engine) Ticket(ctx context.Context, ref tracker.Ref) (facts.Ticket, tracker.Text, error)
- func (e *Engine) TrackerHost(ctx context.Context, name string) (string, error)
- type FactoryPolicy
- type ForgeInstance
- type Git
- type LaneRun
- type LaneState
- type ModelStats
- type PolicyRef
- type RefusedError
- type Replayed
- type RepoPolicy
- type RequiredReport
- type RunRecord
- type ShadowGit
- type ShadowRequest
- type StartRequest
- type Stats
- type Touched
Constants ¶
const AdhocHost = "adhoc"
AdhocHost is the tracker host of work started from a prompt rather than a ticket (ADR-002).
Variables ¶
This section is empty.
Functions ¶
func AdhocTracker ¶
AdhocTracker is the tracker for work started from a prompt: the prompt, kept in ynf's store, is the ticket. There is nothing to comment on or label; the draft pull request carries the work.
func ShadowQuery ¶
ShadowQuery rewrites a lane's intake search to find what it would have taken once closed: open becomes closed, and only tickets closed since the given time are kept.
func VerifyGitHubSignature ¶
VerifyGitHubSignature checks a webhook's X-Hub-Signature-256 against the shared secret. It runs before anything else reads the payload (ADR-003: verify, then shrink, at the edge).
Types ¶
type ActionRecord ¶
type ActionRecord struct {
Action string `json:"action"`
OK bool `json:"ok"`
Detail string `json:"detail,omitempty"`
PR int `json:"pr,omitempty"`
Commit string `json:"commit,omitempty"`
}
ActionRecord is the log entry for a forge action.
type Checker ¶
Checker is a tracker that can check itself without reading a ticket, such as an MCP tracker confirming its server has the tools it is configured to call.
type Connection ¶
type Connection struct {
Kind string `json:"kind"` // forge or tracker
Name string `json:"name"`
Provider string `json:"provider"`
Host string `json:"host"`
OK bool `json:"ok"`
Detail string `json:"detail"`
}
Connection is a forge or tracker the factory works with, and whether ynf can reach it.
type DecisionRecord ¶
type DecisionRecord struct {
Input decide.Input `json:"input"`
Decision decide.Decision `json:"decision"`
Policy PolicyRef `json:"policy"`
}
DecisionRecord is the log entry every decision writes: everything Decide read, and what it returned, so `ynf replay` can recompute it (ADR-006, ADR-011).
type Engine ¶
type Engine struct {
Store store.Store
Forge forge.Forge
// ForgeHost is Forge's host as item keys name it; default github.com.
ForgeHost string
// Forges are the forge instances besides Forge, by host, such as a GitHub Enterprise Server
// (ADR-003). A repository on one is named host/owner/name; one on Forge, owner/name.
Forges map[string]ForgeInstance
// NewForge builds a forge instance the configuration repository declares; nil allows none.
NewForge func(name string, cfg map[string]any) (ForgeInstance, error)
// NewTracker builds a tracker instance the configuration repository declares, returning its
// host; nil allows none.
NewTracker func(name string, cfg map[string]any) (string, tracker.Tracker, error)
// ConfigRepo is the factory's configuration repository (owner/name on the forge), whose
// factory.yaml enrols repositories and whose lanes.yaml every enrolled repository's lanes
// are laid over (ADR-006). Empty: Repos is the enrolment and each repository's lanes stand
// alone.
ConfigRepo string
// Trackers are the tracker instances by host (ADR-003).
Trackers map[string]tracker.Tracker
Git Git
Executor func(name string) (executor.Executor, error)
Repos []string // enrolled repositories
Lanes []string // only these lanes; empty means all
WorkDir string
Owner string
LeaseTTL time.Duration
Heartbeat time.Duration
Poll decide.Poll
RunTimeout time.Duration
Interactive bool // allows the process executor (ADR-007)
// BuildImage builds a ynh agent image for a harness in a worktree (`ynh image --entrypoint
// agent`) and returns its tag. Nil means ynh is not available (ADR-012).
BuildImage func(ctx context.Context, worktree string, cfg policy.Ynh) (string, error)
// HostAutoApprove is --auto-approve asked for by the person at the terminal (ynf start), for
// runs on the host only; HostCapabilities reports the host ynh's capabilities to check it.
HostAutoApprove string
HostCapabilities func(ctx context.Context) (string, error)
// ImageHarness reads what the harness inside an agent image declares, by asking the image's
// own ynh; harness picks one when the image carries several. nil skips reading it.
ImageHarness func(ctx context.Context, image, harness string) (runner.Harness, error)
// InstallHarness installs a harness, a folder or a pin from a repository, into the ynh home
// ynhHome, which is the run's own, and says what was installed, including the id `ynh agent
// run` takes for it. nil runs the lane's harness value as it is, which a pin cannot be.
InstallHarness func(ctx context.Context, src runner.HarnessSource, ynhHome string) (runner.Installed, error)
// ImageCapabilities reports the capabilities version of the ynh inside an agent image; nil
// skips the check.
ImageCapabilities func(ctx context.Context, image string) (string, error)
// DetectYnh reports whether the host's ynh is there and supported, for a lane that names no
// runner (ADR-012); the CLI passes runner.DetectedYnh, which asks once per process. Nil means
// ynh is not detected, and a lane that names ynh is not checked against it.
DetectYnh func(ctx context.Context) runner.Detection
// Getenv reads the variables a lane passes into its runs (run.env). Default os.Getenv.
Getenv func(string) string
// Memory is ynm when it is configured or detected, else nil (ADR-008). It is advisory: what it
// holds goes into a run's task and to people, never into a decision.
Memory memory.Memory
// MemoryNamespace is the namespace for a repository; default factory/<owner>/<name>.
MemoryNamespace func(repo string) string
// MemoryLevel is the ynm level ynf writes at: empty or personal for one person's store,
// distributed for a shared one (ADR-008).
MemoryLevel string
// Spool is the spool root, when one is configured: each run gets a folder and a manifest in
// it (ynr ADR-003). SpoolCollector is true when ynf starts ynr serve for the job, so runs
// start without the operator's OTEL_EXPORTER_OTLP_*.
Spool *spool.Spool
SpoolCollector bool
// ProgressEvery is how often a run in progress is logged; default 30s, negative for never.
ProgressEvery time.Duration
Now func() time.Time
NewID func() string
Log *slog.Logger
// Telemetry is ynf's OpenTelemetry (ADR-011); nil writes none and changes nothing. It is
// write-only: nothing in the engine reads it back.
Telemetry *telemetry.T
// contains filtered or unexported fields
}
Engine runs steps.
func (*Engine) CloseTrackers ¶
func (e *Engine) CloseTrackers()
CloseTrackers ends the trackers that hold a session, such as an MCP server ynf started.
func (*Engine) Connections ¶
func (e *Engine) Connections(ctx context.Context) ([]Connection, error)
Connections lists the forges and trackers ynf works with, each checked (ADR-003, ADR-011): a forge by reading an enrolled repository on it, a GitHub tracker with its forge, and a tracker that can check itself by doing so.
func (*Engine) Enrolled ¶
Enrolled is the repositories ynf works on: the configuration repository's enrolment, or Repos.
func (*Engine) Factory ¶
func (e *Engine) Factory(ctx context.Context) (*FactoryPolicy, error)
Factory reads the configuration repository at its default branch's head, once until ResetPolicies; nil when there is none.
func (*Engine) FlushMemory ¶
FlushMemory sends the memory writes that were queued while ynm was down, oldest first. It is cheap when none are, and it never fails: what cannot be sent stays queued.
func (*Engine) Handle ¶
Handle runs one step for the item at key, starting from ev. A held lease means another instance is working on it, which is not an error.
The step is one trace (ADR-011): its span has the claim, and for each decision a probe, a decide and an act, with the run and every call out to another system beneath them. It links to the item's previous step and to the intake of the event that started it, and leaves its own span on the item for the next step to link to.
func (*Engine) HandleGitHubEvent ¶
HandleGitHubEvent steps every item a webhook touches. An enrolled repository's new tickets are found by sweeping it, since only a lane's own search says whether a ticket belongs to it; a tracked item is stepped at once, with facts probed fresh.
func (*Engine) HostYnh ¶
HostYnh is what detecting ynh on the host found: nothing when no detection is wired.
func (*Engine) LaneID ¶
func (e *Engine) LaneID(rp *RepoPolicy, lane policy.Lane) string
LaneID is the lane's id in telemetry (ADR-006): the id the lane declares, else where it is defined, host first, then its name: github.com/example-org/factory-config#lint-paydown for a lane in the configuration repository, however a repository overrides it, and github.com/eyelock/ynh#docs-refresh for one the repository defines itself.
func (*Engine) LaneRuns ¶
LaneRuns describes how each of a repository's lanes runs (ADR-006, ADR-012). A harness in a published image, or installed beside an inline ynf, is read now and checked against the lane: its budgets may only be tightened, and its sensors only scoped. A harness carried in the repository is read when a run checks it out.
func (*Engine) MemoryQueued ¶
MemoryQueued is how many memory writes are queued and when the oldest was first queued.
func (*Engine) MirrorDelivery ¶
MirrorDelivery records a webhook delivery that never became an event for an item, because it was refused or recognised as one already received: a ynf.forge.changed CloudEvent with the delivery's id when it has one, rejected or deduplicated. subject names the repository when it is known.
func (*Engine) MirrorIntake ¶
func (e *Engine) MirrorIntake(parent context.Context, ev event.Event, itemKey, result string) context.Context
MirrorIntake records a CloudEvent ynf received as a short ynf.intake span carrying one ynf.intake.received event, whether it was accepted, deduplicated or rejected (ynr ADR-003): the one-way mirror of the control plane. itemKey is the item it is for, when it is known. The returned context carries the span, so the step the event starts links to it.
func (*Engine) Policy ¶
Policy reads a repository's lanes from the first factory folder on its default branch, at its head commit, laid key by key over the configuration repository's lanes when there is one (ADR-006): the repository wins.
func (*Engine) PolicyWithLayer ¶
PolicyWithLayer is Policy for a repository's lanes.yaml given as bytes, such as a local file not yet pushed, instead of the one on the forge. It is laid over the configuration repository's lanes and validated as Policy does; nothing is cached. SHA and Dir are empty.
func (*Engine) RequiredChecks ¶
func (e *Engine) RequiredChecks(ctx context.Context) ([]RequiredReport, error)
RequiredChecks reads each enrolled repository's required checks on its default branch, for ynf doctor. A forge that cannot read them is left out.
func (*Engine) ResetPolicies ¶
func (e *Engine) ResetPolicies()
ResetPolicies forgets cached policy, so the next step reads each repository's lanes afresh.
func (*Engine) RunDue ¶
RunDue steps every item whose timer has passed, and returns how many it stepped.
func (*Engine) SetPaused ¶
func (e *Engine) SetPaused(ctx context.Context, repo, lane string, paused bool, reason, by string) error
SetPaused pauses or resumes a lane, recording who and why. Unpausing is deliberate: it takes a reason too, because stop conditions are renegotiated under pressure (ADR-010).
func (*Engine) Settled ¶
Settled reports whether every tracked item is somewhere only a human or the outside world can move it from.
func (*Engine) ShadowRun ¶
ShadowRun runs a lane against closed tickets whose answer is known (FR-26): each candidate is built into a task as start would build it, run in the lane's own runner and containment on the commit before the human fix, and kept beside the human's patch for blind grading. It proposes nothing: no push, commit to a remote, pull request, ticket comment or label, memory write, item or count in stats. Everything it records is under shadow/<run id>/ in ynf's own store.
func (*Engine) Start ¶
Start carries out an instruction. It resolves the ticket, checks where its code goes and that a lane takes it, and fails before creating anything if any check does. Then it creates the item, or picks up the existing one, and steps it until it waits on something outside ynf; with Detach it only schedules it.
func (*Engine) Stats ¶
Stats computes every lane's numbers from its items. Every lane of every enrolled repository is listed, with or without items, so a paused lane is always visible.
func (*Engine) Sweep ¶
Sweep runs every enrolled repository's lane searches and starts a step for each new ticket.
func (*Engine) SweepRepos ¶
SweepRepos is Sweep for some of the enrolled repositories.
type FactoryPolicy ¶
type FactoryPolicy struct {
Repo string
SHA string
Dir string
File *policy.Factory
Lanes []byte // its lanes.yaml; empty when it has none
}
FactoryPolicy is the configuration repository's: its factory.yaml and default lanes, read at a resolved commit.
type ForgeInstance ¶
ForgeInstance is a forge besides the default: its client, the git workspace that clones from and pushes to it with its own token, and the tracker for its issues.
type Git ¶
type Git interface {
Mirror(ctx context.Context, repo string) (string, error)
Checkout(ctx context.Context, mirror, ref, dir string) (string, error)
RemoveCheckout(dir string) error
RemoteHas(ctx context.Context, mirror, branch string) bool
Changed(ctx context.Context, wt string) ([]string, error)
Commit(ctx context.Context, wt, message string) (string, error)
Push(ctx context.Context, wt, repo, branch string) error
Head(ctx context.Context, wt string) (string, error)
RemoteSHA(ctx context.Context, mirror, branch string) (string, error)
PushFastForward(ctx context.Context, wt, repo, branch string) error
}
Git is what the engine needs from the workspace.
type LaneRun ¶
type LaneRun struct {
Lane string `json:"lane"`
Kind string `json:"kind"`
On bool `json:"on"`
Runner string `json:"runner"` // the runner it runs: what it names, else what detection chose
Detected bool `json:"detected,omitempty"` // the lane names no runner, so Runner is what this host resolves to
Resolves string `json:"resolves,omitempty"` // what an unnamed runner resolves to here, such as "ynh (detected 0.10.0)"
Executor string `json:"executor"`
Image string `json:"image,omitempty"`
Harness string `json:"harness,omitempty"` // what the lane names
Model string `json:"model,omitempty"` // the model the lane pins; empty is the vendor's default
Effort string `json:"effort,omitempty"` // the effort the lane asks for; empty is the harness's or vendor's default
Where string `json:"where"` // where the harness comes from
Pin *policy.Pin `json:"pin,omitempty"` // the repository and ref, for a harness pinned from one
Resolved string `json:"resolved,omitempty"` // what a pin installs: name@version at its commit
Read *runner.Harness `json:"read,omitempty"` // the harness, when it could be read
Problem string `json:"problem,omitempty"` // why it could not be read, or does not fit the lane
}
LaneRun is how a lane's runs are made, and the harness it is held to when ynf can read it.
type LaneState ¶
type LaneState struct {
Repo string `json:"repo"`
Lane string `json:"lane"`
Paused bool `json:"paused"`
Reason string `json:"reason,omitempty"`
By string `json:"by,omitempty"`
At time.Time `json:"at"`
}
LaneState is a lane's own record: paused by a human or a stop condition (ADR-010).
type ModelStats ¶
type ModelStats struct {
Model string `json:"model"`
Effort string `json:"effort,omitempty"`
Runs int `json:"runs"`
Converged int `json:"converged"`
Turns int `json:"turns"` // total, over runs that reported them
Tokens int `json:"tokens"` // total, input and output
CacheRead int `json:"cache_read_tokens,omitempty"`
CostUSD float64 `json:"cost_usd,omitempty"`
Proposed int `json:"proposed"`
Merged int `json:"merged"`
Rejected int `json:"rejected"`
}
ModelStats is one model and effort's share of a lane's runs. Proposals, merges and rejections are attributed to the model of the run whose change was proposed.
type PolicyRef ¶
type PolicyRef struct {
Repo string `json:"repo"`
Dir string `json:"dir"`
Ref string `json:"ref"`
SHA string `json:"sha,omitempty"` // the repository's commit its lanes were read at
Config string `json:"config,omitempty"`
ConfigSHA string `json:"config_sha,omitempty"`
Hash string `json:"hash"`
}
PolicyRef says which policy a decision ran under.
type RefusedError ¶
type RefusedError struct{ Reason string }
RefusedError is an instruction ynf refused before creating anything: the reference, the tracker, the repository or the lane did not check out.
func (*RefusedError) Error ¶
func (r *RefusedError) Error() string
type Replayed ¶
type Replayed struct {
EntryID string `json:"entry_id"`
Event string `json:"event"`
Recorded decide.Decision `json:"recorded"`
Replayed decide.Decision `json:"replayed"`
Same bool `json:"same"`
Policy string `json:"policy"` // recorded, or the lane hash it was replayed under
}
Replayed is one recorded decision recomputed.
type RepoPolicy ¶
type RepoPolicy struct {
Repo string
Base string // default branch
SHA string // the commit it was read at
Dir string // the factory folder it came from; empty when the repository has no lanes of its own
Shadowed []string
File *policy.File
// Config is the configuration repository's policy the lanes were laid over, if any; Own is
// the repository's own lanes.yaml, for telling which layer set a value.
Config *FactoryPolicy
Own []byte
}
RepoPolicy is a repository's lane policy, read from its default branch.
type RequiredReport ¶
type RequiredReport struct {
Repo string `json:"repo"`
Branch string `json:"branch"`
Known bool `json:"known"`
Checks []string `json:"checks,omitempty"`
Detail string `json:"detail"`
// Workflows is how many workflow files the default branch has, when the forge could say.
Workflows *int `json:"workflows,omitempty"`
}
RequiredReport is how an enrolled repository's required checks were read: what its default branch requires, or that neither branch protection nor the rulesets could be read, in which case every check on a pull request gates.
type RunRecord ¶
type RunRecord struct {
RunID string `json:"run_id"`
Runner string `json:"runner"`
// RunnerDetected is set when the lane named no runner and detection chose this one, so the
// log and stats can tell a named runner from a detected one (ADR-012).
RunnerDetected bool `json:"runner_detected,omitempty"`
// RunnerFeatures are the features the host's ynh listed, for a run that was detected or ran a
// harness folder by path, so the record says what the run could rely on.
RunnerFeatures []string `json:"runner_features,omitempty"`
Executor string `json:"executor"`
Argv []string `json:"argv"`
Base string `json:"base"`
Exit int `json:"exit"`
Outcome string `json:"outcome"`
Detail string `json:"detail,omitempty"`
Changed []string `json:"changed,omitempty"`
Denied []string `json:"denied,omitempty"` // hosts the egress proxy refused
StepDir string `json:"step_dir"`
Duration string `json:"duration"`
// HarnessPin is the repository and ref a pinned harness was installed from, as the lane wrote
// it; Harness and HarnessSHA say which harness and commit that gave.
HarnessPin string `json:"harness_pin,omitempty"`
// HarnessPinURL is the same pin as ynh was given it, set only when it differs: a bare host and
// path (github.com/org/harness@v1) is cloned from https://github.com/org/harness.
HarnessPinURL string `json:"harness_pin_url,omitempty"`
// Model and Usage are what the runner reports, for comparing outcomes and cost by model and
// effort (ADR-011). ynf's own store is the run history; memory holds only failure patterns.
Model string `json:"model,omitempty"`
runner.Usage
}
RunRecord is the log entry for a run.
type ShadowGit ¶
type ShadowGit interface {
FirstParent(ctx context.Context, mirror, sha string) (string, error)
RangeDiff(ctx context.Context, mirror, from, to string) (string, error)
Diff(ctx context.Context, wt, base string) (string, error)
}
ShadowGit is what shadow mode needs from the workspace besides Git: the commit before a merge, and the two patches to compare.
type ShadowRequest ¶
type ShadowRequest struct {
Lane string
Repos []string // owner/name, as Policy takes them; empty: every enrolled repository with the lane
// Since bounds how recently a candidate was closed; default 90 days.
Since time.Duration
// Limit is how many candidates to run per repository; default 20.
Limit int
// Tickets names candidates directly, in place of the lane's intake.
Tickets []tracker.Ref
}
ShadowRequest is `ynf shadow run`.
type StartRequest ¶
type StartRequest struct {
Ref tracker.Ref // the ticket; empty when Prompt is set
Prompt string // ad hoc work, with no ticket
Labels []string // the prompt's labels, as a ticket's would be: what the lane's templates read
Repo string // owner/name on the forge; a GitHub issue's own repository when empty
Lane string // empty: the repository's only originate lane
Detach bool // only record it, for a running ynf serve to step
}
StartRequest is an instruction (ADR-003): take on this one item now.
type Stats ¶
type Stats struct {
Repo string `json:"repo"`
Lane string `json:"lane"`
States map[string]int `json:"states"`
Proposed int `json:"proposed"` // items that ever had a pull request
Merged int `json:"merged"`
Rejected int `json:"rejected"` // pull requests closed without merging
Yield float64 `json:"yield"` // merged ÷ (merged + rejected); -1 with no decided proposals
Signatures map[string]int `json:"signatures"`
Paused bool `json:"paused"`
Reason string `json:"reason,omitempty"`
// Models breaks the lane's runs down by model and effort (ADR-011): which can do the work,
// at what cost. Read from ynf's own run records.
Models []ModelStats `json:"models,omitempty"`
}
Stats are a lane's factory numbers (ADR-010, ADR-011).
type Touched ¶
type Touched struct {
Repo string `json:"repo"`
// Host is the forge the event came from, read from the payload: github.com and an Enterprise
// Server can share one endpoint. Empty when the payload does not say.
Host string `json:"host,omitempty"`
Issues []int `json:"issues,omitempty"`
PRs []int `json:"prs,omitempty"`
}
Touched is what a webhook is about.