Documentation
¶
Overview ¶
Package pipeline is the run pipeline (design/workstreams/agent/LLD/run-pipeline.md): it turns one leased Task into a harness run and the run into a validated Result. It implements runner.Runner, so the worker's lease loop drives it.
For each job it creates a scratch workspace (0700, always removed; or the exact Root, kept), starts an in-memory git credential helper (package gitcred) for the task's repos (or, in gateway mode, none: git goes through the gateway's proxy), clones them, materialises the pinned skills, renders the prompt (the headless preamble, the rendered template from the API, and the context items fenced with their trust level), builds the harness's environment from an allow-list, runs the harness through its harness.Adapter under the task's limits, validates the Result against the output schema and outcomes (with one repair turn), uploads the redacted, gzipped transcript, and cleans up.
Index ¶
- Constants
- func BundleSchema(doc []byte) ([]byte, error)
- func FailedResult(task, reason string, err error, redact func(string) string) v1alpha1.Result
- func Preamble(taskID, taskSpec string) string
- func Redact(job *runner.Job, s string, extra ...string) string
- func RedactShapes(s string) string
- func ValidateResult(taskID, taskSpec string, spec []byte, outcomes []string, doc []byte) []string
- type APIPrompts
- type APISchemas
- type APITranscripts
- type BundleExtras
- type BundlePrompts
- type BundleSchemas
- type EffectorConfig
- type Environment
- type FetchSkills
- type FileTranscript
- type GatewayCredentialError
- type GitAuth
- type Options
- type Pipeline
- func (p *Pipeline) PrepareBundle(ctx context.Context, job *runner.Job, dir string, x BundleExtras) (err error)
- func (p *Pipeline) Run(ctx context.Context, job *runner.Job, sink runner.Sink) (runner.Outcome, error)
- func (p *Pipeline) StoreEnvironmentLog(ctx context.Context, job *runner.Job, log []byte, extra ...string) error
- func (p *Pipeline) StoreTranscript(ctx context.Context, job *runner.Job, parts [][]byte, extra ...string) (string, error)
- type PrebuiltSkills
- type PromptSource
- type SchemaSource
- type SkillSource
- type TranscriptStore
Constants ¶
const ( EnvHeadless = "FACTORY_HEADLESS" EnvHarness = "FACTORY_HARNESS" EnvTaskID = "FACTORY_TASK_ID" EnvAPI = "FACTORY_API" EnvToken = "FACTORY_TOKEN" EnvOut = "FACTORY_OUT" EnvResult = "FACTORY_RESULT" EnvGHToken = "GH_TOKEN" )
Harness environment variables (run-pipeline.md §Steps 7, the ccf-skills HEADLESS.md contract).
const ( EnvRuntime = "FACTORY_RUNTIME" RuntimeWorkspace = "workspace" )
EnvRuntime names the runtime a run's processes are in: RuntimeWorkspace, a VM workspace (workspace-run.md §System setup), where vmrun sets it through Options.Env. The Effector reads it and refuses any other value.
const ( // DefaultWallClock bounds a run whose TaskSpec sets no limits.maxWallClock (the // worker's drain deadline uses the same default). DefaultWallClock = time.Hour // DefaultTranscriptLimit is the API's transcript cap (contracts.md: ≤ 20 MB gzipped). DefaultTranscriptLimit = 20 << 20 // DefaultSkillsBase prefixes a skills source that isn't a URL ("ccf-skills"). DefaultSkillsBase = "https://github.com/compliance-framework" // DefaultBotName and DefaultBotEmail are the commit author in the rw checkout. DefaultBotName = "ccf-factory[bot]" DefaultBotEmail = "ccf-factory[bot]@users.noreply.github.com" )
Defaults.
const EffectorFile = "effector.json"
EffectorFile is the EffectorConfig's name in the workspace root (outside out/, which the harness writes).
const EnvEffector = "FACTORY_EFFECTOR"
EnvEffector names the variable that carries the path of the run's EffectorConfig, which the Effector CLI and MCP server (`factory`, effector.md) read.
const EnvironmentLogTruncated = "[truncated]\n"
EnvironmentLogTruncated is the line that ends a cut environment log.
const MaxEnvironmentLog = 10 << 20
MaxEnvironmentLog caps the environment log a VM workspace's run uploads (the VM caps it at envrun.MaxLog; past this cap it is cut on a line and ends with EnvironmentLogTruncated).
const PreambleVersion = "2"
PreambleVersion versions Preamble: bump it with any change to the text.
const Redacted = "[REDACTED]"
Redacted replaces every secret in a transcript, a progress summary or an error.
const StackDownTimeout = 3 * time.Minute
StackDownTimeout bounds a VM environment's down script (envrun's DownTimeout, vmrun's --down-timeout): local-dev's compose down stops a dozen containers and removes their volumes, and the showcase profile also deletes a kind cluster, which takes longer than cleanupTimeout. The name predates the environment: it was the host stack's budget.
Variables ¶
This section is empty.
Functions ¶
func BundleSchema ¶
BundleSchema makes a task's output schema self-contained for a harness CLI, which can't resolve the v1alpha1 Result schema by its remote $id. When the document refs that schema (v1alpha1.SchemaBaseURL + ResultSchemaFile, with or without a fragment), the schema is copied into $defs["factory-result"] (without its $schema and $id) and those refs, and the copy's own internal refs, point there. A document that refs nothing remote is returned unchanged; any other remote $ref is an error. The pipeline's own validation still compiles the original documents.
func FailedResult ¶
FailedResult is the failed Result of a run that ended without one of its own, the pipeline's and the VM's (workspace-run.md, decided 2026-10-22): its summary is reason, then ": " and err's message redacted by redact (nil: by the token shapes alone, RedactShapes) when err isn't nil, cut to 500 bytes on a rune boundary.
func Redact ¶
Redact redacts s as the run's redactor does: job's secrets (its git tokens, its agent token, read now, and the lease's run token), extra (secrets the job doesn't hold, such as the supervisor's model logins) and every token shape. The supervisor redacts what a VM workspace reports with it.
func RedactShapes ¶
RedactShapes redacts s by the token shapes alone, as a run that holds no token of its own (a VM workspace's) does.
func ValidateResult ¶
ValidateResult checks a Result document as the pipeline will after the run: the v1alpha1 Result schema, the spec schema (nil: none), the status and the outcome against outcomes. taskSpec names the spec schema's $id. It returns the problems ("/<pointer>: message"), none when the document is valid. `factory result` uses it for fast feedback; the pipeline's own check (with the harness block filled in) still decides.
Types ¶
type APIPrompts ¶
type APIPrompts struct{}
APIPrompts is the default PromptSource: GET /tasks/{id}/prompt and GET /tasks/{id}/context/{i} through the job's dispatcher client.
type APISchemas ¶
type APISchemas struct{}
APISchemas is the default SchemaSource: GET /tasks/{id}/output-schema (getTaskOutputSchema) through the job's dispatcher client, the schema the dispatcher validates the Result with on complete. Only a 404 means no schema; any other failure fails the run, since a Result checked without the schema could be refused on complete without its repair turn.
func (APISchemas) OutputSchema ¶
OutputSchema implements SchemaSource.
type APITranscripts ¶
type APITranscripts struct{}
APITranscripts is the default TranscriptStore: PUT /runs/{id}/transcript through the job's dispatcher client. A lease that names no run, or a job without a client, stores nothing and returns "".
type BundleExtras ¶
type BundleExtras struct {
CA []byte
Dispatcher string
Codex *runbundle.CodexAccount
}
BundleExtras are what a run bundle holds beyond the job (workspace-run.md §pipeline.PrepareBundle): the gateway's CA (the public half), the dispatcher's URL as the VM sees it ("": runbundle.DefaultDispatcherURL) and the codex account, set exactly when the task's harness is codex.
type BundlePrompts ¶
BundlePrompts is the PromptSource of a run bundle: its prompt.md and context/<n>.json (a Gone item is the 404 case). renderPrompt checks the digest, as for the API's.
type BundleSchemas ¶
BundleSchemas is the SchemaSource of a run bundle: its output-schema.json, or nil when the bundle has none (the API answered 404).
func (BundleSchemas) OutputSchema ¶
OutputSchema implements SchemaSource.
type EffectorConfig ¶
type EffectorConfig struct {
Task string `json:"task"`
TaskSpec string `json:"taskSpec,omitempty"`
Provider string `json:"provider,omitempty"`
// Repo is the task's target repo (owner/name).
Repo string `json:"repo,omitempty"`
// Allow is the task's effects.allow (empty: nothing is allowed); Labels its
// effects.labels globs.
Allow []string `json:"allow"`
Labels []string `json:"labels,omitempty"`
// Outcomes are the task's output.outcomes.
Outcomes []string `json:"outcomes,omitempty"`
// RepoDir is the primary checkout (repo/), where git ops run; empty without one.
RepoDir string `json:"repoDir,omitempty"`
// Schema is the file the Result must match (the spec's schema, else the Result schema).
Schema string `json:"schema,omitempty"`
}
EffectorConfig is what the Effector knows about the run's task besides the FACTORY_* variables: the policy for its local pre-check (the dispatcher re-checks it and is authoritative), the repo bare refs (#12) resolve in, the checkout its git ops run in, and what `factory result` validates against.
func EffectorConfigFor ¶
func EffectorConfigFor(t v1alpha1.Task, ws harness.Workspace) EffectorConfig
EffectorConfigFor is the EffectorConfig of a run of t in ws.
type Environment ¶
type Environment interface {
Up(ctx context.Context, ws harness.Workspace, t v1alpha1.Task) (map[string]string, error)
Down(ctx context.Context) error
}
Environment brings up what a task's VM workspace declares in spec.environment (P1 §4, workspace-run.md §The environment step): package envrun implements it. Up runs the setup and up scripts and waits for ready; it returns the variables to add to the harness's environment (the exposes and FACTORY_VERSION_*). ws.Env is the environment the pipeline's own git has (commonEnv), not the harness's. Down is best effort: cleanup calls it once whenever Up was called, even when Up failed part-way, with no deadline of its own (the Environment owns its budget).
type FetchSkills ¶
type FetchSkills struct{}
FetchSkills is the default SkillSource: each pinned skills repo is fetched with the run's git (through the skills cache, by commit) and the selected skills are materialised into the workspace's skills/ (run-pipeline.md §Steps 5). It needs the run's git, so only the pipeline runs it: its Skills method, called on its own, is an error.
type FileTranscript ¶
type FileTranscript struct {
Path string
}
FileTranscript is the TranscriptStore of a VM workspace (workspace-run.md §Sources): it writes the gzip to Path, through a temp file beside it and a rename, and returns no ref: the supervisor uploads it. Path's directory must exist.
type GatewayCredentialError ¶
type GatewayCredentialError struct{ Field string }
GatewayCredentialError is a gateway-mode run's refusal of a job that carries a credential: in a VM workspace every credential lives in the gateway, never in the job. Field names the field (Lease.GitCredential.Token, Git.Reads[0].Token, AgentToken, …), never its value.
func (*GatewayCredentialError) Error ¶
func (e *GatewayCredentialError) Error() string
type GitAuth ¶
type GitAuth int
GitAuth is how the run's git authenticates to the git host (workspace-run.md §Gateway mode).
const ( // GitAuthHelper, the zero value (the host), answers git from the job's credentials through // the in-memory credential helper (package gitcred) on a Unix socket. GitAuthHelper GitAuth = iota // GitAuthGateway (the VM workspace) has no credential helper: no socket and no helper // config. git reaches the git host through the gateway (HTTPS_PROXY in Options.Env), which // adds the credential for the repo in the path. The job carries no credential: Run refuses // one that does (GatewayCredentialError) before any step. GitAuthGateway )
type Options ¶
type Options struct {
// WorkRoot holds PrepareBundle's scratch directories (workRoot/.bundle-<taskID>/), each
// removed after its bundle is written (the host).
WorkRoot string
// Root is the exact workspace root, instead of WorkRoot (the VM workspace: /work). It must
// be absolute, and empty or absent when a run starts (its parent must exist); the run lays
// its tree directly in it and cleanup never removes it, so out/ survives the run. One run
// at a time: a second finds it non-empty and fails its workspace step.
Root string
// GitAuth is how the run's git authenticates. Default: GitAuthHelper.
GitAuth GitAuth
// Env is added to the environment that the pipeline's own git and the harness share,
// after the owner's allow-list (so it wins over Environ) and before the pipeline's own
// variables: the gateway's proxies (HTTP_PROXY, HTTPS_PROXY, NO_PROXY, either case) and CA
// (SSL_CERT_FILE, GIT_SSL_CAINFO, NODE_EXTRA_CA_CERTS, …), a GH_TOKEN placeholder,
// DOCKER_CONFIG. New refuses a variable the pipeline sets itself (HOME, TMPDIR, the XDG
// and PATH ones, the GIT_* and GCM_* ones of the git setup, the credential socket and the
// FACTORY_* ones of the harness's environment), and one that
// could add a credential helper or change what git or the shell runs: every GIT_* name
// but GIT_SSL_CAINFO and GIT_SSL_CAPATH, SSH_ASKPASS, LD_*, DYLD_*, BASH_ENV and ENV. On
// the host it is usually empty.
Env map[string]string
// Adapters are the harness adapters by harness name (task.harness.resolved.name).
Adapters map[string]harness.Adapter
// AgentBinary is the factory-agent executable: the credential helper and the `factory`
// CLI on the harness's PATH run it. Default: os.Executable().
AgentBinary string
// HelperCommand is the credential helper's command line, without the operation git
// appends. Default: AgentBinary fx git-credential.
HelperCommand []string
// Git is the git binary. Default: "git".
Git string
// GitBaseURL is where the task's repos are cloned from (<base>/<owner>/<name>.git).
// Default: https://<the credential's host>.
GitBaseURL string
// SkillsBase prefixes a skills source that isn't a URL. Default: DefaultSkillsBase.
SkillsBase string
// SkillsCache caches skills checkouts by commit (<cache>/<commit>). Empty: no cache.
SkillsCache string
// SocketDir is where the credential socket's directory is created. Default:
// os.TempDir() (Unix socket paths must stay short).
SocketDir string
// Schemas gives spec-specific output schemas. Default: APISchemas, the dispatcher's
// getTaskOutputSchema through the job's client.
Schemas SchemaSource
// Prompts gives the rendered prompt template and the context items. Default:
// APIPrompts, the dispatcher's getTaskPrompt and getTaskContext through the job's client.
Prompts PromptSource
// Skills gives the run its materialised skills. Default: FetchSkills, the pinned skills
// repos fetched with the run's git (through SkillsCache) into the workspace's skills/.
Skills SkillSource
// Transcripts keeps the redacted, gzipped transcript. Default: APITranscripts, the
// dispatcher's uploadTranscript through the job's client.
Transcripts TranscriptStore
// Environment brings up a task's spec.environment (the VM workspace's envrun); nil on the
// host, where such a task's environment is skipped with a warning.
Environment Environment
// Environ is the owner's environment, reduced to the allow-list for the harness.
// Default: os.Environ.
Environ func() []string
// Clock times the run (wall-clock limit, usage); default clock.Real.
Clock clock.Clock
// Logger logs the pipeline's steps; never a token. Default: discard.
Logger *slog.Logger
// TranscriptLimit caps the gzipped transcript. Default: DefaultTranscriptLimit.
TranscriptLimit int64
// DefaultWallClock applies when the task has no maxWallClock. Default:
// DefaultWallClock.
DefaultWallClock time.Duration
// BotName and BotEmail are the commit author set in the writable checkout.
BotName, BotEmail string
// OnStep, when set, is called after each step that ran with its name, its duration and
// its error (nil on success): workspace, credentials, clone, environment (only for
// a task that declares one), skills, schema, prompt, env, prepare, harness, repair (only
// after an invalid Result) and finish (its error is the transcript's). The error is redacted with the run's redactor
// (its tokens and anything shaped like one): its Error() is the redacted text, and it
// unwraps to the step's own error, so errors.Is and errors.As still match. An error
// unwrapped from it is the original, unredacted: report the error OnStep gets, not one
// errors.Unwrap or errors.As returns.
OnStep func(name string, d time.Duration, err error)
// Finished, when set, is called once with the run's Outcome at the end of finish, after
// the transcript is stored and before cleanup (the environment's Down, the credential socket,
// the workspace's removal). It isn't called when Run returns an error.
Finished func(runner.Outcome)
}
Options configure a Pipeline. Exactly one of WorkRoot and Root is required, and Adapters with Root. Run needs Root and GitAuthGateway (the VM workspace); PrepareBundle WorkRoot and GitAuthHelper (supervise, on the host).
type Pipeline ¶
type Pipeline struct {
// contains filtered or unexported fields
}
Pipeline runs leased tasks. It is safe for concurrent use: each run has its own workspace.
func (*Pipeline) PrepareBundle ¶
func (p *Pipeline) PrepareBundle(ctx context.Context, job *runner.Job, dir string, x BundleExtras) (err error)
PrepareBundle writes job's run bundle at dir, on the host, before a VM exists (workspace-run.md §pipeline.PrepareBundle). It fetches the rendered prompt (checked against spec.prompt.digest), each non-repo context item (a 404 is Gone) and the output schema (a 404 is no schema) through the pipeline's sources, with the bounds and refusals of the prompt and schema steps; materialises the skills with the pipeline's git (its gitcred socket lives only for the call) and skills cache into the scratch directory WorkRoot/.bundle-<id> (bundleScratch), which replaces a leftover and is always removed; keeps the job's git credentials without their tokens (runbundle.GitAccess); refuses a bundle any of whose bytes hold one of the job's real secrets (checkSecrets); and, unless ctx is done, calls runbundle.Write, which refuses an existing dir. No token reaches the bundle. Its errors name the step that failed and are redacted as the host steps' are; a caller fails the run with harness-error, as the matching host step would. It refuses a workspace pipeline (Root or GitAuthGateway): the bundle is made on the host.
func (*Pipeline) Run ¶
func (p *Pipeline) Run(ctx context.Context, job *runner.Job, sink runner.Sink) (runner.Outcome, error)
Run implements runner.Runner (run-pipeline.md §Steps).
func (*Pipeline) StoreEnvironmentLog ¶
func (p *Pipeline) StoreEnvironmentLog(ctx context.Context, job *runner.Job, log []byte, extra ...string) error
StoreEnvironmentLog uploads a workspace's environment log (out/environment.log, untrusted) as the transcript is: redacted again with the job's secrets, extra (the supervisor's model logins) and every token shape, and capped at MaxEnvironmentLog, through the job's dispatcher client (PUT /runs/{id}/environment-log). A lease that names no run, or a job without a client, uploads nothing. Its errors name the request.
func (*Pipeline) StoreTranscript ¶
func (p *Pipeline) StoreTranscript(ctx context.Context, job *runner.Job, parts [][]byte, extra ...string) (string, error)
StoreTranscript is the host's transcript step for a run that happened elsewhere (a VM workspace's, whose transcript the supervisor holds as untrusted data): parts' lines are redacted with job's secrets and extra, capped at TranscriptLimit as buildTranscript does, and handed to the TranscriptStore (APITranscripts: PUT /runs/{id}/transcript). It returns the reference.
type PrebuiltSkills ¶
PrebuiltSkills is the SkillSource of skills materialised before the run: a run bundle's skills/ (Dir) and its manifest's list (List, each Dir absolute under Dir), returned as they are. The workspace's Skills is then Dir, which the adapters only read.
func BundleSkills ¶
func BundleSkills(b *runbundle.Bundle) PrebuiltSkills
BundleSkills is the PrebuiltSkills of a run bundle: its skills/ and its manifest's skills.
type PromptSource ¶
type PromptSource interface {
// Prompt is the rendered template, byte-exact (its digest is checked against
// spec.prompt.digest).
Prompt(ctx context.Context, job *runner.Job) ([]byte, error)
// Context is the non-repo context item i. gone is true when the item no longer exists
// (the API's 404); c is then ignored.
Context(ctx context.Context, job *runner.Job, i int) (c apiclient.TaskContext, gone bool, err error)
}
PromptSource gives a task's rendered prompt template and its context items (workspace-run.md §Sources). renderPrompt keeps the digest check, the size caps, the preamble, the fencing and the tickets; a source only fetches. Its errors fail the prompt step as they are, so each names its operation (APIPrompts: the request).
type SchemaSource ¶
SchemaSource gives a task's spec-specific output schema (the TaskSpec's output.schema, a JSON Schema document). A nil document means the TaskSpec declares none: the Result is then checked against the v1alpha1 Result schema and the task's outcomes only, as the dispatcher checks it on complete.
type SkillSource ¶
type SkillSource interface {
Skills(ctx context.Context, job *runner.Job, dst string) (dir string, list []harness.Skill, err error)
}
SkillSource gives a run its materialised skills (workspace-run.md §Sources): dir is the workspace's Skills directory (the plugin directories) and list the selected skills, their Dir absolute under dir. dst is the path of the workspace's own skills/, which the pipeline creates only for FetchSkills: another source that fills it creates it (0700) and returns it; a source with the skills elsewhere returns that directory instead.
type TranscriptStore ¶
type TranscriptStore interface {
Store(ctx context.Context, job *runner.Job, gz []byte) (ref string, err error)
}
TranscriptStore keeps a run's transcript: gz is the redacted, capped, gzipped JSON Lines buildTranscript made. ref is the Outcome's TranscriptRef; "" when the store returns none (nothing was stored, or someone else uploads it). Its errors are logged as they are, so each names its operation (APITranscripts: the request).