pipeline

package
v0.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 29, 2026 License: AGPL-3.0 Imports: 36 Imported by: 0

Documentation

Overview

Package pipeline is the run pipeline (design/workstreams/agent/LLD/run-pipeline.md): it turns one leased Task into a harness run and the run into a validated Result. It implements runner.Runner, so the worker's lease loop drives it.

For each job it creates a scratch workspace (0700, always removed; or the exact Root, kept), starts an in-memory git credential helper (package gitcred) for the task's repos (or, in gateway mode, none: git goes through the gateway's proxy), clones them, materialises the pinned skills, renders the prompt (the headless preamble, the rendered template from the API, and the context items fenced with their trust level), builds the harness's environment from an allow-list, runs the harness through its harness.Adapter under the task's limits, validates the Result against the output schema and outcomes (with one repair turn), uploads the redacted, gzipped transcript, and cleans up.

Index

Constants

View Source
const (
	EnvHeadless = "FACTORY_HEADLESS"
	EnvHarness  = "FACTORY_HARNESS"
	EnvTaskID   = "FACTORY_TASK_ID"
	EnvAPI      = "FACTORY_API"
	EnvToken    = "FACTORY_TOKEN"
	EnvOut      = "FACTORY_OUT"
	EnvResult   = "FACTORY_RESULT"
	EnvGHToken  = "GH_TOKEN"
)

Harness environment variables (run-pipeline.md §Steps 7, the ccf-skills HEADLESS.md contract).

View Source
const (
	EnvRuntime       = "FACTORY_RUNTIME"
	RuntimeWorkspace = "workspace"
)

EnvRuntime names the runtime a run's processes are in: RuntimeWorkspace, a VM workspace (workspace-run.md §System setup), where vmrun sets it through Options.Env. The Effector reads it and refuses any other value.

View Source
const (
	// DefaultWallClock bounds a run whose TaskSpec sets no limits.maxWallClock (the
	// worker's drain deadline uses the same default).
	DefaultWallClock = time.Hour
	// DefaultTranscriptLimit is the API's transcript cap (contracts.md: ≤ 20 MB gzipped).
	DefaultTranscriptLimit = 20 << 20
	// DefaultSkillsBase prefixes a skills source that isn't a URL ("ccf-skills").
	DefaultSkillsBase = "https://github.com/compliance-framework"
	// DefaultBotName and DefaultBotEmail are the commit author in the rw checkout.
	DefaultBotName  = "ccf-factory[bot]"
	DefaultBotEmail = "ccf-factory[bot]@users.noreply.github.com"
)

Defaults.

View Source
const EffectorFile = "effector.json"

EffectorFile is the EffectorConfig's name in the workspace root (outside out/, which the harness writes).

View Source
const EnvEffector = "FACTORY_EFFECTOR"

EnvEffector names the variable that carries the path of the run's EffectorConfig, which the Effector CLI and MCP server (`factory`, effector.md) read.

View Source
const EnvironmentLogTruncated = "[truncated]\n"

EnvironmentLogTruncated is the line that ends a cut environment log.

View Source
const MaxEnvironmentLog = 10 << 20

MaxEnvironmentLog caps the environment log a VM workspace's run uploads (the VM caps it at envrun.MaxLog; past this cap it is cut on a line and ends with EnvironmentLogTruncated).

View Source
const PreambleVersion = "2"

PreambleVersion versions Preamble: bump it with any change to the text.

View Source
const Redacted = "[REDACTED]"

Redacted replaces every secret in a transcript, a progress summary or an error.

View Source
const StackDownTimeout = 3 * time.Minute

StackDownTimeout bounds a VM environment's down script (envrun's DownTimeout, vmrun's --down-timeout): local-dev's compose down stops a dozen containers and removes their volumes, and the showcase profile also deletes a kind cluster, which takes longer than cleanupTimeout. The name predates the environment: it was the host stack's budget.

Variables

This section is empty.

Functions

func BundleSchema

func BundleSchema(doc []byte) ([]byte, error)

BundleSchema makes a task's output schema self-contained for a harness CLI, which can't resolve the v1alpha1 Result schema by its remote $id. When the document refs that schema (v1alpha1.SchemaBaseURL + ResultSchemaFile, with or without a fragment), the schema is copied into $defs["factory-result"] (without its $schema and $id) and those refs, and the copy's own internal refs, point there. A document that refs nothing remote is returned unchanged; any other remote $ref is an error. The pipeline's own validation still compiles the original documents.

func FailedResult

func FailedResult(task, reason string, err error, redact func(string) string) v1alpha1.Result

FailedResult is the failed Result of a run that ended without one of its own, the pipeline's and the VM's (workspace-run.md, decided 2026-10-22): its summary is reason, then ": " and err's message redacted by redact (nil: by the token shapes alone, RedactShapes) when err isn't nil, cut to 500 bytes on a rune boundary.

func Preamble

func Preamble(taskID, taskSpec string) string

Preamble is the headless preamble for a task.

func Redact

func Redact(job *runner.Job, s string, extra ...string) string

Redact redacts s as the run's redactor does: job's secrets (its git tokens, its agent token, read now, and the lease's run token), extra (secrets the job doesn't hold, such as the supervisor's model logins) and every token shape. The supervisor redacts what a VM workspace reports with it.

func RedactShapes

func RedactShapes(s string) string

RedactShapes redacts s by the token shapes alone, as a run that holds no token of its own (a VM workspace's) does.

func ValidateResult

func ValidateResult(taskID, taskSpec string, spec []byte, outcomes []string, doc []byte) []string

ValidateResult checks a Result document as the pipeline will after the run: the v1alpha1 Result schema, the spec schema (nil: none), the status and the outcome against outcomes. taskSpec names the spec schema's $id. It returns the problems ("/<pointer>: message"), none when the document is valid. `factory result` uses it for fast feedback; the pipeline's own check (with the harness block filled in) still decides.

Types

type APIPrompts

type APIPrompts struct{}

APIPrompts is the default PromptSource: GET /tasks/{id}/prompt and GET /tasks/{id}/context/{i} through the job's dispatcher client.

func (APIPrompts) Context

func (APIPrompts) Context(ctx context.Context, job *runner.Job, i int) (apiclient.TaskContext, bool, error)

Context implements PromptSource.

func (APIPrompts) Prompt

func (APIPrompts) Prompt(ctx context.Context, job *runner.Job) ([]byte, error)

Prompt implements PromptSource.

type APISchemas

type APISchemas struct{}

APISchemas is the default SchemaSource: GET /tasks/{id}/output-schema (getTaskOutputSchema) through the job's dispatcher client, the schema the dispatcher validates the Result with on complete. Only a 404 means no schema; any other failure fails the run, since a Result checked without the schema could be refused on complete without its repair turn.

func (APISchemas) OutputSchema

func (APISchemas) OutputSchema(ctx context.Context, job *runner.Job) ([]byte, error)

OutputSchema implements SchemaSource.

type APITranscripts

type APITranscripts struct{}

APITranscripts is the default TranscriptStore: PUT /runs/{id}/transcript through the job's dispatcher client. A lease that names no run, or a job without a client, stores nothing and returns "".

func (APITranscripts) Store

func (APITranscripts) Store(ctx context.Context, job *runner.Job, gz []byte) (string, error)

Store implements TranscriptStore.

type BundleExtras

type BundleExtras struct {
	CA         []byte
	Dispatcher string
	Codex      *runbundle.CodexAccount
}

BundleExtras are what a run bundle holds beyond the job (workspace-run.md §pipeline.PrepareBundle): the gateway's CA (the public half), the dispatcher's URL as the VM sees it ("": runbundle.DefaultDispatcherURL) and the codex account, set exactly when the task's harness is codex.

type BundlePrompts

type BundlePrompts struct {
	B *runbundle.Bundle
}

BundlePrompts is the PromptSource of a run bundle: its prompt.md and context/<n>.json (a Gone item is the 404 case). renderPrompt checks the digest, as for the API's.

func (BundlePrompts) Context

Context implements PromptSource.

func (BundlePrompts) Prompt

func (s BundlePrompts) Prompt(_ context.Context, job *runner.Job) ([]byte, error)

Prompt implements PromptSource.

type BundleSchemas

type BundleSchemas struct {
	B *runbundle.Bundle
}

BundleSchemas is the SchemaSource of a run bundle: its output-schema.json, or nil when the bundle has none (the API answered 404).

func (BundleSchemas) OutputSchema

func (s BundleSchemas) OutputSchema(_ context.Context, job *runner.Job) ([]byte, error)

OutputSchema implements SchemaSource.

type EffectorConfig

type EffectorConfig struct {
	Task     string `json:"task"`
	TaskSpec string `json:"taskSpec,omitempty"`
	Provider string `json:"provider,omitempty"`
	// Repo is the task's target repo (owner/name).
	Repo string `json:"repo,omitempty"`
	// Allow is the task's effects.allow (empty: nothing is allowed); Labels its
	// effects.labels globs.
	Allow  []string `json:"allow"`
	Labels []string `json:"labels,omitempty"`
	// Outcomes are the task's output.outcomes.
	Outcomes []string `json:"outcomes,omitempty"`
	// RepoDir is the primary checkout (repo/), where git ops run; empty without one.
	RepoDir string `json:"repoDir,omitempty"`
	// Schema is the file the Result must match (the spec's schema, else the Result schema).
	Schema string `json:"schema,omitempty"`
}

EffectorConfig is what the Effector knows about the run's task besides the FACTORY_* variables: the policy for its local pre-check (the dispatcher re-checks it and is authoritative), the repo bare refs (#12) resolve in, the checkout its git ops run in, and what `factory result` validates against.

func EffectorConfigFor

func EffectorConfigFor(t v1alpha1.Task, ws harness.Workspace) EffectorConfig

EffectorConfigFor is the EffectorConfig of a run of t in ws.

type Environment

type Environment interface {
	Up(ctx context.Context, ws harness.Workspace, t v1alpha1.Task) (map[string]string, error)
	Down(ctx context.Context) error
}

Environment brings up what a task's VM workspace declares in spec.environment (P1 §4, workspace-run.md §The environment step): package envrun implements it. Up runs the setup and up scripts and waits for ready; it returns the variables to add to the harness's environment (the exposes and FACTORY_VERSION_*). ws.Env is the environment the pipeline's own git has (commonEnv), not the harness's. Down is best effort: cleanup calls it once whenever Up was called, even when Up failed part-way, with no deadline of its own (the Environment owns its budget).

type FetchSkills

type FetchSkills struct{}

FetchSkills is the default SkillSource: each pinned skills repo is fetched with the run's git (through the skills cache, by commit) and the selected skills are materialised into the workspace's skills/ (run-pipeline.md §Steps 5). It needs the run's git, so only the pipeline runs it: its Skills method, called on its own, is an error.

func (FetchSkills) Skills

Skills implements SkillSource. The pipeline runs FetchSkills itself, with the run's git and the skills cache; called on its own it fails.

type FileTranscript

type FileTranscript struct {
	Path string
}

FileTranscript is the TranscriptStore of a VM workspace (workspace-run.md §Sources): it writes the gzip to Path, through a temp file beside it and a rename, and returns no ref: the supervisor uploads it. Path's directory must exist.

func (FileTranscript) Store

func (s FileTranscript) Store(_ context.Context, _ *runner.Job, gz []byte) (string, error)

Store implements TranscriptStore.

type GatewayCredentialError

type GatewayCredentialError struct{ Field string }

GatewayCredentialError is a gateway-mode run's refusal of a job that carries a credential: in a VM workspace every credential lives in the gateway, never in the job. Field names the field (Lease.GitCredential.Token, Git.Reads[0].Token, AgentToken, …), never its value.

func (*GatewayCredentialError) Error

func (e *GatewayCredentialError) Error() string

type GitAuth

type GitAuth int

GitAuth is how the run's git authenticates to the git host (workspace-run.md §Gateway mode).

const (
	// GitAuthHelper, the zero value (the host), answers git from the job's credentials through
	// the in-memory credential helper (package gitcred) on a Unix socket.
	GitAuthHelper GitAuth = iota
	// GitAuthGateway (the VM workspace) has no credential helper: no socket and no helper
	// config. git reaches the git host through the gateway (HTTPS_PROXY in Options.Env), which
	// adds the credential for the repo in the path. The job carries no credential: Run refuses
	// one that does (GatewayCredentialError) before any step.
	GitAuthGateway
)

func (GitAuth) String

func (a GitAuth) String() string

String implements fmt.Stringer.

type Options

type Options struct {
	// WorkRoot holds PrepareBundle's scratch directories (workRoot/.bundle-<taskID>/), each
	// removed after its bundle is written (the host).
	WorkRoot string
	// Root is the exact workspace root, instead of WorkRoot (the VM workspace: /work). It must
	// be absolute, and empty or absent when a run starts (its parent must exist); the run lays
	// its tree directly in it and cleanup never removes it, so out/ survives the run. One run
	// at a time: a second finds it non-empty and fails its workspace step.
	Root string
	// GitAuth is how the run's git authenticates. Default: GitAuthHelper.
	GitAuth GitAuth
	// Env is added to the environment that the pipeline's own git and the harness share,
	// after the owner's allow-list (so it wins over Environ) and before the pipeline's own
	// variables: the gateway's proxies (HTTP_PROXY, HTTPS_PROXY, NO_PROXY, either case) and CA
	// (SSL_CERT_FILE, GIT_SSL_CAINFO, NODE_EXTRA_CA_CERTS, …), a GH_TOKEN placeholder,
	// DOCKER_CONFIG. New refuses a variable the pipeline sets itself (HOME, TMPDIR, the XDG
	// and PATH ones, the GIT_* and GCM_* ones of the git setup, the credential socket and the
	// FACTORY_* ones of the harness's environment), and one that
	// could add a credential helper or change what git or the shell runs: every GIT_* name
	// but GIT_SSL_CAINFO and GIT_SSL_CAPATH, SSH_ASKPASS, LD_*, DYLD_*, BASH_ENV and ENV. On
	// the host it is usually empty.
	Env map[string]string
	// Adapters are the harness adapters by harness name (task.harness.resolved.name).
	Adapters map[string]harness.Adapter
	// AgentBinary is the factory-agent executable: the credential helper and the `factory`
	// CLI on the harness's PATH run it. Default: os.Executable().
	AgentBinary string
	// HelperCommand is the credential helper's command line, without the operation git
	// appends. Default: AgentBinary fx git-credential.
	HelperCommand []string
	// Git is the git binary. Default: "git".
	Git string
	// GitBaseURL is where the task's repos are cloned from (<base>/<owner>/<name>.git).
	// Default: https://<the credential's host>.
	GitBaseURL string
	// SkillsBase prefixes a skills source that isn't a URL. Default: DefaultSkillsBase.
	SkillsBase string
	// SkillsCache caches skills checkouts by commit (<cache>/<commit>). Empty: no cache.
	SkillsCache string
	// SocketDir is where the credential socket's directory is created. Default:
	// os.TempDir() (Unix socket paths must stay short).
	SocketDir string
	// Schemas gives spec-specific output schemas. Default: APISchemas, the dispatcher's
	// getTaskOutputSchema through the job's client.
	Schemas SchemaSource
	// Prompts gives the rendered prompt template and the context items. Default:
	// APIPrompts, the dispatcher's getTaskPrompt and getTaskContext through the job's client.
	Prompts PromptSource
	// Skills gives the run its materialised skills. Default: FetchSkills, the pinned skills
	// repos fetched with the run's git (through SkillsCache) into the workspace's skills/.
	Skills SkillSource
	// Transcripts keeps the redacted, gzipped transcript. Default: APITranscripts, the
	// dispatcher's uploadTranscript through the job's client.
	Transcripts TranscriptStore
	// Environment brings up a task's spec.environment (the VM workspace's envrun); nil on the
	// host, where such a task's environment is skipped with a warning.
	Environment Environment
	// Environ is the owner's environment, reduced to the allow-list for the harness.
	// Default: os.Environ.
	Environ func() []string
	// Clock times the run (wall-clock limit, usage); default clock.Real.
	Clock clock.Clock
	// Logger logs the pipeline's steps; never a token. Default: discard.
	Logger *slog.Logger
	// TranscriptLimit caps the gzipped transcript. Default: DefaultTranscriptLimit.
	TranscriptLimit int64
	// DefaultWallClock applies when the task has no maxWallClock. Default:
	// DefaultWallClock.
	DefaultWallClock time.Duration
	// BotName and BotEmail are the commit author set in the writable checkout.
	BotName, BotEmail string
	// OnStep, when set, is called after each step that ran with its name, its duration and
	// its error (nil on success): workspace, credentials, clone, environment (only for
	// a task that declares one), skills, schema, prompt, env, prepare, harness, repair (only
	// after an invalid Result) and finish (its error is the transcript's). The error is redacted with the run's redactor
	// (its tokens and anything shaped like one): its Error() is the redacted text, and it
	// unwraps to the step's own error, so errors.Is and errors.As still match. An error
	// unwrapped from it is the original, unredacted: report the error OnStep gets, not one
	// errors.Unwrap or errors.As returns.
	OnStep func(name string, d time.Duration, err error)
	// Finished, when set, is called once with the run's Outcome at the end of finish, after
	// the transcript is stored and before cleanup (the environment's Down, the credential socket,
	// the workspace's removal). It isn't called when Run returns an error.
	Finished func(runner.Outcome)
}

Options configure a Pipeline. Exactly one of WorkRoot and Root is required, and Adapters with Root. Run needs Root and GitAuthGateway (the VM workspace); PrepareBundle WorkRoot and GitAuthHelper (supervise, on the host).

type Pipeline

type Pipeline struct {
	// contains filtered or unexported fields
}

Pipeline runs leased tasks. It is safe for concurrent use: each run has its own workspace.

func New

func New(o Options) (*Pipeline, error)

New checks the options and fills in the defaults.

func (*Pipeline) PrepareBundle

func (p *Pipeline) PrepareBundle(ctx context.Context, job *runner.Job, dir string, x BundleExtras) (err error)

PrepareBundle writes job's run bundle at dir, on the host, before a VM exists (workspace-run.md §pipeline.PrepareBundle). It fetches the rendered prompt (checked against spec.prompt.digest), each non-repo context item (a 404 is Gone) and the output schema (a 404 is no schema) through the pipeline's sources, with the bounds and refusals of the prompt and schema steps; materialises the skills with the pipeline's git (its gitcred socket lives only for the call) and skills cache into the scratch directory WorkRoot/.bundle-<id> (bundleScratch), which replaces a leftover and is always removed; keeps the job's git credentials without their tokens (runbundle.GitAccess); refuses a bundle any of whose bytes hold one of the job's real secrets (checkSecrets); and, unless ctx is done, calls runbundle.Write, which refuses an existing dir. No token reaches the bundle. Its errors name the step that failed and are redacted as the host steps' are; a caller fails the run with harness-error, as the matching host step would. It refuses a workspace pipeline (Root or GitAuthGateway): the bundle is made on the host.

func (*Pipeline) Run

func (p *Pipeline) Run(ctx context.Context, job *runner.Job, sink runner.Sink) (runner.Outcome, error)

Run implements runner.Runner (run-pipeline.md §Steps).

func (*Pipeline) StoreEnvironmentLog

func (p *Pipeline) StoreEnvironmentLog(ctx context.Context, job *runner.Job, log []byte, extra ...string) error

StoreEnvironmentLog uploads a workspace's environment log (out/environment.log, untrusted) as the transcript is: redacted again with the job's secrets, extra (the supervisor's model logins) and every token shape, and capped at MaxEnvironmentLog, through the job's dispatcher client (PUT /runs/{id}/environment-log). A lease that names no run, or a job without a client, uploads nothing. Its errors name the request.

func (*Pipeline) StoreTranscript

func (p *Pipeline) StoreTranscript(ctx context.Context, job *runner.Job, parts [][]byte, extra ...string) (string, error)

StoreTranscript is the host's transcript step for a run that happened elsewhere (a VM workspace's, whose transcript the supervisor holds as untrusted data): parts' lines are redacted with job's secrets and extra, capped at TranscriptLimit as buildTranscript does, and handed to the TranscriptStore (APITranscripts: PUT /runs/{id}/transcript). It returns the reference.

type PrebuiltSkills

type PrebuiltSkills struct {
	Dir  string
	List []harness.Skill
}

PrebuiltSkills is the SkillSource of skills materialised before the run: a run bundle's skills/ (Dir) and its manifest's list (List, each Dir absolute under Dir), returned as they are. The workspace's Skills is then Dir, which the adapters only read.

func BundleSkills

func BundleSkills(b *runbundle.Bundle) PrebuiltSkills

BundleSkills is the PrebuiltSkills of a run bundle: its skills/ and its manifest's skills.

func (PrebuiltSkills) Skills

Skills implements SkillSource.

type PromptSource

type PromptSource interface {
	// Prompt is the rendered template, byte-exact (its digest is checked against
	// spec.prompt.digest).
	Prompt(ctx context.Context, job *runner.Job) ([]byte, error)
	// Context is the non-repo context item i. gone is true when the item no longer exists
	// (the API's 404); c is then ignored.
	Context(ctx context.Context, job *runner.Job, i int) (c apiclient.TaskContext, gone bool, err error)
}

PromptSource gives a task's rendered prompt template and its context items (workspace-run.md §Sources). renderPrompt keeps the digest check, the size caps, the preamble, the fencing and the tickets; a source only fetches. Its errors fail the prompt step as they are, so each names its operation (APIPrompts: the request).

type SchemaSource

type SchemaSource interface {
	OutputSchema(ctx context.Context, job *runner.Job) ([]byte, error)
}

SchemaSource gives a task's spec-specific output schema (the TaskSpec's output.schema, a JSON Schema document). A nil document means the TaskSpec declares none: the Result is then checked against the v1alpha1 Result schema and the task's outcomes only, as the dispatcher checks it on complete.

type SkillSource

type SkillSource interface {
	Skills(ctx context.Context, job *runner.Job, dst string) (dir string, list []harness.Skill, err error)
}

SkillSource gives a run its materialised skills (workspace-run.md §Sources): dir is the workspace's Skills directory (the plugin directories) and list the selected skills, their Dir absolute under dir. dst is the path of the workspace's own skills/, which the pipeline creates only for FetchSkills: another source that fills it creates it (0700) and returns it; a source with the skills elsewhere returns that directory instead.

type TranscriptStore

type TranscriptStore interface {
	Store(ctx context.Context, job *runner.Job, gz []byte) (ref string, err error)
}

TranscriptStore keeps a run's transcript: gz is the redacted, capped, gzipped JSON Lines buildTranscript made. ref is the Outcome's TranscriptRef; "" when the store returns none (nothing was stored, or someone else uploads it). Its errors are logged as they are, so each names its operation (APITranscripts: the request).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL