workspace

package
v0.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 25, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Package workspace materializes the per-step/per-build filesystem views a job's steps run against: per-step directories, copied (copy-on-write where the filesystem offers it), built from each step's declared inputs/outputs. It also statically validates a job's declared artifact flow before any step runs.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ClearTaskCaches added in v0.2.0

func ClearTaskCaches(ws *config.WorkspaceConfig, keys []string) (map[string]bool, error)

ClearTaskCaches deletes the caches keyed by keys under ws's root, reporting which were there; ws must name a root, since under a temp one no cache outlives its run.

func CloseBuild

func CloseBuild(bw BuildWorkspace, label string)

CloseBuild is a best-effort cleanup helper for the pipeline/agent packages' deferred Close calls: a cleanup failure must never mask the original error already being returned, so it's only logged.

func CloseSpace

func CloseSpace(space StepSpace, label string)

CloseSpace is CloseBuild's StepSpace counterpart.

func CopyTree

func CopyTree(ctx context.Context, src, dst string) error

CopyTree duplicates a directory tree, exported for the one caller outside this package: a replay forks the workspace of the run it re-executes, rather than editing the baseline it is being compared against.

Same implementation the copy strategy uses for step isolation, so a replay inherits its platform-specific fast paths (clonefile, reflink) instead of growing a second, slower copier.

func Kept

func Kept(space StepSpace) bool

Kept reports whether space survives its own Close (--keep-workspace), so a caller that puts scratch state inside the step's directory can leave it in place for the same postmortem the directory itself is being kept for — see internal/agent's spill directory. False for a space that does not implement the optional capability.

func SaveStepCache

func SaveStepCache(ctx context.Context, bw BuildWorkspace, key string, req StepCacheRequest)

SaveStepCache files a just-finished step's outputs under the key its lookup returned. A zero key means the lookup found no cache to file into, so there is nothing to do.

func TaskCacheKey added in v0.2.0

func TaskCacheKey(pipeline, job, step, path string) string

TaskCacheKey names one cache: Concourse's job + step name + path, under the pipeline that owns the job.

func TaskCacheKeys added in v0.2.0

func TaskCacheKeys(pipeline, job, step string, caches []config.TaskCache) map[string]string

TaskCacheKeys keys every one of a step's caches by its cleaned path, the form AttachCaches takes.

func ValidateArtifactFlow

func ValidateArtifactFlow(cfg *config.Config, job *config.Job) error

ValidateArtifactFlow statically checks that every task/agent/put step's declared inputs name an artifact available by that point in the plan (a resource an earlier get fetches, or an output an earlier task/agent produces), plus an agent step's dir: (which names the artifact it works in). It runs for every job, so a step that declares inputs: [x] where nothing produces x — the classic "this job never fetched anything" mistake — fails here rather than obscurely at run time. It runs no check/in/out command — unlike PlanChains, it is always safe to run, even under --force, which skips PlanChains entirely. get's version:every fans out per resolved version but never changes which artifact *names* exist, so a single linear walk over the plan covers every branch that fan-out could produce.

Crucially, a get starts a *fresh* triggered build (see runTriggeredBuild) whose artifact store is empty except for the resource it fetches — it does not inherit artifacts produced before it. So a get resets the available set to just its own resource rather than adding to it; otherwise this check would pass an input referencing a pre-get artifact that the runtime can't actually see, exactly the late failure it exists to prevent.

Types

type BlobStore

type BlobStore interface {
	HasTree(ctx context.Context, digest string) (bool, error)
	PutTree(ctx context.Context, digest, dir string) error
	GetTree(ctx context.Context, digest, dir string) error
}

BlobStore is remote bytes by content digest: internal/blobstore's shape, declared where it is consumed. The digest is digestTree's, which is what lets what came back be VERIFIED — a blob is never trusted to be what its key claims.

type BuildWorkspace

type BuildWorkspace interface {
	// ResourceDir returns the (empty, freshly created) directory a get step
	// should fetch resource name into.
	ResourceDir(ctx context.Context, name string) (string, error)
	// TaskSpace materializes a task or agent step's working directory: a
	// directory containing only an <input>/ copy or snapshot of each named
	// input plus an empty <output>/ directory for each named output. inputMapping/
	// outputMapping rename a declared name to the plan-artifact name it draws
	// from / is captured as (see config.Step.InputMapping); nil leaves names
	// unmapped. Agent steps carry no author-written mapping, but a cell of a
	// collecting matrix (task or agent) passes the machine-composed
	// coordinate mapping config.CollectedOutputMapping builds.
	TaskSpace(ctx context.Context, label string, inputs, outputs []string, inputMapping, outputMapping map[string]string) (StepSpace, error)
	// PutSpace composes a put step's read view from its declared inputs, or —
	// when all is true (inputs: all) — from every artifact in the build store.
	// Unlike TaskSpace/agent steps, a put step never has outputs of its own.
	PutSpace(ctx context.Context, label string, inputs []string, all bool) (StepSpace, error)
	// GuardSpace composes a when: guard's read view: the same inputs the
	// step's own TaskSpace/PutSpace would materialize — under the same
	// inputMapping, and every artifact in the store when all is true — minus
	// any the build store does not hold.
	//
	// A guard decides whether its step runs at all, and the question it is
	// usually asked — is there anything in this artifact worth publishing —
	// is about an artifact an earlier guarded step may legitimately never
	// have written. Demanding every input first made such a guard FAIL
	// instead of answer: the missing directory errored before the command
	// that would have said "nothing to do" ever ran. Absent is an answer
	// here, not an error.
	//
	// Only "not produced" is lenient. A name the step's own space would
	// reject is kept, so it is reported from the one place that reports it;
	// and the step's own space still requires every declared input, so a
	// misspelled name is still an error the moment the step itself runs.
	GuardSpace(ctx context.Context, label string, inputs []string, inputMapping map[string]string, all bool) (StepSpace, error)
	// PlacedTaskSpace is TaskSpace for a step that runs on a worker, leaving
	// inputs other workers hold on their holders, reported by declared name.
	PlacedTaskSpace(ctx context.Context, label string, inputs, outputs []string, inputMapping, outputMapping map[string]string) (StepSpace, map[string]RemoteArtifact, error)
	// PlacedPutSpace is PutSpace, the same way.
	PlacedPutSpace(ctx context.Context, label string, inputs []string, all bool) (StepSpace, map[string]RemoteArtifact, error)
	// HoldRemote records an output a placed step left on its worker.
	HoldRemote(name string, remote RemoteArtifact) error
	// FetchResource returns the directory for resource name, populated either
	// from the cache entry for cacheKey or by calling fetch. fetch receives the
	// directory to populate and is called at most once; when it is not called,
	// the directory already holds that version's content. With no cache
	// configured, or an empty cacheKey, it is ResourceDir followed by fetch.
	//
	// A cache failure is never a fetch failure: anything that goes wrong with
	// the cache falls back to calling fetch.
	FetchResource(ctx context.Context, name, cacheKey string, fetch func(dir string) error) (string, error)
	// RestoreStep computes a step's action key and, when an entry for it
	// holds every declared output, materializes them into the artifact store.
	// With no durable workspace.root: it answers "miss, no key" — there is
	// nowhere for an entry to outlive the run that wrote it.
	RestoreStep(ctx context.Context, req StepCacheRequest) (StepCacheResult, error)
	// StoreStep files the outputs a step just captured under key. Called only
	// after the step succeeded and space.Capture put its outputs in the store.
	StoreStep(ctx context.Context, key string, req StepCacheRequest) error
	// ResetArtifact replaces whatever the store holds under name with a fresh
	// empty directory, for a collecting across: (see isolatingBuild's).
	ResetArtifact(ctx context.Context, name string) error
	// AttachCaches puts each cache (path relative to dir → TaskCacheKey) in place before a step runs; the func it returns files them back and must run before the space closes.
	AttachCaches(ctx context.Context, dir string, caches map[string]string) func()
	// HasArtifact reports whether name is in the build store, or held by a
	// worker on the build's behalf, so a resume that keeps a get's artifact
	// can refuse a tree that lost it.
	HasArtifact(name string) bool
	// Root is where the build's files are: recorded as the run's workspace,
	// and printed for a resume to continue in.
	Root() string
	Close() error
}

BuildWorkspace is one triggered build's filesystem view: where get steps land pristine resource contents, and where task/agent/put steps draw their per-step workspace from.

type Provider

type Provider interface {
	Validate() error
	// NewBuild creates the workspace for one triggered build (see
	// runTriggeredBuild) — or, called once from RunJob, the job-level build
	// that steps preceding any get run in. Each call is fully independent:
	// it never shares state with a build from a previous call, matching
	// today's per-triggered-build isolation (see runTriggeredBuild's doc).
	NewBuild(ctx context.Context, label string) (BuildWorkspace, error)
	// Probe is Validate without the stale-build sweep, for a provider opened
	// on a root this process may already be building under: the sweep takes
	// every build directory there for a crashed process's leftovers.
	Probe() error
	// Reuse continues a previous run in an existing build tree, or with ""
	// stops doing so. It continues in the BUILD tree: what a provider tears
	// down per step is the step's own view, which a resume does not need —
	// the artifacts finished steps produced live at the build root, and
	// per-step views are materialized from them every time a step runs.
	Reuse(dir string)
	// Resuming is Reuse for one run: a provider whose every NewBuild answers with dir, leaving this one as it was, since a daemon builds other runs beside the one it resumes.
	Resuming(dir string) Provider
	// Spare keeps dirs out of the stale-build sweep, the trees of runs a restart resumes. Called before Validate, which sweeps.
	Spare(dirs []string)
	// AttachArtifactStore wires the two halves of the artifact store into the
	// step cache, reporting whether anything was attached — false with no
	// durable workspace.root:, where there is no local cache to mirror.
	AttachArtifactStore(blobs BlobStore, index StepIndex) bool
	Close() error
}

Provider is built once per CLI invocation from cfg.Workspace (nil taking every default — see NewProvider). It owns whatever on-disk root its strategy needs and fails fast, at startup, on anything that would only surface as a confusing mid-build error otherwise (wrong platform, wrong filesystem, missing binaries).

func NewProvider

func NewProvider(ws *config.WorkspaceConfig, keep bool) (Provider, error)

NewProvider builds the provider a pipeline's workspace: block selects. Every provider isolates: a step's directory is materialized from its own declared inputs:/outputs: and nothing else. ws == nil (no block present) takes every default — strategy copy under a system temp root. (The old shared single-mutable-directory mode, where every step saw everything, is gone: an artifact a step never declared reaching it anyway was ambient data flow, and every flow here is opt-in.)

keep leaves each build's directory on disk instead of removing it at Close, and prints where it is. A build workspace is otherwise destroyed unconditionally, including on the failure path — so the files an agent or task had just edited when it failed, the most useful thing to look at, were always already gone by the time the error reached the terminal.

type RemoteArtifact

type RemoteArtifact struct {
	// Name is what the holder filed the tree under, which the digest binds:
	// PackPaths over `name/` is what it hashed. Empty means the artifact's
	// own name. A tree captured as x through an output_mapping was filed as
	// the step's declared output, and can only be offered by digest under
	// THAT name.
	Name string
	// Digest names the tree as the holder filed it: treedigest's, the name a local copy of it has too, which the holder proves before handing it back.
	Digest string
	// Holder is the worker, as the mapping was written, for the error that
	// names it when the tree cannot be brought home.
	Holder string
	// Pull fills dst — an existing, empty directory the build created — with
	// the tree. Injected rather than imported: this package does not know
	// how a worker is dialled, and must not.
	Pull func(ctx context.Context, dst string) error
}

RemoteArtifact is where an artifact's bytes are when they are not here.

type StepCacheRequest

type StepCacheRequest struct {
	// ContentHash is the step's own hashed content (internal/merkle) — its
	// command or prompt, image, tool grant, declared inputs/outputs and
	// mappings. It identifies the WORK; the input digests below identify what
	// the work is being done to.
	ContentHash string
	// Inputs are the step's declared input names, and InputMapping renames a
	// declared name to the artifact it draws from (see materializeSpace).
	Inputs       []string
	InputMapping map[string]string
	// Outputs are the step's declared output names, and OutputMapping renames
	// a declared name to the artifact it is captured as. They are not part of
	// the key (ContentHash already folds both in) — they are what a hit has to
	// restore, and what a miss will later store.
	Outputs       []string
	OutputMapping map[string]string
}

StepCacheRequest is one step's identity for cache purposes: what the plan says the step IS, and which artifacts it reads.

type StepCacheResult

type StepCacheResult struct {
	Key string
	Hit bool
}

StepCacheResult is what a lookup decided: the key this step's work is filed under, and whether the outputs were already there and have been restored into the artifact store.

func LookupStepCache

func LookupStepCache(ctx context.Context, bw BuildWorkspace, req StepCacheRequest) StepCacheResult

LookupStepCache asks bw to restore a step's outputs, reading an error as "miss, no key" — at the call site, "run the step, store nothing".

func (StepCacheResult) NodeResult

func (r StepCacheResult) NodeResult() map[string]any

NodeResult is what a reused step records in place of the result it did not produce.

Recorded rather than left nil so a succeeded node cannot be read as a step that ran and produced nothing — for an agent step, one whose transcript and token cost are simply absent. The key is included because it is the only handle on which earlier work is being reused.

type StepIndex

type StepIndex interface {
	StepBlobs(ctx context.Context, actionKey string) (map[string]string, error)
	RecordStepBlobs(ctx context.Context, actionKey string, outputs map[string]string) error
}

StepIndex is the durable action-key-to-output-digest mapping: internal/store's shape. The index is truth and stays on the orchestrator; the blob store holds bytes and nothing else.

type StepSpace

type StepSpace interface {
	Dir() string
	// Capture persists the step's declared outputs back into the build's
	// artifact store, so later steps can name them as inputs. Called only
	// after the step itself succeeded.
	Capture(ctx context.Context) error
	// Close removes anything TaskSpace/PutSpace/GuardSpace created for this
	// step.
	Close() error
}

StepSpace is one step's materialized working directory.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL