Documentation
¶
Overview ¶
Package resource runs a resource type's check/in/out shell commands and selects among the versions a check returns.
Index ¶
- func CheckVersions(ctx context.Context, c Call) ([]map[string]any, error)
- func CompileCrons(cfg *config.Config) error
- func CompileExprPrograms(cfg *config.Config) error
- func CompileWebhooks(cfg *config.Config) error
- func ParseVersionJSON(versionJSON string) (map[string]any, error)
- func Place(ctx context.Context, step config.Step) (context.Context, error)
- func Preflight(ctx context.Context, cfg *config.Config, job *config.Job, names []string, ...) []config.Problem
- func Receiver(res config.Resource) (*webhook.Receiver, error)
- func ResetPreflightCache()
- func ResolveVersions(ctx context.Context, cfg *config.Config, step config.Step, ...) (*config.Resource, *config.ResourceType, []map[string]any, error)
- func RunIn(ctx context.Context, c Call) error
- func RunOut(ctx context.Context, c Call) (map[string]any, error)
- func SelectVersion(versions []map[string]any, pinned map[string]string) (map[string]any, error)
- func UnquotedRefs(rt config.ResourceType) []string
- func WithCheckShare(ctx context.Context, share *CheckShare) context.Context
- func WithNow(ctx context.Context, now func() time.Time) context.Context
- func WithPlacerFor(ctx context.Context, fn PlacerFor) context.Context
- type Cache
- type CacheOption
- type Call
- type CheckShare
- type Mode
- type Placer
- type PlacerFor
- type PutInputs
- type Schedule
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CheckVersions ¶
CheckVersions renders c.Type.Config.Check against {"source": source, "version": c.Version}, runs it, and parses stdout as a JSON array into []map[string]any. Ordering (oldest-first) is entirely the check command's responsibility, per Concourse convention — no sorting happens here (see docs/conformance.md and internal/resource/resource_test.go's TestSelectVersion/"latest when unpinned"; Concourse doc: concourse-ci.org/docs/resource-types/implementing/, "check" section).
c.Version is the last version this pipeline recorded for the resource — Concourse's "current version", which lets a check ask its API for exactly what it has not seen (Slack's oldest:, GitHub's since:) instead of guessing a window. It is nil-normalized to an empty map HERE, the single place both backends and every call path agree on, so the first-ever check renders against a present-but-empty map rather than a missing key. Templates render with missingkey=error, so an optional cursor field is spelled {{ index .version "ts" | default "0" }} — the same shape an optional source: field or get param: already uses.
Only the trigger poll loop advances the cursor (see internal/trigger's pollOnce, which records after a successful check AND enqueue, so a failed poll never advances past items nobody saw). The run and plan paths read it and never write it.
c.Resource.Env is the resource INSTANCE's own env: — additional names unioned with the resource TYPE's env: for this call only, so one resource of a shared type can read a credential the type itself doesn't name. See config.Resource.Env for why this is additive, never a replacement.
The backend is chosen once, by backendFor; mcp is the one that reads c.Config, to resolve the referenced mcp_servers: entry.
func CompileCrons ¶
CompileCrons is Cron over every cron resource, for a caller that wants only the refusal.
func CompileExprPrograms ¶
CompileExprPrograms type-checks every expression in the pipeline without running any of them.
It exists because internal/config cannot: that package imports nothing internal and no third-party code but the YAML parser, which is what keeps the config types a leaf every other package can agree on. So a syntax error in an expression is not a load error. This is what makes it a `steps validate` error instead, and preflightResource makes it a preflight one — both before anything polls.
func CompileWebhooks ¶
CompileWebhooks is Receiver over every webhook resource, for a caller that wants only the refusal.
func ParseVersionJSON ¶
ParseVersionJSON decodes a stored version object — the JSON steps itself wrote into resource_checks.version_json — back into the map shape a check template renders against.
It decodes with UseNumber so a numeric field survives as its exact digits rather than a float64. That matters because these values go straight into a template and out over the wire: Slack's ts (1699887654.001200), a snowflake id, a build number. encoding/json's default float64 would render the first as 1.6998876540012e+09 and hand the API something it has never seen. The mcp path already makes the same promise with exactNumbers.
It lives here rather than in the store because both callers that hold a store (internal/trigger, internal/pipeline) must agree on it exactly — and because this package, which owns what a version IS, cannot import that one.
func Place ¶
Place resolves the placer for step and installs it for the stage calls that follow. Without a PlacerFor ctx comes back as it was; an untagged step installs a nil placer, which is what runs a stage here even under a ctx that already carried a worker for some other step.
func Preflight ¶
func Preflight( ctx context.Context, cfg *config.Config, job *config.Job, names []string, settings *config.Preflight, ) []config.Problem
Preflight checks the mcp-backed resource types behind the named resources and returns a problem per broken call. An empty result means every check/ in/out tool those resources use exists on its server and will be called with at least the arguments it requires.
job scopes the out: check to one job's own put steps (see Job.PutSteps): a `steps run` must not be blocked by how a job it is not running spells its put. A nil job judges every put in the pipeline, which is what `steps watch` wants — it will run them all.
A shell-backed resource type checks nothing here: its check is a command, and whether that command works is what running it answers.
func Receiver ¶
Receiver compiles one webhook resource into what internal/webhook runs. Every door a pipeline comes in by calls it, so a mistake is refused where the pipeline is set rather than at the first delivery.
func ResetPreflightCache ¶
func ResetPreflightCache()
ResetPreflightCache forgets every tool list this process has fetched. Tests use it to stay independent of each other.
func ResolveVersions ¶
func ResolveVersions( ctx context.Context, cfg *config.Config, step config.Step, cliPinned map[string]string, supplied func(resourceName string) []map[string]any, ) (*config.Resource, *config.ResourceType, []map[string]any, error)
ResolveVersions determines which version(s) of a get step's resource to fetch. An explicit CLI pin always wins (a single version); otherwise the step's own version: field decides between latest (default, a single version), every (all versions returned by check), or a YAML-pinned version. Both the merkle planner and the executor call ResolveVersions so plan-time hashing and run-time execution stay in lockstep.
supplied lets a caller hand over versions it has already resolved, in which case the check is not run at all. `steps web` does this: its poll asked the resource what was new — precisely, using the cursor — and asking again here would answer a different question (see WithResolvedVersions).
It is a callback rather than a value because this is the only place that knows the RESOLVED resource name: a get may alias it via `resource:`, and the caller holds the versions under the real name. The same reason WithConsumed is a callback, and the same reason both keep this package free of any dependency on the store.
A nil callback, or one returning nil, means nothing was supplied and the check runs as it always has.
func RunIn ¶
RunIn renders c.Type.Config.In against {"source": source, "version": c.Version, "params": c.Params} and executes it with cwd = c.Dir (caller ensures it exists).
c.Params is the get step's own params:, mirroring Concourse, where a get's params tell the resource HOW to fetch — git's depth:/submodules:, s3's unpack: (concourse-ci.org/docs/steps/get/; see docs/conformance.md). It is nil for a get that declares none, which renders as an empty map so {{ .params.x }} on an unset key is empty rather than an error — the same treatment out: already gives a put with no params:.
c.Resource.Env is the resource instance's own env: — see CheckVersions' doc comment.
An mcp-backed type goes to mcpRunIn — see its doc comment for the in:-omitted default (writing version.json) and the materialization convention when in: is set.
func RunOut ¶
RunOut renders c.Type.Config.Out against {"source": source, "params": c.Params} and executes it with cwd = c.Dir. If stdout parses as a JSON object it's returned as result (loosely mirroring check's convention of emitting the version produced); unparsable or empty stdout is not an error — result is simply nil, since many out scripts won't emit anything (see docs/conformance.md and TestConformanceRunOutUnparsableStdoutIsNilNotError in resource_test.go).
c.Resource.Env is the resource instance's own env: — see CheckVersions' doc comment.
An mcp-backed type goes to mcpRunOut. Its out: is itself optional (see validateMCPResourcePuts, which rejects a put step targeting an mcp-backed type with no out: at load time), so this is only ever reached with it set.
func SelectVersion ¶
SelectVersion returns the version matching all key/values in pinned, or (if pinned is empty) the last element of versions (latest, by convention).
func UnquotedRefs ¶
func UnquotedRefs(rt config.ResourceType) []string
UnquotedRefs reports each check/in/out action of a shell-backed type that prints .source, .version or .params without shellquote, as "<stage>: <action>". The roots are the keys the shell backend renders with.
func WithCheckShare ¶
func WithCheckShare(ctx context.Context, share *CheckShare) context.Context
WithCheckShare makes the shell checks under ctx that run on this machine share their answers through share. A check placed on a worker never does: where it runs is a function on the context, which no key can compare.
Types ¶
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
Cache memoizes ResolveVersions results within one RunJob invocation, keyed by the step fields resolution reads (cacheKey). A get step's resolved versions are a pure function of (cfg, step, pinned) — none of which change between the plan-time call (merkle.PlanChains) and the run-time call (the pipeline's runGetStep) within one invocation — so reusing the plan-time result for the run-time call is exact, not approximate, and avoids paying for the check command (and, for an imaged resource type, a docker container) twice per get step per run. A nil *Cache is a valid, always-miss receiver (see ResolveVersionsCached), so callers with no cross-phase reuse to offer (most tests) can simply not construct one.
func NewCache ¶
func NewCache(opts ...CacheOption) *Cache
NewCache returns an empty Cache, scoped to one RunJob invocation. Never share one Cache instance across concurrent RunJob calls (as steps web --max-concurrent > 1 makes possible) — each invocation must get its own, since a cached miss vs. hit is only valid within the single (cfg, pinned) combination one invocation resolves against.
func (*Cache) ResolveVersionsCached ¶
func (c *Cache) ResolveVersionsCached( ctx context.Context, cfg *config.Config, step config.Step, pinned map[string]string, ) (*config.Resource, *config.ResourceType, []map[string]any, error)
ResolveVersionsCached behaves exactly like ResolveVersions, but returns a memoized result for a resource name already resolved earlier through this same cache. A nil c always misses, equivalent to calling ResolveVersions directly.
func (*Cache) Suppressed ¶
Suppressed reports how many versions the consumed filter removed for the gets named step.Get already resolved through this cache, whatever their pins.
It exists so a caller seeing an empty version list can say WHICH empty it is: a check that returned nothing (the source is gone, or broken) versus a check whose every version this job had already taken (idle, and the normal state of a watched resource between arrivals). Those want opposite reactions, and the filter is what made them look alike.
type CacheOption ¶
type CacheOption func(*Cache)
CacheOption configures a Cache at construction.
func WithConsumed ¶
func WithConsumed(consumed func(resourceName string, version map[string]any) bool) CacheOption
WithConsumed filters a `version: every` fan-out down to the versions this job has NOT already taken.
It belongs on the Cache rather than inside ResolveVersions because the Cache is the single point both the plan-time caller (merkle.PlanChains) and the run-time one (the pipeline's runGetStep) go through: filtering here means the two cannot see different lists, which is the invariant that keeps a plan's hashes describing the run that follows. It is also what keeps this package free of any dependency on the store — the caller supplies the answer, this package only applies it.
Only `version: every` is filtered. Every other mode narrows to a single version whose repetition is already governed by the merkle cache.
func WithResolvedVersions ¶
func WithResolvedVersions(resolved func(resourceName string) []map[string]any) CacheOption
WithResolvedVersions supplies versions the caller already knows, so the check for that resource is not run again.
The caller is internal/pipeline reading recorded history (and, for a passed:-constrained get, the green subset of it — see pipeline.loadResourceHistory). Asking the check instead is not merely wasteful but WRONG for any cursor-driven type: the check answers "what is new since the cursor", which is the POLLER's question, and a job asking it gets everything or nothing depending on where the cursor happens to sit.
It rides on the Cache for the same reason WithConsumed does: the Cache is the one seam every resolver goes through — RunJob's planner and executor, and Explain — so none can resolve a different list than another. That is the invariant that keeps a plan's hashes describing the run that follows, and `steps plan` honest about what a run would do.
A nil return means "nothing recorded, go and check" — a resource no poll has ever filed. A non-nil EMPTY slice means the caller resolved none and is honored as such: for a gated resource, "nothing has passed" must fail the get, not fall back to a check the gate never sees.
type Call ¶
type Call struct {
// Config resolves an mcp-backed type's mcp_servers: entry; no other
// backend reads it, so nil is fine whenever the type is not mcp-backed.
Config *config.Config
Type config.ResourceType
// Resource supplies source: and the instance's own env: (see
// config.Resource.Env), unioned with Type's for this call only.
Resource config.Resource
// Version is a check's cursor or the version an in: fetches.
Version map[string]any
// Params is a get's or a put's params:.
Params map[string]any
// Inputs is what a put's out: may read of its build; only expr and
// github read it.
Inputs PutInputs
// Dir is an in:'s destination or an out:'s source tree.
Dir string
}
Call is one stage of one resource. CheckVersions, RunIn and RunOut each read only the fields their stage has.
type CheckShare ¶
type CheckShare struct {
// contains filtered or unexported fields
}
CheckShare runs a check once for every caller asking the identical question inside a window, and hands each the same answer.
A daemon polling many pipelines that watch the same repository ran the same `git ls-remote` once per pipeline per interval. Identical means the same rendered command — which folds in source: and the cursor — run the same way: image, env names, user, network, privilege, limits. Two pipelines whose cursors differ render different commands and share nothing. Only the execution is shared: each caller records its own history and triggers its own jobs, which is narrower than Concourse's opt-in global resources, where pipelines share one version history.
An answer is reused for ttl, which costs a caller at most that much staleness — the same as its poll having happened that much earlier. A failed check is never reused, only coalesced with callers already waiting on it.
func NewCheckShare ¶
func NewCheckShare(ttl time.Duration) *CheckShare
NewCheckShare shares answers for ttl.
type Mode ¶
type Mode string
Mode is how a get step picks among its resource's versions.
The modes, spelled as get.version: spells them.
func VersionMode ¶
VersionMode returns how a get step picks its version, and the pin when that is ModePinned, per Concourse's get.version field convention: unset or the string "latest" means latest, the string "every" means every version, and a map means pinned to a specific version.
type Placer ¶
type Placer func(shell.RunnerSpec) (shell.Runner, error)
Placer builds the runner one resource stage runs on. The stage fills in what it knows — image, tree, env, what to fetch back — and the placer adds the machine.
type PlacerFor ¶
PlacerFor answers the placer for one get or put step, or nil for a step that runs here. The step carries the tag; the caller maps it to a machine.
type PutInputs ¶
PutInputs is what a put's expr out: may read of its build through version(): its input names, and the versions this build's gets fetched for them, by get name. The shell and mcp backends ignore it.
type Schedule ¶
type Schedule struct {
// contains filtered or unexported fields
}
Schedule is one cron resource compiled: when it fires, in which zone, and what its first check does.
func Cron ¶
Cron compiles one cron resource. Every door a pipeline comes in by calls it, so a bad expression is refused where the pipeline is set rather than at the first poll.
func (*Schedule) Check ¶
Check is the versions a poll at now reports, given the last version recorded: one, when a slot has passed since it, or none.
A slot that passed is answered with now rather than with the slot itself, as upstream does: the version says when the job was actually released, and `epoch` in the artifact means that. It is whole seconds because RFC3339 is, and a slot is never finer.
Source Files