Documentation
¶
Index ¶
- Constants
- func EntryExpiry(createdAt time.Time, ttl time.Duration, ttlNever bool) *time.Time
- func EquivalentPriorHash(newHash string, newOutput map[string]string, priors []PriorEntry) string
- func StartPruner(ctx context.Context, store *Store, interval time.Duration)
- type Config
- type Entry
- type HashInput
- type HashInputBlob
- type PriorEntry
- type Store
- func (s *Store) Get(hash string) (*Entry, bool, error)
- func (s *Store) Invalidate(jobID uuid.UUID, taskName string) error
- func (s *Store) InvalidateJob(jobID uuid.UUID) error
- func (s *Store) ListByJob(jobID uuid.UUID) ([]Entry, error)
- func (s *Store) PriorEntriesByTask(jobID uuid.UUID, taskName, excludeHash string) ([]PriorEntry, error)
- func (s *Store) Prune() (int, error)
- func (s *Store) Put(entry *Entry) error
Constants ¶
const ( // ChainTransitive is the default and hashes predecessor identity hashes, // exactly as every hash before this feature did. ChainTransitive = "transitive" // ChainValues excludes predecessor identity hashes from the key while still // hashing predecessor OUTPUTS: "my key is what I consume, not my // predecessors' internal churn". ChainValues = "values" )
Cache chain modes. They mirror jobdef.CacheChainTransitive / jobdef.CacheChainValues and are duplicated here rather than imported: pkg/jobdef's own tests import this package, so a non-test import of pkg/jobdef from internal/cache would be an import cycle in the jobdef test build. TestChainConstantsMatchJobdef pins the two spellings in lockstep.
const HashInputBlobVersion = 1
HashInputBlobVersion is the schema version of the persisted decomposed HashInput blob (see CanonicalJSON). Bump it whenever the on-disk shape changes so a reader (e.g. `caesium why`) can tell two blobs were produced by different serializers and avoid diffing incompatible layouts. It is independent of CacheVersion, which keys the cache itself.
Variables ¶
This section is empty.
Functions ¶
func EntryExpiry ¶
EntryExpiry is the single decision every cache-entry writer makes about when an entry stops being valid. A nil result means "never expires", which Get() already honours (it only rejects a non-nil ExpiresAt in the past).
`cache.ttl: never` (ttlNever) wins over any TTL, including the inherited CAESIUM_CACHE_TTL default: a step keyed purely on a content fingerprint — an infrastructure apply, a build artifact — should not be re-executed because a wall clock moved. A zero/negative ttl likewise means no expiry.
It lives here rather than being reimplemented at each writer because the local lane, its fan-out publisher and the distributed worker all publish entries, and a lane that forgot the ttlNever check would expire exactly the entries the feature exists to keep.
func EquivalentPriorHash ¶
func EquivalentPriorHash(newHash string, newOutput map[string]string, priors []PriorEntry) string
EquivalentPriorHash implements the value-verified short-circuit (design Component 5 / exec-plan item D2).
When a step re-executes because its OWN identity hash changed (a cache MISS — e.g. its image or command changed), its NEW identity hash would normally fold into every downstream task's PredecessorHashes set and force those downstream tasks to re-run too — even if the step produced byte-identical output. That cascade is the cost a content scheduler pays for a no-op code change.
EquivalentPriorHash stops that cascade *only when content equality is PROVEN*. It searches the step's prior successful executions (same job + task name) for one whose persisted Output is byte-identical to the output this run just produced. If found, it returns that prior execution's identity hash — the proven-equivalent identity — which the caller substitutes for the new hash when presenting this step to its downstream consumers. A downstream task whose only changed input was this step's identity then sees an UNCHANGED predecessor hash, cache-hits, and stays green. The skip is proven by digest equality, not inferred.
CRITICAL — cache-correctness invariant. A cache miss must always be safe; a FALSE short-circuit (presenting a prior identity for a step whose output actually changed) would serve a stale downstream result. So this function defaults to re-run on ANY uncertainty and only substitutes when ALL hold:
- newHash is non-empty (the step has a real identity this run).
- newOutput is non-nil — a step that emitted no structured output offers no content to prove equality against; we cannot prove the value is unchanged, so we never short-circuit it.
- A prior candidate exists whose Hash is non-empty, differs from newHash (an identical hash is already a cache hit — nothing to short-circuit), and whose Output is byte-identical to newOutput by canonical comparison.
On any failure of these it returns newHash unchanged: the conservative, always-safe result is to let the changed identity propagate and re-run downstream. The most-recent matching prior wins (candidates are compared by CreatedAt) so the substituted identity is the freshest proven-equal one.
Types ¶
type Config ¶
type Config struct {
Enabled bool
TTL time.Duration
PruneInterval time.Duration
MaxEntries int
// PinDigests is the global default for image-digest pinning. A job or step
// may still override it via cache.pinDigests.
PinDigests bool
// DigestTTL bounds how long a resolved tag->digest mapping is reused.
DigestTTL time.Duration
}
Config holds cache configuration from environment.
func ConfigFromEnv ¶
func ConfigFromEnv() Config
ConfigFromEnv reads cache configuration from environment variables.
type Entry ¶
type Entry struct {
Hash string
JobID uuid.UUID
TaskName string
Result string
Output map[string]string
BranchSelections []string
RunID uuid.UUID
TaskRunID uuid.UUID
// ResolvedImageDigest is the content digest folded into Hash when the
// originating task ran with digest pinning on. Empty when pinning was off.
ResolvedImageDigest string
// HashInputBlob is the canonical, secret-redacted decomposition of the
// HashInput that produced Hash (see cache.HashInput.CanonicalJSON). Stored
// on the cache entry so a cache hit can be explained field-by-field, not
// only attested by the opaque digest. nil when not computed.
HashInputBlob []byte
// Partitions is the normalized partition list a fan-out producer emitted.
// It is what lets a CACHED producer still expand its group: the fan-out
// expansion needs the list, and a cache hit skips the container that would
// have printed it.
//
// nil vs a non-nil empty slice is a meaningful distinction here, and it is
// established at the SOURCE, not invented at this layer: pkgtask.Markers
// (pkg/task/partition.go's partitionAccumulator.finish) returns a non-nil
// []Partition{} when a producer's log explicitly parsed a
// `##caesium::partitions [...]` line — even `[]`, the documented way to
// declare an empty work list — and nil when no such line was ever seen.
// Both writer sites (internal/job/job.go, internal/worker/runtime_executor.go)
// assign that value straight onto this field, and Put/Get preserve it end to
// end (see entryToModel/modelToEntry). So: nil means no partition list was
// ever RECORDED for this entry — every task that is not a fan-out producer,
// and every entry written before this field existed — while a non-nil
// (possibly zero-length) slice means the producer's execution genuinely
// determined the list, even if it came out empty. A cache-hit call site
// facing nil cannot tell "not a producer" from "a pre-fan-out entry for a
// producer whose consumer now expects a group", so it must check separately
// (run.Store.HasFanOutSuccessor) before trusting an empty read as onEmpty.
Partitions []pkgtask.Partition `json:"partitions,omitempty"`
CreatedAt time.Time
ExpiresAt *time.Time
}
Entry represents a cached task result.
type HashInput ¶
type HashInput struct {
JobAlias string
TaskName string
Image string
// ResolvedImageDigest is the content digest (sha256:...) the Image tag
// resolved to when digest pinning is enabled. It is empty when pinning is
// off, in which case the hash is byte-identical to the pre-pinning era and
// only the mutable tag contributes. When set, the digest is folded into the
// key in addition to the tag, so a tag that moves to a new digest yields a
// different hash — a cache miss, never a stale hit.
ResolvedImageDigest string
Command []string
// Env is the step-declared environment after ${CAESIUM_PARAM_*}
// interpolation and predecessor-output injection, and before secret://
// resolution. The substituted values (not the tokens) are what Compute
// folds in, so two runs that differ only in a referenced param miss.
Env map[string]string
WorkDir string
Mounts []container.Mount
ResolvedVolumeMounts []container.VolumeMount
Kubernetes *container.KubernetesSpec
PredecessorHashes []string
PredecessorOutputs map[string]map[string]string
RunParams map[string]string
// Partition is the instance key. Hashed only when non-empty so unfanned
// tasks and pre-fan-out hashes stay byte-identical (no CacheVersion bump).
Partition string
// PartitionFingerprint is the optional per-unit content address.
// Hashed only when non-empty. dependsOn is NOT hashed.
PartitionFingerprint string
// PartitionAttributes are free-form scalar attributes, hashed with sorted
// keys only when the map is non-empty.
PartitionAttributes map[string]string
// Chain selects whether PredecessorHashes enter the key: ChainTransitive
// (the default, and what an empty string means) or ChainValues. It carries
// the resolved jobdef cache config's Chain and is threaded through every
// HashInput construction site so the local, worker and replay lanes cannot
// disagree about one task's identity.
Chain string
CacheVersion int
}
HashInput contains all fields that contribute to a task's identity hash. Control-plane flags such as replaySafe are deliberately absent: they gate orchestration decisions but are not execution inputs and must not bust cache.
func (HashInput) CanonicalJSON ¶
CanonicalJSON serializes the decomposed HashInput to a canonical, secret-redacted JSON blob suitable for persistence and later field-by-field diffing (`caesium why`). It is deterministic: encoding/json emits object keys in sorted order and every slice is stored in the same canonical order Compute() hashes it (predecessor hashes sorted; mounts and volume mounts sorted by their Compute() sort key), so the blob faithfully represents the hashed inputs and a diff never reports a spurious reorder. The returned bytes are bounded by maxHashInputBlobBytes; if the full decomposition would exceed that, a compact oversized marker is returned instead so dqlite write pressure stays bounded.
precomputed is the digest Compute() already produced on the hash write-path; it is embedded inline (so a reader can confirm the blob matches the persisted Hash) and reused rather than recomputed, avoiding a second full SHA-256 pass.
Env values are never stored verbatim: secret:// references are kept as-is (the URI itself is hashed but carries no credential material) and all other values are reduced to a digest, so a credential injected as a literal env value never lands in the blob.
type HashInputBlob ¶
type HashInputBlob struct {
// BlobVersion is HashInputBlobVersion at serialization time.
BlobVersion int `json:"blobVersion"`
// Hash is the Compute() digest this blob decomposes. Storing it inline lets
// a reader confirm the blob matches the persisted TaskRun.Hash before
// trusting the decomposition.
Hash string `json:"hash"`
JobAlias string `json:"jobAlias,omitempty"`
TaskName string `json:"taskName,omitempty"`
Image string `json:"image,omitempty"`
ResolvedImageDigest string `json:"resolvedImageDigest,omitempty"`
Command []string `json:"command,omitempty"`
Env map[string]envBlobValue `json:"env,omitempty"`
WorkDir string `json:"workDir,omitempty"`
Mounts []container.Mount `json:"mounts,omitempty"`
ResolvedVolumeMounts []container.VolumeMount `json:"resolvedVolumeMounts,omitempty"`
Kubernetes *container.KubernetesSpec `json:"kubernetes,omitempty"`
PredecessorHashes []string `json:"predecessorHashes,omitempty"`
PredecessorOutputs map[string]map[string]string `json:"predecessorOutputs,omitempty"`
RunParams map[string]string `json:"runParams,omitempty"`
// Chain records the cache chain mode ONLY when it is ChainValues; the
// default transitive mode writes nothing, so every blob produced before this
// field existed stays byte-identical and HashInputBlobVersion stays 1. It is
// what lets `caesium why` say "predecessor hashes excluded (chain: values)"
// rather than reporting a predecessor-hash change that never discriminated
// the two keys. PredecessorHashes are still recorded verbatim in values mode:
// they are real provenance and useful to a reader, they simply did not enter
// the digest.
Chain string `json:"chain,omitempty"`
// Partition / PartitionFingerprint / PartitionAttributes mirror, field by
// field, the single framed partition_identity record Compute() folds in (see
// HashInput.partitionIdentity). They are kept as three separate JSON fields
// here rather than one nested object because `caesium why` diffs the blob
// field-by-field and a nested object would report a whole-block change for a
// single changed attribute. JSON object fields cannot alias one another, so
// the blob was never subject to the delimiter-forging the hash's old
// line-oriented form allowed — but the SET of fields must stay in lockstep
// with partitionIdentity, or the blob would stop explaining the digest.
// dependsOn appears in neither: it is scheduling, not an execution input.
Partition string `json:"partition,omitempty"`
PartitionFingerprint string `json:"partitionFingerprint,omitempty"`
PartitionAttributes map[string]string `json:"partitionAttributes,omitempty"`
CacheVersion int `json:"cacheVersion"`
// Oversized is set (with Digest/EnvCount/PredecessorOutputCount populated
// and the verbatim fields cleared) when the full decomposition exceeded
// maxHashInputBlobBytes. A reader can still report "inputs changed" via the
// digest but cannot diff field-by-field.
Oversized *oversizedBlob `json:"oversized,omitempty"`
}
HashInputBlob is the canonical, secret-redacted, field-by-field representation of a HashInput that is persisted alongside the opaque digest. It exists so the system can answer *which* input changed between two runs (the basis of `caesium why`), not merely that "the hashes differ". Every field that contributes to Compute() is represented here; env values are redacted (see envBlobValue) but all other fields — including predecessor outputs, which are typed data-contract values, not secrets — are stored verbatim so a reader can show the before/after.
type PriorEntry ¶
type PriorEntry struct {
// Hash is the identity hash the prior successful execution committed to.
Hash string
// Output is the structured output that execution produced. For a
// large-object reference (pkg/task.OutputRef) the value embeds the content
// digest, so byte-equality of this map proves payload-content equality.
Output map[string]string
// CreatedAt orders candidates so the most recent proven-equal prior wins.
CreatedAt int64
}
PriorEntry is a candidate prior successful execution of a task, used to prove a value-verified short-circuit. It carries only the two fields the proof needs: the identity Hash that execution committed to, and the Output it produced. Both come from a persisted cache Entry for the same (job, task), so a match is proof — not a heuristic — that the prior identity produced the same bytes the current re-execution did.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store provides cache operations backed by GORM.
func (*Store) Get ¶
Get retrieves a cache entry by hash. Returns nil, false, nil if not found or expired.
func (*Store) Invalidate ¶
Invalidate removes cache entries for a specific task.
func (*Store) InvalidateJob ¶
InvalidateJob removes all cache entries for a job.
func (*Store) PriorEntriesByTask ¶
func (s *Store) PriorEntriesByTask(jobID uuid.UUID, taskName, excludeHash string) ([]PriorEntry, error)
PriorEntriesByTask returns up to maxPriorEntriesForShortCircuit of the most-recent non-expired cache entries for a (job, task) as PriorEntry candidates for the value-verified short-circuit (EquivalentPriorHash). It deliberately excludes the entry whose Hash equals excludeHash — that is the current re-execution's own (new) identity, which is never its own prior. Only Hash + Output + CreatedAt are needed by the proof, so this is a narrow projection; ordering by created_at DESC with a LIMIT keeps it cheap even for a task with a long run history. A query failure returns the error; the caller treats any failure as "no provable prior" and re-runs (a miss is always safe).