cache

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 7, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// ChainTransitive is the default and hashes predecessor identity hashes,
	// exactly as every hash before this feature did.
	ChainTransitive = "transitive"
	// ChainValues excludes predecessor identity hashes from the key while still
	// hashing predecessor OUTPUTS: "my key is what I consume, not my
	// predecessors' internal churn".
	ChainValues = "values"
)

Cache chain modes. They mirror jobdef.CacheChainTransitive / jobdef.CacheChainValues and are duplicated here rather than imported: pkg/jobdef's own tests import this package, so a non-test import of pkg/jobdef from internal/cache would be an import cycle in the jobdef test build. TestChainConstantsMatchJobdef pins the two spellings in lockstep.

View Source
const HashInputBlobVersion = 1

HashInputBlobVersion is the schema version of the persisted decomposed HashInput blob (see CanonicalJSON). Bump it whenever the on-disk shape changes so a reader (e.g. `caesium why`) can tell two blobs were produced by different serializers and avoid diffing incompatible layouts. It is independent of CacheVersion, which keys the cache itself.

Variables

This section is empty.

Functions

func EntryExpiry

func EntryExpiry(createdAt time.Time, ttl time.Duration, ttlNever bool) *time.Time

EntryExpiry is the single decision every cache-entry writer makes about when an entry stops being valid. A nil result means "never expires", which Get() already honours (it only rejects a non-nil ExpiresAt in the past).

`cache.ttl: never` (ttlNever) wins over any TTL, including the inherited CAESIUM_CACHE_TTL default: a step keyed purely on a content fingerprint — an infrastructure apply, a build artifact — should not be re-executed because a wall clock moved. A zero/negative ttl likewise means no expiry.

It lives here rather than being reimplemented at each writer because the local lane, its fan-out publisher and the distributed worker all publish entries, and a lane that forgot the ttlNever check would expire exactly the entries the feature exists to keep.

func EquivalentPriorHash

func EquivalentPriorHash(newHash string, newOutput map[string]string, priors []PriorEntry) string

EquivalentPriorHash implements the value-verified short-circuit (design Component 5 / exec-plan item D2).

When a step re-executes because its OWN identity hash changed (a cache MISS — e.g. its image or command changed), its NEW identity hash would normally fold into every downstream task's PredecessorHashes set and force those downstream tasks to re-run too — even if the step produced byte-identical output. That cascade is the cost a content scheduler pays for a no-op code change.

EquivalentPriorHash stops that cascade *only when content equality is PROVEN*. It searches the step's prior successful executions (same job + task name) for one whose persisted Output is byte-identical to the output this run just produced. If found, it returns that prior execution's identity hash — the proven-equivalent identity — which the caller substitutes for the new hash when presenting this step to its downstream consumers. A downstream task whose only changed input was this step's identity then sees an UNCHANGED predecessor hash, cache-hits, and stays green. The skip is proven by digest equality, not inferred.

CRITICAL — cache-correctness invariant. A cache miss must always be safe; a FALSE short-circuit (presenting a prior identity for a step whose output actually changed) would serve a stale downstream result. So this function defaults to re-run on ANY uncertainty and only substitutes when ALL hold:

  • newHash is non-empty (the step has a real identity this run).
  • newOutput is non-nil — a step that emitted no structured output offers no content to prove equality against; we cannot prove the value is unchanged, so we never short-circuit it.
  • A prior candidate exists whose Hash is non-empty, differs from newHash (an identical hash is already a cache hit — nothing to short-circuit), and whose Output is byte-identical to newOutput by canonical comparison.

On any failure of these it returns newHash unchanged: the conservative, always-safe result is to let the changed identity propagate and re-run downstream. The most-recent matching prior wins (candidates are compared by CreatedAt) so the substituted identity is the freshest proven-equal one.

func StartPruner

func StartPruner(ctx context.Context, store *Store, interval time.Duration)

StartPruner starts a background goroutine that prunes expired cache entries.

Types

type Config

type Config struct {
	Enabled       bool
	TTL           time.Duration
	PruneInterval time.Duration
	MaxEntries    int
	// PinDigests is the global default for image-digest pinning. A job or step
	// may still override it via cache.pinDigests.
	PinDigests bool
	// DigestTTL bounds how long a resolved tag->digest mapping is reused.
	DigestTTL time.Duration
}

Config holds cache configuration from environment.

func ConfigFromEnv

func ConfigFromEnv() Config

ConfigFromEnv reads cache configuration from environment variables.

type Entry

type Entry struct {
	Hash             string
	JobID            uuid.UUID
	TaskName         string
	Result           string
	Output           map[string]string
	BranchSelections []string
	RunID            uuid.UUID
	TaskRunID        uuid.UUID
	// ResolvedImageDigest is the content digest folded into Hash when the
	// originating task ran with digest pinning on. Empty when pinning was off.
	ResolvedImageDigest string
	// HashInputBlob is the canonical, secret-redacted decomposition of the
	// HashInput that produced Hash (see cache.HashInput.CanonicalJSON). Stored
	// on the cache entry so a cache hit can be explained field-by-field, not
	// only attested by the opaque digest. nil when not computed.
	HashInputBlob []byte
	// Partitions is the normalized partition list a fan-out producer emitted.
	// It is what lets a CACHED producer still expand its group: the fan-out
	// expansion needs the list, and a cache hit skips the container that would
	// have printed it.
	//
	// nil vs a non-nil empty slice is a meaningful distinction here, and it is
	// established at the SOURCE, not invented at this layer: pkgtask.Markers
	// (pkg/task/partition.go's partitionAccumulator.finish) returns a non-nil
	// []Partition{} when a producer's log explicitly parsed a
	// `##caesium::partitions [...]` line — even `[]`, the documented way to
	// declare an empty work list — and nil when no such line was ever seen.
	// Both writer sites (internal/job/job.go, internal/worker/runtime_executor.go)
	// assign that value straight onto this field, and Put/Get preserve it end to
	// end (see entryToModel/modelToEntry). So: nil means no partition list was
	// ever RECORDED for this entry — every task that is not a fan-out producer,
	// and every entry written before this field existed — while a non-nil
	// (possibly zero-length) slice means the producer's execution genuinely
	// determined the list, even if it came out empty. A cache-hit call site
	// facing nil cannot tell "not a producer" from "a pre-fan-out entry for a
	// producer whose consumer now expects a group", so it must check separately
	// (run.Store.HasFanOutSuccessor) before trusting an empty read as onEmpty.
	Partitions []pkgtask.Partition `json:"partitions,omitempty"`
	CreatedAt  time.Time
	ExpiresAt  *time.Time
}

Entry represents a cached task result.

type HashInput

type HashInput struct {
	JobAlias string
	TaskName string
	Image    string
	// ResolvedImageDigest is the content digest (sha256:...) the Image tag
	// resolved to when digest pinning is enabled. It is empty when pinning is
	// off, in which case the hash is byte-identical to the pre-pinning era and
	// only the mutable tag contributes. When set, the digest is folded into the
	// key in addition to the tag, so a tag that moves to a new digest yields a
	// different hash — a cache miss, never a stale hit.
	ResolvedImageDigest string
	Command             []string
	// Env is the step-declared environment after ${CAESIUM_PARAM_*}
	// interpolation and predecessor-output injection, and before secret://
	// resolution. The substituted values (not the tokens) are what Compute
	// folds in, so two runs that differ only in a referenced param miss.
	Env                  map[string]string
	WorkDir              string
	Mounts               []container.Mount
	ResolvedVolumeMounts []container.VolumeMount
	Kubernetes           *container.KubernetesSpec
	PredecessorHashes    []string
	PredecessorOutputs   map[string]map[string]string
	RunParams            map[string]string
	// Partition is the instance key. Hashed only when non-empty so unfanned
	// tasks and pre-fan-out hashes stay byte-identical (no CacheVersion bump).
	Partition string
	// PartitionFingerprint is the optional per-unit content address.
	// Hashed only when non-empty. dependsOn is NOT hashed.
	PartitionFingerprint string
	// PartitionAttributes are free-form scalar attributes, hashed with sorted
	// keys only when the map is non-empty.
	PartitionAttributes map[string]string
	// Chain selects whether PredecessorHashes enter the key: ChainTransitive
	// (the default, and what an empty string means) or ChainValues. It carries
	// the resolved jobdef cache config's Chain and is threaded through every
	// HashInput construction site so the local, worker and replay lanes cannot
	// disagree about one task's identity.
	Chain        string
	CacheVersion int
}

HashInput contains all fields that contribute to a task's identity hash. Control-plane flags such as replaySafe are deliberately absent: they gate orchestration decisions but are not execution inputs and must not bust cache.

func (HashInput) CanonicalJSON

func (h HashInput) CanonicalJSON(precomputed string) ([]byte, error)

CanonicalJSON serializes the decomposed HashInput to a canonical, secret-redacted JSON blob suitable for persistence and later field-by-field diffing (`caesium why`). It is deterministic: encoding/json emits object keys in sorted order and every slice is stored in the same canonical order Compute() hashes it (predecessor hashes sorted; mounts and volume mounts sorted by their Compute() sort key), so the blob faithfully represents the hashed inputs and a diff never reports a spurious reorder. The returned bytes are bounded by maxHashInputBlobBytes; if the full decomposition would exceed that, a compact oversized marker is returned instead so dqlite write pressure stays bounded.

precomputed is the digest Compute() already produced on the hash write-path; it is embedded inline (so a reader can confirm the blob matches the persisted Hash) and reused rather than recomputed, avoiding a second full SHA-256 pass.

Env values are never stored verbatim: secret:// references are kept as-is (the URI itself is hashed but carries no credential material) and all other values are reduced to a digest, so a credential injected as a literal env value never lands in the blob.

func (HashInput) Compute

func (h HashInput) Compute() string

Compute returns the SHA-256 hex digest of the canonicalized input.

type HashInputBlob

type HashInputBlob struct {
	// BlobVersion is HashInputBlobVersion at serialization time.
	BlobVersion int `json:"blobVersion"`
	// Hash is the Compute() digest this blob decomposes. Storing it inline lets
	// a reader confirm the blob matches the persisted TaskRun.Hash before
	// trusting the decomposition.
	Hash string `json:"hash"`

	JobAlias             string                       `json:"jobAlias,omitempty"`
	TaskName             string                       `json:"taskName,omitempty"`
	Image                string                       `json:"image,omitempty"`
	ResolvedImageDigest  string                       `json:"resolvedImageDigest,omitempty"`
	Command              []string                     `json:"command,omitempty"`
	Env                  map[string]envBlobValue      `json:"env,omitempty"`
	WorkDir              string                       `json:"workDir,omitempty"`
	Mounts               []container.Mount            `json:"mounts,omitempty"`
	ResolvedVolumeMounts []container.VolumeMount      `json:"resolvedVolumeMounts,omitempty"`
	Kubernetes           *container.KubernetesSpec    `json:"kubernetes,omitempty"`
	PredecessorHashes    []string                     `json:"predecessorHashes,omitempty"`
	PredecessorOutputs   map[string]map[string]string `json:"predecessorOutputs,omitempty"`
	RunParams            map[string]string            `json:"runParams,omitempty"`

	// Chain records the cache chain mode ONLY when it is ChainValues; the
	// default transitive mode writes nothing, so every blob produced before this
	// field existed stays byte-identical and HashInputBlobVersion stays 1. It is
	// what lets `caesium why` say "predecessor hashes excluded (chain: values)"
	// rather than reporting a predecessor-hash change that never discriminated
	// the two keys. PredecessorHashes are still recorded verbatim in values mode:
	// they are real provenance and useful to a reader, they simply did not enter
	// the digest.
	Chain string `json:"chain,omitempty"`

	// Partition / PartitionFingerprint / PartitionAttributes mirror, field by
	// field, the single framed partition_identity record Compute() folds in (see
	// HashInput.partitionIdentity). They are kept as three separate JSON fields
	// here rather than one nested object because `caesium why` diffs the blob
	// field-by-field and a nested object would report a whole-block change for a
	// single changed attribute. JSON object fields cannot alias one another, so
	// the blob was never subject to the delimiter-forging the hash's old
	// line-oriented form allowed — but the SET of fields must stay in lockstep
	// with partitionIdentity, or the blob would stop explaining the digest.
	// dependsOn appears in neither: it is scheduling, not an execution input.
	Partition            string            `json:"partition,omitempty"`
	PartitionFingerprint string            `json:"partitionFingerprint,omitempty"`
	PartitionAttributes  map[string]string `json:"partitionAttributes,omitempty"`

	CacheVersion int `json:"cacheVersion"`

	// Oversized is set (with Digest/EnvCount/PredecessorOutputCount populated
	// and the verbatim fields cleared) when the full decomposition exceeded
	// maxHashInputBlobBytes. A reader can still report "inputs changed" via the
	// digest but cannot diff field-by-field.
	Oversized *oversizedBlob `json:"oversized,omitempty"`
}

HashInputBlob is the canonical, secret-redacted, field-by-field representation of a HashInput that is persisted alongside the opaque digest. It exists so the system can answer *which* input changed between two runs (the basis of `caesium why`), not merely that "the hashes differ". Every field that contributes to Compute() is represented here; env values are redacted (see envBlobValue) but all other fields — including predecessor outputs, which are typed data-contract values, not secrets — are stored verbatim so a reader can show the before/after.

type PriorEntry

type PriorEntry struct {
	// Hash is the identity hash the prior successful execution committed to.
	Hash string
	// Output is the structured output that execution produced. For a
	// large-object reference (pkg/task.OutputRef) the value embeds the content
	// digest, so byte-equality of this map proves payload-content equality.
	Output map[string]string
	// CreatedAt orders candidates so the most recent proven-equal prior wins.
	CreatedAt int64
}

PriorEntry is a candidate prior successful execution of a task, used to prove a value-verified short-circuit. It carries only the two fields the proof needs: the identity Hash that execution committed to, and the Output it produced. Both come from a persisted cache Entry for the same (job, task), so a match is proof — not a heuristic — that the prior identity produced the same bytes the current re-execution did.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store provides cache operations backed by GORM.

func NewStore

func NewStore(db *gorm.DB) *Store

NewStore creates a new cache store.

func (*Store) Get

func (s *Store) Get(hash string) (*Entry, bool, error)

Get retrieves a cache entry by hash. Returns nil, false, nil if not found or expired.

func (*Store) Invalidate

func (s *Store) Invalidate(jobID uuid.UUID, taskName string) error

Invalidate removes cache entries for a specific task.

func (*Store) InvalidateJob

func (s *Store) InvalidateJob(jobID uuid.UUID) error

InvalidateJob removes all cache entries for a job.

func (*Store) ListByJob

func (s *Store) ListByJob(jobID uuid.UUID) ([]Entry, error)

ListByJob returns all cache entries for a job.

func (*Store) PriorEntriesByTask

func (s *Store) PriorEntriesByTask(jobID uuid.UUID, taskName, excludeHash string) ([]PriorEntry, error)

PriorEntriesByTask returns up to maxPriorEntriesForShortCircuit of the most-recent non-expired cache entries for a (job, task) as PriorEntry candidates for the value-verified short-circuit (EquivalentPriorHash). It deliberately excludes the entry whose Hash equals excludeHash — that is the current re-execution's own (new) identity, which is never its own prior. Only Hash + Output + CreatedAt are needed by the proof, so this is a narrow projection; ordering by created_at DESC with a LIMIT keeps it cheap even for a task with a long run history. A query failure returns the error; the caller treats any failure as "no provable prior" and re-runs (a miss is always safe).

func (*Store) Prune

func (s *Store) Prune() (int, error)

Prune removes expired entries. Returns count of deleted entries.

func (*Store) Put

func (s *Store) Put(entry *Entry) error

Put stores a cache entry.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL