store

package
v0.124.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 6, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

GitStore implements Store on top of a Git repository.

Its mirror clone is bare: fetch-only, with no working tree and no HEAD to manage (git.CloneOrUpdateBareMirror). Publish exports that mirror's tree for the requested revision via git.ExportTree (handling regular files, symlinks and submodules), then decrypts every SOPS-encrypted file the export produced (decrypt.go) before the artifact is published.

OCIStore implements Store on top of an OCI artifact reference (a fixed repository and tag, e.g. "ghcr.io/org/repo:latest"). See the package doc on git_store.go for how OCIStore's role fits into the Phase 2 plan.

Package store maps mutable source references to immutable revisions and materialized artifact directories. GitStore and OCIStore keep source-specific fetching, verification, and export details behind the Store interface.

Index

Constants

View Source
const (
	ArtifactMissing  = "missing"
	ArtifactReplaced = "replaced"
)
View Source
const ArtifactsSubdir = "artifacts"

ArtifactsSubdir is the fixed directory name, relative to a store's base directory, that holds one subdirectory per published revision.

View Source
const ComposeGitCacheSubdir = "compose-git-cache"

ComposeGitCacheSubdir holds Git include stores keyed by remote and reference.

View Source
const LiveSubdir = "live"

LiveSubdir is the fixed directory name, relative to a store's base directory, that holds the mutable copies of deployed files that are updated in place instead of per revision (e.g. bind mounts that are excluded from recreation with recreate.ignore).

View Source
const MirrorSubdir = "mirror"

MirrorSubdir is the fixed directory name, relative to a store's base directory, that holds GitStore's underlying bare mirror clone. Exported so the startup migration pass (internal/migration) can recognize and lay out the same directory a fresh GitStore would use, without importing GitStore itself.

View Source
const SubmodulesSubdir = "submodules"

SubmodulesSubdir is the fixed directory name for cached Git submodule mirrors.

Variables

View Source
var ErrInvalidRevision = errors.New("source store: invalid revision")

ErrInvalidRevision is returned for a Revision that cannot name an artifact directory, e.g. one containing a path separator or "..". Revisions can originate from webhook payloads and must never address a path outside the store's artifacts directory.

View Source
var ErrRevisionNotFound = errors.New("source store: revision not found")

ErrRevisionNotFound is returned by Publish when the requested Revision is not reachable in the store's local state and cannot be materialized without an unbounded fetch. Callers should Resolve the owning reference first, which fetches exactly what is needed to reach it.

Functions

func ArtifactDirName

func ArtifactDirName(revision Revision) string

ArtifactDirName exposes the revision-to-directory-name encoding applied by artifactPath, for callers that need to locate or lay out an artifact directory without going through a Store.

func ArtifactStaleReason added in v0.124.0

func ArtifactStaleReason(dir, deployedAt string) (string, error)

ArtifactStaleReason reports whether tasks deployed at deployedAt lost their artifact directory.

func ListRepositoryDirs added in v0.124.0

func ListRepositoryDirs(dataDir string) ([]string, error)

ListRepositoryDirs returns the absolute path of every store base directory beneath dataDir, or none if dataDir does not exist.

A store base directory is not an immediate child of dataDir: git.GetRepoName (and the OCI equivalent) produce a "<host>/<owner>/<repo>" path, so base directories are found by walking - the same way internal/migration locates legacy repository roots - and recognizing a directory that holds an "artifacts" or "mirror" subdirectory. Descent stops at a base directory, so artifact contents are never walked, and unreadable subtrees are skipped, as are the tombstones of evicted stores (see sourcecache.TombstoneDirName).

func PurgeTombstones added in v0.124.0

func PurgeTombstones(dataDir string) (int, error)

PurgeTombstones removes every tombstone in the tombstone namespace of dataDir and returns how many it removed. Everything in the namespace is garbage, so a tombstone that cannot be removed is retried by the next purge.

func SourceEvictionBlocker added in v0.124.0

func SourceEvictionBlocker(baseDir string) (string, error)

SourceEvictionBlocker returns why the caches of the store at baseDir must not be evicted regardless of its use, or "" if nothing in its layout prevents it.

Stores nested in one of the caches would be evicted with them, so a cache that ever had a store nested in it, or looks like it holds one, blocks eviction for good (see containsNestedStore).

func SourceLastUsed added in v0.124.0

func SourceLastUsed(baseDir string) (time.Time, error)

SourceLastUsed returns when the store at baseDir was last used: the later of the last use recorded by its GC gate (see sourcecache.LastSourceUse) and the newest entry of its artifacts directory, e.g. its latest publication.

func TombstoneSource added in v0.124.0

func TombstoneSource(dataDir, baseDir string, now time.Time) (string, error)

TombstoneSource moves the caches of the store at baseDir (mirror, submodules and artifacts) into a new tombstone in the tombstone namespace of dataDir (see sourcecache.TombstoneDirName), from where PurgeTombstones removes them. Renaming is atomic, so the store never holds partially removed caches, and a tombstone whose removal is interrupted or fails is removed by a later purge independently of the store.

Lock files next to and directly in baseDir, its live data and stores nested in it are kept. baseDir itself is only removed if nothing else is left in it.

The caller must hold the store's eviction lock (see sourcecache.TryAcquireSourceEvictionLock). TombstoneSource returns the tombstone's path, or "" if the store had no caches. If moving a cache fails, the caches moved before it stay in the tombstone and the others in the store, for a later eviction.

Types

type Artifact

type Artifact struct {
	// Revision is the immutable revision this artifact was published for.
	Revision Revision
	// Path is the absolute directory holding the revision's deployable
	// contents.
	Path string
}

Artifact is a materialized, read-only directory for one Revision. Once Publish returns an Artifact, its Path is never written to again by the store that produced it - readers may use it without holding any lock.

func ListArtifacts

func ListArtifacts(baseDir string) ([]Artifact, error)

ListArtifacts returns every artifact already published under baseDir, without requiring a fully constructed Store. GitStore.List and OCIStore.List are both thin wrappers over this same function; it is exported directly for callers (e.g. garbage collection, see internal/gc) that need to inspect a base directory without holding the per-source-type options (remote URL, credentials, ...) a real Store requires.

type GCOptions

type GCOptions struct {
	// RetentionRecords is the number of most-recently-modified unreferenced
	// artifacts that are always kept, regardless of RetentionTTL.
	// Mirrors Flux's --artifact-retention-records.
	RetentionRecords int

	// RetentionTTL is how long an unreferenced artifact beyond
	// RetentionRecords is kept before Sweep is allowed to remove it.
	// Mirrors Flux's --artifact-retention-ttl.
	RetentionTTL time.Duration
}

GCOptions configures artifact garbage collection for one store's base directory (see Sweep). It mirrors Flux's artifact-retention model: RetentionRecords always wins over RetentionTTL, so a repository that is simply quiet for a long time never loses its most recent artifacts.

type GCResult

type GCResult struct {
	// Kept lists every artifact Sweep decided not to remove: referenced
	// (live), within RetentionRecords, or younger than RetentionTTL.
	Kept []Artifact

	// Removed lists every artifact Sweep deleted.
	Removed []Artifact
}

GCResult reports what Sweep did for one base directory.

func Sweep

func Sweep(baseDir string, live set.Set[Revision], opts GCOptions, now time.Time) (GCResult, error)

Sweep removes unreferenced, expired artifacts under baseDir.

live is the hard safety check: any revision present in it - regardless of how it got there - is always kept, no matter its age or position in the retention ranking. Callers are expected to build live from every revision currently referenced by a deployed stack (across every configured Docker context) plus every revision currently being prepared for a deployment that has not finished yet (see the source package's in-flight tracker); Sweep itself has no way to tell the two apart and does not need to.

Artifacts not in live are ranked by their directory's modification time, newest first. The newest RetentionRecords of them are always kept; the rest are removed once older than RetentionTTL, measured against now (an explicit clock so tests are deterministic).

Candidates for removal are never anything outside "<baseDir>/artifacts/<revision>" (via listArtifacts) - the mirror clone, its lock file, the submodule cache, and any in-progress temporary publish directory are never listed as candidates in the first place, so they can never be removed here. Once a candidate is actually removed, its "<path>.lock" sibling - left behind by DeployStack's per-artifact cross-process lock - is removed alongside it, as are its publish record and publish lock file.

A single artifact that fails to stat or remove is recorded as kept and does not stop the sweep of the others; all such errors are joined and returned alongside the (still valid) result.

type GitStore

type GitStore struct {
	// contains filtered or unexported fields
}

GitStore is a Git-backed Store. See the package doc for its current bridging scope.

func NewGitStore

func NewGitStore(opts GitStoreOptions) (*GitStore, error)

NewGitStore returns a GitStore configured by opts.

func (*GitStore) List

func (s *GitStore) List() ([]Artifact, error)

List returns every artifact this store has published.

func (*GitStore) Lookup

func (s *GitStore) Lookup(revision Revision) (Artifact, bool, error)

Lookup returns the already-published artifact for revision, if any.

func (*GitStore) MirrorDir

func (s *GitStore) MirrorDir() string

MirrorDir returns the absolute path of the store's bare mirror clone. Read-only callers that need a *git.Repository handle for history queries (e.g. resolving a different reference, or GetLatestCommit/ GetChangedFilesBetweenCommits-style lookups) can open it directly with git.OpenRepository. Such reads should still hold a shared git.AcquireSharedMirrorLock(mirrorDir) for their duration: Resolve/ Publish only exclude each other via the matching exclusive lock, so an unguarded read can otherwise observe the mirror mid-fetch from a concurrent stack sharing this repository.

func (*GitStore) Publish

func (s *GitStore) Publish(ctx context.Context, revision Revision) (Artifact, error)

Publish exports the mirror's tree at revision into a read-only artifact directory. revision must already be reachable in the mirror - normally guaranteed by a prior Resolve call for the reference that named it. If it is not reachable, Publish returns ErrRevisionNotFound rather than silently doing an unbounded fetch to find it.

func (*GitStore) Resolve

func (s *GitStore) Resolve(ctx context.Context, ref string) (Revision, error)

Resolve fetches ref into the store's mirror and returns the commit SHA it currently points to.

type GitStoreOptions

type GitStoreOptions struct {
	// Log receives debug/info output. Required.
	Log *slog.Logger
	// CloneURL is the repository to fetch from. Required.
	CloneURL string
	// BaseDir is the store's private directory: it holds the mirror clone
	// at "<BaseDir>/mirror" and published artifacts under
	// "<BaseDir>/artifacts/<revision>". Required.
	BaseDir string

	Private                 bool
	SSHPrivateKey           string
	SSHPrivateKeyPassphrase string
	AccessToken             string
	SkipTLSVerify           bool
	ProxyOptions            transport.ProxyOptions
	// CloneSubmodules enables submodule materialization in Publish, via
	// git.ExportTree's own fetch-and-recurse logic. It has no effect on
	// Resolve/the mirror clone itself: the mirror never needs its
	// submodules checked out, since Publish reads tree objects, not the
	// mirror's worktree.
	CloneSubmodules bool
	// SkipAbsoluteSymlinks allows Git includes to omit unrelated absolute
	// links during export. Regular Git sources keep the strict default.
	SkipAbsoluteSymlinks bool
	// Depth is the shallow-clone depth passed to the underlying sync.
	// 0 clones full history.
	Depth int
}

GitStoreOptions configures a GitStore. It mirrors the parameters that git.CloneOrUpdateRepository already takes, so a GitStore can be constructed from the same values callers use today.

type Mirror added in v0.124.0

type Mirror struct {
	// Repository is the mirrored repository in "<host>/<owner>/<repo>" form.
	// A repository used as a submodule by several stores has a mirror in each.
	// It is empty if Err is set and the repository is unknown.
	Repository string
	// Path is the mirror's directory, or the directory that could not be
	// listed if Err is set.
	Path string
	// Owner is the base directory of the store whose GC gate protects the
	// mirror: the store itself, or for a Compose Git include store the store
	// that published the artifacts it is nested in.
	Owner string
	// Err reports why the mirror could not be inspected, e.g. an unreadable
	// configuration.
	Err error
}

Mirror is a bare Git mirror kept by a GitStore.

func ListMirrors added in v0.124.0

func ListMirrors(dataDir string) ([]Mirror, error)

ListMirrors returns the bare Git mirrors of every store beneath dataDir, sorted by repository and path: each store's mirror of its own repository and the mirrors of its Compose Git includes and submodules.

A mirror or directory that cannot be inspected is listed with Err set, so that one broken store neither hides its mirrors nor blocks the others.

type OCIStore

type OCIStore struct {
	// contains filtered or unexported fields
}

OCIStore is an OCI-backed Store.

func NewOCIStore

func NewOCIStore(opts OCIStoreOptions) (*OCIStore, error)

NewOCIStore returns an OCIStore configured by opts.

func (*OCIStore) List

func (s *OCIStore) List() ([]Artifact, error)

List returns every artifact this store has published.

func (*OCIStore) Lookup

func (s *OCIStore) Lookup(revision Revision) (Artifact, bool, error)

Lookup returns the already-published artifact for revision, if any.

func (*OCIStore) Publish

func (s *OCIStore) Publish(ctx context.Context, revision Revision) (Artifact, error)

Publish pulls and extracts the artifact pinned to revision into a read-only artifact directory. revision is passed to the pull as the expected digest, so the artifact actually materialized is always the one Resolve verified - regardless of what the store's tag currently points to.

func (*OCIStore) Resolve

func (s *OCIStore) Resolve(ctx context.Context, ref string) (Revision, error)

Resolve resolves ref - a digest, or empty/"latest" to mean "whatever the store's configured tag currently points to" - against the registry, and verifies it against the store's cosign trust policy before returning it. A digest that fails verification is never returned as a Revision.

type OCIStoreOptions

type OCIStoreOptions struct {
	// Log receives debug/info output. Defaults to slog.Default().
	Log *slog.Logger
	// ArtifactRef is the OCI reference to resolve and pull from,
	// e.g. "ghcr.io/org/repo:latest". Required.
	ArtifactRef string
	// BaseDir is the store's private directory: published artifacts live
	// under "<BaseDir>/artifacts/<digest>". Required.
	BaseDir string
	// CustomTarget selects a non-default ".doco-cd.<target>.y(a)ml" config
	// inside the artifact, matching the deployment's CustomTarget setting.
	CustomTarget string

	TrustPolicy         config.OciTrustPolicy
	TrustPolicyOverride config.OciTrustPolicyOverride
	VerifyMaxWorkers    uint
}

OCIStoreOptions configures an OCIStore.

type Revision

type Revision string

Revision is an immutable identifier for one version of a source: a Git commit SHA, or an OCI manifest digest. Unlike a reference (a branch, tag, or "latest"), resolving the same Revision twice always yields the same content.

func ArtifactRoot

func ArtifactRoot(baseDir, path string) (root string, revision Revision, ok bool)

ArtifactRoot returns the artifact directory ("<baseDir>/artifacts/<revision>") that path is nested under, and the revision it was published at, if any. It performs no filesystem access - it is a pure path computation for callers that only have a descendant path (e.g. a working directory recorded in a container label) and need to recover the artifact root/revision it was published under.

func (Revision) String

func (r Revision) String() string

String implements fmt.Stringer.

type Store

type Store interface {
	// Resolve maps ref (a branch, tag, "latest", or already a Revision) to
	// the Revision it currently points to. It may contact the remote (a Git
	// fetch, a registry HEAD/manifest request) but never modifies a
	// previously published Artifact.
	Resolve(ctx context.Context, ref string) (Revision, error)

	// Publish materializes revision into a read-only Artifact directory and
	// returns it. If an Artifact for revision has already been published,
	// it is returned as-is without redoing the work. Publish is safe to
	// call concurrently for the same or different revisions.
	Publish(ctx context.Context, revision Revision) (Artifact, error)

	// Lookup returns the Artifact for revision if it has already been
	// published, without publishing it. The second return value is false
	// if no artifact exists yet.
	Lookup(revision Revision) (Artifact, bool, error)

	// List returns every Artifact currently published by this store, e.g.
	// for garbage collection or migration.
	List() ([]Artifact, error)
}

Store maps a source's mutable references to immutable revisions, and immutable revisions to materialized artifact directories.

Implementations must make Publish safe for concurrent callers publishing the same Revision (idempotent, at-most-once materialization) and different Revisions (no shared mutable state between them) at the same time. Resolve and Lookup take no lock; they either contact the remote or inspect already-immutable local state.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL