cache

package
v0.10.0-rc13 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 11, 2026 License: MIT Imports: 53 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrHostnameRequired is returned if the given hostName to New is not given.
	ErrHostnameRequired = errors.New("hostName is required")

	// ErrHostnameMustNotContainScheme is returned if the given hostName to New contained a scheme.
	ErrHostnameMustNotContainScheme = errors.New("hostName must not contain scheme")

	// ErrHostnameNotValid is returned if the given hostName to New is not valid.
	ErrHostnameNotValid = errors.New("hostName is not valid")

	// ErrHostnameMustNotContainPath is returned if the given hostName to New contained a path.
	ErrHostnameMustNotContainPath = errors.New("hostName must not contain a path")

	// ErrMigrationInProgress is returned if a migration is already in progress.
	ErrMigrationInProgress = errors.New("migration is already in progress")

	// ErrUntrustedNarInfo is returned by PutNarInfo when requireTrustedSignature
	// is enabled and the submitted narinfo carries no signature that validates
	// against the set of trusted upstream public keys.
	ErrUntrustedNarInfo = errors.New("narinfo has no trusted signature")

	// ErrNarInfoPurged is returned if the narinfo was purged.
	ErrNarInfoPurged = errors.New("the narinfo was purged")

	// ErrCDCDisabled is returned when CDC is required but not enabled.
	ErrCDCDisabled = errors.New("CDC must be enabled and chunk store configured for migration")

	// ErrNarAlreadyChunked is returned when the nar is already chunked.
	ErrNarAlreadyChunked = errors.New("nar is already chunked")

	// ErrNarAlreadyWholeFile is returned by MigrateChunksToNar when the nar is
	// already stored as a whole file (nothing chunked to migrate back).
	ErrNarAlreadyWholeFile = errors.New("nar is already a whole file")

	// ErrNoNarHashToVerify is returned by MigrateChunksToNar when a chunked nar's
	// linked narinfo has no NarHash, so the reconstructed bytes cannot be
	// content-verified before de-chunking. Such nars are skipped, not migrated.
	ErrNoNarHashToVerify = errors.New("no narinfo NarHash to verify reconstructed nar against")

	// ErrNarHashMismatch is returned by MigrateChunksToNar when the bytes
	// reconstructed from chunks do not match the recorded NarHash or size.
	ErrNarHashMismatch = errors.New("reconstructed nar does not match recorded hash or size")

	// ErrMissingChunk is returned by MigrateChunksToNar when one or more chunks
	// referenced by the nar_file are absent from the chunk store or the DB. The
	// NAR cannot be reconstructed and should be purged so it can be re-fetched.
	ErrMissingChunk = errors.New("one or more chunks missing from store")
)
View Source 
var ErrBadRequest = errors.New("bad request")

ErrBadRequest is returned when the client sends an invalid build trace body.

Functions

func CDCChunkingLockTTL 

func CDCChunkingLockTTL() time.Duration

CDCChunkingLockTTL exposes the chunking lock TTL so out-of-package maintenance tooling (fsck's residue reclaimer) can recognise an in-flight chunking operation and leave it untouched rather than mistaking it for residue.

func IsUploadOnly added in v0.9.0 

func IsUploadOnly(ctx context.Context) bool

IsUploadOnly checks if the context specifies skipping upstream checks.

func LinkedNarinfoNarHash 

func LinkedNarinfoNarHash(
	ctx context.Context,
	dbClient *database.Client,
	narFileID int,
	narURL nar.URL,
) (*nixhash.HashWithEncoding, error)

LinkedNarinfoNarHash returns the parsed NarHash of the narinfo linked to the given nar_file, or (nil, nil) when no link exists or the narinfo has no recorded NarHash. It is the single source of truth for de-chunk recoverability: both the CDC drain and fsck's residue reclaimer use it so they agree on which chunked NARs are verifiable (recoverable) versus un-de-chunkable.

func MigrateNarInfo added in v0.8.0 

func MigrateNarInfo(
	ctx context.Context,
	locker lock.Locker,
	dbClient *database.Client,
	narInfoStore storage.NarInfoStore,
	hash string,
	ni *narinfo.NarInfo,
) error

MigrateNarInfo migrates a single narinfo from storage to the database. It uses distributed locking to coordinate with other instances (if a distributed locker is provided). This function is used both by Cache.MigrateNarInfoToDatabase and the CLI migrate-narinfo command.

Parameters:

  • ctx: Context for the operation
  • locker: Distributed locker for coordination (can be in-memory for single-instance)
  • db: Database querier (legacy; kept for the pre-lock double-check until §11 fully retires the Querier surface)
  • dbClient: Ent-backed client used by storeNarInfoInDatabase for the actual UPSERT pipeline
  • narInfoStore: Optional storage backend to delete from after migration (nil to skip deletion)
  • hash: The narinfo hash to migrate
  • ni: The parsed narinfo to migrate

Returns an error if migration fails. Returns nil if the narinfo is already migrated or if another instance is currently migrating it.

func PrimeMetrics 

func PrimeMetrics(ctx context.Context)

PrimeMetrics records a zero-valued measurement on every counter instrument in this package so the corresponding time series are exported from startup, rather than only appearing after the first real event. Without this, an idle instance scraped at /metrics is missing documented counters such as ncps_nar_served_total and ncps_narinfo_served_total (GitHub issue #1337).

It must be called after the global OTel meter provider has been installed; when no provider is configured the measurements are dropped, making this a harmless no-op. Adding zero never inflates the counts: the first real event still advances the counter from 0 to 1.

func WithUploadOnly added in v0.9.0 

func WithUploadOnly(ctx context.Context) context.Context

WithUploadOnly returns a context that instructs the cache to skip upstream checks.

Types

type Cache 

type Cache struct {
	// contains filtered or unexported fields
}

Cache represents the main cache service.

func New 

func New(
	ctx context.Context,
	hostName string,
	dbClient *database.Client,

	configStore storage.ConfigStore,
	narInfoStore storage.NarInfoStore,
	narStore storage.NarStore,
	secretKeyPath string,
	downloadLocker lock.Locker,
	cacheLocker lock.RWLocker,
	downloadLockTTL time.Duration,
	downloadPollTimeout time.Duration,
	cacheLockTTL time.Duration,
) (*Cache, error)

New returns a new Cache.

func (*Cache) AddCDCDeletedCleanupCronJob 

func (c *Cache) AddCDCDeletedCleanupCronJob(ctx context.Context, schedule cron.Schedule)

AddCDCDeletedCleanupCronJob adds a periodic job to delete old compressed NAR files after chunking is complete and the delay has passed.

func (*Cache) AddCDCLazyRecoveryCronJob 

func (c *Cache) AddCDCLazyRecoveryCronJob(
	ctx context.Context,
	schedule cron.Schedule,
	batchSize int,
)

AddCDCLazyRecoveryCronJob adds a periodic job to recover CDC rows that failed to chunk due to restart or other issues.

func (*Cache) AddInflightStagingGCCronJob 

func (c *Cache) AddInflightStagingGCCronJob(ctx context.Context, schedule cron.Schedule)

AddInflightStagingGCCronJob registers the periodic staging GC sweep. It binds only the logger from ctx, not ctx itself: each sweep derives a fresh shutdown-bound context, so a request/startup-scoped registration ctx being cancelled can never silently disable later sweeps.

func (*Cache) AddLRUCronJob added in v0.0.11 

func (c *Cache) AddLRUCronJob(ctx context.Context, schedule cron.Schedule)

AddLRUCronJob adds a job for LRU.

func (*Cache) AddUpstreamCaches added in v0.0.11 

func (c *Cache) AddUpstreamCaches(ctx context.Context, ucs ...*upstream.Cache)

AddUpstreamCaches adds one or more upstream caches with lazy loading support.

func (*Cache) BackgroundMigrateNarToChunks added in v0.9.0 

func (c *Cache) BackgroundMigrateNarToChunks(ctx context.Context, narURL nar.URL)

BackgroundMigrateNarToChunks migrates a traditional NAR blob to content-defined chunks in the background.

func (*Cache) CheckAndFixNarInfo added in v0.9.3 

func (c *Cache) CheckAndFixNarInfo(ctx context.Context, hash string) error

checkAndFixNarInfo checks if a NarInfo exists for the given hash, and if so, ensures its FileSize matches the actual NAR size.

func (*Cache) Close added in v0.8.0 

func (c *Cache) Close()

Close waits for all background operations to complete.

func (*Cache) DeleteNar added in v0.0.4 

func (c *Cache) DeleteNar(ctx context.Context, narURL nar.URL) error

DeleteNar deletes the nar from the store.

func (*Cache) DeleteNarInfo added in v0.0.4 

func (c *Cache) DeleteNarInfo(ctx context.Context, hash string) error

DeleteNarInfo deletes the narInfo from the store.

func (*Cache) GetBuildTrace 

func (c *Cache) GetBuildTrace(ctx context.Context, drvName, outputName string) ([]byte, error)

GetBuildTrace retrieves a stored build trace entry and returns its JSON representation. The response is reconstructed from structured DB columns and all stored signatures so that ncps's own signature is always present.

func (*Cache) GetCDCBackgroundWorkers added in v0.9.4 

func (c *Cache) GetCDCBackgroundWorkers() int

GetCDCBackgroundWorkers returns the number of background workers for lazy chunking.

func (*Cache) GetCDCDeleteDelay 

func (c *Cache) GetCDCDeleteDelay() time.Duration

GetCDCDeleteDelay returns the delay before deleting compressed NAR files after chunking.

func (*Cache) GetCDCLazyChunkingEnabled added in v0.9.4 

func (c *Cache) GetCDCLazyChunkingEnabled() bool

GetCDCLazyChunkingEnabled returns whether lazy chunking is enabled.

func (*Cache) GetConfig added in v0.8.0 

func (c *Cache) GetConfig() *config.Config

GetConfig returns the configuration instance. It's useful for testing the behavior of ncps.

func (*Cache) GetHealthChecker added in v0.3.0 

func (c *Cache) GetHealthChecker() *healthcheck.HealthChecker

GetHealthChecker returns the instance of haelth checker used by the cache. It's useful for testing the behavior of ncps.

func (*Cache) GetHealthyUpstreamCount added in v0.6.0 

func (c *Cache) GetHealthyUpstreamCount() int

GetHealthyUpstreamCount returns the number of healthy upstream caches.

func (*Cache) GetHealthyUpstreamCountLocked added in v0.9.0 

func (c *Cache) GetHealthyUpstreamCountLocked() int

GetHealthyUpstreamCountLocked returns the number of healthy upstream caches. It assumes the caller holds at least a read lock on upstreamCachesMu.

func (*Cache) GetHostname 

func (c *Cache) GetHostname() string

GetHostname returns the hostname.

func (*Cache) GetNar 

func (c *Cache) GetNar(ctx context.Context, narURL nar.URL) (nar.URL, int64, io.ReadCloser, error)

GetNar returns the nar given a hash and compression from the store. If the nar is not found in the store, it's pulled from an upstream, stored in the stored and finally returned. The returned narURL reflects any mutations made during serving (e.g. TransparentZstd cleared when zstd stream not available). NOTE: It's the caller responsibility to close the body.

func (*Cache) GetNarFileSize added in v0.9.0 

func (c *Cache) GetNarFileSize(ctx context.Context, nu nar.URL) (int64, error)

GetNarFileSize returns the size of the NAR file from the database if it exists.

func (*Cache) GetNarInfo 

func (c *Cache) GetNarInfo(ctx context.Context, hash string) (*narinfo.NarInfo, error)

GetNarInfo returns the narInfo given a hash from the store. If the narInfo is not found in the store, it's pulled from an upstream, stored in the stored and finally returned.

func (*Cache) GetPinnedClosureHashes 

func (c *Cache) GetPinnedClosureHashes(ctx context.Context) (map[string]struct{}, error)

GetPinnedClosureHashes computes all hashes that are protected by pinned closures. This includes the pinned root hashes and all their transitive references.

func (*Cache) HasBuildTrace 

func (c *Cache) HasBuildTrace(ctx context.Context, drvName, outputName string) bool

HasBuildTrace returns true if a build trace entry for (drvName, outputName) exists in the database.

func (*Cache) HasNarFileRecord added in v0.9.0 

func (c *Cache) HasNarFileRecord(ctx context.Context, narURL nar.URL) (bool, error)

HasNarFileRecord checks if a NAR file record exists in the database, regardless of chunking completion status. This is used for coordination to allow progressive streaming while chunking is in progress.

func (*Cache) HasNarInChunks added in v0.9.0 

func (c *Cache) HasNarInChunks(ctx context.Context, narURL nar.URL) (bool, error)

HasNarInChunks returns true if the NAR is already in chunks and chunking is complete.

func (*Cache) HasNarInStore added in v0.9.3 

func (c *Cache) HasNarInStore(ctx context.Context, narURL nar.URL) bool

hasNarInStore checks if the NAR exists in the storage, handling the .nar.zst fallback for CompressionTypeNone.

func (*Cache) InflightStagingEnabled 

func (c *Cache) InflightStagingEnabled() bool

InflightStagingEnabled reports whether in-flight NAR staging is active. It is true only when the feature flag is set AND the locker is distributed.

func (*Cache) InflightStagingPartSize 

func (c *Cache) InflightStagingPartSize() int64

InflightStagingPartSize returns the size in bytes of each staging part-object.

func (*Cache) InflightStagingRetention 

func (c *Cache) InflightStagingRetention() time.Duration

InflightStagingRetention returns the grace period before staging part-objects are reclaimed after the final representation is committed.

func (*Cache) IsNarInfoPinned 

func (c *Cache) IsNarInfoPinned(ctx context.Context, hash string) (bool, error)

IsNarInfoPinned checks if a narinfo hash is pinned.

func (*Cache) IsNarServable 

func (c *Cache) IsNarServable(ctx context.Context, narURL nar.URL) (bool, error)

IsNarServable reports whether the NAR's actual bytes are present and therefore servable, mirroring how GetNar determines existence — a whole-file in the store or chunks present. It is tri-state, like statNarInStore: a confirmed absence is (false, nil) while an ambiguous/transient storage error is (false, err) so callers do NOT mistake an undeterminable result for "absent". The mere existence of a nar_file row (e.g. GetNarFileSize/HasNarFileRecord) does NOT make a NAR servable; callers gating an existence answer (e.g. the NAR HEAD handler) MUST use this rather than the DB record alone.

func (*Cache) ListPinnedClosures 

func (c *Cache) ListPinnedClosures(ctx context.Context) ([]*ent.PinnedClosure, error)

ListPinnedClosures returns all pinned closures.

func (*Cache) MigrateChunksToNar 

func (c *Cache) MigrateChunksToNar(ctx context.Context, narURL *nar.URL, forceReclaim bool) error

MigrateChunksToNar is the reverse of MigrateNarToChunks: it reconstructs a CDC-chunked NAR into a whole file so a deployment can exit CDC. It reconstructs the NAR from its chunks, verifies the assembled bytes against the linked narinfo's recorded NarHash (and size), writes the whole file to the NAR store via narStore.PutNar, then flips the nar_file record to the whole-file representation (total_chunks=0, chunk links removed). The write-then-flip ordering keeps an interrupted run recoverable.

Orphaned chunks (no longer referenced by any nar_file) are NOT deleted by default: an in-flight chunk-serve that started before the flip may still be reading chunk files, and deleting them mid-stream would truncate that transfer. They are left for the regular GC to reclaim. When forceReclaim is true the caller asserts traffic is drained (e.g. a maintenance-window run), and orphaned chunks are reclaimed immediately and dedup-safely (chunks still referenced by another nar_file are retained).

It returns ErrNarAlreadyWholeFile when there is nothing chunked to migrate, ErrNoNarHashToVerify when the linked narinfo lacks a NarHash (the NAR is left chunked rather than de-chunked unverified), and ErrNarHashMismatch when the reconstructed bytes do not match the recorded hash/size.

func (*Cache) MigrateNarInfoToDatabase added in v0.8.0 

func (c *Cache) MigrateNarInfoToDatabase(
	ctx context.Context,
	hash string,
	ni *narinfo.NarInfo,
	deleteFromStorage bool,
) error

MigrateNarInfoToDatabase migrates a single narinfo from storage to the database. It uses distributed locking to coordinate with other instances (if Redis is configured). This is a convenience wrapper around MigrateNarInfo for use within the Cache.

func (*Cache) MigrateNarToChunks added in v0.9.0 

func (c *Cache) MigrateNarToChunks(ctx context.Context, narURL *nar.URL) error

MigrateNarToChunks migrates a traditional NAR blob to content-defined chunks. narURL is taken by pointer because storeNarWithCDC normalizes compression to "none".

func (*Cache) NormalizeChunkedNarInfoURL 

func (c *Cache) NormalizeChunkedNarInfoURL(ctx context.Context, narURL *nar.URL) error

NormalizeChunkedNarInfoURL repairs a recoverable inconsistent chunked NAR: a chunked nar_file whose referencing narinfo carries a valid NarHash but advertises a non-Compression:none URL (e.g. nar/<H>.nar.xz). It relinks (if needed) and rewrites every referencing narinfo's URL to the Compression:none form so the persisted metadata matches the chunk-backed storage. It touches no chunks and leaves the nar_file chunked — it is the safe, CDC-state-independent half of fsck's residue repair, reusing the same narinfo-normalization the CDC drain performs during its de-chunk flip.

func (*Cache) PinClosure 

func (c *Cache) PinClosure(ctx context.Context, hash string) error

PinClosure pins a closure by its narinfo hash. The narinfo and all its transitive references will be protected from LRU eviction.

func (*Cache) PublicKey 

func (c *Cache) PublicKey() signature.PublicKey

PublicKey returns the public key of the server.

func (*Cache) PurgeChunkedNar 

func (c *Cache) PurgeChunkedNar(ctx context.Context, narURL *nar.URL) error

PurgeChunkedNar deletes a permanently broken chunked NAR entry so its hash can be re-fetched from upstream on the next GetNar request. It acquires the same migration lock as MigrateChunksToNar to serialize with concurrent migration attempts on the same hash.

Unlike the normal de-chunk path, chunk objects are always reclaimed immediately: the NAR was never serveable (its reconstructed bytes do not match the recorded hash), so there is no valid in-flight chunk-serve that could be truncated. Chunk deletion is dedup-safe: a chunk still referenced by another nar_file is not deleted.

The linked narinfo record is left intact so the next GetNarInfo can succeed and trigger a clean upstream re-fetch.

func (*Cache) PutBuildTrace 

func (c *Cache) PutBuildTrace(ctx context.Context, drvName, outputName string, r io.Reader) error

PutBuildTrace stores a build trace entry from an uploaded JSON body. It parses the body, validates that the URL params match the body key, appends ncps's own signature, then upserts into the database.

func (*Cache) PutNar added in v0.0.3 

func (c *Cache) PutNar(ctx context.Context, narURL nar.URL, r io.ReadCloser) error

PutNar records the NAR (given as an io.Reader) into the store.

func (*Cache) PutNarInfo added in v0.0.3 

func (c *Cache) PutNarInfo(ctx context.Context, hash string, r io.ReadCloser) error

PutNarInfo records the narInfo (given as an io.Reader) into the store and signs it.

func (*Cache) RegisterUpstreamMetrics added in v0.6.0 

func (c *Cache) RegisterUpstreamMetrics(m metric.Meter) error

RegisterUpstreamMetrics register metrics related to upstream caches.

func (*Cache) SetCDCConfiguration added in v0.9.0 

func (c *Cache) SetCDCConfiguration(enabled bool, minSize, avgSize, maxSize uint32) error

SetCDCConfiguration enables and configures CDC.

func (*Cache) SetCDCDeleteDelay 

func (c *Cache) SetCDCDeleteDelay(delay time.Duration)

SetCDCDeleteDelay sets the delay before deleting compressed NAR files after chunking.

func (*Cache) SetCDCLazyChunking 

func (c *Cache) SetCDCLazyChunking(enabled bool, workers int)

SetCDCLazyChunking configures lazy chunking behavior.

func (*Cache) SetCacheRequireTrustedSignature 

func (c *Cache) SetCacheRequireTrustedSignature(requireTrustedSignature bool)

SetCacheRequireTrustedSignature configures whether PutNarInfo rejects narinfos lacking a valid trusted upstream signature.

func (*Cache) SetCacheSignNarinfo added in v0.2.0 

func (c *Cache) SetCacheSignNarinfo(shouldSignNarinfo bool)

SetCacheSignNarinfo configure ncps to sign or not sign narinfos.

func (*Cache) SetChunkStore added in v0.9.0 

func (c *Cache) SetChunkStore(cs chunk.Store)

SetChunkStore sets the chunk store.

func (*Cache) SetChunkWaitTimeout 

func (c *Cache) SetChunkWaitTimeout(d time.Duration)

SetChunkWaitTimeout overrides the per-chunk wait bound used by progressive CDC streaming. A non-positive value resets it to defaultChunkWaitTimeout. Operators on high-latency storage can raise it; those behind a short gateway timeout can lower it to fail fast and let the client retry.

func (*Cache) SetInflightStaging 

func (c *Cache) SetInflightStaging(enabled bool, retention time.Duration, partSize int64, distributed bool)

SetInflightStaging configures in-flight NAR staging. partSize is the size of each staging part-object in bytes. distributed reports whether the configured download locker is distributed (Redis); staging is only meaningful in that case, because a single-instance deployment can never have a cross-pod waiter.

func (*Cache) SetMaxSize added in v0.0.11 

func (c *Cache) SetMaxSize(maxSize uint64)

SetMaxSize sets the maxsize of the cache. This will be used by the LRU cronjob to automatically clean-up the store.

func (*Cache) SetRecordAgeIgnoreTouch added in v0.0.6 

func (c *Cache) SetRecordAgeIgnoreTouch(d time.Duration)

SetRecordAgeIgnoreTouch changes the duration at which a record is considered up to date and a touch is not invoked.

func (*Cache) SetTempDir added in v0.3.0 

func (c *Cache) SetTempDir(d string) error

SetTempDir sets the temporary directory.

func (*Cache) SetupCron added in v0.0.11 

func (c *Cache) SetupCron(ctx context.Context, timezone *time.Location)

SetupCron creates a cron instance in the cache.

func (*Cache) StartCron added in v0.0.11 

func (c *Cache) StartCron(ctx context.Context)

StartCron starts the cron scheduler in its own go-routine, or no-op if already started.

func (*Cache) UnpinClosure 

func (c *Cache) UnpinClosure(ctx context.Context, hash string) error

UnpinClosure unpins a closure by its narinfo hash.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.