cache

package
v0.10.0-rc8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 30, 2026 License: MIT Imports: 52 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrHostnameRequired is returned if the given hostName to New is not given.
	ErrHostnameRequired = errors.New("hostName is required")

	// ErrHostnameMustNotContainScheme is returned if the given hostName to New contained a scheme.
	ErrHostnameMustNotContainScheme = errors.New("hostName must not contain scheme")

	// ErrHostnameNotValid is returned if the given hostName to New is not valid.
	ErrHostnameNotValid = errors.New("hostName is not valid")

	// ErrHostnameMustNotContainPath is returned if the given hostName to New contained a path.
	ErrHostnameMustNotContainPath = errors.New("hostName must not contain a path")

	// ErrMigrationInProgress is returned if a migration is already in progress.
	ErrMigrationInProgress = errors.New("migration is already in progress")

	// ErrCDCDisabled is returned when CDC is required but not enabled.
	ErrCDCDisabled = errors.New("CDC must be enabled and chunk store configured for migration")

	// ErrNarAlreadyChunked is returned when the nar is already chunked.
	ErrNarAlreadyChunked = errors.New("nar is already chunked")

	// ErrNarAlreadyWholeFile is returned by MigrateChunksToNar when the nar is
	// already stored as a whole file (nothing chunked to migrate back).
	ErrNarAlreadyWholeFile = errors.New("nar is already a whole file")

	// ErrNoNarHashToVerify is returned by MigrateChunksToNar when a chunked nar's
	// linked narinfo has no NarHash, so the reconstructed bytes cannot be
	// content-verified before de-chunking. Such nars are skipped, not migrated.
	ErrNoNarHashToVerify = errors.New("no narinfo NarHash to verify reconstructed nar against")

	// ErrNarHashMismatch is returned by MigrateChunksToNar when the bytes
	// reconstructed from chunks do not match the recorded NarHash or size.
	ErrNarHashMismatch = errors.New("reconstructed nar does not match recorded hash or size")
)
View Source 
var ErrBadRequest = errors.New("bad request")

ErrBadRequest is returned when the client sends an invalid build trace body.

Functions

func IsUploadOnly added in v0.9.0 

func IsUploadOnly(ctx context.Context) bool

IsUploadOnly checks if the context specifies skipping upstream checks.

func MigrateNarInfo added in v0.8.0 

func MigrateNarInfo(
	ctx context.Context,
	locker lock.Locker,
	dbClient *database.Client,
	narInfoStore storage.NarInfoStore,
	hash string,
	ni *narinfo.NarInfo,
) error

MigrateNarInfo migrates a single narinfo from storage to the database. It uses distributed locking to coordinate with other instances (if a distributed locker is provided). This function is used both by Cache.MigrateNarInfoToDatabase and the CLI migrate-narinfo command.

Parameters:

  • ctx: Context for the operation
  • locker: Distributed locker for coordination (can be in-memory for single-instance)
  • db: Database querier (legacy; kept for the pre-lock double-check until §11 fully retires the Querier surface)
  • dbClient: Ent-backed client used by storeNarInfoInDatabase for the actual UPSERT pipeline
  • narInfoStore: Optional storage backend to delete from after migration (nil to skip deletion)
  • hash: The narinfo hash to migrate
  • ni: The parsed narinfo to migrate

Returns an error if migration fails. Returns nil if the narinfo is already migrated or if another instance is currently migrating it.

func WithUploadOnly added in v0.9.0 

func WithUploadOnly(ctx context.Context) context.Context

WithUploadOnly returns a context that instructs the cache to skip upstream checks.

Types

type Cache 

type Cache struct {
	// contains filtered or unexported fields
}

Cache represents the main cache service.

func New 

func New(
	ctx context.Context,
	hostName string,
	dbClient *database.Client,

	configStore storage.ConfigStore,
	narInfoStore storage.NarInfoStore,
	narStore storage.NarStore,
	secretKeyPath string,
	downloadLocker lock.Locker,
	cacheLocker lock.RWLocker,
	downloadLockTTL time.Duration,
	downloadPollTimeout time.Duration,
	cacheLockTTL time.Duration,
) (*Cache, error)

New returns a new Cache.

func (*Cache) AddCDCDeletedCleanupCronJob 

func (c *Cache) AddCDCDeletedCleanupCronJob(ctx context.Context, schedule cron.Schedule)

AddCDCDeletedCleanupCronJob adds a periodic job to delete old compressed NAR files after chunking is complete and the delay has passed.

func (*Cache) AddCDCLazyRecoveryCronJob 

func (c *Cache) AddCDCLazyRecoveryCronJob(
	ctx context.Context,
	schedule cron.Schedule,
	batchSize int,
)

AddCDCLazyRecoveryCronJob adds a periodic job to recover stuck NAR files that failed to chunk due to restart or other issues.

func (*Cache) AddLRUCronJob added in v0.0.11 

func (c *Cache) AddLRUCronJob(ctx context.Context, schedule cron.Schedule)

AddLRUCronJob adds a job for LRU.

func (*Cache) AddUpstreamCaches added in v0.0.11 

func (c *Cache) AddUpstreamCaches(ctx context.Context, ucs ...*upstream.Cache)

AddUpstreamCaches adds one or more upstream caches with lazy loading support.

func (*Cache) BackgroundMigrateNarToChunks added in v0.9.0 

func (c *Cache) BackgroundMigrateNarToChunks(ctx context.Context, narURL nar.URL)

BackgroundMigrateNarToChunks migrates a traditional NAR blob to content-defined chunks in the background.

func (*Cache) CheckAndFixNarInfo added in v0.9.3 

func (c *Cache) CheckAndFixNarInfo(ctx context.Context, hash string) error

checkAndFixNarInfo checks if a NarInfo exists for the given hash, and if so, ensures its FileSize matches the actual NAR size.

func (*Cache) Close added in v0.8.0 

func (c *Cache) Close()

Close waits for all background operations to complete.

func (*Cache) DeleteNar added in v0.0.4 

func (c *Cache) DeleteNar(ctx context.Context, narURL nar.URL) error

DeleteNar deletes the nar from the store.

func (*Cache) DeleteNarInfo added in v0.0.4 

func (c *Cache) DeleteNarInfo(ctx context.Context, hash string) error

DeleteNarInfo deletes the narInfo from the store.

func (*Cache) GetBuildTrace 

func (c *Cache) GetBuildTrace(ctx context.Context, drvName, outputName string) ([]byte, error)

GetBuildTrace retrieves a stored build trace entry and returns its JSON representation. The response is reconstructed from structured DB columns and all stored signatures so that ncps's own signature is always present.

func (*Cache) GetCDCBackgroundWorkers added in v0.9.4 

func (c *Cache) GetCDCBackgroundWorkers() int

GetCDCBackgroundWorkers returns the number of background workers for lazy chunking.

func (*Cache) GetCDCDeleteDelay 

func (c *Cache) GetCDCDeleteDelay() time.Duration

GetCDCDeleteDelay returns the delay before deleting compressed NAR files after chunking.

func (*Cache) GetCDCLazyChunkingEnabled added in v0.9.4 

func (c *Cache) GetCDCLazyChunkingEnabled() bool

GetCDCLazyChunkingEnabled returns whether lazy chunking is enabled.

func (*Cache) GetConfig added in v0.8.0 

func (c *Cache) GetConfig() *config.Config

GetConfig returns the configuration instance. It's useful for testing the behavior of ncps.

func (*Cache) GetHealthChecker added in v0.3.0 

func (c *Cache) GetHealthChecker() *healthcheck.HealthChecker

GetHealthChecker returns the instance of haelth checker used by the cache. It's useful for testing the behavior of ncps.

func (*Cache) GetHealthyUpstreamCount added in v0.6.0 

func (c *Cache) GetHealthyUpstreamCount() int

GetHealthyUpstreamCount returns the number of healthy upstream caches.

func (*Cache) GetHealthyUpstreamCountLocked added in v0.9.0 

func (c *Cache) GetHealthyUpstreamCountLocked() int

GetHealthyUpstreamCountLocked returns the number of healthy upstream caches. It assumes the caller holds at least a read lock on upstreamCachesMu.

func (*Cache) GetHostname 

func (c *Cache) GetHostname() string

GetHostname returns the hostname.

func (*Cache) GetNar 

func (c *Cache) GetNar(ctx context.Context, narURL nar.URL) (nar.URL, int64, io.ReadCloser, error)

GetNar returns the nar given a hash and compression from the store. If the nar is not found in the store, it's pulled from an upstream, stored in the stored and finally returned. The returned narURL reflects any mutations made during serving (e.g. TransparentZstd cleared when zstd stream not available). NOTE: It's the caller responsibility to close the body.

func (*Cache) GetNarFileSize added in v0.9.0 

func (c *Cache) GetNarFileSize(ctx context.Context, nu nar.URL) (int64, error)

GetNarFileSize returns the size of the NAR file from the database if it exists.

func (*Cache) GetNarInfo 

func (c *Cache) GetNarInfo(ctx context.Context, hash string) (*narinfo.NarInfo, error)

GetNarInfo returns the narInfo given a hash from the store. If the narInfo is not found in the store, it's pulled from an upstream, stored in the stored and finally returned.

func (*Cache) GetPinnedClosureHashes 

func (c *Cache) GetPinnedClosureHashes(ctx context.Context) (map[string]struct{}, error)

GetPinnedClosureHashes computes all hashes that are protected by pinned closures. This includes the pinned root hashes and all their transitive references.

func (*Cache) HasBuildTrace 

func (c *Cache) HasBuildTrace(ctx context.Context, drvName, outputName string) bool

HasBuildTrace returns true if a build trace entry for (drvName, outputName) exists in the database.

func (*Cache) HasNarFileRecord added in v0.9.0 

func (c *Cache) HasNarFileRecord(ctx context.Context, narURL nar.URL) (bool, error)

HasNarFileRecord checks if a NAR file record exists in the database, regardless of chunking completion status. This is used for coordination to allow progressive streaming while chunking is in progress.

func (*Cache) HasNarInChunks added in v0.9.0 

func (c *Cache) HasNarInChunks(ctx context.Context, narURL nar.URL) (bool, error)

HasNarInChunks returns true if the NAR is already in chunks and chunking is complete.

func (*Cache) HasNarInStore added in v0.9.3 

func (c *Cache) HasNarInStore(ctx context.Context, narURL nar.URL) bool

hasNarInStore checks if the NAR exists in the storage, handling the .nar.zst fallback for CompressionTypeNone.

func (*Cache) IsNarInfoPinned 

func (c *Cache) IsNarInfoPinned(ctx context.Context, hash string) (bool, error)

IsNarInfoPinned checks if a narinfo hash is pinned.

func (*Cache) ListPinnedClosures 

func (c *Cache) ListPinnedClosures(ctx context.Context) ([]*ent.PinnedClosure, error)

ListPinnedClosures returns all pinned closures.

func (*Cache) MigrateChunksToNar 

func (c *Cache) MigrateChunksToNar(ctx context.Context, narURL *nar.URL, forceReclaim bool) error

MigrateChunksToNar is the reverse of MigrateNarToChunks: it reconstructs a CDC-chunked NAR into a whole file so a deployment can exit CDC. It reconstructs the NAR from its chunks, verifies the assembled bytes against the linked narinfo's recorded NarHash (and size), writes the whole file to the NAR store via narStore.PutNar, then flips the nar_file record to the whole-file representation (total_chunks=0, chunk links removed). The write-then-flip ordering keeps an interrupted run recoverable.

Orphaned chunks (no longer referenced by any nar_file) are NOT deleted by default: an in-flight chunk-serve that started before the flip may still be reading chunk files, and deleting them mid-stream would truncate that transfer. They are left for the regular GC to reclaim. When forceReclaim is true the caller asserts traffic is drained (e.g. a maintenance-window run), and orphaned chunks are reclaimed immediately and dedup-safely (chunks still referenced by another nar_file are retained).

It returns ErrNarAlreadyWholeFile when there is nothing chunked to migrate, ErrNoNarHashToVerify when the linked narinfo lacks a NarHash (the NAR is left chunked rather than de-chunked unverified), and ErrNarHashMismatch when the reconstructed bytes do not match the recorded hash/size.

func (*Cache) MigrateNarInfoToDatabase added in v0.8.0 

func (c *Cache) MigrateNarInfoToDatabase(
	ctx context.Context,
	hash string,
	ni *narinfo.NarInfo,
	deleteFromStorage bool,
) error

MigrateNarInfoToDatabase migrates a single narinfo from storage to the database. It uses distributed locking to coordinate with other instances (if Redis is configured). This is a convenience wrapper around MigrateNarInfo for use within the Cache.

func (*Cache) MigrateNarToChunks added in v0.9.0 

func (c *Cache) MigrateNarToChunks(ctx context.Context, narURL *nar.URL) error

MigrateNarToChunks migrates a traditional NAR blob to content-defined chunks. narURL is taken by pointer because storeNarWithCDC normalizes compression to "none".

func (*Cache) PinClosure 

func (c *Cache) PinClosure(ctx context.Context, hash string) error

PinClosure pins a closure by its narinfo hash. The narinfo and all its transitive references will be protected from LRU eviction.

func (*Cache) PublicKey 

func (c *Cache) PublicKey() signature.PublicKey

PublicKey returns the public key of the server.

func (*Cache) PutBuildTrace 

func (c *Cache) PutBuildTrace(ctx context.Context, drvName, outputName string, r io.Reader) error

PutBuildTrace stores a build trace entry from an uploaded JSON body. It parses the body, validates that the URL params match the body key, appends ncps's own signature, then upserts into the database.

func (*Cache) PutNar added in v0.0.3 

func (c *Cache) PutNar(ctx context.Context, narURL nar.URL, r io.ReadCloser) error

PutNar records the NAR (given as an io.Reader) into the store.

func (*Cache) PutNarInfo added in v0.0.3 

func (c *Cache) PutNarInfo(ctx context.Context, hash string, r io.ReadCloser) error

PutNarInfo records the narInfo (given as an io.Reader) into the store and signs it.

func (*Cache) RegisterUpstreamMetrics added in v0.6.0 

func (c *Cache) RegisterUpstreamMetrics(m metric.Meter) error

RegisterUpstreamMetrics register metrics related to upstream caches.

func (*Cache) SetCDCConfiguration added in v0.9.0 

func (c *Cache) SetCDCConfiguration(enabled bool, minSize, avgSize, maxSize uint32) error

SetCDCConfiguration enables and configures CDC.

func (*Cache) SetCDCDeleteDelay 

func (c *Cache) SetCDCDeleteDelay(delay time.Duration)

SetCDCDeleteDelay sets the delay before deleting compressed NAR files after chunking.

func (*Cache) SetCDCLazyChunking 

func (c *Cache) SetCDCLazyChunking(enabled bool, workers int)

SetCDCLazyChunking configures lazy chunking behavior.

func (*Cache) SetCacheSignNarinfo added in v0.2.0 

func (c *Cache) SetCacheSignNarinfo(shouldSignNarinfo bool)

SetCacheSignNarinfo configure ncps to sign or not sign narinfos.

func (*Cache) SetChunkStore added in v0.9.0 

func (c *Cache) SetChunkStore(cs chunk.Store)

SetChunkStore sets the chunk store.

func (*Cache) SetChunkWaitTimeout 

func (c *Cache) SetChunkWaitTimeout(d time.Duration)

SetChunkWaitTimeout overrides the per-chunk wait bound used by progressive CDC streaming. A non-positive value resets it to defaultChunkWaitTimeout. Operators on high-latency storage can raise it; those behind a short gateway timeout can lower it to fail fast and let the client retry.

func (*Cache) SetMaxSize added in v0.0.11 

func (c *Cache) SetMaxSize(maxSize uint64)

SetMaxSize sets the maxsize of the cache. This will be used by the LRU cronjob to automatically clean-up the store.

func (*Cache) SetRecordAgeIgnoreTouch added in v0.0.6 

func (c *Cache) SetRecordAgeIgnoreTouch(d time.Duration)

SetRecordAgeIgnoreTouch changes the duration at which a record is considered up to date and a touch is not invoked.

func (*Cache) SetTempDir added in v0.3.0 

func (c *Cache) SetTempDir(d string) error

SetTempDir sets the temporary directory.

func (*Cache) SetupCron added in v0.0.11 

func (c *Cache) SetupCron(ctx context.Context, timezone *time.Location)

SetupCron creates a cron instance in the cache.

func (*Cache) StartCron added in v0.0.11 

func (c *Cache) StartCron(ctx context.Context)

StartCron starts the cron scheduler in its own go-routine, or no-op if already started.

func (*Cache) UnpinClosure 

func (c *Cache) UnpinClosure(ctx context.Context, hash string) error

UnpinClosure unpins a closure by its narinfo hash.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.