database

package
v0.11.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 31, 2026 License: GPL-3.0 Imports: 23 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	CacheEntryArtifact = "artifact"
	CacheEntryMetadata = "metadata"
)

Kinds of CacheEntry.

View Source 
const (
	SourceUpstream = "upstream"
	SourceLocal    = "local"
)

Artifact sources. Upstream artifacts are a cache of a registry and may be evicted; local artifacts were pushed to the proxy and are only removed by a delete.

View Source 
const SchemaVersion = 1

Variables

View Source 
var ErrLocalArtifactExists = errors.New("a pushed artifact already exists")

ErrLocalArtifactExists is returned when an artifact would replace one that was pushed to the proxy. Pushed artifacts are only removed by deleting them.

Functions

func Exists 

func Exists(path string) bool

func VersionFromPURL added in v0.7.0 

func VersionFromPURL(p string) string

VersionFromPURL extracts the decoded version string from a PURL.

PURL percent-encodes characters that are not safe in a path component, so a Debian version like "7.91+dfsg1-2ubuntu0.1" is stored as "pkg:deb/nmap@7.91%2Bdfsg1-2ubuntu0.1". The raw substring after "@" is therefore not the version: it must be percent-decoded before being displayed or used to build a URL, otherwise "%2B" leaks into the UI and round-tripping the value back into a PURL double-encodes it.

e.g., "pkg:npm/lodash@4.17.21" -> "4.17.21"

Types

type Artifact 

type Artifact struct {
	ID             int64          `db:"id" json:"id"`
	VersionPURL    string         `db:"version_purl" json:"version_purl"`
	Filename       string         `db:"filename" json:"filename"`
	UpstreamURL    string         `db:"upstream_url" json:"upstream_url"`
	StoragePath    sql.NullString `db:"storage_path" json:"storage_path,omitempty"`
	ContentHash    sql.NullString `db:"content_hash" json:"content_hash,omitempty"`
	Size           sql.NullInt64  `db:"size" json:"size,omitempty"`
	ContentType    sql.NullString `db:"content_type" json:"content_type,omitempty"`
	FetchedAt      sql.NullTime   `db:"fetched_at" json:"fetched_at,omitempty"`
	HitCount       int64          `db:"hit_count" json:"hit_count"`
	LastAccessedAt sql.NullTime   `db:"last_accessed_at" json:"last_accessed_at,omitempty"`
	CreatedAt      time.Time      `db:"created_at" json:"created_at"`
	UpdatedAt      time.Time      `db:"updated_at" json:"updated_at"`
	Source         string         `db:"source" json:"source"`
	Manifest       sql.NullString `db:"manifest" json:"manifest,omitempty"`
}

Artifact represents a cached artifact in the database. This table is proxy-specific and not part of git-pkgs.

func (*Artifact) IsCached 

func (a *Artifact) IsCached() bool

IsCached returns true if the artifact has been fetched and stored locally.

type CacheEntry added in v0.10.0 

type CacheEntry struct {
	Kind           string         `db:"kind"`
	Key            string         `db:"cache_key"`
	Name           string         `db:"name"`
	StoragePath    sql.NullString `db:"storage_path"`
	Size           sql.NullInt64  `db:"size"`
	LastAccessedAt sql.NullTime   `db:"last_accessed_at"`
	FetchedAt      sql.NullTime   `db:"fetched_at"`
	UpdatedAt      sql.NullTime   `db:"updated_at"`
}

CacheEntry is a cached object size eviction can remove: an artifact fetched from upstream or a metadata document. For an artifact Key is its version PURL and Name its filename; for metadata Key is the ecosystem and Name the cache key. FetchedAt is only read for artifacts, UpdatedAt for metadata not backfilled yet.

type CacheStats 

type CacheStats struct {
	TotalPackages   int64
	TotalVersions   int64
	TotalArtifacts  int64
	TotalSize       int64
	TotalHits       int64
	EcosystemCounts map[string]int64
}

type CachedArtifact added in v0.7.0 

type CachedArtifact struct {
	Ecosystem   string
	StoragePath string
	Artifact    artifacts.Artifact
	Integrity   sql.NullString
}

CachedArtifact contains the fields needed to serve a cached artifact.

type DB 

type DB struct {
	*sqlx.DB
	// contains filtered or unexported fields
}

DB is the proxy database. The embedded pool takes every write. For SQLite it holds a single connection, as SQLite allows one writer at a time, and reads go to a separate read-only pool so they don't queue behind writes or behind each other (WAL mode lets readers run alongside the writer). For Postgres and in-memory SQLite both are the same pool.

Get, Select, Query and QueryRow (and their Context forms) use the reader. Exec, transactions and everything else use the writer, so a read inside a transaction goes through the transaction and sees its own writes.

func Create 

func Create(path string) (*DB, error)

func Open 

func Open(path string) (*DB, error)

func OpenOrCreate 

func OpenOrCreate(path string) (*DB, error)

func OpenPostgres 

func OpenPostgres(url string) (*DB, error)

func OpenPostgresOrCreate 

func OpenPostgresOrCreate(url string) (*DB, error)

func (*DB) AcquireFetchLease added in v0.11.0 

func (db *DB) AcquireFetchLease(key, owner string, ttl time.Duration) (bool, error)

AcquireFetchLease takes the lease on key for owner unless another owner holds one that has neither expired nor failed, and reports whether it did.

func (*DB) BackfillMetadataLastAccessed added in v0.10.0 

func (db *DB) BackfillMetadataLastAccessed(limit int) (int, error)

BackfillMetadataLastAccessed sets last_accessed_at from the write time on up to limit rows that have none, in one transaction, and reports how many it set. Rows served in the meantime keep the time of that hit.

func (*DB) BatchHits added in v0.9.1 

func (db *DB) BatchHits(interval time.Duration, logger *slog.Logger)

BatchHits makes RecordArtifactHit buffer hits and write them every interval. An interval of zero or less leaves every hit written immediately. Close writes whatever is still pending. Failed writes are logged to logger.

func (*DB) CachedArtifactNames added in v0.11.0 

func (db *DB) CachedArtifactNames(ecosystem, filename string) ([]string, error)

CachedArtifactNames returns the names of the packages of ecosystem that have a cached artifact called filename, for protocols whose file names identify an artifact without its package name.

func (*DB) ClaimJobRun added in v0.11.0 

func (db *DB) ClaimJobRun(name string, minGap time.Duration) (bool, error)

ClaimJobRun records that the named job starts now unless it started less than minGap ago, and reports whether the caller should run it. Instances sharing the database each keep their own timer for a scheduled job, and this lets only the first of them run it per interval.

func (*DB) ClearArtifactCache 

func (db *DB) ClearArtifactCache(versionPURL, filename, storagePath string) (bool, error)

ClearArtifactCache marks an artifact uncached if its record still points at storagePath, queues storagePath for deletion in the same transaction, and reports whether it did. A record a newer fetch has moved elsewhere is left alone, so a clear never orphans the object that fetch committed. If queueing fails, the record keeps its path.

func (*DB) ClearExpiredArtifact added in v0.10.0 

func (db *DB) ClearExpiredArtifact(versionPURL, filename, storagePath string, cutoff time.Time) (bool, error)

ClearExpiredArtifact marks an artifact uncached like ClearArtifactCache, but only while it is still expired before cutoff, so an artifact served between the scan and the clear stays cached. It queues storagePath for deletion in the same transaction: if queueing fails, the record keeps its path and a later sweep tries again, instead of the object staying in storage with nothing left pointing at it.

func (*DB) Close added in v0.9.1 

func (db *DB) Close() error

Close writes any batched hits, then closes the database.

func (*DB) CountCachedPackages 

func (db *DB) CountCachedPackages(ecosystem string) (int64, error)

func (*DB) CountCachedPackagesByEcosystem added in v0.9.1 

func (db *DB) CountCachedPackagesByEcosystem() (map[string]int64, error)

CountCachedPackagesByEcosystem counts packages with at least one stored artifact.

func (*DB) CountSearchResults 

func (db *DB) CountSearchResults(query string, ecosystem string) (int64, error)

func (*DB) CreateSchema 

func (db *DB) CreateSchema() error

func (*DB) DeleteExpiredFetchLeases added in v0.11.0 

func (db *DB) DeleteExpiredFetchLeases(grace time.Duration) (int64, error)

DeleteExpiredFetchLeases removes leases that expired more than grace ago, left by instances that stopped while fetching.

func (*DB) DeleteLocalArtifacts added in v0.10.0 

func (db *DB) DeleteLocalArtifacts(versionPURL, filename string) (int, error)

DeleteLocalArtifacts removes the pushed artifacts of a version, or only the one named filename when it is set, and queues their stored objects for deletion in the same transaction, so a failed queue keeps the records instead of orphaning the objects. Artifacts cached from an upstream are left alone. It returns how many records it removed.

func (*DB) DeleteVulnerabilitiesForPackage 

func (db *DB) DeleteVulnerabilitiesForPackage(ecosystem, name string) error

func (*DB) Dialect 

func (db *DB) Dialect() Dialect

func (*DB) DiscardArtifact added in v0.9.0 

func (db *DB) DiscardArtifact(versionPURL, filename, storagePath string) error

DiscardArtifact clears the record and queues storagePath as ClearArtifactCache does, for callers that must not delete an object another request may be reading.

func (*DB) DiscardMetadataCache added in v0.11.0 

func (db *DB) DiscardMetadataCache(ecosystem, name, storagePath, digest string) error

DiscardMetadataCache deletes a metadata cache row whose stored document failed its integrity check, if it still points at storagePath with digest, and queues the path for deletion in the same transaction. A row a newer fetch has rewritten is left alone.

func (*DB) EnsureArtifactsTable 

func (db *DB) EnsureArtifactsTable() error

EnsureArtifactsTable adds the artifacts table to an existing database (e.g., a git-pkgs database) if it doesn't already exist.

func (*DB) EnsureMetadataCacheTable 

func (db *DB) EnsureMetadataCacheTable() error

EnsureMetadataCacheTable creates the metadata_cache table if it doesn't exist.

func (*DB) EvictMetadataCache added in v0.10.0 

func (db *DB) EvictMetadataCache(ecosystem, name, storagePath string, lastAccessed sql.NullTime) (bool, error)

EvictMetadataCache deletes a metadata cache row if it still points at storagePath and has not been used since lastAccessed, queueing the path in the same transaction so a crash before the blob is deleted cannot orphan it.

func (*DB) FailFetchLease added in v0.11.0 

func (db *DB) FailFetchLease(key, owner string) error

FailFetchLease marks owner's lease on key as failed. Instances waiting on it stop waiting, and the next AcquireFetchLease takes it at once.

func (*DB) FetchLeaseHolder added in v0.11.0 

func (db *DB) FetchLeaseHolder(key string) (owner string, failed, ok bool, err error)

FetchLeaseHolder returns the owner of the lease on key, and whether its fetch failed. ok is false when no lease is held or it has expired.

func (*DB) FindStoredContent added in v0.11.0 

func (db *DB) FindStoredContent(hash string) (*StoredContent, error)

FindStoredContent returns a cached artifact whose content has the sha256 hash, under any name, or nil if none is stored. Pushed artifacts are left out.

func (*DB) FlushHits added in v0.10.0 

func (db *DB) FlushHits() error

FlushHits writes hits buffered by BatchHits now, so a caller about to read last_accessed_at sees them.

func (*DB) ForgetMetadataDocument added in v0.11.0 

func (db *DB) ForgetMetadataDocument(ecosystem, name, from, to string) error

ForgetMetadataDocument points the metadata cache row for ecosystem and name at to, provided it is still at from, and clears what described the document at from: its digest, validators and fetch time. The row stays, so the next lookup fetches the document again rather than serving or revalidating it.

func (*DB) Get added in v0.11.0 

func (db *DB) Get(dest any, query string, args ...any) error

Get runs a read on the reader pool.

func (*DB) GetArtifact 

func (db *DB) GetArtifact(versionPURL, filename string) (*Artifact, error)

func (*DB) GetArtifactByPath 

func (db *DB) GetArtifactByPath(storagePath string) (*Artifact, error)

func (*DB) GetArtifactEcosystem added in v0.10.0 

func (db *DB) GetArtifactEcosystem(versionPURL string) (string, error)

GetArtifactEcosystem returns the packages.ecosystem value of the package an artifact version belongs to, or "" when it has no package record.

func (*DB) GetArtifactsByVersionPURL 

func (db *DB) GetArtifactsByVersionPURL(versionPURL string) ([]Artifact, error)

func (*DB) GetCacheStats 

func (db *DB) GetCacheStats() (*CacheStats, error)

func (*DB) GetCachedArtifact added in v0.7.0 

func (db *DB) GetCachedArtifact(packagePURL, versionPURL, filename string) (*CachedArtifact, error)

GetCachedArtifact returns the fields needed to serve a cached artifact.

func (*DB) GetCachedArtifactCount 

func (db *DB) GetCachedArtifactCount() (int64, error)

func (*DB) GetContext added in v0.11.0 

func (db *DB) GetContext(ctx context.Context, dest any, query string, args ...any) error

GetContext runs a read on the reader pool.

func (*DB) GetDuePendingDeletes added in v0.9.0 

func (db *DB) GetDuePendingDeletes(cutoff time.Time, limit int) ([]string, error)

GetDuePendingDeletes returns up to limit paths queued before cutoff, oldest first. A path a record points at again is skipped: a metadata document is stored at a path fixed by its key, so one fetched again after eviction reuses the path queued for it.

func (*DB) GetEcosystemStats added in v0.9.1 

func (db *DB) GetEcosystemStats() ([]EcosystemStats, error)

GetEcosystemStats returns per-ecosystem cache and download totals, ordered by accumulated download volume descending. Ecosystems with rows in packages but nothing cached are included with zeroed artifact counters.

Artifacts evicted from the cache no longer contribute: eviction clears the size column, so their historical hits drop out of the accumulated total.

func (*DB) GetEnrichmentStats 

func (db *DB) GetEnrichmentStats() (*EnrichmentStats, error)

func (*DB) GetEvictableCacheSize added in v0.10.0 

func (db *DB) GetEvictableCacheSize() (int64, error)

GetEvictableCacheSize returns the bytes max_size limits: cached upstream artifacts and cached metadata documents.

func (*DB) GetLeastRecentlyUsedArtifacts 

func (db *DB) GetLeastRecentlyUsedArtifacts(limit int) ([]Artifact, error)

func (*DB) GetLeastRecentlyUsedCacheEntries added in v0.10.0 

func (db *DB) GetLeastRecentlyUsedCacheEntries(limit int) ([]CacheEntry, error)

GetLeastRecentlyUsedCacheEntries returns up to limit artifacts and metadata documents, least recently used first. An artifact counts as used when it was fetched until it is first served, so a fresh download does not go before every metadata document. Each kind is read in order on its own and the two are merged here, which keeps the queries simple for both dialects.

func (*DB) GetLocalArtifacts added in v0.10.0 

func (db *DB) GetLocalArtifacts(packagePURL string) ([]LocalArtifact, error)

GetLocalArtifacts returns the artifacts pushed to the proxy for a package, ordered by version and filename.

func (*DB) GetMetadataCache 

func (db *DB) GetMetadataCache(ecosystem, name string) (*MetadataCacheEntry, error)

func (*DB) GetMetadataCacheStats added in v0.10.0 

func (db *DB) GetMetadataCacheStats() ([]MetadataCacheStats, error)

GetMetadataCacheStats returns the number and total size of cached metadata documents per ecosystem, largest first. Ecosystem names are normalized so aliases such as "gem" and "rubygems" share one row. Rows written before sizes were recorded count as entries with no size.

func (*DB) GetMetadataDate added in v0.10.0 

func (db *DB) GetMetadataDate(ecosystem, proxyURL, name string) (*MetadataDate, error)

GetMetadataDate returns the date recorded for a package's metadata as rewritten for proxyURL, or nil when there is none.

func (*DB) GetMostPopularPackages 

func (db *DB) GetMostPopularPackages(limit int) ([]PopularPackage, error)

func (*DB) GetPackageByEcosystemName 

func (db *DB) GetPackageByEcosystemName(ecosystem, name string) (*Package, error)

func (*DB) GetPackageByPURL 

func (db *DB) GetPackageByPURL(purl string) (*Package, error)

func (*DB) GetPackagesNeedingVulnSync 

func (db *DB) GetPackagesNeedingVulnSync(limit int, minAge time.Duration) ([]Package, error)

func (*DB) GetRecentlyCachedPackages 

func (db *DB) GetRecentlyCachedPackages(limit int) ([]RecentPackage, error)

func (*DB) GetRetentionCandidates added in v0.10.0 

func (db *DB) GetRetentionCandidates(fromID, toID int64, cutoff time.Time) ([]RetentionCandidate, error)

GetRetentionCandidates returns the cached artifacts with fromID < id <= toID that expired before cutoff. Bounding the scan by id keeps each query short, which matters on SQLite, where every request shares one connection.

func (*DB) GetTotalCacheSize 

func (db *DB) GetTotalCacheSize() (int64, error)

func (*DB) GetVersionByPURL 

func (db *DB) GetVersionByPURL(purl string) (*Version, error)

func (*DB) GetVersionsByPackagePURL 

func (db *DB) GetVersionsByPackagePURL(packagePURL string) ([]Version, error)

func (*DB) GetVulnCountForPackage 

func (db *DB) GetVulnCountForPackage(ecosystem, name string) (int64, error)

func (*DB) GetVulnerabilitiesForPackage 

func (db *DB) GetVulnerabilitiesForPackage(ecosystem, name string) ([]Vulnerability, error)

func (*DB) GetVulnsSyncedAt 

func (db *DB) GetVulnsSyncedAt(ecosystem, name string) (time.Time, error)

func (*DB) HasColumn 

func (db *DB) HasColumn(table, column string) (bool, error)

HasColumn checks if a column exists in a table.

func (*DB) HasTable 

func (db *DB) HasTable(name string) (bool, error)

HasTable checks if a table exists in the database.

func (*DB) InsertMetadataDate added in v0.10.0 

func (db *DB) InsertMetadataDate(ecosystem, proxyURL, name string, date MetadataDate) (bool, error)

InsertMetadataDate records the first date of a package's metadata and reports whether it did; it leaves a date another replica recorded first.

func (*DB) ListCachedPackages 

func (db *DB) ListCachedPackages(ecosystem string, sortBy string, limit int, offset int) ([]PackageListItem, error)

func (*DB) MarkArtifactCached 

func (db *DB) MarkArtifactCached(versionPURL, filename, storagePath, contentHash string, size int64, contentType string) error

func (*DB) MaxArtifactID added in v0.10.0 

func (db *DB) MaxArtifactID() (int64, error)

MaxArtifactID returns the highest artifact id, or zero for an empty table.

func (*DB) MigrateSchema 

func (db *DB) MigrateSchema() error

MigrateSchema applies any unapplied migrations in order. For a fully migrated database this executes a single SELECT query. Instances sharing a Postgres database run it one after the other, so each sees the migrations the one before it recorded.

func (*DB) MoveMetadataCache added in v0.11.0 

func (db *DB) MoveMetadataCache(ecosystem, name, from, to, digest string) (bool, error)

MoveMetadataCache points the metadata cache row for ecosystem and name at to, provided it is still at from and records digest, and reports whether it did. The caller has copied the document from from to to.

func (*DB) OptimizeForBulkWrites 

func (db *DB) OptimizeForBulkWrites() error

func (*DB) OptimizeForReads 

func (db *DB) OptimizeForReads() error

func (*DB) Path 

func (db *DB) Path() string

func (*DB) PutUpstreamHold added in v0.11.0 

func (db *DB) PutUpstreamHold(host string, until time.Time) error

PutUpstreamHold records that requests to host are held until until. A hold already recorded for longer is kept.

func (*DB) Query added in v0.11.0 

func (db *DB) Query(query string, args ...any) (*sql.Rows, error)

Query runs a read on the reader pool.

func (*DB) QueryContext added in v0.11.0 

func (db *DB) QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error)

QueryContext runs a read on the reader pool.

func (*DB) QueryRow added in v0.11.0 

func (db *DB) QueryRow(query string, args ...any) *sql.Row

QueryRow runs a read on the reader pool.

func (*DB) QueryRowContext added in v0.11.0 

func (db *DB) QueryRowContext(ctx context.Context, query string, args ...any) *sql.Row

QueryRowContext runs a read on the reader pool.

func (*DB) QueuePendingDelete added in v0.9.0 

func (db *DB) QueuePendingDelete(path string) error

QueuePendingDelete queues a storage path no record points at any more. Queueing a path again, as reclaim does after a failed delete, restarts its grace period and moves it behind the rest.

func (*DB) Rebind 

func (db *DB) Rebind(query string) string

func (*DB) RecordArtifactHit 

func (db *DB) RecordArtifactHit(versionPURL, filename string) error

func (*DB) RecordCachedArtifact added in v0.11.0 

func (db *DB) RecordCachedArtifact(pkg *Package, v *Version, a *Artifact) error

RecordCachedArtifact records an artifact fetched from an upstream together with its package and version, in one transaction. If the artifact was pushed in the meantime, nothing is written and ErrLocalArtifactExists is returned, so the push's version integrity is kept as well.

func (*DB) RecordLocalArtifact added in v0.11.0 

func (db *DB) RecordLocalArtifact(pkg *Package, v *Version, oneFilePerVersion bool, a *Artifact) error

RecordLocalArtifact records a pushed artifact in one transaction: the package if it is missing, the version, and the artifact. An artifact cached from an upstream under the same name is replaced and its stored object queued for deletion; a pushed one is never replaced, and the whole record is refused with ErrLocalArtifactExists, so of two concurrent pushes of the same file exactly one wins.

When oneFilePerVersion is false, an existing version record is left as it is. Otherwise its integrity is set to v.Integrity, and the push is refused if the version already holds a pushed artifact under any filename, since aliases of one name (Composer's are case-insensitive) share the version record and its integrity. The version upsert locks that record first, so concurrent pushes of aliases see each other.

func (*DB) RecordMetadataHit added in v0.10.0 

func (db *DB) RecordMetadataHit(ecosystem, name string) error

RecordMetadataHit marks a cached metadata document as served now. Like RecordArtifactHit it is buffered when BatchHits is on.

func (*DB) RefreshMetadataCache added in v0.10.0 

func (db *DB) RefreshMetadataCache(ecosystem, name, digest string, fetchedAt time.Time) (bool, error)

RefreshMetadataCache sets fetched_at on the metadata cache row for ecosystem and name, provided the row still records digest, and reports whether it did. It is for an upstream 304, which leaves the stored document as it is.

func (*DB) RekeyCachedArtifact added in v0.11.0 

func (db *DB) RekeyCachedArtifact(old *Artifact, pkg *Package, version *Version) (bool, error)

RekeyCachedArtifact moves one unchanged upstream record without replacing an existing destination. Access history and stored bytes are preserved.

func (*DB) ReleaseFetchLease added in v0.11.0 

func (db *DB) ReleaseFetchLease(key, owner string) error

ReleaseFetchLease drops owner's lease on key.

func (*DB) ReleaseJobRun added in v0.11.0 

func (db *DB) ReleaseJobRun(name string) error

ReleaseJobRun drops the named job's last claim, so the next attempt on any instance runs it. It is for a run that was interrupted before it finished.

func (*DB) RemovePendingDelete added in v0.9.0 

func (db *DB) RemovePendingDelete(path string) error

RemovePendingDelete drops a deleted path from the queue.

func (*DB) RenewFetchLease added in v0.11.0 

func (db *DB) RenewFetchLease(key, owner string, ttl time.Duration) (bool, error)

RenewFetchLease extends owner's lease on key by ttl from now, and reports whether owner still held it.

func (*DB) SchemaVersion 

func (db *DB) SchemaVersion() (int, error)

func (*DB) SearchPackages 

func (db *DB) SearchPackages(query string, ecosystem string, limit int, offset int) ([]SearchResult, error)

func (*DB) Select added in v0.11.0 

func (db *DB) Select(dest any, query string, args ...any) error

Select runs a read on the reader pool.

func (*DB) SelectContext added in v0.11.0 

func (db *DB) SelectContext(ctx context.Context, dest any, query string, args ...any) error

SelectContext runs a read on the reader pool.

func (*DB) SetVersionPublishedAt added in v0.8.0 

func (db *DB) SetVersionPublishedAt(versionPURL, packagePURL string, publishedAt time.Time) error

SetVersionPublishedAt records a version's publish time, creating the versions row if the proxy has not seen the version yet. It only writes published_at, so it never disturbs enrichment data on an existing row.

func (*DB) SetVulnsSyncedAt 

func (db *DB) SetVulnsSyncedAt(ecosystem, name string) error

func (*DB) SwapMetadataDate added in v0.10.0 

func (db *DB) SwapMetadataDate(ecosystem, proxyURL, name string, old, next MetadataDate) (bool, error)

SwapMetadataDate replaces a package's date with next if it is still old, and reports whether it did; it leaves a date another replica swapped in.

func (*DB) TryLock added in v0.11.0 

func (db *DB) TryLock(ctx context.Context, name string) (release func(), ok bool, err error)

TryLock takes the named lock if no other instance holds it. When ok is true the caller must call release once done. The lock lives in a connection held for that time, and Postgres frees it if the process dies.

func (*DB) UpsertArtifact 

func (db *DB) UpsertArtifact(a *Artifact) error

func (*DB) UpsertMetadataCache 

func (db *DB) UpsertMetadataCache(entry *MetadataCacheEntry) error

UpsertMetadataCache writes entry and marks it accessed now, since a document is stored because a request asked for it.

func (*DB) UpsertPackage 

func (db *DB) UpsertPackage(pkg *Package) error

func (*DB) UpsertVersion 

func (db *DB) UpsertVersion(v *Version) error

func (*DB) UpsertVulnerability 

func (db *DB) UpsertVulnerability(v *Vulnerability) error

func (*DB) UpstreamHolds added in v0.11.0 

func (db *DB) UpstreamHolds(now time.Time) (map[string]time.Time, error)

UpstreamHolds returns the holds that have not ended by now, and deletes those that have.

func (*DB) WithLock added in v0.11.0 

func (db *DB) WithLock(ctx context.Context, name string, fn func() error) error

WithLock runs fn while holding the named lock, waiting for another instance that holds it.

type Dialect 

type Dialect string
const (
	DialectSQLite   Dialect = "sqlite"
	DialectPostgres Dialect = "postgres"
)

type EcosystemStats added in v0.9.1 

type EcosystemStats struct {
	Ecosystem       string `db:"ecosystem"`
	Packages        int64  `db:"packages"`
	Versions        int64  `db:"versions"`
	Artifacts       int64  `db:"artifacts"`
	CacheSize       int64  `db:"cache_size"`
	Downloads       int64  `db:"downloads"`
	DownloadedBytes int64  `db:"downloaded_bytes"`
}

EcosystemStats aggregates cache and download activity for one ecosystem.

DownloadedBytes is the accumulated download volume: every cache hit on an artifact served its full size, so the sum of hit_count * size is the number of bytes the proxy has handed to clients from cache for this ecosystem.

type EnrichmentStats 

type EnrichmentStats struct {
	TotalPackages        int64
	EnrichedPackages     int64
	VulnSyncedPackages   int64
	TotalVulnerabilities int64
	CriticalVulns        int64
	HighVulns            int64
	MediumVulns          int64
	LowVulns             int64
}

type LocalArtifact added in v0.10.0 

type LocalArtifact struct {
	Artifact
	Integrity sql.NullString `db:"integrity"`
}

LocalArtifact is a pushed artifact with the integrity of its version.

type MetadataCacheEntry 

type MetadataCacheEntry struct {
	ID              int64          `db:"id" json:"id"`
	Ecosystem       string         `db:"ecosystem" json:"ecosystem"`
	Name            string         `db:"name" json:"name"`
	StoragePath     string         `db:"storage_path" json:"storage_path"`
	ETag            sql.NullString `db:"etag" json:"etag,omitempty"`
	Link            sql.NullString `db:"link" json:"link,omitempty"`
	ContentType     sql.NullString `db:"content_type" json:"content_type,omitempty"`
	ContentEncoding sql.NullString `db:"content_encoding" json:"content_encoding,omitempty"`
	ContentDigest   sql.NullString `db:"content_digest" json:"content_digest,omitempty"`
	Size            sql.NullInt64  `db:"size" json:"size,omitempty"`
	LastModified    sql.NullTime   `db:"last_modified" json:"last_modified,omitempty"`
	FetchedAt       sql.NullTime   `db:"fetched_at" json:"fetched_at,omitempty"`
	LastAccessedAt  sql.NullTime   `db:"last_accessed_at" json:"last_accessed_at,omitempty"`
	CreatedAt       time.Time      `db:"created_at" json:"created_at"`
	UpdatedAt       time.Time      `db:"updated_at" json:"updated_at"`
}

MetadataCacheEntry represents a cached metadata blob for offline serving.

type MetadataCacheStats added in v0.10.0 

type MetadataCacheStats struct {
	Ecosystem string
	Entries   int64
	Size      int64
}

MetadataCacheStats is one ecosystem's share of the metadata cache.

type MetadataDate added in v0.10.0 

type MetadataDate struct {
	ETag         string
	LastModified time.Time
}

MetadataDate is the Last-Modified a package's rewritten metadata is served with, and the ETag of the rewrite it dates.

Rewritten metadata has no date of its own, so the proxy assigns one when a package starts serving a new rewrite. Keeping it in the database rather than in process memory is what lets replicas sharing the database agree on it: a date one replica gave out is never given by another to a different rewrite.

type Package 

type Package struct {
	ID            int64          `db:"id" json:"id"`
	PURL          string         `db:"purl" json:"purl"`
	Ecosystem     string         `db:"ecosystem" json:"ecosystem"`
	Name          string         `db:"name" json:"name"`
	LatestVersion sql.NullString `db:"latest_version" json:"latest_version,omitempty"`
	License       sql.NullString `db:"license" json:"license,omitempty"`
	Description   sql.NullString `db:"description" json:"description,omitempty"`
	Homepage      sql.NullString `db:"homepage" json:"homepage,omitempty"`
	RepositoryURL sql.NullString `db:"repository_url" json:"repository_url,omitempty"`
	RegistryURL   sql.NullString `db:"registry_url" json:"registry_url,omitempty"`
	SupplierName  sql.NullString `db:"supplier_name" json:"supplier_name,omitempty"`
	SupplierType  sql.NullString `db:"supplier_type" json:"supplier_type,omitempty"`
	Source        sql.NullString `db:"source" json:"source,omitempty"`
	EnrichedAt    sql.NullTime   `db:"enriched_at" json:"enriched_at,omitempty"`
	VulnsSyncedAt sql.NullTime   `db:"vulns_synced_at" json:"vulns_synced_at,omitempty"`
	CreatedAt     time.Time      `db:"created_at" json:"created_at"`
	UpdatedAt     time.Time      `db:"updated_at" json:"updated_at"`
}

Package represents a package in the database. Schema is compatible with git-pkgs.

type PackageListItem 

type PackageListItem struct {
	Ecosystem     string         `db:"ecosystem"`
	Name          string         `db:"name"`
	LatestVersion sql.NullString `db:"latest_version"`
	License       sql.NullString `db:"license"`
	Hits          int64          `db:"hits"`
	Size          int64          `db:"size"`
	CachedAt      sql.NullString `db:"cached_at"`
	VulnCount     int64          `db:"vuln_count"`
}

type PopularPackage 

type PopularPackage struct {
	Ecosystem string `db:"ecosystem"`
	Name      string `db:"name"`
	Hits      int64  `db:"hits"`
	Size      int64  `db:"size"`
}

type RecentPackage 

type RecentPackage struct {
	Ecosystem   string    `db:"ecosystem"`
	Name        string    `db:"name"`
	VersionPURL string    `db:"version_purl"`
	CachedAt    time.Time `db:"fetched_at"`
	Size        int64     `db:"size"`
	// Version is derived from VersionPURL rather than selected, so that the
	// PURL percent-encoding is decoded (e.g. "%2B" back to "+").
	Version string `db:"-"`
}

type RetentionCandidate added in v0.10.0 

type RetentionCandidate struct {
	ID             int64         `db:"id"`
	VersionPURL    string        `db:"version_purl"`
	Filename       string        `db:"filename"`
	StoragePath    string        `db:"storage_path"`
	Size           sql.NullInt64 `db:"size"`
	FetchedAt      sql.NullTime  `db:"fetched_at"`
	LastAccessedAt sql.NullTime  `db:"last_accessed_at"`
	Ecosystem      string        `db:"ecosystem"`
	PackagePURL    string        `db:"package_purl"`
}

RetentionCandidate is a cached artifact that has not been accessed since a cutoff, together with the package record it belongs to.

func (RetentionCandidate) ExpiredBefore added in v0.10.0 

func (c RetentionCandidate) ExpiredBefore(cutoff time.Time) bool

ExpiredBefore reports whether the artifact was neither fetched nor accessed at or after cutoff. Both are checked: clearing a record keeps its last access time, so a record fetched again after an eviction has a fresh fetch time next to an old access time.

type SearchResult 

type SearchResult struct {
	Ecosystem     string         `db:"ecosystem"`
	Name          string         `db:"name"`
	LatestVersion sql.NullString `db:"latest_version"`
	License       sql.NullString `db:"license"`
	Hits          int64          `db:"hits"`
	Size          int64          `db:"size"`
	CachedAt      sql.NullString `db:"cached_at"`
}

type StoredContent added in v0.11.0 

type StoredContent struct {
	StoragePath string         `db:"storage_path"`
	Size        int64          `db:"size"`
	ContentType sql.NullString `db:"content_type"`
}

StoredContent is a stored object with a given content hash.

type Version 

type Version struct {
	ID          int64          `db:"id" json:"id"`
	PURL        string         `db:"purl" json:"purl"`
	PackagePURL string         `db:"package_purl" json:"package_purl"`
	License     sql.NullString `db:"license" json:"license,omitempty"`
	PublishedAt sql.NullTime   `db:"published_at" json:"published_at,omitempty"`
	Integrity   sql.NullString `db:"integrity" json:"integrity,omitempty"`
	Yanked      bool           `db:"yanked" json:"yanked"`
	Source      sql.NullString `db:"source" json:"source,omitempty"`
	EnrichedAt  sql.NullTime   `db:"enriched_at" json:"enriched_at,omitempty"`
	CreatedAt   time.Time      `db:"created_at" json:"created_at"`
	UpdatedAt   time.Time      `db:"updated_at" json:"updated_at"`
}

Version represents a package version in the database. Schema is compatible with git-pkgs.

func (*Version) DisplayPURL added in v0.7.0 

func (v *Version) DisplayPURL() string

DisplayPURL returns the PURL with its path components percent-decoded, for showing in the UI. The stored PURL keeps the canonical encoding (which is what the API and all lookups use); this is only a readable rendering, so that a version like "7.91+dfsg1-2ubuntu0.1" is not shown as "7.91%2Bdfsg1-2ubuntu0.1" and an npm scope is shown as "@babel" rather than "%40babel". Qualifiers and subpath keep their encoding, since decoding those would be ambiguous.

func (*Version) EscapedVersion added in v0.7.0 

func (v *Version) EscapedVersion() string

EscapedVersion returns the version escaped for use as a single URL path segment.

Version returns decoded text, which is what should be shown to a user but is not safe to drop into a link: html/template preserves reserved characters and existing escapes in a URL, so "release/1" would split into two path segments, "v1?build" would start a query string, and a literal "%2B" would be read back as "+". Escaping here and decoding in splitWildcardPath round-trips the value, so the link resolves to the version that was stored.

func (*Version) Version 

func (v *Version) Version() string

Version extracts the version string from the PURL. e.g., "pkg:npm/lodash@4.17.21" -> "4.17.21"

type Vulnerability 

type Vulnerability struct {
	ID           int64           `db:"id" json:"id"`
	VulnID       string          `db:"vuln_id" json:"vuln_id"`
	Ecosystem    string          `db:"ecosystem" json:"ecosystem"`
	PackageName  string          `db:"package_name" json:"package_name"`
	Severity     sql.NullString  `db:"severity" json:"severity,omitempty"`
	Summary      sql.NullString  `db:"summary" json:"summary,omitempty"`
	FixedVersion sql.NullString  `db:"fixed_version" json:"fixed_version,omitempty"`
	CVSSScore    sql.NullFloat64 `db:"cvss_score" json:"cvss_score,omitempty"`
	References   sql.NullString  `db:"references" json:"references,omitempty"`
	FetchedAt    sql.NullTime    `db:"fetched_at" json:"fetched_at,omitempty"`
	CreatedAt    time.Time       `db:"created_at" json:"created_at"`
	UpdatedAt    time.Time       `db:"updated_at" json:"updated_at"`
}

Vulnerability represents a cached vulnerability record.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.