sqliteindex

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 4, 2026 License: LGPL-2.1 Imports: 23 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CountBlobRefs 

func CountBlobRefs(entriesArr []core.TreeEntryObj) (map[core.HashObj]BlobRefObj, error)

CountBlobRefs aggregates tree entries into blob refcount and size. A size mismatch for the same hash means the tree is internally inconsistent.

func NewVersion 

func NewVersion(key string, version string, sourceHashObj core.HashObj, sourceSizeBytes uint64, treeHashObj core.HashObj, upstreamDeletedFlag bool) core.VersionObj

NewVersion builds core.VersionObj with current UTC IngestTS and other fields from arguments.

Types

type BlobRefObj 

type BlobRefObj struct {
	Refcount  uint64
	SizeBytes uint64
}

BlobRefObj stores blob refcount and size for reachability and logical usage accounting.

type BlobRefRowObj 

type BlobRefRowObj struct {
	Hash core.HashObj
	Ref  BlobRefObj
}

BlobRefRowObj is one blob_refs row for keyset paging.

type Obj 

type Obj struct {
	// contains filtered or unexported fields
}

Obj stores the SQLite metadata index for versions, detection, blob_refs, artifacts and history. Reads go through the pool; writes are serialized by the external storage.writeMu. Transactional mutations go through TxObj and WithTx.

func Open 

func Open(ctx context.Context, pathToFile string) (*Obj, error)

Open validates the path, opens SQLite with WAL/synchronous=NORMAL, runs migrations and format checks. Format or schema mismatch stops open.

func (*Obj) BlobRefHashesPage 

func (obj *Obj) BlobRefHashesPage(ctx context.Context, afterHash []byte, limit int) ([]core.HashObj, error)

BlobRefHashesPage returns a keyset page of live blob hashes for GC reachable-set membership.

func (*Obj) BlobReferenced 

func (obj *Obj) BlobReferenced(ctx context.Context, hashObj core.HashObj) (bool, error)

BlobReferenced reports whether a blob has live references.

func (*Obj) BlobRefs 

func (obj *Obj) BlobRefs(ctx context.Context) (map[core.HashObj]BlobRefObj, error)

BlobRefs loads the full blob_refs table into a map for rebuild and integrity paths; paged callers use *Page.

func (*Obj) BlobRefsPage 

func (obj *Obj) BlobRefsPage(ctx context.Context, afterHash []byte, limit int) ([]BlobRefRowObj, error)

BlobRefsPage returns a keyset page of blob_hash, refcount, and size_bytes. It has no refcount filter so integrity checks also see refcount=0 rows.

func (*Obj) CheckSQLiteIntegrity 

func (obj *Obj) CheckSQLiteIntegrity(ctx context.Context) ([]string, error)

CheckSQLiteIntegrity runs quick_check and foreign_key_check, returning discovered issues. An empty list means a consistent DB; used during open and --inspect.

func (*Obj) Checkpoint 

func (obj *Obj) Checkpoint(ctx context.Context) error

Checkpoint force-runs wal_checkpoint(TRUNCATE) with a timeout and errors on incomplete results.

func (*Obj) Close 

func (obj *Obj) Close() error

Close closes the index with a background context.

func (*Obj) CloseContext 

func (obj *Obj) CloseContext(ctx context.Context) error

CloseContext runs the final checkpoint(TRUNCATE), then closes migrations, prepared cache and DB. The method is safe for a nil receiver; statement cache is closed before DB.

func (*Obj) ContentChecksum 

func (obj *Obj) ContentChecksum(ctx context.Context) (core.HashObj, error)

ContentChecksum is deterministic blake3 over active key, version, and tree_hash rows in canonical order. It is the mirror content fingerprint used to compare nodes and brothers.

func (*Obj) CountTable 

func (obj *Obj) CountTable(ctx context.Context, tableName string) (uint64, error)

CountTable returns COUNT(*) for whitelisted tables used by --inspect. Other table names are rejected against injection.

func (*Obj) CountVersionsByKey 

func (obj *Obj) CountVersionsByKey(ctx context.Context, key string, includeDeleted bool) (uint64, error)

CountVersionsByKey returns key version count through SQL COUNT without loading rows.

func (*Obj) DeleteUnreferencedBlobRefs 

func (obj *Obj) DeleteUnreferencedBlobRefs(ctx context.Context, hashArr []core.HashObj) error

DeleteUnreferencedBlobRefs deletes refcount=0 blob_refs rows for specified hashes in batches. It records accounting deletion after physical blob GC.

func (*Obj) DistinctVersionKeys 

func (obj *Obj) DistinctVersionKeys(ctx context.Context, afterKey string, limit int) ([]string, error)

DistinctVersionKeys returns one keyset page of unique keys with key > afterKey. It supports prune without loading all versions for all keys into RAM under writeMu.

func (*Obj) GetArtifact 

func (obj *Obj) GetArtifact(ctx context.Context, keyObj core.ArtifactKeyObj) (core.ArtifactObj, bool, error)

GetArtifact returns a materialized artifact by full identity key; missing rows return false, nil.

func (*Obj) GetDetection 

func (obj *Obj) GetDetection(ctx context.Context, key string, version string) (core.DetectionObj, bool, error)

GetDetection returns version detection; missing rows return false, nil.

func (*Obj) GetKeySource 

func (obj *Obj) GetKeySource(ctx context.Context, key string) (core.KeySourceObj, bool, error)

GetKeySource returns a binding by key name; the second result reports whether it exists.

func (*Obj) GetVersion 

func (obj *Obj) GetVersion(ctx context.Context, key string, version string) (core.VersionObj, bool, error)

GetVersion returns a version by key and version; missing rows return false, nil.

func (*Obj) LatestVersion 

func (obj *Obj) LatestVersion(ctx context.Context, key string) (core.VersionObj, bool, error)

LatestVersion returns the newest active version without release_notes; no row returns false, nil. Semver keys use semver max to match go-proxy @latest, while raw versions use upstream_seq order. For mixed keys, the later source-listing position wins between semver max and raw top.

func (*Obj) ListArtifacts 

func (obj *Obj) ListArtifacts(ctx context.Context, key string, version string) ([]core.ArtifactObj, error)

ListArtifacts returns all materialized artifacts for a version for release_detail Artifacts. Order is not guaranteed; consumers sort when needed.

func (*Obj) ListKeySources 

func (obj *Obj) ListKeySources(ctx context.Context) ([]core.KeySourceObj, error)

ListKeySources returns all bindings for boot name-to-URL checks.

func (*Obj) ListPublishFeed 

func (obj *Obj) ListPublishFeed(ctx context.Context, key string, limit int) ([]core.FeedEventObj, error)

ListPublishFeed returns newest-first publish events with release notes and the first-publish flag. An empty key means all keys; Atom feeds use this.

func (*Obj) ListVersions 

func (obj *Obj) ListVersions(ctx context.Context, key string, includeDeleted bool) ([]core.VersionObj, error)

ListVersions returns the full service newest-first list without reading release_notes. Use GetVersion or page/keyset views when notes are needed; includeDeleted=false hides tombstones.

func (*Obj) ListVersionsKeyset 

func (obj *Obj) ListVersionsKeyset(ctx context.Context, key string, includeDeleted bool, afterSeq int64, afterVersion string, limit int) ([]core.VersionObj, error)

ListVersionsKeyset returns a newest-first keyset page by afterSeq and afterVersion cursor. Empty afterVersion means first page; the path is O(log n + limit) through idx_versions_key_seq without OFFSET scans. The compound (seq, version) cursor tolerates duplicate seq values, so UNIQUE(key, upstream_seq) is unnecessary.

func (*Obj) ListVersionsKeysetBefore 

func (obj *Obj) ListVersionsKeysetBefore(ctx context.Context, key string, includeDeleted bool, beforeSeq int64, beforeVersion string, limit int) ([]core.VersionObj, error)

ListVersionsKeysetBefore returns up to limit versions strictly newer than the (beforeSeq, beforeVersion) cursor, ordered ASCENDING (closest to the cursor first). The serve layer reverses for newest-first display and trims the keyset overshoot. Powers the "newer" pager direction of the key page without OFFSET.

func (*Obj) ListVersionsPage 

func (obj *Obj) ListVersionsPage(ctx context.Context, key string, includeDeleted bool, limit int, offset int) ([]core.VersionObj, error)

ListVersionsPage returns one newest-first page through LIMIT/OFFSET without loading the full set.

func (*Obj) MaxUpstreamSeq 

func (obj *Obj) MaxUpstreamSeq(ctx context.Context, key string) (int64, error)

MaxUpstreamSeq returns the maximum upstream_seq for a key across all rows, deleted included; 0 means none.

func (*Obj) PruneHistory 

func (obj *Obj) PruneHistory(ctx context.Context, maxEvents uint) (int64, error)

PruneHistory deletes oldest events beyond maxEvents, bounding the append-only source for Atom. maxEvents==0 means unlimited; newest maxEvents by id are kept, and an underfilled set is unchanged.

func (*Obj) ReferencedBytes 

func (obj *Obj) ReferencedBytes(ctx context.Context) (uint64, error)

ReferencedBytes returns logical durable content size from live blob references. It is deterministic and drops immediately on delete, unlike physical DiskSpaceUsage with LSM lag.

func (*Obj) RewriteSet 

func (obj *Obj) RewriteSet(ctx context.Context, key string, version string) ([]core.HashObj, error)

RewriteSet returns version rewrite-set blob hashes sorted by blob_hash.

func (*Obj) SmallestArtifactByDescriptor 

func (obj *Obj) SmallestArtifactByDescriptor(ctx context.Context, materializerID string, artifactKind string, formatVersion uint32) (core.ArtifactObj, bool, error)

SmallestArtifactByDescriptor returns the smallest artifact for a materializer, kind, and format version. Startup self-test uses one cheap artifact per codec; kind is required because zip and tar.gz share materializer and format. Missing rows return false, nil.

func (*Obj) TreeHashCountsPage 

func (obj *Obj) TreeHashCountsPage(ctx context.Context, afterHash []byte, limit int) ([]TreeHashCountObj, error)

TreeHashCountsPage returns one keyset page of tree_hash counts for integrity and repair.

func (*Obj) TreeHashesPage 

func (obj *Obj) TreeHashesPage(ctx context.Context, afterHash []byte, limit int) ([]core.HashObj, error)

TreeHashesPage returns one keyset page of unique tree hashes. It feeds GC reachable-set membership without loading all versions into RAM.

func (*Obj) TreeReferenced 

func (obj *Obj) TreeReferenced(ctx context.Context, hashObj core.HashObj) (bool, error)

TreeReferenced reports whether any version references a tree hash for GC reachability.

func (*Obj) UnreferencedBlobs 

func (obj *Obj) UnreferencedBlobs(ctx context.Context, hashArr []core.HashObj) (map[core.HashObj]struct{}, error)

UnreferencedBlobs returns the subset of hashArr with refcount=0 rows as deletion candidates. Queries are batched by sqlMaxBindArgs; hashes without blob_refs rows are handled by another path.

func (*Obj) UpdateArtifactDigest 

func (obj *Obj) UpdateArtifactDigest(ctx context.Context, artifactObj core.ArtifactObj) error

UpdateArtifactDigest updates digest metadata by identity primary key and clears the degraded state. file_path is cleared because the old hot file is named by the old body_hash and must be rebuilt. This is UPDATE, not upsert: missing rows are not resurrected.

func (*Obj) UpdateArtifactPath 

func (obj *Obj) UpdateArtifactPath(ctx context.Context, artifactObj core.ArtifactObj, filePath string) error

UpdateArtifactPath attaches a hot file to an artifact and updates sha digests. WHERE also checks body_hash, size, and format_version so stale identities are not attached.

func (*Obj) Vacuum 

func (obj *Obj) Vacuum(ctx context.Context) error

Vacuum rebuilds the DB file and returns freed pages. SQLite requires running outside a transaction; storage.writeMu serializes the call, and free disk must cover DB size.

func (*Obj) VersionsByIngest 

func (obj *Obj) VersionsByIngest(ctx context.Context, afterObj core.VersionObj, afterFlag bool, limitValue uint64) ([]core.VersionObj, error)

VersionsByIngest walks versions by service ingest order and intentionally skips release_notes.

func (*Obj) WithTx 

func (obj *Obj) WithTx(ctx context.Context, useFunc func(*TxObj) error) (err error)

WithTx runs useFunc in a transaction: nil error commits, non-nil rolls back. Panics are rolled back and converted to errors; callers provide write serialization.

type TreeHashCountObj 

type TreeHashCountObj struct {
	Hash  core.HashObj
	Count uint64
}

TreeHashCountObj is one GROUP BY row with tree hash and referencing version count.

type TxObj 

type TxObj struct {
	// contains filtered or unexported fields
}

TxObj opens operations inside one transaction; mutators live here, not on Obj.

func (*TxObj) AddBlobRefs 

func (obj *TxObj) AddBlobRefs(ctx context.Context, entriesArr []core.TreeEntryObj) error

AddBlobRefs increments refcount and records size for blobs from tree entries through batched upsert.

func (*TxObj) AddHistory 

func (obj *TxObj) AddHistory(ctx context.Context, key string, version string, eventType string, treeHashObj core.HashObj, bodyHashObj core.HashObj, message string) error

AddHistory appends a feed event; zero tree or body hashes are stored as SQL NULL.

func (*TxObj) ArtifactPaths 

func (obj *TxObj) ArtifactPaths(ctx context.Context, key string, version string) ([]string, error)

ArtifactPaths returns non-empty hot-file paths for a version for transactional prune or replace.

func (*TxObj) ClearUpstreamDeleted 

func (obj *TxObj) ClearUpstreamDeleted(ctx context.Context, key string, version string) error

ClearUpstreamDeleted clears the tombstone when the version appears upstream again.

func (*TxObj) CopyVersionMetadata 

func (obj *TxObj) CopyVersionMetadata(ctx context.Context, oldObj core.VersionObj, historicalVersion string) error

CopyVersionMetadata clones a version under a historical tag with detection, rewrite_set, and artifact metadata. file_path is cleared so hot files are rebuilt under the new tag.

func (*TxObj) DeleteVersionRow 

func (obj *TxObj) DeleteVersionRow(ctx context.Context, key string, version string) error

DeleteVersionRow physically deletes a version row; FK cascade cleans dependent rows.

func (*TxObj) GetVersion 

func (obj *TxObj) GetVersion(ctx context.Context, key string, version string) (core.VersionObj, bool, error)

GetVersion returns a version by key and version within a transaction; missing rows return false, nil.

func (*TxObj) InsertArtifacts 

func (obj *TxObj) InsertArtifacts(ctx context.Context, key string, version string, artifactArr []core.ArtifactObj) error

InsertArtifacts upserts version artifact metadata in batches within sqlMaxBindArgs. Identity conflicts update all mutable fields.

func (*TxObj) InsertDetection 

func (obj *TxObj) InsertDetection(ctx context.Context, key string, version string, detectionObj core.DetectionObj) error

InsertDetection upserts version detection results.

func (*TxObj) InsertKeySource 

func (obj *TxObj) InsertKeySource(ctx context.Context, keySourceObj core.KeySourceObj) error

InsertKeySource upserts a binding and updates all fields on key conflict. listing_mode is intentionally excluded from conflict updates; only UpdateKeySourceListingMode changes it.

func (*TxObj) InsertRewriteSet 

func (obj *TxObj) InsertRewriteSet(ctx context.Context, key string, version string, hashArr []core.HashObj) error

InsertRewriteSet fully replaces the version rewrite set with delete plus batched insert. The set contains blobs whose paths need rewriting during materialization.

func (*TxObj) InsertVersion 

func (obj *TxObj) InsertVersion(ctx context.Context, versionObj core.VersionObj) error

InsertVersion upserts a version row and updates all fields on key/version conflict.

func (*TxObj) MarkUpstreamDeleted 

func (obj *TxObj) MarkUpstreamDeleted(ctx context.Context, key string, version string) error

MarkUpstreamDeleted marks a version as deleted upstream without physically deleting the row.

func (*TxObj) MaxUpstreamSeq 

func (obj *TxObj) MaxUpstreamSeq(ctx context.Context, key string) (int64, error)

MaxUpstreamSeq is the in-transaction variant used by publish to assign a fallback position.

func (*TxObj) NextHistoricalVersion 

func (obj *TxObj) NextHistoricalVersion(ctx context.Context, historyPrefix string) (string, error)

NextHistoricalVersion returns the next monotonic historical tag and increments the globals counter. Missing counters are initialized from the maximum existing tag for the prefix.

func (*TxObj) ReplaceBlobRefs 

func (obj *TxObj) ReplaceBlobRefs(ctx context.Context, refsObj map[core.HashObj]BlobRefObj) error

ReplaceBlobRefs fully rebuilds blob_refs from the supplied set.

func (*TxObj) SetHealPending 

func (obj *TxObj) SetHealPending(ctx context.Context, key string, version string, pending bool) error

SetHealPending marks whether a version needs damaged content recovery.

func (*TxObj) SubtractBlobRefs 

func (obj *TxObj) SubtractBlobRefs(ctx context.Context, entriesArr []core.TreeEntryObj) error

SubtractBlobRefs decrements blob refcount from tree entries through one CTE UPDATE. Underflow is an error protecting accounting from desync; refcount=0 rows remain for later GC.

func (*TxObj) TouchVersionVerified 

func (obj *TxObj) TouchVersionVerified(ctx context.Context, key string, version string, verifiedTS time.Time, upstreamRef string) error

TouchVersionVerified updates deep-verification time and adopts a non-empty upstream_ref. A zero verifiedTS leaves verified_ts unchanged, allowing ref adoption without download.

func (*TxObj) UpdateKeySourceListingMode 

func (obj *TxObj) UpdateKeySourceListingMode(ctx context.Context, key string, mode string) error

UpdateKeySourceListingMode pins the git listing mode for an existing binding. A missing row is a no-op: the binding appears at classification time before any git listing.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.