sql

package
v1.66.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 13, 2026 License: Apache-2.0 Imports: 55 Imported by: 0

Documentation

Overview

Package sql is the index of a store: the commits, versions and pins of its sets with their retention state, the presence filters, the store settings and the archive index of the pack store, in one database that is SQLite for a local store and Postgres for a server.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Index 

type Index struct {
	backup.ObjectStore

	// DefaultPolicy is the retention policy sets inherit when the store
	// has none set; nil means retention.Default.
	DefaultPolicy *retention.Policy
	// Now is the clock; nil means time.Now.
	Now func() time.Time
	// Logger is where the index reports; nil means slog.Default.
	Logger *slog.Logger
	// Grouping, when set, names the group each set is collected in: a
	// group carries everything its sets reach, whatever another group
	// reaches too. nil counts every set of the store together.
	Grouping func(ctx context.Context) (map[int64]int64, error)
	// contains filtered or unexported fields
}

Index is the store index over one database.

func New 

func New(location string, store backup.ObjectStore, opts ...Option) *Index

New returns an index over store at location: a directory holds an SQLite database and postgres:// names a Postgres server. Open connects and applies the pending migrations.

func NewMemory 

func NewMemory(name string, store backup.ObjectStore, opts ...Option) *Index

NewMemory returns an index over an in-memory SQLite database that lives as long as the index is open; name keeps two of them apart.

func (*Index) Abandon 

func (x *Index) Abandon(name string, within time.Duration) (bool, error)

Abandon implements pack.ClaimIndex.

func (*Index) AddObjects 

func (x *Index) AddObjects(name string, records []pack.IndexRecord) error

AddObjects implements pack.ClaimIndex. The archive's row is locked for the insert, so a finalize or an abandon cannot come between the check and the rows.

func (*Index) BeginCommit 

func (x *Index) BeginCommit(ctx context.Context, name string) (*backup.CommitGrant, error)

BeginCommit implements backup.CommitGate: the set is created if needed, with the initial retention of the context's session, it must be active, and the grant carries the store policy.

func (*Index) BeginSession 

func (x *Index) BeginSession(s *backup.Session) error

BeginSession implements pack.SessionIndex.

func (*Index) BuildPendingPresence 

func (x *Index) BuildPendingPresence(ctx context.Context) (int, error)

BuildPendingPresence builds the filter of every set whose newest live commit has none and reports how many it built.

func (*Index) BuildPresence 

func (x *Index) BuildPresence(ctx context.Context, commit *proto.Ref) error

BuildPresence builds and stores the presence filter of an indexed commit and records its logical size.

func (*Index) Claims 

func (x *Index) Claims(id string) ([]pack.Claim, error)

Claims implements pack.ClaimIndex.

func (*Index) Close 

func (x *Index) Close() error

Close releases the database; one passed through WithDB stays open.

func (*Index) CommitDetails 

func (x *Index) CommitDetails(ctx context.Context, backupSet string, notAfter time.Time, count int) ([]index.CommitDetail, error)

CommitDetails is CommitInfo with what the index knows about each commit beyond the object, for a caller reporting on a set rather than reading it back.

func (*Index) CommitDetailsBefore 

func (x *Index) CommitDetailsBefore(ctx context.Context, backupSet string, before time.Time, limit int) ([]index.CommitDetail, error)

CommitDetailsBefore pages through a set's live, complete commits by receipt, newest first: those received before before, or all of them when it is zero. It returns at most limit, every one when limit is zero, and then the others received at the same instant as the last, so the last one's ReceivedAtNs is the before of the next page.

func (*Index) CommitInfo 

func (x *Index) CommitInfo(ctx context.Context, backupSet string, notAfter time.Time, count int) ([]*proto.Commit, error)

CommitInfo lists the live, complete commits of a set, newest first.

func (*Index) CommitSession 

func (x *Index) CommitSession(id string) error

CommitSession implements pack.SessionIndex: the session must still be live, then its pending archives become committed and its lost ones go.

func (*Index) CommitSizes 

func (x *Index) CommitSizes(ctx context.Context, backupSet string, notAfter time.Time, count int) ([]*proto.Commit, []*proto.CommitSize, error)

CommitSizes implements backup.CommitSizer.

func (*Index) CountCommits 

func (x *Index) CountCommits(ctx context.Context, backupSet string, period proto.Period, from, to time.Time, zone string, deleted bool) ([]*proto.CommitCount, error)

CountCommits implements backup.Retention.

func (*Index) CountObjects 

func (x *Index) CountObjects() (uint64, uint64, error)

CountObjects implements pack.ArchiveIndex: the object rows and the distinct refs among them.

func (*Index) CountSets 

func (x *Index) CountSets(ctx context.Context, states ...string) (int, error)

CountSets counts the sets in one of the states, every set when none is given.

func (*Index) Delete 

func (x *Index) Delete(ctx context.Context, ref *proto.Ref) error

Delete implements backup.ObjectStore: objects are retired through retention, never deleted directly.

func (*Index) DeleteArchives 

func (x *Index) DeleteArchives(names []string) error

DeleteArchives implements pack.ArchiveIndex; the objects go with the archives.

func (*Index) DeleteCommit 

func (x *Index) DeleteCommit(ctx context.Context, ref *proto.Ref) error

DeleteCommit implements backup.Retention.

func (*Index) DeleteSet 

func (x *Index) DeleteSet(ctx context.Context, name string, erase bool) error

DeleteSet implements backup.Retention.

func (*Index) EndSession 

func (x *Index) EndSession(id string) ([]string, error)

EndSession implements pack.SessionIndex: the session's uncommitted archives go with it.

func (*Index) EscrowedKeys 

func (x *Index) EscrowedKeys(ctx context.Context) ([]backup.EscrowedKey, error)

EscrowedKeys implements backup.KeyEscrow when the store does.

func (*Index) FileInfo 

func (x *Index) FileInfo(ctx context.Context, backupSet string, name string, notAfter time.Time, count int) ([]*proto.TreeNode, error)

FileInfo lists the versions of a path that a live commit of the set contains, newest first.

func (*Index) FillMissingSizes 

func (x *Index) FillMissingSizes(ctx context.Context) (int, error)

FillMissingSizes records the logical size and file count of every commit missing either, and reports how many it filled. A tombstoned commit is skipped: the file versions it held may already be pruned, so its size can no longer be worked out.

func (*Index) FinalizeArchive 

func (x *Index) FinalizeArchive(name string, within time.Duration, created time.Time) error

FinalizeArchive implements pack.ClaimIndex.

func (*Index) GetCommitDetail 

func (x *Index) GetCommitDetail(ctx context.Context, backupSet string, ref *proto.Ref) (index.CommitDetail, error)

GetCommitDetail returns the set's live, complete commit ref names, as CommitDetails describes it, or backup.ErrNotFound.

func (*Index) GetDefaultPolicy 

func (x *Index) GetDefaultPolicy(ctx context.Context) (policy retention.Policy, stored bool, err error)

GetDefaultPolicy returns the retention policy sets inherit until they set their own, and whether the operator stored it or it is the built-in default.

func (*Index) GetPolicy 

func (x *Index) GetPolicy(ctx context.Context, name string) (index.SetRetention, error)

GetPolicy reports a set's retention.

func (*Index) GetSession 

func (x *Index) GetSession(id string) (*backup.Session, error)

GetSession implements pack.SessionIndex.

func (*Index) GetSet 

func (x *Index) GetSet(ctx context.Context, name string) (index.SetInfo, error)

GetSet returns the named set as ListSets describes it, or backup.ErrNotFound.

func (*Index) GetStorePolicy 

func (x *Index) GetStorePolicy(ctx context.Context) (index.StorePolicy, error)

GetStorePolicy returns the store's policy; a store that never had one set reports version 0 with the default parameters.

func (*Index) GetTrashedCommit 

func (x *Index) GetTrashedCommit(ctx context.Context, backupSet string, ref *proto.Ref) (*proto.TrashedCommit, error)

GetTrashedCommit returns the set's commit ref names while it waits in the trash, as TrashedCommits describes it, or backup.ErrNotFound.

func (*Index) HasAll 

func (x *Index) HasAll(ctx context.Context, refs []*proto.Ref) ([]bool, error)

HasAll implements backup.HasAller as the store's Has would answer.

func (*Index) Holds 

func (x *Index) Holds(ref *proto.Ref, id string) (bool, error)

Holds implements pack.ClaimIndex.

func (*Index) IndexArchive 

func (x *Index) IndexArchive(info pack.ArchiveInfo, index pack.IndexFile) error

IndexArchive implements pack.ArchiveIndex; a pending archive needs a live session.

func (*Index) IndexCommits 

func (x *Index) IndexCommits(ctx context.Context, refs []*proto.Ref, dryRun bool) ([]index.IndexedSet, error)

IndexCommits indexes the commits refs name, already in the store, without a rebuild: by set in receipt order, as ReIndex does, those that name no set under a placeholder set. It refuses the lot if a ref is not a commit, is tombstoned or already has a row (index.ErrIndexed). With dryRun it writes nothing and reports what it would do.

func (*Index) LatestCommit 

func (x *Index) LatestCommit(ctx context.Context, backupSet string) (*proto.Ref, error)

LatestCommit returns the set's newest commit that is neither tombstoned nor deleted, partial or not, or backup.ErrNotFound.

func (*Index) ListSessions 

func (x *Index) ListSessions() ([]*backup.Session, error)

ListSessions implements pack.SessionIndex.

func (*Index) ListSets 

func (x *Index) ListSets(ctx context.Context) ([]index.SetInfo, error)

ListSets returns every set by name, with the sizes MeasureSets and the last garbage collection stored for it.

func (*Index) LocateCopies 

func (x *Index) LocateCopies(refs []*proto.Ref, scope pack.Scope) (map[string][]pack.IndexLocation, error)

LocateCopies implements pack.ArchiveIndex.

func (*Index) LocateObject 

func (x *Index) LocateObject(ref *proto.Ref, scope pack.Scope, exclude ...string) (pack.IndexLocation, error)

LocateObject implements pack.ArchiveIndex: the committed archives plus the pending archives of the scope's session.

func (*Index) LocateRecords 

func (x *Index) LocateRecords(ctx context.Context, refs []*proto.Ref) ([]*proto.LocatedRun, error)

LocateRecords implements backup.RecordLocator when the store does, and otherwise locates nothing.

func (*Index) LocateTombstones 

func (x *Index) LocateTombstones(refs []*proto.Ref, scope pack.Scope) (map[string][]pack.IndexLocation, error)

LocateTombstones implements pack.ArchiveIndex through the index of tombstone rows alone.

func (*Index) LookupArchive 

func (x *Index) LookupArchive(name string) (pack.ArchiveInfo, bool, error)

LookupArchive implements pack.ArchiveIndex.

func (*Index) Lost 

func (x *Index) Lost(id string) ([]*proto.Ref, error)

Lost implements pack.ClaimIndex.

func (*Index) LostVersions 

func (x *Index) LostVersions(ctx context.Context, name string) ([]index.LostVersion, error)

LostVersions returns the versions of a set the store cannot read and the commits each one damages. A restore of those commits cannot produce that path.

func (*Index) MarkDamaged 

func (x *Index) MarkDamaged(ctx context.Context, setID int64, paths []string) error

MarkDamaged records that the paths of a set hold content the store lost, so the next backup reads them again. An unknown set is ignored, since a set that is gone has nothing to repair.

func (*Index) MarkLost 

func (x *Index) MarkLost(ctx context.Context, versions []backup.FilePath) error

MarkLost records that stored versions cannot be read any more. Every commit whose range covers one holds a file it cannot restore.

func (*Index) MarkRescan 

func (x *Index) MarkRescan(ctx context.Context, setID int64) error

MarkRescan records that a whole set must be read again, for damage no path could be resolved for.

func (*Index) MeasureSets 

func (x *Index) MeasureSets(ctx context.Context) ([]index.SetInfo, error)

MeasureSets stores the sizes of every set whose live commits changed since they were last measured, or that was never measured, which ListSets, QuerySets and GetSet then report. It returns the sets it measured as those describe them.

func (*Index) Open 

func (x *Index) Open() error

Open connects to the database and applies the pending migrations.

func (*Index) OpenArchive 

func (x *Index) OpenArchive(name, id string) error

OpenArchive implements pack.ClaimIndex.

func (*Index) OrphanCommits 

func (x *Index) OrphanCommits(ctx context.Context) ([]index.OrphanCommit, error)

OrphanCommits lists, by ref, the commits the store's committed archives hold that the index has no row for and no tombstone retires.

func (*Index) PathsOfFiles 

func (x *Index) PathsOfFiles(ctx context.Context, refs []*proto.Ref) ([]backup.FilePath, error)

PathsOfFiles returns where the given file objects are stored, by set and path. A file object no live version references any more yields nothing.

func (*Index) PendingArchives 

func (x *Index) PendingArchives(id string) ([]string, error)

PendingArchives implements pack.SessionIndex.

func (*Index) Pins 

func (x *Index) Pins(ctx context.Context) ([]*proto.PinInfo, error)

Pins implements backup.Retention.

func (*Index) PinsOf 

func (x *Index) PinsOf(ctx context.Context, targets ...*proto.Ref) ([]*proto.PinInfo, error)

PinsOf returns the pins Pins lists that hold one of the targets.

func (*Index) Presence 

func (x *Index) Presence(ctx context.Context, scope backup.PresenceScope, backupSet string) ([]*proto.PresenceFilter, error)

Presence implements backup.PresenceIndex.

func (*Index) Put 

func (x *Index) Put(ctx context.Context, object *proto.Object) error

Put stores an object and indexes a commit or pin.

func (*Index) PutEscrowedKey 

func (x *Index) PutEscrowedKey(ctx context.Context, key backup.EscrowedKey) error

PutEscrowedKey implements backup.KeyEscrow when the store does.

func (*Index) QuerySets 

func (x *Index) QuerySets(ctx context.Context, q index.SetQuery) ([]index.SetInfo, error)

QuerySets returns a page of the sets q picks, described as ListSets describes them. The last one's Name and PhysicalSize are the After and AfterSize of the next page.

func (*Index) ReIndex 

func (x *Index) ReIndex(ctx context.Context) (backup.ReIndexReport, error)

ReIndex rebuilds the caches from the archives in three passes: tombstones, then pins, then commits per set in receipt order. Sets it has to create come up with retention paused.

func (*Index) Reachable 

func (x *Index) Reachable(ctx context.Context, sets []string, ref *proto.Ref) (bool, error)

Reachable reports whether a commit of one of the named sets references ref, which is what a caller limited to those sets may read.

func (*Index) ReadDir 

func (x *Index) ReadDir(ctx context.Context, backupSet string, dir string, notAfter time.Time) ([]*proto.TreeNode, error)

ReadDir lists what a set held directly under dir at notAfter, sorted by name. Directory entries carry a name and a ref only.

func (*Index) ReadDirAfter 

func (x *Index) ReadDirAfter(ctx context.Context, backupSet string, dir string, notAfter time.Time, after string, limit int) ([]*proto.TreeNode, error)

ReadDirAfter pages through what ReadDir lists, ordered by path byte by byte rather than by name: the entries whose paths sort after after, at most limit, every one when limit is zero. An entry's path is proto.JoinPath(dir, its name), and the last one's is the after of the next page.

func (*Index) ReadRecords 

func (x *Index) ReadRecords(ctx context.Context, refs []*proto.Ref) ([]*proto.Object, error)

ReadRecords implements backup.RecordReader when the store does, and reads nothing otherwise.

func (*Index) RecordSetSizes 

func (x *Index) RecordSetSizes(ctx context.Context, report *pack.CollectReport) error

RecordSetSizes records what a garbage collection attributed to each set: the physical, deduplicated, alone, exclusive and deduplicated alone sizes; a set it did not name holds nothing of its own. What it attributed to a set the index does not hold is added to report.Unattributed.

func (*Index) References 

func (x *Index) References(ctx context.Context, ref *proto.Ref) (bool, error)

References implements backup.RefScope: a commit, tree or file ref is readable once an indexed commit of any set the caller sees names it.

func (*Index) ReferencesAll 

func (x *Index) ReferencesAll(ctx context.Context, refs []*proto.Ref) ([]bool, error)

ReferencesAll implements backup.RefScope in one query.

func (*Index) ReindexChurned 

func (x *Index) ReindexChurned(ctx context.Context) ([]index.Reindexed, error)

ReindexChurned rebuilds, one at a time and without blocking writes, the indexes of every table of the index whose rows changed by at least 30% of its live rows, and at least 10,000, since its last reindex, and reports each table it rebuilt. It is meant for the end of a maintenance run that churned the index, such as a compaction, retirement or collection. Only one ReindexChurned of a database schema runs at a time; another returns nothing. On SQLite it does nothing.

func (*Index) Retire 

func (x *Index) Retire(ctx context.Context, now time.Time) (int, error)

Retire implements backup.Retirer as RetireCommits does.

func (*Index) RetireCommits 

func (x *Index) RetireCommits(ctx context.Context, now time.Time) ([]index.Retired, error)

RetireCommits gives every retired commit past its window a tombstone, drops the rows of its set no other commit holds, and makes a closing set with nothing left deleted. An active set whose retention is paused is skipped; a deleted set proceeds regardless. It returns the commits it tombstoned, those before a failure included.

func (*Index) RetiredCommits 

func (x *Index) RetiredCommits(ctx context.Context, refs []*proto.Ref) ([]index.Retired, error)

RetiredCommits reports the retirement of each of the commits whose set the caller sees, in the order given, leaving out the others; a commit live now, unretired or never retired, is RetiredLive.

func (*Index) RootOwner 

func (x *Index) RootOwner(ctx context.Context) (func(root []byte) pack.Attribution, error)

RootOwner returns the lookup a garbage collection attributes with: it answers a root, a commit or a pin holding one, with the set it belongs to and the group that set is collected in, and anything else with nothing.

func (*Index) SetDefaultPolicy 

func (x *Index) SetDefaultPolicy(ctx context.Context, p *retention.Policy) error

SetDefaultPolicy stores the retention policy sets inherit until they set their own, nil for the built-in default, and re-evaluates every set.

func (*Index) SetPolicy 

func (x *Index) SetPolicy(ctx context.Context, name string, p *retention.Policy) error

SetPolicy stores a set's policy, nil to inherit the store's default, clamped by the limits, and re-evaluates the set. Setting a policy also resumes retention paused by a rebuild.

func (*Index) SetStorePolicy 

func (x *Index) SetStorePolicy(ctx context.Context, policy storekey.Policy, acknowledge bool, now time.Time) (index.StorePolicy, error)

SetStorePolicy stores policy under the next version and, when acknowledge is set, records that the operator has saved the store key.

func (*Index) SetWindows 

func (x *Index) SetWindows(ctx context.Context, w index.Windows) error

SetWindows stores the trash window and re-evaluates every set; a negative window is refused.

func (*Index) SharedNow 

func (x *Index) SharedNow(ctx context.Context) (time.Time, error)

SharedNow implements pack.SharedClock.

func (*Index) Stamp 

func (x *Index) Stamp(ctx context.Context, object *proto.Object) error

Stamp assigns the receipt time to a commit or pin and the set id to a commit, replacing whatever they carried.

func (*Index) StorePolicy 

func (x *Index) StorePolicy(ctx context.Context) (*storekey.Policy, error)

StorePolicy implements backup.PolicySource: nil until an operator set a policy.

func (*Index) TombstonedCommits 

func (x *Index) TombstonedCommits(ctx context.Context, name string, since time.Time) ([]*proto.Ref, error)

TombstonedCommits returns the set's tombstoned commits, oldest first: those tombstoned at or after since, or all when it is zero.

func (*Index) TouchSession 

func (x *Index) TouchSession(id string, at time.Time) error

TouchSession implements pack.SessionIndex.

func (*Index) TrashedCommits 

func (x *Index) TrashedCommits(ctx context.Context, backupSet string, before time.Time, limit int) ([]*proto.TrashedCommit, error)

TrashedCommits implements backup.Retention.

func (*Index) UndeleteCommit 

func (x *Index) UndeleteCommit(ctx context.Context, ref *proto.Ref) error

UndeleteCommit implements backup.Retention.

func (*Index) UndeleteSet 

func (x *Index) UndeleteSet(ctx context.Context, name string) error

UndeleteSet implements backup.Retention.

func (*Index) Unpin 

func (x *Index) Unpin(ctx context.Context, ref *proto.Ref) error

Unpin implements backup.Retention.

func (*Index) UnretireCommits 

func (x *Index) UnretireCommits(ctx context.Context, refs []*proto.Ref, dryRun bool) ([]index.Unretired, error)

UnretireCommits brings tombstoned commits of active sets back to live while the store still holds everything they reach, and has each set's policy decide on them again. A commit missing an object is reported and stays tombstoned. dryRun changes nothing and reports what the policies would keep.

func (*Index) VersionsBefore 

func (x *Index) VersionsBefore(ctx context.Context, backupSet string, name string, before time.Time, limit int) ([]index.Version, error)

VersionsBefore pages through the versions of a path that a live commit of the set contains, newest first: those first held before before, or all of them when it is zero, at most limit, every one when limit is zero. The last one's From is the before of the next page.

func (*Index) WalkHeaders 

func (x *Index) WalkHeaders(ctx context.Context, t proto.ObjectType, fn func(*proto.ObjectHeader) error) error

WalkHeaders implements backup.HeaderWalker when the store does.

func (*Index) Windows 

func (x *Index) Windows(ctx context.Context) (index.Windows, error)

Windows returns the store's trash window.

type Option 

type Option func(*Index)

Option configures an index before it is opened.

func WithDB 

func WithDB(db *sql.DB, d string) Option

WithDB opens the index over a database the caller connected, instead of the location's; d is dialect.SQLite or dialect.Postgres. Open still applies the migrations unless WithoutMigrations says otherwise, and Close leaves db open.

func WithoutMigrations 

func WithoutMigrations() Option

WithoutMigrations makes Open trust that the schema is current, for a caller that migrates the database itself before opening indexes over it.

Source Files

View all Source files

Directories

Path Synopsis
ent
archive
commitrow
damagedpath
deletedref
enttest
file
hook
migrate
object
pin
predicate
reindex
runtime
schema
Package schema is the ent schema of the index: store settings, sets with their commits, versions, refs and retention state, pins, tombstoned refs and the archive index, in one database.
 Package schema is the ent schema of the index: store settings, sets with their commits, versions, refs and retention state, pins, tombstoned refs and the archive index, in one database.
session
set
setref
settings
tree
walkedarchive
Package mapping turns index rows into the protos and structs the rest of goback speaks.
 Package mapping turns index rows into the protos and structs the rest of goback speaks.
gen
Package migrations holds the versioned migrations of the index, one directory per dialect, and applies them on open.
 Package migrations holds the versioned migrations of the index, one directory per dialect, and applies them on open.
gen command
Command gen appends a migration to every dialect's directory when the ent schema changed.
 Command gen appends a migration to every dialect's directory when the ent schema changed.
generate
Package generate writes the migration an ent schema change needs, for goback's own index and for anything built on goback that keeps its own schema the same way.
 Package generate writes the migration an ent schema change needs, for goback's own index and for anything built on goback that keeps its own schema the same way.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.