indexstore

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 11, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// RunStatusRunning indicates the run is currently in progress.
	RunStatusRunning = "running"
	// RunStatusSuccess indicates the run completed successfully.
	RunStatusSuccess = "success"
	// RunStatusPartial indicates the run completed with partial results.
	RunStatusPartial = "partial"
	// RunStatusFailed indicates the run failed.
	RunStatusFailed = "failed"
	// RunStatusFailedResumable indicates the run failed after writing a resumable checkpoint.
	RunStatusFailedResumable = "failed-resumable"
)
View Source 
const (
	// EventCategoryInfo for informational events.
	EventCategoryInfo = "info"
	// EventCategoryWarning for warning events.
	EventCategoryWarning = "warning"
	// EventCategoryError for error events.
	EventCategoryError = "error"
	// EventCategoryThrottle for rate-limiting events.
	EventCategoryThrottle = "throttle"
	// EventCategoryAccess for permission/access events.
	EventCategoryAccess = "access"
)
View Source 
const (
	QueryChangeKindAdded   = "added"
	QueryChangeKindChanged = "changed"
)
View Source 
const CanonicalSQLiteQuarantinePrefix = ".gonimbus-sqlite-quarantine-"

CanonicalSQLiteQuarantinePrefix marks retained exact-epoch capture residue in the index directory. Ordinary open/recovery must discover these entries.

View Source 
const SchemaVersion = 8

Variables

View Source 
var ErrInvalidPrefixStat = errors.New("invalid prefix stat: values must be non-negative")

ErrInvalidPrefixStat is returned when prefix stat values are invalid.

Functions

func BatchInsertPrefixStats 

func BatchInsertPrefixStats(ctx context.Context, db *sql.DB, stats []PrefixStatRow) error

BatchInsertPrefixStats inserts multiple prefix stat records in a single transaction.

Returns ErrInvalidPrefixStat if any record has negative numeric values.

func BatchUpdateHeadEnrichment 

func BatchUpdateHeadEnrichment(ctx context.Context, db *sql.DB, updates []HeadEnrichmentUpdate) error

BatchUpdateHeadEnrichment updates only HEAD-derived enrichment columns.

func BatchUpsertObjects 

func BatchUpsertObjects(ctx context.Context, db *sql.DB, objects []ObjectRow) error

BatchUpsertObjects inserts or updates multiple objects in a single transaction.

This is more efficient than individual UpsertObject calls for bulk ingestion.

func ConfigureDurableConn 

func ConfigureDurableConn(ctx context.Context, conn *sql.Conn) error

ConfigureDurableConn asserts the local durability configuration on a specific pinned connection: WAL journal mode, busy_timeout, and synchronous=FULL, with the resolved synchronous level verified rather than inherited from the driver default. A durability-authoritative store that pins one *sql.Conn as its sole mutation authority calls this so the connection-local FULL guarantee holds on the exact connection that performs its write transactions. synchronous and busy_timeout are connection-local; journal_mode is database-global but is reasserted here so the pinned writer verifies the full configuration itself.

The caller is responsible for having opened a local file: target: this function verifies the resolved PRAGMA values and returns an error if the connection cannot carry WAL+FULL, so a non-local connection fails closed.

This is not redundant with the pool-level configuration done at Open, even though the reflow store keeps SetMaxOpenConns(1): synchronous and busy_timeout are per-connection, not persisted in the database file, so a connection the pool opens or reopens (for example after the prior physical connection is dropped) starts at the driver default. A durability-authoritative writer that pins one connection must therefore assert and verify the setting on that exact connection rather than trust that the pool handed back the one Open configured.

func CountObjects 

func CountObjects(ctx context.Context, db *sql.DB, indexSetID string, includeDeleted bool) (int64, error)

CountObjects returns the count of objects in an index set. If includeDeleted is false, only counts objects where deleted_at IS NULL.

func DeleteIndexSet 

func DeleteIndexSet(ctx context.Context, db *sql.DB, indexSetID string) error

DeleteIndexSet deletes an IndexSet and all associated data.

func DeriveRelKey 

func DeriveRelKey(baseURI, fullKey string) string

DeriveRelKey derives the relative key from a full object key and base URI.

The base URI should be in the form "s3://bucket/prefix/" and the key should be the full object key. Returns the key relative to the base prefix, normalized to never start with "/".

Examples: 

DeriveRelKey("s3://bucket/prefix/", "prefix/file.txt") => "file.txt"
DeriveRelKey("s3://bucket/a/b/", "a/b/c/d.txt") => "c/d.txt"
DeriveRelKey("s3://bucket/", "file.txt") => "file.txt"

func InsertPrefixStat 

func InsertPrefixStat(ctx context.Context, db *sql.DB, stat PrefixStatRow) error

InsertPrefixStat inserts a prefix stat record.

Prefix stats are per-run (not upserted) to maintain history. Returns ErrInvalidPrefixStat if any numeric values are negative.

func MarkIndexRunResuming 

func MarkIndexRunResuming(ctx context.Context, db *sql.DB, runID string) error

MarkIndexRunResuming promotes a failed-resumable run back to running for a same-run resume attempt. It fails closed if the run is no longer resumable.

func MarkIndexRunResumingWithEvents 

func MarkIndexRunResumingWithEvents(ctx context.Context, db *sql.DB, runID string, events []RunEvent) error

MarkIndexRunResumingWithEvents atomically promotes a failed-resumable run back to running and records the resume lifecycle events.

func MarkObjectsDeletedNotSeenInRun 

func MarkObjectsDeletedNotSeenInRun(ctx context.Context, db *sql.DB, indexSetID, runID string, runStartedAt time.Time) (int64, error)

MarkObjectsDeletedNotSeenInRun marks objects as deleted if they were not seen in the given run.

This implements soft-delete semantics: objects that existed in previous runs but were not seen in the current run are marked with deleted_at = run.started_at.

IMPORTANT: This should only be called for SUCCESSFUL runs. For partial runs, missing objects may be due to incomplete traversal (throttling, access denied), not actual deletions. Calling this on partial runs will incorrectly mark objects as deleted.

For partial runs, consider: - Skipping soft-delete entirely (recommended default) - Using an explicit "reconcile deletes" mode with user confirmation - Only soft-deleting objects in prefixes that were fully traversed

func Migrate 

func Migrate(ctx context.Context, db *sql.DB) error

Migrate creates (or upgrades) the index schema in-place.

v0.1.3 starts with a minimal schema that supports: - index set identity + run provenance - current object rows (upserted) - prefix summary stats (tree-backed)

func NewEventID 

func NewEventID() string

NewEventID returns an identifier for a new run event.

func NewRunID 

func NewRunID() string

NewRunID returns an identifier for a new run, in the run_<digits> form the hub layout and the SQLite schema both expect.

func ObjectRowChanged 

func ObjectRowChanged(existing *ObjectRow, candidate ObjectRow) bool

ObjectRowChanged reports whether an incoming LIST row changes the persisted current-state fields that define object identity for delta reporting.

func Open 

func Open(ctx context.Context, cfg Config) (*sql.DB, error)

Open opens (and creates if needed) a pure-Go SQLite-backed index database.

Notes: - Local file paths are created if parent directories do not exist. - For local DBs, WAL and busy_timeout are applied for predictable CLI behavior. - Remote libsql URLs require rebuilding with -tags gonimbus_libsql.

func OpenLocalMutableCanonical 

func OpenLocalMutableCanonical(ctx context.Context, path string, bound *os.File, beforeMutation func() error) (*sql.DB, error)

OpenLocalMutableCanonical opens an existing local SQLite file through its concrete canonical pathname. SQLite therefore uses that name for its lock, WAL, and shared-memory namespace. A per-connection VFS attests SQLite's exact main-file handle against bound and revalidates the callback and main path at SQLite access and I/O boundaries. This ordinary writer path requires transaction sidecars to have been absent at caller validation; VFS registration reasserts that absence and will not adopt an intervening WAL, SHM, rollback, master, or statement journal. The VFS exclusively reserves each new sidecar from the retained main path plus known suffixes (WAL and rollback journal from open-type flags; SHM before xShmMap) through a retained no-follow directory binding, then requires SQLite's exact handle to match that reservation. Exact-epoch cleanup captures the bound object before unlinking it. The VFS remains installed until the connection closes.

func OpenLocalReadOnly 

func OpenLocalReadOnly(ctx context.Context, path string) (*sql.DB, error)

OpenLocalReadOnly opens an existing local SQLite database without creating parent directories, changing journal mode, running migrations, or creating SQLite sidecars. The connection is bound to a no-follow file handle before SQLite sees it, so a pathname swap cannot substitute unverified metadata. Callers must reject transaction sidecars before and after this inspection.

func PurgeDeletedObjects 

func PurgeDeletedObjects(ctx context.Context, db *sql.DB, indexSetID string, olderThan time.Time) (int64, error)

PurgeDeletedObjects permanently removes objects marked as deleted.

This can be used for garbage collection of soft-deleted objects. The olderThan parameter specifies the minimum age of deleted_at for purging.

func QuarantineSQLiteTransactionResidue 

func QuarantineSQLiteTransactionResidue(dbPath string) ([]string, error)

QuarantineSQLiteTransactionResidue returns discoverable quarantine-prefix names beside dbPath without deleting them. Ordinary close may surface this list as blocking residue while leaving every entry intact.

func QueryCanonicalObjects 

func QueryCanonicalObjects(ctx context.Context, db *sql.DB, params QueryParams) ([]CanonicalOutputRecord, CanonicalQueryStats, error)

QueryCanonicalObjects returns canonical-by-ETag output over the same filtered row set as QueryObjects. Existing filters are applied before ETag grouping.

func QueryHeadEnrichmentCandidates 

func QueryHeadEnrichmentCandidates(ctx context.Context, db *sql.DB, params QueryParams) ([]HeadEnrichmentCandidate, QueryStats, error)

QueryHeadEnrichmentCandidates returns post-filter rows that are eligible for HEAD enrichment. Storage class, size, deleted, glob, and regex semantics match QueryObjects; rows are returned in rel_key order.

func QueryObjectCount 

func QueryObjectCount(ctx context.Context, db *sql.DB, params QueryParams) (int64, error)

QueryObjectCount counts objects matching the query without materializing results.

This is optimized for --count scenarios: - When no Pattern/KeyRegex: uses COUNT(*) at DB level (fast) - Otherwise: streams rows and counts matches (constant memory)

Optimization: prefix pushdown is applied when Pattern has a derivable prefix.

func QueryObjects 

func QueryObjects(ctx context.Context, db *sql.DB, params QueryParams) ([]QueryResult, QueryStats, error)

QueryObjects queries the objects_current table with the given filters.

Pattern matching uses doublestar semantics (same as crawl). Results are returned in rel_key order for deterministic output.

Optimization: when Pattern has a derivable prefix (e.g., "data/2025/**"), prefix pushdown is applied via SQL LIKE to use the primary key index.

Timestamp parse failures are handled gracefully: the row is included with nil timestamp fields, and the count is tracked in QueryStats. Callers should check stats.TimestampParseErrors and warn if non-zero.

func RecordAccessDenied 

func RecordAccessDenied(ctx context.Context, db *sql.DB, runID string, key, prefix string) error

RecordAccessDenied records an access denied event for a key or prefix.

func RecordPartialRun 

func RecordPartialRun(ctx context.Context, db *sql.DB, runID string, reason string) error

RecordPartialRun records a partial run event and updates run status.

func RecordRunEvent 

func RecordRunEvent(ctx context.Context, db *sql.DB, event RunEvent) error

RecordRunEvent records an event for an IndexRun.

func RecordScopeViolation 

func RecordScopeViolation(ctx context.Context, db *sql.DB, runID string, key string, expectedPrefix string) error

RecordScopeViolation records an event when the index builder encounters an object key outside the configured base prefix. This indicates a bug or misconfiguration and should cause the run to be treated as partial.

func RecordThrottling 

func RecordThrottling(ctx context.Context, db *sql.DB, runID, prefix string) error

RecordThrottling records a rate-limiting event for a prefix.

func RejectLiveSQLiteTransactionSidecars 

func RejectLiveSQLiteTransactionSidecars(dbPath string) error

RejectLiveSQLiteTransactionSidecars fails only when live SQLite sidecar names remain (WAL, SHM, rollback/master/statement journals). Retained `.gonimbus-sqlite-quarantine-*` captures are not live hot state: content was destroyed via the bound descriptor and directory-entry removal is deferred to receipt-backed recovery. This helper never deletes residue. Canonical readers and mutable writers that must refuse unrecovered residue use RejectSQLiteTransactionSidecars (all quarantine-prefix names) instead.

func RejectSQLiteTransactionSidecars 

func RejectSQLiteTransactionSidecars(dbPath string) error

RejectSQLiteTransactionSidecars fails when any transaction artifact is present, including discoverable quarantine residue. Ordinary open and registration use this fail-closed gate.

func RestoreDeletedObjects 

func RestoreDeletedObjects(ctx context.Context, db *sql.DB, indexSetID string, deletedAfter time.Time) (int64, error)

RestoreDeletedObjects restores soft-deleted objects by clearing deleted_at.

This can be used to undo soft-deletes, for example if a run was marked as partial and we want to restore the previous state.

func SQLiteTransactionSidecars 

func SQLiteTransactionSidecars(dbPath string) ([]string, error)

SQLiteTransactionSidecars returns transaction artifacts associated with an index database. Any returned entry means an immutable base-file snapshot is unsafe because committed or in-flight state may live outside index.db.

This function is read-only inventory. Quarantine-prefix names are reported as blocking residue; it never deletes, truncates, or reclaims directory entries. Directory-entry removal requires a separately authorized recovery transaction that validates a durable receipt or exact binding — never prefix possession, emptiness, or whole-set authority alone.

func UpdateIndexRunStatus 

func UpdateIndexRunStatus(ctx context.Context, db *sql.DB, runID string, status RunStatus, snapshotAt *time.Time) error

UpdateIndexRunStatus updates the status and end time of an IndexRun.

func UpdateIndexRunStatusWithEvents 

func UpdateIndexRunStatusWithEvents(ctx context.Context, db *sql.DB, runID string, status RunStatus, snapshotAt *time.Time, events []RunEvent) error

UpdateIndexRunStatusWithEvents atomically updates an index run's terminal status and records its associated lifecycle/audit events.

func UpdateIndexRunStatusWithEventsAndSoftDelete 

func UpdateIndexRunStatusWithEventsAndSoftDelete(ctx context.Context, db *sql.DB, runID string, status RunStatus, snapshotAt *time.Time, events []RunEvent, softDelete *IndexRunSoftDelete) (int64, error)

UpdateIndexRunStatusWithEventsAndSoftDelete atomically updates status, records lifecycle/audit events, and optionally marks objects not seen in the run as deleted. It returns the number of soft-deleted objects.

func UpsertObject 

func UpsertObject(ctx context.Context, db *sql.DB, obj ObjectRow) error

UpsertObject inserts or updates an object in objects_current.

If the object exists, updates last_seen_run_id, last_seen_at, and clears deleted_at. If the object doesn't exist, inserts a new row.

func ValidateCurrentSchemaReadOnly 

func ValidateCurrentSchemaReadOnly(ctx context.Context, db *sql.DB) error

ValidateCurrentSchemaReadOnly verifies that db has exactly the schema this binary understands. It never upgrades older schemas; callers must retain those artifacts until an explicit migration operation is authorized.

Types

type BuildParams 

type BuildParams struct {
	SourceType         string
	PathDateExtraction *PathDateExtraction
	SchemaVersion      int
	GonimbusVersion    string

	// Match parameters that affect index contents.
	Includes      []string
	Excludes      []string
	IncludeHidden bool
	FiltersHash   string // Hash of filters (size, modified, key_regex)
	ScopeHash     string // Hash of build.scope config
}

BuildParams captures parameters that affect index contents.

Any change in index_build_hash requires a new IndexSet to ensure index consistency and proper query semantics.

type CanonicalObjectGroup 

type CanonicalObjectGroup struct {
	ETag       string
	Canonical  QueryResult
	Alternates []QueryResult
}

CanonicalObjectGroup holds one ETag group after canonical selection.

type CanonicalOutputRecord 

type CanonicalOutputRecord struct {
	Group       *CanonicalObjectGroup
	Passthrough *QueryResult
}

CanonicalOutputRecord is one output record in canonical-by-ETag mode. Rows with empty ETag pass through as standard object records.

type CanonicalQueryStats 

type CanonicalQueryStats struct {
	QueryStats
	CanonicalGroups int
	PassthroughRows int
	TotalRecords    int
	// AvailableRecords is the logical record count before an output limit.
	AvailableRecords int
	// AvailablePassthroughRows counts matching empty-ETag rows before an output
	// limit. Receipt framing uses it to fail closed even when a limit would have
	// hidden those rows from the returned slice.
	AvailablePassthroughRows int
	Truncated                bool
}

CanonicalQueryStats holds summary counts for canonical-by-ETag output.

type CanonicalTieBreak 

type CanonicalTieBreak string

CanonicalTieBreak selects the canonical row for an ETag group. 

const (
	CanonicalTieBreakMinKey      CanonicalTieBreak = "min-key"
	CanonicalTieBreakMinModified CanonicalTieBreak = "min-modified"
	CanonicalTieBreakMaxModified CanonicalTieBreak = "max-modified"
)

type Config 

type Config struct {
	// Path is a local filesystem path to the index database.
	// If set, it is converted into a libsql-compatible DSN (file:<path>).
	Path string

	// URL is a libsql/Turso URL, e.g. libsql://your-db.turso.io.
	URL string

	// AuthToken is appended to URL-based DSNs as authToken=... when not already present.
	AuthToken string

	// SynchronousFull requests PRAGMA synchronous=FULL on the local connection,
	// set and verified rather than inherited from the driver default. It is used
	// by durability-authoritative stores (the reflow checkpoint store) whose
	// terminal state is the resume authority; leave it false for rebuildable
	// stores (index builds) that tolerate the driver default under WAL.
	SynchronousFull bool
}

type DeletedObjectStats 

type DeletedObjectStats struct {
	TotalDeleted  int64
	OldestDeleted *time.Time
	NewestDeleted *time.Time
}

DeletedObjectStats returns statistics about deleted objects in an index set.

func GetDeletedObjectStats 

func GetDeletedObjectStats(ctx context.Context, db *sql.DB, indexSetID string) (*DeletedObjectStats, error)

GetDeletedObjectStats retrieves statistics about soft-deleted objects.

type EventCategory 

type EventCategory string

EventCategory groups events by type.

type EventType 

type EventType string

EventType identifies specific event types. 

const (
	EventTypePrefixSkipped  EventType = "prefix_skipped"
	EventTypeAccessDenied   EventType = "access_denied"
	EventTypeRateLimited    EventType = "rate_limited"
	EventTypePartial        EventType = "partial_run"
	EventTypeUnknownError   EventType = "unknown_error"
	EventTypeObjectAdded    EventType = "object_added"
	EventTypeObjectDeleted  EventType = "object_deleted"
	EventTypePrefixListed   EventType = "prefix_listed"
	EventTypeRunStarted     EventType = "run_started"
	EventTypeRunCompleted   EventType = "run_completed"
	EventTypeScopeViolation EventType = "scope_violation"
)

type GCParams 

type GCParams struct {
	// MaxAge removes index sets older than this duration.
	// Zero means no age-based cleanup.
	MaxAge time.Duration

	// KeepLast keeps at least this many index sets per base URI.
	// Zero means no minimum.
	KeepLast int

	// DryRun if true, reports what would be deleted without deleting.
	DryRun bool
}

GCParams specifies parameters for garbage collection.

type GCResult 

type GCResult struct {
	// IndexSetsRemoved is the count of index sets deleted.
	IndexSetsRemoved int

	// ObjectsRemoved is the count of objects deleted.
	ObjectsRemoved int64

	// BytesFreed is an estimate of bytes freed based on active object sizes.
	// Note: this reflects indexed object sizes, not actual DB file shrinkage.
	BytesFreed int64

	// Candidates lists index sets that would be/were removed.
	Candidates []IndexListEntry
}

GCResult contains the results of a garbage collection run.

func GarbageCollect 

func GarbageCollect(ctx context.Context, db *sql.DB, params GCParams) (*GCResult, error)

GarbageCollect removes old index sets based on the given parameters.

The cleanup strategy: 1. If MaxAge > 0: remove index sets created more than MaxAge ago 2. If KeepLast > 0: keep at least KeepLast index sets per base URI

KeepLast takes precedence: even if an index set is older than MaxAge, it won't be removed if it's within the KeepLast threshold for its base URI.

type HeadEnrichmentCandidate 

type HeadEnrichmentCandidate struct {
	RelKey         string
	SizeBytes      int64
	StorageClass   *string
	HeadEnrichedAt *time.Time
}

HeadEnrichmentCandidate is an object row selected for HEAD enrichment.

type HeadEnrichmentUpdate 

type HeadEnrichmentUpdate struct {
	IndexSetID     string
	RelKey         string
	ArchiveStatus  *string
	RestoreState   *string
	RestoreExpiry  *time.Time
	ContentType    *string
	HeadEnrichedAt time.Time
}

HeadEnrichmentUpdate contains HEAD-derived fields that may be written by enrichment. It intentionally excludes LIST-derived fields such as storage_class, size, ETag, and last_modified.

type IndexListEntry 

type IndexListEntry struct {
	IndexSetID       string
	BaseURI          string
	Provider         string
	CreatedAt        time.Time
	ObjectCount      int64
	TotalSizeBytes   int64
	RunCount         int
	LatestRunID      string
	LatestRunAt      *time.Time
	LatestStatus     string
	LatestSourceType string
}

IndexListEntry provides summary info for the index list command.

func ListIndexSetsWithStats 

func ListIndexSetsWithStats(ctx context.Context, db *sql.DB) ([]IndexListEntry, error)

ListIndexSetsWithStats returns all IndexSets with summary statistics.

type IndexRun 

type IndexRun struct {
	RunID            string
	IndexSetID       string
	StartedAt        time.Time
	EndedAt          *time.Time
	AcquiredAt       time.Time
	SourceType       string
	SourceSnapshotAt *time.Time
	Status           RunStatus
}

IndexRun represents a single index build execution.

Each IndexRun is scoped to an IndexSet and represents a discrete snapshot of the source at a specific point in time.

func CreateIndexRun 

func CreateIndexRun(ctx context.Context, db *sql.DB, indexSetID string, sourceType string) (*IndexRun, error)

CreateIndexRun creates a new IndexRun in running status.

func GetIndexRun 

func GetIndexRun(ctx context.Context, db *sql.DB, runID string) (*IndexRun, error)

GetIndexRun retrieves an IndexRun by ID.

func ListIndexRuns 

func ListIndexRuns(ctx context.Context, db *sql.DB, indexSetID string) ([]IndexRun, error)

ListIndexRuns lists all runs for an IndexSet.

type IndexRunSoftDelete 

type IndexRunSoftDelete struct {
	IndexSetID   string
	RunID        string
	RunStartedAt time.Time
}

IndexRunSoftDelete describes the successful-run soft-delete mutation that can be committed with an index-run status transition.

type IndexSet 

type IndexSet struct {
	IndexSetID      string
	BaseURI         string
	Provider        string
	StorageProvider string
	CloudProvider   string
	RegionKind      string
	Region          string
	Endpoint        string
	EndpointHost    string
	IndexBuildHash  string
	CreatedAt       time.Time
}

IndexSet represents a unique index identity.

An IndexSet is identified by (base_uri, provider_identity, index_build_hash). Different build params (filters, path-date extraction, etc.) produce different IndexSets.

func FindOrCreateIndexSet 

func FindOrCreateIndexSet(ctx context.Context, db *sql.DB, params IndexSetParams) (*IndexSet, bool, error)

FindOrCreateIndexSet finds an existing IndexSet or creates a new one.

Returns the IndexSet and whether it was created (true) or found existing (false).

func GetIndexSet 

func GetIndexSet(ctx context.Context, db *sql.DB, indexSetID string) (*IndexSet, error)

GetIndexSet retrieves an IndexSet by ID.

func GetIndexSetByBaseURI 

func GetIndexSetByBaseURI(ctx context.Context, db *sql.DB, baseURI string) (*IndexSet, error)

GetIndexSetByBaseURI finds an IndexSet by its base_uri.

If multiple index sets exist for the same base_uri (different build params), returns the most recently created one. Returns nil if no matching index set is found.

func ListIndexSets 

func ListIndexSets(ctx context.Context, db *sql.DB, baseURI string) ([]IndexSet, error)

ListIndexSets lists all IndexSets, optionally filtered by base URI.

type IndexSetIdentityBuild 

type IndexSetIdentityBuild struct {
	SourceType      string   `json:"source_type"`
	SchemaVersion   int      `json:"schema_version"`
	GonimbusVersion string   `json:"gonimbus_version,omitempty"`
	Includes        []string `json:"includes"`
	Excludes        []string `json:"excludes,omitempty"`
	IncludeHidden   bool     `json:"include_hidden"`
	FiltersHash     string   `json:"filters_hash,omitempty"`
	ScopeHash       string   `json:"scope_hash,omitempty"`
}

IndexSetIdentityBuild captures build parameters that affect index identity.

type IndexSetIdentityPathDate 

type IndexSetIdentityPathDate struct {
	Method       string `json:"method"`
	Regex        string `json:"regex,omitempty"`
	SegmentIndex int    `json:"segment_index,omitempty"`
}

IndexSetIdentityPathDate captures path date extraction identity fields.

type IndexSetIdentityPayload 

type IndexSetIdentityPayload struct {
	BaseURI         string                    `json:"base_uri"`
	Provider        string                    `json:"provider"`
	StorageProvider string                    `json:"storage_provider,omitempty"`
	CloudProvider   string                    `json:"cloud_provider,omitempty"`
	RegionKind      string                    `json:"region_kind,omitempty"`
	Region          string                    `json:"region,omitempty"`
	EndpointHost    string                    `json:"endpoint_host,omitempty"`
	Build           IndexSetIdentityBuild     `json:"build"`
	PathDate        *IndexSetIdentityPathDate `json:"path_date,omitempty"`
}

IndexSetIdentityPayload captures the canonical identity for hashing.

type IndexSetIdentityResult 

type IndexSetIdentityResult struct {
	IndexSetID      string
	DirName         string
	CanonicalJSON   string
	CanonicalSHA256 string
}

IndexSetIdentityResult contains derived identity fields.

func ComputeIndexSetID 

func ComputeIndexSetID(params IndexSetParams) (*IndexSetIdentityResult, error)

ComputeIndexSetID computes canonical identity details for an IndexSet.

type IndexSetParams 

type IndexSetParams struct {
	BaseURI         string
	Provider        string
	StorageProvider string
	CloudProvider   string
	RegionKind      string
	Region          string
	Endpoint        string
	EndpointHost    string // Explicit endpoint host; if empty and Endpoint is set, derived from Endpoint
	BuildParams     BuildParams
}

IndexSetParams contains parameters for creating or finding an IndexSet.

type IndexSetSummary 

type IndexSetSummary struct {
	IndexSetID string
	BaseURI    string
	Provider   string
	CreatedAt  time.Time

	// Object statistics (from objects_current)
	TotalObjects   int64
	ActiveObjects  int64
	DeletedObjects int64
	TotalSizeBytes int64

	// Run statistics
	TotalRuns           int
	SuccessfulRuns      int
	PartialRuns         int
	FailedRuns          int
	FailedResumableRuns int
	LatestRun           *IndexRun
}

IndexSetSummary provides aggregate statistics for an IndexSet.

func GetIndexSetSummary 

func GetIndexSetSummary(ctx context.Context, db *sql.DB, indexSetID string) (*IndexSetSummary, error)

GetIndexSetSummary retrieves aggregate statistics for an IndexSet.

type ObjectDeltaBaseline 

type ObjectDeltaBaseline struct {
	IndexSetID        string
	BaselineRunID     string
	BaselineStartedAt time.Time
	CreatedAt         time.Time
}

ObjectDeltaBaseline records the legacy migration boundary for an index set.

func GetObjectDeltaBaseline 

func GetObjectDeltaBaseline(ctx context.Context, db *sql.DB, indexSetID string) (*ObjectDeltaBaseline, error)

GetObjectDeltaBaseline returns the legacy migration baseline for an index set. A nil baseline means the index set was created with delta tracking available.

type ObjectRow 

type ObjectRow struct {
	IndexSetID       string
	RelKey           string
	SizeBytes        int64
	LastModified     *time.Time
	ETag             string
	StorageClass     *string
	ArchiveStatus    *string
	RestoreState     *string
	RestoreExpiry    *time.Time
	ContentType      *string
	HeadEnrichedAt   *time.Time
	FirstSeenRunID   string
	FirstSeenAt      time.Time
	LastChangedRunID string
	LastChangedAt    time.Time
	LastSeenRunID    string
	LastSeenAt       time.Time
	DeletedAt        *time.Time
}

ObjectRow represents a row in the objects_current table.

func GetObject 

func GetObject(ctx context.Context, db *sql.DB, indexSetID, relKey string) (*ObjectRow, error)

GetObject retrieves a single object by index set and relative key.

func ListObjectsForRun 

func ListObjectsForRun(ctx context.Context, db *sql.DB, indexSetID, runID string) ([]ObjectRow, error)

ListObjectsForRun returns active objects whose latest observation belongs to the requested completed run. Results are ordered by rel_key for deterministic post-pass consumers.

type PathDateExtraction 

type PathDateExtraction struct {
	Method       string // "regex" or "segment"
	Regex        string // For regex method
	SegmentIndex int    // For segment method (0-indexed)
}

PathDateExtraction configures extracting dates from object key paths.

type PrefixStatRow 

type PrefixStatRow struct {
	IndexSetID      string
	RunID           string
	Prefix          string
	Depth           int
	ObjectsDirect   int64
	BytesDirect     int64
	CommonPrefixes  int64
	Truncated       bool
	TruncatedReason string
}

PrefixStatRow represents a row in the prefix_stats table.

func GetLatestPrefixStats 

func GetLatestPrefixStats(ctx context.Context, db *sql.DB, indexSetID string) ([]PrefixStatRow, error)

GetLatestPrefixStats retrieves prefix stats from the most recent successful run.

func GetPrefixStats 

func GetPrefixStats(ctx context.Context, db *sql.DB, indexSetID, runID string) ([]PrefixStatRow, error)

GetPrefixStats retrieves prefix stats for a run.

type QueryParams 

type QueryParams struct {
	// IndexSetID limits query to a specific index set.
	// Required.
	IndexSetID string

	// Pattern is a doublestar glob pattern to match against rel_key.
	// Optional. If empty, matches all keys.
	Pattern string

	// KeyRegex is a regex pattern to match against rel_key.
	// Applied after glob pattern (if both specified).
	// Optional.
	KeyRegex string

	// MinSize filters objects >= this size in bytes.
	// Optional. Zero means no minimum.
	MinSize int64

	// MaxSize filters objects <= this size in bytes.
	// Optional. Zero means no maximum.
	MaxSize int64

	// ModifiedAfter filters objects modified after this time.
	// Optional. Zero time means no lower bound.
	ModifiedAfter time.Time

	// ModifiedBefore filters objects modified before this time.
	// Optional. Zero time means no upper bound.
	ModifiedBefore time.Time

	// StorageClasses filters objects by exact, provider-raw storage_class values.
	// Empty means no storage class filter. Values match non-null rows only.
	StorageClasses []string

	// EnrichedAfter filters objects whose head_enriched_at is at or after this time.
	// Optional. Zero time means no enrichment timestamp lower bound.
	EnrichedAfter time.Time

	// IncludeDeleted includes soft-deleted objects in results.
	// Default: false (only non-deleted objects).
	IncludeDeleted bool

	// SinceRun filters to current rows first seen or meaningfully changed after
	// a validated run boundary. This is a forward delta over latest state, not
	// historical reconstruction.
	SinceRun *SinceRunFilter

	// Limit caps the number of results returned.
	// Optional. Zero means no limit.
	Limit int

	// CanonicalTieBreak selects the canonical row for ETag groups.
	// Optional. Empty means CanonicalTieBreakMinKey.
	CanonicalTieBreak CanonicalTieBreak
}

QueryParams specifies filters for querying indexed objects.

type QueryResult 

type QueryResult struct {
	RelKey           string
	SizeBytes        int64
	LastModified     *time.Time
	ETag             string
	StorageClass     *string
	ArchiveStatus    *string
	RestoreState     *string
	RestoreExpiry    *time.Time
	ContentType      *string
	HeadEnrichedAt   *time.Time
	FirstSeenRunID   string
	FirstSeenAt      *time.Time
	LastChangedRunID string
	LastChangedAt    *time.Time
	ChangeKind       string
	DeletedAt        *time.Time
}

QueryResult holds a single object from the query.

type QueryStats 

type QueryStats struct {
	// TimestampParseErrors is the count of rows with unparseable timestamps.
	// These rows are included in results but with nil timestamp fields.
	TimestampParseErrors int64
	// Examined is the number of rows actually evaluated by the reader.
	Examined int64
	// Matched is the number of examined rows that satisfied all filters.
	Matched int64
	// SegmentsWalked is the number of durable segments whose row stream was
	// entered. SQLite readers leave segment counters at zero.
	SegmentsWalked int
	// SegmentsVerified is the number of durable segment digests verified before
	// any row from the segment was exposed.
	SegmentsVerified int
	// SegmentsManifestPruned is the number of durable segments skipped using
	// bounds from the already-verified manifest.
	SegmentsManifestPruned int
}

QueryStats holds statistics about the query execution.

type RunEvent 

type RunEvent struct {
	EventID       string
	RunID         string
	OccurredAt    time.Time
	EventType     string
	EventCategory string
	Detail        *string
	Key           *string
	Prefix        *string
	ErrorCode     *string
}

RunEvent represents an event during an IndexRun.

Events provide structured diagnostic information, especially for partial runs.

func ListRunEvents 

func ListRunEvents(ctx context.Context, db *sql.DB, runID string, category *EventCategory) ([]RunEvent, error)

ListRunEvents retrieves events for a run.

type RunStatus 

type RunStatus string

RunStatus represents the status of an IndexRun.

type SinceRunFilter 

type SinceRunFilter struct {
	RunID                    string
	StartedAt                time.Time
	VerifiedDescendantRunIDs []string
	SameRun                  bool
}

SinceRunFilter is a resolved query boundary for --since-run.

func ResolveSinceRunFilter 

func ResolveSinceRunFilter(ctx context.Context, db *sql.DB, indexSetID, runID string) (*SinceRunFilter, error)

ResolveSinceRunFilter validates a --since-run boundary for an index set and returns the timestamp-resolved filter used by all query paths.

type TopLevelObjectSummary 

type TopLevelObjectSummary struct {
	Prefix         string
	ObjectCount    int64
	TotalSizeBytes int64
}

TopLevelObjectSummary provides object distribution for a single run.

func GetTopLevelObjectSummaryForRun 

func GetTopLevelObjectSummaryForRun(ctx context.Context, db *sql.DB, indexSetID, runID string) ([]TopLevelObjectSummary, error)

GetTopLevelObjectSummaryForRun returns active objects seen by a specific run, grouped by the first relative-key segment. Root-level objects use an empty prefix.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.