reconcile

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 25, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package reconcile diffs a schema's S3 parquet objects against its manifest (issue #203). It reports orphans — live objects the manifest does not list, classified by filename shape — and dangling entries — manifest paths whose object is gone. Optional repair appends delta-shaped orphans back to the manifest when the coverage + Postgres-liveness guard proves they are the #197 flush failure mode (data exists nowhere else), and promotes complete init-shaped base orphan sets into the base tier once coverage, eviction safety, and the no-resurrection proof all hold (#292); provable compaction leftovers are instead classified for GC, which also deletes merged-base and _tmp/ orphans left behind by compaction rewrites (#188).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type DuckStatsReader

type DuckStatsReader struct {
	DB     *sql.DB
	Bucket string
}

DuckStatsReader recomputes parquet stats over a DuckDB session with httpfs and S3 credentials already configured (cdc.NewDuckExporter's DB). Keep the pool at one connection: the exporter's S3 SETs are session-scoped and only reach the connection they ran on (#285).

func (*DuckStatsReader) FileColumns

func (d *DuckStatsReader) FileColumns(ctx context.Context, key string) (map[string]string, error)

FileColumns probes the parquet footer for the #256 column stamp, exactly as cdc-init's initStampColumns does for its own entries.

func (*DuckStatsReader) FileStats

func (d *DuckStatsReader) FileStats(ctx context.Context, key string) (compaction.MergeStats, error)

func (*DuckStatsReader) UncoveredRows

func (d *DuckStatsReader) UncoveredRows(ctx context.Context, key string, listedKeys []string) ([]compaction.UncoveredRow, error)

type GCStateStore

type GCStateStore interface {
	Load(ctx context.Context, schemaID int16) (map[string]int64, string, error)
	Save(ctx context.Context, schemaID int16, state map[string]int64, etag string) (string, error)
}

GCStateStore persists per-schema first-unlisted sighting timestamps (key -> unix ms) with optimistic concurrency. GC deletes an orphan only after it has been observed unlisted for longer than the grace period — LastModified alone cannot express "unlisted duration" (#188 follow-up: an old source freshly spliced out by the compactor would otherwise be deleted inside the in-flight-reader window).

type LiveRowChecker

type LiveRowChecker interface {
	MissingLiveRows(ctx context.Context, schemaID int16, rowIDs []string) ([]string, error)
	LiveRowCount(ctx context.Context, schemaID int16) (int64, error)
}

LiveRowChecker reports Postgres entity-store liveness. MissingLiveRows returns which of the given row ids are NOT live (a row absent from every manifest-listed parquet AND missing from Postgres was deleted — its tombstone won a compaction merge and was dropped, so re-appending would resurrect it; see classifyDeltaOrphan in repair.go). LiveRowCount returns the number of live rows for a schema — the right-hand side of init promotion's coverage identity (#292): the orphan set may replace the base tier only if it provably covers every one of these rows.

type Locker

type Locker interface {
	TryLock(ctx context.Context, schemaID int16) (locked bool, unlock func(), err error)
}

Locker serializes reconciliation against the live flusher per schema. unlock is non-nil exactly when locked is true.

type ManifestGCStateStore

type ManifestGCStateStore struct {
	Store    manifest.Store
	Resolver manifest.PathResolver
}

ManifestGCStateStore persists GC sighting state next to the schema's manifest (at "<manifest path>.gc-state") through the same etag-aware store, inheriting its optimistic concurrency incl. the If-None-Match create guard. The state object lives under the manifest prefix, so it never pollutes the data-prefix listing the reconciler classifies.

func (*ManifestGCStateStore) Load

func (s *ManifestGCStateStore) Load(ctx context.Context, schemaID int16) (map[string]int64, string, error)

func (*ManifestGCStateStore) Save

func (s *ManifestGCStateStore) Save(ctx context.Context, schemaID int16, state map[string]int64, etag string) (string, error)

type ManifestStore

type ManifestStore interface {
	Load(ctx context.Context, schemaID int16) (*manifest.Manifest, string, error)
	Save(ctx context.Context, schemaID int16, m *manifest.Manifest, etag string) (string, error)
}

ManifestStore loads and saves one schema's manifest with optimistic concurrency. Load must have LoadOrCreate semantics: a schema without a manifest reconciles as empty instead of erroring.

type ObjectDeleter

type ObjectDeleter interface {
	DeleteObject(ctx context.Context, key string) error
}

ObjectDeleter deletes a single object by bucket-relative key.

type ObjectInfo

type ObjectInfo struct {
	Key          string
	Size         int64
	LastModified time.Time
}

ObjectInfo is one listed S3 object with the metadata reconcile needs: Size feeds repaired FileEntry.SizeBytes, LastModified feeds the GC grace check.

type ObjectLister

type ObjectLister interface {
	ListObjects(ctx context.Context, prefix string) ([]ObjectInfo, error)
}

ObjectLister lists every object under a prefix. Implementations own pagination — callers always receive the complete listing.

type ObjectReader

type ObjectReader interface {
	GetObject(ctx context.Context, params *s3.GetObjectInput, optFns ...func(*s3.Options)) (*s3.GetObjectOutput, error)
}

ObjectReader streams an object's raw bytes — the surface the #347 checksum scrub re-hashes through cdc.ObjectSHA256. It is declared here rather than reused from cdc so the reconciler names its own dependency; *s3.Client satisfies it, as does cdc.S3GetClient's shape.

type Options

type Options struct {
	Repair  bool          // append delta-shaped orphans back to the manifest
	GC      bool          // delete base-shaped and _tmp orphans past the grace period
	GCGrace time.Duration // minimum object age before GC may delete it
	// VerifyStamps compares every listed entry's #256 column stamp against
	// the object footer; divergence is the byte-truth breach the read-path
	// stamp short-circuit cannot see.
	VerifyStamps bool
	// VerifyChecksums re-hashes every stamped entry's object and compares the
	// digest with its #347 content stamp — the byte-integrity layer under
	// VerifyStamps' shape check, and the only offline detector of silent
	// parquet mis-decode corruption. One full GET per stamped entry.
	VerifyChecksums bool
	MaxETagRetries  int // manifest save retries on optimistic-concurrency conflict
	// AllowEmptyManifestSchemas waives the #463 empty-manifest GC guard for
	// exactly these schemas. Schema-explicit by design: one mis-pointed
	// manifest template resolves EVERY schema's manifest empty, so a global
	// override would wave the whole fleet through the very failure the
	// guard exists to stop.
	AllowEmptyManifestSchemas []int16
}

Options selects the reconcile actions. The zero value is a read-only report.

type OrphanClass

type OrphanClass int

OrphanClass is the filename-shape class of an unlisted parquet object. The class decides the recovery direction: delta orphans may carry data that exists nowhere else and are candidates for repair (guarded by row coverage and Postgres liveness), while merged-base, _tmp, and init-shaped base orphans are all eligible for GC. Init-shaped orphans became GC candidates in #290: cdc-init now holds the same per-schema advisory lock reconcile takes, so under that lock an init-shaped orphan is provably not from an in-flight init — it is either a failed manifest publish or a file superseded by a later init run. Since #292 an init-shaped set that --repair can prove complete is promoted into the base tier instead of collected. Unknown shapes are reported and never touched.

const (
	ClassDelta      OrphanClass = iota
	ClassBaseInit               // {minRowID}_{maxRowID}.parquet (cdc-init export)
	ClassBaseMerged             // base-{uuid}.parquet (compaction rewrite)
	ClassTmp
	ClassUnknown
)

type PGAdvisoryLocker

type PGAdvisoryLocker struct {
	DB *sql.DB
}

PGAdvisoryLocker takes the flusher's per-schema advisory lock (pg_try_advisory_lock(schemaID, schemaID)). It delegates to cdc.TrySchemaLock, the single source of truth for the pinned-connection acquire/release dance: on a pool, acquire and release could land on different connections, and a session-scoped lock released on the wrong session silently fails, so the lock pins one physical connection for its lifetime and closes it to unlock.

func (*PGAdvisoryLocker) TryLock

func (l *PGAdvisoryLocker) TryLock(ctx context.Context, schemaID int16) (bool, func(), error)

type PGLiveRows

type PGLiveRows struct {
	DB    *sql.DB
	Table string // entity_main table name
}

PGLiveRows implements LiveRowChecker over the entity main table. Liveness mirrors cdc-init's export filter (init.go): the row exists under the (schema_id, row_id) primary key AND ltbase_deleted_at IS NULL — writes can soft-delete by setting the column, and treating such rows as live would let repair re-append data whose tombstone compaction already dropped.

func (*PGLiveRows) LiveRowCount

func (p *PGLiveRows) LiveRowCount(ctx context.Context, schemaID int16) (int64, error)

LiveRowCount counts live rows with the same liveness predicate as MissingLiveRows, mirroring cdc-init's export filter: the counting identity in init promotion (#292) is only sound if numerator and denominator use one definition of "live".

func (*PGLiveRows) MissingLiveRows

func (p *PGLiveRows) MissingLiveRows(ctx context.Context, schemaID int16, rowIDs []string) ([]string, error)

type Reconciler

type Reconciler struct {
	Lister     ObjectLister
	Deleter    ObjectDeleter
	Manifests  ManifestStore
	Objects    ObjectReader   // required by Opts.VerifyChecksums; nil also leaves repaired and promoted entries unstamped
	Stats      StatsReader    // may be nil unless Opts.Repair or Opts.VerifyStamps
	LiveRows   LiveRowChecker // may be nil unless Opts.Repair
	Locker     Locker
	Schemas    SchemaEnumerator
	GCStates   GCStateStore // may be nil unless Opts.GC
	Now        func() time.Time
	Bucket     string
	DataPrefix string
	Logger     *zap.Logger
	Opts       Options
}

Reconciler diffs S3 parquet objects against per-schema manifests and optionally repairs (append delta orphans) or garbage-collects (delete compaction leftovers). See the package comment for the recovery model.

func (*Reconciler) Run

func (r *Reconciler) Run(ctx context.Context) (Report, error)

Run reconciles every enumerated schema. Per-schema failures are recorded in the report and do not abort the run; the returned error is reserved for failures that prevent reconciling anything at all.

type RegistrySchemaEnumerator

type RegistrySchemaEnumerator struct {
	DB             *sql.DB
	Table          string
	SchemaIDFilter int // 0 = all schemas
}

RegistrySchemaEnumerator enumerates schema IDs from the schema registry table, mirroring cdc-init's getSchemaIDsToInit: every registered schema reconciles, not just those with pending CDC work.

func (*RegistrySchemaEnumerator) SchemaIDs

func (e *RegistrySchemaEnumerator) SchemaIDs(ctx context.Context) ([]int16, error)

type Report

type Report struct {
	Schemas []SchemaReport
}

Report is a full reconcile run across schemas.

func (Report) HasResidualDiscrepancies

func (r Report) HasResidualDiscrepancies() bool

HasResidualDiscrepancies reports whether anything actionable is left after repair and GC: orphans not repaired, promoted, or deleted, dangling entries, skipped schemas, unknown shapes, stamp divergences (--verify-stamps), checksum divergences (--verify-checksums), or per-schema failures. Unverifiable paths are informational — they cannot be proven inconsistent from this run, and so is the unstamped-entry count, which reports missing coverage rather than a discrepancy.

func (Report) Render

func (r Report) Render(w io.Writer)

Render writes the human-readable report: exact keys per class per schema.

type ResolverManifestStore

type ResolverManifestStore struct {
	Store    manifest.Store
	Resolver manifest.PathResolver
}

ResolverManifestStore implements ManifestStore over a manifest.Store and PathResolver. Load has LoadOrCreate semantics — a schema whose manifest does not exist yet reconciles as empty instead of erroring (the compaction ManifestProvider errors on a missing manifest, which is wrong for reconciliation: an empty manifest with live objects is exactly the all-orphans case the tool must report).

func (*ResolverManifestStore) Load

func (*ResolverManifestStore) Save

func (s *ResolverManifestStore) Save(ctx context.Context, schemaID int16, m *manifest.Manifest, etag string) (string, error)

type S3ObjectStore

type S3ObjectStore struct {
	Client s3ObjectAPI
	Bucket string
}

S3ObjectStore implements ObjectLister and ObjectDeleter over an *s3.Client (or any compatible API subset).

func (*S3ObjectStore) DeleteObject

func (s *S3ObjectStore) DeleteObject(ctx context.Context, key string) error

DeleteObject deletes one object by key.

func (*S3ObjectStore) ListObjects

func (s *S3ObjectStore) ListObjects(ctx context.Context, prefix string) ([]ObjectInfo, error)

ListObjects lists all objects under prefix, following continuation tokens until the listing is exhausted.

type SchemaEnumerator

type SchemaEnumerator interface {
	SchemaIDs(ctx context.Context) ([]int16, error)
}

SchemaEnumerator yields the schema IDs to reconcile.

type SchemaReport

type SchemaReport struct {
	SchemaID     int16
	Skipped      bool // advisory lock not acquired; nothing inspected
	DeltaOrphans []string
	BaseOrphans  []string
	TmpOrphans   []string
	Unknown      []string // unrecognized shapes; reported, never repaired or deleted
	Dangling     []string // manifest entries with no live object; removal stays manual
	Unverifiable []string // manifest paths this listing cannot prove absent
	// ObjectsSeen counts this schema's classified parquet objects in
	// storage; ManifestEntries counts its manifest's file entries as
	// loaded (#463). Together they distinguish "the manifest genuinely
	// lists nothing" from "the manifest failed to resolve" — N objects
	// against 0 entries is the mis-pointed-template signature --gc refuses.
	ObjectsSeen     int
	ManifestEntries int
	Repaired        []string // delta orphans appended to the manifest (--repair)
	// DeltaLeftovers are delta orphans the repair guard classified as
	// compaction leftovers (no uncovered rows, or every uncovered row
	// deleted in Postgres): never appended, GC-eligible under --gc.
	DeltaLeftovers []string
	Deleted        []string // leftover/merged-base/tmp orphans removed (--gc)
	// PromotedBase are init-shaped base orphans promoted into the manifest
	// base tier after the coverage + eviction-safety proof (--repair, #292).
	PromotedBase []string
	// InitPromotionRefusal explains why an init-shaped orphan set was NOT
	// promoted. Promotion is all-or-nothing over the set, so one reason
	// covers every file; empty when promotion succeeded or never ran. The
	// refused files stay ordinary GC candidates.
	InitPromotionRefusal string
	// StampDivergences are listed entries whose stamp no longer matches the
	// object footer (--verify-stamps); each is a byte-truth breach requiring
	// operator action — restamp via rewrite, or investigate the overwrite.
	StampDivergences []string
	// ChecksumDivergences are listed entries whose object bytes no longer hash
	// to the entry's #347 content stamp (--verify-checksums): silent
	// corruption, actionable exactly like a stamp divergence.
	ChecksumDivergences []string
	// SkippedUnstamped counts entries --verify-checksums could not cover
	// because they carry no checksum — either legacy (never backfilled) or a
	// best-effort write-side hash that failed. It is coverage observability,
	// not a discrepancy: it never affects the exit code, but a "clean"
	// verdict over mostly-unstamped entries means far less than a clean
	// verdict over stamped ones.
	SkippedUnstamped int
	Err              error // per-schema failure; other schemas still reconcile
}

SchemaReport is one schema's reconcile outcome. Key slices hold bucket-relative keys except Unverifiable, which preserves the manifest path verbatim (the raw path is what the operator must inspect).

func (SchemaReport) Residual

func (s SchemaReport) Residual() bool

Residual reports whether this schema still has actionable discrepancies after repair and GC.

type StatsReader

type StatsReader interface {
	FileStats(ctx context.Context, key string) (compaction.MergeStats, error)
	UncoveredRows(ctx context.Context, key string, listedKeys []string) ([]compaction.UncoveredRow, error)
	FileColumns(ctx context.Context, key string) (map[string]string, error)
}

StatsReader inspects one parquet file's contents: FileStats recomputes manifest metadata; UncoveredRows returns the rows whose newest version no listed file supersedes, with a tombstone flag (the repair guard's version-aware coverage probe) — with no listed keys it enumerates every distinct row id in the file (#292 init promotion). FileColumns probes the footer for the #256 column stamp. All take bucket-relative keys. Only consulted under --repair, plus FileColumns under --verify-stamps.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL