Documentation
¶
Overview ¶
Package reconcile diffs a schema's S3 parquet objects against its manifest (issue #203). It reports orphans — live objects the manifest does not list, classified by filename shape — and dangling entries — manifest paths whose object is gone. Optional repair appends delta-shaped orphans back to the manifest when the coverage + Postgres-liveness guard proves they are the #197 flush failure mode (data exists nowhere else), and promotes complete init-shaped base orphan sets into the base tier once coverage, eviction safety, and the no-resurrection proof all hold (#292); provable compaction leftovers are instead classified for GC, which also deletes merged-base and _tmp/ orphans left behind by compaction rewrites (#188).
Index ¶
- type DuckStatsReader
- func (d *DuckStatsReader) FileColumns(ctx context.Context, key string) (map[string]string, error)
- func (d *DuckStatsReader) FileStats(ctx context.Context, key string) (compaction.MergeStats, error)
- func (d *DuckStatsReader) UncoveredRows(ctx context.Context, key string, listedKeys []string) ([]compaction.UncoveredRow, error)
- type GCStateStore
- type LiveRowChecker
- type Locker
- type ManifestGCStateStore
- type ManifestStore
- type ObjectDeleter
- type ObjectInfo
- type ObjectLister
- type ObjectReader
- type Options
- type OrphanClass
- type PGAdvisoryLocker
- type PGLiveRows
- type Reconciler
- type RegistrySchemaEnumerator
- type Report
- type ResolverManifestStore
- type S3ObjectStore
- type SchemaEnumerator
- type SchemaReport
- type StatsReader
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DuckStatsReader ¶
DuckStatsReader recomputes parquet stats over a DuckDB session with httpfs and S3 credentials already configured (cdc.NewDuckExporter's DB). Keep the pool at one connection: the exporter's S3 SETs are session-scoped and only reach the connection they ran on (#285).
func (*DuckStatsReader) FileColumns ¶
FileColumns probes the parquet footer for the #256 column stamp, exactly as cdc-init's initStampColumns does for its own entries.
func (*DuckStatsReader) FileStats ¶
func (d *DuckStatsReader) FileStats(ctx context.Context, key string) (compaction.MergeStats, error)
func (*DuckStatsReader) UncoveredRows ¶
func (d *DuckStatsReader) UncoveredRows(ctx context.Context, key string, listedKeys []string) ([]compaction.UncoveredRow, error)
type GCStateStore ¶
type GCStateStore interface {
Load(ctx context.Context, schemaID int16) (map[string]int64, string, error)
Save(ctx context.Context, schemaID int16, state map[string]int64, etag string) (string, error)
}
GCStateStore persists per-schema first-unlisted sighting timestamps (key -> unix ms) with optimistic concurrency. GC deletes an orphan only after it has been observed unlisted for longer than the grace period — LastModified alone cannot express "unlisted duration" (#188 follow-up: an old source freshly spliced out by the compactor would otherwise be deleted inside the in-flight-reader window).
type LiveRowChecker ¶
type LiveRowChecker interface {
MissingLiveRows(ctx context.Context, schemaID int16, rowIDs []string) ([]string, error)
LiveRowCount(ctx context.Context, schemaID int16) (int64, error)
}
LiveRowChecker reports Postgres entity-store liveness. MissingLiveRows returns which of the given row ids are NOT live (a row absent from every manifest-listed parquet AND missing from Postgres was deleted — its tombstone won a compaction merge and was dropped, so re-appending would resurrect it; see classifyDeltaOrphan in repair.go). LiveRowCount returns the number of live rows for a schema — the right-hand side of init promotion's coverage identity (#292): the orphan set may replace the base tier only if it provably covers every one of these rows.
type Locker ¶
type Locker interface {
TryLock(ctx context.Context, schemaID int16) (locked bool, unlock func(), err error)
}
Locker serializes reconciliation against the live flusher per schema. unlock is non-nil exactly when locked is true.
type ManifestGCStateStore ¶
type ManifestGCStateStore struct {
Store manifest.Store
Resolver manifest.PathResolver
}
ManifestGCStateStore persists GC sighting state next to the schema's manifest (at "<manifest path>.gc-state") through the same etag-aware store, inheriting its optimistic concurrency incl. the If-None-Match create guard. The state object lives under the manifest prefix, so it never pollutes the data-prefix listing the reconciler classifies.
type ManifestStore ¶
type ManifestStore interface {
Load(ctx context.Context, schemaID int16) (*manifest.Manifest, string, error)
Save(ctx context.Context, schemaID int16, m *manifest.Manifest, etag string) (string, error)
}
ManifestStore loads and saves one schema's manifest with optimistic concurrency. Load must have LoadOrCreate semantics: a schema without a manifest reconciles as empty instead of erroring.
type ObjectDeleter ¶
ObjectDeleter deletes a single object by bucket-relative key.
type ObjectInfo ¶
ObjectInfo is one listed S3 object with the metadata reconcile needs: Size feeds repaired FileEntry.SizeBytes, LastModified feeds the GC grace check.
type ObjectLister ¶
type ObjectLister interface {
ListObjects(ctx context.Context, prefix string) ([]ObjectInfo, error)
}
ObjectLister lists every object under a prefix. Implementations own pagination — callers always receive the complete listing.
type ObjectReader ¶
type ObjectReader interface {
GetObject(ctx context.Context, params *s3.GetObjectInput, optFns ...func(*s3.Options)) (*s3.GetObjectOutput, error)
}
ObjectReader streams an object's raw bytes — the surface the #347 checksum scrub re-hashes through cdc.ObjectSHA256. It is declared here rather than reused from cdc so the reconciler names its own dependency; *s3.Client satisfies it, as does cdc.S3GetClient's shape.
type Options ¶
type Options struct {
Repair bool // append delta-shaped orphans back to the manifest
GC bool // delete base-shaped and _tmp orphans past the grace period
GCGrace time.Duration // minimum object age before GC may delete it
// VerifyStamps compares every listed entry's #256 column stamp against
// the object footer; divergence is the byte-truth breach the read-path
// stamp short-circuit cannot see.
VerifyStamps bool
// VerifyChecksums re-hashes every stamped entry's object and compares the
// digest with its #347 content stamp — the byte-integrity layer under
// VerifyStamps' shape check, and the only offline detector of silent
// parquet mis-decode corruption. One full GET per stamped entry.
VerifyChecksums bool
MaxETagRetries int // manifest save retries on optimistic-concurrency conflict
// AllowEmptyManifestSchemas waives the #463 empty-manifest GC guard for
// exactly these schemas. Schema-explicit by design: one mis-pointed
// manifest template resolves EVERY schema's manifest empty, so a global
// override would wave the whole fleet through the very failure the
// guard exists to stop.
AllowEmptyManifestSchemas []int16
}
Options selects the reconcile actions. The zero value is a read-only report.
type OrphanClass ¶
type OrphanClass int
OrphanClass is the filename-shape class of an unlisted parquet object. The class decides the recovery direction: delta orphans may carry data that exists nowhere else and are candidates for repair (guarded by row coverage and Postgres liveness), while merged-base, _tmp, and init-shaped base orphans are all eligible for GC. Init-shaped orphans became GC candidates in #290: cdc-init now holds the same per-schema advisory lock reconcile takes, so under that lock an init-shaped orphan is provably not from an in-flight init — it is either a failed manifest publish or a file superseded by a later init run. Since #292 an init-shaped set that --repair can prove complete is promoted into the base tier instead of collected. Unknown shapes are reported and never touched.
const ( ClassDelta OrphanClass = iota ClassBaseInit // {minRowID}_{maxRowID}.parquet (cdc-init export) ClassBaseMerged // base-{uuid}.parquet (compaction rewrite) ClassTmp ClassUnknown )
type PGAdvisoryLocker ¶
PGAdvisoryLocker takes the flusher's per-schema advisory lock (pg_try_advisory_lock(schemaID, schemaID)). It delegates to cdc.TrySchemaLock, the single source of truth for the pinned-connection acquire/release dance: on a pool, acquire and release could land on different connections, and a session-scoped lock released on the wrong session silently fails, so the lock pins one physical connection for its lifetime and closes it to unlock.
type PGLiveRows ¶
PGLiveRows implements LiveRowChecker over the entity main table. Liveness mirrors cdc-init's export filter (init.go): the row exists under the (schema_id, row_id) primary key AND ltbase_deleted_at IS NULL — writes can soft-delete by setting the column, and treating such rows as live would let repair re-append data whose tombstone compaction already dropped.
func (*PGLiveRows) LiveRowCount ¶
LiveRowCount counts live rows with the same liveness predicate as MissingLiveRows, mirroring cdc-init's export filter: the counting identity in init promotion (#292) is only sound if numerator and denominator use one definition of "live".
func (*PGLiveRows) MissingLiveRows ¶
type Reconciler ¶
type Reconciler struct {
Lister ObjectLister
Deleter ObjectDeleter
Manifests ManifestStore
Objects ObjectReader // required by Opts.VerifyChecksums; nil also leaves repaired and promoted entries unstamped
Stats StatsReader // may be nil unless Opts.Repair or Opts.VerifyStamps
LiveRows LiveRowChecker // may be nil unless Opts.Repair
Locker Locker
Schemas SchemaEnumerator
GCStates GCStateStore // may be nil unless Opts.GC
Now func() time.Time
Bucket string
DataPrefix string
Logger *zap.Logger
Opts Options
}
Reconciler diffs S3 parquet objects against per-schema manifests and optionally repairs (append delta orphans) or garbage-collects (delete compaction leftovers). See the package comment for the recovery model.
type RegistrySchemaEnumerator ¶
type RegistrySchemaEnumerator struct {
DB *sql.DB
Table string
SchemaIDFilter int // 0 = all schemas
}
RegistrySchemaEnumerator enumerates schema IDs from the schema registry table, mirroring cdc-init's getSchemaIDsToInit: every registered schema reconciles, not just those with pending CDC work.
type Report ¶
type Report struct {
Schemas []SchemaReport
}
Report is a full reconcile run across schemas.
func (Report) HasResidualDiscrepancies ¶
HasResidualDiscrepancies reports whether anything actionable is left after repair and GC: orphans not repaired, promoted, or deleted, dangling entries, skipped schemas, unknown shapes, stamp divergences (--verify-stamps), checksum divergences (--verify-checksums), or per-schema failures. Unverifiable paths are informational — they cannot be proven inconsistent from this run, and so is the unstamped-entry count, which reports missing coverage rather than a discrepancy.
type ResolverManifestStore ¶
type ResolverManifestStore struct {
Store manifest.Store
Resolver manifest.PathResolver
}
ResolverManifestStore implements ManifestStore over a manifest.Store and PathResolver. Load has LoadOrCreate semantics — a schema whose manifest does not exist yet reconciles as empty instead of erroring (the compaction ManifestProvider errors on a missing manifest, which is wrong for reconciliation: an empty manifest with live objects is exactly the all-orphans case the tool must report).
type S3ObjectStore ¶
type S3ObjectStore struct {
Client s3ObjectAPI
Bucket string
}
S3ObjectStore implements ObjectLister and ObjectDeleter over an *s3.Client (or any compatible API subset).
func (*S3ObjectStore) DeleteObject ¶
func (s *S3ObjectStore) DeleteObject(ctx context.Context, key string) error
DeleteObject deletes one object by key.
func (*S3ObjectStore) ListObjects ¶
func (s *S3ObjectStore) ListObjects(ctx context.Context, prefix string) ([]ObjectInfo, error)
ListObjects lists all objects under prefix, following continuation tokens until the listing is exhausted.
type SchemaEnumerator ¶
SchemaEnumerator yields the schema IDs to reconcile.
type SchemaReport ¶
type SchemaReport struct {
SchemaID int16
Skipped bool // advisory lock not acquired; nothing inspected
DeltaOrphans []string
BaseOrphans []string
TmpOrphans []string
Unknown []string // unrecognized shapes; reported, never repaired or deleted
Dangling []string // manifest entries with no live object; removal stays manual
Unverifiable []string // manifest paths this listing cannot prove absent
// ObjectsSeen counts this schema's classified parquet objects in
// storage; ManifestEntries counts its manifest's file entries as
// loaded (#463). Together they distinguish "the manifest genuinely
// lists nothing" from "the manifest failed to resolve" — N objects
// against 0 entries is the mis-pointed-template signature --gc refuses.
ObjectsSeen int
ManifestEntries int
Repaired []string // delta orphans appended to the manifest (--repair)
// DeltaLeftovers are delta orphans the repair guard classified as
// compaction leftovers (no uncovered rows, or every uncovered row
// deleted in Postgres): never appended, GC-eligible under --gc.
DeltaLeftovers []string
Deleted []string // leftover/merged-base/tmp orphans removed (--gc)
// PromotedBase are init-shaped base orphans promoted into the manifest
// base tier after the coverage + eviction-safety proof (--repair, #292).
PromotedBase []string
// InitPromotionRefusal explains why an init-shaped orphan set was NOT
// promoted. Promotion is all-or-nothing over the set, so one reason
// covers every file; empty when promotion succeeded or never ran. The
// refused files stay ordinary GC candidates.
InitPromotionRefusal string
// StampDivergences are listed entries whose stamp no longer matches the
// object footer (--verify-stamps); each is a byte-truth breach requiring
// operator action — restamp via rewrite, or investigate the overwrite.
StampDivergences []string
// ChecksumDivergences are listed entries whose object bytes no longer hash
// to the entry's #347 content stamp (--verify-checksums): silent
// corruption, actionable exactly like a stamp divergence.
ChecksumDivergences []string
// SkippedUnstamped counts entries --verify-checksums could not cover
// because they carry no checksum — either legacy (never backfilled) or a
// best-effort write-side hash that failed. It is coverage observability,
// not a discrepancy: it never affects the exit code, but a "clean"
// verdict over mostly-unstamped entries means far less than a clean
// verdict over stamped ones.
SkippedUnstamped int
Err error // per-schema failure; other schemas still reconcile
}
SchemaReport is one schema's reconcile outcome. Key slices hold bucket-relative keys except Unverifiable, which preserves the manifest path verbatim (the raw path is what the operator must inspect).
func (SchemaReport) Residual ¶
func (s SchemaReport) Residual() bool
Residual reports whether this schema still has actionable discrepancies after repair and GC.
type StatsReader ¶
type StatsReader interface {
FileStats(ctx context.Context, key string) (compaction.MergeStats, error)
UncoveredRows(ctx context.Context, key string, listedKeys []string) ([]compaction.UncoveredRow, error)
FileColumns(ctx context.Context, key string) (map[string]string, error)
}
StatsReader inspects one parquet file's contents: FileStats recomputes manifest metadata; UncoveredRows returns the rows whose newest version no listed file supersedes, with a tombstone flag (the repair guard's version-aware coverage probe) — with no listed keys it enumerates every distinct row id in the file (#292 init promotion). FileColumns probes the footer for the #256 column stamp. All take bucket-relative keys. Only consulted under --repair, plus FileColumns under --verify-stamps.