reconcile

package
v0.143.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 12, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package reconcile compares one dataset across N database instances in a single streaming pass and classifies every difference against an explicit consistency policy.

Alignment reuses DALgo recordops.DiffFunc, a K-way merge over key-ordered streams that retains one record per input: memory is O(N) records plus the bounded detail lists, and the cost is one pass over every input rather than N×(N−1)/2 pairwise diffs. Every source is a merge candidate against an empty baseline, so each emitted ID carries each instance's full post-policy row and authority is applied here, by policy, not by merge position.

The package is read-only by construction: it consumes rows and returns a verdict. It never writes, repairs or synchronizes data.

Index

Constants

View Source 
const (
	// DefaultLimit bounds each detail list when Rules.Limit is zero.
	DefaultLimit = 100
	// MaximumLimit is the largest accepted Rules.Limit.
	MaximumLimit = 10000
)
View Source 
const ProfilesFolder = "reconciliations"

ProfilesFolder is the project folder that holds reconciliation profiles, one `<id>.json` per profile.

View Source 
const SnapshotCaveat = "instances were read independently, not from one snapshot: writes that landed between reads can appear as differences"

SnapshotCaveat is reported when instances were not read from one snapshot.

Variables

View Source 
var ErrInvalidRules = errors.New("reconcile: invalid rules")

ErrInvalidRules reports a policy that cannot be evaluated.

Functions

func SortCandidates 

func SortCandidates(candidates []Candidate)

SortCandidates orders candidates by environment then catalog.

Types

type Candidate 

type Candidate struct {
	Environment string                  `json:"environment"`
	Catalog     string                  `json:"catalog"`
	Family      string                  `json:"family"`
	Kind        datatug.EnvironmentKind `json:"kind,omitempty"`
	Tenants     []string                `json:"tenants,omitempty"`
}

Candidate is one catalog in one environment, as the project records it.

func (Candidate) Instance 

func (c Candidate) Instance() Instance

Instance returns the reconciliation identity of the candidate. The instance id is the environment id, which is what Perimeter policies and execution receipts name.

type ColumnStatus 

type ColumnStatus struct {
	Column    string            `json:"column"`
	Policy    FieldPolicy       `json:"policy"`
	MissingIn []string          `json:"missingIn,omitempty"`
	HiddenIn  []string          `json:"hiddenIn,omitempty"`
	Types     map[string]string `json:"types,omitempty"`
	// Compared is false when a type conflict excluded the column entirely.
	Compared bool `json:"compared"`
}

ColumnStatus reports schema drift for one non-key column: instances that lack it, hide it by policy, or declare a different type. Columns that are present with one type everywhere are not listed.

type FieldDifference 

type FieldDifference struct {
	Column   string       `json:"column"`
	Policy   FieldPolicy  `json:"policy"`
	Expected bool         `json:"expected"`
	Groups   []ValueGroup `json:"groups"`
	// Deviating lists instances attributed with the difference: in baseline
	// mode those outside the baseline's group; in consensus mode those
	// outside the unique largest group. Empty with NoConsensus when the
	// largest groups tie.
	Deviating   []string `json:"deviating,omitempty"`
	NoConsensus bool     `json:"noConsensus,omitempty"`
}

FieldDifference is one field whose values disagree across instances.

type FieldPolicy 

type FieldPolicy string

FieldPolicy states the consistency expectation for one field. 

const (
	// FieldIdentical requires byte-for-byte equal typed values.
	FieldIdentical FieldPolicy = "identical"
	// FieldNormalized compares strings after trimming, collapsing inner
	// whitespace and case-folding; other types compare as identical.
	FieldNormalized FieldPolicy = "normalized"
	// FieldInstanceSpecific expects values to differ: differences are
	// reported as expected, never as violations.
	FieldInstanceSpecific FieldPolicy = "instance-specific"
	// FieldIgnored excludes the field from comparison and from results.
	FieldIgnored FieldPolicy = "ignored"
)

type Instance 

type Instance struct {
	ID          string            `json:"id"`
	Tenant      string            `json:"tenant,omitempty"`
	Environment string            `json:"environment,omitempty"`
	Family      string            `json:"family,omitempty"`
	TenantScope string            `json:"tenantScope,omitempty"`
	Labels      map[string]string `json:"labels,omitempty"`
}

Instance identifies one compared database instance along independent dimensions. Tenant and environment are labels, not a hierarchy: one tenant may have several instances, and one instance may hold several tenants (TenantScope then names the slice compared, for example a WHERE predicate already applied by the caller).

type InstanceResult 

type InstanceResult struct {
	Instance      Instance     `json:"instance"`
	Status        Status       `json:"status"`
	Authoritative bool         `json:"authoritative,omitempty"`
	Unavailable   *Unavailable `json:"unavailable,omitempty"`
	ReadAt        string       `json:"readAt,omitempty"`
	RowCount      int          `json:"rowCount"`
	RowsFiltered  bool         `json:"rowsFiltered,omitempty"`
	HiddenColumns []string     `json:"hiddenColumns,omitempty"`
	// DriftColumns lists columns this instance could not be fully compared on
	// because of schema drift (see Result.Columns).
	DriftColumns []string `json:"driftColumns,omitempty"`
	// Counts are per (record, instance) attributions.
	Missing      int `json:"missing"`
	Extra        int `json:"extra"`
	Conflicts    int `json:"conflicts"`
	Expected     int `json:"expected"`
	Inconclusive int `json:"inconclusive"`
}

InstanceResult is one instance's share of the verdict.

type InstanceSelector 

type InstanceSelector struct {
	Family       string                  `json:"family,omitempty"`
	Kind         datatug.EnvironmentKind `json:"kind,omitempty"`
	Tenants      []string                `json:"tenants,omitempty"`
	Environments []string                `json:"environments,omitempty"`
}

InstanceSelector chooses instances either by dimension (family, optional kind and tenants) or by an explicit list of environment ids. Both may be combined: an explicit list narrows the dimension match.

func (InstanceSelector) Select 

func (s InstanceSelector) Select(candidates []Candidate) ([]Candidate, error)

Select returns the candidates matching the selector, one per environment, sorted by environment id. Two catalogs of the same family in one environment are ambiguous and refused: name the environment's catalog in the project instead of guessing.

func (InstanceSelector) Validate 

func (s InstanceSelector) Validate() error

Validate requires a family or an explicit environment list.

type Mode 

type Mode string

Mode selects how authority is assigned. 

const (
	// ModeBaseline compares every instance to one explicitly named
	// authoritative instance. Deviations are attributed to the candidate.
	ModeBaseline Mode = "baseline"
	// ModeConsensus groups equal values and reports minority groups as
	// outliers. Consensus never declares the majority correct.
	ModeConsensus Mode = "consensus"
)

type Presence 

type Presence string

Presence states which instances must hold each record. 

const (
	// PresenceRequired: every instance holds every record.
	PresenceRequired Presence = "required"
	// PresenceSubset (baseline mode): candidates may lack baseline records but
	// must not hold records the baseline lacks.
	PresenceSubset Presence = "subset"
	// PresenceSuperset (baseline mode): candidates must hold every baseline
	// record and may hold more.
	PresenceSuperset Presence = "superset"
	// PresenceAny: presence differences are expected.
	PresenceAny Presence = "any"
)

type Profile 

type Profile struct {
	ID          string           `json:"id"`
	Title       string           `json:"title"`
	Description string           `json:"description,omitempty"`
	QueryID     string           `json:"queryId"`
	Instances   InstanceSelector `json:"instances"`
	Rules       Rules            `json:"rules"`
}

Profile is a saved, reviewable reconciliation: which dataset query, which instances, which rule. It is a plain project file.

func (Profile) Validate 

func (p Profile) Validate() error

Validate checks the profile's shape and that its rules can be evaluated.

type RecordDiscrepancy 

type RecordDiscrepancy struct {
	// Ordinal is the record's 1-based position in key order. It identifies
	// the record in groups-only results, where Key is omitted.
	Ordinal int                      `json:"ordinal"`
	Key     []apicontract.TypedValue `json:"key,omitempty"`
	// Present lists instances holding the record.
	Present []string `json:"present"`
	// Missing lists instances lacking the record; MissingExpected says
	// whether the presence rule allows it.
	Missing         []string `json:"missing,omitempty"`
	MissingExpected bool     `json:"missingExpected,omitempty"`
	// Extra lists instances holding a record the baseline lacks.
	Extra         []string `json:"extra,omitempty"`
	ExtraExpected bool     `json:"extraExpected,omitempty"`
	// PresenceNoConsensus: in consensus mode, as many instances hold the
	// record as lack it; both sides are listed and neither is presumed right.
	PresenceNoConsensus bool `json:"presenceNoConsensus,omitempty"`
	// Inconclusive lists instances whose absence cannot be judged because
	// policy filtered their rows.
	Inconclusive []string          `json:"inconclusive,omitempty"`
	Fields       []FieldDifference `json:"fields,omitempty"`
}

RecordDiscrepancy is one logical record that is not consistent everywhere.

type Result 

type Result struct {
	Mode       Mode             `json:"mode"`
	Baseline   string           `json:"baseline,omitempty"`
	Key        []string         `json:"key"`
	Presence   Presence         `json:"presence"`
	Disclosure ValueDisclosure  `json:"disclosure"`
	Status     Status           `json:"status"`
	Instances  []InstanceResult `json:"instances"`
	Columns    []ColumnStatus   `json:"columns"`
	Summary    Summary          `json:"summary"`
	// Unexpected, Expected and Inconclusive hold the first Limit records of
	// each class in key order.
	Unexpected   []RecordDiscrepancy `json:"unexpected"`
	Expected     []RecordDiscrepancy `json:"expected"`
	Inconclusive []RecordDiscrepancy `json:"inconclusive"`
	Truncated    bool                `json:"truncated"`
	Caveats      []string            `json:"caveats"`
}

Result is the structured outcome of one dataset reconciliation.

func Reconcile 

func Reconcile(sources []Source, rules Rules) (Result, error)

Reconcile compares sources under rules. It returns an error only for an invalid policy or a malformed source stream; unavailable sources, schema drift and policy limitations are reported in the Result.

type Rules 

type Rules struct {
	// Key is the semantic identity: column names whose values identify the
	// same logical record across instances. Never assume surrogate primary
	// keys match across databases.
	Key          []string               `json:"key"`
	Mode         Mode                   `json:"mode"`
	Baseline     string                 `json:"baseline,omitempty"`
	Presence     Presence               `json:"presence,omitempty"`
	DefaultField FieldPolicy            `json:"defaultField,omitempty"`
	Fields       map[string]FieldPolicy `json:"fields,omitempty"`
	Disclosure   ValueDisclosure        `json:"disclosure,omitempty"`
	// Limit bounds each detail list (unexpected, expected, inconclusive).
	// Counts in Summary are always complete.
	Limit int `json:"limit,omitempty"`
}

Rules is the reconciliation policy for one dataset.

type Source 

type Source struct {
	Instance    Instance
	Columns     []apicontract.Column
	Rows        recordsetcompare.OrderedRows
	Info        apicontract.CompareSideInfo
	Unavailable *Unavailable
	// ReadAt is when the source was read (RFC 3339), reported back verbatim.
	ReadAt string
	// Snapshot is true when every row came from one consistent snapshot of
	// this instance. Cross-instance reads are never one snapshot.
	Snapshot bool
}

Source is one instance's post-policy recordset. Rows must be strictly ordered by Rules.Key using apicontract.TypedKeySortKey (see recordsetcompare.OrderRecordset for callers holding a materialized recordset). Info.Limitations carries the access policy already applied by the caller: hidden columns are never compared, and filtered rows make absences inconclusive.

type Status 

type Status string

Status is a reconciliation verdict. 

const (
	StatusConsistent    Status = "consistent"
	StatusDiscrepancies Status = "discrepancies"
	// StatusPartial: no unexpected differences in what was visible, but
	// policy hid rows or columns, so the result is not a full proof.
	StatusPartial Status = "partial"
	// StatusInconclusive: the instance (or run) could not be compared.
	StatusInconclusive Status = "inconclusive"
)

type Summary 

type Summary struct {
	Instances           int `json:"instances"`
	ComparedInstances   int `json:"comparedInstances"`
	Records             int `json:"records"`
	Consistent          int `json:"consistent"`
	WithUnexpected      int `json:"withUnexpected"`
	WithExpectedOnly    int `json:"withExpectedOnly"`
	WithInconclusive    int `json:"withInconclusive"`
	MissingRecords      int `json:"missingRecords"`
	ExtraRecords        int `json:"extraRecords"`
	ConflictingRecords  int `json:"conflictingRecords"`
	ExpectedDifferences int `json:"expectedDifferences"`
}

Summary carries complete counts regardless of Limit.

type Unavailable 

type Unavailable struct {
	Reason  UnavailableReason `json:"reason"`
	Message string            `json:"message,omitempty"`
}

Unavailable marks a source that could not be read. An unavailable source is never treated as matching: its instance is reported inconclusive.

type UnavailableReason 

type UnavailableReason string

UnavailableReason says why a source contributed no rows. 

const (
	UnavailablePermissionDenied UnavailableReason = "permission_denied"
	UnavailableExecutionFailed  UnavailableReason = "execution_failed"
	UnavailableNotExamined      UnavailableReason = "not_examined"
)

type ValueDisclosure 

type ValueDisclosure string

ValueDisclosure controls whether observed values appear in results. 

const (
	// DiscloseValues includes post-policy visible values in value groups.
	DiscloseValues ValueDisclosure = "values"
	// DiscloseGroupsOnly reports only which instances agree, never values,
	// hashes of values or record keys (records are identified by ordinal). Use it when results leave the trust boundary (for
	// example to an AI service).
	DiscloseGroupsOnly ValueDisclosure = "groups-only"
)

type ValueGroup 

type ValueGroup struct {
	Value         *apicontract.TypedValue `json:"value,omitempty"`
	Instances     []string                `json:"instances"`
	Authoritative bool                    `json:"authoritative,omitempty"`
}

ValueGroup is a set of instances holding the same (normalized) value.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.