migration

package
v0.1.16 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 30, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Index

Constants

View Source
const (
	StreamName    = "migrations"
	ConsumerGroup = "migration-workers"
)

Exported stream constants so callers (cmd_serve, tests) use the same values.

Variables

This section is empty.

Functions

func ApplyTenantMigration

func ApplyTenantMigration(ctx context.Context, pool *pgxpool.Pool, pgSchema string, s *schema.APISchema) error

ApplyTenantMigration converges the resource tables in pgSchema to the tenant's schema, using the real migration engine (pkg/schemadiff): introspect the live state, build the desired state, diff into a typed plan, and apply it through the production-safe executor (lock_timeout + retry, NOT VALID/VALIDATE, CONCURRENTLY partitioning, data-preserving renames).

This REPLACES the historical converger (CREATE TABLE / ADD COLUMN IF NOT EXISTS), which lost data on a rename, silently discarded NOT NULL, no-op'd a type change and took locks unguarded (docs/MIGRATION_DIAG.md). The engine now migrates for real: it detects what changed and applies it safely.

This is the PURELY ADDITIVE entry point: it gates EVERY drop (no approval is possible through it), exactly the v1 policy. It is used by tenant registration (a brand-new tenant has no drops anyway) and the Redis migration worker — an AUTOMATED process must NEVER drop data without prior, recorded human approval, so the worker can only ever create/add/alter/rename. To apply an explicitly approved destructive drop, use ApplyTenantMigrationApproved (the control-plane PUT / CLI path), which is the only place a drop is ever executed.

v1 policy (documented):

  • Provisioning a NEW tenant is identical to before: the diff against an empty schema is all CreateTable, applied as it always was.
  • Re-applying an UNCHANGED schema is a true no-op: the diff is empty, no DDL runs (introspect is the only DB read).
  • DROP operations are NEVER applied through THIS function — they are logged and left as drift, exactly the converger's "never removes anything" contract. This guarantees no data loss and that a minor modeling gap can never spuriously undo a converger artifact. A field removed from the schema leaves its column in place (docs/MIGRATION_DIAG.md case D, unchanged).
  • Every NON-drop change IS applied faithfully: a rename preserves data, a real NOT NULL is enforced (over populated data it fails loudly and rolls back atomically — never the converger's silent NULL-accepting divergence), a type change is a real ALTER … TYPE. Planning concerns (backfill/transformational) are logged before applying.

Tables physically present but NOT declared as resources — the engine's own auth_*/files tables, or a resource dropped from the schema — are excluded from the diff (managedSubset) so the additive migration only ever touches resource tables. (The approval-aware path additionally SURFACES a removed-resource table as a gated/approvable DropTable; see ApplyTenantMigrationApproved.)

The signature is unchanged, so every additive call site (tenant registration, the Redis worker) is untouched.

func PersistTenantSchema

func PersistTenantSchema(ctx context.Context, pool *pgxpool.Pool, tenantID string, s *schema.APISchema) error

PersistTenantSchema records an APPLIED schema on the tenant's control-plane row + version history, exactly as the fan-out and the control-plane PUT do. The UPDATE on public.tenants fires the schema_updated trigger (migrations/001_control_plane.sql → pg_notify), so a RUNNING engine invalidates its per-tenant caches and starts serving the new columns hot.

DOC-1 (CONSUMER-PATH-S1): the single-tenant CLI `migrate --tenant` applied the DDL but never persisted the schema, so the live engine kept validating against the OLD one — a freshly migrated field answered `422 unknown_field` until a restart, while the SAME change through the control-plane PUT worked hot. One migrate path, one persistence contract. Best-effort at the call site: the DDL is already applied; a persistence failure is reported, never rolls the DDL back.

Types

type ApplyOutcome

type ApplyOutcome struct {
	AppliedDrops       []string // destructive keys applied (explicitly approved)
	GatedDrops         []string // destructive keys present but NOT approved (drift)
	UnmatchedApprovals []string // approved keys that matched no destructive op
	// ExternalDrift are consumer-owned objects (no schema version ever declared
	// them — ENG-9) present in the database but out of migration scope: never
	// proposed as drops, never approvable, left untouched.
	ExternalDrift []string
	// Unapplied lists DECLARED operations that are STILL PENDING after the apply,
	// computed by RE-INTROSPECTING the live database — never by trusting the
	// executor's own log (ENG-13). A non-empty Unapplied means declared and applied
	// have DIVERGED and the apply is PARTIAL: it must never be reported as success.
	//
	// This is the class-level guard, not a per-operation one: whatever the reason an
	// operation did not land (a tolerated foreign-key failure, a rename skipped
	// because its target already existed, a future tolerant path nobody has written
	// yet), the verification sees the DATABASE and reports the gap.
	Unapplied []string
	// UnvalidatedFKs lists foreign keys that were added but left NOT VALID because
	// pre-existing rows violate them. They DO protect every new write; the historical
	// rows need fixing, then a manual VALIDATE. Not a divergence (the constraint
	// exists), so it does not make the apply partial — but it is reported, never
	// only logged.
	UnvalidatedFKs []string
	// NoChange is true when NO DDL was applied — the schema was already converged, or
	// the only pending operations were gated drops. It is the noop signal the
	// multi-tenant orchestrator uses to distinguish an "already up to date" tenant
	// from one it actually migrated.
	NoChange bool
}

ApplyOutcome reports what a migration apply did with the data-losing drops in the plan: which approved drops it APPLIED, which it GATED (present but not approved, kept as drift), and which approval tokens matched NOTHING (a typo or an already-applied drop). It carries no error semantics — an apply that fails returns an error instead.

func ApplyTenantMigrationApproved

func ApplyTenantMigrationApproved(ctx context.Context, pool *pgxpool.Pool, pgSchema string, s *schema.APISchema, approved []string) (*ApplyOutcome, error)

ApplyTenantMigrationApproved is the APPROVAL-AWARE apply: it converges pgSchema to the desired schema and, for the data-losing drops (DropTable / DropColumn), applies ONLY those whose approval key (schemadiff.DestructiveKey) appears in `approved`. Every other drop stays gated as drift, exactly as the additive policy. With an empty `approved`, it is fail-safe: NOTHING is dropped (identical net effect to the additive path, plus visibility of what COULD be approved).

This is the ONLY function that ever executes a destructive drop, and only by explicit, enumerated consent. It is wired to the control-plane PUT (after a dry-run preview) and the CLI `migrate --approve-drops`. It is NEVER reachable from the Redis worker (which must not auto-approve).

Preview↔apply consistency is structural: the plan is recomputed FRESH against the live database here, and a drop is applied iff its key is approved. A destructive drop that appeared since the preview carries a different (un-approved) key, so it is gated automatically — a new, unreviewed drop can never slip through.

func (*ApplyOutcome) Partial

func (o *ApplyOutcome) Partial() bool

Partial reports whether the apply left DECLARED changes unapplied — i.e. the tenant's schema now claims a shape the database does not have. Every reporter (CLI, control plane, admin API, fan-out) must treat a partial apply as a FAILURE, never as ✓.

type DestructiveImpact

type DestructiveImpact struct {
	Key      string `json:"key"`
	Kind     string `json:"kind"`
	Tenants  int    `json:"tenants"`   // tenants where this drop appears
	RowsLost int64  `json:"rows_lost"` // total rows lost across those tenants
}

DestructiveImpact is the AGGREGATE impact of one approved/gated drop across all tenants in a dry-run — "how much data would be lost, in how many tenants".

type DestructiveOp

type DestructiveOp struct {
	Key       string `json:"key"`        // approval token: "<table>" or "<table>.<column>"
	Kind      string `json:"kind"`       // "table" | "column"
	Table     string `json:"table"`      // the (post-rename) table name
	Column    string `json:"column"`     // "" for a table drop
	RowsLost  int64  `json:"rows_lost"`  // rows whose data is destroyed (table: all rows; column: rows with a non-null value)
	TableRows int64  `json:"table_rows"` // total rows in the table (context)
	Approved  bool   `json:"approved"`   // whether this op's key is in the supplied approval set
	Summary   string `json:"summary"`    // human one-liner with the impact
}

DestructiveOp is one data-losing drop in a preview, with the IMPACT of applying it (how much data is destroyed) and the KEY an operator enumerates to approve it.

type FanoutOptions

type FanoutOptions struct {
	// Schema is the desired schema applied to EVERY targeted tenant.
	Schema *schema.APISchema
	// TenantIDs is an explicit subset; empty means ALL tenants in public.tenants.
	// The caller (CLI) must opt into "all" deliberately — it never defaults silently.
	TenantIDs []string
	// ApprovedDrops are destructive-drop keys approved for EVERY targeted tenant
	// (a mass drop). Empty (the default) is additive: every drop is gated.
	ApprovedDrops []string
	// DryRun classifies + measures impact without applying or persisting anything.
	DryRun bool
	// OnTenant, if set, is called after each tenant completes (for CLI streaming).
	OnTenant func(TenantFanoutResult)
}

FanoutOptions configures a fan-out.

type FanoutResult

type FanoutResult struct {
	RunID          string               `json:"run_id"`
	DryRun         bool                 `json:"dry_run"`
	Total          int                  `json:"total"`
	Applied        int                  `json:"applied"`
	Noop           int                  `json:"noop"`
	Failed         int                  `json:"failed"`
	MissingTenants []string             `json:"missing_tenants,omitempty"` // requested ids not in public.tenants
	Results        []TenantFanoutResult `json:"results"`
}

FanoutResult summarizes a fan-out run.

func RunFanout

func RunFanout(ctx context.Context, pool *pgxpool.Pool, opts FanoutOptions) (*FanoutResult, error)

RunFanout applies opts.Schema to the targeted tenants, one at a time, recording a per-tenant result and CONTINUING past any failure. The only error it returns is a setup failure (enumerating the tenants); a per-tenant migration failure is captured in the result, never propagated as the function's error — that is the resilience contract. A re-run resumes: converged tenants are noops, failed ones retry.

func (*FanoutResult) AggregateDestructive

func (r *FanoutResult) AggregateDestructive() []DestructiveImpact

AggregateDestructive sums the data-losing drops across every tenant's preview (use after a dry-run). It is the informed-consent surface for a MASS destructive change.

type FanoutStatus

type FanoutStatus string

FanoutStatus is the per-tenant outcome of a fan-out.

const (
	// FanoutApplied — DDL was applied to this tenant (in a dry-run: WOULD be applied).
	FanoutApplied FanoutStatus = "applied"
	// FanoutNoop — already converged (empty diff), or only gated drops were pending.
	FanoutNoop FanoutStatus = "noop"
	// FanoutFailed — the tenant's migration errored (or its lock could not be acquired).
	// The tenant is left in its previous state and will be retried on a resume.
	FanoutFailed FanoutStatus = "failed"
)

type MigrationWorker

type MigrationWorker struct {

	// BlockTime is how long XReadGroup waits for new messages before looping.
	// Default: 5s. Set to a smaller value in tests.
	BlockTime time.Duration

	// BackoffBase is the unit delay before re-enqueueing a failed job.
	// Delays: BackoffBase * 1, BackoffBase * 2, BackoffBase * 4 (exponential).
	// Default: 1s. Set to a smaller value in tests.
	BackoffBase time.Duration

	// LockRetryDelay is how long to wait before re-enqueueing a job whose tenant
	// advisory lock is held by another worker. Default: 2s.
	LockRetryDelay time.Duration

	// ConsumerName identifies this worker in the Redis consumer group.
	// Must be unique per running worker instance. Default: "worker-1".
	ConsumerName string
	// contains filtered or unexported fields
}

MigrationWorker consumes jobs from the Redis Stream and applies tenant migrations.

func NewMigrationWorker

func NewMigrationWorker(r *redis.Client, db *pgxpool.Pool, cache *tenant.SchemaCache) *MigrationWorker

func (*MigrationWorker) Run

func (w *MigrationWorker) Run(ctx context.Context)

Run starts the consumer loop. It blocks until ctx is cancelled.

type Preview

type Preview struct {
	PGSchema string `json:"pg_schema"`
	// Empty is true when the schema is already converged — nothing to do.
	Empty bool `json:"empty"`
	// Apply are the SAFE operations that will run (creates, adds, alters, renames, FK
	// adds, and any FK drop required as a consequence of an approved table drop). The
	// data-losing drops are NOT here — they live in Destructive with their impact.
	Apply []string `json:"apply,omitempty"`
	// Destructive are the data-losing drops with impact and approval status. A drop is
	// applied only when Approved is true; otherwise it is gated (drift).
	Destructive []DestructiveOp `json:"destructive,omitempty"`
	// Drift are the SAFE drops (index/constraint removals) left as additive drift —
	// not approvable in v1, they simply stay.
	Drift []string `json:"drift,omitempty"`
	// Concerns are backfill/transformational risks on PRE-EXISTING tables (e.g. a NOT
	// NULL added over populated data), surfaced before applying.
	Concerns []string `json:"concerns,omitempty"`
	// UnmatchedApprovals are supplied approval tokens that matched no destructive op
	// in this plan (a typo, or an already-applied drop).
	UnmatchedApprovals []string `json:"unmatched_approvals,omitempty"`
	// External are consumer-owned objects (no deployed schema version ever declared
	// them — ENG-9): present in the database, out of migration scope, never proposed
	// as drops and never approvable. Informative only.
	External []string `json:"external,omitempty"`
}

Preview is the classified result of a dry run.

func PreviewTenantMigration

func PreviewTenantMigration(ctx context.Context, pool *pgxpool.Pool, pgSchema string, s *schema.APISchema, approved []string) (*Preview, error)

PreviewTenantMigration computes and classifies the migration plan that would converge pgSchema to s, WITHOUT applying anything. `approved` is the (optional) set of destructive-drop keys being considered — pass nil to see every drop gated (the "what could I approve" view), or a set to see exactly what an apply with that set would do. For each data-losing drop it queries the live table for the impact (rows that would be lost).

func (*Preview) HasDestructive

func (p *Preview) HasDestructive() bool

HasDestructive reports whether the preview contains any data-losing drop (whether or not it is approved) — the signal that an apply needs informed consent.

func (*Preview) PendingDestructive

func (p *Preview) PendingDestructive() bool

PendingDestructive reports whether any data-losing drop is present but NOT approved — i.e. an apply with this approval set would still leave a gated drop.

type TenantFanoutResult

type TenantFanoutResult struct {
	TenantID     string       `json:"tenant_id"`
	PGSchema     string       `json:"pg_schema"`
	Status       FanoutStatus `json:"status"`
	AppliedDrops []string     `json:"applied_drops,omitempty"`
	GatedDrops   []string     `json:"gated_drops,omitempty"`
	Error        string       `json:"error,omitempty"`
	DurationMS   int64        `json:"duration_ms"`
	// Preview is populated on a dry run (the classified plan + per-drop impact).
	Preview *Preview `json:"preview,omitempty"`
}

TenantFanoutResult is the outcome for one tenant.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL