Documentation
¶
Index ¶
- Constants
- func ApplyTenantMigration(ctx context.Context, pool *pgxpool.Pool, pgSchema string, s *schema.APISchema) error
- func PersistTenantSchema(ctx context.Context, pool *pgxpool.Pool, tenantID string, s *schema.APISchema) error
- type ApplyOutcome
- type DestructiveImpact
- type DestructiveOp
- type FanoutOptions
- type FanoutResult
- type FanoutStatus
- type MigrationWorker
- type Preview
- type TenantFanoutResult
Constants ¶
const ( StreamName = "migrations" ConsumerGroup = "migration-workers" )
Exported stream constants so callers (cmd_serve, tests) use the same values.
Variables ¶
This section is empty.
Functions ¶
func ApplyTenantMigration ¶
func ApplyTenantMigration(ctx context.Context, pool *pgxpool.Pool, pgSchema string, s *schema.APISchema) error
ApplyTenantMigration converges the resource tables in pgSchema to the tenant's schema, using the real migration engine (pkg/schemadiff): introspect the live state, build the desired state, diff into a typed plan, and apply it through the production-safe executor (lock_timeout + retry, NOT VALID/VALIDATE, CONCURRENTLY partitioning, data-preserving renames).
This REPLACES the historical converger (CREATE TABLE / ADD COLUMN IF NOT EXISTS), which lost data on a rename, silently discarded NOT NULL, no-op'd a type change and took locks unguarded (docs/MIGRATION_DIAG.md). The engine now migrates for real: it detects what changed and applies it safely.
This is the PURELY ADDITIVE entry point: it gates EVERY drop (no approval is possible through it), exactly the v1 policy. It is used by tenant registration (a brand-new tenant has no drops anyway) and the Redis migration worker — an AUTOMATED process must NEVER drop data without prior, recorded human approval, so the worker can only ever create/add/alter/rename. To apply an explicitly approved destructive drop, use ApplyTenantMigrationApproved (the control-plane PUT / CLI path), which is the only place a drop is ever executed.
v1 policy (documented):
- Provisioning a NEW tenant is identical to before: the diff against an empty schema is all CreateTable, applied as it always was.
- Re-applying an UNCHANGED schema is a true no-op: the diff is empty, no DDL runs (introspect is the only DB read).
- DROP operations are NEVER applied through THIS function — they are logged and left as drift, exactly the converger's "never removes anything" contract. This guarantees no data loss and that a minor modeling gap can never spuriously undo a converger artifact. A field removed from the schema leaves its column in place (docs/MIGRATION_DIAG.md case D, unchanged).
- Every NON-drop change IS applied faithfully: a rename preserves data, a real NOT NULL is enforced (over populated data it fails loudly and rolls back atomically — never the converger's silent NULL-accepting divergence), a type change is a real ALTER … TYPE. Planning concerns (backfill/transformational) are logged before applying.
Tables physically present but NOT declared as resources — the engine's own auth_*/files tables, or a resource dropped from the schema — are excluded from the diff (managedSubset) so the additive migration only ever touches resource tables. (The approval-aware path additionally SURFACES a removed-resource table as a gated/approvable DropTable; see ApplyTenantMigrationApproved.)
The signature is unchanged, so every additive call site (tenant registration, the Redis worker) is untouched.
func PersistTenantSchema ¶
func PersistTenantSchema(ctx context.Context, pool *pgxpool.Pool, tenantID string, s *schema.APISchema) error
PersistTenantSchema records an APPLIED schema on the tenant's control-plane row + version history, exactly as the fan-out and the control-plane PUT do. The UPDATE on public.tenants fires the schema_updated trigger (migrations/001_control_plane.sql → pg_notify), so a RUNNING engine invalidates its per-tenant caches and starts serving the new columns hot.
DOC-1 (CONSUMER-PATH-S1): the single-tenant CLI `migrate --tenant` applied the DDL but never persisted the schema, so the live engine kept validating against the OLD one — a freshly migrated field answered `422 unknown_field` until a restart, while the SAME change through the control-plane PUT worked hot. One migrate path, one persistence contract. Best-effort at the call site: the DDL is already applied; a persistence failure is reported, never rolls the DDL back.
Types ¶
type ApplyOutcome ¶
type ApplyOutcome struct {
AppliedDrops []string // destructive keys applied (explicitly approved)
GatedDrops []string // destructive keys present but NOT approved (drift)
UnmatchedApprovals []string // approved keys that matched no destructive op
// ExternalDrift are consumer-owned objects (no schema version ever declared
// them — ENG-9) present in the database but out of migration scope: never
// proposed as drops, never approvable, left untouched.
ExternalDrift []string
// Unapplied lists DECLARED operations that are STILL PENDING after the apply,
// computed by RE-INTROSPECTING the live database — never by trusting the
// executor's own log (ENG-13). A non-empty Unapplied means declared and applied
// have DIVERGED and the apply is PARTIAL: it must never be reported as success.
//
// This is the class-level guard, not a per-operation one: whatever the reason an
// operation did not land (a tolerated foreign-key failure, a rename skipped
// because its target already existed, a future tolerant path nobody has written
// yet), the verification sees the DATABASE and reports the gap.
Unapplied []string
// UnvalidatedFKs lists foreign keys that were added but left NOT VALID because
// pre-existing rows violate them. They DO protect every new write; the historical
// rows need fixing, then a manual VALIDATE. Not a divergence (the constraint
// exists), so it does not make the apply partial — but it is reported, never
// only logged.
UnvalidatedFKs []string
// NoChange is true when NO DDL was applied — the schema was already converged, or
// the only pending operations were gated drops. It is the noop signal the
// multi-tenant orchestrator uses to distinguish an "already up to date" tenant
// from one it actually migrated.
NoChange bool
}
ApplyOutcome reports what a migration apply did with the data-losing drops in the plan: which approved drops it APPLIED, which it GATED (present but not approved, kept as drift), and which approval tokens matched NOTHING (a typo or an already-applied drop). It carries no error semantics — an apply that fails returns an error instead.
func ApplyTenantMigrationApproved ¶
func ApplyTenantMigrationApproved(ctx context.Context, pool *pgxpool.Pool, pgSchema string, s *schema.APISchema, approved []string) (*ApplyOutcome, error)
ApplyTenantMigrationApproved is the APPROVAL-AWARE apply: it converges pgSchema to the desired schema and, for the data-losing drops (DropTable / DropColumn), applies ONLY those whose approval key (schemadiff.DestructiveKey) appears in `approved`. Every other drop stays gated as drift, exactly as the additive policy. With an empty `approved`, it is fail-safe: NOTHING is dropped (identical net effect to the additive path, plus visibility of what COULD be approved).
This is the ONLY function that ever executes a destructive drop, and only by explicit, enumerated consent. It is wired to the control-plane PUT (after a dry-run preview) and the CLI `migrate --approve-drops`. It is NEVER reachable from the Redis worker (which must not auto-approve).
Preview↔apply consistency is structural: the plan is recomputed FRESH against the live database here, and a drop is applied iff its key is approved. A destructive drop that appeared since the preview carries a different (un-approved) key, so it is gated automatically — a new, unreviewed drop can never slip through.
func (*ApplyOutcome) Partial ¶
func (o *ApplyOutcome) Partial() bool
Partial reports whether the apply left DECLARED changes unapplied — i.e. the tenant's schema now claims a shape the database does not have. Every reporter (CLI, control plane, admin API, fan-out) must treat a partial apply as a FAILURE, never as ✓.
type DestructiveImpact ¶
type DestructiveImpact struct {
Key string `json:"key"`
Kind string `json:"kind"`
Tenants int `json:"tenants"` // tenants where this drop appears
RowsLost int64 `json:"rows_lost"` // total rows lost across those tenants
}
DestructiveImpact is the AGGREGATE impact of one approved/gated drop across all tenants in a dry-run — "how much data would be lost, in how many tenants".
type DestructiveOp ¶
type DestructiveOp struct {
Key string `json:"key"` // approval token: "<table>" or "<table>.<column>"
Kind string `json:"kind"` // "table" | "column"
Table string `json:"table"` // the (post-rename) table name
Column string `json:"column"` // "" for a table drop
RowsLost int64 `json:"rows_lost"` // rows whose data is destroyed (table: all rows; column: rows with a non-null value)
TableRows int64 `json:"table_rows"` // total rows in the table (context)
Approved bool `json:"approved"` // whether this op's key is in the supplied approval set
Summary string `json:"summary"` // human one-liner with the impact
}
DestructiveOp is one data-losing drop in a preview, with the IMPACT of applying it (how much data is destroyed) and the KEY an operator enumerates to approve it.
type FanoutOptions ¶
type FanoutOptions struct {
// Schema is the desired schema applied to EVERY targeted tenant.
Schema *schema.APISchema
// TenantIDs is an explicit subset; empty means ALL tenants in public.tenants.
// The caller (CLI) must opt into "all" deliberately — it never defaults silently.
TenantIDs []string
// ApprovedDrops are destructive-drop keys approved for EVERY targeted tenant
// (a mass drop). Empty (the default) is additive: every drop is gated.
ApprovedDrops []string
// DryRun classifies + measures impact without applying or persisting anything.
DryRun bool
// OnTenant, if set, is called after each tenant completes (for CLI streaming).
OnTenant func(TenantFanoutResult)
}
FanoutOptions configures a fan-out.
type FanoutResult ¶
type FanoutResult struct {
RunID string `json:"run_id"`
DryRun bool `json:"dry_run"`
Total int `json:"total"`
Applied int `json:"applied"`
Noop int `json:"noop"`
Failed int `json:"failed"`
MissingTenants []string `json:"missing_tenants,omitempty"` // requested ids not in public.tenants
Results []TenantFanoutResult `json:"results"`
}
FanoutResult summarizes a fan-out run.
func RunFanout ¶
func RunFanout(ctx context.Context, pool *pgxpool.Pool, opts FanoutOptions) (*FanoutResult, error)
RunFanout applies opts.Schema to the targeted tenants, one at a time, recording a per-tenant result and CONTINUING past any failure. The only error it returns is a setup failure (enumerating the tenants); a per-tenant migration failure is captured in the result, never propagated as the function's error — that is the resilience contract. A re-run resumes: converged tenants are noops, failed ones retry.
func (*FanoutResult) AggregateDestructive ¶
func (r *FanoutResult) AggregateDestructive() []DestructiveImpact
AggregateDestructive sums the data-losing drops across every tenant's preview (use after a dry-run). It is the informed-consent surface for a MASS destructive change.
type FanoutStatus ¶
type FanoutStatus string
FanoutStatus is the per-tenant outcome of a fan-out.
const ( // FanoutApplied — DDL was applied to this tenant (in a dry-run: WOULD be applied). FanoutApplied FanoutStatus = "applied" // FanoutNoop — already converged (empty diff), or only gated drops were pending. FanoutNoop FanoutStatus = "noop" // FanoutFailed — the tenant's migration errored (or its lock could not be acquired). // The tenant is left in its previous state and will be retried on a resume. FanoutFailed FanoutStatus = "failed" )
type MigrationWorker ¶
type MigrationWorker struct {
// BlockTime is how long XReadGroup waits for new messages before looping.
// Default: 5s. Set to a smaller value in tests.
BlockTime time.Duration
// BackoffBase is the unit delay before re-enqueueing a failed job.
// Delays: BackoffBase * 1, BackoffBase * 2, BackoffBase * 4 (exponential).
// Default: 1s. Set to a smaller value in tests.
BackoffBase time.Duration
// LockRetryDelay is how long to wait before re-enqueueing a job whose tenant
// advisory lock is held by another worker. Default: 2s.
LockRetryDelay time.Duration
// ConsumerName identifies this worker in the Redis consumer group.
// Must be unique per running worker instance. Default: "worker-1".
ConsumerName string
// contains filtered or unexported fields
}
MigrationWorker consumes jobs from the Redis Stream and applies tenant migrations.
func NewMigrationWorker ¶
func NewMigrationWorker(r *redis.Client, db *pgxpool.Pool, cache *tenant.SchemaCache) *MigrationWorker
func (*MigrationWorker) Run ¶
func (w *MigrationWorker) Run(ctx context.Context)
Run starts the consumer loop. It blocks until ctx is cancelled.
type Preview ¶
type Preview struct {
PGSchema string `json:"pg_schema"`
// Empty is true when the schema is already converged — nothing to do.
Empty bool `json:"empty"`
// Apply are the SAFE operations that will run (creates, adds, alters, renames, FK
// adds, and any FK drop required as a consequence of an approved table drop). The
// data-losing drops are NOT here — they live in Destructive with their impact.
Apply []string `json:"apply,omitempty"`
// Destructive are the data-losing drops with impact and approval status. A drop is
// applied only when Approved is true; otherwise it is gated (drift).
Destructive []DestructiveOp `json:"destructive,omitempty"`
// Drift are the SAFE drops (index/constraint removals) left as additive drift —
// not approvable in v1, they simply stay.
Drift []string `json:"drift,omitempty"`
// Concerns are backfill/transformational risks on PRE-EXISTING tables (e.g. a NOT
// NULL added over populated data), surfaced before applying.
Concerns []string `json:"concerns,omitempty"`
// UnmatchedApprovals are supplied approval tokens that matched no destructive op
// in this plan (a typo, or an already-applied drop).
UnmatchedApprovals []string `json:"unmatched_approvals,omitempty"`
// External are consumer-owned objects (no deployed schema version ever declared
// them — ENG-9): present in the database, out of migration scope, never proposed
// as drops and never approvable. Informative only.
External []string `json:"external,omitempty"`
}
Preview is the classified result of a dry run.
func PreviewTenantMigration ¶
func PreviewTenantMigration(ctx context.Context, pool *pgxpool.Pool, pgSchema string, s *schema.APISchema, approved []string) (*Preview, error)
PreviewTenantMigration computes and classifies the migration plan that would converge pgSchema to s, WITHOUT applying anything. `approved` is the (optional) set of destructive-drop keys being considered — pass nil to see every drop gated (the "what could I approve" view), or a set to see exactly what an apply with that set would do. For each data-losing drop it queries the live table for the impact (rows that would be lost).
func (*Preview) HasDestructive ¶
HasDestructive reports whether the preview contains any data-losing drop (whether or not it is approved) — the signal that an apply needs informed consent.
func (*Preview) PendingDestructive ¶
PendingDestructive reports whether any data-losing drop is present but NOT approved — i.e. an apply with this approval set would still leave a gated drop.
type TenantFanoutResult ¶
type TenantFanoutResult struct {
TenantID string `json:"tenant_id"`
PGSchema string `json:"pg_schema"`
Status FanoutStatus `json:"status"`
AppliedDrops []string `json:"applied_drops,omitempty"`
GatedDrops []string `json:"gated_drops,omitempty"`
Error string `json:"error,omitempty"`
DurationMS int64 `json:"duration_ms"`
// Preview is populated on a dry run (the classified plan + per-drop impact).
Preview *Preview `json:"preview,omitempty"`
}
TenantFanoutResult is the outcome for one tenant.