tenantexport

package
v0.30.38 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 7, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package tenantexport implements per-tenant data export as a pair of iterate-and-serialize algorithms, one per storage engine xolu uses (SQLite, Pebble) -- not a reconstruction of a valid subset database file. Each source (a SQLite table, a Pebble store) becomes one JSON file: every row or key/value pair in that source, serialized, no schema/constraint/index fidelity attempted or needed.

Design settled directly with the team (2026-08-03), correcting an earlier, substantially overcomplicated draft that tried to reconstruct a byte-valid SQLite file containing only one tenant's tables (ATTACH + CREATE TABLE AS SELECT, preserving schema/indexes). That problem doesn't need solving: a JSON export doesn't care about SQL structure fidelity, only about the data itself.

See docs/proposals/tenant-export.md for the full design (async job flow, blob-backed result, TTL, throttling). Wired into pkg/server/blob_export_handlers.go's POST/GET .../blob/export routes.

Index

Constants

View Source
const ExportKeyPrefix = "export-"

ExportKeyPrefix is the fixed prefix every export blob is stored under (see handleBlobExportStart's own exportKey construction, "export-%d.zip") -- SweepExpiredExports lists exactly this prefix, never touching a caller's own blobs that happen to start differently.

Variables

View Source
var LocStoreTables = []SQLiteTableSpec{
	{Name: "locations", TenantFiltered: false},
	{Name: "loc_patterns", TenantFiltered: false},
	{Name: "loc_capacity", TenantFiltered: false},
	{Name: "fences", TenantFiltered: false},
	{Name: "loc_fence_capacity", TenantFiltered: false},
	{Name: "loc_fence_membership", TenantFiltered: false},
	{Name: "loc_assignment", TenantFiltered: false},
	{Name: "loc_journal", TenantFiltered: false},
}

LocStoreTables is every table in loc's own dedicated per-tenant file (storelayout.TenantLocDir) -- the whole FILE is already one tenant's own (T-115, wave 9), so none of these need a tenant_id filter, unlike PrimaryStoreTables' global tables.

View Source
var ObjStoreTables = []SQLiteTableSpec{
	{Name: "obj_subjects", TenantFiltered: false},
	{Name: "obj_position", TenantFiltered: false},
	{Name: "obj_journal", TenantFiltered: false},
}

ObjStoreTables is every table in obj's own dedicated per-tenant file (storelayout.TenantObjDir) -- same reasoning as LocStoreTables.

View Source
var PrimaryStoreTables = []SQLiteTableSpec{

	{Name: "nodes", NameFunc: func(t uint16) string { return tenant.TenantID(t).NodesTableName() }},
	{Name: "edges", NameFunc: func(t uint16) string { return tenant.TenantID(t).EdgePropsTableName() }},
	{Name: "graph", NameFunc: func(t uint16) string { return tenant.TenantID(t).GraphTableName() }},
	{Name: "eseq", NameFunc: func(t uint16) string { return tenant.TenantID(t).EdgeSeqTableName() }},
	{Name: "nseq", NameFunc: func(t uint16) string { return tenant.TenantID(t).NodeSeqTableName() }},
	{Name: "e_sch", NameFunc: func(t uint16) string { return tenant.TenantID(t).EdgeSchemaTableName() }},
	{Name: "n_sch", NameFunc: func(t uint16) string { return tenant.TenantID(t).NodeSchemaTableName() }},

	{Name: "bal_accounts", TenantPrefixed: true},
	{Name: "bal_balances", TenantPrefixed: true},
	{Name: "bal_checkpoints", TenantPrefixed: true},
	{Name: "bal_journal", TenantPrefixed: true},
	{Name: "bal_seal", TenantPrefixed: true},

	{Name: "cal_bookings", TenantFiltered: true},
	{Name: "cal_calendars", TenantFiltered: true},
	{Name: "cal_participants", TenantFiltered: true},
	{Name: "cal_ord_seq", TenantFiltered: true},

	{Name: "dxp_defs", TenantFiltered: true},
	{Name: "dxp_id_seq", TenantFiltered: true},
	{Name: "dxp_txn", TenantFiltered: true},

	{Name: "event_defs", TenantFiltered: true},
	{Name: "event_delivery_log", TenantFiltered: true},

	{Name: "fsm_definitions", TenantFiltered: true},
	{Name: "fsm_history", TenantFiltered: true},
	{Name: "fsm_id_seq", TenantFiltered: true},
	{Name: "fsm_machines", TenantFiltered: true},
	{Name: "fsm_terminal_states", TenantFiltered: true},

	{Name: "gen_definitions", TenantFiltered: true},
	{Name: "sequences", TenantFiltered: true},

	{Name: "entity_meta", TenantFiltered: true},
}

PrimaryStoreTables is every table in a tenant's own primary store (store/xolu.db, shared or per-file per SQLitePerFileTenants) that holds real tenant data -- verified directly against each subsystem's own real query/schema code (grep for actual WHERE tenant_id=... use or, where no query happened to be caught that way, the literal CREATE TABLE itself -- cal_participants specifically was confirmed via schema, not a query match), not assumed from naming convention alone.

Deliberately excluded, and why:

  • t0000_efts*/t0000_nfts* (full-text search indexes): derived, rebuildable from the entity data already being exported: not source data.
  • schema_version, schema_version_v2, tenants: server/system-level tables, not scoped to any one tenant.

This list is authoritative until a new subsystem adds tables of its own -- extend it there, not by guessing the pattern holds.

Functions

func ExportPebbleStore

func ExportPebbleStore(ctx context.Context, dir, name, outDir string) (int, error)

ExportPebbleStore iterates every key/value pair in the Pebble database at dir and writes them as a JSON array to outDir/<name>.json. dir is expected to already be scoped to one tenant (storelayout.TenantBalRollupDir, TenantCalDir, TenantTSDir, etc. each return a per-tenant directory) -- there is no tenant_id filtering step here because none is needed, unlike the SQLite side's shared tables.

Opens its own read-only handle rather than requiring an existing *pebble.DB be passed in -- Pebble does not support opening the same directory read-write more than once, and this export is meant to run alongside the primitive's own live handle (already open read-write for normal request traffic), not compete with it for the same *pebble.DB instance.

func ExportPebbleStores

func ExportPebbleStores(ctx context.Context, specs []PebbleStoreSpec, outDir string) (map[string]int, error)

ExportPebbleStores runs ExportPebbleStore for every spec in order, stopping at the first error. A spec whose Dir does not exist (a tenant that has never used that primitive, e.g. no cal bookings ever made) is skipped, not an error -- an absent store is a legitimate "no data" case, not a failure, and pebble.Open on a missing directory would otherwise create an empty one as a side effect of merely checking.

func ExportSQLiteTable

func ExportSQLiteTable(ctx context.Context, db *sql.DB, tenantID uint16, spec SQLiteTableSpec, outDir string) (int, error)

ExportSQLiteTable runs one table's export: SELECT (filtered by tenant_id when TenantFiltered) every row, and write it as a JSON array to outDir/<spec.Name>.json. Column names and values are taken directly from the driver's own row description -- no struct mapping, no assumption about a table's schema beyond what the database itself reports at query time.

Returns the number of rows written and any error. An empty table (zero rows) still produces a valid file containing "[]", not an error and not a skipped file -- a caller reconstructing a tenant's full export later should not have to distinguish "this table had no data" from "this table was never exported".

func ExportSQLiteTables

func ExportSQLiteTables(ctx context.Context, db *sql.DB, tenantID uint16, specs []SQLiteTableSpec, outDir string) (map[string]int, error)

ExportSQLiteTables runs ExportSQLiteTable for every spec in order, stopping at the first error. Returns a map of table name to row count for every table that completed successfully before any failure -- a caller can log partial progress even on a failed run.

func SweepExpiredExports

func SweepExpiredExports(ctx context.Context, bs *blob.Store, ttl time.Duration) (gc.Report, error)

SweepExpiredExports deletes every blob under ExportKeyPrefix in bs whose StoredAt is older than ttl. Implements the same one-store, one-sweep shape as blob.GCWorker.Sweep -- a caller enumerating multiple tenants' stores (see pkg/server's blobManager.Sweep for the established iteration pattern this mirrors) calls this once per store and aggregates the returned gc.Report itself, the same way blobManager already aggregates blob.GCWorker's own reports.

A per-key deletion failure is counted in Report.Errors and does not stop the sweep -- one export blob that fails to delete (a permissions issue, a concurrent delete already in flight) should not prevent every other expired export in the same store from being cleaned up.

Types

type ErrTenantExportInFlight

type ErrTenantExportInFlight struct {
	ExistingTicket string
}

ErrTenantExportInFlight is returned by Submit when the given tenant already has a running export job -- the per-tenant throttle. The caller should return this to the client as a 429 with the existing ticket, not silently start a redundant second export of the same tenant's data.

func (*ErrTenantExportInFlight) Error

func (e *ErrTenantExportInFlight) Error() string

type Job

type Job struct {
	Ticket     string
	TenantID   tenant.TenantID
	Status     JobStatus
	BlobKey    string // set once Status == JobComplete
	Error      string // set once Status == JobFailed
	CreatedAt  time.Time
	FinishedAt time.Time
}

Job is one tenant's export job, tracked by ticket ID.

type JobManager

type JobManager struct {
	// contains filtered or unexported fields
}

JobManager tracks export jobs in memory and bounds how many run concurrently. Not persisted across a process restart -- a job in flight when the server restarts is simply lost, matching this package's own scope (data collection and job tracking, not a durable work queue); a caller needing that guarantee is a different, larger piece of work than what was asked for here.

func NewJobManager

func NewJobManager(maxConcurrent int) *JobManager

NewJobManager creates a job tracker allowing at most maxConcurrent export jobs to actually run (query/iterate/package) at once, server-wide, regardless of how many tenants have called Submit.

func (*JobManager) Status

func (m *JobManager) Status(ticket string) (*Job, bool)

Status returns the job for ticket, or nil, false if no such ticket is known (never existed, or -- not currently implemented -- was pruned; this package does not yet age out old completed jobs from memory).

func (*JobManager) Submit

func (m *JobManager) Submit(tenantID tenant.TenantID, work func() (*PackageResult, error)) (string, error)

Submit starts a new export job for tenantID, running work in a background goroutine bounded by the manager's own concurrency semaphore. Returns the new job's ticket ID immediately -- work has not necessarily started yet if the semaphore is currently full; Status() distinguishes "queued behind the concurrency limit" from "actually running" only implicitly (both report JobRunning -- this package does not currently expose queued-vs-executing as a separate state, since a caller polling Status has no actionable difference between the two).

Returns *ErrTenantExportInFlight, not a ticket, if tenantID already has a running job -- the per-tenant throttle. work is called with no arguments; a caller closes over whatever primaryDB/basePath/blobStore it needs (this keeps JobManager itself free of any dependency on ExportTenant's own signature, so it can track any long-running, eventually-blob-producing job, not just this specific export shape).

type JobStatus

type JobStatus string

JobStatus is an export job's lifecycle state.

const (
	JobRunning  JobStatus = "running"
	JobComplete JobStatus = "complete"
	JobFailed   JobStatus = "failed"
)

type PackageResult

type PackageResult struct {
	// Key is the blob key the export was stored under.
	Key string
	// SHA256 is the stored content's hash, as returned by the blob
	// store itself.
	SHA256 string
	// Bytes is the packaged zip's own size.
	Bytes int64
}

PackageResult describes a completed, blob-stored export.

func ExportTenant

func ExportTenant(ctx context.Context, primaryDB *sql.DB, basePath string, tenantID tenant.TenantID, bs *blob.Store, exportKey string) (*PackageResult, error)

ExportTenant runs a complete export for one tenant: every table in its primary store (filtered where the table is shared, unfiltered where it's already tenant-scoped), every table in its dedicated loc/obj files if those files exist, every per-tenant Pebble store (ts, cal's occupancy index, bal's rollup) if that store's directory exists, packages the whole collection into one zip, and stores it as a blob under exportKey via bs.

Staging happens in a fresh temp directory created directly under the tenant's own root (storelayout.TenantRoot) -- not the OS temp directory -- per the design settled directly with the team: keeping staging co-located with the tenant's own data rather than a shared system temp path. Removed unconditionally on return, success or failure, via defer -- a failed export must not leave partial JSON files behind under the tenant's own directory.

primaryDB is the *sql.DB already open against this tenant's primary store (the caller's own handle -- either the tenant's per-file db or the shared db, per SQLitePerFileTenants; ExportTenant does not open or care which mode is in effect, it just queries what it's given).

func PackageAndStore

func PackageAndStore(ctx context.Context, srcDir string, bs *blob.Store, key string) (*PackageResult, error)

PackageAndStore zips every file directly inside srcDir (non- recursive -- the JSON files ExportSQLiteTables/ExportPebbleStores write are already flat, one file per table/store) and stores the result as a blob under key via bs.Put. srcDir itself is not removed; the caller owns cleanup (see this package's own doc comment on the temp-directory convention -- staged under the tenant's own directory, moved into blobs, then removed by the caller once this returns successfully).

type PebbleKV

type PebbleKV struct {
	Key   string `json:"key_b64"`
	Value string `json:"value_b64"`
}

PebbleKV is one key/value pair from a Pebble store, base64-encoded.

Unconditional base64 for both fields, not a per-value guess at UTF-8-vs-binary: unlike the SQLite side (where this package knows specific TEXT columns hold JSON/decimal strings), a Pebble store's key and value encoding is internal to whichever primitive owns it (bal's rollup deltas, cal's occupancy bitmap, ts's own encoding) -- this package has no reason to assume any of it is text, and guessing wrong per-key would make the same store's export inconsistently shaped from one key to the next.

type PebbleStoreSpec

type PebbleStoreSpec struct {
	Dir  string
	Name string
}

PebbleStoreSpec names one per-tenant Pebble store to export -- Dir is its own directory (already tenant-scoped, e.g. storelayout.TenantBalRollupDir(base, tenantID)), Name is the output file's own base name (e.g. "bal_rollup").

type SQLiteTableSpec

type SQLiteTableSpec struct {
	// Name is a label for this table, used for the output JSON
	// filename and in error messages. When NameFunc is nil, Name is
	// ALSO the literal table name to query (optionally prefixed, see
	// TenantPrefixed) -- when NameFunc is set, Name is label-only and
	// NameFunc alone determines the real table name.
	Name string
	// NameFunc, when set, is the authoritative source for this
	// table's real name -- e.g. tenant.TenantID.NodesTableName,
	// which this package MUST call directly rather than re-derive the
	// same "prefix + suffix" pattern by hand. This is not a style
	// preference: a hand-rolled version of this exact logic already
	// caused two real bugs in this package (a missing table,
	// NodeSchemaTableName's own t<XXXX>_n_sch, that a generic
	// prefix+suffix builder had no way to know about since the
	// abbreviation "n_sch" isn't Name+prefix-shaped at all; and a
	// query against the literal, unprefixed string "nodes", which
	// doesn't exist -- both caught by TestIntegration_BlobExport_
	// FullAsyncFlow against a real server, not assumed away).
	// TenantPrefixed and TenantFiltered are both ignored when NameFunc
	// is set -- NameFunc already encodes the correct scoping.
	NameFunc func(tenantID uint16) string
	// TenantPrefixed is true when this table's REAL name is Name with
	// the tenant's own table prefix prepended
	// (tenant.TenantID.TablePrefix(), e.g. "bal_accounts" ->
	// "t0000_bal_accounts") -- verified directly against source for
	// every table this applies to (pkg/bal/store.go's own
	// accountsTable/journalTable/balancesTable and siblings in
	// rollup.go/seal.go, all "s.prefix + literal suffix", confirmed
	// simple with no abbreviation surprises, unlike the entity/graph
	// tables above). Used only when NameFunc is nil.
	TenantPrefixed bool
	// TenantFiltered is true when this table is shared across all
	// tenants and needs a "WHERE tenant_id = ?" filter (e.g.
	// cal_bookings, dxp_txn, fsm_machines -- confirmed directly
	// against their own real WHERE clauses, not assumed). Used only
	// when NameFunc is nil.
	TenantFiltered bool
}

SQLiteTableSpec describes one table to export.

type TenantDataSummary added in v0.30.38

type TenantDataSummary struct {
	// Primary holds one entry per PrimaryStoreTables spec, keyed by
	// spec.Name.
	Primary map[string]int `json:"primary"`
	// Loc holds one entry per LocStoreTables spec (loc.db). Present
	// but zero-valued for every entry when the tenant has never used
	// /loc at all (loc.db doesn't exist yet) -- distinct from a
	// tenant that used /loc and then deleted everything, which this
	// summary cannot and does not try to distinguish.
	Loc map[string]int `json:"loc"`
	// Obj holds one entry per ObjStoreTables spec (obj.db) -- this is
	// what closes XOT209's own emptiness-check gap for obj, ahead of
	// or independent of that item's own general REST enumeration
	// endpoint.
	Obj map[string]int `json:"obj"`
	// TS, CalIndex, and BalRollup are key counts in the three
	// per-tenant Pebble stores ExportTenant itself exports
	// (storelayout.TenantTSDir/TenantCalDir/TenantBalRollupDir).
	TS        int `json:"ts"`
	CalIndex  int `json:"cal_index"`
	BalRollup int `json:"bal_rollup"`
	// Blob is the tenant's own uploaded blob key count -- NOT part of
	// ExportTenant's own captured set (blob.Store there is only ever
	// the destination for a packaged export, never a source of
	// counted data; see this file's own XOT216 note on Summarize).
	// Always 0 when Summarize's own bs parameter is nil (blob
	// disabled on this server, or unavailable for this call) -- a
	// disabled primitive summarizing to zero, matching every other
	// per-store field's own "never used this primitive" treatment,
	// not a sentinel meaning "unknown."
	Blob int `json:"blob"`
}

TenantDataSummary is per-store row/key counts across every place a tenant's own data can live -- the same set ExportTenant itself captures, no more and no less. A tenant summary where Empty() reports true is empty by that same real definition, not a fifth, separately-maintained notion of what counts.

func Summarize added in v0.30.38

func Summarize(ctx context.Context, primaryDB *sql.DB, basePath string, tenantID tenant.TenantID, bs *blob.Store) (*TenantDataSummary, error)

func (*TenantDataSummary) Empty added in v0.30.38

func (s *TenantDataSummary) Empty() bool

Empty reports whether every count in s is genuinely zero -- the single boolean xoluman's own document named as the minimum contract, alongside the full per-store breakdown its document had a mild preference for ("would be more useful"). This summary provides both from one computation rather than forcing a caller to choose.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL