Documentation
¶
Overview ¶
Package tenantexport implements per-tenant data export as a pair of iterate-and-serialize algorithms, one per storage engine xolu uses (SQLite, Pebble) -- not a reconstruction of a valid subset database file. Each source (a SQLite table, a Pebble store) becomes one JSON file: every row or key/value pair in that source, serialized, no schema/constraint/index fidelity attempted or needed.
Design settled directly with the team (2026-08-03), correcting an earlier, substantially overcomplicated draft that tried to reconstruct a byte-valid SQLite file containing only one tenant's tables (ATTACH + CREATE TABLE AS SELECT, preserving schema/indexes). That problem doesn't need solving: a JSON export doesn't care about SQL structure fidelity, only about the data itself.
See docs/proposals/tenant-export.md for the full design (async job flow, blob-backed result, TTL, throttling). Wired into pkg/server/blob_export_handlers.go's POST/GET .../blob/export routes.
Index ¶
- Constants
- Variables
- func ExportPebbleStore(ctx context.Context, dir, name, outDir string) (int, error)
- func ExportPebbleStores(ctx context.Context, specs []PebbleStoreSpec, outDir string) (map[string]int, error)
- func ExportSQLiteTable(ctx context.Context, db *sql.DB, tenantID uint16, spec SQLiteTableSpec, ...) (int, error)
- func ExportSQLiteTables(ctx context.Context, db *sql.DB, tenantID uint16, specs []SQLiteTableSpec, ...) (map[string]int, error)
- func SweepExpiredExports(ctx context.Context, bs *blob.Store, ttl time.Duration) (gc.Report, error)
- type ErrTenantExportInFlight
- type Job
- type JobManager
- type JobStatus
- type PackageResult
- type PebbleKV
- type PebbleStoreSpec
- type SQLiteTableSpec
- type TenantDataSummary
Constants ¶
const ExportKeyPrefix = "export-"
ExportKeyPrefix is the fixed prefix every export blob is stored under (see handleBlobExportStart's own exportKey construction, "export-%d.zip") -- SweepExpiredExports lists exactly this prefix, never touching a caller's own blobs that happen to start differently.
Variables ¶
var LocStoreTables = []SQLiteTableSpec{ {Name: "locations", TenantFiltered: false}, {Name: "loc_patterns", TenantFiltered: false}, {Name: "loc_capacity", TenantFiltered: false}, {Name: "fences", TenantFiltered: false}, {Name: "loc_fence_capacity", TenantFiltered: false}, {Name: "loc_fence_membership", TenantFiltered: false}, {Name: "loc_assignment", TenantFiltered: false}, {Name: "loc_journal", TenantFiltered: false}, }
LocStoreTables is every table in loc's own dedicated per-tenant file (storelayout.TenantLocDir) -- the whole FILE is already one tenant's own (T-115, wave 9), so none of these need a tenant_id filter, unlike PrimaryStoreTables' global tables.
var ObjStoreTables = []SQLiteTableSpec{ {Name: "obj_subjects", TenantFiltered: false}, {Name: "obj_position", TenantFiltered: false}, {Name: "obj_journal", TenantFiltered: false}, }
ObjStoreTables is every table in obj's own dedicated per-tenant file (storelayout.TenantObjDir) -- same reasoning as LocStoreTables.
var PrimaryStoreTables = []SQLiteTableSpec{ {Name: "nodes", NameFunc: func(t uint16) string { return tenant.TenantID(t).NodesTableName() }}, {Name: "edges", NameFunc: func(t uint16) string { return tenant.TenantID(t).EdgePropsTableName() }}, {Name: "graph", NameFunc: func(t uint16) string { return tenant.TenantID(t).GraphTableName() }}, {Name: "eseq", NameFunc: func(t uint16) string { return tenant.TenantID(t).EdgeSeqTableName() }}, {Name: "nseq", NameFunc: func(t uint16) string { return tenant.TenantID(t).NodeSeqTableName() }}, {Name: "e_sch", NameFunc: func(t uint16) string { return tenant.TenantID(t).EdgeSchemaTableName() }}, {Name: "n_sch", NameFunc: func(t uint16) string { return tenant.TenantID(t).NodeSchemaTableName() }}, {Name: "bal_accounts", TenantPrefixed: true}, {Name: "bal_balances", TenantPrefixed: true}, {Name: "bal_checkpoints", TenantPrefixed: true}, {Name: "bal_journal", TenantPrefixed: true}, {Name: "bal_seal", TenantPrefixed: true}, {Name: "cal_bookings", TenantFiltered: true}, {Name: "cal_calendars", TenantFiltered: true}, {Name: "cal_participants", TenantFiltered: true}, {Name: "cal_ord_seq", TenantFiltered: true}, {Name: "dxp_defs", TenantFiltered: true}, {Name: "dxp_id_seq", TenantFiltered: true}, {Name: "dxp_txn", TenantFiltered: true}, {Name: "event_defs", TenantFiltered: true}, {Name: "event_delivery_log", TenantFiltered: true}, {Name: "fsm_definitions", TenantFiltered: true}, {Name: "fsm_history", TenantFiltered: true}, {Name: "fsm_id_seq", TenantFiltered: true}, {Name: "fsm_machines", TenantFiltered: true}, {Name: "fsm_terminal_states", TenantFiltered: true}, {Name: "gen_definitions", TenantFiltered: true}, {Name: "sequences", TenantFiltered: true}, {Name: "entity_meta", TenantFiltered: true}, }
PrimaryStoreTables is every table in a tenant's own primary store (store/xolu.db, shared or per-file per SQLitePerFileTenants) that holds real tenant data -- verified directly against each subsystem's own real query/schema code (grep for actual WHERE tenant_id=... use or, where no query happened to be caught that way, the literal CREATE TABLE itself -- cal_participants specifically was confirmed via schema, not a query match), not assumed from naming convention alone.
Deliberately excluded, and why:
- t0000_efts*/t0000_nfts* (full-text search indexes): derived, rebuildable from the entity data already being exported: not source data.
- schema_version, schema_version_v2, tenants: server/system-level tables, not scoped to any one tenant.
This list is authoritative until a new subsystem adds tables of its own -- extend it there, not by guessing the pattern holds.
Functions ¶
func ExportPebbleStore ¶
ExportPebbleStore iterates every key/value pair in the Pebble database at dir and writes them as a JSON array to outDir/<name>.json. dir is expected to already be scoped to one tenant (storelayout.TenantBalRollupDir, TenantCalDir, TenantTSDir, etc. each return a per-tenant directory) -- there is no tenant_id filtering step here because none is needed, unlike the SQLite side's shared tables.
Opens its own read-only handle rather than requiring an existing *pebble.DB be passed in -- Pebble does not support opening the same directory read-write more than once, and this export is meant to run alongside the primitive's own live handle (already open read-write for normal request traffic), not compete with it for the same *pebble.DB instance.
func ExportPebbleStores ¶
func ExportPebbleStores(ctx context.Context, specs []PebbleStoreSpec, outDir string) (map[string]int, error)
ExportPebbleStores runs ExportPebbleStore for every spec in order, stopping at the first error. A spec whose Dir does not exist (a tenant that has never used that primitive, e.g. no cal bookings ever made) is skipped, not an error -- an absent store is a legitimate "no data" case, not a failure, and pebble.Open on a missing directory would otherwise create an empty one as a side effect of merely checking.
func ExportSQLiteTable ¶
func ExportSQLiteTable(ctx context.Context, db *sql.DB, tenantID uint16, spec SQLiteTableSpec, outDir string) (int, error)
ExportSQLiteTable runs one table's export: SELECT (filtered by tenant_id when TenantFiltered) every row, and write it as a JSON array to outDir/<spec.Name>.json. Column names and values are taken directly from the driver's own row description -- no struct mapping, no assumption about a table's schema beyond what the database itself reports at query time.
Returns the number of rows written and any error. An empty table (zero rows) still produces a valid file containing "[]", not an error and not a skipped file -- a caller reconstructing a tenant's full export later should not have to distinguish "this table had no data" from "this table was never exported".
func ExportSQLiteTables ¶
func ExportSQLiteTables(ctx context.Context, db *sql.DB, tenantID uint16, specs []SQLiteTableSpec, outDir string) (map[string]int, error)
ExportSQLiteTables runs ExportSQLiteTable for every spec in order, stopping at the first error. Returns a map of table name to row count for every table that completed successfully before any failure -- a caller can log partial progress even on a failed run.
func SweepExpiredExports ¶
SweepExpiredExports deletes every blob under ExportKeyPrefix in bs whose StoredAt is older than ttl. Implements the same one-store, one-sweep shape as blob.GCWorker.Sweep -- a caller enumerating multiple tenants' stores (see pkg/server's blobManager.Sweep for the established iteration pattern this mirrors) calls this once per store and aggregates the returned gc.Report itself, the same way blobManager already aggregates blob.GCWorker's own reports.
A per-key deletion failure is counted in Report.Errors and does not stop the sweep -- one export blob that fails to delete (a permissions issue, a concurrent delete already in flight) should not prevent every other expired export in the same store from being cleaned up.
Types ¶
type ErrTenantExportInFlight ¶
type ErrTenantExportInFlight struct {
ExistingTicket string
}
ErrTenantExportInFlight is returned by Submit when the given tenant already has a running export job -- the per-tenant throttle. The caller should return this to the client as a 429 with the existing ticket, not silently start a redundant second export of the same tenant's data.
func (*ErrTenantExportInFlight) Error ¶
func (e *ErrTenantExportInFlight) Error() string
type Job ¶
type Job struct {
Ticket string
TenantID tenant.TenantID
Status JobStatus
BlobKey string // set once Status == JobComplete
Error string // set once Status == JobFailed
CreatedAt time.Time
FinishedAt time.Time
}
Job is one tenant's export job, tracked by ticket ID.
type JobManager ¶
type JobManager struct {
// contains filtered or unexported fields
}
JobManager tracks export jobs in memory and bounds how many run concurrently. Not persisted across a process restart -- a job in flight when the server restarts is simply lost, matching this package's own scope (data collection and job tracking, not a durable work queue); a caller needing that guarantee is a different, larger piece of work than what was asked for here.
func NewJobManager ¶
func NewJobManager(maxConcurrent int) *JobManager
NewJobManager creates a job tracker allowing at most maxConcurrent export jobs to actually run (query/iterate/package) at once, server-wide, regardless of how many tenants have called Submit.
func (*JobManager) Status ¶
func (m *JobManager) Status(ticket string) (*Job, bool)
Status returns the job for ticket, or nil, false if no such ticket is known (never existed, or -- not currently implemented -- was pruned; this package does not yet age out old completed jobs from memory).
func (*JobManager) Submit ¶
func (m *JobManager) Submit(tenantID tenant.TenantID, work func() (*PackageResult, error)) (string, error)
Submit starts a new export job for tenantID, running work in a background goroutine bounded by the manager's own concurrency semaphore. Returns the new job's ticket ID immediately -- work has not necessarily started yet if the semaphore is currently full; Status() distinguishes "queued behind the concurrency limit" from "actually running" only implicitly (both report JobRunning -- this package does not currently expose queued-vs-executing as a separate state, since a caller polling Status has no actionable difference between the two).
Returns *ErrTenantExportInFlight, not a ticket, if tenantID already has a running job -- the per-tenant throttle. work is called with no arguments; a caller closes over whatever primaryDB/basePath/blobStore it needs (this keeps JobManager itself free of any dependency on ExportTenant's own signature, so it can track any long-running, eventually-blob-producing job, not just this specific export shape).
type PackageResult ¶
type PackageResult struct {
// Key is the blob key the export was stored under.
Key string
// SHA256 is the stored content's hash, as returned by the blob
// store itself.
SHA256 string
// Bytes is the packaged zip's own size.
Bytes int64
}
PackageResult describes a completed, blob-stored export.
func ExportTenant ¶
func ExportTenant(ctx context.Context, primaryDB *sql.DB, basePath string, tenantID tenant.TenantID, bs *blob.Store, exportKey string) (*PackageResult, error)
ExportTenant runs a complete export for one tenant: every table in its primary store (filtered where the table is shared, unfiltered where it's already tenant-scoped), every table in its dedicated loc/obj files if those files exist, every per-tenant Pebble store (ts, cal's occupancy index, bal's rollup) if that store's directory exists, packages the whole collection into one zip, and stores it as a blob under exportKey via bs.
Staging happens in a fresh temp directory created directly under the tenant's own root (storelayout.TenantRoot) -- not the OS temp directory -- per the design settled directly with the team: keeping staging co-located with the tenant's own data rather than a shared system temp path. Removed unconditionally on return, success or failure, via defer -- a failed export must not leave partial JSON files behind under the tenant's own directory.
primaryDB is the *sql.DB already open against this tenant's primary store (the caller's own handle -- either the tenant's per-file db or the shared db, per SQLitePerFileTenants; ExportTenant does not open or care which mode is in effect, it just queries what it's given).
func PackageAndStore ¶
func PackageAndStore(ctx context.Context, srcDir string, bs *blob.Store, key string) (*PackageResult, error)
PackageAndStore zips every file directly inside srcDir (non- recursive -- the JSON files ExportSQLiteTables/ExportPebbleStores write are already flat, one file per table/store) and stores the result as a blob under key via bs.Put. srcDir itself is not removed; the caller owns cleanup (see this package's own doc comment on the temp-directory convention -- staged under the tenant's own directory, moved into blobs, then removed by the caller once this returns successfully).
type PebbleKV ¶
PebbleKV is one key/value pair from a Pebble store, base64-encoded.
Unconditional base64 for both fields, not a per-value guess at UTF-8-vs-binary: unlike the SQLite side (where this package knows specific TEXT columns hold JSON/decimal strings), a Pebble store's key and value encoding is internal to whichever primitive owns it (bal's rollup deltas, cal's occupancy bitmap, ts's own encoding) -- this package has no reason to assume any of it is text, and guessing wrong per-key would make the same store's export inconsistently shaped from one key to the next.
type PebbleStoreSpec ¶
PebbleStoreSpec names one per-tenant Pebble store to export -- Dir is its own directory (already tenant-scoped, e.g. storelayout.TenantBalRollupDir(base, tenantID)), Name is the output file's own base name (e.g. "bal_rollup").
type SQLiteTableSpec ¶
type SQLiteTableSpec struct {
// Name is a label for this table, used for the output JSON
// filename and in error messages. When NameFunc is nil, Name is
// ALSO the literal table name to query (optionally prefixed, see
// TenantPrefixed) -- when NameFunc is set, Name is label-only and
// NameFunc alone determines the real table name.
Name string
// NameFunc, when set, is the authoritative source for this
// table's real name -- e.g. tenant.TenantID.NodesTableName,
// which this package MUST call directly rather than re-derive the
// same "prefix + suffix" pattern by hand. This is not a style
// preference: a hand-rolled version of this exact logic already
// caused two real bugs in this package (a missing table,
// NodeSchemaTableName's own t<XXXX>_n_sch, that a generic
// prefix+suffix builder had no way to know about since the
// abbreviation "n_sch" isn't Name+prefix-shaped at all; and a
// query against the literal, unprefixed string "nodes", which
// doesn't exist -- both caught by TestIntegration_BlobExport_
// FullAsyncFlow against a real server, not assumed away).
// TenantPrefixed and TenantFiltered are both ignored when NameFunc
// is set -- NameFunc already encodes the correct scoping.
NameFunc func(tenantID uint16) string
// TenantPrefixed is true when this table's REAL name is Name with
// the tenant's own table prefix prepended
// (tenant.TenantID.TablePrefix(), e.g. "bal_accounts" ->
// "t0000_bal_accounts") -- verified directly against source for
// every table this applies to (pkg/bal/store.go's own
// accountsTable/journalTable/balancesTable and siblings in
// rollup.go/seal.go, all "s.prefix + literal suffix", confirmed
// simple with no abbreviation surprises, unlike the entity/graph
// tables above). Used only when NameFunc is nil.
TenantPrefixed bool
// TenantFiltered is true when this table is shared across all
// tenants and needs a "WHERE tenant_id = ?" filter (e.g.
// cal_bookings, dxp_txn, fsm_machines -- confirmed directly
// against their own real WHERE clauses, not assumed). Used only
// when NameFunc is nil.
TenantFiltered bool
}
SQLiteTableSpec describes one table to export.
type TenantDataSummary ¶ added in v0.30.38
type TenantDataSummary struct {
// Primary holds one entry per PrimaryStoreTables spec, keyed by
// spec.Name.
Primary map[string]int `json:"primary"`
// Loc holds one entry per LocStoreTables spec (loc.db). Present
// but zero-valued for every entry when the tenant has never used
// /loc at all (loc.db doesn't exist yet) -- distinct from a
// tenant that used /loc and then deleted everything, which this
// summary cannot and does not try to distinguish.
Loc map[string]int `json:"loc"`
// Obj holds one entry per ObjStoreTables spec (obj.db) -- this is
// what closes XOT209's own emptiness-check gap for obj, ahead of
// or independent of that item's own general REST enumeration
// endpoint.
Obj map[string]int `json:"obj"`
// TS, CalIndex, and BalRollup are key counts in the three
// per-tenant Pebble stores ExportTenant itself exports
// (storelayout.TenantTSDir/TenantCalDir/TenantBalRollupDir).
TS int `json:"ts"`
CalIndex int `json:"cal_index"`
BalRollup int `json:"bal_rollup"`
// Blob is the tenant's own uploaded blob key count -- NOT part of
// ExportTenant's own captured set (blob.Store there is only ever
// the destination for a packaged export, never a source of
// counted data; see this file's own XOT216 note on Summarize).
// Always 0 when Summarize's own bs parameter is nil (blob
// disabled on this server, or unavailable for this call) -- a
// disabled primitive summarizing to zero, matching every other
// per-store field's own "never used this primitive" treatment,
// not a sentinel meaning "unknown."
Blob int `json:"blob"`
}
TenantDataSummary is per-store row/key counts across every place a tenant's own data can live -- the same set ExportTenant itself captures, no more and no less. A tenant summary where Empty() reports true is empty by that same real definition, not a fifth, separately-maintained notion of what counts.
func (*TenantDataSummary) Empty ¶ added in v0.30.38
func (s *TenantDataSummary) Empty() bool
Empty reports whether every count in s is genuinely zero -- the single boolean xoluman's own document named as the minimum contract, alongside the full per-store breakdown its document had a mild preference for ("would be more useful"). This summary provides both from one computation rather than forcing a caller to choose.