Documentation
¶
Overview ¶
Package export implements the `export` subcommand's core: reading the current state of every configured workspace out of the repository and writing it, one table per entity, to a pluggable Sink (BigQuery today). The Exporter is sink-agnostic — it describes a table as typed columns plus rows of natural Go values and streams those rows into the Sink, which owns all backend-specific concerns (schema evolution, full-refresh, encoding). This keeps a future Cloud Storage sink a drop-in without touching the read/normalize logic here.
Rows are streamed rather than assembled: a table is opened with BeginTable, fed with AppendRows as the rows are read, and published with Commit. That is what keeps the exporter's peak memory bounded by one read's worth of rows instead of by the workspace's total volume — job_run_events alone reaches gigabytes on a busy workspace, which is enough to end the process.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func WriteTable ¶ added in v0.3.0
func WriteTable(ctx context.Context, sink Sink, namespace, name string, columns []Column, rows []map[string]any) error
WriteTable is the one-shot form of the streaming protocol: open the table, append every row, publish. It exists so a caller with the whole table already in hand does not have to repeat the abort-on-failure bookkeeping, which is what keeps a half-written refresh from ever reaching the destination.
Types ¶
type Column ¶
type Column struct {
// Name is the column name (fixed columns are snake_case; custom fields are
// "field_<id>").
Name string
// Type is the logical column type.
Type ColumnType
// Repeated marks an array column (ARRAY<Type>).
Repeated bool
// Nullable marks a nullable column. A non-nullable, non-repeated column is
// REQUIRED in the sink's schema.
Nullable bool
}
Column describes one output column.
type ColumnType ¶
type ColumnType string
ColumnType is the backend-neutral logical type of a Table column. A Sink maps it to its own type system (e.g. BigQuery STRING/INT64/...).
const ( TypeString ColumnType = "STRING" TypeInt ColumnType = "INT64" TypeFloat ColumnType = "FLOAT64" TypeBool ColumnType = "BOOL" TypeTimestamp ColumnType = "TIMESTAMP" )
type Exporter ¶
type Exporter struct {
// contains filtered or unexported fields
}
Exporter reads workspace data from the repository and writes it to a Sink. It holds no mutable state across a run.
func New ¶
func New(repo interfaces.Repository, sink Sink, opts ...Option) *Exporter
New builds an Exporter. repo and sink are required; opts are optional. The per-workspace privacy policy travels on each Target, not here.
func (*Exporter) Run ¶
Run exports every target. A failure on one target is logged and collected but does not stop the others; all collected failures are returned joined so the caller (and its error reporter) sees every one.
Single-instance assumption: Run is designed to be invoked by ONE process at a time and takes no distributed lock. Each table's refresh is staged in full before it replaces the destination, so overlapping runs cannot interleave into a half-written table — every snapshot is complete, and the destination ends up holding whichever run swapped last. This is a deliberate, documented constraint (the export is a singly-run batch job, e.g. a scheduled task with no overlap), not an oversight — see docs/export.md.
type Option ¶
type Option func(*Exporter)
Option customizes an Exporter.
func WithTablePrefix ¶
WithTablePrefix prepends prefix to every table name. It is empty in production (tables are named exactly cases/actions/...); tests use it to write uniquely-named tables into a shared dataset without recreating the dataset.
type Sink ¶
type Sink interface {
// BeginTable starts a full refresh of namespace.name with the given schema.
BeginTable(ctx context.Context, namespace, name string, columns []Column) (TableWriter, error)
io.Closer
}
Sink is a destination that fully replaces (洗替) a table's schema and rows. Implementations MUST make each BeginTable/Commit pair a full refresh of the named table within the given namespace, and MUST leave the destination at its previous contents when the pair ends in Abort or in any error.
type TableWriter ¶ added in v0.3.0
type TableWriter interface {
// AppendRows stages rows. It may flush to the backend at its discretion,
// so the caller must not assume anything is durable until Commit.
AppendRows(ctx context.Context, rows []map[string]any) error
// Commit publishes everything staged as the table's new full contents.
Commit(ctx context.Context) error
// Abort discards what was staged and leaves the destination untouched.
// It is a no-op after a successful Commit, so it is safe to defer.
Abort(ctx context.Context)
}
TableWriter is one table's in-progress full refresh. Row values are natural Go types keyed by column name; a missing key is NULL. Backend-specific encoding (e.g. TIMESTAMP -> microseconds) is the Sink's job.
A writer is used by a single goroutine and exactly once: append as many batches as the caller has, then either Commit or Abort.
type Target ¶
type Target struct {
Entry *model.WorkspaceEntry
Namespace string
// ExcludePrivate, when true, omits this workspace's private Cases (and their
// Actions / Memos). It is resolved per workspace by the caller.
ExcludePrivate bool
}
Target binds one workspace to its destination namespace (a BigQuery dataset) and its per-workspace privacy policy.
Source Files