Documentation
¶
Overview ¶
Package externalwrite implements writing rows of a query/LOAD into the backing files of a writable external table. A table becomes writable when it is created with a WRITE_FILE_PATTERN option; each parallel write pipeline owns one ExternalWriter and produces exactly one output file.
Index ¶
Constants ¶
const ( FormatCSV = "csv" FormatJSONLine = "jsonline" )
const MinUniqueRandomDigits = 6
MinUniqueRandomDigits is the smallest %nN width that qualifies as a per-writer uniqueness directive: fewer digits (e.g. %1N with 10 outcomes) collide between parallel pipelines with realistic probability.
Variables ¶
This section is empty.
Functions ¶
func ExpandFilePattern ¶
ExpandFilePattern expands a strftime(3) pattern into a concrete path.
In addition to the standard strftime directives it supports two MatrixOne extensions used by writable external tables:
%nN -> n random decimal digits (n is a decimal count, e.g. %6N -> "492013") %U -> a freshly generated UUID
t is the timestamp the pattern is evaluated against (typically the statement start time). Time directives are rendered in UTC, so every pipeline — local or on a remote CN whose host may run in a different OS time zone — expands the same instant to the same path. It is the caller's responsibility to make the pattern produce unique names across parallel writers (use %U or %nN); the standard directives alone are not unique.
func PatternHasUniqueDirective ¶
PatternHasUniqueDirective reports whether the pattern contains a directive that yields a distinct value per writer (%U, or %nN with n >= MinUniqueRandomDigits). Patterns without one expand to the same path in every parallel pipeline of a statement (and in every statement within the same time-directive granularity), so concurrent writers would clobber each other; DDL validation rejects them.
Types ¶
type ExternalWriter ¶
type ExternalWriter interface {
// WriteBatch encodes every row of bat and appends it to the output file.
// The file is created lazily on the first non-empty batch, so a pipeline
// that never receives rows produces no file.
WriteBatch(ctx context.Context, bat *batch.Batch) error
// Close flushes and finalizes the output file and returns the number of
// rows written. It is a no-op (rowsWritten == 0) if no file was opened.
Close(ctx context.Context) (rowsWritten uint64, err error)
// Abort discards the output file instead of finalizing it. Used on pipeline
// failure so a half-written file never becomes visible to readers of the
// external table. Best-effort; a no-op if no file was opened.
Abort(ctx context.Context)
}
ExternalWriter encodes batches and appends them to a single backing file.
func NewExternalWriter ¶
func NewExternalWriter(proc *process.Process, cfg WriterConfig) ExternalWriter
NewExternalWriter builds an ExternalWriter for one pipeline. Defaults are filled for any unset CSV formatting option.
type WriterConfig ¶
type WriterConfig struct {
// Pattern is the WRITE_FILE_PATTERN strftime template. It must resolve to a
// stage:// URL after expansion.
Pattern string
// Format is "csv" or "jsonline".
Format string
// Attrs are the column names in output order (used for the CSV header and as
// JSONLine object keys).
Attrs []string
// CSV formatting. Defaults mirror LOAD/SELECT INTO OUTFILE defaults.
FieldTerminator []byte // default ","
LineTerminator []byte // default "\n"
LineStartingBy []byte // LINES STARTED BY prefix, written before each record
// Comment is the table's COMMENT marker. The reader skips a line whose raw
// prefix matches it (before LINES STARTING BY is consumed), so when set the
// writer encloses the first field of any row whose unenclosed line prefix
// would match — only then, never otherwise. Empty disables the marker.
Comment []byte
EnclosedBy byte // default '"' (the external reader's enclosure)
// EscapedBy is the FIELDS ESCAPED BY character (default '\\'). The writer
// doubles it wherever it appears in a value, matching the reader's
// unescaping. NoEscape disables escaping entirely (ESCAPED BY ”), which
// the reader mirrors by skipping unescape processing.
EscapedBy byte
NoEscape bool
Header bool // write a CSV header line
// Stmt is the timestamp the pattern is evaluated against (statement start).
Stmt time.Time
// TimeZone is used to render TIMESTAMP values; defaults to UTC.
TimeZone *time.Location
}
WriterConfig describes how one external-table write pipeline should encode and place its output file.
Source Files