Documentation
¶
Overview ¶
Package ingest provides micro-batch accumulation and flushing to object storage.
Index ¶
- type Config
- type Ingester
- func (ing *Ingester) DeferManifestCommit()
- func (ing *Ingester) FlushAll(ctx context.Context) error
- func (ing *Ingester) Ingest(ctx context.Context, rows []map[string]any) error
- func (ing *Ingester) PendingFiles() []catalog.PendingFile
- func (ing *Ingester) RefuseWith(err error)
- func (ing *Ingester) Start()
- func (ing *Ingester) Stop(ctx context.Context) error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
MaxBufferSize int // max bytes before flush (default 128 MB)
MaxBufferRows int // max rows before flush (default 1M)
FlushInterval time.Duration // max time before flush (default 60s)
RowGroupSize int // rows per row group in Parquet (default 128K)
MinFlushRows int // min rows to flush on timer (default 100; 0 = no minimum)
}
Config configures the micro-batch ingester.
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig returns default ingest configuration.
type Ingester ¶
type Ingester struct {
// contains filtered or unexported fields
}
Ingester accumulates rows and periodically flushes them as Parquet files.
func New ¶
func New(cat *catalog.Catalog, tableName string, schema parquet.Schema, partKeys []string, cfg Config) *Ingester
New creates a new Ingester for the given table.
func (*Ingester) DeferManifestCommit ¶ added in v0.18.20
func (ing *Ingester) DeferManifestCommit()
DeferManifestCommit stops this Ingester from registering its flushed files in the manifest, holding them in PendingFiles instead.
It exists for one caller: a DML statement, which must commit the rows it WRITES and the delete markers that remove the rows they REPLACE in one CAS or neither (#691). The default — a manifest commit per flushed file — makes an UPDATE two independent commits, so a marker commit refused at the end left the replacement rows beside the originals, reporting an error over a table that now had both.
Call it before the first Ingest, and never on an Ingester whose background flusher is Start()ed: the pending list is drained by the caller, not by a timer. The files are already durable in the object store when they land here; only their manifest entries are held.
func (*Ingester) PendingFiles ¶ added in v0.18.20
func (ing *Ingester) PendingFiles() []catalog.PendingFile
PendingFiles returns the flushed files whose manifest entries are being held, and clears the list.
func (*Ingester) RefuseWith ¶ added in v0.18.6
Ingest adds rows to the buffer. Rows are partitioned based on partition key values. Each row is validated against the table schema before buffering. RefuseWith arms this Ingester to reject every Ingest and FlushAll with err. Used by the constructor's caller for a schema the ingest door refuses.
Source Files