Documentation
¶
Overview ¶
Package ingest provides micro-batch accumulation and flushing to object storage.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
MaxBufferSize int // max bytes before flush (default 128 MB)
MaxBufferRows int // max rows before flush (default 1M)
FlushInterval time.Duration // max time before flush (default 60s)
RowGroupSize int // rows per row group in Parquet (default 128K)
MinFlushRows int // min rows to flush on timer (default 100; 0 = no minimum)
}
Config configures the micro-batch ingester.
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig returns default ingest configuration.
type Ingester ¶
type Ingester struct {
// contains filtered or unexported fields
}
Ingester accumulates rows and periodically flushes them as Parquet files.
func New ¶
func New(cat *catalog.Catalog, tableName string, schema parquet.Schema, partKeys []string, cfg Config) *Ingester
New creates a new Ingester for the given table.
func (*Ingester) RefuseWith ¶ added in v0.18.6
Ingest adds rows to the buffer. Rows are partitioned based on partition key values. Each row is validated against the table schema before buffering. RefuseWith arms this Ingester to reject every Ingest and FlushAll with err. Used by the constructor's caller for a schema the ingest door refuses.
Source Files
Click to show internal directories.
Click to hide internal directories.