Documentation
¶
Index ¶
- func NewParquetFieldTypeMapper() warehouse.FieldTypeMapper[SpecificParquetType]
- type CSVFormatOption
- type FilesDriver
- func (sd *FilesDriver) AddColumn(table string, field *arrow.Field) error
- func (sd *FilesDriver) Close() error
- func (sd *FilesDriver) CreateTable(table string, schema *arrow.Schema) error
- func (sd *FilesDriver) MissingColumns(table string, schema *arrow.Schema) ([]*arrow.Field, error)
- func (sd *FilesDriver) Write(ctx context.Context, table string, schema *arrow.Schema, rows []map[string]any) error
- type FilesOption
- type Format
- type FormatWriter
- type ParquetFormatOption
- type SpecificParquetType
- type StreamUploader
- type Upload
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewParquetFieldTypeMapper ¶ added in v0.49.1
func NewParquetFieldTypeMapper() warehouse.FieldTypeMapper[SpecificParquetType]
NewParquetFieldTypeMapper creates a mapper that supports parquet nodes.
Types ¶
type CSVFormatOption ¶
type CSVFormatOption func(*csvFormat)
CSVFormatOption configures the CSV format behavior.
func Gzip ¶
func Gzip(level int) CSVFormatOption
Gzip enables gzip compression with the provided level.
func WithCompression ¶
func WithCompression(opt CSVFormatOption) CSVFormatOption
WithCompression is a pass-through to keep configuration options grouped.
type FilesDriver ¶ added in v0.49.1
type FilesDriver struct {
// contains filtered or unexported fields
}
FilesDriver writes warehouse rows into a keyed spool and flushes them to remote storage.
func NewFilesDriver ¶ added in v0.49.1
func NewFilesDriver( _ context.Context, spoolFactory spools.Factory, kv storage.KV, uploader StreamUploader, format Format, opts ...FilesOption, ) (*FilesDriver, error)
func (*FilesDriver) AddColumn ¶ added in v0.49.1
func (sd *FilesDriver) AddColumn(table string, field *arrow.Field) error
AddColumn is a no-op for spool drivers.
func (*FilesDriver) Close ¶ added in v0.49.1
func (sd *FilesDriver) Close() error
Close gracefully shuts down the driver.
func (*FilesDriver) CreateTable ¶ added in v0.49.1
func (sd *FilesDriver) CreateTable(table string, schema *arrow.Schema) error
CreateTable is a no-op for spool drivers.
func (*FilesDriver) MissingColumns ¶ added in v0.49.1
MissingColumns always returns an empty slice for spool drivers.
type FilesOption ¶ added in v0.49.1
type FilesOption func(*FilesDriver)
FilesOption is a functional option for configuring FilesDriver.
func WithPathTemplate ¶ added in v0.40.0
func WithPathTemplate(tmplStr string) FilesOption
WithPathTemplate sets the path template string for remote object keys.
func WithSpoolEncoderDecoder ¶ added in v0.49.1
func WithSpoolEncoderDecoder( encoder encoding.EncoderFunc, decoder encoding.DecoderFunc, ) FilesOption
WithSpoolEncoderDecoder sets the spool payload codec used for frame serialization/deserialization.
type Format ¶
type Format interface {
Extension() string
NewWriter(w io.Writer, schema *arrow.Schema) (FormatWriter, error)
}
Format defines how data is serialized to files.
func NewCSVFormat ¶
func NewCSVFormat(opts ...CSVFormatOption) Format
NewCSVFormat creates a new CSV format implementation.
func NewParquetFormat ¶ added in v0.49.1
func NewParquetFormat(opts ...ParquetFormatOption) Format
NewParquetFormat creates a parquet format implementation.
type FormatWriter ¶ added in v0.49.1
FormatWriter writes rows for a single output stream.
type ParquetFormatOption ¶ added in v0.49.1
type ParquetFormatOption func(*parquetFormat)
ParquetFormatOption configures parquet format behavior.
func WithParquetCompression ¶ added in v0.49.1
func WithParquetCompression(codec compress.Codec) ParquetFormatOption
WithParquetCompression sets parquet compression codec.
type SpecificParquetType ¶ added in v0.49.1
type SpecificParquetType struct {
Node parquet.Node
FormatFunc func(i any, m arrow.Metadata) (any, error)
}
SpecificParquetType represents a parquet node with formatting behavior.
type StreamUploader ¶ added in v0.49.1
StreamUploader handles uploading streams to a destination.
func NewBlobUploader ¶
func NewBlobUploader(bucket *blob.Bucket) StreamUploader
NewBlobUploader creates a new StreamUploader that uploads files to a blob bucket.
func NewFilesystemUploader ¶
func NewFilesystemUploader(destDir string) (StreamUploader, error)
NewFilesystemUploader creates a new StreamUploader that writes to a destination directory.
Source Files