importer

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 24, 2025 License: MIT Imports: 15 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Field

type Field struct {
	Name     string `json:"name"`
	Type     string `json:"type"`
	Nullable bool   `json:"nullable"`
}

Field represents a single column in a schema

type FileStats

type FileStats struct {
	RecordCount int64 `json:"record_count"`
	FileSize    int64 `json:"file_size"`
	ColumnCount int   `json:"column_count"`
}

FileStats contains statistics about a Parquet file

type ImportRequest

type ImportRequest struct {
	ParquetFile    string
	TableIdent     table.Identifier
	NamespaceIdent table.Identifier
	Schema         *Schema
	Overwrite      bool
	PartitionBy    []string
}

ImportRequest contains all parameters for importing a table

type ImportResult

type ImportResult struct {
	TableIdent    table.Identifier
	RecordCount   int64
	DataSize      int64
	TableLocation string
}

ImportResult contains the results of a table import

type ParquetImporter

type ParquetImporter struct {
	// contains filtered or unexported fields
}

ParquetImporter handles importing Parquet files into Iceberg tables

func NewParquetImporter

func NewParquetImporter(cfg *config.Config) (*ParquetImporter, error)

NewParquetImporter creates a new Parquet importer

func (*ParquetImporter) Close

func (p *ParquetImporter) Close() error

Close closes the importer and releases resources

func (*ParquetImporter) GetTableLocation

func (p *ParquetImporter) GetTableLocation(tableIdent table.Identifier) string

GetTableLocation returns the location where table data would be stored

func (*ParquetImporter) ImportTable

func (p *ParquetImporter) ImportTable(ctx context.Context, req ImportRequest) (*ImportResult, error)

ImportTable imports a Parquet file into an Iceberg table

func (*ParquetImporter) InferSchema

func (p *ParquetImporter) InferSchema(parquetFile string) (*Schema, *FileStats, error)

InferSchema reads a Parquet file and infers the schema

type Schema

type Schema struct {
	Fields []Field `json:"fields"`
}

Schema represents a simplified table schema

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL