dataloader

package
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 28, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewCsvReader added in v0.16.0 

func NewCsvReader(r io.ReadCloser) (*csvReader, error)

NewCsvReader creates a csvReader from a given ReadCloser.

The interpretation of the bytes of the supplied reader is a little murky. If there is a UTF8, UTF16LE or UTF16BE BOM as the first bytes read, then the BOM is stripped and the remaining contents of the reader are treated as that encoding. If we are not in any of those marked encodings, then some of the bytes go uninterpreted until we get to the SQL layer. It is currently the case that newlines must be encoded as a '0xa' byte.

func NewStringPrefixReader added in v0.16.0 

func NewStringPrefixReader(prefix string, reader io.Reader) *stringPrefixReader

NewStringPrefixReader creates a new stringPrefixReader that first returns the data in |prefix| and then returns data from |reader|.

Types

type BinaryDataLoader added in v1.3.0 

type BinaryDataLoader struct {
	// contains filtered or unexported fields
}

BinaryDataLoader tracks the state of a load data operation from a data source in the binary COPY format: a file header, followed by tuples consisting of a 16-bit field count and each field as a 32-bit byte length (-1 for NULL) and the field data in its binary send format, terminated by a trailer word of -1. Field values are decoded with each column type's binary receive function.

func NewBinaryDataLoader added in v1.3.0 

func NewBinaryDataLoader(colNames []string, tableSch sql.Schema) (*BinaryDataLoader, error)

NewBinaryDataLoader creates a new BinaryDataLoader that will produce rows for the schema provided.

func (*BinaryDataLoader) Children added in v1.3.0 

func (bdl *BinaryDataLoader) Children() []sql.Node

func (*BinaryDataLoader) Finish added in v1.3.0 

func (bdl *BinaryDataLoader) Finish(ctx *sql.Context) (*LoadDataResults, error)

Finish completes the current load data operation and finalizes the data that has been inserted.

func (*BinaryDataLoader) IsReadOnly added in v1.3.0 

func (bdl *BinaryDataLoader) IsReadOnly() bool

func (*BinaryDataLoader) Resolved added in v1.3.0 

func (bdl *BinaryDataLoader) Resolved() bool

func (*BinaryDataLoader) RowIter added in v1.3.0 

func (bdl *BinaryDataLoader) RowIter(ctx *sql.Context, r sql.Row) (sql.RowIter, error)

func (*BinaryDataLoader) Schema added in v1.3.0 

func (bdl *BinaryDataLoader) Schema(ctx *sql.Context) sql.Schema

func (*BinaryDataLoader) SetNextDataChunk added in v1.3.0 

func (bdl *BinaryDataLoader) SetNextDataChunk(ctx *sql.Context, data *bufio.Reader) error

func (*BinaryDataLoader) String added in v1.3.0 

func (bdl *BinaryDataLoader) String() string

func (*BinaryDataLoader) WithChildren added in v1.3.0 

func (bdl *BinaryDataLoader) WithChildren(ctx *sql.Context, children ...sql.Node) (sql.Node, error)

type BinaryDataWriter added in v1.3.0 

type BinaryDataWriter struct{}

BinaryDataWriter encodes rows for a COPY TO operation in the binary format. Values are expected to already be encoded in each type's binary send format (as produced for binary result values on the wire); this writer adds the file header, per-tuple field counts and lengths, and the file trailer that make up the binary COPY format.

func NewBinaryDataWriter added in v1.3.0 

func NewBinaryDataWriter() *BinaryDataWriter

NewBinaryDataWriter creates a new BinaryDataWriter.

func (*BinaryDataWriter) WriteFooter added in v1.3.0 

func (bdw *BinaryDataWriter) WriteFooter() ([]byte, error)

WriteFooter implements the DataWriter interface. The binary format ends with a trailer consisting of a 16-bit word containing -1, distinguishing a complete file from a truncated one (a tuple would begin with a non-negative field count).

func (*BinaryDataWriter) WriteHeader added in v1.3.0 

func (bdw *BinaryDataWriter) WriteHeader() ([]byte, error)

WriteHeader implements the DataWriter interface. The binary format always begins with a header, consisting of the format signature, a flags field, and a header extension area (empty, since we don't include OIDs or define any extensions).

func (*BinaryDataWriter) WriteRow added in v1.3.0 

func (bdw *BinaryDataWriter) WriteRow(vals [][]byte) ([]byte, error)

WriteRow implements the DataWriter interface. Each tuple consists of a 16-bit field count, followed by each field as a 32-bit byte length (-1 for NULL, with no value bytes following) and the field data in its binary send format.

type CsvDataLoader 

type CsvDataLoader struct {
	// contains filtered or unexported fields
}

CsvDataLoader is an implementation of DataLoader that reads data from chunks of CSV files and inserts them into a table.

func NewCsvDataLoader 

func NewCsvDataLoader(colNames []string, sch sql.Schema, delimiter string, header bool) (*CsvDataLoader, error)

NewCsvDataLoader creates a new DataLoader instance that will produce rows for the schema provided. |header| is true, the first line of the data will be treated as a header and ignored. If |delimiter| is not the empty string, it will be used as the delimiter separating value.

func (*CsvDataLoader) Children added in v0.17.0 

func (cdl *CsvDataLoader) Children() []sql.Node

func (*CsvDataLoader) Finish 

func (cdl *CsvDataLoader) Finish(ctx *sql.Context) (*LoadDataResults, error)

Finish implements the DataLoader interface

func (*CsvDataLoader) IsReadOnly added in v0.17.0 

func (cdl *CsvDataLoader) IsReadOnly() bool

func (*CsvDataLoader) Resolved added in v0.17.0 

func (cdl *CsvDataLoader) Resolved() bool

func (*CsvDataLoader) RowIter added in v0.17.0 

func (cdl *CsvDataLoader) RowIter(ctx *sql.Context, r sql.Row) (sql.RowIter, error)

func (*CsvDataLoader) Schema added in v0.17.0 

func (cdl *CsvDataLoader) Schema(ctx *sql.Context) sql.Schema

func (*CsvDataLoader) SetNextDataChunk added in v0.17.0 

func (cdl *CsvDataLoader) SetNextDataChunk(ctx *sql.Context, data *bufio.Reader) error

func (*CsvDataLoader) String added in v0.17.0 

func (cdl *CsvDataLoader) String() string

func (*CsvDataLoader) WithChildren added in v0.17.0 

func (cdl *CsvDataLoader) WithChildren(ctx *sql.Context, children ...sql.Node) (sql.Node, error)

type CsvDataWriter added in v1.3.0 

type CsvDataWriter struct {
	// contains filtered or unexported fields
}

CsvDataWriter encodes rows for a COPY TO operation in the CSV format, mirroring the format accepted by CsvDataLoader.

func NewCsvDataWriter added in v1.3.0 

func NewCsvDataWriter(colNames []string, delimiterChar string, header bool) *CsvDataWriter

NewCsvDataWriter creates a new CsvDataWriter for the columns named, using the specified |delimiterChar|. If |header| is true, a header line containing the column names will be returned by WriteHeader.

func (*CsvDataWriter) WriteFooter added in v1.3.0 

func (cdw *CsvDataWriter) WriteFooter() ([]byte, error)

WriteFooter implements the DataWriter interface.

func (*CsvDataWriter) WriteHeader added in v1.3.0 

func (cdw *CsvDataWriter) WriteHeader() ([]byte, error)

WriteHeader implements the DataWriter interface.

func (*CsvDataWriter) WriteRow added in v1.3.0 

func (cdw *CsvDataWriter) WriteRow(vals [][]byte) ([]byte, error)

WriteRow implements the DataWriter interface.

type DataLoader 

type DataLoader interface {
	sql.ExecSourceRel

	// SetNextDataChunk sets the next data chunk to be processed by the DataLoader.  Data records
	// are not guaranteed to start and end cleanly on chunk boundaries, so implementations must recognize incomplete
	// records and save them to prepend on the next processed chunk.
	SetNextDataChunk(ctx *sql.Context, data *bufio.Reader) error

	// Finish finalizes the current load operation and cleans up any resources used. Implementations should check that
	// the last call to LoadChunk did not end with an incomplete record and return an error to the caller if so. The
	// returned LoadDataResults describe the load operation, including how many rows were inserted.
	Finish(ctx *sql.Context) (*LoadDataResults, error)
}

DataLoader allows callers to insert rows from multiple chunks into a table. Rows encoded in each chunk will not necessarily end cleanly on a chunk boundary, so DataLoader implementations must handle recognizing partial, or incomplete records, and saving that partial record until the next call to LoadChunk, so that it may be prefixed with the incomplete record.

type DataWriter added in v1.3.0 

type DataWriter interface {
	// WriteHeader returns the encoded header line for the data, or nil if no header was requested.
	WriteHeader() ([]byte, error)

	// WriteRow returns the encoded bytes for the row given. |vals| contains the encoded value of each column
	// (text-encoded for the text and CSV formats, binary-encoded for the binary format), with nil indicating a
	// NULL value.
	WriteRow(vals [][]byte) ([]byte, error)

	// WriteFooter returns the encoded footer for the data, or nil if this format has no footer.
	WriteFooter() ([]byte, error)
}

DataWriter encodes rows of data for a COPY TO operation, one row at a time. Rows are provided as the text-encoded values of each column, with a nil value indicating NULL. Each call to WriteRow returns the encoded bytes for a single row (including the row terminator), which callers either stream to the client as CopyData messages or write to a file.

type LoadDataResults 

type LoadDataResults struct {
	// RowsLoaded contains the total number of rows inserted during a load data operation.
	RowsLoaded int32
}

LoadDataResults contains the results of a load data operation, including the number of rows loaded.

type TabularDataLoader 

type TabularDataLoader struct {
	// contains filtered or unexported fields
}

TabularDataLoader tracks the state of a load data operation from a tabular data source.

func NewTabularDataLoader 

func NewTabularDataLoader(colNames []string, tableSch sql.Schema, delimiterChar, nullChar string, header bool) (*TabularDataLoader, error)

NewTabularDataLoader creates a new TabularDataLoader to insert into the specified |table| using the specified |delimiterChar| and |nullChar|. If |header| is true, the first line of the data will be treated as a header and ignored.

func (*TabularDataLoader) Children added in v0.17.0 

func (tdl *TabularDataLoader) Children() []sql.Node

func (*TabularDataLoader) Finish 

func (tdl *TabularDataLoader) Finish(ctx *sql.Context) (*LoadDataResults, error)

Finish completes the current load data operation and finalizes the data that has been inserted.

func (*TabularDataLoader) IsReadOnly added in v0.17.0 

func (tdl *TabularDataLoader) IsReadOnly() bool

func (*TabularDataLoader) Resolved added in v0.17.0 

func (tdl *TabularDataLoader) Resolved() bool

func (*TabularDataLoader) RowIter added in v0.17.0 

func (tdl *TabularDataLoader) RowIter(ctx *sql.Context, r sql.Row) (sql.RowIter, error)

func (*TabularDataLoader) Schema added in v0.17.0 

func (tdl *TabularDataLoader) Schema(ctx *sql.Context) sql.Schema

func (*TabularDataLoader) SetNextDataChunk added in v0.17.0 

func (tdl *TabularDataLoader) SetNextDataChunk(ctx *sql.Context, data *bufio.Reader) error

func (*TabularDataLoader) String added in v0.17.0 

func (tdl *TabularDataLoader) String() string

func (*TabularDataLoader) WithChildren added in v0.17.0 

func (tdl *TabularDataLoader) WithChildren(ctx *sql.Context, children ...sql.Node) (sql.Node, error)

type TabularDataWriter added in v1.3.0 

type TabularDataWriter struct {
	// contains filtered or unexported fields
}

TabularDataWriter encodes rows for a COPY TO operation in the text format, mirroring the format accepted by TabularDataLoader.

func NewTabularDataWriter added in v1.3.0 

func NewTabularDataWriter(colNames []string, delimiterChar, nullChar string, header bool) *TabularDataWriter

NewTabularDataWriter creates a new TabularDataWriter for the columns named, using the specified |delimiterChar| and |nullChar|. If |header| is true, a header line containing the column names will be returned by WriteHeader.

func (*TabularDataWriter) WriteFooter added in v1.3.0 

func (tdw *TabularDataWriter) WriteFooter() ([]byte, error)

WriteFooter implements the DataWriter interface.

func (*TabularDataWriter) WriteHeader added in v1.3.0 

func (tdw *TabularDataWriter) WriteHeader() ([]byte, error)

WriteHeader implements the DataWriter interface.

func (*TabularDataWriter) WriteRow added in v1.3.0 

func (tdw *TabularDataWriter) WriteRow(vals [][]byte) ([]byte, error)

WriteRow implements the DataWriter interface.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.