Documentation
¶
Overview ¶
Package copyfrom turns a stream of values into a pgx.CopyFrom source.
It knows nothing about CSV. A RowSource is anything that yields values one at a time - decode.Typed over a file, a reader over an XLSX sheet, a paged API client, a generator - and Copy maps each value onto the columns a COPY was given:
source, err := copyfrom.NewCopy(rows, len(columns), func(dst []any, e *entity) []any {
return append(dst, e.ID, e.Name)
})
if err != nil {
return err
}
_, err = tx.CopyFrom(ctx, pgx.Identifier{table}, columns, source)
// The source's error first: it names the line, pgx's does not.
if srcErr := source.Err(); srcErr != nil {
return fmt.Errorf("read input at line %d: %w", source.Line(), srcErr)
}
if err != nil {
return err
}
That is the whole point of the package boundary. The layer that owns the database imports this one and never mentions the file format; the layer that parses imports decode and never mentions the database. Neither package imports the other, so the separation holds whether or not anyone remembers it.
pgx is not imported either. Go interfaces are structural, so the method set Next/Values/Err is the entire contract, and the version of pgx stays the application's choice.
ValidateColumns is here for the same reason: a staging table built from a file's own header puts untrusted names on the path to CREATE TABLE, and deciding what an identifier may look like is a database question, not a parsing one.
Errors wrap csvcopy.ErrSchema for wiring the calling code got wrong, and csvcopy.ErrParse for a header that cannot be used. See the csvcopy package doc.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrInvalidColumns = fmt.Errorf("%w: header has unusable column names", csvcopy.ErrParse)
ErrInvalidColumns reports a header that must not reach a DDL statement.
Wraps csvcopy.ErrParse: it is a property of the file, and the caller who quarantines bad files wants this one quarantined too.
Functions ¶
func ValidateColumns ¶
ValidateColumns rejects a header whose names cannot safely be used to build a statement.
A staging table is created from the file's own column names, which makes those names untrusted input on the path to CREATE TABLE. A file with a column called
x" ); DROP TABLE clients; --
is not a hypothetical; it is a text file, and anyone can write one. This refuses the shapes that break a statement or change its meaning even after quoting is considered: an empty name, a duplicate, two names differing only in case, a name over 63 bytes, one starting with a digit, and anything outside [A-Za-z0-9_].
The case rule is the one that needs a word. Quoted, "ID" and "id" are two legal columns; unquoted, Postgres folds both to id and the CREATE TABLE fails with `column "id" specified more than once` - which is the statement this function is asked about. A header that stays unambiguous only as long as nobody drops the quotes is a header to map explicitly, the same argument that refuses prose names.
Every violation is reported at once rather than the first one, so one run tells the whole story of the file instead of one name per attempt.
What it does not know is the reserved words of your Postgres version - select, table, user and the rest, which are a moving list this package would have to carry and keep current for a gain quoting already provides. That is one more reason the rule below holds rather than an exception to it.
Deliberately strict otherwise, and it rejects more than injection: "date of birth" and any non-ASCII name are refused too. A file whose column names are prose is a file whose names should be mapped explicitly, not quoted and hoped for. Passing this is not a licence to skip quoting - build identifiers with pgx.Identifier{name}.Sanitize() regardless. For CopyFrom, pgx quotes them itself.
An empty header - the empty-file case - passes: there is nothing to build from and nothing to be unsafe with.
Types ¶
type Copy ¶
type Copy[T any] struct { // contains filtered or unexported fields }
Copy adapts a RowSource to pgx.CopyFromSource.
Only the mapping of a value onto its columns lives here; where the values came from is none of this layer's business.
func NewCopy ¶
func NewCopy[T any](src RowSource[T], columns int, encode func(dst []any, item T) []any) (*Copy[T], error)
NewCopy wraps src, laying each value out with encode.
columns is how many values encode appends; it only sizes the backing slice, so being wrong costs an allocation rather than correctness. encode must append in the same order as the column list passed to pgx.CopyFrom - the two are a pair, and Postgres cannot notice when they disagree if the types are compatible.
A nil src or encode is csvcopy.ErrSchema, the same as a nil reader or convert elsewhere in the package: it is wiring the calling code got wrong, and no input file will fix it.
Use the pointer it returns. Copying the value gives two sources sharing one row buffer and one counter, so Rows stops meaning anything on either of them.
Example ¶
The two layers meeting: decode turns the file into values, this package lays a value out across the columns pgx.CopyFrom is given. Neither knows the other's concerns - swap decode for a source over XLSX and nothing here changes.
const file = "name;id\nAlice;1\nBob;2\n"
rows, err := decode.NewTyped(strings.NewReader(file), toEntity)
if err != nil {
panic(err)
}
source, err := NewCopy(rows, 2, func(dst []any, e *entity) []any {
return append(dst, e.ID, e.Name)
})
if err != nil {
panic(err)
}
// tx.CopyFrom(ctx, pgx.Identifier{"people"}, []string{"id", "name"}, source)
for source.Next() {
values, _ := source.Values()
fmt.Println(values...)
}
if err = source.Err(); err != nil {
panic(err)
}
Output: 1 Alice 2 Bob
func (*Copy[T]) Line ¶
Line is the line of the file the current row came from, or zero when the source does not have lines - one over an API or a generator does not.
Asked of the source through an interface rather than required by RowSource, so a source that cannot answer does not have to declare a method returning nothing useful.
It exists because the advice for a failed CopyFrom is to read the source's error first, since that is the one naming the line. Without this, following that advice meant keeping the decode.Typed value in a second variable purely to ask it - and the obvious code, which passes the source straight into NewCopy, could not.
func (*Copy[T]) Record ¶
Record is the raw record behind the current row, or nil when the source does not keep one. The counterpart to Line, and asked for the same way.
func (*Copy[T]) Rows ¶
Rows is the number of rows handed out so far. Counted here rather than asked of the source, which need not track it.
func (*Copy[T]) Values ¶
Values lays out the current value.
The result is stored back, so a row wider than columns grows the slice once instead of reallocating on every row. Safe to reuse because pgx encodes the row before pulling the next one.
Calling it before the first Next is csvcopy.ErrSchema. There is no row to lay out then, and encoding the zero value would hand back a plausible-looking row of empty values with no error on it. pgx.CopyFrom always calls Next first and never sees this; a caller driving the source by hand can, which is the only reason it is checked.
type RowSource ¶
RowSource is anything that yields values one at a time: decode.Typed, or a source of your own over XLSX, an API or a generator. Nothing in this package knows about CSV, and that is the point of it being its own package.
The interface is a pull, not a push, and that is the point. pgx.CopyFrom drives the loop itself - it asks for a row, encodes it into its send buffer, and only then asks for the next one. A source that pushes rows into a channel or a callback would need a goroutine between it and CopyFrom, and with it error handling across a goroutine boundary inside an open transaction.
Source Files