Documentation
¶
Overview ¶
Package tablecsv inspects and corrects a CSV file before a table is registered over it: the defects a query engine cannot read through (a carriage-return line ending, a line break inside a cell, a legacy code page), whether each is correctable, the reason and remedy a refusal names, and the normalization that writes the corrected version. It knows nothing about registrations; tableregister asks it and acts on the answer.
Index ¶
Constants ¶
const ColumnType = "VARCHAR"
ColumnType is the only column type a Hive CSV table admits. Declaring a table with any other type is refused by Trino itself -- "Hive CSV storage format only supports VARCHAR (unbounded)" -- so this is the connector's rule rather than a choice the platform makes, and a join against a typed warehouse column needs a CAST.
Variables ¶
var ErrEmptyHeader = errors.New("the file has no header row, so the table has no column names")
ErrEmptyHeader means the file had no header row to take columns from.
var ErrUncorrectable = errors.New("the file cannot be corrected")
ErrUncorrectable marks a file that cannot be read as a table and cannot be corrected here either: the reason names what is wrong and where it has to be fixed. A registrar answers it as a refusal the caller can act on.
Functions ¶
Types ¶
type Column ¶
Column is one column a CSV header declares, as the table over the file records it. Type is recorded even though Hive CSV admits exactly one: a reader of the record should not have to know the connector's rule to know what a query will get back, and a stored type is what a later format would vary.
func ColumnsFrom ¶
ColumnsFrom names the columns a header record declares. It is separate from the header read so that a refusal describing a file calls its columns what the table over that file would have called them, without parsing the header a second time.
type Defect ¶
type Defect struct {
// Rows is how many records carry a field with a line break inside it.
Rows int `json:"rows,omitempty"`
// Columns names the columns those line breaks are in, in header order.
Columns []string `json:"columns,omitempty"`
// Encoding names what the bytes appear to be when they are not readable as
// UTF-8 text, and is empty when they are. A NUL byte is one of the ways
// they are not, whether or not the rest of the file is valid UTF-8.
Encoding string `json:"encoding,omitempty"`
// LineEndings names what the file's lines end in when that is not
// something a line-based reader splits on, and is empty when it is.
LineEndings string `json:"line_endings,omitempty"`
// HeaderFields is how many fields the header row declares, and is what a
// ragged record is short or long of.
HeaderFields int `json:"header_fields,omitempty"`
// Ragged names the records whose field count differs from the header's,
// in file order and at most maxNamedRecords of them.
Ragged []string `json:"ragged_records,omitempty"`
// Unreadable is the parse error that stopped a read of this file before
// its end, and is empty when every record parsed.
Unreadable string `json:"unreadable,omitempty"`
}
Defect is why a CSV cannot be read as a table the way it is stored. A nil *Defect means it can.
func Inspect ¶
Inspect reports why a CSV cannot be read by a line-based reader, or nil when it can.
Three conditions refuse. Lines that end in something the reader does not split on run the records that end in one together into a single record, a line break inside a field tears the record, and bytes that are not UTF-8 reach every cell as replacement characters. None of them is visible in the result of a query over the table, so all three are answered before one exists.
The scan records two further things that do not refuse a file by themselves: records whose field count differs from the header's, and a parse error that stops the read before the end. A file carrying only one of those registers today and goes on registering, but neither can be put right by the correction, so a defect found alongside one of them is refused rather than offered a repair that would then decline (#1449).
The line endings are settled first and the record scan runs over the translated bytes, because the scan is itself a line-based reader: over a carriage-return file it sees one record, reports the file as a single torn row, and names columns made out of the header line joined to the line after it.
func (*Defect) Correctable ¶
Correctable reports whether the platform can produce a corrected version of this file itself.
It is the same question Normalize answers, asked before the offer is made rather than after it is taken up. Three things say no: bytes in an encoding the platform does not convert, because everything else about the file is read through that encoding and would be corrected into mojibake; a record whose field count differs from the header's, because padding a short one invents data and truncating a long one discards it; and a parse that does not reach the end of the file, because the correction has to read every record to write it back.
func (*Defect) Reason ¶
Reason is what the person is told: what the file's lines end in when that is not a newline, how many rows carry a line break inside a cell, which columns those are in, and what the bytes appear to be when they are not UTF-8.
The line endings come first, because they decide how the rest of the file is read.
func (*Defect) Remedy ¶
remedy is what has to happen to a file the platform cannot correct itself. Bytes it cannot read are re-exported; records that do not match the header, or that it cannot parse through, are fixed where the file was written, because only whoever wrote it knows what the missing field held.
type NormalizeReport ¶
type NormalizeReport struct {
// RowsRepaired is how many records had a line break taken out of a field.
RowsRepaired int `json:"rows_repaired"`
// FromEncoding names the encoding the bytes were converted from, and is
// empty when they were already UTF-8.
FromEncoding string `json:"from_encoding,omitempty"`
// FromLineEndings names the line endings the file was rewritten from, and
// is empty when its records already ended in a newline.
FromLineEndings string `json:"from_line_endings,omitempty"`
}
NormalizeReport is what a correction changed.
func Normalize ¶
func Normalize(content []byte) ([]byte, NormalizeReport, error)
Normalize rewrites a CSV so a line-based reader gets the records the file actually holds: UTF-8 with no byte-order mark, one record per newline, and every field on one line.
It is a decode and a re-emit, never a repair of the record structure. A record whose field count differs from the header is refused rather than adjusted: padding a short record invents data and truncating a long one discards it, and neither is a correction the platform can make on somebody's behalf.
Source Files