Documentation
¶
Overview ¶
Package hive reads a dataset's partition columns from its files' paths, as Hive lays a dataset out and Polars reads one: each directory below the dataset's root named key=value gives every file beneath it a column key of value (step 347).
Index ¶
Constants ¶
const DefaultPartition = "__HIVE_DEFAULT_PARTITION__"
DefaultPartition is the value Hive writes for a null key.
Variables ¶
This section is empty.
Functions ¶
func Escape ¶ added in v0.6.8
Escape is a key's name or value as a path segment holds it, as Polars writes one (step 367): a byte below a space, DEL, a byte past ASCII, and `/`, `=`, a space, `%` and `:` percent-encoded, every other byte as it is. ParseFor decodes it.
func Infer ¶
Infer is the type of a key's values, as Polars infers a partition key's: Int64, or Int128 past it, for integers; Float64 for numbers among them one with a point, an exponent, inf or NaN; Boolean for true and false in any case; with dates, Date for dates and Datetime for instants, in microseconds and in UTC where one has an offset; String for anything else, and for a key with no value.
Types ¶
type Partitions ¶
type Partitions struct {
Keys []string
Text [][]string // by file, then key
Null [][]bool
// contains filtered or unexported fields
}
Partitions are a dataset's partition keys, in path order, and each file's values of them: the text its path spells, percent-decoded, and whether the value is null, Hive's default partition or nothing.
func Parse ¶
func Parse(root string, paths []string) (*Partitions, error)
Parse reads the key=value directories of each path below root, or of the whole of it where root is "", as Polars reads a directory's files and a list's; every path's the same keys in the same order, as Polars refuses paths whose keys differ. A directory not of that form is passed over, and so is root's own name.
func ParseFor ¶
func ParseFor(op, root string, paths []string) (*Partitions, error)
ParseFor is Parse for the scan op, which its errors and Values' name (step 360).
func (*Partitions) Types ¶
func (p *Partitions) Types(given map[string]dtype.DataType, files *dtype.Schema, dates bool) []dtype.DataType
Types are the keys' types: given's where it names the key; a file column's of the same name, which the key's values replace, as Polars reads them; or the type the key's values infer (Infer).
func (*Partitions) Values ¶
Values are each file's partition values as columns of one row, by file then key, of the types: each key's values cast from their text, strictly, a value its type cannot take refused, as Polars refuses it. An empty value is the empty string in a key of text, and null in any other, as Polars reads `k=` (step 367).
Source Files