Documentation
¶
Overview ¶
Package dtype is ursus's type system: DataType, Field and Schema.
It is a public package and the definition site for these types. The root package re-exports everything here by type alias, so ursus.Int64 and ursus.Schema are the spellings you should normally write. Import dtype directly only when implementing a custom scan source.
(It is public rather than internal for a mundane but real reason: an alias to an internal type renders on pkg.go.dev as a bare line with no methods documented, and DataType and Schema are the two most-read types in the library.)
DataType is comparable ¶
DataType is a small comparable struct, so `a == b` and `map[DataType]T` both work — the kernel registry and the type-promotion table depend on it. Nested and parameterised types keep their payload behind an interned pointer, so pointer equality is semantic equality. See intern.go.
Its fields are unexported for exactly that reason: an exported Inner or Fields would let a caller construct a DataType that bypasses interning, at which point `==` silently starts returning false for equal types. Use the constructors.
Index ¶
- Variables
- func CanCast(from, to DataType) bool
- func FormatDecimal(unscaled string, scale uint8) string
- func FormatTemporal(d DataType, ticks int64) string
- func LooksTemporal(d DataType, s string) bool
- func ParseTemporal(d DataType, s string) (int64, bool)
- type DataType
- func Array(inner DataType, size int) DataType
- func Datetime(u TimeUnit, tz string) DataType
- func Decimal(precision, scale uint8) DataType
- func Duration(u TimeUnit) DataType
- func Enum(cats ...string) DataType
- func List(inner DataType) DataType
- func Promote(a, b DataType) (DataType, bool)
- func Struct(fields ...Field) DataType
- func Time(u TimeUnit) DataType
- func (d DataType) BitWidth() int
- func (d DataType) Categories() []string
- func (d DataType) CategoryIndex(s string) (uint32, bool)
- func (d DataType) Equal(o DataType) bool
- func (d DataType) Fields() []Field
- func (d DataType) FromTime(t time.Time) (int64, bool)
- func (d DataType) HasStringStorage() bool
- func (d DataType) ID() TypeID
- func (d DataType) Inner() DataType
- func (d DataType) IsBool() bool
- func (d DataType) IsFixedWidth() bool
- func (d DataType) IsFloat() bool
- func (d DataType) IsHashable() bool
- func (d DataType) IsInteger() bool
- func (d DataType) IsNested() bool
- func (d DataType) IsNull() bool
- func (d DataType) IsNumeric() bool
- func (d DataType) IsOrdered() bool
- func (d DataType) IsSignedInteger() bool
- func (d DataType) IsString() bool
- func (d DataType) IsTemporal() bool
- func (d DataType) IsUnsignedInteger() bool
- func (d DataType) NanosPerTick() (int64, bool)
- func (d DataType) Physical() DataType
- func (d DataType) Precision() uint8
- func (d DataType) Scale() uint8
- func (d DataType) Size() int
- func (d DataType) String() string
- func (d DataType) TimeUnit() TimeUnit
- func (d DataType) TimeZone() string
- func (d DataType) ToDuration(ticks int64) (time.Duration, bool)
- func (d DataType) ToTime(ticks int64) (time.Time, bool)
- type Field
- type Interval
- func (i Interval) AddTo(t time.Time) time.Time
- func (i Interval) Days() int32
- func (i Interval) Err() error
- func (i Interval) IsCalendar() bool
- func (i Interval) IsZero() bool
- func (i Interval) Months() int32
- func (i Interval) Nanos() int64
- func (i Interval) Neg() Interval
- func (i Interval) Negative() bool
- func (i Interval) String() string
- func (i Interval) TruncateTo(t time.Time) time.Time
- type Schema
- func (s *Schema) All() iter.Seq2[int, Field]
- func (s *Schema) ByName(name string) (Field, bool)
- func (s *Schema) Drop(names ...string) *Schema
- func (s *Schema) Equal(o *Schema) bool
- func (s *Schema) Field(i int) Field
- func (s *Schema) FieldSlice() []Field
- func (s *Schema) Has(name string) bool
- func (s *Schema) IndexOf(name string) int
- func (s *Schema) Len() int
- func (s *Schema) Names() []string
- func (s *Schema) Select(names []string) (*Schema, error)
- func (s *Schema) String() string
- type TimeUnit
- type TypeID
Constants ¶
This section is empty.
Variables ¶
var ( Null = DataType{/* contains filtered or unexported fields */} Bool = DataType{/* contains filtered or unexported fields */} Int8 = DataType{/* contains filtered or unexported fields */} Int16 = DataType{/* contains filtered or unexported fields */} Int32 = DataType{/* contains filtered or unexported fields */} Int64 = DataType{/* contains filtered or unexported fields */} Uint8 = DataType{/* contains filtered or unexported fields */} Uint16 = DataType{/* contains filtered or unexported fields */} Uint32 = DataType{/* contains filtered or unexported fields */} Uint64 = DataType{/* contains filtered or unexported fields */} Float32 = DataType{/* contains filtered or unexported fields */} Float64 = DataType{/* contains filtered or unexported fields */} Int128 = DataType{/* contains filtered or unexported fields */} String = DataType{/* contains filtered or unexported fields */} Binary = DataType{/* contains filtered or unexported fields */} Date = DataType{/* contains filtered or unexported fields */} )
--- Simple types ------------------------------------------------------------
Declared as vars rather than consts because DataType is a struct. They are never mutated; treat them as constants.
var EmptySchema = &Schema{index: map[string]int{}}
EmptySchema is the schema with no columns.
Functions ¶
func CanCast ¶
CanCast reports whether an explicit Cast from `from` to `to` is defined.
This is broader than Promote: a cast may lose information (Int64 → Int8, Float64 → Int32), because the user asked for it. What it may not do is be meaningless — there is no cast from Struct to Int64.
Non-strict casts turn unrepresentable values into nulls rather than erroring; that is a kernel-level concern, not a type-level one.
func FormatDecimal ¶
FormatDecimal places the decimal point in an unscaled integer's digits.
FormatDecimal("1234", 2) == "12.34"
FormatDecimal("5", 3) == "0.005"
FormatDecimal("-5", 2) == "-0.05"
It takes the digits as a string rather than an Int128 so that dtype does not depend on the i128 package for what is purely presentation. Callers hold the scale (from DataType.Scale) and the digits (from Int128.String).
Without this a Decimal renders as its storage — 12.34 printed as "1234" — which is not a rounding error but a factor of 10^scale, and looks entirely plausible.
func FormatTemporal ¶
FormatTemporal renders one stored tick count of type d.
The formats are ISO 8601, which is what every other tool in the ecosystem reads and what the CSV writer emits — so a frame printed to a terminal and a frame written to a file agree, and neither shows the raw integer that this function exists to stop appearing.
func LooksTemporal ¶
LooksTemporal reports whether s could be one of the layouts ParseTemporal accepts for d. Used by CSV schema inference, which must be conservative: a column of bare integers like 20240101 must NOT become dates.
func ParseTemporal ¶
ParseTemporal reads a textual instant into a tick count of type d.
It accepts the layouts FormatTemporal produces, plus the common relaxations: a space instead of the T separator, and a missing timezone (read as UTC). That asymmetry is deliberate — a writer should be strict and a reader forgiving, because the reader is the one handed files it did not create.
Types ¶
type DataType ¶
type DataType struct {
// contains filtered or unexported fields
}
DataType is a logical column type.
It is a comparable value type: `a == b` is exact type equality, and DataType works as a map key. Parameterised and nested types keep their payload in an interned *typeExt, so two independently-constructed List(Int64) values share one pointer and compare equal.
The zero value is Null.
func Datetime ¶
Datetime is an instant at the given resolution. An empty tz means naive (wall-clock, no zone); otherwise tz is an IANA name such as "UTC".
The distinction matters: converting a zone changes the wall clock and keeps the instant, while replacing a zone keeps the wall clock and changes the instant.
func Decimal ¶
Decimal is a fixed-point decimal with the given precision (total digits) and scale (digits after the point), stored in 128 bits.
func Enum ¶
Enum is a string type whose category set is frozen and part of the type. Physically it is a Uint32 index into cats.
Enum is internable precisely because its categories never change. Categorical — whose mapping grows as data arrives — is not, which is why it is deferred until there is a per-context string cache to own the mapping.
func Promote ¶
Promote returns the common type two operands are cast to before a binary operation, or ok=false if there is no lossless-enough common type.
The rule is: never silently lose information that the user is likely to care about, and never silently produce a wrong answer.
Null + T → T (a null literal adopts the other side) same type → that type intN + intM (same sign) → the wider intN + uintM → the smallest SIGNED type holding both signed + uint64 → Int128 (see below) Int128 + any integer → Int128 float + float → the wider int + float → a float wide enough for the int's mantissa anything else → not ok
Uint64 combined with a signed type has no common type at 64 bits: the choices would be to wrap (silently wrong for large values) or to route through Float64 (silently lossy above 2^53), and Polars wraps. ursus used to reject the combination outright and make the user cast.
Int128 removes the dilemma — every uint64 and every int64 fits — so the combination now promotes losslessly. That was a free side effect of widening integer `sum` to 128 bits, and it is the better answer: an explicit cast that exists only to work around a missing type is a tax, not a safety feature.
Bool is not numeric here: `true + true` is rejected. Comparisons of two Bools are handled by the caller before reaching Promote.
func (DataType) BitWidth ¶
BitWidth returns the storage width in bits, or 0 if the type is not fixed width.
func (DataType) Categories ¶
Categories returns an Enum's category set. The result aliases interned storage and must not be mutated.
func (DataType) CategoryIndex ¶
CategoryIndex returns the physical index of an Enum category.
func (DataType) Equal ¶
Equal reports exact type equality. It is identical to ==, and exists because `a.Equal(b)` reads better in conditionals and because a future non-comparable type would keep the call sites working.
func (DataType) Fields ¶
Fields returns a Struct's fields. The result aliases interned storage and must not be mutated.
func (DataType) FromTime ¶
FromTime converts an instant into a tick count of type d, flooring toward negative infinity so the mapping stays monotone across the epoch.
func (DataType) HasStringStorage ¶
HasStringStorage reports whether values are stored as offsets + character data, i.e. whether Column.Strings() is the right accessor. This is the storage question, and it deliberately excludes Enum.
func (DataType) IsFixedWidth ¶
IsFixedWidth reports whether values occupy a constant number of bits, i.e. whether the column has a flat values buffer and no offsets.
func (DataType) IsHashable ¶
IsHashable reports whether values can be group-by keys or deduplicated.
Currently identical to IsOrdered, and deliberately a separate name: they are different questions and will diverge (a Struct could be hashable field-by-field long before anyone defines a total order over it).
func (DataType) IsNumeric ¶
IsNumeric reports whether arithmetic is defined on the type.
Bool is deliberately excluded: `true + true` is a bug in every query it appears in, and rejecting it at type-resolution time is more useful than promoting to integer the way C does. Decimal is included; its kernels arrive later.
func (DataType) IsOrdered ¶
IsOrdered reports whether ursus can compute a total order over this type.
It is the single source of truth for Sort, Min/Max, ordering comparisons and GroupBy, all of which previously carried their own copies of this list — five of them, already disagreeing about Binary, Enum and Decimal. Divergence there is worse than it sounds: the plan layer's copies decide what is REJECTED AT PLAN TIME, and the kernel layer's copies decide what actually works, so a mismatch either rejects a working feature or lets a query through to fail at runtime.
Enum is ordered by its CATEGORY INDEX, not lexically. For a frozen category set the index order is the order the user declared — Enum("low","med","high") sorts low < med < high, which is almost always the intent and is not expressible lexically. Categorical, whose mapping grows at runtime, would make the same choice non-deterministic, which is one reason it stays unconstructible.
func (DataType) IsSignedInteger ¶
func (DataType) IsString ¶
IsString reports whether the type's logical values are strings, including Enum.
Careful: an Enum is logically a string but is stored as a Uint32 index, so a caller that uses IsString to decide how to READ a column will reach for the string accessor and find nothing there. Use it for semantic questions ("can this be compared to a string literal?") and Physical() for storage questions.
func (DataType) IsTemporal ¶
func (DataType) IsUnsignedInteger ¶
func (DataType) NanosPerTick ¶
NanosPerTick returns the length of one stored tick in nanoseconds, and whether d is a temporal type with a defined tick length.
This is the conversion factor every temporal cast, parse and format needs, in one place. Date is included at whole days: it stores days since the epoch, so its tick is 86400 seconds even though it carries no TimeUnit of its own.
The widest value is Date's 8.64e13, so a tick count times this stays inside int64 for any instant within ~10^5 years of the epoch.
func (DataType) Physical ¶
Physical returns the storage type used to represent d.
Kernels dispatch on the physical type: Date and Int32 share integer kernels, Datetime and Duration share Int64 kernels, Enum indexes are Uint32. Only operations that need calendar or category semantics look at the logical type.
func (DataType) String ¶
String renders the type with its parameters.
This output is load-bearing: it appears in golden Explain files, in ursustest failure messages, and in every type error. It must stay stable.
func (DataType) TimeUnit ¶
TimeUnit returns the resolution of a Time, Datetime or Duration; Second otherwise.
func (DataType) TimeZone ¶
TimeZone returns a Datetime's IANA zone name, or "" if naive or not a Datetime.
func (DataType) ToDuration ¶
ToDuration converts a stored tick count of a Duration type into a time.Duration. Reports false for any other type.
type Field ¶
Field is a named, typed column slot.
Nullable is a static property of the schema, not a count of nulls actually present: a non-nullable field is one where the engine may skip validity handling entirely. Operations propagate it conservatively — an arithmetic result is nullable if either operand is, a non-strict cast is always nullable, IsNull is never nullable.
Field is comparable, because DataType is.
func Of ¶
Of builds a nullable field. Nullable is the safe default: claiming a column has no nulls when it does causes silent wrong answers, while the reverse only costs a little speed.
func (Field) AsNullable ¶
AsNullable returns a copy of f that is nullable.
func (Field) Rename ¶
Rename returns a copy of f with a new name. Used on every naming path — Alias, Name().Prefix, join suffixing.
type Interval ¶
type Interval struct {
// contains filtered or unexported fields
}
Interval is a calendar-aware span of time.
Why time.Duration is not enough ¶
A Duration is a fixed number of nanoseconds. "1 month" is not: it is 28, 29, 30 or 31 days depending on where you start. Neither is "1 day", in any zone with daylight saving — it is 23, 24 or 25 hours depending on the date. A type that stores only nanoseconds cannot express either, so it cannot express the windows a temporal group-by is asked for.
Three fields, and they are not interchangeable ¶
months, days and nanos are stored separately and applied in that order, because each obeys a different rule:
months calendar arithmetic with END-OF-MONTH CLAMPING days calendar days — 23, 24 or 25 hours across a DST boundary nanos absolute elapsed time
So Every("1d") is NOT FromDuration(24*time.Hour), and the difference is visible on exactly two days a year in any DST zone. Storing 1 day as 86,400,000,000,000 nanos would collapse the distinction at construction time, where it can never be recovered. This is Arrow's month-day-nano interval layout, chosen for the same reason.
Errors are carried, not returned ¶
Every("banana") has to do something, and returning (Interval, error) would break the one call shape the API is for — an option struct field. So a parse failure is carried in the value and surfaced when the query is built, which is the deferred error rule the rest of the library already follows.
func DaysInterval ¶
DaysInterval builds a calendar interval of n days.
func Every ¶
Every parses an interval string: "1y", "3mo", "1w", "2d", "6h", "15m", "30s", "1ns". Components may be concatenated — "1h30m" — and a leading "-" negates the whole thing.
Units: ns, us, ms, s, m, h, d, w, mo, q, y. Note "m" is MINUTES and "mo" is months, which is the one ambiguity in the notation and the reason the unit table is ordered longest-first.
func FromDuration ¶
FromDuration builds an absolute interval. The result is never calendar-aware — 24*time.Hour is twenty-four hours, not one day.
func IntervalOf ¶
IntervalOf rebuilds an interval from its three components.
It exists for the kernel. An expression argument travels as a *expr.Lit and litreflect only builds those from primitive kinds, so an Interval crosses that boundary as three plain integers and is reassembled here — rather than teaching the literal path about a struct type, which would be a second construction route for a value whose whole point is that its three fields are not interchangeable.
func MonthsInterval ¶
Months builds a calendar interval of n months. Years and quarters are months.
func (Interval) AddTo ¶
AddTo advances t by the interval, in t's own location.
The three components are applied in order — months, then days, then nanos — and the order is part of the contract, not an implementation detail. Adding one month and one day to January 31st gives February 29th then March 1st in a leap year; the other order gives February 1st then March 1st. Both are defensible; only one can be the answer, and this is the one Arrow, Polars and PostgreSQL all give.
func (Interval) Err ¶
Err reports a parse failure from Every. Nil for every Interval built any other way.
func (Interval) IsCalendar ¶
IsCalendar reports whether the interval's length depends on WHERE IT STARTS.
True for months and for days, and the days half is the part that is easy to get wrong: a day is a calendar day, so in a DST zone it is not a fixed span either. Anything built from a Duration is never calendar.
func (Interval) Months ¶
Months, Days and Nanos are the three components. Exported as accessors rather than fields so the zero value stays meaningful and nobody constructs a half-valid Interval that skipped Every's validation.
func (Interval) Neg ¶
Neg reverses the interval. Used to walk a window grid backwards and to turn a window's width into the offset from its end to its start.
func (Interval) String ¶
String renders the interval in Every's own notation, so String and Every round-trip.
func (Interval) TruncateTo ¶
TruncateTo floors t to the start of the window containing it, on a grid of this interval anchored at the Unix epoch, in t's own location.
Local, not UTC ¶
The grid is built from wall-clock components, so truncating by "1d" in America/New_York gives local midnight rather than 00:00 UTC. That is what makes Every("1d") different from FromDuration(24*time.Hour), which floors the absolute instant and is left alone precisely because a Duration IS absolute.
Mixed intervals are refused by the caller ¶
An interval with two non-zero components has no well-defined grid — "every 1 month and 3 days" does not tile the line — so a window grid must be built from exactly one component. GroupByDynamic rejects the rest; this function uses the coarsest component present so it is total rather than panicking on input it cannot get.
type Schema ¶
type Schema struct {
// contains filtered or unexported fields
}
Schema is an ordered, name-unique sequence of Fields.
A *Schema is IMMUTABLE once constructed and is shared freely between plan nodes, batches and goroutines. Nothing in ursus mutates one. Accessors that hand back a slice therefore return a fresh copy — see Names and FieldSlice.
Column order is meaningful (it is the output column order) but there is no row index: ursus follows Polars in treating row order as a property of the data rather than an addressable label space.
func MustSchema ¶
MustSchema is NewSchema for tests and package-level vars, panicking on duplicates.
func NewSchema ¶
NewSchema builds a schema, rejecting duplicate column names.
Duplicates are an error rather than a silent last-wins or auto-suffix because every downstream lookup is by name: tolerating them would make Col("x") ambiguous and the ambiguity would surface far from its cause.
func (*Schema) Drop ¶
Drop returns a schema without the named columns. Unknown names are ignored, because Drop is used to express "remove these if present" during rewrites.
func (*Schema) Field ¶
Field returns the i'th field. It panics if i is out of range, matching slice indexing: an out-of-range column ordinal is always an ursus bug, never user error.
func (*Schema) FieldSlice ¶
FieldSlice returns the fields in order, as a FRESH slice the caller may mutate.
func (*Schema) Names ¶
Names returns the column names in order, as a FRESH slice the caller may mutate. Projection pushdown sorts and filters the result in place.
type TimeUnit ¶
type TimeUnit uint8
TimeUnit is the resolution of a Time, Datetime or Duration.
type TypeID ¶
type TypeID uint8
TypeID identifies a type's shape. It is the switch tag for every dtype-erased dispatch in the engine.
const ( TypeNull TypeID = iota TypeBool TypeInt8 TypeInt16 TypeInt32 TypeInt64 TypeUint8 TypeUint16 TypeUint32 TypeUint64 TypeFloat32 TypeFloat64 TypeDecimal TypeString TypeBinary TypeDate // days since the Unix epoch, int32 TypeTime // time of day since midnight, int64 in the type's unit TypeDatetime // instant since the Unix epoch, int64 in the type's unit TypeDuration // elapsed time, int64 in the type's unit TypeList // variable length, one child type TypeArray // fixed length, one child type TypeStruct // named fields TypeEnum // frozen category set, part of the type // TypeInt128 is the accumulator and output type of integer Sum. Constructible. // // Categorical and Uint128 remain reserved: Categorical cannot be interned the // way Enum can, because its category mapping grows at runtime, and Uint128 has // no use while Int128 covers every unsigned value with room to spare. TypeCategorical TypeInt128 TypeUint128 // TypeIDCount is one past the last TypeID. Tables indexed by TypeID are // declared [TypeIDCount]T. TypeIDCount )
Source Files