histimport

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package histimport parses the history exports of other apps (TezLab, ABRP) into drive and charge candidates. It is pure: no database, no network, no logging of row values. The caller (the import API) does the dedupe against stored rows and the writes.

The export formats are not documented anywhere public, so each source is a Format: a table of header aliases per field, built from the columns we assume and meant to be edited once real exports are in hand (slice 2b). The parser fails closed: a required field with no matching header refuses the file (MissingColumnsError, header names only), a unit or zone the file does not state is asked of the user (NeedChoiceError), a row that fails a check is skipped and counted by reason, and a file with more than 20% failing rows is refused (TooManyBadRowsError).

Index

Constants

View Source
const (
	FStart, FEnd         = "start", "end"
	FTime                = "time"
	FSessionID           = "session"
	FDistance            = "distance"
	FStartSoC, FEndSoC   = "start_soc", "end_soc"
	FSoC                 = "soc"
	FStartOdo, FEndOdo   = "start_odometer", "end_odometer"
	FOdometer            = "odometer"
	FStartLat, FStartLon = "start_lat", "start_lon"
	FEndLat, FEndLon     = "end_lat", "end_lon"
	FLat, FLon           = "lat", "lon"
	FMaxSpeed, FAvgSpeed = "max_speed", "avg_speed"
	FSpeed               = "speed"
	FEnergyUsed          = "energy_used"
	FEnergyAdded         = "energy_added"
	FMaxPower, FAvgPower = "max_power", "avg_power"
	FPower               = "power"
	FCost, FCurrency     = "cost", "currency"
	FVehicle             = "vehicle"
)

Canonical field names.

View Source
const (
	ReasonBadNumber    = "bad_number"
	ReasonBadTime      = "bad_time"
	ReasonMissing      = "missing_value"
	ReasonEndBeforeBeg = "ended_before_started"
	ReasonFuture       = "ended_in_future"
	ReasonTooOld       = "before_2021"
	ReasonTooLong      = "too_long"
	ReasonSoCRange     = "soc_out_of_range"
	ReasonDistance     = "bad_distance"
	ReasonSpeed        = "speed_too_high"
	ReasonEnergy       = "energy_out_of_range"
	ReasonPower        = "power_out_of_range"
	ReasonRunTooLong   = "run_too_long"

	ReasonDuplicate = "duplicate_in_file"
	ReasonOtherVeh  = "other_vehicle"
	ReasonParked    = "parked"
	ReasonEmpty     = "empty_session"
)

Skip reasons. The first group counts toward MaxBadRowFraction (the row or session is wrong); the second is expected noise.

View Source
const (
	MaxRows         = 500_000
	MaxCols         = 200
	MaxSessions     = 100_000 // drives or charges one file may produce; a decade of ten drives a day is under 40k
	MaxCSVBytes     = 200 << 20
	MaxXLSXZipBytes = 50 << 20
	MaxXLSXUnzipped = 200 << 20
)

Caps on what a file may hold. The upload is capped at 50 MB by the caller; these bound what it can expand to.

View Source
const AliasProvenance = "assumed: no published column list for TezLab or ABRP exports (checked Nov 13)"

Provenance of the header aliases, checked Nov 13: neither TezLab (support article 86, "Exporting Data"), Tessie (article 68, "Import from TezLab") nor ABRP (help article 7182124 "My Drives", and the feedback threads on export separators, temperatures and odometer) publishes a column list, a unit or a time format. So every alias below is an assumption drawn from the vocabulary exports of this kind use, and none is pinned to a real file. The parser is built for that: a file whose required columns do not match is refused with the header names it has and the names we expected (RequiredColumns), never imported in part.

View Source
const MaxBadRowFraction = 0.20

MaxBadRowFraction: a file with more failing rows than this is refused as a wrong mapping, not noise (design §2).

Variables

View Source
var (
	TezLabDrives = Format{
		Source: "tezlab_import",
		Kind:   KindDrives,
		Fields: []Field{
			{Name: FStart, Dim: DimTime, Required: true, Aliases: []string{"start time", "started at", "start", "start date", "departure time", "begin time", "date"}},
			{Name: FEnd, Dim: DimTime, Required: true, Aliases: []string{"end time", "ended at", "end", "end date", "arrival time", "finish time"}},
			{Name: FDistance, Dim: DimDistance, Required: true, Aliases: []string{"distance", "drive distance", "trip distance", "distance driven"}},
			{Name: FStartSoC, Dim: DimSoC, Aliases: []string{"start battery", "start soc", "start battery level", "starting battery", "starting soc", "battery start"}},
			{Name: FEndSoC, Dim: DimSoC, Aliases: []string{"end battery", "end soc", "end battery level", "ending battery", "ending soc", "battery end"}},
			{Name: FStartOdo, Dim: DimDistance, Aliases: []string{"start odometer", "odometer start", "starting odometer"}},
			{Name: FEndOdo, Dim: DimDistance, Aliases: []string{"end odometer", "odometer end", "ending odometer"}},
			{Name: FStartLat, Dim: DimAngle, Aliases: []string{"start latitude", "start lat", "origin latitude"}},
			{Name: FStartLon, Dim: DimAngle, Aliases: []string{"start longitude", "start lon", "start lng", "origin longitude"}},
			{Name: FEndLat, Dim: DimAngle, Aliases: []string{"end latitude", "end lat", "destination latitude"}},
			{Name: FEndLon, Dim: DimAngle, Aliases: []string{"end longitude", "end lon", "end lng", "destination longitude"}},
			{Name: FMaxSpeed, Dim: DimSpeed, Aliases: []string{"max speed", "maximum speed", "top speed"}},
			{Name: FAvgSpeed, Dim: DimSpeed, Aliases: []string{"avg speed", "average speed"}},
			{Name: FEnergyUsed, Dim: DimEnergy, Aliases: []string{"energy used", "energy consumed", "consumed energy"}},
			{Name: FVehicle, Dim: DimText, Aliases: []string{"vehicle", "vehicle name", "car", "vin"}},
		},
	}
	TezLabCharges = Format{
		Source:      "tezlab_import",
		Kind:        KindCharges,
		EnergyBasis: "unknown",
		Fields: []Field{
			{Name: FStart, Dim: DimTime, Required: true, Aliases: []string{"start time", "started at", "start", "start date", "begin time", "date"}},
			{Name: FEnd, Dim: DimTime, Required: true, Aliases: []string{"end time", "ended at", "end", "end date", "finish time"}},
			{Name: FStartSoC, Dim: DimSoC, Aliases: []string{"start battery", "start soc", "start battery level", "starting battery", "starting soc", "battery start"}},
			{Name: FEndSoC, Dim: DimSoC, Aliases: []string{"end battery", "end soc", "end battery level", "ending battery", "ending soc", "battery end"}},
			{Name: FEnergyAdded, Dim: DimEnergy, Aliases: []string{"energy added", "added energy", "charge energy added", "energy", "charged"}},
			{Name: FMaxPower, Dim: DimPower, Aliases: []string{"max power", "peak power", "maximum power", "max charge rate"}},
			{Name: FAvgPower, Dim: DimPower, Aliases: []string{"avg power", "average power", "avg charge rate"}},
			{Name: FStartLat, Dim: DimAngle, Aliases: []string{"latitude", "lat", "location latitude"}},
			{Name: FStartLon, Dim: DimAngle, Aliases: []string{"longitude", "lon", "lng", "location longitude"}},
			{Name: FCost, Dim: DimMoney, Aliases: []string{"cost", "total cost", "charge cost", "price"}},
			{Name: FCurrency, Dim: DimText, Aliases: []string{"currency", "cost currency"}},
			{Name: FVehicle, Dim: DimText, Aliases: []string{"vehicle", "vehicle name", "car", "vin"}},
		},
	}
)

TezLab exports one CSV per kind (Activity tab: Drives, Charges). Its columns are not published, so these aliases are assumptions (see AliasProvenance); a spelling that turns up in a real file is added here, never guessed in the parser. The energy basis of its charges is unknown, so it is recorded as "unknown" and kept out of efficiency ratios.

View Source
var ABRPPoints = Format{
	Source: "abrp_import",
	Kind:   KindPoints,
	Fields: []Field{
		{Name: FTime, Dim: DimTime, Required: true, Aliases: []string{"time", "timestamp", "date", "datetime", "date time", "utc"}},
		{Name: FLat, Dim: DimAngle, Required: true, Aliases: []string{"latitude", "lat"}},
		{Name: FLon, Dim: DimAngle, Required: true, Aliases: []string{"longitude", "lon", "lng", "long"}},
		{Name: FSoC, Dim: DimSoC, Required: true, Aliases: []string{"soc", "state of charge", "battery level", "battery", "battery soc"}},
		{Name: FSpeed, Dim: DimSpeed, Aliases: []string{"speed", "vehicle speed", "gps speed"}},
		{Name: FOdometer, Dim: DimDistance, Aliases: []string{"odometer", "odo"}},
		{Name: FSessionID, Dim: DimText, Aliases: []string{"drive id", "session", "session id", "trip id", "trip", "drive"}},
		{Name: FVehicle, Dim: DimText, Aliases: []string{"vehicle", "vehicle name", "car", "vin"}},
	},
}

ABRP's Drive & charge history export is an .xlsx of recorded points (design §1; the format is reported by users, not documented). The aliases are assumptions (see AliasProvenance). Drives and charges are derived from the points (points.go); points do not go into vehicle_state. SoC on a Rivian is whole percent only, so energy from SoC steps is coarse by about 1.3 kWh.

View Source
var ErrTooBig = errors.New("histimport: file is too big")

ErrTooBig: the file exceeds a cap (rows, columns, bytes).

Functions

func ExternalID

func ExternalID(source, rivianVehicleID, kind string, start time.Time) string

ExternalID is the deterministic id of an imported session: the same physical session gives the same id on every import of any file of that source (design §3). kind is "d" for a drive, "c" for a charge.

Types

type Charge

type Charge struct {
	StartedAt, EndedAt     time.Time
	StartSoCPct, EndSoCPct float64
	EnergyAddedKWh         float64
	// EnergyBasis is what EnergyAddedKWh measures, from the format: "battery", "wall" or "unknown". EnergyEstimated: it came from SoC × pack.
	EnergyBasis     string
	EnergyEstimated bool
	MaxPowerKW      float64
	AvgPowerKW      float64
	Lat, Lon        float64
	SessionType     string // "dc" when the peak power is at least 25 kW, else "ac"
	Cost            float64
	Currency        string
	HasCost         bool
	Source          string
	VehicleLabel    string
}

Charge is a parsed charge.

type ColumnSpec added in v0.41.24

type ColumnSpec struct {
	Field    string   `json:"field"`
	Examples []string `json:"examples"`
}

ColumnSpec is one required column of a format: the canonical field and the header spellings the parser accepts for it (first few).

type Dim

type Dim int

Dim is the physical dimension of a field, which says what its header unit can be.

const (
	DimText Dim = iota
	DimTime
	DimDistance // km or mi
	DimSpeed    // km/h or mph
	DimEnergy   // kWh or Wh
	DimPower    // kW or W
	DimSoC      // percent, or a fraction
	DimAngle
	DimMoney
)

type Drive

type Drive struct {
	StartedAt, EndedAt             time.Time
	StartSoCPct, EndSoCPct         float64
	StartOdometerMi, EndOdometerMi float64
	DistanceMi                     float64
	StartLat, StartLon             float64
	EndLat, EndLon                 float64
	MaxSpeedMph, AvgSpeedMph       float64
	// EnergyUsedKWh is the pack-side energy: the file's own consumed energy when its header says so, else (start SoC - end SoC) × the pack
	// given in Options. EnergyEstimated marks the second case; zero with no pack.
	EnergyUsedKWh   float64
	EnergyEstimated bool
	RoutePolyline   string // precision 5; only for sources that carry points
	Source          string
	VehicleLabel    string // the file's own label for the vehicle, "" when none
}

Drive is a parsed drive, in the units and zone the rest of Rivolt uses (miles, kWh, percent, UTC). It maps onto drives.Drive; the import API sets the id, vehicle and import batch.

type ExpectedLayout added in v0.41.24

type ExpectedLayout struct {
	Kind    string       `json:"kind"` // "drives", "charges" or "points"
	Columns []ColumnSpec `json:"columns"`
}

ExpectedLayout is one file layout of a source and the columns it requires.

func RequiredColumns added in v0.41.24

func RequiredColumns(formats []Format) []ExpectedLayout

RequiredColumns lists, per format, the required fields and up to three accepted spellings of each, in the format's own order. It carries no file content, so it can go to the user.

type Field

type Field struct {
	Name     string
	Aliases  []string
	Dim      Dim
	Required bool
}

Field is one canonical field and the header spellings that mean it. Aliases are compared after normalising (see baseName): case, spaces, punctuation and any unit word are ignored, so "Distance (km)", "distance_km" and "DISTANCE" are all "distance".

type Format

type Format struct {
	Source      string // "tezlab_import", "abrp_import": the Source of the rows it produces
	Kind        Kind
	Fields      []Field
	EnergyBasis string // what the file's charge energy measures: "battery", "wall" or "unknown" (recorded on the row, never guessed)
}

Format describes one export file: which source it comes from, what its rows are, and its columns. Formats are data, so a corrected or additional spelling is a one-line change (slice 2b pins them to real exports).

type Kind

type Kind int

Kind is what a table's rows are.

const (
	KindDrives  Kind = iota // one row per drive
	KindCharges             // one row per charge
	KindPoints              // one row per recorded point; drives and charges are derived from runs of points
)

type MissingColumnsError

type MissingColumnsError struct {
	Missing []string // canonical names of the required fields not found
	Header  []string
}

MissingColumnsError refuses a file with no header for a required field. It carries header names only, never row values, so it can be shown to the user and sent to support.

func (*MissingColumnsError) Error

func (e *MissingColumnsError) Error() string

type NeedChoiceError

type NeedChoiceError struct {
	Choices  []string // "distance_unit" (km|mi), "energy_unit" (kwh|wh), "timezone", "vehicle"
	Vehicles []string // for "vehicle": the labels the file holds
}

NeedChoiceError: the file does not say something the numbers depend on, so the user must (the preview asks). Never guessed.

func (*NeedChoiceError) Error

func (e *NeedChoiceError) Error() string

type Options

type Options struct {
	Location     *time.Location // zone of times that carry none
	DistanceUnit string         // "km" or "mi", for headers that state no unit
	EnergyUnit   string         // "kwh" or "wh", for energy and power headers that state none
	PackKWh      float64        // usable pack, for energy derived from SoC; 0 leaves such energy out
	Vehicle      string         // the file's own label of the vehicle to keep, when the file holds several
	Now          time.Time      // for the no-future check; zero means time.Now()
}

Options are what the user confirmed in the preview. A zero value is "not stated": the parser asks (NeedChoiceError) when the file does not state it either.

type Report

type Report struct {
	Rows      int            // data rows read
	Accepted  int            // sessions produced
	Skipped   map[string]int // sessions or rows left out, by reason (a points file counts points for parse failures and sessions for the rest)
	BadRows   int            // rows behind failing sessions: what MaxBadRowFraction is measured on
	Ambiguous int            // local times inside a DST fall-back hour, kept as the first instant
	// DroppedCoords counts session positions that were off the globe (or 0,0) and were left out; the session itself is kept.
	DroppedCoords int
	// Assumptions the numbers rest on, shown in the preview.
	SoCScale     string // "percent" or "fraction" (file said, or every value was ≤ 1)
	DistanceUnit string
	EnergyUnit   string
	Vehicles     []string // distinct vehicle labels in the file (at most 20)
	First, Last  time.Time
}

Report says what happened to every row, so the preview can show that new + skipped = rows.

type Result

type Result struct {
	Drives  []Drive
	Charges []Charge
	Report  Report
}

Result is a parsed file.

func Parse

func Parse(t Table, f Format, o Options) (*Result, error)

Parse reads a whole table in the given format. Memory is bounded by the sessions it produces, plus one run of points for KindPoints.

type Table

type Table interface {
	Header() []string
	// Next returns the next non-empty row, or io.EOF.
	Next() ([]string, error)
}

Table is a header plus a stream of rows. The row slice returned by Next is only valid until the next call.

func OpenCSV

func OpenCSV(src io.Reader) (Table, error)

OpenCSV reads a CSV with a header row. It skips a UTF-8 byte order mark and picks the delimiter (comma, semicolon or tab) from the header line.

func OpenXLSX

func OpenXLSX(src io.ReaderAt, size int64) (Table, error)

OpenXLSX reads the first worksheet of the workbook in src (size bytes). It refuses a zip over MaxXLSXZipBytes, one that declares more than MaxXLSXUnzipped, and one that actually expands past it.

type TooManyBadRowsError

type TooManyBadRowsError struct {
	Bad, Rows int
	Skipped   map[string]int
}

TooManyBadRowsError refuses a file whose rows mostly fail: its columns do not mean what the format says.

func (*TooManyBadRowsError) Error

func (e *TooManyBadRowsError) Error() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL