ulog

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 28, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

README

ulog

CI Go Reference

A pure-Go reader, writer, and analysis library for the PX4 ULog format. It reads file-defined schemas dynamically, with reflection and generics available as optional conveniences for Go types.

go get github.com/sunfish-robotics/ulog

Usage

The package documentation contains executable examples for the common workflows:

  • stream dynamically typed records without materialising the complete log;
  • decode selected formats into typed Go structs;
  • write records from typed Go structs;
  • load a file into column-oriented datasets and export individual datasets as CSV with pkg/dataset; and
  • convert datasets to Apache Arrow or Parquet with pkg/columnar.

ULog F format messages remain authoritative. FormatFor[T], Decode[T], and Register[T] provide optional typed adapters without requiring a matching Go struct to read a file. Numeric arrays and nested formats use flattened paths such as q[0] and position.x. Character arrays remain one string-valued field; scalar char and uint8_t arrays remain byte-valued. Typed string fields declare their wire width explicitly with a tag such as ulog:"name,char[80]".

The root ulog and pkg/dataset packages use only the Go standard library. Importing pkg/columnar adds Apache Arrow.

Compatibility

Wire codecs have golden-byte tests. A committed file written by pyulog is read during ordinary Go tests, while a separate pinned CI job verifies both directions semantically:

Go writer → pyulog reader
Go writer → pyulog rewrite → Go reader

The pinned environment, test implementation, fixture, and provenance live together under tests.

Current scope

  • ULog version 1 writing and forward-compatible header and flag-bit reading
  • dynamic primitive, fixed-array, and nested-format data
  • streaming and eager reads
  • typed and dynamic writing
  • information and multi-information, parameters and defaults, logging, and dropouts
  • CSV, Arrow record batches, and Parquet output

Appended data sections are rejected rather than silently misread. Multi-information, default-parameter, and tagged-log writing are available at the lower-level pkg/wire boundary but do not yet have root-package writer conveniences.

License

This project is released under the Apache License, Version 2.0.

Documentation

Overview

Package ulog reads and writes PX4 ULog streams.

Each ULog file defines the formats of its own data records. Reader resolves those definitions and subscriptions as it advances. The current record is replaced on the next successful read, while metadata, parameters, logs, and dropouts remain available through the reader's accessors. The reader accepts future file versions and ignores unknown message types, but currently rejects unknown incompatibility flags and logs marked as containing appended data.

FormatsFor, FormatFor, Register, and Decode adapt named Go structs to that dynamic model. Exported fields become lower_snake_case by default; a “ulog” struct tag can rename a field or exclude it with “-”. The adapter supports ULog primitive types, fixed-size arrays, and nested named structs. A string field requires its wire width in a tag such as ulog:"name,char[80]". Dynamic records expose character arrays as strings with trailing NUL padding removed. Scalar characters and uint8 arrays remain byte-valued.

Use Reader when records can be processed as a stream. Package dataset loads a complete file into typed columns, and package columnar converts those datasets to Apache Arrow or Parquet.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func Decode

func Decode[T any](record Record) (T, error)

Decode maps the leading fields of record into T. It requires the flattened field names, order, and primitive wire types derived by FormatsFor, but does not require Record.Name to match T. Additional trailing record fields are ignored so that T can decode compatible schema extensions.

Example
package main

import (
	"fmt"
	"os"

	"github.com/sunfish-robotics/ulog"
)

func main() {
	type vehicleAttitude struct {
		Timestamp uint64
		Q         [4]float32
	}

	source, err := os.Open("flight.ulg")
	if err != nil {
		panic(err)
	}
	defer func() {
		if err := source.Close(); err != nil {
			panic(err)
		}
	}()

	reader, err := ulog.NewReader(source)
	if err != nil {
		panic(err)
	}
	for reader.Next() {
		if reader.Record().Name() != "vehicle_attitude" {
			continue
		}
		attitude, err := ulog.Decode[vehicleAttitude](reader.Record())
		if err != nil {
			panic(err)
		}
		fmt.Printf("%d: %v\n", attitude.Timestamp, attitude.Q)
	}
	if err := reader.Err(); err != nil {
		panic(err)
	}
}

Types

type DefaultParameter

type DefaultParameter struct {
	KeyValue
	// Types contains one or more applicable default scopes.
	Types DefaultParameterTypes
}

DefaultParameter is a parameter's default value for the scopes in DefaultParameter.Types. If a log has no default for a given parameter and scope, ULog defines its parameter value as the default.

type DefaultParameterTypes

type DefaultParameterTypes uint8

DefaultParameterTypes identifies the independent configuration scopes to which a DefaultParameter applies. A value may apply to both scopes.

const (
	// DefaultParameterSystemWide marks a system-wide default value.
	DefaultParameterSystemWide DefaultParameterTypes = 1 << 0
	// DefaultParameterCurrentConfiguration marks a default for the current configuration.
	DefaultParameterCurrentConfiguration DefaultParameterTypes = 1 << 1
)

type Dropout

type Dropout struct {
	// Duration is the period for which logging messages were lost.
	Duration time.Duration
}

Dropout describes a period during which logging messages were lost, often because the logging device could not keep up.

type Field

type Field struct {
	// Name is the case-sensitive field name used on the wire.
	Name string
	// Type is a ULog primitive type or the name of another [Format].
	Type Type
	// ArrayLength is the fixed element count, or zero for a scalar.
	ArrayLength int
}

Field describes one member of a Format. A non-primitive Field.Type names another format. Field.ArrayLength is zero for a scalar.

type FieldValue

type FieldValue struct {
	// Name is the flattened field path.
	Name string
	// Type is the primitive wire type of Value.
	Type Type
	// ArrayLength is the fixed byte width of a character array, or zero for a
	// scalar value.
	ArrayLength int
	// Value has the Go scalar type corresponding to Type. Character arrays are
	// strings with trailing NUL padding removed.
	Value any
}

FieldValue is one dynamically decoded value from a Record. FieldValue.Name is flattened: numeric arrays and nested formats use paths such as "q[0]" and "position.x". Character arrays remain one string-valued field.

type Format

type Format struct {
	// Name is the case-sensitive name used by subscriptions and nested fields.
	Name string
	// Fields contains at least one field in wire order.
	Fields []Field
}

Format is the self-described wire schema for one kind of data record. Fields remain in wire order.

func FormatFor

func FormatFor[T any]() (*Format, error)

FormatFor derives the root ULog Format for T. Use FormatsFor when nested definitions must also be written.

func FormatsFor

func FormatsFor[T any]() ([]Format, error)

FormatsFor derives every ULog Format needed to encode T. T must be a named struct; nested definitions precede the root definition in the returned slice. See the package documentation for the field and struct-tag mapping.

func ParseFormat

func ParseFormat(definition string) (*Format, error)

ParseFormat parses one "name:type field;" definition. It validates the grammar, names, duplicate fields, and primitive array sizes, but does not resolve nested Format names, detect cycles, or require the timestamp field needed by a subscription.

func (Format) String

func (f Format) String() string

String returns f in canonical "name:type field;" form.

type Header struct {
	// Version is the file-format version declared by the log.
	Version uint8
	// Timestamp is when logging started, in microseconds.
	Timestamp uint64
}

Header contains the ULog version and the logging start time from the fixed file header.

type KeyValue

type KeyValue struct {
	// Name is the case-sensitive information key or parameter name.
	Name string
	// Type is the primitive ULog type of Value.
	Type Type
	// ArrayLength is the fixed element count, or zero for a scalar.
	ArrayLength int
	// Value is a Go scalar, primitive slice, or string matching Type and ArrayLength.
	Value any
}

KeyValue is a decoded information or parameter entry. KeyValue.Value has the Go scalar or slice type selected by KeyValue.Type; character arrays are strings. KeyValue.ArrayLength is zero for a scalar.

type LogEntry

type LogEntry struct {
	// Level is the message severity.
	Level LogLevel
	// Timestamp is the message timestamp in microseconds.
	Timestamp uint64
	// Message is the logged text.
	Message string
	// Tag is an application-defined source identifier when Tagged is true.
	Tag uint16
	// Tagged reports whether Tag was present on the wire.
	Tagged bool
}

LogEntry is one printf-style ULog text message. LogEntry.Timestamp is in microseconds. LogEntry.Tag identifies an application-defined source only when LogEntry.Tagged is true.

type LogLevel

type LogLevel uint8

LogLevel is the severity of a ULog text message.

const (
	// LogLevelEmergency indicates that the system is unusable.
	LogLevelEmergency LogLevel = '0'
	// LogLevelAlert indicates that action must be taken immediately.
	LogLevelAlert LogLevel = '1'
	// LogLevelCritical indicates a critical condition.
	LogLevelCritical LogLevel = '2'
	// LogLevelError indicates an error condition.
	LogLevelError LogLevel = '3'
	// LogLevelWarning indicates a warning condition.
	LogLevelWarning LogLevel = '4'
	// LogLevelNotice indicates a normal but significant condition.
	LogLevelNotice LogLevel = '5'
	// LogLevelInfo indicates an informational message.
	LogLevelInfo LogLevel = '6'
	// LogLevelDebug indicates a debug message.
	LogLevelDebug LogLevel = '7'
)

type MultiInformationGroup added in v0.2.0

type MultiInformationGroup struct {
	// Name is the case-sensitive information key shared by Values.
	Name string
	// Values contains the independently typed entries in wire order.
	Values []MultiInformationValue
}

MultiInformationGroup contains one ordered group of multi-information values with the same name. The first value starts the group; subsequent values were marked as continuations on the wire. Each value retains its own declared type and array length.

type MultiInformationValue added in v0.2.0

type MultiInformationValue struct {
	KeyValue
	// IsArray reports whether the wire declaration included an array length.
	IsArray bool
}

MultiInformationValue is one independently typed value from a MultiInformationGroup. IsArray distinguishes scalar values from arrays, including PX4's empty char[0] separator values.

type RawStream

type RawStream struct {
	// contains filtered or unexported fields
}

RawStream writes payloads for one dynamic Format registered by Writer.RegisterFormat.

func (*RawStream) Write

func (s *RawStream) Write(payload []byte) error

Write writes one format-defined payload. Top-level trailing padding may be omitted as permitted by ULog; all other fields must be encoded.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader streams format-resolved Record values from ULog. As Reader.Next advances, it also collects information, multi-information groups, parameters, logs, and dropouts.

Example
package main

import (
	"fmt"
	"os"

	"github.com/sunfish-robotics/ulog"
)

func main() {
	source, err := os.Open("flight.ulg")
	if err != nil {
		panic(err)
	}
	defer func() {
		if err := source.Close(); err != nil {
			panic(err)
		}
	}()

	reader, err := ulog.NewReader(source)
	if err != nil {
		panic(err)
	}
	for reader.Next() {
		record := reader.Record()
		timestamp, err := record.Value("timestamp")
		if err != nil {
			panic(err)
		}
		fmt.Printf("%s[%d] timestamp=%v\n", record.Name(), record.MultiID(), timestamp)
	}
	if err := reader.Err(); err != nil {
		panic(err)
	}
}

func NewReader

func NewReader(source io.Reader) (*Reader, error)

NewReader consumes the fixed file header from source and checks its magic bytes. It accepts later file-format versions for forward compatibility and does not close source.

func (*Reader) DefaultParameters

func (r *Reader) DefaultParameters() []DefaultParameter

DefaultParameters returns independent copies of the parameter defaults encountered so far, in file order. Missing defaults are not synthesised.

func (*Reader) Dropouts

func (r *Reader) Dropouts() []Dropout

Dropouts returns the periods of lost logging messages encountered so far, in file order.

func (*Reader) Err

func (r *Reader) Err() error

Err returns the first error encountered by Reader.Next, or nil after a clean end of stream.

func (*Reader) Header

func (r *Reader) Header() Header

Header returns the version and logging start time consumed by NewReader. It is available before the first call to Reader.Next.

func (*Reader) Information

func (r *Reader) Information() []KeyValue

Information returns independent copies of the typed metadata entries encountered so far, in file order.

func (*Reader) Logs

func (r *Reader) Logs() []LogEntry

Logs returns the tagged and untagged text messages encountered so far, in file order.

func (*Reader) MultiInformation added in v0.2.0

func (r *Reader) MultiInformation() []MultiInformationGroup

MultiInformation returns independent copies of the grouped multi-information values encountered so far, in the order each group started. Every value keeps the type and array length declared by its own wire message.

func (*Reader) Next

func (r *Reader) Next() bool

Next consumes messages until it reaches the next data record. It returns false at end of stream or after the first error; call Reader.Err to distinguish the two. Definition, subscription, metadata, log, and dropout messages update the reader's state without producing a record. Unknown message types are skipped unless the log advertises an unsupported incompatibility feature.

func (*Reader) Parameters

func (r *Reader) Parameters() []KeyValue

Parameters returns independent copies of the initial parameter values and later changes encountered so far, in file order.

func (*Reader) Record

func (r *Reader) Record() Record

Record returns the Record selected by the most recent successful Reader.Next. Its value is unchanged after Next returns false.

type Record

type Record struct {
	// contains filtered or unexported fields
}

Record is one ULog data message paired with the subscription and Format needed to interpret it.

func (Record) Bytes

func (r Record) Bytes() []byte

Bytes returns an independent copy of the raw bytes described by Record.Format, excluding the subscription ID and enclosing message header. The payload may omit trailing top-level padding permitted by ULog.

func (Record) Fields

func (r Record) Fields() []ScalarField

Fields returns every flattened, non-padding value in the selected format, in wire order. A compatible older record may omit trailing fields; Record.Value reports those fields as unavailable.

func (Record) Format

func (r Record) Format() Format

Format returns an independent copy of the dynamic schema selected by the record's subscription.

func (Record) MessageID

func (r Record) MessageID() uint16

MessageID returns the runtime subscription identifier. It is only meaningful within this log.

func (Record) MultiID

func (r Record) MultiID() uint8

MultiID returns the instance identifier for the subscribed format. Zero is the first and default instance.

func (Record) Name

func (r Record) Name() string

Name returns the case-sensitive Format.Name selected by the record's subscription.

func (Record) Value

func (r Record) Value(name string) (any, error)

Value decodes a field by its flattened path. It reports an error when the field is unknown, omitted as trailing data, or truncated.

func (Record) Values

func (r Record) Values() ([]FieldValue, error)

Values decodes every available non-padding value in wire order. Missing trailing fields are omitted to support compatible schema extension.

type ScalarField

type ScalarField struct {
	// Name is the flattened field path.
	Name string
	// Type is the field's primitive wire type.
	Type Type
	// ArrayLength is the fixed byte width of a character array, or zero for a
	// scalar value.
	ArrayLength int
}

ScalarField describes one flattened, non-padding value in a Record.

type Stream

type Stream[T any] struct {
	// contains filtered or unexported fields
}

Stream writes values of T to one ULog subscription created by Register.

func Register

func Register[T any](writer *Writer, options ...StreamOption) (*Stream[T], error)

Register derives the formats required by T using the same mapping as FormatsFor, defines them, then registers a typed Stream. T must contain an exported scalar field that maps to the case-sensitive ULog name "timestamp" and wire type uint64_t. Registration must finish before the data section starts. Values are encoded in field order without Go struct padding.

Example
package main

import (
	"bytes"
	"fmt"

	"github.com/sunfish-robotics/ulog"
)

func main() {
	type sensorSample struct {
		Timestamp uint64
		Pressure  float32
		Valid     bool
	}

	var destination bytes.Buffer
	writer, err := ulog.NewWriter(&destination, ulog.WithStartTimestamp(1_000_000))
	if err != nil {
		panic(err)
	}
	stream, err := ulog.Register[sensorSample](writer)
	if err != nil {
		panic(err)
	}
	if err := stream.Write(sensorSample{Timestamp: 1_000_100, Pressure: 101.25, Valid: true}); err != nil {
		panic(err)
	}
	if err := writer.Close(); err != nil {
		panic(err)
	}

	reader, err := ulog.NewReader(bytes.NewReader(destination.Bytes()))
	if err != nil {
		panic(err)
	}
	var samples int
	for reader.Next() {
		samples++
	}
	if err := reader.Err(); err != nil {
		panic(err)
	}
	fmt.Println(samples)

}
Output:
1

func (*Stream[T]) Write

func (s *Stream[T]) Write(value T) error

Write encodes value using the format fixed by Register. The first stream write emits every registered subscription and starts the data section.

type StreamOption

type StreamOption func(*streamConfig) error

StreamOption configures Register or Writer.RegisterFormat.

func WithFormatName

func WithFormatName(name string) StreamOption

WithFormatName sets the root format name used by Register or Writer.RegisterFormat. For typed streams it replaces the name derived from the Go type; nested format names are unchanged.

func WithMultiID

func WithMultiID(multiID uint8) StreamOption

WithMultiID identifies one of several streams using the same format. Zero is the first and default instance.

type Type

type Type string

Type identifies a primitive ULog type or the name of another Format.

const (
	// TypeInt8 identifies a signed 8-bit integer.
	TypeInt8 Type = "int8_t"
	// TypeUint8 identifies an unsigned 8-bit integer.
	TypeUint8 Type = "uint8_t"
	// TypeInt16 identifies a signed 16-bit integer.
	TypeInt16 Type = "int16_t"
	// TypeUint16 identifies an unsigned 16-bit integer.
	TypeUint16 Type = "uint16_t"
	// TypeInt32 identifies a signed 32-bit integer.
	TypeInt32 Type = "int32_t"
	// TypeUint32 identifies an unsigned 32-bit integer.
	TypeUint32 Type = "uint32_t"
	// TypeInt64 identifies a signed 64-bit integer.
	TypeInt64 Type = "int64_t"
	// TypeUint64 identifies an unsigned 64-bit integer.
	TypeUint64 Type = "uint64_t"
	// TypeFloat32 identifies a 32-bit IEEE-754 floating-point value.
	TypeFloat32 Type = "float"
	// TypeFloat64 identifies a 64-bit IEEE-754 floating-point value.
	TypeFloat64 Type = "double"
	// TypeBool identifies a one-byte Boolean value.
	TypeBool Type = "bool"
	// TypeChar identifies a one-byte character.
	TypeChar Type = "char"
)

func (Type) IsPrimitive

func (t Type) IsPrimitive() bool

IsPrimitive reports whether t is one of ULog's built-in scalar types.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer serialises a ULog stream. All definitions, information, parameters, and stream registrations must be complete before the first data or log write. Writer methods and registered stream writes are safe for concurrent use; concurrent writes are serialised, but their relative order is unspecified.

func NewWriter

func NewWriter(destination io.Writer, options ...WriterOption) (*Writer, error)

NewWriter immediately writes the ULog file header and required flag-bits message to destination. Neither Writer.Close nor a failed constructor closes destination.

func (*Writer) Close

func (w *Writer) Close() error

Close writes any subscriptions not yet emitted, then prevents further writes. It does not close the underlying destination and is safe to call more than once. Close returns the first destination write error, including on later calls.

func (*Writer) Define

func (w *Writer) Define(format Format) error

Define validates and writes a dynamic Format. Definitions must be added before the first data or log write. Repeating an identical definition is a no-op; redefining the same name differently returns an error.

func (*Writer) RegisterFormat

func (w *Writer) RegisterFormat(format Format, options ...StreamOption) (*RawStream, error)

RegisterFormat validates and defines format, then registers a RawStream. The format must contain a scalar uint64_t timestamp field. Definitions referenced by nested fields must first be added with Writer.Define. Registration must finish before the data section starts.

func (*Writer) WriteDropout

func (w *Writer) WriteDropout(duration time.Duration) error

WriteDropout marks a period in which logging messages were lost. duration must be a non-negative whole number of milliseconds no greater than 65,535 ms. The first call starts the data section.

func (*Writer) WriteInformation

func (w *Writer) WriteInformation(name string, value any) error

WriteInformation writes one initial metadata entry before the data section. value may be a non-empty string or a scalar Go value supported by the ULog primitive mapping. Calls after the first data or log write return an error.

func (*Writer) WriteLog

func (w *Writer) WriteLog(level LogLevel, timestamp uint64, message string) error

WriteLog writes untagged printf-style output in the data section. timestamp is in microseconds. The first call starts the data section and prevents further definitions, registrations, initial information, or initial parameters.

func (*Writer) WriteParameter

func (w *Writer) WriteParameter(name string, value any) error

WriteParameter writes a vehicle parameter's value at the start of logging. ULog permits int32 and float32 values. Calls after the first data or log write return an error.

type WriterOption

type WriterOption func(*writerConfig) error

WriterOption configures NewWriter.

func WithStartTimestamp

func WithStartTimestamp(timestamp uint64) WriterOption

WithStartTimestamp records when logging started, in microseconds, in the file header. Without this option NewWriter writes zero.

Directories

Path Synopsis
pkg
columnar
Package columnar converts dataset.Dataset values to Apache Arrow and Parquet.
Package columnar converts dataset.Dataset values to Apache Arrow and Parquet.
dataset
Package dataset loads ULog streams into eager, column-oriented views.
Package dataset loads ULog streams into eager, column-oriented views.
wire
Package wire defines the byte-level types used by the PX4 ULog file format.
Package wire defines the byte-level types used by the PX4 ULog file format.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL