gnss-convert

gnss-convert: GNSS Data Conversion & Processing Tool

gnss-convert is a comprehensive command-line tool for converting GNSS data between different formats and performing various data processing operations. It consolidates functionality from multiple specialized conversion tools into a single, unified interface with modern CLI patterns.

Features

• Format Conversion: Convert between RINEX, BINEX, Novatel, SBF, RTCM3, SP3, Parquet, and TileDB formats
• TileDB Operations: Consolidate, query, and export TileDB arrays
• Data Validation: Validate GNSS data integrity and format compliance
• Availability Analysis: Compute data availability and perform analysis
• Batch Processing: Process multiple files and datasets efficiently
• Cloud-Native: Support for local filesystems and cloud storage (AWS S3)
• Structured Logging: Comprehensive logging with configurable levels

Installation

Prerequisites

• Go 1.21 or later
• For TileDB support: CGO-enabled build (automatic with Makefile)

Building from Source

The project includes a comprehensive Makefile with automatic TileDB dependency management:

# Clone the repository
git clone <repository-url>
cd gnsstools/cmd/gnss-convert

# Build with TileDB support (recommended)
make build

# Build without TileDB support
make build-without-tiledb

# Install to system
make install

TileDB Dependency Management

The Makefile automatically handles TileDB dependencies:

# Setup TileDB for current platform
make tiledb-setup

# Check TileDB installation
make tiledb-check

# Inspect available platforms
make tiledb-inspect-releases

# Check version compatibility
make tiledb-check-compatibility

Manual Build (without Makefile)

# Simple build (requires TileDB to be installed separately)
go build -o gnss-convert .

# Build without TileDB support
go build -tags no_tiledb -o gnss-convert .

Usage

Basic Commands

# Show help and available commands
gnss-convert --help

# Show version information
gnss-convert version

# Convert RINEX to Parquet
gnss-convert convert --input rinex:./SITE0010.25o --output parquet:./output/

# Convert RINEX to TileDB
gnss-convert convert --input rinex:./SITE0010.25o --output tiledb:./array/

# Convert to cloud storage
gnss-convert convert --input rinex:./SITE0010.25o --output tiledb:s3://bucket/path/

Available Commands

• convert: Convert GNSS data between different formats
• tiledb: TileDB-specific operations (consolidate, query, export)
• validate: Validate GNSS data integrity and format compliance
• info: Show information about GNSS data files
• version: Show version information

Convert Command Options

gnss-convert convert [flags]

Flags:
  --input string        Input format and file path (e.g., rinex:/path/to/file.obs)
  --output string       Output format and path (e.g., parquet:/path/to/root or tiledb:/path/to/array)
  --session string      Session ID for the data (optional)
  --filter-system string Comma-separated list of GNSS systems (e.g., GPS,GLONASS)
  --log-level string    Log level (debug, info, warn, error) (default "info")
  --log-format string   Log format (json, console) (default "console")
  --rinex-version string RINEX version for output (2.11, 3.05, 4.02) - defaults to 4.02

Supported Input/Output Combinations

The following table shows the current status of input/output format combinations:

Note: RINEX files come in three types:

• OBS: Observation data (satellite measurements)
• NAV: Navigation data (ephemeris, clock corrections)
• MET: Meteorological data (weather observations)

Legend:

• ✅ IMPLEMENTED: Fully functional and tested
• 🔄 PLANNED: In development roadmap
• 🚫 NOT PLANNED: Not currently in scope

Phase Development:

• Phase 1 ✅: RINEX OBS/BINEX input → RINEX OBS/Parquet/TileDB output
• Phase 2 🔄: Novatel/SBF/RTCM3 input → RINEX OBS/Parquet/TileDB + RTCM3/JSON output + RINEX NAV/MET support
• Phase 3 🔄: SP3 input → Parquet/TileDB + advanced features

Examples

# Convert RINEX observation file to Parquet
gnss-convert convert --input rinex:data.obs --output parquet:output/

# Convert with specific GNSS systems
gnss-convert convert --input rinex:data.obs --output parquet:output/ --filter-system GPS,GLONASS

# Convert with custom session ID
gnss-convert convert --input rinex:data.obs --output tiledb:array/ --session SITE001

# Convert with debug logging
gnss-convert convert --input rinex:data.obs --output parquet:output/ --log-level debug

# Convert BINEX to RINEX with specific version
gnss-convert convert --input binex:data.bnx --output rinex:output.obs --rinex-version 2.11

# Convert RINEX to RINEX with version upgrade
gnss-convert convert --input rinex:data_v2.obs --output rinex:data_v4.obs --rinex-version 4.02

# Convert SBF to RINEX
gnss-convert convert --input sbf:data.sbf --output rinex:output.obs

# Convert SBF to Parquet
gnss-convert convert --input sbf:data.sbf --output parquet:output/

# Convert SBF to TileDB
gnss-convert convert --input sbf:data.sbf --output tiledb:array/

Development

Makefile Targets

The project includes a comprehensive Makefile with the following targets:

Build Targets

• build: Build with TileDB support (requires TileDB installation)
• build-without-tiledb: Build without TileDB support
• install: Install to system
• clean: Clean build artifacts and TileDB installation
• release: Clean and build for release

TileDB Targets

• tiledb-setup: Download and setup TileDB library
• tiledb-check: Check if TileDB is properly installed
• tiledb-info: Show TileDB configuration and status
• tiledb-clean: Remove TileDB installation
• tiledb-inspect-releases: Inspect available platforms in releases.csv
• tiledb-check-compatibility: Check TileDB-Go and TileDB-C++ compatibility

Development Targets

• test: Run tests
• test-coverage: Run tests with coverage reporting
• lint: Run linter
• lint-fix: Run linter and fix issues
• deps: Show dependency information
• deps-update: Update dependencies
• deps-vendor: Vendor dependencies
• bench: Run benchmarks

Development Commands

• dev: Run the tool in development mode
• dev-convert: Run convert command in development mode
• dev-tiledb: Run tiledb command in development mode
• dev-validate: Run validate command in development mode
• dev-info: Run info command in development mode

Documentation & Docker

• docs: Generate documentation
• docker-build: Build Docker image
• docker-run: Run Docker container

Development Workflow

# Setup development environment
make tiledb-setup

# Build and test
make build
make test

# Run with debug logging
./bin/gnss-convert convert --input rinex:test.obs --output parquet:output/ --debug

# Development mode (run without building)
make dev-convert -- --input rinex:test.obs --output parquet:output/

# Clean and rebuild
make clean
make build

AWS Credential Setup

If you are using AWS SSO, you can login with the following command:

aws sso login --profile my-sso-profile

You can export the credentials with the following command:

# For Linux or macOS (bash/zsh)
eval $(aws configure export-credentials --profile default --format env)

Don't forget to set the AWS_REGION environment variable to the region of the bucket you are writing to.

export AWS_REGION=us-east-2

TileDB Compatibility

This tool uses TileDB-Go v0.36.0, which requires TileDB-C++ version 2.28.1. The Makefile automatically downloads and configures the correct version for your platform.

Supported Platforms

• macOS: ARM64 (Apple Silicon) and x86_64 (Intel)
• Linux: ARM64 (aarch64) and x86_64
• Windows: x86_64 (with MinGW64/UCRT64 support)

Version Information

# Check compatibility
make tiledb-check-compatibility

# Inspect available platforms
make tiledb-inspect-releases

# Show current configuration
make tiledb-info