gnssconv

gnssconv: GNSS Data Conversion & Processing Tool

gnssconv is a command-line tool for converting GNSS data from RINEX to modern data formats like Apache Parquet and TileDB. Planning to add support for other formats in the future.

Features

• Input Formats: -RINEX Conversion: Reads RINEX versions 2, 3, and 4.
• Output Formats:
  • Apache Parquet: Creates Hive-partitioned datasets ideal for use with query engines like DuckDB, AWS Athena, and Spark.
  • TileDB: Writes to the TileDB array format, a powerful engine for storing and accessing dense and sparse multi-dimensional arrays.
• Cloud-Native: Natively supports writing output to local filesystems or to Amazon S3.

Installation

To build the gnssconv tool from source, run the following command from the project's root directory:

go build -o gnssconv .

Usage

gnssconv convert  \
--input rinex:./SITE0010.25o \
--output tiledb:s3://<bucket>/root/path 

Configuration

Options:
  --input FORMAT:PATH      Input format and file path (e.g., rinex:/path/to/file.obs)
  --output FORMAT:PATH      Output format and path (e.g., parquet:/path/to/root or tiledb:/path/to/array)
  --session SESSION_ID      Session ID for the data (optional)
  --filter-system SYSTEMS   Comma-separated list of GNSS systems (e.g., GPS,GLONASS)

AWS Credential Setup

If you are using AWS SSO, you can login with the following command:

aws sso login --profile my-sso-profile

You can export the credentials with the following command:

# For Linux or macOS (bash/zsh)
eval $(aws configure export-credentials --profile default --format env)

Don't forget to set the AWS_REGION environment variable to the region of the bucket you are writing to.

export AWS_REGION=us-east-2