tiledbgnss

package
v0.37.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 14, 2026 License: Apache-2.0 Imports: 18 Imported by: 4

README

TileDB GNSS: Efficient Storage and Access for GNSS Data

This package provides tools for writing to and reading from TileDB arrays using schemas optimized for common GNSS (Global Navigation Satellite System) data types. TileDB's efficient storage engine and flexible querying capabilities make it ideal for managing large-scale GNSS datasets.

Benefits:

  • Scalability: Handle massive GNSS datasets with ease.
  • Performance: Optimized for fast data retrieval and analysis.
  • Cloud-Native: Seamless integration with cloud storage.
  • Multi-Dimensional Efficiently query GNSS data by time, system, satellite, and signal.

Installation

Prerequisites:

  • Golang (compatible versions: >=1.23)
  • TileDB C++ Library (link)
  • TileDB-Go (link)

Install TileDB C++ Library (Mac OS X):

From Release (version 2.27):

curl --location -o tiledb.tar.gz https://github.com/TileDB-Inc/TileDB/releases/download/2.27.2/tiledb-macos-arm64-2.27.2-1757013.tar.gz \
&& sudo tar -C /usr/local -xSf tiledb.tar.gz

From Source (version 2.27):

git clone https://github.com/TileDB-Inc/TileDB.git -b release-2.27
cd TileDB
mkdir build && cd build
cmake -DTILEDB_WERROR=OFF -DTILEDB_VCPKG=ON -DTILEDB_VERBOSE=OFF -DTILEDB_S3=ON -DTILEDB_SERIALIZATION=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local ..
sudo make -j$(sysctl -n hw.ncpu) tiledb install

Install TileDB-Go Library

 go get -v github.com/TileDB-Inc/TileDB-Go@v0.35.0

Note: Golang and C++ versions must be compatible, check here for compatability

For Developers

Golang

Enable CGO CGO_ENABLED=1 when building projects that include TileDB.

Make sure either gcc or clang are installed when building with TileDB.

Mac OS X

Add the path to libtiledb.dylib to DYLD_LIBRARY_PATH

LINUX

Add the path to the C++ library to LD_LIBRARY_PATH, commonly located at /usr/local/lib

Run ldconfig to configure and update the dynamic linker run-time bindings.

VSCode Setup Mac OSX

Add the following parameter to your settings.json file.

"go.toolsEnvVars": {
    "DYLD_LIBRARY_PATH": "$DYLD_LIBRARY_PATH:/usr/local/lib"
},

Note: This assumes libtiledb.dylib is installed in /usr/local/lib

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ArrayExists added in v0.17.2 

func ArrayExists(arrayPath string) bool

Check if the TileDB array exists at the given path

func Consolidate added in v0.37.1 

func Consolidate(dir, region string, mode ConsolidationMode, fragConfig *FragmentConsolidationConfig) error

Consolidate runs consolidation for the given mode on a TileDB array.

func ConsolidateAll added in v0.37.1 

func ConsolidateAll(dir, region string, fragConfig *FragmentConsolidationConfig) error

ConsolidateAll runs all three consolidation modes in sequence: fragments -> fragment_meta -> commits.

func ConsolidateMetadata 

func ConsolidateMetadata(dir string, region string) error

ConsolidateMetadata consolidates fragment metadata (backward-compatible wrapper).

func ConsolidateOverlapping added in v0.37.1 

func ConsolidateOverlapping(dir, region string, groups []OverlapGroup, vacuum bool, vacuumInterval int) error

ConsolidateOverlapping consolidates each overlap group's fragments into a single fragment using targeted ConsolidateFragments. When vacuum is true, fragments are vacuumed every vacuumInterval groups to prevent unbounded growth of the __fragments directory, and a full cleanup (commits + fragment_meta) is performed at the end.

func ConvertBytesToUint16 

func ConvertBytesToUint16(bytes []byte) (uint16, error)

ConvertBytesToUint16 converts a slice of two bytes to a uint16 using BigEndian byte order.

func ConvertUint16ToBytes 

func ConvertUint16ToBytes(num uint16) []byte

ConvertUint16ToBytes converts a uint16 to a slice of two bytes assuming BigEndian byte order.

func CreateArray 

func CreateArray(srcArrayPath, destArrayPath string, region string) (err error)

CreateArray creates a new TileDB array using the schema from the provided path to a reference array

func CreateTDCPArray 

func CreateTDCPArray(arr, region string) (err error)

func DeleteOverlappingFragments added in v0.37.1 

func DeleteOverlappingFragments(arrayURI, region string, plan *DeleteResult) error

DeleteOverlappingFragments deletes all fragments marked for deletion in the plan using a single tiledb.DeleteFragmentsList call.

func GPSWeekSowToTime 

func GPSWeekSowToTime(gpsWeek uint16, tow int32) (dt time.Time)

func GetTimeRange added in v0.16.2 

func GetTimeRange(arr string, region string) (start time.Time, end time.Time, err error)

For a given array, return the time range of the data

func ReadMetArray 

func ReadMetArray(arr, region string, queryParams MetQueryParams) (epochs []met.Epoch, err error)

func ReadMetV2Array 

func ReadMetV2Array(arr, region string, queryParams MetV2QueryParams) (epochs []met.Epoch, err error)

func ReadNavArray 

func ReadNavArray(arr string, region string, queryParams NavQueryParams) (ephems []ephemeris.GenericEphemeris, err error)

func ReadNavV2Array 

func ReadNavV2Array(arr string, region string, queryParams NavV2QueryParams) (ephems []ephemeris.GenericEphemeris, err error)

func ReadObsV3Array 

func ReadObsV3Array(arr string, region string, queryParams ObsQueryParams) (epochs []observation.Epoch, err error)

ReadObsV3Array reads GNSS observations to a TileDB array

func ReadPosArray 

func ReadPosArray()

func Vacuum added in v0.37.1 

func Vacuum(dir, region string, mode ConsolidationMode) error

Vacuum runs vacuum for the given mode on a TileDB array.

func VacuumAll added in v0.37.1 

func VacuumAll(dir, region string) error

VacuumAll runs all three vacuum modes in sequence: fragments -> fragment_meta -> commits.

func VacuumMetadata 

func VacuumMetadata(dir string, region string) error

VacuumMetadata vacuums fragment metadata (backward-compatible wrapper).

func WriteBottleArray 

func WriteBottleArray(arr string, region string, bSlice []bottle.Bottle3DFloat64, opts ...WriteOptions) (err error)

WriteBottleArray writes strain observations to a TileDB array

func WriteMetArray 

func WriteMetArray(arr, edid, region string, epochs []met.Epoch, opts ...WriteOptions) (err error)

WriteMetArray - writes meteorolgical data to a TileDB array

func WriteMetV2Array 

func WriteMetV2Array(arr, region string, epochs []met.Epoch, opts ...WriteOptions) (err error)

WriteMetArray - writes meteorolgical data to a TileDB array

func WriteNavArray 

func WriteNavArray(arr string, edid string, region string, ephs []ephemeris.GenericEphemeris, opts ...WriteOptions) (err error)

WriteNavArray -

func WriteNavV2Array 

func WriteNavV2Array(arr string, region string, ephs []ephemeris.GenericEphemeris, opts ...WriteOptions) (err error)

WriteNavV2Array -

func WriteObsV3Array 

func WriteObsV3Array(arr string, region string, epochs []observation.Epoch, opts ...WriteOptions) (err error)

WriteObsV3Array writes GNSS observations to a TileDB array

func WritePosArray 

func WritePosArray(arr string, region string, positionEpochs []position.GAGEPosition, opts ...WriteOptions) (err error)

func WriteSP3Array 

func WriteSP3Array(arr string, region string, tag string, epochs []precise.SP3Epoch) (err error)

DEPRECATED: WriteSP3Array is the legacy function for backward compatibility Use WriteSP3Records instead

func WriteSP3Records added in v0.31.0 

func WriteSP3Records(arrayURI string, region string, records []precise.SP3Record) error

DEPRECATED: Use pkg/geodata/gnsstiledb.Client.WritePreciseOrbits() instead. WriteSP3Records writes SP3Record format to TileDB with support for: - Optional velocity fields (nullable) - Product metadata (AC, type, frame, reference point) - Event and prediction flags

func WriteTDCPArray 

func WriteTDCPArray(arr, region, edid string, tdcpResults []tdcp.Velocity, opts ...WriteOptions) (err error)

Types

type ConsolidationMode added in v0.37.1 

type ConsolidationMode string

ConsolidationMode specifies what to consolidate or vacuum. 

const (
	ModeFragments    ConsolidationMode = "fragments"
	ModeFragmentMeta ConsolidationMode = "fragment_meta"
	ModeCommits      ConsolidationMode = "commits"
)

type DeleteDecision added in v0.37.1 

type DeleteDecision struct {
	Fragment FragmentDetail `json:"fragment"`
	Action   string         `json:"action"` // "keep" or "delete"
	Reason   string         `json:"reason"`
}

DeleteDecision records whether a single fragment should be kept or deleted.

type DeleteGroupResult added in v0.37.1 

type DeleteGroupResult struct {
	Group     OverlapGroup     `json:"group"`
	Decisions []DeleteDecision `json:"decisions"`
}

DeleteGroupResult holds the deletion decisions for one overlap group.

type DeleteResult added in v0.37.1 

type DeleteResult struct {
	Groups           []DeleteGroupResult `json:"groups"`
	TotalKeep        int                 `json:"total_keep"`
	TotalDelete      int                 `json:"total_delete"`
	ReclaimableBytes uint64              `json:"reclaimable_bytes"`
}

DeleteResult is the complete deletion plan across all overlap groups.

func PlanOverlapDeletions added in v0.37.1 

func PlanOverlapDeletions(groups []OverlapGroup) *DeleteResult

PlanOverlapDeletions determines which fragments to keep and which to delete in each overlap group. Strategy: keep the fragment with the highest WriteEnd (newest write). Tie-break by CellCount (most data), then Index (lowest).

type FragmentConsolidationConfig added in v0.37.1 

type FragmentConsolidationConfig struct {
	Steps        uint32
	StepMinFrags uint32
	StepMaxFrags uint32
}

FragmentConsolidationConfig controls step-based fragment consolidation.

func DefaultFragmentConfig added in v0.37.1 

func DefaultFragmentConfig() FragmentConsolidationConfig

DefaultFragmentConfig returns sensible defaults for fragment consolidation.

type FragmentDetail added in v0.37.1 

type FragmentDetail struct {
	Index      uint32 `json:"index"`
	URI        string `json:"uri"`
	TimeStart  int64  `json:"time_start"` // non-empty domain min on "time" dimension (ms)
	TimeEnd    int64  `json:"time_end"`   // non-empty domain max on "time" dimension (ms)
	CellCount  uint64 `json:"cell_count"`
	Size       uint64 `json:"size"`
	WriteStart uint64 `json:"write_start"` // fragment write timestamp
	WriteEnd   uint64 `json:"write_end"`   // fragment write timestamp
}

FragmentDetail holds metadata about a single TileDB fragment.

type FragmentReport added in v0.37.1 

type FragmentReport struct {
	ArrayURI       string           `json:"array_uri"`
	TotalFragments uint32           `json:"total_fragments"`
	TotalCells     uint64           `json:"total_cells"`
	TotalSize      uint64           `json:"total_size"`
	Fragments      []FragmentDetail `json:"fragments"`
	OverlapGroups  []OverlapGroup   `json:"overlap_groups"` // only groups with 2+ fragments
}

FragmentReport contains the full analysis of fragments in an array.

func AnalyzeFragments added in v0.37.1 

func AnalyzeFragments(dir, region string, strategies ...GroupingStrategy) (*FragmentReport, error)

AnalyzeFragments inspects all fragments in a TileDB array. If a GroupingStrategy is provided, fragments are grouped accordingly. If no strategy is provided, GroupExact is used, where fragments that share the same start and end times are grouped.

type GroupingStrategy added in v0.37.1 

type GroupingStrategy int

GroupingStrategy defines how fragments are grouped together. 

const (
	// GroupExact matches only if Start AND End times are identical.
	// BEST FOR: Deduplication / Deletion.
	GroupExact GroupingStrategy = iota

	// GroupOverlaps matches if time ranges overlap or touch.
	// BEST FOR: Consolidation / Vacuuming.
	GroupOverlaps
)

type MetQueryParams 

type MetQueryParams struct {
	EDID    []StringRange `json:"edid"`
	Time    []TimeRange   `json:"time"`
	Key     []StringRange `json:"key"`
	Ordered bool          `json:"ordered"` // Ignored: met reads always use ROW_MAJOR (variable-length buffers require it)
}

type MetV2QueryParams 

type MetV2QueryParams struct {
	Time    []TimeRange   `json:"time"`
	Key     []StringRange `json:"key"`
	Ordered bool          `json:"ordered"` // Ignored: met reads always use ROW_MAJOR (variable-length buffers require it)
}
type NavQueryParams struct {
	EDID    []StringRange `json:"edid"`
	Time    []TimeRange   `json:"time"`
	Key     []StringRange `json:"key"`
	Ordered bool          `json:"ordered"` // Ignored: nav reads always use ROW_MAJOR (variable-length buffers require it)
}
type NavV2QueryParams struct {
	Time    []TimeRange   `json:"time"`
	Sys     []Uint8Range  `json:"system"`
	SvID    []Uint8Range  `json:"svid"`
	Key     []StringRange `json:"key"`
	Ordered bool          `json:"ordered"` // Ignored: nav reads always use ROW_MAJOR (variable-length buffers require it)
}

type ObsQueryParams 

type ObsQueryParams struct {
	Time    []TimeRange   `json:"time"`
	Sys     []Uint8Range  `json:"system"`
	SvID    []Uint8Range  `json:"svid"`
	Obs     []StringRange `json:"obs"`
	Ordered bool          `json:"ordered"` // Use ROW_MAJOR layout (enables C-level dedup, returns sorted by dims)
}

type ObsV2QueryParams 

type ObsV2QueryParams struct {
	EDID      []StringRange `json:"edid"`
	Time      []TimeRange   `json:"time"`
	Satellite []StringRange `json:"sat"`
	Obs       []StringRange `json:"obs"`
	Ordered   bool          `json:"ordered"` // Use ROW_MAJOR layout (enables C-level dedup, returns sorted by dims)
}

type OverlapGroup added in v0.37.1 

type OverlapGroup struct {
	Fragments []FragmentDetail `json:"fragments"`
	TimeStart int64            `json:"time_start"` // group union start
	TimeEnd   int64            `json:"time_end"`   // group union end
}

OverlapGroup represents a set of fragments with overlapping time ranges.

func FindGroups added in v0.37.1 

func FindGroups(fragments []FragmentDetail, strategy GroupingStrategy) []OverlapGroup

FindGroups organizes fragments into overlapping groups based on the strategy.

type StringRange 

type StringRange struct {
	Start string `json:"start"`
	End   string `json:"end"`
}

type TimeRange 

type TimeRange struct {
	Start time.Time `json:"start"`
	End   time.Time `json:"end"`
}

func GetDateArranged added in v0.16.2 

func GetDateArranged(start time.Time, end time.Time) []TimeRange

type Uint8Range 

type Uint8Range struct {
	Start uint8 `json:"start"`
	End   uint8 `json:"end"`
}

type WriteOptions added in v0.37.1 

type WriteOptions struct {
	Ordered bool // Use ROW_MAJOR layout instead of UNORDERED
}

WriteOptions controls write query behaviour. Pass as variadic parameter to Write* functions; zero value preserves current (UNORDERED) behaviour.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.