gpkgio

package
v0.2.13 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 27, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package gpkgio reads and writes gobi Frames as OGC GeoPackage (SQLite) files.

GeoPackage 1.3 (application_id 0x47504B47, user_version 10300) is the target — every WriteFile emits a file that QGIS, GDAL / ogr2ogr, and other GeoPackage-aware tools recognize as a compliant feature GeoPackage. Each geometry is stored as the standard GPB blob (magic + version + flags + SRS ID + optional envelope + WKB payload) per spec §2.1.3.1.

The package offers three entry points:

  • ReadFile materializes a single layer as a Frame. Peak memory scales with the layer size; good for small/medium layers.

  • ReadFileChunksFunc streams a layer as record-batch-sized Frames, releasing arrow buffers after each callback. Peak memory is bounded regardless of layer size.

  • ScanFile returns a gobi.LazyFrame — participates in the optimizer's projection pushdown (SELECT column list is narrowed to what the plan actually uses) and streams under the hood. Predicate-pushdown-to-SQL is not implemented yet; callers who need SQL-side filtering can use ReadOptions.Where directly.

Write is transactional and prepared-statement-based: rows batch into transactions of WriteOptions.BatchSize (default 1000). The standard GeoPackage RTree spatial index (rtree_<layer>_<geomcol>) is created and populated inline during the insert loop; gobi maintains it from Go rather than via SpatiaLite triggers, since pure-Go modernc.org/sqlite doesn't ship ST_MinX/ST_MaxX/ST_IsEmpty.

Multi-layer GeoPackages are supported: each WriteFile appends its layer to the target file, leaving other layers untouched. Use WriteOptions.Replace to overwrite an existing layer of the same name.

Index

Constants

This section is empty.

Variables

View Source
var ErrInvalidHeader = errors.New("gpkg: invalid geometry header")

ErrInvalidHeader is returned when a geometry blob lacks the "GP" magic.

Functions

func DecodeGeometry

func DecodeGeometry(b []byte) (geometry.Geometry, error)

DecodeGeometry decodes a GeoPackage geometry blob (header + WKB) into a geometry.Geometry, attaching the header's SRS as a CRS if it maps to a known one.

func ReadFile

func ReadFile(path string, opts *ReadOptions) (*gobi.Frame, error)

ReadFile opens the GeoPackage at path and reads one layer's rows into a Frame. The geometry column (if present) is returned as a Binary column tagged with gobi.GeometryField metadata carrying the layer's SRS.

The layer's arrow schema is inferred from the SQLite column affinities: INTEGER → Int64, REAL → Float64, TEXT → String, BLOB → Binary. The geometry column always comes out as Binary regardless of affinity so the WKB payload round-trips through WriteFile without re-encoding.

For streaming reads, ReadFileChunksFunc is the callback-based counterpart — bounded memory regardless of layer size.

func ReadFileChunksFunc

func ReadFileChunksFunc(path string, opts *ReadOptions, fn func(*gobi.Frame) error) error

ReadFileChunksFunc streams the rows of one layer through fn one batch at a time. Peak memory is bounded to one batch (default 64k rows). Returning a non-nil error from fn aborts the read.

Batch boundaries are just row-count-based; there's no attempt to align with SQLite's B-tree pages or the RTree. If you need predicate-driven skipping, use ScanFile instead — it drives the LazyFrame optimizer which pushes WHERE clauses into the SQL.

func ReadSchema

func ReadSchema(path string, opts *ReadOptions) (*arrow.Schema, error)

ReadSchema opens path just far enough to build the arrow schema for the target layer. Cheap — it queries the SQLite catalog (pragma_table_info) without scanning any rows.

Layer selection follows the same rules as ReadFile: opts.Layer if set, or the file's single feature table otherwise. Multiple feature tables with no opts.Layer is an error.

func ScanFile

func ScanFile(path string, opts *ReadOptions) *gobi.LazyFrame

ScanFile returns a LazyFrame that reads path/opts when Collect is called. The scan participates in gobi's optimizer for all three pushdown categories:

  • Projection pushdown: Frame.Select() above the scan restricts the SQL SELECT column list at Collect time, reducing what SQLite decodes.
  • Predicate pushdown: Frame.FilterExpr() above the scan is translated to a SQL WHERE fragment via gobi.ExprToSQL. Parts that translate (col-vs-lit comparisons, AND, NOT) run in SQLite and skip both decode + row materialization; parts that don't (custom nodes, unsupported ops) stay in the executor. Splits at top-level ANDs via gobi.SplitConjuncts so partial translation still delivers a real speedup.
  • Streaming: reads flow one batch at a time through ReadFileChunksFunc, so peak memory stays bounded even on large layers.

The GeoPackage RTree (created by WriteFile) is not consulted at scan build time. It would kick in automatically if the SQL WHERE referenced the fid via an RTree join — a follow-up capability that requires spatial predicate awareness in ExprToSQL.

func WriteFile

func WriteFile(df *gobi.Frame, path string, opts *WriteOptions) error

WriteFile writes df to a GeoPackage file at path, creating the file if it doesn't exist. When the file already exists, the layer is appended (or replaced when opts.Replace is true) — other layers stay untouched.

The Frame's geometry column (if any) is detected via the arrow schema metadata that gobi.GeometryField() sets; alternatively opts.GeomCol can name it explicitly. Non-geometry columns are written as native SQLite affinity types based on the Frame's arrow schema:

Int8/16/32/64, Uint8/16/32/64 → INTEGER
Float32/64                    → REAL
Bool                          → INTEGER (0 or 1)
String, LargeString           → TEXT
Binary, LargeBinary           → BLOB
Timestamp                     → TEXT (ISO 8601)
other                         → TEXT via string(v) fallback

Batch inserts run in a single transaction of opts.BatchSize rows each. Prepared statements are reused across the write.

Types

type Feature

type Feature struct {
	Attributes map[string]any
	Geometry   geometry.Geometry
}

Feature is one row from a feature table.

type FeatureTable

type FeatureTable struct {
	Name     string
	GeomCol  string
	SRID     int32
	GeomType string
}

FeatureTable describes one feature table registered in gpkg_geometry_columns.

type GeoPackage

type GeoPackage struct {
	// contains filtered or unexported fields
}

GeoPackage is an open GeoPackage database.

func Open

func Open(path string) (*GeoPackage, error)

Open opens the GeoPackage file at path.

func (*GeoPackage) Close

func (g *GeoPackage) Close() error

Close releases the database handle.

func (*GeoPackage) FeatureTables

func (g *GeoPackage) FeatureTables() ([]FeatureTable, error)

FeatureTables returns every registered feature table.

func (*GeoPackage) ReadFeatures

func (g *GeoPackage) ReadFeatures(table string) ([]Feature, error)

ReadFeatures returns every row of the named feature table with its geometry decoded into a geometry.Geometry.

type ReadOptions

type ReadOptions struct {
	// Layer names the feature table to read. Required when the file
	// has more than one feature table; optional otherwise.
	Layer string

	// Columns optionally restricts the read to these column names.
	// nil / empty means "read every column". The geometry column
	// is always included when it's present in the source layer,
	// even if it isn't listed here — GeoPackage's whole point is
	// geometry, and dropping it silently would surprise users. If
	// you truly want an attribute-only projection, use
	// `SELECT *` directly against the underlying sql.DB.
	Columns []string

	// Where is an optional SQL WHERE fragment appended verbatim
	// after WHERE in the SELECT. Callers are responsible for their
	// own quoting — this is a lower-level knob than a full
	// gobi.Expr predicate. ScanFile lifts predicate pushdown into
	// this automatically for expressions the SQL translator can
	// handle; ReadFile leaves it to callers.
	//
	// Use `?` placeholders in the fragment and supply matching
	// values via WhereArgs — driver-safe parameterization. Callers
	// concatenating literal values into the string are responsible
	// for their own escaping.
	Where string

	// WhereArgs holds the positional bind arguments for `?` markers
	// in Where. Filled automatically when ScanFile pushes a
	// translated predicate down; callers who set Where by hand can
	// supply their own args here.
	WhereArgs []any

	// Limit caps the number of rows returned. 0 = unlimited.
	Limit int64

	// Allocator overrides the arrow allocator used for the produced
	// Frame's columns. nil = memory.DefaultAllocator. Provided for
	// callers who pool arrow buffers across pipelines.
	Allocator memory.Allocator
	// contains filtered or unexported fields
}

ReadOptions controls ReadFile / ScanFile behavior.

Empty / zero fields fall back to sensible defaults: Layer picks the only feature table when the file has one, or errors otherwise.

type WriteOptions

type WriteOptions struct {
	// Layer names the feature table to create/replace. Required.
	// Must be a valid SQLite identifier (letters, digits, underscore,
	// not starting with a digit) — WriteFile will reject anything
	// else so we can safely interpolate it into DDL without a full
	// quoting round-trip.
	Layer string

	// GeomCol names the geometry column in the output layer.
	// Defaults to the first Frame column tagged as WKB geometry
	// (via gobi.GeometryField). If the Frame has no geometry
	// column at all, the file is written as an attribute-only
	// table (still valid GeoPackage, just no gpkg_geometry_columns
	// entry for this layer).
	GeomCol string

	// SRID is the spatial reference identifier stamped into every
	// row's GPB header + registered in gpkg_contents. Defaults to
	// the value carried by the geometry column's schema metadata
	// (gobi.MetaGeometryCRS); if that's unset, defaults to 4326.
	//
	// The corresponding row in gpkg_spatial_ref_sys is inserted
	// on-demand — the default 4326 (WGS84) is always registered
	// even for attribute-only layers so the file passes strict
	// validators.
	SRID int32

	// BatchSize is the number of rows per transaction commit. Higher
	// values reduce commit overhead but grow the WAL / journal on
	// crash-safety trade-off. 1000 is a reasonable default matching
	// GDAL's ogr2ogr for GPKG output.
	BatchSize int

	// SkipRTree, when true, disables creation of the standard
	// GeoPackage rtree_<layer>_<geomcol> shadow table. Skipped
	// automatically when the Frame has no geometry column.
	//
	// The RTree is created by default because the per-insert
	// overhead is small and it unlocks fast bbox filtering on
	// downstream reads (both from gobi and from external tools
	// like QGIS / GDAL). Use this opt-out only when you're
	// writing intermediate scratch files where read-side
	// filtering doesn't matter.
	SkipRTree bool

	// Replace, when true, drops any pre-existing layer with the
	// same name (feature table + gpkg_contents row + gpkg_geometry_columns
	// row + RTree shadow tables + triggers) before creating the new
	// one. When false, WriteFile errors if the layer already exists —
	// the safer default that keeps us from silently clobbering data
	// in an existing multi-layer GeoPackage.
	Replace bool
}

WriteOptions controls WriteFile / Writer behavior.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL