Deprecated: No longer in active development
README
¶
dp-geodata-api
⚠️ This repository will be archived in August 2024 as it is no longer in development. ⚠️
Census Atlas geodata API and database.
Further Docs
-
Environment Variables
Database connection details and feature flags are controlled with environment variables. Almost all processes depend on these variables being set correctly, so be careful to verify before running anything risky.
-
Postgres/PostGIS
The system is designed around Postgres with PostGIS extensions. The database may be run as a native local process, within a container, or as an RDS instance.
- local container
- local native instance (docs to be written)
- RDS provisioning (see dp-setup repo)
-
Data Ingest
The "ingest" process takes source data files and loads them into the database. For best results, the ingest should be applied to a local native postgres instance. Running against RDS directly or against a containerised postgres instance is too slow.
Source data files should be in place, and the postgres environment variables must be exported before running the ingest script (
indigestion.sh).- source data files
- running
-
Export/Import
The export/import processes copies a database. Our main application is to export a locally ingested database for import into RDS.
-
API
The API presents a specific interface for querying the database. It is mainly used by the front end application.
The API may be run as a local process, within a container, or on some temporary EC2 instances we have in place for the moment. The plan is that instance of the API will run within the standard develop and prod environments.
-
CLIs
These clis are mostly for developers, and may not be fully caught up with features of the API.
- geodata cli
- cantabular cli
- geobb (to generate static
geoLookup.jsonfile)
-
Testing
-
Terraform
- S3 and misc
- RDS (see dp-setup repo)
-
How-Tos
Getting started
- Run
make debug
To enable postgres and census queries, set ENABLE_DATABASE, and the postgres environment variables.
To lookup the postgres password in Secrets Manager, set FI_PG_SECRET_ID instead of PGPASSWORD,
and make sure to set AWS_REGION and any other AWS environment variables.
Auto generated code
swagger.yaml is used to generate code via make generate
Particularly api/api.go (and similar files) shouldn't be directly edited.
Configuration
| Environment variable | Default | Description |
|---|---|---|
| BIND_ADDR | :25252 | The host and port to bind to |
| GRACEFUL_SHUTDOWN_TIMEOUT | 5s | The graceful shutdown timeout in seconds (time.Duration format) |
| HEALTHCHECK_INTERVAL | 30s | Time between self-healthchecks (time.Duration format) |
| HEALTHCHECK_CRITICAL_TIMEOUT | 90s | Time to wait until an unhealthy dependent propagates its state to make this app unhealthy (time.Duration format) |
| ENABLE_DATABASE | false | Enable postgres and census query functionality |
| AWS_REGION | used by AWS SDK when ENABLE_DATABASE is true and PGPASSWORD is empty | |
| PGHOST | postgres host when ENABLE_DATABASE is true | |
| PGPORT | postgres port when ENABLE_DATABASE is true | |
| PGUSER | postgres user when ENABLE_DATABASE is true | |
| PGPASSWORD | postgres password when ENABLE_DATABASE is true (also see FI_PG_SECRET_ID) | |
| PGDATABASE | postgres database when ENABLE_DATABASE is true | |
| FI_PG_SECRET_ID | ARN of key holding postgres password if PGPASSWORD is empty | |
| DO_CORS | false | Add Access-Control-Allow-Origin: * to headers if true (not needed in develop / prod) |
Contributing
See CONTRIBUTING for details.
License
Copyright © 2022, Office for National Statistics (https://www.ons.gov.uk)
Released under MIT license, see LICENSE for details.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package api provides primitives to interact with the openapi HTTP API.
|
Package api provides primitives to interact with the openapi HTTP API. |
|
The cache package provides content caching and a mechanism to prevent cache stampedes.
|
The cache package provides content caching and a mechanism to prevent cache stampedes. |
|
cmd
|
|
|
cantabular
command
|
|
|
creatschema
command
|
|
|
dp-geodata-api
command
|
|
|
filescli
command
|
|
|
gentiles
command
|
|
|
geobb
command
|
|
|
geodata
command
|
|
|
data-tiles
|
|
|
cmd/extract-categories
command
|
|
|
cmd/fake-data
command
|
|
|
cmd/generate-breaks
command
|
|
|
cmd/generate-tiles
command
|
|
|
cmd/geolookup
command
|
|
|
cmd/map-categories
command
|
|
|
cmd/normalise
command
|
|
|
cmd/recode-lads
command
|
|
|
cmd/rename-msoas
command
|
|
|
cmd/split-geojson
command
|
|
|
cmd/split-metrics
command
|
|
|
cmd/xls2csv
command
|
|
|
content
Load and save content.json
|
Load and save content.json |
|
dataingest
|
|
|
addtodb
command
|
|
|
geoname
command
|
|
|
postcode
command
|
|
|
spatial
command
|
|
|
spatial/longlatgeom
command
|
|
|
features
|
|
|
pkg
|
|
|
database
The database package is super thin layer on sql that provides a Checker.
|
The database package is super thin layer on sql that provides a Checker. |
|
table
The table package implements a simplistic 2-dimensional array that can be populated one cell at a time, and output as a CSV.
|
The table package implements a simplistic 2-dimensional array that can be populated one cell at a time, and output as a CSV. |
|
Package Swagger provides primitives to interact the openapi HTTP API.
|
Package Swagger provides primitives to interact the openapi HTTP API. |