README
¶
dp-filter-api
Getting started
mongo
- Run
brew install mongodb - Run
brew services start mongodb
Kafka
- Run
brew install zookeeper - Download kafka version 0.10.2.1
- Run
./kafka-server-start.sh ../config/server.properties
Scripts for updating and debugging Kafka can be found here(dp-data-tools)
Configuration
- Run
brew install neo4j - Disable authentication in the config
- Run
brew services start neo4j
| Environment variable | Default | Description |
|---|---|---|
| BIND_ADDR | :22100 | The host and port to bind to |
| HOST | http://localhost:22100 | The host name used to build URLs |
| KAFKA_ADDR | localhost:9092 | The kafka broker addresses (can be comma separated) |
| KAFKA_MAX_BYTES | 2000000 | The maximum permitted size of a message. Should be set equal to or smaller than the broker's message.max.bytes |
| KAFKA_SEC_PROTO | unset | if set to TLS, kafka connections will use TLS [1] |
| KAFKA_SEC_CLIENT_KEY | unset | PEM for the client key [1] |
| KAFKA_SEC_CLIENT_CERT | unset | PEM for the client certificate [1] |
| KAFKA_SEC_CA_CERTS | unset | CA cert chain for the server cert [1] |
| KAFKA_SEC_SKIP_VERIFY | false | ignores server certificate issues if true [1] |
| FILTER_JOB_SUBMITTED_TOPIC | filter-job-submitted | The kafka topic to write messages to |
| MONGODB_BIND_ADDR | localhost:27017 | The MongoDB bind address |
| MONGODB_USERNAME | The MongoDB Username | |
| MONGODB_PASSWORD | The MongoDB Password | |
| MONGODB_DATABASE | filters | The MongoDB database |
| MONGODB_COLLECTIONS | FiltersCollection:filters, OutputsCollection:filterOutputs | The MongoDB collections |
| MONGODB_REPLICA_SET | The name of the MongoDB replica set | |
| MONGODB_ENABLE_READ_CONCERN | false | Switch to use (or not) majority read concern |
| MONGODB_ENABLE_WRITE_CONCERN | true | Switch to use (or not) majority write concern |
| MONGODB_CONNECT_TIMEOUT | 5s | The timeout when connecting to MongoDB (time.Duration format) |
| MONGODB_QUERY_TIMEOUT | 15s | The timeout for querying MongoDB (time.Duration format) |
| MONGODB_IS_SSL | false | Switch to use (or not) TLS when connecting to mongodb |
| SHUTDOWN_TIMEOUT | 5s | The graceful shutdown timeout (time.Duration format) |
| DATASET_API_URL | http://localhost:22000 | The URL of the Dataset API |
| HEALTHCHECK_INTERVAL | 30s | Time between self-healthchecks (time.Duration format) |
| HEALTHCHECK_CRITICAL_TIMEOUT | 90s | The time taken for the health changes from warning state to critical due to subsystem check failures |
| OTEL_BATCH_TIMEOUT | 5s | Interval between pushes to OT Collector |
| OTEL_EXPORTER_OTLP_ENDPOINT | http://localhost:4317 | URL for OpenTelemetry endpoint |
| OTEL_SERVICE_NAME | "dp-filter-api" | Service name to report to telemetry tools |
| SERVICE_AUTH_TOKEN | FD0108EA-825D-411C-9B1D-41EF7727F465 | The token used to identify this service when authenticating |
| ZEBEDEE_URL | "http://localhost:8082" | Zebedee URL |
| ENABLE_PRIVATE_ENDPOINTS | false | true if private endpoints should be enabled |
| DOWNLOAD_SERVICE_URL | http://localhost:23600 | The URL of the download service |
| DOWNLOAD_SERVICE_SECRET_KEY | QB0108EZ-825D-412C-9B1D-41EF7747F462 | The service token for the download service |
| ENABLE_URL_REWRITING | false | Feature flag to enable URL rewriting |
Notes:
- For more info, see the kafka TLS examples documentation
Healthchecking
Currently checked each $HEALTHCHECK_INTERVAL and reported on endpoint /healthcheck:
- Neo4j
- MongoDB
- Dataset API
Acceptance tests
To run the acceptance tests for this project, use the following commands
- Start Kafka and Mongodb
- Run the dp-filter-api using
make acceptance - Run the tests in dp-api-test
Contributing
See CONTRIBUTING for details.
License
Copyright © 2016-2022, Office for National Statistics (https://www.ons.gov.uk)
Released under MIT license, see LICENSE for details.
Documentation
¶
There is no documentation for this package.
Source Files
Directories
Click to show internal directories.
Click to hide internal directories.