dp-sitemap

command module

v0.3.0 Latest Latest Go to latest Published: Dec 18, 2023 License: MIT Imports: 8 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/ONSdigital/dp-sitemap

Links

Open Source Insights

README ¶

dp-sitemap

This repo holds all information/code regarding sitemap (for SEO and other purposes).

Structure of robot.json

Holds allow/deny list for different user-agents.

{
    "Googlebot": {
      "AllowList": ["/googleallow1", "/googleallow2"],
      "DenyList":  ["/googledeny"]
    },
    "Bingbot": {
        "AllowList": ["/bingcontent"],
        "DenyList":  ["/bingdeny1", "/bingdeny2"]
    },
      "*": {
        "AllowList": ["/"],
        "DenyList":  ["/private"]
    }
}

Getting started

Run make debug

The service runs in the background consuming messages from Kafka. An example event can be created using the helper script, make produce.

Dependencies

Requires running…
- kafka
No further dependencies other than those defined in go.mod

Configuration

Environment variable	Default	Description
BIND_ADDR	localhost:	The host and port to bind to
GRACEFUL_SHUTDOWN_TIMEOUT	5s	The graceful shutdown timeout in seconds (`time.Duration` format)
HEALTHCHECK_INTERVAL	30s	Time between self-healthchecks (`time.Duration` format)
HEALTHCHECK_CRITICAL_TIMEOUT	90s	Time to wait until an unhealthy dependent propagates its state to make this app unhealthy (`time.Duration` format)
KAFKA_ADDR	"localhost:9092"	The address of Kafka (accepts list)
KAFKA_OFFSET_OLDEST	true	Start processing Kafka messages in order from the oldest in the queue
KAFKA_NUM_WORKERS	1	The maximum number of parallel kafka consumers
KAFKA_SEC_PROTO	unset	if set to `TLS`, kafka connections will use TLS (kafka TLS doc)
KAFKA_SEC_CA_CERTS	unset	CA cert chain for the server cert (kafka TLS doc)
KAFKA_SEC_CLIENT_KEY	unset	PEM for the client key (kafka TLS doc)
KAFKA_SEC_CLIENT_CERT	unset	PEM for the client certificate (kafka TLS doc)
KAFKA_SEC_SKIP_VERIFY	false	ignores server certificate issues if `true` (kafka TLS doc)
KAFKA_CONTENT_UPDATED_GROUP	dp-sitemap	The consumer group this application to consume topic messages
KAFKA_CONTENT_UPDATED_TOPIC	content-updated	The name of the topic to consume messages from

Healthcheck

The /health endpoint returns the current status of the service. Dependent services are health checked on an interval defined by the HEALTHCHECK_INTERVAL environment variable.

On a development machine a request to the health check endpoint can be made by:

curl localhost:8125/health