Monitoring

The Monitoring component is in charge of the collection, process and storage of various signals (i.e. logs, metrics and distributed traces). It is extensively based upon OpenTelemetry, with its Collector.

[!CAUTION]

This component is an internal work mostly used for development purposes. It is used for production purposes too, i.e. on Capture The Flag events.

Nonetheless, we do not include it in the repositories we are actively maintaining.

Architecture

The Monitoring service's architecture currently provides:

• OpenTelemetry Collector
• Jaeger UI
• Prometheus

The multiple parts passes information in a non-cyclic way to avoid deadlocks (DAG or dependencies), as summarized in the following diagram.

Cold Extract

For research and/or development purposes, the architecture provide way to perform an extraction of the OpenTelemetry data.

1. Activate cold extract:

pulumi config set cold-extract true

1. Use the Monitoring through the OTEL Collector endpoint.

2. Run the extractor:

mkdir extract
go run cmd/extractor/main.go \
  --namespace $(pulumi stack export namespace) \
  --pvc-name $(pulumi stack export otel-cold-extract-pvc-name) \
  --directory extract

TODO list

• Add Grafana (require Prometheus)
• Add AlertManager (require Prometheus)
• Think of Dashboard as Code, and if it is relevant to extract them (so technically how ? With Docker manifests inspection ? Service discovery and custom protocol ?) automatically ?