peeringdb-plus

module
v1.18.14 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 3, 2026 License: BSD-3-Clause

README

PeeringDB Plus

CI Go Reference

A high-performance, globally distributed, read-only mirror of PeeringDB data. PeeringDB Plus periodically re-fetches every PeeringDB object, stores the result in SQLite with edge replication via LiteFS on Fly.io, and serves the same dataset through five coexisting API surfaces.

Live instance: https://peeringdb-plus.fly.dev


Why

The upstream PeeringDB API is the canonical source for peering coordination data, but it is single-region and rate-limited. PeeringDB Plus offers:

  • Low-latency reads from the nearest Fly.io region (LiteFS replicates the SQLite WAL to every replica).
  • Multiple wire formats from a single dataset — drop-in PeeringDB API compatibility, OpenAPI REST, GraphQL, ConnectRPC/gRPC, and a Web UI all read from the same ent.Client.
  • Mandatory observability — OpenTelemetry traces, metrics, and structured logs are first-class, not an afterthought.
  • No CGO, no Java, no orchestrator — a single static Go binary plus an out-of-process LiteFS FUSE mount.

API surfaces

All five surfaces are mounted on the same HTTP server and read from the same SQLite database.

Surface Path Description
Web UI /ui/ templ + htmx search, detail pages, ASN comparison; content-negotiates terminal vs browser
GraphQL /graphql gqlgen via entgql; interactive playground on GET, queries on POST
REST /rest/v1/ OpenAPI-compliant (entrest); spec at /rest/v1/openapi.json
PeeringDB Compat /api/ Drop-in replacement for api.peeringdb.com — same paths, same envelope
ConnectRPC / gRPC /peeringdb.v1.*/ Get / List / Stream RPCs for all 13 entity types; reflection + health checks enabled

GET / returns a JSON service-discovery document; browsers are redirected to the Web UI, terminal clients receive plain help text.

See docs/API.md for the full surface catalogue, including filter semantics, ordering guarantees, divergences, and the response-memory envelope governing pdbcompat list responses.

Quick start

Local (Go)
git clone https://github.com/dotwaffle/peeringdb-plus.git
cd peeringdb-plus
go build -o peeringdb-plus ./cmd/peeringdb-plus
./peeringdb-plus
# Listening on :8080, syncing from api.peeringdb.com every hour

The first sync takes 30-60 seconds against the public PeeringDB API. While it runs, /healthz returns 200 and /readyz returns 503; once the database is populated, /readyz flips to 200 and every API surface is usable.

Local (Docker)
docker build -t peeringdb-plus .
docker run -p 8080:8080 -v pdbdata:/data peeringdb-plus

The image stores the database at /data/peeringdb-plus.db; mount a volume to persist data across container restarts.

Verify it's working
curl -s http://localhost:8080/healthz                    # liveness
curl -sI http://localhost:8080/readyz                    # readiness (200 once first sync completes)
curl -s http://localhost:8080/api/net/1 | head -c 500    # PeeringDB-compatible API
open http://localhost:8080/ui/                           # Web UI
open http://localhost:8080/graphql                       # GraphQL playground

For the full first-30-minutes walkthrough, see docs/GETTING-STARTED.md.

Usage examples

PeeringDB-compatible API
# List networks
curl https://peeringdb-plus.fly.dev/api/net

# Fetch a specific network by PeeringDB numeric ID
curl https://peeringdb-plus.fly.dev/api/net/42

# Search with query parameters (depth/limit/skip/fields/since/q all supported)
curl 'https://peeringdb-plus.fly.dev/api/net?q=cloudflare&limit=5'
ConnectRPC / gRPC

ConnectRPC, gRPC, and gRPC-Web all share the same endpoints. Reflection (grpcreflect.NewHandlerV1 / V1Alpha) and the standard gRPC health check service are enabled, so grpcurl and gRPC-aware load balancers work out-of-the-box.

# List all services
grpcurl -plaintext localhost:8080 list

# Get a single network by ID
buf curl --protocol grpc --http2-prior-knowledge \
  http://localhost:8080/peeringdb.v1.NetworkService/GetNetwork \
  -d '{"id": 42}'

# Stream all networks (server-streaming, no manual pagination)
buf curl --protocol grpc --http2-prior-knowledge \
  http://localhost:8080/peeringdb.v1.NetworkService/StreamNetworks \
  -d '{"asn": 15169}'

Streams accept the same filters as List* RPCs; the grpc-total-count response header carries the approximate total. Server-side timeout defaults to 60s (PDBPLUS_STREAM_TIMEOUT) and clients can cancel at any time.

GraphQL
curl -X POST http://localhost:8080/graphql \
  -H 'Content-Type: application/json' \
  -d '{"query":"{ networks(first: 3) { edges { node { id name asn } } } }"}'

Configuration

PeeringDB Plus is configured exclusively via environment variables, validated at startup with fail-fast diagnostics. Operationally-relevant defaults:

Variable Default Purpose
PDBPLUS_LISTEN_ADDR :8080 HTTP listen address
PDBPLUS_DB_PATH ./peeringdb-plus.db SQLite database file path
PDBPLUS_PEERINGDB_URL https://api.peeringdb.com Upstream PeeringDB API
PDBPLUS_PEERINGDB_API_KEY (unset) Optional — raises rate limit and shortens default sync interval
PDBPLUS_SYNC_MODE incremental incremental (delta) or full (re-fetch); full is the operator escape-hatch
PDBPLUS_SYNC_INTERVAL 1h (15m if API key set) Time between sync cycles
PDBPLUS_RESPONSE_MEMORY_LIMIT 128MiB pdbcompat list pre-flight 413 budget
PDBPLUS_PUBLIC_TIER public Anonymous-caller tier; set users only for private deployments

The full catalogue (sync, observability, LiteFS, Fly.io, CSP, security headers, and OAuth-gated visibility) lives in docs/CONFIGURATION.md. Standard OTEL_* env vars are honoured via OpenTelemetry autoexport.

Documentation

Doc Topic
docs/GETTING-STARTED.md First-30-minutes walkthrough: prerequisites, build, first run, verification
docs/ARCHITECTURE.md Component diagram, data flow, code-generation pipeline, middleware chain, privacy layer, sampling matrix
docs/API.md All five API surfaces, filter semantics, ordering, cross-entity traversal, divergences
docs/CONFIGURATION.md Full environment-variable catalogue with validation rules
docs/DEVELOPMENT.md Local dev workflow, code generation, conventions, sibling-file pattern
docs/TESTING.md Test layout, fixtures, parity harness, live tests against beta.peeringdb.com
docs/DEPLOYMENT.md Fly.io rollout, asymmetric fleet topology, LiteFS operations
CONTRIBUTING.md How to propose changes, run the verification suite, and submit a PR

Technology

  • Language: Go 1.26.1+
  • ORM / codegen: entgo drives all five API surfaces from a single set of schemas in ent/schema/ (entgql + entrest + entproto)
  • Database: modernc.org/sqlite (pure Go, no CGO) plus LiteFS for edge replication
  • RPC: ConnectRPC — gRPC, gRPC-Web, and the Connect protocol on the same handlers
  • GraphQL: gqlgen wired through entgql
  • Web UI: templ + htmx + Tailwind CSS — zero JavaScript build toolchain
  • Observability: OpenTelemetry with the stdlib log/slog bridge; per-route trace sampling; per-VM runtime metrics
  • Platform: Fly.io with an asymmetric primary / replica process-group fleet
  • Container base: Chainguard minimal images (cgr.dev/chainguard/go, cgr.dev/chainguard/glibc-dynamic)

Development

go build ./...                    # Build all packages
go test -race ./...               # Run tests with race detector
go generate ./...                 # Full codegen pipeline (ent + buf + templ)
golangci-lint run                 # Lint
govulncheck ./...                 # Vulnerability scan

CI runs the full pipeline plus a generated-code-drift check on every PR. See docs/DEVELOPMENT.md for the conventions, including the sibling-file pattern that protects hand-edited schema methods from being overwritten by cmd/pdb-schema-generate.

Contributing

PeeringDB Plus is open source. Bug reports, feature suggestions, and pull requests are welcome — please read CONTRIBUTING.md before opening a PR.

License

BSD 3-Clause. See LICENSE.

Directories

Path Synopsis
cmd
loadtest command
Command loadtest is an operator tool that exercises every API surface of a peeringdb-plus deployment (default https://peeringdb-plus.fly.dev) for capacity validation, dashboard warmup, and load reproduction.
Command loadtest is an operator tool that exercises every API surface of a peeringdb-plus deployment (default https://peeringdb-plus.fly.dev) for capacity validation, dashboard warmup, and load reproduction.
pdb-compat-allowlist command
Command pdb-compat-allowlist reads the ent schema graph and emits internal/pdbcompat/allowlist_gen.go with:
Command pdb-compat-allowlist reads the ent schema graph and emits internal/pdbcompat/allowlist_gen.go with:
pdb-fixture-port command
Command pdb-fixture-port ports Django fixture blocks from peeringdb/peeringdb's src/peeringdb_server/management/commands/ pdb_api_test.py into Go struct literals committed to internal/testutil/parity/fixtures.go.
Command pdb-fixture-port ports Django fixture blocks from peeringdb/peeringdb's src/peeringdb_server/management/commands/ pdb_api_test.py into Go struct literals committed to internal/testutil/parity/fixtures.go.
pdb-schema-extract command
pdb-schema-extract parses PeeringDB Django serializer and model Python source to produce an intermediate JSON schema representation.
pdb-schema-extract parses PeeringDB Django serializer and model Python source to produce an intermediate JSON schema representation.
pdb-schema-generate command
pdb-schema-generate reads an intermediate JSON schema produced by pdb-schema-extract and generates entgo schema Go files.
pdb-schema-generate reads an intermediate JSON schema produced by pdb-schema-extract and generates entgo schema Go files.
pdbcompat-check command
Command pdbcompat-check fetches responses from the PeeringDB API and compares their structure against local golden files to detect drift.
Command pdbcompat-check fetches responses from the PeeringDB API and compares their structure against local golden files to detect drift.
peeringdb-plus command
Package main is the entry point for the peeringdb-plus application.
Package main is the entry point for the peeringdb-plus application.
ent
poc
schema
Package schema-level hooks live here.
Package schema-level hooks live here.
schematypes
Package schematypes hosts Go value types referenced by ent schemas via field.JSON.
Package schematypes hosts Go value types referenced by ent schemas via field.JSON.
gen
Package graph provides the GraphQL resolver layer for the PeeringDB Plus API.
Package graph provides the GraphQL resolver layer for the PeeringDB Plus API.
model
Package model defines custom GraphQL model types that are not generated by ent.
Package model defines custom GraphQL model types that are not generated by ent.
internal
buildinfo
Package buildinfo exposes the build-time version string used by both the PeeringDB User-Agent and the OTel resource so they stay in lockstep.
Package buildinfo exposes the build-time version string used by both the PeeringDB User-Agent and the OTel resource so they stay in lockstep.
config
Package config loads application configuration from environment variables.
Package config loads application configuration from environment variables.
conformance
Package conformance provides structural comparison of JSON API responses for validating PeeringDB compatibility layer output against the real PeeringDB API.
Package conformance provides structural comparison of JSON API responses for validating PeeringDB compatibility layer output against the real PeeringDB API.
database
Package database provides SQLite database setup for the ent ORM client.
Package database provides SQLite database setup for the ent ORM client.
graphql
Package graphql provides the HTTP handler factory for the PeeringDB Plus GraphQL API.
Package graphql provides the HTTP handler factory for the PeeringDB Plus GraphQL API.
grpcserver
Package grpcserver provides ConnectRPC service handlers for the PeeringDB gRPC API.
Package grpcserver provides ConnectRPC service handlers for the PeeringDB gRPC API.
health
Package health provides HTTP handlers for liveness and readiness checks.
Package health provides HTTP handlers for liveness and readiness checks.
httperr
Package httperr provides RFC 9457 Problem Details for HTTP API error responses.
Package httperr provides RFC 9457 Problem Details for HTTP API error responses.
litefs
Package litefs provides utilities for detecting the role of the current node in a LiteFS cluster (primary vs replica).
Package litefs provides utilities for detecting the role of the current node in a LiteFS cluster (primary vs replica).
middleware
Package middleware provides HTTP middleware for the PeeringDB Plus server.
Package middleware provides HTTP middleware for the PeeringDB Plus server.
otel
Package otel initializes the OpenTelemetry trace, metric, and log pipelines.
Package otel initializes the OpenTelemetry trace, metric, and log pipelines.
pdbcompat
Package pdbcompat provides a PeeringDB-compatible REST API layer that translates Django-style query parameters to ent predicates and serializes ent entities to PeeringDB's exact JSON response format.
Package pdbcompat provides a PeeringDB-compatible REST API layer that translates Django-style query parameters to ent predicates and serializes ent entities to PeeringDB's exact JSON response format.
pdbcompat/parity
Package parity holds regression tests that lock v1.16 pdbcompat semantics against future drift.
Package parity holds regression tests that lock v1.16 pdbcompat semantics against future drift.
pdbcompat/schemaannot
Package schemaannot exposes ent schema annotation types consumed by ent/schema/*.go to describe pdbcompat's Path A allowlist (Phase 70 D-01) and per-edge FILTER_EXCLUDE (D-03).
Package schemaannot exposes ent schema annotation types consumed by ent/schema/*.go to describe pdbcompat's Path A allowlist (Phase 70 D-01) and per-edge FILTER_EXCLUDE (D-03).
peeringdb
Package peeringdb provides a client for the PeeringDB API with rate limiting, pagination, retry logic, and response types for all 13 PeeringDB object types.
Package peeringdb provides a client for the PeeringDB API with rate limiting, pagination, retry logic, and response types for all 13 PeeringDB object types.
privctx
Package privctx propagates the visibility tier of the caller through a Go context.
Package privctx propagates the visibility tier of the caller through a Go context.
privfield
This file exists so future exported symbols in the package can accrete documentation here without crowding the Redact contract in privfield.go.
This file exists so future exported symbols in the package can accrete documentation here without crowding the Redact contract in privfield.go.
sync
Package sync orchestrates data synchronization from PeeringDB into the local SQLite database using the ent ORM.
Package sync orchestrates data synchronization from PeeringDB into the local SQLite database using the ent ORM.
testutil
Package testutil provides shared test helpers for ent client setup.
Package testutil provides shared test helpers for ent client setup.
testutil/parity
Package parity holds upstream-ported fixture data for the Phase 72 parity regression test suite.
Package parity holds upstream-ported fixture data for the Phase 72 parity regression test suite.
testutil/seed
Package seed provides deterministic test data seeding for PeeringDB entity types.
Package seed provides deterministic test data seeding for PeeringDB entity types.
unifold
Package unifold folds Unicode strings into a normalised ASCII-lowercase form suitable for diacritic-insensitive equality and substring matching.
Package unifold folds Unicode strings into a normalised ASCII-lowercase form suitable for diacritic-insensitive equality and substring matching.
visbaseline
Package visbaseline captures and diffs unauthenticated vs authenticated PeeringDB API responses for all 13 object types, with a strict no-PII-in-repo guarantee enforced by the Redact function in this package.
Package visbaseline captures and diffs unauthenticated vs authenticated PeeringDB API responses for all 13 object types, with a strict no-PII-in-repo guarantee enforced by the Redact function in this package.
web
web/templates
templ: version: v0.3.1001
templ: version: v0.3.1001
web/termrender
Package termrender provides terminal client detection and ANSI text rendering for serving styled text responses to CLI clients like curl, wget, and HTTPie.
Package termrender provides terminal client detection and ANSI text rendering for serving styled text responses to CLI clients like curl, wget, and HTTPie.
Package schema contains the intermediate PeeringDB JSON schema and generate directives for the schema extraction pipeline.
Package schema contains the intermediate PeeringDB JSON schema and generate directives for the schema extraction pipeline.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL