d3d-api

d3d-api

Go REST + WebSocket backend for d3dweb. Stores directed acyclic graphs (DAGs) in PostgreSQL and provides real-time collaborative editing via a WebSocket relay.

Tech stack

Prerequisites

• Go 1.23+
• Docker (for local PostgreSQL)
• gin for live-reload during development (optional: go install github.com/codegangsta/gin@latest)

Local development setup

1. Start PostgreSQL

make postgres-start

That starts the existing pg container, or creates one if it is missing:

docker run -d --name pg -p 5432:5432 -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=samus postgres:16-alpine

POSTGRES_DB=samus matters because the DSN below names that database. It only takes effect when the data directory is initialized, so a container created without it keeps failing with database "samus" does not exist. Create the database once by hand rather than recreating the container:

docker exec pg psql -U postgres -c 'CREATE DATABASE samus;'

The schema itself needs no setup. Init applies the embedded goose migrations on startup (app/db/postgres/migrations/), so an empty database is enough.

2. Configure the API

Copy the template; your copy is gitignored, so edit it freely:

cp samus_dev.toml.example samus_dev.toml

It defaults to auth_provider = "localauth" (no LDAP server needed) and points at the container from step 1:

[postgres]
    dsn = "postgres://postgres:postgres@localhost:5432/samus?sslmode=disable"

Alternatively, spell out the connection as discrete fields under [postgresql] (a DSN is assembled for you, with the password properly escaped):

[postgresql]
    address  = "localhost:5432"
    user     = "samus"
    password = "dev-password-change-in-prod"
    database = "samus"

A non-empty dsn always wins when both forms are present.

Configuring neither is not the same as omitting the section: the DSN then falls back to a passwordless postgres://localhost:5432/samus, which will not authenticate against the container from step 1.

3. Create the first user

localauth stores bcrypt hashed users in the users table:

go run . createUser admin changeme --config samus_dev.toml

4. Run the API

go run . --config samus_dev.toml

Or with live-reload, which also starts PostgreSQL for you:

make start-api-service

That runs gin as a reverse proxy: gin listens on :3000 and rebuilds on change, while the app itself binds bind_addr (:3001). Make your requests against http://localhost:3000 so you get the reloading proxy. Running go run . directly means no proxy, so use :3001 instead.

Without --config, the app reads ./samus.toml — the LDAP config, which has no [postgres] section. Note that gin forwards everything after run to the built binary as arguments, so the flag belongs there directly (gin run --config samus_dev.toml); passing a filename such as samus.go silently consumes the first argument slot and the app falls back to ./samus.toml.

API reference

All endpoints except /auth/login and /shares/exchange require a Bearer <jwt> token in the Authorization header (or ?token= / ?api-key= query params).

Auth

DAGs

Real-time collaboration

The WebSocket relay forwards two message types:

• diagram:updated — broadcast to all peers when a save occurs (echo-prevented via clientId)
• presence — relayed to other peers in the room (cursor position, node selection, display name)

Diagram history

Share links

Share tokens are signed with the same signing_key as regular auth tokens but carry iss: "d3d-share". The role claim is enforced server-side:

• view — POST /dag/:dag/update returns 403; WS is allowed (presence only)
• edit — full read/write access

Revoked JTIs are stored in the share_denylist table and checked on WS upgrade and token exchange.

Anonymous display names (e.g. "Teal Fox") are generated on share creation and returned by /shares/exchange so view-only users appear in the presence HUD with a readable identity.

Nodes / Edges / Menus

Auth providers

Set auth_provider in your config to one of (defaults to ldap if unset):

Running tests

go test ./...

The app/collab/ package has unit tests for the hub room registry and write pump.

Database tables

The schema lives in app/db/postgres/migrations/ and is applied automatically on startup by embedded goose migrations. To add a new migration, drop a NNNN_name.sql file in that directory.

Migrating from RethinkDB

cmd/migrate-rb2pg copies data from an existing RethinkDB database into Postgres. It is idempotent (rows already present are skipped), so it is safe to run more than once.

go run ./cmd/migrate-rb2pg \
  -rb-address localhost:28015 \
  -rb-database samus \
  -pg-dsn "postgres://postgres:postgres@localhost:5432/samus?sslmode=disable"

Behavior notes:

• Preserves UUIDs and timestamps exactly; RethinkDB zero-time values become NULL.
• diagram / snapshot_json (stored as JSON strings in RethinkDB) are validated and normalized into jsonb.
• Tables missing in RethinkDB are skipped; rows whose parent DAG is gone are reported as orphans and skipped.
• Per-table counts are logged (inserted / already present / orphaned / failed); a non-zero exit code indicates real failures.

For the full step-by-step production cutover (freeze, preflight, migrate, verify, rollback), see docs/cutover-runbook.md.

Deployment

Deploys run through Vercel's Git integration, not GitHub Actions: pushing to main produces a production deployment and opening a pull request produces a preview deployment. There is no deploy workflow in this repository to trigger by hand.

vercel.json builds Dockerfile.vercel with @vercel/container and supplies the non-secret settings (AUTH_PROVIDER=localauth, the LOG_* flags, TLS disabled, empty LDAP/OAuth blocks). The image is a two-stage build: golang:1.23 compiles the binary, and an Alpine runtime layer adds gomplate.

The container does not ship a config file. entrypoint.sh builds one at startup:

export BIND_ADDR="0.0.0.0:${PORT:-8081}"
export POSTGRES_DSN="${DATABASE_URL}"
gomplate --file samus.tmpl --out samus.toml
exec ./main --config ./samus.toml

So the database connection comes from DATABASE_URL — not POSTGRES_DSN, POSTGRES_URL, or any of the other names the Neon integration provides. Renaming or unsetting DATABASE_URL makes the app fall back to a passwordless postgres://localhost:5432/samus, which fails to connect.

Postgres is Neon, attached through the Vercel integration, which injects DATABASE_URL, POSTGRES_URL, PG*, and friends. SIGNING_KEY is set as a project environment variable.

Migrations need no deploy step. Postgres.Init runs the embedded goose migrations before the app serves anything, and BuildApp calls log.Fatal if they fail — a deployment that is serving traffic has necessarily applied every migration in app/db/postgres/migrations/.

Preview deployments share the production database. Every Postgres variable is scoped to both Production and Preview with the same value, so a preview deployment runs migrations against, and writes to, production data. A migration on an unmerged branch reaches production as soon as the preview boots. Giving preview its own Neon branch would remove this.

To deploy a build of arbitrary code outside Git (a rollback rehearsal, a debug build), the image runs anywhere: supply a samus.toml with a [postgres] dsn (or a discrete [postgresql] block), or set DATABASE_URL and let entrypoint.sh render one from samus.tmpl.

CI/CD

GitHub Actions drives the quality gates. Deploys are handled by Vercel (see Deployment), so no workflow here builds or ships the app:

Repository secrets: VERCEL_TOKEN for the registry prune job (see below). The secret scan uses the automatic GITHUB_TOKEN; no extra secret needed. Deploy credentials live in Vercel, not here.

Vercel container registry pruning

Vercel deployments push one image per commit to the project's registry repository, and nothing removes the old ones. The repository has a hard cap on image count; once it is reached, every build fails at the push step with denied: repository has reached the maximum allowed number of images — the image builds fine, only the upload is rejected.

The prune workflow keeps the 25 newest images and deletes the rest. The rule is count-based rather than age-based because the cap itself is a count: "older than N days" can still overflow during a burst of deployments. Deleting an image does not delete its deployment, but rolling back to a deployment whose image is gone requires a rebuild.

The same logic runs locally when a build is blocked and you do not want to wait for the schedule:

DRY_RUN=1 scripts/prune-vercel-images.sh   # report only
scripts/prune-vercel-images.sh             # prune to the newest 25
KEEP=10 scripts/prune-vercel-images.sh     # keep fewer

Repository secret required: VERCEL_TOKEN, created at https://vercel.com/account/tokens and scoped to the smetroids-projects team. Locally the script uses your vercel login session instead.

Branch protection on main should require the CI workflow to pass before merging.

Pre-commit hooks

A .pre-commit-config.yaml runs local quality gates before every commit:

• gitleaks — secrets scanning
• golangci-lint — linting (.golangci.yml)
• gofmt, go mod tidy drift check
• generic file hygiene (trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files)

Install once:

make precommit-install   # pre-commit install

Run manually at any time:

make precommit-run       # pre-commit run --all-files

The gitleaks allowlist (public test fixtures in app/auth/ldap/ldap_test.go, docs samples) lives in .gitleaks.toml.