daemon

daemon

The backend of the Cyber ICE Box platform: one Go service behind api.<domain> that serves the HTTP API of every frontend (main, ID, admin, exercises, event) and drives the laboratory infrastructure through enrolled agents.

What it does

• Accounts and sessions: registration, sign-in (password and Google OAuth), bot check (Cloudflare Turnstile or reCAPTCHA), roles and permissions (RBAC).
• Exercise catalog and events: exercises, challenges and flags, stands, teams, participants, scoring, submissions, event pages and forms.
• Mail and notifications: SMTP providers, templates, broadcasts, the inbox, signals and delivery journals.
• Analytics: platform and per-event analytics with PDF and CSV reports.
• Lab infrastructure: enrolls laboratory agents, deploys stands and test labs, issues lab access links and VPN configs, collects lab monitoring and traffic.
• Background jobs (River): stand launch, mail delivery, data retention, garbage collection of media and test labs, and more (internal/jobs).

Architecture

Domain-driven layers; the rules are enforced by convention and by lint tools (see CLAUDE.md for the full set).

Conventions worth knowing:

• One whole-entity repository per aggregate; sqlc rows and JSON (de)serialization stay inside it.
• Authorization is a single point: every route is gated with RequirePermission(perm); make vet fails on a route without a gate.
• Request identity travels as one rbac.Claims bundle in the context.
• Time is always a parameter (now time.Time), so tests control the clock.

Stack: Gin, pgx/v5 with sqlc, golang-migrate, River, gomock, testcontainers, zerolog.

Requirements

• Go 1.27 (see go.mod).
• PostgreSQL.
• Docker, for the integration tests (they use testcontainers and skip themselves when Docker is unavailable) and for make sqlcGenerate.
• The backend imports the agent contract and a few shared packages from the laboratory repository. go.mod pins its version (there is no replace). To build against a checkout at ../laboratory run make work, which writes ../go.work (gitignored; CI never uses it). See CONTRIBUTING.md for the pin.

cybericebox/
  daemon/
  laboratory/

The statistics-contract work on feat/proxy-traffic-contract is validated against the local laboratory revision 8f55694c66dbc18abe9022cd4928e62e1189f2e8 using this workspace workflow. The remote module pin still predates its new protobuf fields; standalone CI and release builds need that producer revision published and the pin updated first. No local-path replacement is committed.

In the existing event Usage API, Labs[].LabInitiatedAttempts counts permitted VPN flows started by the laboratory separately from participant Attempts. FirstAt and LastAt are nullable participant observations. Per-lab byte totals cover the whole event: inner IP packets for VPN, HTTP bodies or relayed WebSocket data payloads for proxy. They are separate from period-based WireGuard session totals. Explicit coverage preserves collector gaps; incomplete observations do not prove per-lab absence. Raw coverage and collector identities remain private.

Running locally

1. Start PostgreSQL (and, if you need them, an S3-compatible store and a mail catcher). The infrastructure repository has a local stand for this.
2. Create ./.env from the table below. It is git-ignored; never commit it.
3. Run:

make run-local   # sources ./.env, then go run ./cmd/daemon

The daemon reads configuration from the process environment only (there is no .env loader in the binary), so make run-local sources the file first. Migrations are applied on boot with golang-migrate. In ENV=development (set it in .env for a local run; the default is production) they are read from internal/delivery/repository/postgres/migrations; in any other environment from migrations next to the binary.

The service answers only on the API_HOST host. For local work route that host to the process (for example with a hosts entry or the local edge proxy), or send the Host header explicitly. GET /api/health is a public liveness probe.

Configuration

All settings are environment variables. Values below are placeholders; durations use Go syntax (30s, 24h). A bot check is mandatory in production (CAPTCHA_PROVIDER other than none).

The image takes the minimum: DOMAIN, SUPPORT_EMAIL, POSTGRES_PASSWORD and the secret keys (plus the external endpoints, the first admin and the bot check keys). Every other variable below is optional with the default shown: it can be overridden without a rebuild, and our deployment templates do not list it. The test lab lease has one setting, EXERCISE_TEST_DEPLOY_TTL (the former CALENDAR_TEST_LAB_LEASE is gone). The contact, privacy and security mailboxes belong to the frontends; the daemon needs none of them.

General and HTTP server

The daemon refuses to start without a valid DOMAIN. The labels api, id, admin, exercises and www are reserved in code, and EVENT_RESERVED_TAGS_EXTRA (comma-separated lower case DNS labels, default empty; the deployment passes labs,vpn,ctl, which belong to the laboratory) adds more; a bad label stops the start. Creating or renaming an event to a reserved tag is refused with error code 21142 (Event tag is reserved for a platform address). CORS allows exactly the frontend hosts, the landing host and https://<tag>.<DOMAIN>. The same rule is in the frontends (deploy/base-domain.sh, hosts.ts) and in infrastructure (cibconf.py); internal/config/testdata/base-domain-vectors.json holds the shared test vectors, and its copies in the other repositories must stay identical.

| HTTP_SERVER_HOST | 0.0.0.0 | Listen host. | | HTTP_SERVER_PORT | 8080, none when TLS is on | Plain-HTTP listener. Unset it is off when TLS is on (the standard container: HTTPS only) and 8080 otherwise (unprivileged). Set to an empty value to turn it off; with it empty and TLS off the daemon refuses to start. | | HTTP_SERVER_READ_TIMEOUT | 10s | Read timeout. | | HTTP_SERVER_WRITE_TIMEOUT | 10s | Write timeout. | | HTTP_SERVER_IDLE_TIMEOUT | 300s | Idle keep-alive timeout; keep it under the load balancer's 350 s idle timeout. | | HTTP_SERVER_MAX_HEADER_MB | 1 | Max header size in MiB. | | MAX_REQUEST_BODY_BYTES | 10485760 | Cap of every request body (10 MiB); an upload route states its own larger cap. | | TRUSTED_PROXIES | none | Comma-separated CIDRs or addresses of the proxies in front of the daemon (ingress, CDN). The client address is read from X-Forwarded-For only for requests from them; with none listed the connection address is used and the header is ignored. Behind a proxy set it, or every client shares the proxy address in the per-address limits. | | HTTP_SERVER_HTTPS_PORT | 8443 | TLS listener (HTTP/2); runs only when both HTTP_SERVER_TLS_CERT_FILE and HTTP_SERVER_TLS_KEY_FILE are set. | | HTTP_SERVER_TLS_CERT_FILE | /tls/tls.crt when it and the key exist | PEM server certificate chain. Both cert and key set: TLS on; exactly one set: the daemon refuses to start. | | HTTP_SERVER_TLS_KEY_FILE | /tls/tls.key when it and the certificate exist | PEM server key. The certificate, key and client CA are reread when a file changes (a renewal needs no restart); a broken renewal keeps the previous one in service. | | HTTP_SERVER_TLS_MIN_VERSION | 1.2 | Lowest TLS version: 1.2 or 1.3. | | HTTP_SERVER_TLS_CLIENT_CA_FILE | /aop/ca.crt when TLS is on and it exists | PEM bundle of the roots (and intermediates) that signed the client certificates (for example the Cloudflare Authenticated Origin Pulls CA). | | HTTP_SERVER_TLS_CLIENT_AUTH | require when the default CA file was found, else off | off, optional (a presented certificate is verified, none is fine; a presented one that does not verify is refused) or require (no valid certificate: the handshake is refused). optional and require need TLS on and HTTP_SERVER_TLS_CLIENT_CA_FILE, else the daemon refuses to start. | | HTTP_SERVER_HEALTH_PORT | 8081 in the image, none (off) otherwise | When set, an extra plain-HTTP listener on HEALTH_BIND that serves only GET /api/health (the kubelet cannot present a client certificate). Never route it. | | HEALTH_BIND | 0.0.0.0 | Address of the health listener and of the internal listener; the deploy sets the pod IP. | | HTTP_SERVER_INTERNAL_PORT | none (off) | When set, a second plain-HTTP listener on HEALTH_BIND with the same API handler, for in-cluster callers (the event-frontend server rendering, INTERNAL_API_ORIGIN). Restrict who reaches it with a NetworkPolicy. | | HTTP_SERVER_HTTP3_ENABLED | false | HTTP/3 (QUIC over UDP) next to HTTPS: the same certificate, handler and middleware chain, the same read, write, idle and header limits. Needs TLS. On, every HTTPS response carries Alt-Svc: h3=":<advertised port>"; ma=86400. UDP flood posture: 0-RTT off, handshake timeout 5 s, 100 request streams per connection, and a Retry (source address validation) for new connections once more than 100 per second arrive (burst 200), so spoofed floods cost one small packet each and no state. | | HTTP_SERVER_HTTP3_PORT | the HTTPS port | UDP port the daemon listens on (a Service/NLB UDP listener forwards to it). | | HTTP_SERVER_HTTP3_ADVERTISED_PORT | 443 | Port in Alt-Svc: the one clients reach outside the load balancer, which differs from the pod port (8443). |

Client address: the only use is the session IP (login session, visible to the user only). A request whose client certificate was verified (HTTP_SERVER_TLS_CLIENT_AUTH require, or optional with a presented and verified certificate) takes it from CF-Connecting-IP (a missing or invalid value falls back to the connection address); the header is never trusted on any other connection (plain, internal listener, no or unverified certificate), where TRUSTED_PROXIES / X-Forwarded-For apply as before.

Removed with no alias: HTTP_SERVER_TLS_ENABLED, HTTP_SERVER_TLS_PORT (now HTTP_SERVER_HTTPS_PORT), HTTP_SERVER_TLS_CLIENT_AUTH as a boolean (now off|optional|require), HTTP_SERVER_TLS_CA_FILE (now HTTP_SERVER_TLS_CLIENT_CA_FILE), and the built-in /certificates/* and /aop/ca.crt defaults; HTTP_SERVER_HEALTH_PORT is 8081 in the image (a local run has none). The TLS paths and client auth now default as the table says, so the deploy passes none of the listener settings; the settings stay overridable (an explicitly set value, even an empty one, always wins). A certificate or key path that is set but missing, or optional/require client auth without the CA file, stops the start with an error; the start-up log names the active mode (http, https, https+client-auth-<mode>).

Database

Connection budget: one replica opens up to POSTGRES_MAX_CONNS + POSTGRES_JOB_MAX_CONNS (40 by default), so N replicas need N × 40 + 20 (reserved for administration; the leader also holds one connection of its own outside the pools for the leadership lock, and a push to an agent holds one more while it runs) ≤ the server's max_connections. The daemon logs this arithmetic at startup and warns when a single replica already does not fit.

Authentication and secrets

Generate a key with openssl rand -hex 32. Keep every key stable: changing one makes data sealed with it unreadable.

Sessions

The cookie is the session: __Host-session (HttpOnly, Secure, SameSite=Strict, host-only on the API host) holds the session id, the user id, the sign-in time and the expiry, encrypted and authenticated with SESSION_ENCRYPTION_KEY. No role and no name. While the session is not revoked, no database read is needed to trust it. The earlier HMAC-signed cookies are not accepted: everyone signs in again after the upgrade.

Per request, in this order: (1) decrypt the cookie and check its expiry, (2) look the session id up in the in-memory revocation set, (3) the per-user rate limit, (4) one query for the caller's global role and blocked flag. A garbage, expired or revoked cookie is a 401 that never touches the database. Event roles stay in the handlers.

last_seen is written per session: the first request of a 30 s window writes it at once (asynchronously), later ones only update memory, and at the end of the window the latest time is written (GREATEST, never backward). The replica flushes what it holds on SIGTERM.

Every end of a session (sign-out, "end this session", sign-out everywhere, password change, block, eviction beyond SESSION_MAX_PER_USER) deletes the session row and writes a row in session_revocations in one statement: seq, session_id, user_id, revoked_at (database time) and expires_at (when the cookie would die by itself: the smaller of sign-in + SESSION_ABSOLUTE_TTL and last_seen + SESSION_IDLE_TTL + 1 min). Every replica loads the unexpired rows before it serves and polls the table every second (WHERE revoked_at > watermark - 10 s; the overlap covers late commits, the session id dedupes). Revocation therefore reaches every replica within about a second. A replica whose poll fails for longer than 30 s answers signed-in requests with 503 until the poll recovers. A worker deletes rows past expires_at.

Bot check (CAPTCHA_PROVIDER)

One provider for the whole platform: the sign-in, sign-up and password-reset forms use it. The token travels in the JSON body as RecaptchaToken whichever provider issued it. A token must have been solved on a platform frontend host. The frontends take the same choice from NEXT_PUBLIC_CAPTCHA_PROVIDER and NEXT_PUBLIC_CAPTCHA_SITE_KEY.

Only the chosen provider's keys are required; any other value, or missing keys, stops the start. RECAPTCHA_SCORE (default 0.5) is the minimum accepted reCAPTCHA score; below 0.3 (0 accepts every bot) is fatal in production.

Mail

Rotating a secrets key. Each *_SECRETS_KEY may be a keyring of id:hex entries (an id is 1 to 16 characters of a-z, 0-9, _, -), for example 2026b:<hex>,2025a:<hex>. The first key encrypts; every key opens. A ciphertext names the key that sealed it (v1:<id>:...), and values written before keyrings (no id) open with the keys of the ring in turn. To rotate: put the new key first and keep the old one after it; remove the old one when nothing is sealed with it any more (a value whose key is missing from the ring fails with the missing key id). Secrets are also bound to their row (a VPN config to its user and scope, an exercise secret to its variant, device and variable), so a ciphertext copied elsewhere does not open.

Sender and transport settings live in the database: SMTP providers are managed in the admin (Mail settings) and their passwords are sealed with PLATFORM_SECRETS_KEY. The environment transport is only a bootstrap fallback.

Fallback rule: providers saved in the database always win. The SMTP_* environment transport is used only while no provider is enabled.

Object storage

S3-compatible store (MinIO, AWS S3) for user avatars. An empty STORAGE_ENDPOINT disables storage; avatar upload and serving then become unavailable.

Laboratory agents

Lab infrastructure is served by one or more laboratory agents. Agents live in the database and are enrolled, never configured with certificate files.

• In the admin (Infrastructure, Agents): add an agent with its endpoint and the one-time enrollment token from the cluster administrator. The platform generates its own mutual-TLS key and the signing key pair of the lab access tokens; the agent signs the client certificate (the tenant identity is its CN). Both private keys are stored encrypted with PLATFORM_SECRETS_KEY. Certificates are renewed automatically after two thirds of their lifetime; the access key can be rotated.
• Bootstrap from the environment: with AGENT_ENDPOINT and AGENT_ENROLLMENT_TOKEN set and no live agent with that endpoint, the daemon enrolls it once at start-up and stores it like any other agent. The token is ignored afterwards (remove it). A failed enrollment is logged and the daemon still starts.
• Without any agent the daemon runs fine (catalog, event setup, exercise authoring); whatever needs to deploy a lab fails with ErrInfrastructureUnavailable.

What a laboratory offers is not configured here: each agent reports it (device state persistence and its limits, the image cache, the scheduler, the lab and VPN endpoints, its certificate) and the platform keeps the last report per agent (shown in the admin agent list). The exercise editor offers state persistence when some enabled agent offers it, and a deploy is refused when the agent that holds the team's group does not. A client certificate is renewed after two thirds of its own lifetime, and a rotated-out access key stays at the agent for three times the longest link its proxy accepts (at least 15 minutes). The proxy limits (longest link, longest session) come from the agent too: a web link never promises more than the proxy allows (a warning is logged once per agent when LAB_ACCESS_TOKEN_TTL or LAB_SESSION_TTL is above them), and image prewarm skips an agent that reports its image cache is off. The agents also describe their hardware (the most CPU and memory per device, devices per lab, the sizing of a lab group's own pods: the VPN by its users, the gateway by its internet labs) and the product rules are the platform's (RESOURCES_* below): an agent whose device maxima are below the frame or that allows fewer than 32 devices per lab is flagged "does not meet the platform requirements" in the admin agent list (MeetsRequirements, Unmet) and is not used for planning or placement. A device carries a preset (one of the sizes above; there is no custom size) and the agent is always sent explicit, equal requests and limits; the VPN and gateway sizes of a group are rounded up to the next preset size (its CPU by the same rule; when that CPU is below what the formula needs, the next size up) before they are sent and reserved, and the largest device an agent can place is mapped down to the largest allowed size. A draft always saves and its response lists the devices outside the frame; publishing needs every device inside the frame or covered by an approved elevation (a platform admin approves per exercise the requested block or a smaller allowed one, up to the ceiling; a later version keeps the approval while every device stays at or below the approved block). The event plan (GET /events/:id/manage/resource-plan) counts the devices of the tasks (the largest variant of each) plus each group's VPN, sized by the event's maximum team size, and its gateway, sized by the group's internet labs, with the agents' formula; the group is created with those sizes, and an elevated task is placed only on an agent whose maxima fit it.

The variables AGENT_TLS_*, AGENT_ACCESS_PRIVATE_KEY, AGENT_ACCESS_KEY_ID and LAB_ACCESS_PRIVATE_KEY were removed and have no replacement: keys are per agent and stored in the database.

Exercises and stands

Resource calendar

The backend owns a calendar of lab resources (internal/useCase/resourceCalendar, internal/model/resourceCalendar). The calendar owns the reservations; the agents hold them as cluster reservations, split per agent (see below).

• Time is cut into 15-minute slots. The capacity is the recorded capacity of the agents that are used (enabled, meeting the platform requirements, with a recorded capacity; no tenant quota means no limit on that resource). Feasibility is checked by packing, never by adding free room: a team (its lab group) stays whole on one agent, the fewest agents are used first, then agent priority, elevated tasks only go to agents whose device maxima fit them. Agents do not report per-node room yet, so an agent counts as one node (PerNodeRoomReported: false); they do not report maintenance windows either, so the calendar shows none.
• An event reservation is set by a platform admin only: size = the event's resource plan per team x teams + the buffer + the organizer's estimate for future dynamic tasks; window = from the stand deploy lead (plus CALENDAR_LEAD_MARGIN) to the event end + the tail gap (CALENDAR_TAIL_GAP, never shorter, the admin may set more). A reservation that does not fit by packing is refused (409) unless the admin allows the conflict; it is then kept, shown as not covered, and an alarm is raised. Conflicts are resolved by hand: a changed reservation keeps every team that still fits where it is, and nothing is moved automatically.
• An organizer sees allocated vs used (never an agent) and sends change requests (size, window, estimate, with a reason) that the admin approves or rejects; approving extends the existing reservation. A new task of a running event deploys only if its reservation holds it for all teams, otherwise it is refused with "not enough reserved resources, request an extension" (72508).
• Readiness alarms are real entities (not_placed, agent_lost, agent_shrunk, not_connected; the last escalates 24 h and 2 h before the deploy lead and at it). Raising one notifies the super admins through the inbox and writes the error journal (lab_readiness). A periodic job (resource_calendar_check, every minute) re-evaluates them and only adds the teams that had no agent when capacity appears.
• Test labs: a guaranteed minimum pool (an admin setting, always on, never reserved by events); above it any room no event has reserved; otherwise the author gets the nearest free window ("no free resources now, the nearest window is from HH:MM") and can book it (15 minutes to 8 hours, at most 14 days ahead, 3 at once). The booking reserves room in the calendar and covers the lab started inside it.
• Cluster reservations: every calendar reservation is split into parts, one per agent of its placement (ceil(size x units / teams), the calendar window), kept in cluster_reservations with stable ids; a test lab started by the pool or free room gets its own run part (the run length the author chose: 15/30/45 min, 1/2 h up to the lease maximum, GET /api/exercises/test-runs/lengths, LeaseMinutes on the deploy and on the event preview start), and each author one group part on the agent of their tu- group. Every change is sent to the agents' reservation API after its commit; a River job (cluster_reservations_reconcile, every 5 min) retries what failed and deletes orphans of this instance. While labs run in a part, a shrink never goes below what they use nor ends earlier than the agent's ending notice (cluster_shrink_held), unless the admin forced it; a part not sent for 15 min raises cluster_sync_failed. Every group and lab carries its part id, and a new group goes to the agent of its team's part. The agent's monitoring drives the rest: the ending notice of a test run is shown to the author (EndingAt), of an event part during the event tells the admins and organizers; at the end the test lab ends like at its lease and the stands release their labs, without a failure. An agent without the reservation API is marked incompatible (no new groups). A held retire (a part with groups dropped from the placement without force) stays alive and raises the same alarm (retire_held); a part never starts after its window, and the units of destroyed team groups are released.
• Rollout of the reservations (one release): first the laboratory agents with the reservation API, then the backend. Groups made before the release carry no part: a group whose reservation part is on another agent fails with a placement mismatch and is recreated on the right agent (stands at the next deploy, test groups with the author's next lab); the backend never pins a unit to an agent where it has no part.
• Admin API (super admin, infrastructure.read / infrastructure.write): /api/infrastructure/calendar/* (timeline, capacity, stats, settings, alarms, change requests, event reservation; every reservation lists its Parts). Organizer: /api/events/:id/manage/resources. Author: /api/exercises/test-labs/*, /api/exercises/test-runs/lengths. See the swagger.

Outgoing mail security

• An event SMTP (set by an organizer) can only name a public host: an IP literal or a name in a loopback, private, link-local (cloud metadata), carrier-grade NAT or other special range is refused when saved, and again at connect time after the name is resolved (a name that later resolves inside is never connected). Ports are limited to SMTP_ALLOWED_PORTS (default 25,465,587,2525).
• The stored SMTP password is used only for the stored connection: when the host, port, username or TLS mode changes, the password has to be entered again (error 22109); a test never sends the stored password to a changed host.
• STARTTLS is required: a server that does not offer it is refused before any credential is sent. SMTP_INSECURE=true lifts this for the env transport only (a development mail catcher). Every send has connection and conversation deadlines. Send errors are reported as one fixed line per kind (connection failed, TLS failed, a reply code); the raw error is logged, never returned.

Limits and timings

Error journal

The platform error journal records what broke: 5xx, panics, 403 (with the permission that refused), 429 (with the limiter), failed job attempts and discards, a stalled or growing job queue, mail that failed after its retries, and the laboratory (an agent offline, a failed lab deploy, a certificate close to its end, component errors reported by the agent). Errors are grouped by fingerprint (kind + route template or job kind + normalized message) with a count, first and last time and a few recent samples. 404s are only counted per day: per route template for a handler 404, one counter for every unmatched path (paths are never stored). 401 is not recorded. No IP address is ever read or stored; secrets, passwords, tokens and e-mail addresses are scrubbed from messages. Only super admins see it (platform.errors.read / platform.errors.write, routes under /api/admin/errors).

Notifications go to the Telegram chat ids and the e-mail list that super admins keep in the journal settings (the e-mail list defaults to every super admin). A chat id the bot may not write to (it answers 403) is marked failing in the settings, never dropped. One message per fingerprint per ERROR_JOURNAL_NOTIFY_COOLDOWN: a storm is one message with a count.

Every request and response carries an X-Request-ID header (a client's own id is kept when it is 8-64 letters, digits, ., _ or -); the journal stores it with the sample.

Rate limits and retention

No limit is keyed on a client address (a whole on-site event sits behind one router). Mail sent by organizers and admins (invitations, broadcasts, notifications) is never limited; only mail an account triggers itself is.

The RETENTION_* periods (all but RETENTION_BATCH_SIZE) are defaults: periods saved in the platform settings of the platform panel override them on the next purge pass, without a restart.

The retention defaults are the published periods of the Privacy Policy; change them only together with the policy text.

Make targets

Test data

make seed fills the development database with data for live end-to-end checks. It drives the application's own use cases and repositories (no raw SQL), reads the same .env as make run-local, and needs the schema to be current (start the daemon once after an upgrade). It refuses to run when ENV=production, and when the database host is not local unless --allow-remote is given. It sends no mail: no signals are published.

make seed                                    # small profile, event starts in 30 minutes
make seed ARGS="--profile load --start-in 10m --reveal as_ready"
make seed ARGS="--delete --dry-run"          # list what cleanup would remove
make seed ARGS="--delete"                    # remove everything the seeder created

What it creates:

• Accounts: two organizers (platform role admin; seed-org1 owns the event, seed-org2 is its manager) and the team members seed-tNN-mK@seed.cybericebox.test (member 1 is the captain). All are active and verified, with one shared password. The password is written only to the gitignored .seed-credentials (mode 0600, with the account list); it is never printed or logged, and a re-run keeps it.
• Ten catalog exercises, tagged seed, named [seed] …, published: six with a lab (one to three devices, a switch, VPN with a DHCP range, a state-persistent device, one with two per-team variants) and four static; flags are fixed (ICE{seed_<task>}), tags web, crypto, forensics, net, osint, misc, hints of every level. Lab devices use nginx:alpine, httpd:alpine, redis:alpine and traefik/whoami: every device must keep running by itself, so a bare alpine or busybox does not do.
• One event [seed] Live Check: team participation, open registration, public boards, published now and starting after --start-in; all exercises attached and shown on the board with points, hint costs, four challenge groups and prerequisites (two chains); dynamic infrastructure allowed; the teams are built, so they form at the start. The infrastructure flag cannot change after creation: delete and re-seed to flip it.

It is idempotent: a re-run finds everything by its stable name, skips what is unchanged, publishes a new version only of an exercise whose definition changed, adds missing members, and moves the start to --start-in from now while the event has not started. Once the event has started, the schedule and the reveal mode stay.

Cleanup removes only what carries the marker: events whose tag starts with seed and whose name with [seed] , catalog exercises tagged seed and named [seed] …, and accounts on seed.cybericebox.test. Events go first (an exercise cannot go while an event uses it), accounts last, deleted the way the platform deletes an account (the address is freed, sessions are revoked). An exercise that an event outside the seed uses is kept and reported.

Database and migrations

• SQL queries live in internal/delivery/repository/postgres/queries/*.sql; sqlc generates the Go code next to them. Edit the SQL, run make sqlcGenerate, never edit generated files.
• Migrations are numbered pairs (NNNN_name.up.sql and .down.sql) in internal/delivery/repository/postgres/migrations and are applied automatically on boot by golang-migrate. The Docker image carries them in /app/migrations.
• The SQL itself is covered by testcontainers integration tests (*_integration_test.go), because mocks cannot catch schema drift.

Error catalog

Every API error carries a numeric code (FullCode = informCode*10000 + objectCode*100 + detailCode, see pkg/err). The backend is the single source of truth: error-catalog/errors.en.json maps each code to its English message and errors.uk.json holds the Ukrainian text. Frontends localize by code and must not copy codes by hand.

• One error variable is one call site with a unique detail code inside its object code; make vet fails on duplicates.
• After adding or renaming an error run make error-catalog, and add the Ukrainian message.
• Changing a detail code changes the full code and breaks clients that key on it; call it out in the commit message.

API documentation

The OpenAPI spec is generated by swag from the handler annotations into internal/delivery/controller/http/handler/apidocs. The Swagger UI is served at /docs on the API host in every environment except production. Regenerate the spec with make swagger after changing handler DTOs or annotations.

Docker image and CI

The Dockerfile builds a static binary (CGO off, cross-compiled on the build platform) and copies it with the migrations into a slim Alpine image. The laboratory module comes through go mod download at the version go.mod pins, so a plain build works:

docker buildx build --build-arg LABORATORY_VERSION=$(.github/scripts/lab-pin.sh version) -t cybericebox/daemon:local .

The image carries the label org.cybericebox.laboratory.commit (the pinned laboratory version); infrastructure compares it with LABORATORY_IMAGE_TAG when it renders a deploy.

GitHub Actions publish multi-arch (linux/amd64, linux/arm64) images to Docker Hub as cybericebox/daemon. The cycle is described in CONTRIBUTING.md:

Registry credentials (DOCKERHUB_USERNAME, DOCKERHUB_TOKEN, with delete rights) are organisation secrets.

Several replicas

The daemon runs behind an HPA with any number of replicas that share only Postgres.

• Leader. One replica, the holder of a session advisory lock (pg_try_advisory_lock on a connection of its own, internal/leader), runs the loops that poll the laboratory agents or write shared state on a timer: the monitoring stream of every agent (with its maintenance polling, capacity, link and error reports) and the error journal's periodic checks (queue, certificates, SMTP providers). The others retry every 5 s. The leader checks its lock connection every 5 s; when it is lost the loops stop and the replica goes back to waiting. On SIGTERM the loops stop first and the lock is released at once, so another replica takes over within one retry. Every replica still holds the agent connections (requests and jobs use them) and runs the job workers (River has its own leader election for periodic jobs).
• Pushes to one agent. Cluster reservation pushes (SyncAgent) can start on any replica (a request or the reconcile job); each push also holds a per-agent advisory lock, so two replicas never list and write one agent at the same time.
• Error journal. Capture, grouping and the notification decision are in the database: one message per group per cooldown (claimed in the capture transaction) and one per cooldown for the whole noisy kind (403), whichever replica captured. The live stream of the admin page crosses replicas through a change log table (error_journal_changes): every recorded occurrence or status change adds a row last in its transaction, under one advisory lock, so seq order is commit order. While a replica has at least one open stream, one poller of that replica reads the rows after its cursor every 3 s (one indexed query on a pool connection, no held connection) and shows them to the local streams; with no stream nothing is polled. A replica skips the rows it wrote itself (it showed them at once). Rows older than a day are purged.
• Still per replica (in memory): the HTTP rate limiters and in-flight limiters (the limits above are per replica, so the effective limit grows with the replica count), the spike counter of the journal (a spike needs ERROR_JOURNAL_SPIKE_THRESHOLD occurrences on one replica), the 403 fold and the 404 counters (each replica writes its own totals; the writes add up), the session last_seen batching, and the open live streams (each browser is attached to one replica).

Deployment

Deployment and cluster configuration: see the infrastructure repository.

License

Licensed under the Apache License, Version 2.0. See LICENSE.

Copyright 2026 CyberICEBox