deploy/

directory
v1.6.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 29, 2026 License: MIT

README

deploy/

Operational scripts and host configuration for running Nekomari. None of this is needed to use the project — it is the tooling used to deploy, verify and operate the reference instance, kept in-tree so the steps are reproducible rather than folklore.

No credentials belong in this directory. Tokens and passwords come from the panel or the environment; see docs/SECRETS.md for why, and for the incident that rule came from.

Deploy

Script What it does
agent-footprint.sh Measures what the agent costs the host it runs on: resident memory, CPU over a window, the agent's own socket byte counters, disk writes, threads, file descriptors and its effective capabilities. Takes [unit] [system|user] [window-seconds] and works for a user unit with the agent running as another account. Redacts the agent token out of ExecStart. Results and the resulting minimum-spec recommendation are in docs/AGENT-FOOTPRINT.md.
deploy-verify.sh Proves a published release actually deploys: downloads the assets, checks SHA256SUMS.txt, starts the server on its own port and data directory, completes the first-run install via the API, connects an agent, confirms the node reports. Cleans up after itself. Runs in CI as the verify job.
verify-image.sh Pulls a published container image and proves the binary inside it actually starts. Listing tags needs read:packages, but that is not the question — the question is whether the binary executes on the image's own base, which is what broke in v0.1.1 (glibc binary in a musl base).
verify-settings-persist.sh Runs a real upgrade (old image → new image) against a copy of production data and diffs the settings. Evidence that an update does not reset anything, rather than an assurance.
test-volume-footgun.sh Reproduces the anonymous-volume trap: install without -v, recreate the container, watch the settings disappear.
test-persist-warning.sh Checks the startup warning fires on an anonymous volume and stays quiet on a named volume and a bind mount.
docker-compose.yml The reference container setup: host networking off, bound to 127.0.0.1, bind-mounted ./data.
../Dockerfile.agent The agent image (ghcr.io/aone2233/nekomari-agent), built by docker.yml from the release's komari-agent-linux-* assets. Its /data volume is the agent's working directory — the panel's Docker install command mounts it so a container replacement does not reset the traffic ledger or the node identity.
nginx-nekomari.conf Host nginx vhost. WebSocket-safe (agents hold long-lived connections) and forwards CF-Connecting-IP so the panel records the real visitor rather than Cloudflare's edge.
install-node-agent.sh Installs the agent on a node: refuses to run if one is already active, verifies the download against SHA256SUMS.txt, retires older agent units, writes a systemd unit. Works as root or through passwordless sudo. It accepts -t <token> (what the panel hands out) but moves the token into <install-dir>/.agent-credentials (0600) and starts the agent with --token-file, so the node's identity is not in ExecStart, ps or systemctl show. Use --install-token-file PATH to place the credential elsewhere; a re-run with no token reuses the file an earlier install wrote. --dry-run writes the credential and prints the command without downloading or touching the service.
install-node-agent.test.sh Proves the above with the real script: an offline harness confining every write to a temp directory, asserting that each spelling of the token argument (-t v, --token=v) ends up in a mode-0600 file and never in the unit's ExecStart, that a re-run reuses the stored credential, and that a missing identity still fails instead of starting a node with none. Runs in CI as the deploy-scripts job.
install-staged-agent.sh Installs an already copied binary over the running agent: finds the unit rather than the process name (a pgrep -f komari-agent also matches the shell running the script, and a leftover docker run of the agent image looks like a host process), verifies the hash, keeps the old binary, preserves owner and mode, restores cap_net_raw if the node had it, restarts and confirms. This is how the v0.1.16 fleet upgrade was done — copy the release asset to each node first, which also covers nodes that cannot reach the release assets.
upgrade-agent.sh Downloads the release asset on the node and installs it, comparing SHA-256 against SHA256SUMS.txt (the agent has no --version, so a version check cannot decide whether an upgrade is needed).
enable-webssh-all.sh Re-applies units without --disable-web-ssh. That flag also disables remote command execution, so it is opt-in per node. Tokens come from NEKOMARI_NODE_TOKENS.
set-month-rotate.sh Sets --month-rotate to a node's billing day and adopts the previous agent's net_static.json. Without it the panel divides a since-boot counter by the plan limit and reports inflated traffic.

Check

Script What it does
healthcheck.py Post-deployment check of the running instance: host services, container, agent unit, databases, API reachability, which nodes are actually reporting, retained history, the ip-info endpoints, private-IP rejection, active theme.
panel-probe.sh The always-on version of the check above: every 10 minutes it records the origin TTFB (the only number that implicates the panel, so it has the tight threshold) plus the public TTFB with the cf-ray POP and cache status, one line per run. A sample only counts as healthy when curl succeeded, the status is 200 and the body is a valid success envelope — a fast 500 or a refused connection exits nonzero and names the reason instead of looking fast. Installed with hosts/panel-probe.{service,timer}. docs/PERFORMANCE.md explains how to read it.
panel-probe.test.sh Proves panel-probe.sh can actually fail: local listeners serve a 200 envelope, a 200 with an HTML or {"status":"error"} body, a 500 and a deliberately slow response, and the assertions pin the exit code and the named reason for each, plus a refused connection and a zero time_starttransfer. Offline and deterministic; PROBE_LOG points at a temp file, so it never touches /var/log.
node_status.py Which nodes are live. Distinguishes a restored-but-silent node from a reconnected one.
ping-task-stats.py Per-node loss and latency for one ping task. Picks the newest series tag shape automatically — the pre-fork agents tagged {"task_id":"N"} and this fork's tag {"protocol":"tcp","task_id":"N"}, so matching only the old shape returns history that stopped when the old instance was retired and makes every node look dead.
ping-tasks-audit-all.py Fleet-wide: which tasks pair a node with a target its address family cannot reach, and which carry references to deleted nodes.
fix-ping-family.py Removes those mismatches and stale references. Reads address families from the database, not /api/nodes, which does not expose them. --apply to write; dry run by default.
ping-store-locate.py Where ping results are stored and at what resolution, for when a chart's granularity is mistaken for packet loss.
ping-history-depth.py How much ping history is actually retained.
check-node-freshness.py Which tasks are producing fresh samples and whether a given node is in any of them. Answers "is this node actually contributing anything" after the address-family filter means an IPv6-only node legitimately reports on some targets and not others.
test-family-skip-live.py End-to-end: re-adds an IPv6-only node to an IPv4 task and confirms the panel skips it without configuration help.
unhide-node.py Unhides a node once it is genuinely contributing data. Dry run by default, and refuses while nothing is fresh — a hidden node with no data is usually hidden for a reason.
webhook_receiver.py A throwaway HTTP endpoint that records every request it receives, used to prove the notifier really dispatched an alert rather than trusting the UI's "sent" state.
traffic_explain.py Prints, per node, the counters the panel divides by the limit and the resulting percentage — so a traffic figure can be checked against the provider's dashboard instead of guessed at.
theme-contract-check.mjs Runs the installed theme's own zod schemas against the live ip-info responses and exits non-zero on any rejection. It pulls the schema block out of whatever theme build is on disk, so the contract is re-derived rather than transcribed. Run it before shipping any change to web/api/ipinfo.
inventory-monitoring.sh Lists every monitoring agent on a host. Kept because two systems coexisted here and are easy to confuse.
inspect-node-traffic.sh, inspect-netstatic.sh Per-interface kernel counters and the agent's netstatic coverage.
Why there is no hand-written contract validator

There was one: validate_contract.py, a Python mirror of the theme's zod schemas. It was deleted. It required classification to have type and label and nothing else, because that is what the server sent — so it validated the implementation against itself and passed while the real theme rejected the same payload for a missing classification.source. The IP 信息 tab stayed invisible and this script said everything was fine.

A transcription of a contract you cannot see is not a check. theme-contract-check.mjs extracts the schema from the installed bundle and runs it with the theme's own bundled zod, so it cannot agree with a bug in the server.

Browser checks

Run with Playwright against a live panel. Each drives the real UI rather than the API, because that is the only way to see what a user sees.

Setup

Needs Node 20+ and a Chromium download (~115 MB). From this directory:

npm install          # installs playwright (declared in deploy/package.json)
npm run setup        # playwright install chromium
Running

All four take the same arguments — panel URL, username, password, and optionally an output directory for screenshots:

node ip_panel_check.js    https://your-panel admin 'password' /tmp
node ip_panel_capture.js  https://your-panel admin 'password'
node favicon_ui_check.js  https://your-panel admin 'password' /tmp
node admin_version_check.js https://your-panel admin 'password'

The password goes on the command line, so it is visible in ps while the script runs and lands in your shell history. That is acceptable for a throwaway test account and not for a real one; see docs/SECRETS.md.

A panel with private_site enabled answers anonymous requests with 401, so the scripts log in through /api/login first rather than driving a login form.

Script What it does
ip_panel_check.js Logs in, opens a node detail, reports which tabs render and every ip-info response the browser saw.
ip_panel_capture.js Installs a fetch interceptor before app code runs, so the exact bodies the theme parses are recorded. This is what separates "the server sent something the theme rejects" from "the theme gates the UI elsewhere".
favicon_ui_check.js Uploads a favicon through the admin UI and reports whether the icon actually changed.
admin_version_check.js Confirms the admin version banner queries this repository, not upstream's.

Each exits non-zero if its own expectations fail, so they can be wired into a script; none of them are part of CI, because all four need a running panel and real credentials.

Themes

Script What it does
upload_theme.py Uploads a theme archive through the server's chunked upload API, so its own extract-and-validate path runs rather than files being written into the data volume by hand.
rebrand_theme.py Rebrands a third-party theme's user-visible strings for Nekomari, while deliberately leaving the theme identifier alone — that string is the theme's contract with the server.

Cloudflare

Script What it does
cloudflare_dns.py Lists, shows or points a hostname at the origin. Reuses the token cloudflared tunnel login wrote, so no separate secret is needed.
retire-monitor-dns.py Removes the DNS record fronting the retired CF-Server-Monitor worker. Documents why it cannot finish the job: worker-managed records are read-only through the DNS API.
update-cloudflare-ips.sh Regenerates the Cloudflare range list that the vhost's geo block includes, from Cloudflare's published lists. --check compares without writing. --ufw additionally brings UFW's 80/443 allowlist in line with the same list — it only adds: missing ranges are added, and ranges Cloudflare no longer publishes are printed with the command to remove them rather than deleted, because silently dropping firewall rules is worse than leaving one extra.

The vhost trusts CF-Connecting-IP only when the connection really came from Cloudflare, and UFW admits 80/443 only from the same ranges. The two are complementary: the geo block stops a forged header, the firewall stops the direct connection that would carry it. Keep both halves in step — v4 and v6 are separate rule sets, and a half-configured pair fails quietly. When only the v4 rules exist, nothing breaks while the origin record is an A record, but the day an AAAA is added those origin fetches are dropped before nginx and show up as intermittent 522s with nothing in the origin's logs. --ufw exists because that gap was found in production.

CI maintenance

Actions pinned to a Node 20 runtime still run, but GitHub forces them onto Node 24 and warns on every job. These four make the upgrade a re-run rather than a research task, and they check the things that only fail at run time.

Script What it does
check-action-versions.py Which pinned actions are behind their latest release. Reads the list out of the workflows, so it cannot drift from what is used.
check-action-breaking.py Surfaces the breaking-change lines between the pinned version and the latest, before jumping majors.
check-action-inputs.py Confirms every with: input a workflow passes still exists on the pinned action. A dropped input only fails when the workflow runs — for release.yml that means at release time, which is the worst moment to find out.
bump-actions.py Rewrites the pins to each action's latest major. Targets come from the actions themselves, because they differ per publisher — docker/build-push-action was already past the version actions/checkout needed. Dry run by default.

Watch out for one thing when bumping setup-go: go-version-file must point at the higher of the two modules' requirements. The root go.mod says 1.25.0 and agent/go.mod says 1.26.0, and both are built from the same job. Pointing at the root installed too old a toolchain and the agent tests failed with go.mod requires go >= 1.26.0. Older versions of the action tolerated the mismatch; v7 installs exactly what the file asks for, which is correct behaviour that simply exposed the wrong file being named.

Retiring the previous stack

See docs/RETIRE-CF-PROBE.md for the full record. These are kept because the rollback path matters more than the removal path.

Script What it does
retire-cf-probe.sh Per host: backs up, disables, removes, and sweeps for residual traces. --dry-run supported.
retire-cf-probe-all.sh Drives the above across the fleet.
verify-cf-probe-retired.sh Confirms cf-probe is gone and that the Nekomari agent is still active.

Token rotation

Script What it does
rotate-all-tokens.py Issues a new token for every node through admin:editClient and writes a push map to a 0600 file. Tokens never reach stdout.

Pushing the new tokens to each host depends on where the SSH keys live, so it is done per-reachability rather than by one script: some hosts are reachable from the panel host, others only from a workstation, one only over IPv6, and two need a sudo password. That variation is why the ad-hoc push scripts were removed once used — keeping them implied a single supported path that does not exist.

Script authentication, and what 2FA changes

Every script here that talks to the panel logs in through nekomari_auth.py, which handles TOTP. Enabling 2FA on the account breaks any script that logs in with just a username and password — they start returning 401 2FA code is required. That is not a bug in the scripts; it is 2FA working.

export NEKOMARI_PASSWORD='<panel password>'
export NEKOMARI_2FA_SECRET='<base32 secret from enrolment>'
sudo -n env NEKOMARI_PASSWORD="$NEKOMARI_PASSWORD" \
            NEKOMARI_2FA_SECRET="$NEKOMARI_2FA_SECRET" \
            python3 deploy/<script>.py

nekomari_auth.py implements RFC 6238 directly rather than depending on a TOTP library, so the scripts stay runnable on a host with nothing installed. If the variable is missing while 2FA is on, login fails with a message naming the variable instead of a bare 401.

enable-2fa.py enrols the account and prints the secret and otpauth:// URI; that secret is the only way back in short of the disable-2fa CLI.

hosts/

Per-host configuration that does not fit the generic installer.

File Why it is separate
oc424.service The reference panel host. Uses the restored node's own token rather than --auto-discovery, so the panel keeps that node's group, tags and history.
macwan.service Runs as a user unit: that host has no passwordless sudo. Also documents that PrivateTmp and NoNewPrivileges must stay unset or a file capability stops working.
tender-guard.service The NOSLA Tokyo node. Runs as a non-root komari user with AmbientCapabilities=CAP_NET_RAW (a file capability would be defeated by NoNewPrivileges/PrivateTmp), carries its own node token rather than --auto-discovery, and sets --month-rotate to its own billing day.
nekomari-agent.openrc The OpenRC equivalent for hosts without systemd — first used by JPKD2 (Alpine 3.19, LXC). Takes the endpoint, token and flags from /etc/conf.d/nekomari-agent (mode 0600) and supervises with supervise-daemon. The generic installer is systemd-only, so this is what to copy on Alpine, and it pairs with --prefer-ip-version 4 on a host with no IPv6.
pzyc.sh Cannot fetch release assets (the CDN resets TLS), so it installs a separately fetched, checksum-verified binary. Takes its token from NEKOMARI_AGENT_TOKEN.
macwan-webhook-sink.service An artificial notification channel used to prove alerts are really dispatched.

experiments/

One-off probes kept because they captured knowledge that was otherwise expensive to obtain. Not part of any operational path.

File What it established
capability-probe.sh That NoNewPrivileges=yes and PrivateTmp=yes each defeat a setcap cap_net_raw file capability — with /proc/<pid>/status still reporting CAP_NET_RAW in CapEff. This is why MAC-WAN's ICMP tasks failed while an identical command from a login shell worked.
rawicmp-probe.go Minimal raw-ICMP socket probe, used to separate "the capability does not work here" from "the agent does something else".

Directories

Path Synopsis
Minimal raw-ICMP socket probe, built and run with the same file capability as the agent.
Minimal raw-ICMP socket probe, built and run with the same file capability as the agent.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL