towerdef/docs/DEPLOY.md
Claude faf0cfa603 Add co-op server, deploy docs, e2e suite and wire in the music tracks
- server/: Node (ws) static server + authoritative co-op rooms; dual-stack
  listen, proxy-aware client IP and Origin check, keep-alive pings, per-socket
  rate limits, graceful shutdown; 59 node:test tests
- docs/DEPLOY.md + deploy/netwall.service: Debian 13 LXC, systemd, nginx
  Proxy Manager walk-through, path prefix, troubleshooting
- e2e/: Playwright suite (smoke, touch, end flow, two-browser co-op incl.
  reconnect, 14-size layout matrix); npm run e2e
- Music: tracks.json now points at the uploaded theme/battle/boss/win/lose
  files; stingers fade out when a new game starts
- PAUSED indicator, refreshed icon

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tb8pW2KAN4oFkh1aK1BfD8
2026-10-09 19:57:27 +00:00

287 lines
18 KiB
Markdown

# Deploying Netwall
Netwall is one Node process: it serves the static game from `public/` and a WebSocket endpoint at `/ws` that runs
the co-op rooms. There is no database and no build step; games live in memory (a restart ends every running game).
```
phone / laptop ──https──▶ nginx Proxy Manager (TLS, :443) ──http + WebSocket──▶ Debian 13 LXC: node :8080
◀──wss── (HOST=::, IPv4 + IPv6)
```
What was actually run while writing this: the server and its test suite (Node 20.20.2 and 24.21), the real
`node server/index.js` over IPv4 and IPv6 on loopback (`curl`, a two-player WebSocket session, `SIGTERM` exit 0), the
IPv6-disabled case in a throwaway network namespace, and `systemd-analyze verify` on the unit (syntax only). The
proxy behaviour was only *simulated* by sending the headers a proxy would send (`Host`, `X-Forwarded-For`,
`X-Forwarded-Host`, `Origin`) straight to the server. **Never run here:** nginx Proxy Manager or any nginx, a started
systemd unit, a Proxmox LXC, a real browser or phone. Everything about NPM below is what its documented UI and
nginx's behaviour lead one to expect; labels differ between NPM versions, so check each step against your version.
## 1. Requirements
* Debian 13 (trixie) LXC, with its own IPv4 (and optionally IPv6) address on a network the proxy can reach.
* Node 18 or newer. Debian 13 ships Node 20.19 as `nodejs`. The server is plain CommonJS with one runtime
dependency (`ws`).
* A DNS name for the game (for example `game.example.com`) pointing at the proxy.
## 2. Install in the LXC
Run as root inside the container.
```sh
apt update
apt install -y nodejs npm # Debian 13: Node 20.19
node --version
# a dedicated, login-less user that only runs the game
useradd --system --home-dir /opt/netwall --shell /usr/sbin/nologin netwall
# copy the project to /opt/netwall (from your workstation, for example):
# rsync -a --exclude node_modules --exclude test --exclude .git ./ root@<lxc>:/opt/netwall/
# it needs server/, public/, package.json, package-lock.json (docs/ and deploy/ are optional)
cd /opt/netwall
npm install --omit=dev # only "ws"; `npm ci --omit=dev` also works and follows the lockfile exactly
chown -R root:root /opt/netwall # the service user only needs read access
# quick manual test as the service user (Ctrl-C to stop)
runuser -u netwall -- env PORT=8080 node server/index.js
```
When run directly the server prints `http://localhost:PORT/` and one URL per non-internal interface address
(IPv4 as `http://a.b.c.d:PORT/`, IPv6 as `http://[addr]:PORT/`, link-local `fe80::` skipped).
## 3. systemd service
```sh
install -m 0644 /opt/netwall/deploy/netwall.service /etc/systemd/system/netwall.service
systemctl daemon-reload
systemctl enable --now netwall
systemctl status netwall
journalctl -u netwall -f # live log
```
`deploy/netwall.service` runs as user `netwall`, `Restart=on-failure`, and sets the environment below. Edit the
`Environment=` lines (or use `systemctl edit netwall` and put them in an override) and `systemctl restart netwall`.
Hardening in the unit: `NoNewPrivileges`, an empty capability set (port 8080 needs none), `RestrictAddressFamilies`,
`RestrictRealtime`, `RestrictSUIDSGID`, `LockPersonality`, `SystemCallArchitectures=native`, `UMask=0077`. They need
no mount namespaces, so they should work in an LXC without extra settings; the unit has not been started in one yet,
so if `systemctl status netwall` shows a sandboxing error, comment the offending line out. A second block (`ProtectSystem=strict`, `PrivateTmp`, ...) is
left commented out: in an unprivileged Proxmox LXC it fails with `status=226/NAMESPACE` unless the container has
**Options > Features > Nesting** enabled. Turn nesting on, uncomment the block and restart if you want it.
Do not add `MemoryDenyWriteExecute=yes`; V8's JIT needs writable+executable memory.
## 4. Configuration
Environment variables, read when the server is started directly (`node server/index.js`):
| Variable | Default | Meaning |
|---|---|---|
| `PORT` | `8080` | TCP port. `0` picks a free one (the banner prints the real port). |
| `HOST` | `::` | Address to bind. `::` is **one dual-stack socket serving IPv4 and IPv6** (`ipv6Only:false`). `0.0.0.0` = IPv4 only. `127.0.0.1` = this container only. Any explicit address is used as given. |
| `TRUST_PROXY` | `auto` | `auto`: believe `X-Forwarded-For` / `X-Real-IP` / `X-Forwarded-Host` only when the TCP peer is loopback, RFC1918 (`10/8`, `172.16/12`, `192.168/16`), `fc00::/7` or link-local. `1` (also `true`, `yes`, `on`): always believe them. `0` (also `false`, `no`, `off`): never. |
| `ALLOWED_ORIGINS` | empty | Comma separated list of extra origins allowed to open the WebSocket, for example `https://game.example.com,https://play.example.org` (list every public name if the game has more than one). See "Origin check". |
| `MAX_CONN_PER_IP` | `40` | Open WebSockets per client IP; more get HTTP `429` on the upgrade. `0` disables the cap. |
| `GRACE_MS` | `60000` | How long a disconnected player keeps their slot (so a phone can come back after a screen lock or a network switch). |
Fixed in code (options of `createServer()`, not environment variables): at most 50 rooms, 20 snapshots/s per room,
a WebSocket ping every 25 s, client frames at most 2048 bytes, 60 messages/s per socket (burst 120), a socket that does
not `join` within 10 s is closed. When all 50 rooms exist, a room in which nobody is online (it is only being held for
its `GRACE_MS`) is deleted to make room for a new one, longest-abandoned first, so on a full server the grace period is
not guaranteed; rooms with somebody online are never evicted (the newcomer gets `too_many_rooms`).
What the server does, relevant to a proxy setup:
* **Real client IP** is used only for the log and the per-IP cap. With `TRUST_PROXY=auto` and a private/loopback peer
it comes from `X-Forwarded-For` (else `X-Real-IP`); otherwise it is the TCP peer. From `X-Forwarded-For` the server
takes the **rightmost entry that is not a private address**: proxies such as NPM *append* the address they saw to
whatever header the client sent, so the entries further left are the client's own claim and are never believed (a
chain that is private all the way gives its rightmost entry; a client with a private LAN address cannot be told
apart from your own proxy hops, so it can still choose its own address). IPv4-mapped IPv6 peers (`::ffff:a.b.c.d`) are shown
as plain IPv4. `X-Forwarded-Proto` is never used.
* **Player ids.** A player's id (`pid`, kept in the browser's session storage) is what lets a browser take its slot
back after a disconnect, so the server only ever sends it to its owner; the other player is shown under an opaque,
stable alias.
* **Keep-alive.** Every 25 s the server sends a WebSocket protocol ping to each socket and terminates a socket that did
not answer the previous one (browsers answer automatically). The ping traffic also keeps nginx's `proxy_read_timeout`
(default 60 s) from firing on an idle but healthy game. A terminated player stays in the room for `GRACE_MS`.
* **Origin check** (WebSocket upgrade only, path `/ws`). A request without `Origin` is allowed (non-browser clients).
Otherwise the origin host must equal the `Host` header, **or** the first entry of `X-Forwarded-Host` (only from a
trusted proxy), **or** be listed in `ALLOWED_ORIGINS`. Comparison is on lower-case `host[:port]` with the scheme
ignored and ports 80/443 treated as "no port" (so listing `https://game.example.com` also admits
`http://game.example.com`). Everything else gets `403`; the reason is logged.
* **Path prefix.** The server only knows `/ws` and `/`; the browser builds the socket URL relative to the page, so a
proxy that strips a prefix works (section 7).
* `GET /healthz` (and `HEAD`) returns `200 ok`. Static files are only served from `public/`, with
`Cache-Control: no-cache` (revalidated by ETag), range requests (needed for audio on iOS Safari) and a strict
Content-Security-Policy. The server does not compress responses.
* Logs go to stdout (journald): room created/deleted, players joining/leaving, refused WebSockets. Player IP
addresses appear in the log. There is no per-request access log.
* **Shutdown.** `SIGTERM`/`SIGINT`: all sockets are closed with code 1001, the process exits 0 (within about a
second, forced after 5 s). Running games are lost; clients that reconnect are told the room no longer exists.
## 5. nginx Proxy Manager (NPM)
In NPM, **Hosts > Proxy Hosts > Add Proxy Host**.
**Details tab**
| Field | Value |
|---|---|
| Domain Names | `game.example.com` |
| Scheme | `http` |
| Forward Hostname / IP | the LXC's IP address (for example `192.168.1.50`) or a name NPM can resolve |
| Forward Port | `8080` |
| Cache Assets | **off** (the game files must revalidate; the server already sends `no-cache`) |
| Block Common Exploits | optional, can stay on |
| **Websockets Support** | **on** (required: it adds the `Upgrade` / `Connection` headers and HTTP/1.1 to the upstream) |
**SSL tab**: SSL Certificate = *Request a new SSL Certificate* (Let's Encrypt), then switch on **Force SSL** and
**HTTP/2 Support** (HSTS is optional), agree to the terms, Save. Browsers fall back to HTTP/1.1 for the WebSocket;
that is normal.
**Advanced tab** (Custom Nginx Configuration): paste
```nginx
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
```
Not strictly needed (the server pings every 25 s), but it makes a long-idle game survive proxies and middleboxes
with short idle timers. Save, then open `https://game.example.com/`.
NPM is expected to pass the original `Host` header through unchanged (not verified here); then the browser's `Origin`
and the `Host` the server sees match and no configuration is needed. If the server log shows
`refused websocket ... does not match Host` (HTTP 403 on the upgrade), or if you set `Host` yourself in a custom
config, either make the proxy send `X-Forwarded-Host` with the public name, or set
`ALLOWED_ORIGINS=https://game.example.com`.
Optional, in the same Advanced box, to compress text for phones on mobile data (the Node server does not):
```nginx
gzip on;
gzip_types text/css text/javascript application/javascript application/json image/svg+xml;
```
## 6. Dual stack (IPv4 + IPv6)
* **The app side is automatic.** `HOST=::` (the default) is a single dual-stack socket: IPv4 clients arrive as
`::ffff:a.b.c.d` and are logged as plain IPv4. Check both families from inside the LXC:
```sh
curl -i http://127.0.0.1:8080/healthz
curl -gi 'http://[::1]:8080/healthz'
```
* **Public side.** Publish an `AAAA` record for `game.example.com` pointing at the proxy's IPv6 address, next to the
`A` record, **after** ports 80 and 443 really answer over IPv6 (router/firewall rules, NPM's container having an
IPv6 address). A published but broken `AAAA` breaks IPv6 visitors and certificate issuance. The proxy-to-LXC hop
can stay IPv4; that is independent of what visitors use.
* **Firewall.** Only the proxy should reach port 8080 (everybody else goes through TLS). Inside the LXC, for example
with nftables (`apt install nftables`; keep your SSH rule):
```text
# /etc/nftables.conf (excerpt)
table inet filter {
chain input {
type filter hook input priority 0; policy drop;
ct state established,related accept
iif lo accept
meta l4proto { icmp, ipv6-icmp } accept
tcp dport 22 accept
ip saddr 192.168.1.10 tcp dport 8080 accept # the NPM host (IPv4)
ip6 saddr fd00::10 tcp dport 8080 accept # the NPM host (IPv6), if it reaches the LXC over IPv6
}
}
```
`systemctl enable --now nftables`. The Proxmox firewall (Datacenter / Node / CT > Firewall) can do the same.
`TRUST_PROXY=auto` is meant for exactly this setup: only private-range peers (the proxy) are believed. If the port
is reachable by other LAN machines they could forge `X-Forwarded-For`; the effect is limited to the per-IP cap
and the log.
* **IPv6 disabled in the container.** If IPv6 is switched off by sysctl (`net.ipv6.conf.all.disable_ipv6=1`) the
`::` socket still binds and still serves IPv4 (tested in a throwaway Linux network namespace with
`disable_ipv6=1`, not in an LXC). Only a kernel without IPv6 at all fails to bind `::`
(`EAFNOSUPPORT`, `EADDRNOTAVAIL`, ...); the server then logs
`IPv6 is not available (...); falling back to 0.0.0.0 (IPv4 only)` and listens on IPv4. You can also set
`HOST=0.0.0.0`. `EADDRINUSE` (something else on the port) is an error and never triggers the fallback.
## 7. Under a path prefix (`https://example.com/netwall/`)
The client never uses absolute URLs: it opens `new URL('ws', location.href)`, so at `https://example.com/netwall/`
it connects to `wss://example.com/netwall/ws`. The proxy must **strip the prefix** so the server sees `/` and `/ws`,
and the page must be opened **with the trailing slash** (`/netwall/`, not `/netwall`, or the relative URLs resolve to
the wrong place). The behaviour you need is this nginx block (a trailing slash on `proxy_pass` replaces the matched
prefix):
```nginx
location = /netwall { return 301 /netwall/; }
location /netwall/ {
proxy_pass http://192.168.1.50:8080/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
```
In NPM the dialog differs between versions (Proxy Host > **Custom locations** tab, or the Advanced tab of a host).
Whichever you use, the result must be the block above; check with `curl -i https://example.com/netwall/healthz`
(expect `200 ok`). The simpler route is a dedicated sub-domain (`game.example.com`), which needs none of this.
Origin check: the browser's `Origin` is `https://example.com` and `Host` is `example.com`, so it passes unchanged.
## 8. Verify
```sh
# from anywhere
curl -i https://game.example.com/healthz # 200 ok
curl -I https://game.example.com/ # 200, Content-Security-Policy etc.
curl -6 -I https://game.example.com/ # over IPv6 (once the AAAA record exists)
# WebSocket through the proxy (run in /opt/netwall, uses the installed "ws").
# (Run here only against the local server, with the URL and origin pointed at it.)
node -e "
const WebSocket=require('ws');
const ws=new WebSocket('wss://game.example.com/ws',{origin:'https://game.example.com'});
ws.on('open',()=>ws.send(JSON.stringify({t:'join',pid:'deploy-check-0001',name:'check'})));
ws.on('message',d=>{const m=JSON.parse(d);console.log(m.t,m.room||'');if(m.t==='joined'){ws.close();}});
ws.on('unexpected-response',(q,r)=>{console.log('HTTP',r.statusCode);process.exit(1)});
ws.on('error',e=>{console.log(e.message);process.exit(1)});
"
# prints: joined XXXX
```
Then open the URL on two phones, create a room on one and enter the 4-letter code on the other.
## 9. Troubleshooting
| Symptom | Cause and fix |
|---|---|
| Page loads, but the game says it cannot reach the server / WebSocket never opens | **Websockets Support** is off in the Proxy Host, or something in front strips the `Upgrade` header. Turn it on. |
| WebSocket upgrade answered with **403** | Origin check: the browser's `Origin` host differs from the `Host` the server sees. The log says `refused websocket ... Origin "..." does not match Host "..."`. Fix the proxy so `Host` is the public name, or set `ALLOWED_ORIGINS=https://game.example.com` (comma separated for several) and restart. |
| Upgrade answered with **429** | More than `MAX_CONN_PER_IP` sockets from one IP. If every visitor appears to come from the proxy, `TRUST_PROXY` is `0`, or the proxy connects from a non-private address; set `TRUST_PROXY=1`. |
| WebSocket dies after exactly 60 s | Should not happen with the defaults: the server's 25 s pings keep nginx's 60 s `proxy_read_timeout` fed. If it does, something else sits in the path with its own idle timer (a CDN, a second proxy, a firewall) that is not passing the pings, or the proxy's read timeout is shorter than 25 s. Add the `proxy_read_timeout`/`proxy_send_timeout` snippet (section 5) and check every hop. |
| Players are dropped every few seconds and reconnect | The current client treats a socket as dead after 5 s without data (snapshots arrive 20 times a second). Something is buffering or throttling the stream; look at the proxy's buffering settings and at CPU use of the LXC. |
| **502 Bad Gateway** | NPM cannot reach the upstream. `systemctl status netwall`; from the NPM host run `curl -i http://<lxc>:8080/healthz`; check forward IP/port, the firewall (section 6) and that `HOST` is not `127.0.0.1`. |
| `listen EADDRINUSE` in the log, service restarts in a loop | Another process uses the port (`ss -ltnp | grep 8080`), or two copies are running. Change `PORT` and the NPM forward port together. |
| `IPv6 is not available (...); falling back to 0.0.0.0` | The kernel has no IPv6 at all. IPv4 keeps working; nothing to do, or set `HOST=0.0.0.0` to silence it. With IPv6 merely disabled by sysctl the `::` bind still works. |
| IPv6 visitors cannot connect, IPv4 can | The `AAAA` record points somewhere that does not answer on 443, or NPM/the firewall is not listening on IPv6. Test `curl -6 -I https://game.example.com/`. The app itself is dual-stack. |
| `status=226/NAMESPACE` when starting the service | You enabled the optional namespace-based hardening in an LXC without Nesting. Enable Proxmox **Options > Features > Nesting** for the container, or comment that block out again. |
| Old version of the game after an update | NPM *Cache Assets* is on, or a browser/service layer ignores `no-cache`. Switch Cache Assets off, hard-reload. |
| Phone loses the game when it sleeps | Expected: the socket dies. The client reconnects with the same player id and gets its slot back if that happens within `GRACE_MS` (60 s by default). Raise `GRACE_MS` for longer. |
| `Cannot find module 'ws'` | `npm install --omit=dev` was not run in `/opt/netwall`. |
## 10. Updating
```sh
systemctl stop netwall
# replace server/ and public/ (rsync as in section 2), then
cd /opt/netwall && npm install --omit=dev
systemctl start netwall
```
Running games end with the restart. For a quiet moment: `journalctl -u netwall | grep -E 'created|deleted'`
shows which rooms are open.