Upstream/docs/DEPLOY.md
jvanvik cc0524de5e Smoke e2e, docs consistency pass, status summary
Built: e2e/run.js + lib.js (playwright-core, in-process server, temp DB): desktop flow, phone touch portrait/landscape, layout matrix on 5 viewports x pop/map/trade/world (overflow + console errors), WebSocket push incoming-banner without reload. 61/61 checks, ~54 s, identical over 4 runs; the push and overflow checks were shown to fail when the behaviour is removed.
Docs: fixed stale numbers/claims in DESIGN, DECISIONS, BALANCE, API (undocumented routes, rate limits, notification types), DEPLOY, NEXT (status-at-a-glance on top). Deploy path exercised in a scratch dir: migrations, healthz, signup/build, admin CLI, SIGKILL catch-up, backup restore.

Tested: npm test 207/207, npm run e2e 61/61, bot run 50x14d INVARIANTS OK.
NOT tested: Chromium only; no systemd/nginx Proxy Manager/Node 22.13 run; e2e does not cover units, alliances, messages, trade or end-game flows beyond empty-state rendering; no nb, PWA or polling-fallback e2e; docs/workflows/ still empty.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2MJaQ5LXAHzGnTDgceT14
2026-10-11 20:34:35 +02:00

404 lines
29 KiB
Markdown

# Deploying UPSTREAM
UPSTREAM is one Node process: it serves the static client from `public/`, the REST API under `/api/v1/` and a
WebSocket push channel at `/ws` (docs/PROTOCOL.md). Unlike NETWALL it **keeps state**: one SQLite file, owned by that process. A restart loses
nothing: overdue events are caught up in order before the server accepts traffic.
```
phone / laptop --https--> nginx Proxy Manager (TLS, :443) --http (+ WebSocket)--> Debian 13 LXC: node :8080
/opt/upstream (code, read-only)
/var/lib/upstream (game.db, backups/)
```
Same shape as NETWALL (`docs/DEPLOY.md` there): same proxy, same hardening notes, same env vars for the network
layer. The differences are the Node version (`node:sqlite` needs 22.13+), a state directory and backups.
**What was and was not run while writing this** (so you know what to check on the real box):
* **Run:** `systemd-analyze verify` on the three unit files (syntax and directives only), the CLI `tools/admin.js`
and its tests on a dev machine (Node 24.21), a backup, verify and prune cycle, `sh -n` on `deploy/upstream-admin`.
Re-checked on 2026-10-11 with the code as it is now (Node 24.21, scratch directory, no systemd): first boot with `DB_PATH`/`PORT`/`HOST`
(migrations 001-004 applied, `created world 1`), `/healthz`, `/` headers, signup + build over the API, `admin.js` status / worlds / users /
operators / inspect-town / events / replay-events / backup / verify-backup / ban / unban / grant / end-world (refusal without `--yes`),
SIGTERM and restart (state, queue and password kept, no migrations re-applied; tokens revoked by `ban` stay revoked), SIGKILL then restart after a build fell due (`caught up 1 overdue event(s)`, the build level was applied), and a restore by copying a snapshot to a fresh `DB_PATH` and starting on it (section 7: it logged `caught up 1 overdue event(s)`).
* **Taken from NETWALL's docs, not re-run for UPSTREAM:** everything about nginx Proxy Manager (labels, header
behaviour, "Client addresses"). NETWALL's guide records what was observed against NPM v2.16.0.
* **Never run:** a Proxmox LXC, a started unit under systemd, the timer firing on its own, a Node install from the
tarball or NodeSource on Debian 13, `rrsync`, Let's Encrypt, a restore on the real box. Treat those steps as a
checklist to confirm, and fix this file when reality differs.
## 1. Requirements
* Debian 13 (trixie) LXC with its own IPv4 (optionally IPv6) address the proxy can reach. 1 core, 512 MB RAM and 4 GB
disk are plenty for a small world; the database grows with the event and notification history (measure it with
`upstream-admin status`), and snapshots multiply it by the retention count.
* **Node 22.13 or newer.** Debian 13 ships `nodejs` **20.19.2** (checked on tracker.debian.org: stable has 20.19.2,
testing and unstable have 24.21). Node 20 has no `node:sqlite`, so `apt install nodejs` is **not enough**. `package.json`
says `"engines": {"node": ">=22.13"}`; 22.5 to 22.12 only have `node:sqlite` behind `--experimental-sqlite`, which
the unit does not pass. Development is on Node 24.21.
* A DNS name for the game (for example `upstream.example.com`) pointing at the proxy.
### Getting a suitable Node (pick one)
**A. Official tarball (no third-party apt repo, nothing piped into a shell).** Pick a current 24.x (Active LTS) or
22.x from https://nodejs.org/en/download and put its version in `V`:
```sh
apt update && apt install -y curl xz-utils ca-certificates
V=24.21.0 # example: use the current 24.x (or >= 22.13) you picked
ARCH=linux-x64 # linux-arm64 on ARM hosts (uname -m: aarch64)
cd /tmp
curl -fsSLO https://nodejs.org/dist/v$V/node-v$V-$ARCH.tar.xz
curl -fsSLO https://nodejs.org/dist/v$V/SHASUMS256.txt
grep " node-v$V-$ARCH.tar.xz\$" SHASUMS256.txt | sha256sum -c - # must print: ... OK
tar -C /opt -xJf node-v$V-$ARCH.tar.xz
ln -sfn /opt/node-v$V-$ARCH /opt/node
ln -sf /opt/node/bin/node /usr/local/bin/node
ln -sf /opt/node/bin/npm /usr/local/bin/npm
ln -sf /opt/node/bin/npx /usr/local/bin/npx
```
(For extra assurance also verify the signature of `SHASUMS256.txt` as described on nodejs.org.) An upgrade later is
a new tarball and `ln -sfn` to it, then `systemctl restart upstream`. `upstream.service` starts `/usr/bin/env node`, and
systemd's default PATH lists `/usr/local/bin` before `/usr/bin`, so the tarball wins over any distro Node.
**B. NodeSource apt repository** (follows their documentation; not run here). It puts Node in `/usr/bin/node`:
```sh
apt install -y ca-certificates curl gnupg
install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_24.x nodistro main" > /etc/apt/sources.list.d/nodesource.list
apt update && apt install -y nodejs # includes npm; do not also install Debian's npm package
```
**Verify either way** (all three must succeed):
```sh
node --version # v22.13.0 or newer
node -e "require('node:sqlite'); console.log('node:sqlite ok')"
which -a node # the first one is the one the unit will use
```
On 22.x/24.x Node may print `ExperimentalWarning: SQLite is an experimental feature` to stderr; that is harmless (it
shows up in the journal once per start). `No such built-in module: node:sqlite` means Node is too old (or it is the
distro's 20.x earlier on the PATH). If `node:sqlite` ever misbehaves, `DECISIONS.md` D1 describes the one-file swap
to `better-sqlite3` in `server/db.js`; that would add a native dependency and a build toolchain to the LXC.
## 2. Create the LXC (Proxmox)
Container creation, the Proxmox UI way (CLI `pct create` works too):
* Template `debian-13-standard`, **unprivileged**, hostname `upstream`, 1 core, 512 MB RAM (+256 MB swap), 4-8 GB disk.
* Network: a static address (or a DHCP reservation) on a bridge the NPM host can reach. The proxy forwards to that
address, so it must not change.
* Features: **Nesting** is optional; tick it only if you want the namespace-based hardening block in the unit
(section 3). Everything else works without it.
* Start on boot: yes. Timezone does not matter: the game uses UTC ms, the backup timer uses the container's clock.
* Proxmox's own CT backups (`vzdump`) are not a substitute for the snapshots in section 7: a running-container copy of
a SQLite file is crash-consistent at best. Use them for the OS and let the verified snapshots be the data backup.
## 3. Install in the LXC
Run as root inside the container (Node from section 1 is already in place).
```sh
apt install -y rsync # for the copy below (or use git)
# a dedicated, login-less user that only runs the game
useradd --system --home-dir /var/lib/upstream --shell /usr/sbin/nologin upstream
# code: read-only for the service user, owned by root
mkdir -p /opt/upstream
# from your workstation:
# rsync -a --delete --exclude node_modules --exclude .git --exclude data --exclude e2e ./ root@<lxc>:/opt/upstream/
# (needs server/, public/, tools/, deploy/, docs/, package.json, package-lock.json; `test/` is optional)
cd /opt/upstream
npm ci --omit=dev # only "ws"; follows the lockfile exactly
chown -R root:root /opt/upstream
# the admin helper on the PATH
install -m 0755 /opt/upstream/deploy/upstream-admin /usr/local/bin/upstream-admin
# optional settings / secrets file (read by the unit and by upstream-admin)
install -d -m 0750 -o root -g upstream /etc/upstream
install -m 0640 -o root -g upstream /dev/null /etc/upstream/upstream.env
# edit it: SIGNUP_CODE=choose-a-long-random-string
# ALLOWED_ORIGINS=https://upstream.example.com
# the units
install -m 0644 /opt/upstream/deploy/upstream.service /opt/upstream/deploy/upstream-backup.service /opt/upstream/deploy/upstream-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now upstream upstream-backup.timer
systemctl status upstream
journalctl -u upstream -f
```
The first start prints `applied migrations: 001_init.sql, ...`, `created world 1` and the listen banner with
`database: /var/lib/upstream/game.db`. Quick manual run as the service user instead of systemd (Ctrl-C to stop):
`runuser -u upstream -- env PORT=8080 DB_PATH=/var/lib/upstream/game.db node /opt/upstream/server/index.js`.
### What the units do
* `deploy/upstream.service`: user `upstream`, `WorkingDirectory=/opt/upstream`, `StateDirectory=upstream` (systemd creates
`/var/lib/upstream` owned by that user, mode 0700, so the game can write nowhere else), `Restart=on-failure`,
`EnvironmentFile=-/etc/upstream/upstream.env` (overrides the `Environment=` lines), `UMask=0077` (database and
snapshots hold password hashes: owner-only).
* Hardening that works in any container: `NoNewPrivileges`, empty capability set (port 8080 needs none),
`RestrictAddressFamilies`, `RestrictRealtime`, `RestrictSUIDSGID`, `LockPersonality`, `SystemCallArchitectures`.
No `MemoryDenyWriteExecute` (V8 would crash).
* The namespace block (`ProtectSystem=strict`, `ReadWritePaths=/var/lib/upstream`, `PrivateTmp`, ...) is commented out.
In an unprivileged LXC it fails with `status=226/NAMESPACE` unless **Options > Features > Nesting** is on. Because the
app writes to disk, `ProtectSystem=strict` needs the `ReadWritePaths=` line that is already next to it in the unit;
without it the database cannot be opened.
* Shutdown: SIGTERM stops the scheduler timer, closes the listener and the database and exits 0 in about a second.
Even a SIGKILL or power cut is survivable (WAL replay on the next open, then overdue events are caught up).
* `deploy/upstream-backup.service` (oneshot) and `.timer`: see section 7.
## 4. Configuration
Environment variables read by `node server/index.js`. Put changes in `/etc/upstream/upstream.env` (`KEY=value`, one per
line, no `export`) or in a `systemctl edit upstream` override, then `systemctl restart upstream`.
| Variable | Default (unit) | Meaning |
|---|---|---|
| `PORT` | `8080` | TCP port. Unprivileged, so no capabilities needed. `0` picks a free one (banner prints it). |
| `HOST` | `::` | Bind address. `::` = one dual-stack socket for IPv4 and IPv6 (falls back to `0.0.0.0` if the kernel has no IPv6). `127.0.0.1` = this container only. |
| `DB_PATH` | `/var/lib/upstream/game.db` | SQLite file. The code default is `./data/game.db`; the unit overrides it. Its directory is created if missing, it must be writable by the service user. `-wal` and `-shm` files appear next to it: always copy via a snapshot (section 7), never with `cp` while running. |
| `TRUST_PROXY` | `auto` | `auto`: believe `X-Forwarded-For` / `X-Forwarded-Host` only from loopback / private-range peers. `1`: always (then firewall the port). `0`: never. Used for the per-IP rate limits and the log. |
| `TRUSTED_PROXIES` | empty | Extra IPs / CIDRs `auto` believes: needed when the proxy reaches the LXC from a global IPv6, Tailscale / CGNAT `100.64.0.0/10` or another routable address. |
| `ALLOWED_ORIGINS` | empty | Extra origins for the WebSocket origin check, comma separated, with the port when it is not 80/443. The REST API uses bearer tokens (no cookies) and does not check origins. |
| `MAX_CONN_PER_IP` | `40` | Open WebSockets per client address (HTTP connections are not capped). `0` disables. |
| `WORLD_SPEED` | unset (1) | World speed multiplier, **first boot only**, when the database has no world yet. |
| `WORLD_RADIUS` | unset (50) | Map radius in tiles (coordinates `-R..R`, so 50 = 101 x 101), **first boot only**. |
| `WORLD_NAME` | unset (`World 1`) | Name of the first world, **first boot only**. |
| `SIGNUP_CODE` | empty | If set, `POST /auth/signup` needs `"invite": "<this>"` in the body or answers `signup_closed` (403). Empty = open signup. Keep it in `/etc/upstream/upstream.env`, not in the unit file. |
| `UPSTREAM_DIR`, `UPSTREAM_ENV_FILE` | `/opt/upstream`, `/etc/upstream/upstream.env` | Only for the `upstream-admin` wrapper. |
Fixed in code (options of `createServer()`): per-IP and per-token rate limits (login and signup are stricter: about one
per 10 s sustained, burst 8, per client address), 32 KB request bodies, idempotency keys pruned hourly.
### First boot
An empty or missing database is created, all migrations are applied, and **one world is created** from `WORLD_SPEED`,
`WORLD_RADIUS`, `WORLD_NAME` (seed random). After that those variables are ignored: changing them does not touch the
existing world. More worlds, a different seed, ending or resetting a world are admin tasks (section 6). The first
player to sign up becomes an ordinary operator; there is no special admin account, admin work is the CLI on the box.
## 5. nginx Proxy Manager
In NPM: **Hosts > Proxy Hosts > Add Proxy Host**.
| Tab / field | Value |
|---|---|
| Details: Domain Names | `upstream.example.com` |
| Details: Scheme | `http` |
| Details: Forward Hostname / IP | the LXC's address (for example `192.168.1.51`), not `127.0.0.1` when NPM runs elsewhere or in its own container |
| Details: Forward Port | `8080` |
| Details: Cache Assets | **off** (static files are `no-cache` and revalidated by ETag; the game must see new versions) |
| Details: Block Common Exploits | optional |
| Details: **Websockets Support** | **on.** The REST game works without it, but `/ws` (live push) fails with HTTP 426 when this is off. |
| Custom Locations | none needed |
| SSL | Request a new Let's Encrypt certificate, **Force SSL**, **HTTP/2** (HSTS optional) |
| Advanced (gear) | nothing required; optional gzip below |
Optional, Advanced box, to compress JS/JSON for phones (the Node server does not compress; NPM's own `gzip on` only
covers HTML):
```nginx
gzip on;
gzip_types text/css text/javascript application/javascript application/json application/manifest+json image/svg+xml;
```
Notes carried over from NETWALL's setup (observed there against NPM v2.16.0, not re-run for UPSTREAM):
* NPM sends `Host` **without the public port** and no `X-Forwarded-Host`. On a public port other than 443 the browser
`Origin` then differs from `Host`; list the origin with its port in `ALLOWED_ORIGINS` (matters for `/ws`).
* A proxy on a non-private address (global IPv6, Tailscale) is not believed by `TRUST_PROXY=auto`: every player then
shares one rate-limit budget and the log shows the proxy's address. Fix: `TRUSTED_PROXIES=<its address or CIDR>`.
* Clients on your own LAN can pick their address through NPM's `real_ip` handling, so the per-IP limits protect against
public abuse, not against LAN users.
* **Idle timeout:** nginx's `proxy_read_timeout` is 60 s by default (NETWALL's notes say 90 s for NPM; not re-checked). NETWALL never
idles because it streams 20 snapshots a second; UPSTREAM's socket is quiet for long stretches. `server/push.js` sends a protocol ping
every 30 s (`wsPingMs`), below either value, which is reasoning, not a measurement behind a real NPM.
Firewall and dual stack: identical to NETWALL (`docs/DEPLOY.md` there, section 6): only the proxy needs to reach port
8080. `curl -i http://127.0.0.1:8080/healthz` and `curl -gi 'http://[::1]:8080/healthz'` check both families.
## 6. Operating: `upstream-admin`
`tools/admin.js` is the operator CLI (Node core only, uses the game's own `db.js`, `state.js`, `rules.js`).
`deploy/upstream-admin` runs it as the service user against the service's database (it reads `DB_PATH` from
`/etc/upstream/upstream.env`). **Do not run `node tools/admin.js` as root**: SQLite would create `game.db-wal` and
`-shm` owned by root and the game could no longer open its database (symptom in section 10).
```sh
upstream-admin help
upstream-admin status # file, size, migrations, overdue events
upstream-admin worlds
upstream-admin create-world --name "Summer" --speed 2 --radius 60 --seed 1234 --end-in 90d
upstream-admin end-world 1 --yes # flag flip only, see below
upstream-admin users [--banned] # no hashes or tokens are ever printed
upstream-admin operators [--world 1]
upstream-admin ban mallory # sets banned, revokes all tokens: old tokens give 401, login 403
upstream-admin unban mallory
upstream-admin revoke-tokens ada # log out everywhere
upstream-admin inspect-operator ada # (--world N when the name exists in several worlds, --id N by operator id)
upstream-admin inspect-town 12 # resources now (rules.resourcesAt), buildings, queue, units if the table exists
upstream-admin events --pending [--type buildDone --town 12 --limit 100]
upstream-admin replay-events --until +6h # dry run: what the scheduler would process, in order
upstream-admin replay-events --until +6h --simulate # + run the real handlers on a scratch copy, report outcome and follow-ups
upstream-admin backup /var/lib/upstream/backups --keep 14
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db
upstream-admin grant 12 --fiber 5000 --capital 2000 --yes # testing only
upstream-admin reset-world 1 --yes-destroy --backup /var/lib/upstream/backups --new-seed
upstream-admin migrate # apply pending migrations without starting the server
```
All commands take `--json` (machine output) and `--db FILE` (otherwise `$DB_PATH`, otherwise `./data/game.db`).
Exit codes: `0` ok, `1` failed, `2` usage error **or a destructive command missing its flag**, `3` not found.
Safety rules the CLI follows:
* Destructive or test-only commands refuse without a flag: `end-world` and `grant` need `--yes`; `reset-world`
needs `--yes-destroy` (without it, it prints what it would delete and exits 2). `ban`, `unban`, `revoke-tokens` are
reversible and need none.
* `end-world` just sets the world's status to `ended` (and `end_at` to now): signup and join stop and every game command answers
`world_ended`, but it writes no ranking snapshot (`GET /worlds/:id/results` stays `available: false`), cancels no pending event and
closes no trade offer. The natural end (a backbone winner, then the `worldEnd` event two world days later) does all of that.
* `reset-world` deletes the world's game rows (everything with a `world_id`, plus tables pointing at them: towns,
buildings, events, operators, alliances, messages, ... discovered from the schema, so tables added by later stages
are covered) and **keeps accounts and tokens**; players must join the world again. It resets `start_at`, clears
`end_at` and sets the world running. It rolls back if it would leave dangling references. Stop the game first
(`systemctl stop upstream`): the server does not know the world was swept while it is running. Use `--backup DIR`.
* Everything runs on a running server's database: writes are single short `BEGIN IMMEDIATE` transactions, the
connection is always closed, and SQLite's 5 s busy timeout covers collisions with the game (`database is locked`
from the CLI just means: try again).
* `grant` adds to the town's stored amounts (valid at its `res_at`); the lazy model carries them forward. Amounts above
the Rack hall cap are kept but stop growing, like a lowered cap.
* `inspect-town` shows values as of now **ignoring** overdue events; it says so when there are any (server stopped).
## 7. Backups and restore
`upstream-backup.timer` runs `upstream-backup.service` daily at 04:15 (10 min jitter, `Persistent=true`). It runs
`tools/admin.js backup /var/lib/upstream/backups --keep 14` as the service user. A snapshot is:
1. `VACUUM INTO` a temporary `.partial-*` file: a consistent copy as of one moment, safe while the game runs (WAL readers do
not block the writer; the game keeps its own 5 s busy timeout);
2. verified: opened read-only, `PRAGMA integrity_check` = `ok`, `worlds` and `schema_migrations` present;
3. published under a dated name `upstream-YYYYMMDDTHHMMSSZ.db` (UTC) with a hard link, so an existing file is never
replaced (a same-second collision becomes `...Z-1.db`), mode 0600;
4. only then older snapshots are pruned: `--keep N` keeps the newest N, `--keep-days D` keeps those younger than D days,
both = survive if either keeps it, neither = no pruning. The newest snapshot and any file not matching the naming
pattern are never deleted. A failed verification leaves nothing behind and prunes nothing, and the unit fails
(`systemctl status upstream-backup`, `journalctl -u upstream-backup`).
Snapshots are plain SQLite files (journal mode `delete`), not `-wal` dependent: copy a single file.
```sh
systemctl list-timers upstream-backup.timer # next run
systemctl start upstream-backup # one right now
journalctl -u upstream-backup -n 20
ls -l /var/lib/upstream/backups
```
Change the schedule or retention with `systemctl edit upstream-backup.timer` / `upstream-backup.service` (for every
6 hours: `OnCalendar=*-*-* 00/6:15:00` and a larger `--keep`). Before an upgrade or a `reset-world`, take one by hand.
### Off-box copy
Snapshots on the same disk die with the container. Pull them from another machine (NAS, workstation, the Proxmox host),
for example from cron or a timer there:
```sh
rsync -a --ignore-existing root@<lxc>:/var/lib/upstream/backups/ /srv/backups/upstream/
```
`--ignore-existing` makes it append-only on your side (nothing already stored is overwritten, and snapshots pruned on
the LXC stay on the backup host until you decide otherwise). Better than a root key: a dedicated key restricted in
`authorized_keys` to `command="rrsync -ro /var/lib/upstream/backups",restrict ...` (`rrsync` comes with the `rsync`
package; not run here). The files are owner-only (`upstream`), so the reading user needs root or a group ACL.
### Restore
The restore is a file copy, and the time model makes it clean: the restored world simply fast-forwards. Every event that
fell due since the snapshot is processed in order at start-up, each at its own due time. What is lost is what players did after
the snapshot (builds started, signups, messages) and their tokens: players log in again.
```sh
systemctl stop upstream-backup.timer upstream
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db # exit 0 or stop here
mkdir -p /var/lib/upstream/pre-restore
mv /var/lib/upstream/game.db* /var/lib/upstream/pre-restore/ 2>/dev/null # keep the old one, incl. -wal/-shm
install -o upstream -g upstream -m 0600 /var/lib/upstream/backups/upstream-20261011T041500Z.db /var/lib/upstream/game.db
systemctl start upstream upstream-backup.timer
journalctl -u upstream -n 20 # look for "caught up N overdue event(s)"
upstream-admin status
```
Restoring onto another machine works the same way (install code and Node, copy the file to `DB_PATH`). Migrations in
the snapshot are checksummed: if the code is newer than the snapshot, the missing migrations are applied at boot; run
code that is **older** than the snapshot's last migration (`upstream-admin status` shows it) only if you know the schema
still fits, the server does not check for that.
## 8. Upgrading
```sh
upstream-admin backup /var/lib/upstream/backups --keep 14 # 1. snapshot first
rsync -a --delete --exclude node_modules --exclude .git --exclude data ./ root@<lxc>:/opt/upstream/ # 2. new code (from your workstation)
cd /opt/upstream && npm ci --omit=dev && chown -R root:root /opt/upstream # 3. dependencies, ownership
systemctl restart upstream # 4. migrations apply at boot
journalctl -u upstream -n 30 # 5. "applied migrations: ..." then the listen banner
curl -i http://127.0.0.1:8080/healthz
```
* Migrations are `server/migrations/NNN_name.sql`, applied in order inside a transaction each, **recorded with a
checksum and never edited after they shipped**. If one that was applied has changed, the server refuses to start
(`migration 001_init.sql was edited after it was applied`): restore that file from git and ship a **new** numbered
migration instead. `upstream-admin status` lists applied and pending migrations; `upstream-admin migrate` applies
them without starting the game.
* Each migration runs in its own transaction: a failing one rolls back and the server exits, earlier ones of the same
boot stay applied. Going back to the old code after a migration committed means restoring the pre-upgrade snapshot
(section 7).
* Updated unit files: `install -m 0644 ...`, `systemctl daemon-reload`, `systemctl restart upstream`.
* Players reconnecting after the restart need no action; clients get `serverTime` in every response and refetch.
## 9. Health, logs, checks
```sh
curl -i https://upstream.example.com/healthz # 200 ok (only says the process answers HTTP)
curl -I https://upstream.example.com/ # 200 + Content-Security-Policy
upstream-admin status # database, migrations, events pending / OVERDUE
journalctl -u upstream -f # live log; -p err for failures only
journalctl -u upstream --since "1 hour ago" | grep -E "failed|refused"
```
The log has: applied migrations, created world, caught-up events, `refused rate_limited(...) ip=...`, and
`event N (type) failed: ...` for a handler that threw (that event is marked `failed` and the queue moves on; list them
with `upstream-admin events --status failed`). Player IP addresses appear in the log; passwords and tokens never do.
A healthy server has **no overdue events** in `status`: the timer processes events within milliseconds of their due time.
## 10. Troubleshooting
| Symptom | Cause and fix |
|---|---|
| Service dies at start: `No such built-in module: node:sqlite` | Node is older than 22.13, or the distro's Node 20 is first on the PATH. `node --version`, `which -a node`, section 1. |
| `ExperimentalWarning: SQLite is an experimental feature` in the journal | Harmless. Printed once per start on some Node versions. |
| `SQLITE_CANTOPEN` / `unable to open database file` | `DB_PATH`'s directory is not writable by `upstream`: wrong path in `upstream.env`, `/var/lib/upstream` created by hand with other ownership (`chown -R upstream:upstream /var/lib/upstream`), or `ProtectSystem=strict` enabled without `ReadWritePaths=/var/lib/upstream`. |
| `attempt to write a readonly database` / `unable to open database file` right after using the CLI | `admin.js` was run as root and left root-owned `game.db-wal` / `-shm`. Stop the game, `chown upstream:upstream /var/lib/upstream/game.db*`, start. Use `upstream-admin`, which switches user. |
| `migration NNN_x.sql was edited after it was applied` | An applied migration changed on disk (checksum mismatch). Restore it from git; changes go in a new migration. |
| `UPSTREAM failed to start: ...` with `EADDRINUSE` | Another process on the port (`ss -ltnp \| grep 8080`) or two copies. Change `PORT` and the NPM forward port together. |
| `IPv6 is not available (...); falling back to 0.0.0.0` | The kernel has no IPv6. IPv4 keeps working; set `HOST=0.0.0.0` to silence it. |
| `status=226/NAMESPACE` | The optional namespace hardening is on in an LXC without Nesting: enable **Options > Features > Nesting** and restart the CT, or comment the block out. |
| `Cannot find module 'ws'` | `npm ci --omit=dev` was not run in `/opt/upstream`. |
| 502 Bad Gateway from NPM | Upstream unreachable: `systemctl status upstream`, `curl -i http://<lxc>:8080/healthz` from the NPM host, forward IP/port, firewall. |
| Page loads but a new deploy still shows the old client | NPM *Cache Assets* is on, or a service layer ignores `no-cache`. Switch it off, hard-reload. |
| Signup answers 403 `signup_closed` | `SIGNUP_CODE` is set. Send `{"invite": "<code>"}` in the signup body, or unset it and restart. |
| Login/signup answers 429 `rate_limited` for many players | They all arrive from one address: the proxy is not trusted so the per-IP budget is shared (log says `ip=<the proxy>`). Set `TRUSTED_PROXIES`. Login and signup allow about one request per 10 s per address, burst 8; a whole classroom behind one NAT can hit that. |
| Game seems frozen: builds never finish | `upstream-admin status`: `OVERDUE` events mean the scheduler is not running (server stopped or stuck) or handlers fail: `upstream-admin events --status failed`, journal `event N ... failed`. A restart catches up everything in order. |
| Banned user still plays | They cannot: `ban` revokes every token and blocks login. Check the username spelling (`upstream-admin users --banned`). |
| `database is locked` from the CLI | The game held a write lock longer than the 5 s busy timeout. Retry. |
| `upstream-backup.service` failed | `journalctl -u upstream-backup`. Typical: disk full, or the integrity check of the new snapshot failed (then nothing was kept and nothing pruned; check the disk and the `game.db` itself). |
| Backups not appearing | `systemctl list-timers upstream-backup.timer`; the timer must be **enabled** (`systemctl enable --now upstream-backup.timer`), not just installed. |
| World reset by mistake | Restore the snapshot taken before (section 7), or the one `reset-world --backup` wrote. |
| WebSocket upgrade answered 426 / 403 / 429 | 426: **Websockets Support** off in NPM. 403: origin check, see `ALLOWED_ORIGINS` and the notes in section 5. 429: more than `MAX_CONN_PER_IP` sockets from one address. |