Upstream/docs/DEPLOY.md
jvanvik cc0524de5e Smoke e2e, docs consistency pass, status summary
Built: e2e/run.js + lib.js (playwright-core, in-process server, temp DB): desktop flow, phone touch portrait/landscape, layout matrix on 5 viewports x pop/map/trade/world (overflow + console errors), WebSocket push incoming-banner without reload. 61/61 checks, ~54 s, identical over 4 runs; the push and overflow checks were shown to fail when the behaviour is removed.
Docs: fixed stale numbers/claims in DESIGN, DECISIONS, BALANCE, API (undocumented routes, rate limits, notification types), DEPLOY, NEXT (status-at-a-glance on top). Deploy path exercised in a scratch dir: migrations, healthz, signup/build, admin CLI, SIGKILL catch-up, backup restore.

Tested: npm test 207/207, npm run e2e 61/61, bot run 50x14d INVARIANTS OK.
NOT tested: Chromium only; no systemd/nginx Proxy Manager/Node 22.13 run; e2e does not cover units, alliances, messages, trade or end-game flows beyond empty-state rendering; no nb, PWA or polling-fallback e2e; docs/workflows/ still empty.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2MJaQ5LXAHzGnTDgceT14
2026-10-11 20:34:35 +02:00

29 KiB

Deploying UPSTREAM

UPSTREAM is one Node process: it serves the static client from public/, the REST API under /api/v1/ and a WebSocket push channel at /ws (docs/PROTOCOL.md). Unlike NETWALL it keeps state: one SQLite file, owned by that process. A restart loses nothing: overdue events are caught up in order before the server accepts traffic.

 phone / laptop --https--> nginx Proxy Manager (TLS, :443) --http (+ WebSocket)--> Debian 13 LXC: node :8080
                                                                                    /opt/upstream      (code, read-only)
                                                                                    /var/lib/upstream  (game.db, backups/)

Same shape as NETWALL (docs/DEPLOY.md there): same proxy, same hardening notes, same env vars for the network layer. The differences are the Node version (node:sqlite needs 22.13+), a state directory and backups.

What was and was not run while writing this (so you know what to check on the real box):

  • Run: systemd-analyze verify on the three unit files (syntax and directives only), the CLI tools/admin.js and its tests on a dev machine (Node 24.21), a backup, verify and prune cycle, sh -n on deploy/upstream-admin. Re-checked on 2026-10-11 with the code as it is now (Node 24.21, scratch directory, no systemd): first boot with DB_PATH/PORT/HOST (migrations 001-004 applied, created world 1), /healthz, / headers, signup + build over the API, admin.js status / worlds / users / operators / inspect-town / events / replay-events / backup / verify-backup / ban / unban / grant / end-world (refusal without --yes), SIGTERM and restart (state, queue and password kept, no migrations re-applied; tokens revoked by ban stay revoked), SIGKILL then restart after a build fell due (caught up 1 overdue event(s), the build level was applied), and a restore by copying a snapshot to a fresh DB_PATH and starting on it (section 7: it logged caught up 1 overdue event(s)).
  • Taken from NETWALL's docs, not re-run for UPSTREAM: everything about nginx Proxy Manager (labels, header behaviour, "Client addresses"). NETWALL's guide records what was observed against NPM v2.16.0.
  • Never run: a Proxmox LXC, a started unit under systemd, the timer firing on its own, a Node install from the tarball or NodeSource on Debian 13, rrsync, Let's Encrypt, a restore on the real box. Treat those steps as a checklist to confirm, and fix this file when reality differs.

1. Requirements

  • Debian 13 (trixie) LXC with its own IPv4 (optionally IPv6) address the proxy can reach. 1 core, 512 MB RAM and 4 GB disk are plenty for a small world; the database grows with the event and notification history (measure it with upstream-admin status), and snapshots multiply it by the retention count.
  • Node 22.13 or newer. Debian 13 ships nodejs 20.19.2 (checked on tracker.debian.org: stable has 20.19.2, testing and unstable have 24.21). Node 20 has no node:sqlite, so apt install nodejs is not enough. package.json says "engines": {"node": ">=22.13"}; 22.5 to 22.12 only have node:sqlite behind --experimental-sqlite, which the unit does not pass. Development is on Node 24.21.
  • A DNS name for the game (for example upstream.example.com) pointing at the proxy.

Getting a suitable Node (pick one)

A. Official tarball (no third-party apt repo, nothing piped into a shell). Pick a current 24.x (Active LTS) or 22.x from https://nodejs.org/en/download and put its version in V:

apt update && apt install -y curl xz-utils ca-certificates
V=24.21.0                                  # example: use the current 24.x (or >= 22.13) you picked
ARCH=linux-x64                             # linux-arm64 on ARM hosts (uname -m: aarch64)
cd /tmp
curl -fsSLO https://nodejs.org/dist/v$V/node-v$V-$ARCH.tar.xz
curl -fsSLO https://nodejs.org/dist/v$V/SHASUMS256.txt
grep " node-v$V-$ARCH.tar.xz\$" SHASUMS256.txt | sha256sum -c -       # must print: ... OK
tar -C /opt -xJf node-v$V-$ARCH.tar.xz
ln -sfn /opt/node-v$V-$ARCH /opt/node
ln -sf /opt/node/bin/node /usr/local/bin/node
ln -sf /opt/node/bin/npm  /usr/local/bin/npm
ln -sf /opt/node/bin/npx  /usr/local/bin/npx

(For extra assurance also verify the signature of SHASUMS256.txt as described on nodejs.org.) An upgrade later is a new tarball and ln -sfn to it, then systemctl restart upstream. upstream.service starts /usr/bin/env node, and systemd's default PATH lists /usr/local/bin before /usr/bin, so the tarball wins over any distro Node.

B. NodeSource apt repository (follows their documentation; not run here). It puts Node in /usr/bin/node:

apt install -y ca-certificates curl gnupg
install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_24.x nodistro main" > /etc/apt/sources.list.d/nodesource.list
apt update && apt install -y nodejs        # includes npm; do not also install Debian's npm package

Verify either way (all three must succeed):

node --version                                        # v22.13.0 or newer
node -e "require('node:sqlite'); console.log('node:sqlite ok')"
which -a node                                         # the first one is the one the unit will use

On 22.x/24.x Node may print ExperimentalWarning: SQLite is an experimental feature to stderr; that is harmless (it shows up in the journal once per start). No such built-in module: node:sqlite means Node is too old (or it is the distro's 20.x earlier on the PATH). If node:sqlite ever misbehaves, DECISIONS.md D1 describes the one-file swap to better-sqlite3 in server/db.js; that would add a native dependency and a build toolchain to the LXC.

2. Create the LXC (Proxmox)

Container creation, the Proxmox UI way (CLI pct create works too):

  • Template debian-13-standard, unprivileged, hostname upstream, 1 core, 512 MB RAM (+256 MB swap), 4-8 GB disk.
  • Network: a static address (or a DHCP reservation) on a bridge the NPM host can reach. The proxy forwards to that address, so it must not change.
  • Features: Nesting is optional; tick it only if you want the namespace-based hardening block in the unit (section 3). Everything else works without it.
  • Start on boot: yes. Timezone does not matter: the game uses UTC ms, the backup timer uses the container's clock.
  • Proxmox's own CT backups (vzdump) are not a substitute for the snapshots in section 7: a running-container copy of a SQLite file is crash-consistent at best. Use them for the OS and let the verified snapshots be the data backup.

3. Install in the LXC

Run as root inside the container (Node from section 1 is already in place).

apt install -y rsync                               # for the copy below (or use git)

# a dedicated, login-less user that only runs the game
useradd --system --home-dir /var/lib/upstream --shell /usr/sbin/nologin upstream

# code: read-only for the service user, owned by root
mkdir -p /opt/upstream
#   from your workstation:
#   rsync -a --delete --exclude node_modules --exclude .git --exclude data --exclude e2e ./ root@<lxc>:/opt/upstream/
#   (needs server/, public/, tools/, deploy/, docs/, package.json, package-lock.json; `test/` is optional)
cd /opt/upstream
npm ci --omit=dev                                   # only "ws"; follows the lockfile exactly
chown -R root:root /opt/upstream

# the admin helper on the PATH
install -m 0755 /opt/upstream/deploy/upstream-admin /usr/local/bin/upstream-admin

# optional settings / secrets file (read by the unit and by upstream-admin)
install -d -m 0750 -o root -g upstream /etc/upstream
install -m 0640 -o root -g upstream /dev/null /etc/upstream/upstream.env
#   edit it:  SIGNUP_CODE=choose-a-long-random-string
#             ALLOWED_ORIGINS=https://upstream.example.com

# the units
install -m 0644 /opt/upstream/deploy/upstream.service /opt/upstream/deploy/upstream-backup.service /opt/upstream/deploy/upstream-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now upstream upstream-backup.timer
systemctl status upstream
journalctl -u upstream -f

The first start prints applied migrations: 001_init.sql, ..., created world 1 and the listen banner with database: /var/lib/upstream/game.db. Quick manual run as the service user instead of systemd (Ctrl-C to stop): runuser -u upstream -- env PORT=8080 DB_PATH=/var/lib/upstream/game.db node /opt/upstream/server/index.js.

What the units do

  • deploy/upstream.service: user upstream, WorkingDirectory=/opt/upstream, StateDirectory=upstream (systemd creates /var/lib/upstream owned by that user, mode 0700, so the game can write nowhere else), Restart=on-failure, EnvironmentFile=-/etc/upstream/upstream.env (overrides the Environment= lines), UMask=0077 (database and snapshots hold password hashes: owner-only).
  • Hardening that works in any container: NoNewPrivileges, empty capability set (port 8080 needs none), RestrictAddressFamilies, RestrictRealtime, RestrictSUIDSGID, LockPersonality, SystemCallArchitectures. No MemoryDenyWriteExecute (V8 would crash).
  • The namespace block (ProtectSystem=strict, ReadWritePaths=/var/lib/upstream, PrivateTmp, ...) is commented out. In an unprivileged LXC it fails with status=226/NAMESPACE unless Options > Features > Nesting is on. Because the app writes to disk, ProtectSystem=strict needs the ReadWritePaths= line that is already next to it in the unit; without it the database cannot be opened.
  • Shutdown: SIGTERM stops the scheduler timer, closes the listener and the database and exits 0 in about a second. Even a SIGKILL or power cut is survivable (WAL replay on the next open, then overdue events are caught up).
  • deploy/upstream-backup.service (oneshot) and .timer: see section 7.

4. Configuration

Environment variables read by node server/index.js. Put changes in /etc/upstream/upstream.env (KEY=value, one per line, no export) or in a systemctl edit upstream override, then systemctl restart upstream.

Variable Default (unit) Meaning
PORT 8080 TCP port. Unprivileged, so no capabilities needed. 0 picks a free one (banner prints it).
HOST :: Bind address. :: = one dual-stack socket for IPv4 and IPv6 (falls back to 0.0.0.0 if the kernel has no IPv6). 127.0.0.1 = this container only.
DB_PATH /var/lib/upstream/game.db SQLite file. The code default is ./data/game.db; the unit overrides it. Its directory is created if missing, it must be writable by the service user. -wal and -shm files appear next to it: always copy via a snapshot (section 7), never with cp while running.
TRUST_PROXY auto auto: believe X-Forwarded-For / X-Forwarded-Host only from loopback / private-range peers. 1: always (then firewall the port). 0: never. Used for the per-IP rate limits and the log.
TRUSTED_PROXIES empty Extra IPs / CIDRs auto believes: needed when the proxy reaches the LXC from a global IPv6, Tailscale / CGNAT 100.64.0.0/10 or another routable address.
ALLOWED_ORIGINS empty Extra origins for the WebSocket origin check, comma separated, with the port when it is not 80/443. The REST API uses bearer tokens (no cookies) and does not check origins.
MAX_CONN_PER_IP 40 Open WebSockets per client address (HTTP connections are not capped). 0 disables.
WORLD_SPEED unset (1) World speed multiplier, first boot only, when the database has no world yet.
WORLD_RADIUS unset (50) Map radius in tiles (coordinates -R..R, so 50 = 101 x 101), first boot only.
WORLD_NAME unset (World 1) Name of the first world, first boot only.
SIGNUP_CODE empty If set, POST /auth/signup needs "invite": "<this>" in the body or answers signup_closed (403). Empty = open signup. Keep it in /etc/upstream/upstream.env, not in the unit file.
UPSTREAM_DIR, UPSTREAM_ENV_FILE /opt/upstream, /etc/upstream/upstream.env Only for the upstream-admin wrapper.

Fixed in code (options of createServer()): per-IP and per-token rate limits (login and signup are stricter: about one per 10 s sustained, burst 8, per client address), 32 KB request bodies, idempotency keys pruned hourly.

First boot

An empty or missing database is created, all migrations are applied, and one world is created from WORLD_SPEED, WORLD_RADIUS, WORLD_NAME (seed random). After that those variables are ignored: changing them does not touch the existing world. More worlds, a different seed, ending or resetting a world are admin tasks (section 6). The first player to sign up becomes an ordinary operator; there is no special admin account, admin work is the CLI on the box.

5. nginx Proxy Manager

In NPM: Hosts > Proxy Hosts > Add Proxy Host.

Tab / field Value
Details: Domain Names upstream.example.com
Details: Scheme http
Details: Forward Hostname / IP the LXC's address (for example 192.168.1.51), not 127.0.0.1 when NPM runs elsewhere or in its own container
Details: Forward Port 8080
Details: Cache Assets off (static files are no-cache and revalidated by ETag; the game must see new versions)
Details: Block Common Exploits optional
Details: Websockets Support on. The REST game works without it, but /ws (live push) fails with HTTP 426 when this is off.
Custom Locations none needed
SSL Request a new Let's Encrypt certificate, Force SSL, HTTP/2 (HSTS optional)
Advanced (gear) nothing required; optional gzip below

Optional, Advanced box, to compress JS/JSON for phones (the Node server does not compress; NPM's own gzip on only covers HTML):

gzip on;
gzip_types text/css text/javascript application/javascript application/json application/manifest+json image/svg+xml;

Notes carried over from NETWALL's setup (observed there against NPM v2.16.0, not re-run for UPSTREAM):

  • NPM sends Host without the public port and no X-Forwarded-Host. On a public port other than 443 the browser Origin then differs from Host; list the origin with its port in ALLOWED_ORIGINS (matters for /ws).
  • A proxy on a non-private address (global IPv6, Tailscale) is not believed by TRUST_PROXY=auto: every player then shares one rate-limit budget and the log shows the proxy's address. Fix: TRUSTED_PROXIES=<its address or CIDR>.
  • Clients on your own LAN can pick their address through NPM's real_ip handling, so the per-IP limits protect against public abuse, not against LAN users.
  • Idle timeout: nginx's proxy_read_timeout is 60 s by default (NETWALL's notes say 90 s for NPM; not re-checked). NETWALL never idles because it streams 20 snapshots a second; UPSTREAM's socket is quiet for long stretches. server/push.js sends a protocol ping every 30 s (wsPingMs), below either value, which is reasoning, not a measurement behind a real NPM.

Firewall and dual stack: identical to NETWALL (docs/DEPLOY.md there, section 6): only the proxy needs to reach port 8080. curl -i http://127.0.0.1:8080/healthz and curl -gi 'http://[::1]:8080/healthz' check both families.

6. Operating: upstream-admin

tools/admin.js is the operator CLI (Node core only, uses the game's own db.js, state.js, rules.js). deploy/upstream-admin runs it as the service user against the service's database (it reads DB_PATH from /etc/upstream/upstream.env). Do not run node tools/admin.js as root: SQLite would create game.db-wal and -shm owned by root and the game could no longer open its database (symptom in section 10).

upstream-admin help
upstream-admin status                          # file, size, migrations, overdue events
upstream-admin worlds
upstream-admin create-world --name "Summer" --speed 2 --radius 60 --seed 1234 --end-in 90d
upstream-admin end-world 1 --yes                # flag flip only, see below
upstream-admin users [--banned]                # no hashes or tokens are ever printed
upstream-admin operators [--world 1]
upstream-admin ban mallory                     # sets banned, revokes all tokens: old tokens give 401, login 403
upstream-admin unban mallory
upstream-admin revoke-tokens ada               # log out everywhere
upstream-admin inspect-operator ada            # (--world N when the name exists in several worlds, --id N by operator id)
upstream-admin inspect-town 12                 # resources now (rules.resourcesAt), buildings, queue, units if the table exists
upstream-admin events --pending [--type buildDone --town 12 --limit 100]
upstream-admin replay-events --until +6h       # dry run: what the scheduler would process, in order
upstream-admin replay-events --until +6h --simulate   # + run the real handlers on a scratch copy, report outcome and follow-ups
upstream-admin backup /var/lib/upstream/backups --keep 14
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db
upstream-admin grant 12 --fiber 5000 --capital 2000 --yes          # testing only
upstream-admin reset-world 1 --yes-destroy --backup /var/lib/upstream/backups --new-seed
upstream-admin migrate                         # apply pending migrations without starting the server

All commands take --json (machine output) and --db FILE (otherwise $DB_PATH, otherwise ./data/game.db). Exit codes: 0 ok, 1 failed, 2 usage error or a destructive command missing its flag, 3 not found.

Safety rules the CLI follows:

  • Destructive or test-only commands refuse without a flag: end-world and grant need --yes; reset-world needs --yes-destroy (without it, it prints what it would delete and exits 2). ban, unban, revoke-tokens are reversible and need none.
  • end-world just sets the world's status to ended (and end_at to now): signup and join stop and every game command answers world_ended, but it writes no ranking snapshot (GET /worlds/:id/results stays available: false), cancels no pending event and closes no trade offer. The natural end (a backbone winner, then the worldEnd event two world days later) does all of that.
  • reset-world deletes the world's game rows (everything with a world_id, plus tables pointing at them: towns, buildings, events, operators, alliances, messages, ... discovered from the schema, so tables added by later stages are covered) and keeps accounts and tokens; players must join the world again. It resets start_at, clears end_at and sets the world running. It rolls back if it would leave dangling references. Stop the game first (systemctl stop upstream): the server does not know the world was swept while it is running. Use --backup DIR.
  • Everything runs on a running server's database: writes are single short BEGIN IMMEDIATE transactions, the connection is always closed, and SQLite's 5 s busy timeout covers collisions with the game (database is locked from the CLI just means: try again).
  • grant adds to the town's stored amounts (valid at its res_at); the lazy model carries them forward. Amounts above the Rack hall cap are kept but stop growing, like a lowered cap.
  • inspect-town shows values as of now ignoring overdue events; it says so when there are any (server stopped).

7. Backups and restore

upstream-backup.timer runs upstream-backup.service daily at 04:15 (10 min jitter, Persistent=true). It runs tools/admin.js backup /var/lib/upstream/backups --keep 14 as the service user. A snapshot is:

  1. VACUUM INTO a temporary .partial-* file: a consistent copy as of one moment, safe while the game runs (WAL readers do not block the writer; the game keeps its own 5 s busy timeout);
  2. verified: opened read-only, PRAGMA integrity_check = ok, worlds and schema_migrations present;
  3. published under a dated name upstream-YYYYMMDDTHHMMSSZ.db (UTC) with a hard link, so an existing file is never replaced (a same-second collision becomes ...Z-1.db), mode 0600;
  4. only then older snapshots are pruned: --keep N keeps the newest N, --keep-days D keeps those younger than D days, both = survive if either keeps it, neither = no pruning. The newest snapshot and any file not matching the naming pattern are never deleted. A failed verification leaves nothing behind and prunes nothing, and the unit fails (systemctl status upstream-backup, journalctl -u upstream-backup).

Snapshots are plain SQLite files (journal mode delete), not -wal dependent: copy a single file.

systemctl list-timers upstream-backup.timer      # next run
systemctl start upstream-backup                  # one right now
journalctl -u upstream-backup -n 20
ls -l /var/lib/upstream/backups

Change the schedule or retention with systemctl edit upstream-backup.timer / upstream-backup.service (for every 6 hours: OnCalendar=*-*-* 00/6:15:00 and a larger --keep). Before an upgrade or a reset-world, take one by hand.

Off-box copy

Snapshots on the same disk die with the container. Pull them from another machine (NAS, workstation, the Proxmox host), for example from cron or a timer there:

rsync -a --ignore-existing root@<lxc>:/var/lib/upstream/backups/ /srv/backups/upstream/

--ignore-existing makes it append-only on your side (nothing already stored is overwritten, and snapshots pruned on the LXC stay on the backup host until you decide otherwise). Better than a root key: a dedicated key restricted in authorized_keys to command="rrsync -ro /var/lib/upstream/backups",restrict ... (rrsync comes with the rsync package; not run here). The files are owner-only (upstream), so the reading user needs root or a group ACL.

Restore

The restore is a file copy, and the time model makes it clean: the restored world simply fast-forwards. Every event that fell due since the snapshot is processed in order at start-up, each at its own due time. What is lost is what players did after the snapshot (builds started, signups, messages) and their tokens: players log in again.

systemctl stop upstream-backup.timer upstream
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db     # exit 0 or stop here
mkdir -p /var/lib/upstream/pre-restore
mv /var/lib/upstream/game.db* /var/lib/upstream/pre-restore/ 2>/dev/null                # keep the old one, incl. -wal/-shm
install -o upstream -g upstream -m 0600 /var/lib/upstream/backups/upstream-20261011T041500Z.db /var/lib/upstream/game.db
systemctl start upstream upstream-backup.timer
journalctl -u upstream -n 20          # look for "caught up N overdue event(s)"
upstream-admin status

Restoring onto another machine works the same way (install code and Node, copy the file to DB_PATH). Migrations in the snapshot are checksummed: if the code is newer than the snapshot, the missing migrations are applied at boot; run code that is older than the snapshot's last migration (upstream-admin status shows it) only if you know the schema still fits, the server does not check for that.

8. Upgrading

upstream-admin backup /var/lib/upstream/backups --keep 14         # 1. snapshot first
rsync -a --delete --exclude node_modules --exclude .git --exclude data ./ root@<lxc>:/opt/upstream/    # 2. new code (from your workstation)
cd /opt/upstream && npm ci --omit=dev && chown -R root:root /opt/upstream                               # 3. dependencies, ownership
systemctl restart upstream                                                                              # 4. migrations apply at boot
journalctl -u upstream -n 30                                      # 5. "applied migrations: ..." then the listen banner
curl -i http://127.0.0.1:8080/healthz
  • Migrations are server/migrations/NNN_name.sql, applied in order inside a transaction each, recorded with a checksum and never edited after they shipped. If one that was applied has changed, the server refuses to start (migration 001_init.sql was edited after it was applied): restore that file from git and ship a new numbered migration instead. upstream-admin status lists applied and pending migrations; upstream-admin migrate applies them without starting the game.
  • Each migration runs in its own transaction: a failing one rolls back and the server exits, earlier ones of the same boot stay applied. Going back to the old code after a migration committed means restoring the pre-upgrade snapshot (section 7).
  • Updated unit files: install -m 0644 ..., systemctl daemon-reload, systemctl restart upstream.
  • Players reconnecting after the restart need no action; clients get serverTime in every response and refetch.

9. Health, logs, checks

curl -i https://upstream.example.com/healthz          # 200 ok (only says the process answers HTTP)
curl -I https://upstream.example.com/                 # 200 + Content-Security-Policy
upstream-admin status                                 # database, migrations, events pending / OVERDUE
journalctl -u upstream -f                             # live log; -p err for failures only
journalctl -u upstream --since "1 hour ago" | grep -E "failed|refused"

The log has: applied migrations, created world, caught-up events, refused rate_limited(...) ip=..., and event N (type) failed: ... for a handler that threw (that event is marked failed and the queue moves on; list them with upstream-admin events --status failed). Player IP addresses appear in the log; passwords and tokens never do. A healthy server has no overdue events in status: the timer processes events within milliseconds of their due time.

10. Troubleshooting

Symptom Cause and fix
Service dies at start: No such built-in module: node:sqlite Node is older than 22.13, or the distro's Node 20 is first on the PATH. node --version, which -a node, section 1.
ExperimentalWarning: SQLite is an experimental feature in the journal Harmless. Printed once per start on some Node versions.
SQLITE_CANTOPEN / unable to open database file DB_PATH's directory is not writable by upstream: wrong path in upstream.env, /var/lib/upstream created by hand with other ownership (chown -R upstream:upstream /var/lib/upstream), or ProtectSystem=strict enabled without ReadWritePaths=/var/lib/upstream.
attempt to write a readonly database / unable to open database file right after using the CLI admin.js was run as root and left root-owned game.db-wal / -shm. Stop the game, chown upstream:upstream /var/lib/upstream/game.db*, start. Use upstream-admin, which switches user.
migration NNN_x.sql was edited after it was applied An applied migration changed on disk (checksum mismatch). Restore it from git; changes go in a new migration.
UPSTREAM failed to start: ... with EADDRINUSE Another process on the port (ss -ltnp | grep 8080) or two copies. Change PORT and the NPM forward port together.
IPv6 is not available (...); falling back to 0.0.0.0 The kernel has no IPv6. IPv4 keeps working; set HOST=0.0.0.0 to silence it.
status=226/NAMESPACE The optional namespace hardening is on in an LXC without Nesting: enable Options > Features > Nesting and restart the CT, or comment the block out.
Cannot find module 'ws' npm ci --omit=dev was not run in /opt/upstream.
502 Bad Gateway from NPM Upstream unreachable: systemctl status upstream, curl -i http://<lxc>:8080/healthz from the NPM host, forward IP/port, firewall.
Page loads but a new deploy still shows the old client NPM Cache Assets is on, or a service layer ignores no-cache. Switch it off, hard-reload.
Signup answers 403 signup_closed SIGNUP_CODE is set. Send {"invite": "<code>"} in the signup body, or unset it and restart.
Login/signup answers 429 rate_limited for many players They all arrive from one address: the proxy is not trusted so the per-IP budget is shared (log says ip=<the proxy>). Set TRUSTED_PROXIES. Login and signup allow about one request per 10 s per address, burst 8; a whole classroom behind one NAT can hit that.
Game seems frozen: builds never finish upstream-admin status: OVERDUE events mean the scheduler is not running (server stopped or stuck) or handlers fail: upstream-admin events --status failed, journal event N ... failed. A restart catches up everything in order.
Banned user still plays They cannot: ban revokes every token and blocks login. Check the username spelling (upstream-admin users --banned).
database is locked from the CLI The game held a write lock longer than the 5 s busy timeout. Retry.
upstream-backup.service failed journalctl -u upstream-backup. Typical: disk full, or the integrity check of the new snapshot failed (then nothing was kept and nothing pruned; check the disk and the game.db itself).
Backups not appearing systemctl list-timers upstream-backup.timer; the timer must be enabled (systemctl enable --now upstream-backup.timer), not just installed.
World reset by mistake Restore the snapshot taken before (section 7), or the one reset-world --backup wrote.
WebSocket upgrade answered 426 / 403 / 429 426: Websockets Support off in NPM. 403: origin check, see ALLOWED_ORIGINS and the notes in section 5. 429: more than MAX_CONN_PER_IP sockets from one address.