Built: e2e/run.js + lib.js (playwright-core, in-process server, temp DB): desktop flow, phone touch portrait/landscape, layout matrix on 5 viewports x pop/map/trade/world (overflow + console errors), WebSocket push incoming-banner without reload. 61/61 checks, ~54 s, identical over 4 runs; the push and overflow checks were shown to fail when the behaviour is removed. Docs: fixed stale numbers/claims in DESIGN, DECISIONS, BALANCE, API (undocumented routes, rate limits, notification types), DEPLOY, NEXT (status-at-a-glance on top). Deploy path exercised in a scratch dir: migrations, healthz, signup/build, admin CLI, SIGKILL catch-up, backup restore. Tested: npm test 207/207, npm run e2e 61/61, bot run 50x14d INVARIANTS OK. NOT tested: Chromium only; no systemd/nginx Proxy Manager/Node 22.13 run; e2e does not cover units, alliances, messages, trade or end-game flows beyond empty-state rendering; no nb, PWA or polling-fallback e2e; docs/workflows/ still empty. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q2MJaQ5LXAHzGnTDgceT14
29 KiB
Deploying UPSTREAM
UPSTREAM is one Node process: it serves the static client from public/, the REST API under /api/v1/ and a
WebSocket push channel at /ws (docs/PROTOCOL.md). Unlike NETWALL it keeps state: one SQLite file, owned by that process. A restart loses
nothing: overdue events are caught up in order before the server accepts traffic.
phone / laptop --https--> nginx Proxy Manager (TLS, :443) --http (+ WebSocket)--> Debian 13 LXC: node :8080
/opt/upstream (code, read-only)
/var/lib/upstream (game.db, backups/)
Same shape as NETWALL (docs/DEPLOY.md there): same proxy, same hardening notes, same env vars for the network
layer. The differences are the Node version (node:sqlite needs 22.13+), a state directory and backups.
What was and was not run while writing this (so you know what to check on the real box):
- Run:
systemd-analyze verifyon the three unit files (syntax and directives only), the CLItools/admin.jsand its tests on a dev machine (Node 24.21), a backup, verify and prune cycle,sh -nondeploy/upstream-admin. Re-checked on 2026-10-11 with the code as it is now (Node 24.21, scratch directory, no systemd): first boot withDB_PATH/PORT/HOST(migrations 001-004 applied,created world 1),/healthz,/headers, signup + build over the API,admin.jsstatus / worlds / users / operators / inspect-town / events / replay-events / backup / verify-backup / ban / unban / grant / end-world (refusal without--yes), SIGTERM and restart (state, queue and password kept, no migrations re-applied; tokens revoked bybanstay revoked), SIGKILL then restart after a build fell due (caught up 1 overdue event(s), the build level was applied), and a restore by copying a snapshot to a freshDB_PATHand starting on it (section 7: it loggedcaught up 1 overdue event(s)). - Taken from NETWALL's docs, not re-run for UPSTREAM: everything about nginx Proxy Manager (labels, header behaviour, "Client addresses"). NETWALL's guide records what was observed against NPM v2.16.0.
- Never run: a Proxmox LXC, a started unit under systemd, the timer firing on its own, a Node install from the
tarball or NodeSource on Debian 13,
rrsync, Let's Encrypt, a restore on the real box. Treat those steps as a checklist to confirm, and fix this file when reality differs.
1. Requirements
- Debian 13 (trixie) LXC with its own IPv4 (optionally IPv6) address the proxy can reach. 1 core, 512 MB RAM and 4 GB
disk are plenty for a small world; the database grows with the event and notification history (measure it with
upstream-admin status), and snapshots multiply it by the retention count. - Node 22.13 or newer. Debian 13 ships
nodejs20.19.2 (checked on tracker.debian.org: stable has 20.19.2, testing and unstable have 24.21). Node 20 has nonode:sqlite, soapt install nodejsis not enough.package.jsonsays"engines": {"node": ">=22.13"}; 22.5 to 22.12 only havenode:sqlitebehind--experimental-sqlite, which the unit does not pass. Development is on Node 24.21. - A DNS name for the game (for example
upstream.example.com) pointing at the proxy.
Getting a suitable Node (pick one)
A. Official tarball (no third-party apt repo, nothing piped into a shell). Pick a current 24.x (Active LTS) or
22.x from https://nodejs.org/en/download and put its version in V:
apt update && apt install -y curl xz-utils ca-certificates
V=24.21.0 # example: use the current 24.x (or >= 22.13) you picked
ARCH=linux-x64 # linux-arm64 on ARM hosts (uname -m: aarch64)
cd /tmp
curl -fsSLO https://nodejs.org/dist/v$V/node-v$V-$ARCH.tar.xz
curl -fsSLO https://nodejs.org/dist/v$V/SHASUMS256.txt
grep " node-v$V-$ARCH.tar.xz\$" SHASUMS256.txt | sha256sum -c - # must print: ... OK
tar -C /opt -xJf node-v$V-$ARCH.tar.xz
ln -sfn /opt/node-v$V-$ARCH /opt/node
ln -sf /opt/node/bin/node /usr/local/bin/node
ln -sf /opt/node/bin/npm /usr/local/bin/npm
ln -sf /opt/node/bin/npx /usr/local/bin/npx
(For extra assurance also verify the signature of SHASUMS256.txt as described on nodejs.org.) An upgrade later is
a new tarball and ln -sfn to it, then systemctl restart upstream. upstream.service starts /usr/bin/env node, and
systemd's default PATH lists /usr/local/bin before /usr/bin, so the tarball wins over any distro Node.
B. NodeSource apt repository (follows their documentation; not run here). It puts Node in /usr/bin/node:
apt install -y ca-certificates curl gnupg
install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_24.x nodistro main" > /etc/apt/sources.list.d/nodesource.list
apt update && apt install -y nodejs # includes npm; do not also install Debian's npm package
Verify either way (all three must succeed):
node --version # v22.13.0 or newer
node -e "require('node:sqlite'); console.log('node:sqlite ok')"
which -a node # the first one is the one the unit will use
On 22.x/24.x Node may print ExperimentalWarning: SQLite is an experimental feature to stderr; that is harmless (it
shows up in the journal once per start). No such built-in module: node:sqlite means Node is too old (or it is the
distro's 20.x earlier on the PATH). If node:sqlite ever misbehaves, DECISIONS.md D1 describes the one-file swap
to better-sqlite3 in server/db.js; that would add a native dependency and a build toolchain to the LXC.
2. Create the LXC (Proxmox)
Container creation, the Proxmox UI way (CLI pct create works too):
- Template
debian-13-standard, unprivileged, hostnameupstream, 1 core, 512 MB RAM (+256 MB swap), 4-8 GB disk. - Network: a static address (or a DHCP reservation) on a bridge the NPM host can reach. The proxy forwards to that address, so it must not change.
- Features: Nesting is optional; tick it only if you want the namespace-based hardening block in the unit (section 3). Everything else works without it.
- Start on boot: yes. Timezone does not matter: the game uses UTC ms, the backup timer uses the container's clock.
- Proxmox's own CT backups (
vzdump) are not a substitute for the snapshots in section 7: a running-container copy of a SQLite file is crash-consistent at best. Use them for the OS and let the verified snapshots be the data backup.
3. Install in the LXC
Run as root inside the container (Node from section 1 is already in place).
apt install -y rsync # for the copy below (or use git)
# a dedicated, login-less user that only runs the game
useradd --system --home-dir /var/lib/upstream --shell /usr/sbin/nologin upstream
# code: read-only for the service user, owned by root
mkdir -p /opt/upstream
# from your workstation:
# rsync -a --delete --exclude node_modules --exclude .git --exclude data --exclude e2e ./ root@<lxc>:/opt/upstream/
# (needs server/, public/, tools/, deploy/, docs/, package.json, package-lock.json; `test/` is optional)
cd /opt/upstream
npm ci --omit=dev # only "ws"; follows the lockfile exactly
chown -R root:root /opt/upstream
# the admin helper on the PATH
install -m 0755 /opt/upstream/deploy/upstream-admin /usr/local/bin/upstream-admin
# optional settings / secrets file (read by the unit and by upstream-admin)
install -d -m 0750 -o root -g upstream /etc/upstream
install -m 0640 -o root -g upstream /dev/null /etc/upstream/upstream.env
# edit it: SIGNUP_CODE=choose-a-long-random-string
# ALLOWED_ORIGINS=https://upstream.example.com
# the units
install -m 0644 /opt/upstream/deploy/upstream.service /opt/upstream/deploy/upstream-backup.service /opt/upstream/deploy/upstream-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now upstream upstream-backup.timer
systemctl status upstream
journalctl -u upstream -f
The first start prints applied migrations: 001_init.sql, ..., created world 1 and the listen banner with
database: /var/lib/upstream/game.db. Quick manual run as the service user instead of systemd (Ctrl-C to stop):
runuser -u upstream -- env PORT=8080 DB_PATH=/var/lib/upstream/game.db node /opt/upstream/server/index.js.
What the units do
deploy/upstream.service: userupstream,WorkingDirectory=/opt/upstream,StateDirectory=upstream(systemd creates/var/lib/upstreamowned by that user, mode 0700, so the game can write nowhere else),Restart=on-failure,EnvironmentFile=-/etc/upstream/upstream.env(overrides theEnvironment=lines),UMask=0077(database and snapshots hold password hashes: owner-only).- Hardening that works in any container:
NoNewPrivileges, empty capability set (port 8080 needs none),RestrictAddressFamilies,RestrictRealtime,RestrictSUIDSGID,LockPersonality,SystemCallArchitectures. NoMemoryDenyWriteExecute(V8 would crash). - The namespace block (
ProtectSystem=strict,ReadWritePaths=/var/lib/upstream,PrivateTmp, ...) is commented out. In an unprivileged LXC it fails withstatus=226/NAMESPACEunless Options > Features > Nesting is on. Because the app writes to disk,ProtectSystem=strictneeds theReadWritePaths=line that is already next to it in the unit; without it the database cannot be opened. - Shutdown: SIGTERM stops the scheduler timer, closes the listener and the database and exits 0 in about a second. Even a SIGKILL or power cut is survivable (WAL replay on the next open, then overdue events are caught up).
deploy/upstream-backup.service(oneshot) and.timer: see section 7.
4. Configuration
Environment variables read by node server/index.js. Put changes in /etc/upstream/upstream.env (KEY=value, one per
line, no export) or in a systemctl edit upstream override, then systemctl restart upstream.
| Variable | Default (unit) | Meaning |
|---|---|---|
PORT |
8080 |
TCP port. Unprivileged, so no capabilities needed. 0 picks a free one (banner prints it). |
HOST |
:: |
Bind address. :: = one dual-stack socket for IPv4 and IPv6 (falls back to 0.0.0.0 if the kernel has no IPv6). 127.0.0.1 = this container only. |
DB_PATH |
/var/lib/upstream/game.db |
SQLite file. The code default is ./data/game.db; the unit overrides it. Its directory is created if missing, it must be writable by the service user. -wal and -shm files appear next to it: always copy via a snapshot (section 7), never with cp while running. |
TRUST_PROXY |
auto |
auto: believe X-Forwarded-For / X-Forwarded-Host only from loopback / private-range peers. 1: always (then firewall the port). 0: never. Used for the per-IP rate limits and the log. |
TRUSTED_PROXIES |
empty | Extra IPs / CIDRs auto believes: needed when the proxy reaches the LXC from a global IPv6, Tailscale / CGNAT 100.64.0.0/10 or another routable address. |
ALLOWED_ORIGINS |
empty | Extra origins for the WebSocket origin check, comma separated, with the port when it is not 80/443. The REST API uses bearer tokens (no cookies) and does not check origins. |
MAX_CONN_PER_IP |
40 |
Open WebSockets per client address (HTTP connections are not capped). 0 disables. |
WORLD_SPEED |
unset (1) | World speed multiplier, first boot only, when the database has no world yet. |
WORLD_RADIUS |
unset (50) | Map radius in tiles (coordinates -R..R, so 50 = 101 x 101), first boot only. |
WORLD_NAME |
unset (World 1) |
Name of the first world, first boot only. |
SIGNUP_CODE |
empty | If set, POST /auth/signup needs "invite": "<this>" in the body or answers signup_closed (403). Empty = open signup. Keep it in /etc/upstream/upstream.env, not in the unit file. |
UPSTREAM_DIR, UPSTREAM_ENV_FILE |
/opt/upstream, /etc/upstream/upstream.env |
Only for the upstream-admin wrapper. |
Fixed in code (options of createServer()): per-IP and per-token rate limits (login and signup are stricter: about one
per 10 s sustained, burst 8, per client address), 32 KB request bodies, idempotency keys pruned hourly.
First boot
An empty or missing database is created, all migrations are applied, and one world is created from WORLD_SPEED,
WORLD_RADIUS, WORLD_NAME (seed random). After that those variables are ignored: changing them does not touch the
existing world. More worlds, a different seed, ending or resetting a world are admin tasks (section 6). The first
player to sign up becomes an ordinary operator; there is no special admin account, admin work is the CLI on the box.
5. nginx Proxy Manager
In NPM: Hosts > Proxy Hosts > Add Proxy Host.
| Tab / field | Value |
|---|---|
| Details: Domain Names | upstream.example.com |
| Details: Scheme | http |
| Details: Forward Hostname / IP | the LXC's address (for example 192.168.1.51), not 127.0.0.1 when NPM runs elsewhere or in its own container |
| Details: Forward Port | 8080 |
| Details: Cache Assets | off (static files are no-cache and revalidated by ETag; the game must see new versions) |
| Details: Block Common Exploits | optional |
| Details: Websockets Support | on. The REST game works without it, but /ws (live push) fails with HTTP 426 when this is off. |
| Custom Locations | none needed |
| SSL | Request a new Let's Encrypt certificate, Force SSL, HTTP/2 (HSTS optional) |
| Advanced (gear) | nothing required; optional gzip below |
Optional, Advanced box, to compress JS/JSON for phones (the Node server does not compress; NPM's own gzip on only
covers HTML):
gzip on;
gzip_types text/css text/javascript application/javascript application/json application/manifest+json image/svg+xml;
Notes carried over from NETWALL's setup (observed there against NPM v2.16.0, not re-run for UPSTREAM):
- NPM sends
Hostwithout the public port and noX-Forwarded-Host. On a public port other than 443 the browserOriginthen differs fromHost; list the origin with its port inALLOWED_ORIGINS(matters for/ws). - A proxy on a non-private address (global IPv6, Tailscale) is not believed by
TRUST_PROXY=auto: every player then shares one rate-limit budget and the log shows the proxy's address. Fix:TRUSTED_PROXIES=<its address or CIDR>. - Clients on your own LAN can pick their address through NPM's
real_iphandling, so the per-IP limits protect against public abuse, not against LAN users. - Idle timeout: nginx's
proxy_read_timeoutis 60 s by default (NETWALL's notes say 90 s for NPM; not re-checked). NETWALL never idles because it streams 20 snapshots a second; UPSTREAM's socket is quiet for long stretches.server/push.jssends a protocol ping every 30 s (wsPingMs), below either value, which is reasoning, not a measurement behind a real NPM.
Firewall and dual stack: identical to NETWALL (docs/DEPLOY.md there, section 6): only the proxy needs to reach port
8080. curl -i http://127.0.0.1:8080/healthz and curl -gi 'http://[::1]:8080/healthz' check both families.
6. Operating: upstream-admin
tools/admin.js is the operator CLI (Node core only, uses the game's own db.js, state.js, rules.js).
deploy/upstream-admin runs it as the service user against the service's database (it reads DB_PATH from
/etc/upstream/upstream.env). Do not run node tools/admin.js as root: SQLite would create game.db-wal and
-shm owned by root and the game could no longer open its database (symptom in section 10).
upstream-admin help
upstream-admin status # file, size, migrations, overdue events
upstream-admin worlds
upstream-admin create-world --name "Summer" --speed 2 --radius 60 --seed 1234 --end-in 90d
upstream-admin end-world 1 --yes # flag flip only, see below
upstream-admin users [--banned] # no hashes or tokens are ever printed
upstream-admin operators [--world 1]
upstream-admin ban mallory # sets banned, revokes all tokens: old tokens give 401, login 403
upstream-admin unban mallory
upstream-admin revoke-tokens ada # log out everywhere
upstream-admin inspect-operator ada # (--world N when the name exists in several worlds, --id N by operator id)
upstream-admin inspect-town 12 # resources now (rules.resourcesAt), buildings, queue, units if the table exists
upstream-admin events --pending [--type buildDone --town 12 --limit 100]
upstream-admin replay-events --until +6h # dry run: what the scheduler would process, in order
upstream-admin replay-events --until +6h --simulate # + run the real handlers on a scratch copy, report outcome and follow-ups
upstream-admin backup /var/lib/upstream/backups --keep 14
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db
upstream-admin grant 12 --fiber 5000 --capital 2000 --yes # testing only
upstream-admin reset-world 1 --yes-destroy --backup /var/lib/upstream/backups --new-seed
upstream-admin migrate # apply pending migrations without starting the server
All commands take --json (machine output) and --db FILE (otherwise $DB_PATH, otherwise ./data/game.db).
Exit codes: 0 ok, 1 failed, 2 usage error or a destructive command missing its flag, 3 not found.
Safety rules the CLI follows:
- Destructive or test-only commands refuse without a flag:
end-worldandgrantneed--yes;reset-worldneeds--yes-destroy(without it, it prints what it would delete and exits 2).ban,unban,revoke-tokensare reversible and need none. end-worldjust sets the world's status toended(andend_atto now): signup and join stop and every game command answersworld_ended, but it writes no ranking snapshot (GET /worlds/:id/resultsstaysavailable: false), cancels no pending event and closes no trade offer. The natural end (a backbone winner, then theworldEndevent two world days later) does all of that.reset-worlddeletes the world's game rows (everything with aworld_id, plus tables pointing at them: towns, buildings, events, operators, alliances, messages, ... discovered from the schema, so tables added by later stages are covered) and keeps accounts and tokens; players must join the world again. It resetsstart_at, clearsend_atand sets the world running. It rolls back if it would leave dangling references. Stop the game first (systemctl stop upstream): the server does not know the world was swept while it is running. Use--backup DIR.- Everything runs on a running server's database: writes are single short
BEGIN IMMEDIATEtransactions, the connection is always closed, and SQLite's 5 s busy timeout covers collisions with the game (database is lockedfrom the CLI just means: try again). grantadds to the town's stored amounts (valid at itsres_at); the lazy model carries them forward. Amounts above the Rack hall cap are kept but stop growing, like a lowered cap.inspect-townshows values as of now ignoring overdue events; it says so when there are any (server stopped).
7. Backups and restore
upstream-backup.timer runs upstream-backup.service daily at 04:15 (10 min jitter, Persistent=true). It runs
tools/admin.js backup /var/lib/upstream/backups --keep 14 as the service user. A snapshot is:
VACUUM INTOa temporary.partial-*file: a consistent copy as of one moment, safe while the game runs (WAL readers do not block the writer; the game keeps its own 5 s busy timeout);- verified: opened read-only,
PRAGMA integrity_check=ok,worldsandschema_migrationspresent; - published under a dated name
upstream-YYYYMMDDTHHMMSSZ.db(UTC) with a hard link, so an existing file is never replaced (a same-second collision becomes...Z-1.db), mode 0600; - only then older snapshots are pruned:
--keep Nkeeps the newest N,--keep-days Dkeeps those younger than D days, both = survive if either keeps it, neither = no pruning. The newest snapshot and any file not matching the naming pattern are never deleted. A failed verification leaves nothing behind and prunes nothing, and the unit fails (systemctl status upstream-backup,journalctl -u upstream-backup).
Snapshots are plain SQLite files (journal mode delete), not -wal dependent: copy a single file.
systemctl list-timers upstream-backup.timer # next run
systemctl start upstream-backup # one right now
journalctl -u upstream-backup -n 20
ls -l /var/lib/upstream/backups
Change the schedule or retention with systemctl edit upstream-backup.timer / upstream-backup.service (for every
6 hours: OnCalendar=*-*-* 00/6:15:00 and a larger --keep). Before an upgrade or a reset-world, take one by hand.
Off-box copy
Snapshots on the same disk die with the container. Pull them from another machine (NAS, workstation, the Proxmox host), for example from cron or a timer there:
rsync -a --ignore-existing root@<lxc>:/var/lib/upstream/backups/ /srv/backups/upstream/
--ignore-existing makes it append-only on your side (nothing already stored is overwritten, and snapshots pruned on
the LXC stay on the backup host until you decide otherwise). Better than a root key: a dedicated key restricted in
authorized_keys to command="rrsync -ro /var/lib/upstream/backups",restrict ... (rrsync comes with the rsync
package; not run here). The files are owner-only (upstream), so the reading user needs root or a group ACL.
Restore
The restore is a file copy, and the time model makes it clean: the restored world simply fast-forwards. Every event that fell due since the snapshot is processed in order at start-up, each at its own due time. What is lost is what players did after the snapshot (builds started, signups, messages) and their tokens: players log in again.
systemctl stop upstream-backup.timer upstream
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db # exit 0 or stop here
mkdir -p /var/lib/upstream/pre-restore
mv /var/lib/upstream/game.db* /var/lib/upstream/pre-restore/ 2>/dev/null # keep the old one, incl. -wal/-shm
install -o upstream -g upstream -m 0600 /var/lib/upstream/backups/upstream-20261011T041500Z.db /var/lib/upstream/game.db
systemctl start upstream upstream-backup.timer
journalctl -u upstream -n 20 # look for "caught up N overdue event(s)"
upstream-admin status
Restoring onto another machine works the same way (install code and Node, copy the file to DB_PATH). Migrations in
the snapshot are checksummed: if the code is newer than the snapshot, the missing migrations are applied at boot; run
code that is older than the snapshot's last migration (upstream-admin status shows it) only if you know the schema
still fits, the server does not check for that.
8. Upgrading
upstream-admin backup /var/lib/upstream/backups --keep 14 # 1. snapshot first
rsync -a --delete --exclude node_modules --exclude .git --exclude data ./ root@<lxc>:/opt/upstream/ # 2. new code (from your workstation)
cd /opt/upstream && npm ci --omit=dev && chown -R root:root /opt/upstream # 3. dependencies, ownership
systemctl restart upstream # 4. migrations apply at boot
journalctl -u upstream -n 30 # 5. "applied migrations: ..." then the listen banner
curl -i http://127.0.0.1:8080/healthz
- Migrations are
server/migrations/NNN_name.sql, applied in order inside a transaction each, recorded with a checksum and never edited after they shipped. If one that was applied has changed, the server refuses to start (migration 001_init.sql was edited after it was applied): restore that file from git and ship a new numbered migration instead.upstream-admin statuslists applied and pending migrations;upstream-admin migrateapplies them without starting the game. - Each migration runs in its own transaction: a failing one rolls back and the server exits, earlier ones of the same boot stay applied. Going back to the old code after a migration committed means restoring the pre-upgrade snapshot (section 7).
- Updated unit files:
install -m 0644 ...,systemctl daemon-reload,systemctl restart upstream. - Players reconnecting after the restart need no action; clients get
serverTimein every response and refetch.
9. Health, logs, checks
curl -i https://upstream.example.com/healthz # 200 ok (only says the process answers HTTP)
curl -I https://upstream.example.com/ # 200 + Content-Security-Policy
upstream-admin status # database, migrations, events pending / OVERDUE
journalctl -u upstream -f # live log; -p err for failures only
journalctl -u upstream --since "1 hour ago" | grep -E "failed|refused"
The log has: applied migrations, created world, caught-up events, refused rate_limited(...) ip=..., and
event N (type) failed: ... for a handler that threw (that event is marked failed and the queue moves on; list them
with upstream-admin events --status failed). Player IP addresses appear in the log; passwords and tokens never do.
A healthy server has no overdue events in status: the timer processes events within milliseconds of their due time.
10. Troubleshooting
| Symptom | Cause and fix |
|---|---|
Service dies at start: No such built-in module: node:sqlite |
Node is older than 22.13, or the distro's Node 20 is first on the PATH. node --version, which -a node, section 1. |
ExperimentalWarning: SQLite is an experimental feature in the journal |
Harmless. Printed once per start on some Node versions. |
SQLITE_CANTOPEN / unable to open database file |
DB_PATH's directory is not writable by upstream: wrong path in upstream.env, /var/lib/upstream created by hand with other ownership (chown -R upstream:upstream /var/lib/upstream), or ProtectSystem=strict enabled without ReadWritePaths=/var/lib/upstream. |
attempt to write a readonly database / unable to open database file right after using the CLI |
admin.js was run as root and left root-owned game.db-wal / -shm. Stop the game, chown upstream:upstream /var/lib/upstream/game.db*, start. Use upstream-admin, which switches user. |
migration NNN_x.sql was edited after it was applied |
An applied migration changed on disk (checksum mismatch). Restore it from git; changes go in a new migration. |
UPSTREAM failed to start: ... with EADDRINUSE |
Another process on the port (ss -ltnp | grep 8080) or two copies. Change PORT and the NPM forward port together. |
IPv6 is not available (...); falling back to 0.0.0.0 |
The kernel has no IPv6. IPv4 keeps working; set HOST=0.0.0.0 to silence it. |
status=226/NAMESPACE |
The optional namespace hardening is on in an LXC without Nesting: enable Options > Features > Nesting and restart the CT, or comment the block out. |
Cannot find module 'ws' |
npm ci --omit=dev was not run in /opt/upstream. |
| 502 Bad Gateway from NPM | Upstream unreachable: systemctl status upstream, curl -i http://<lxc>:8080/healthz from the NPM host, forward IP/port, firewall. |
| Page loads but a new deploy still shows the old client | NPM Cache Assets is on, or a service layer ignores no-cache. Switch it off, hard-reload. |
Signup answers 403 signup_closed |
SIGNUP_CODE is set. Send {"invite": "<code>"} in the signup body, or unset it and restart. |
Login/signup answers 429 rate_limited for many players |
They all arrive from one address: the proxy is not trusted so the per-IP budget is shared (log says ip=<the proxy>). Set TRUSTED_PROXIES. Login and signup allow about one request per 10 s per address, burst 8; a whole classroom behind one NAT can hit that. |
| Game seems frozen: builds never finish | upstream-admin status: OVERDUE events mean the scheduler is not running (server stopped or stuck) or handlers fail: upstream-admin events --status failed, journal event N ... failed. A restart catches up everything in order. |
| Banned user still plays | They cannot: ban revokes every token and blocks login. Check the username spelling (upstream-admin users --banned). |
database is locked from the CLI |
The game held a write lock longer than the 5 s busy timeout. Retry. |
upstream-backup.service failed |
journalctl -u upstream-backup. Typical: disk full, or the integrity check of the new snapshot failed (then nothing was kept and nothing pruned; check the disk and the game.db itself). |
| Backups not appearing | systemctl list-timers upstream-backup.timer; the timer must be enabled (systemctl enable --now upstream-backup.timer), not just installed. |
| World reset by mistake | Restore the snapshot taken before (section 7), or the one reset-world --backup wrote. |
| WebSocket upgrade answered 426 / 403 / 429 | 426: Websockets Support off in NPM. 403: origin check, see ALLOWED_ORIGINS and the notes in section 5. 429: more than MAX_CONN_PER_IP sockets from one address. |