Upstream/docs/DEPLOY.md
jvanvik 3c23c51902 Phase 4: close loot/refund leaks, landing-site garrisons, speed-scaled shipping, update/offsite tooling, broad e2e
Built:
- Economy: raid loot capped to the room the raiders have at home (rest stays with the victim, "lootLeft" in the report); trade escrow reserves its room so refunds always fit; trade accept refuses with no_room_for_goods when the goods would not fit. Loot destroyed per 30-day run 31-61k -> ~1k, refunds clipped ~27k -> 0 (docs/BALANCE.md, 3 seeds).
- End game: members of the holding alliance garrison landing sites (migration 005_site_garrison.sql), garrison counts in site defence, 72 h capture shield, withdraw route, POP view lists garrisoned units. Shipping capacity scales with world speed (3x world: winner 76-98 -> ~30-36 days after sites open). Bot runs show real contests (84-194 attacks on held sites at 1x).
- Ops: deploy/update.sh (backup, stop, ff-merge, npm ci --ignore-scripts, health check, rollback), deploy/upstream-offsite (rsync only), admin end-world snapshots results, EPIPE fix, docs/workflows/ filled in.
- e2e 62 -> 163 checks: units, raids/reports, alliance+messages, trade, end game, nb, PWA, reconnect, slow polling fallback (opt-in). Two app bugs found: message push ignored, "New" button stayed English.

Tested: npm test 274/274, openapi check, npm run e2e 163/163 (137 s), bot runs 50x30d seeds 1-3 and 140-day/60-day speed-3 end-game runs INVARIANTS OK, adversarial review pass (findings fixed or documented).
NOT tested: update.sh on a real box (systemd, real npm, runuser; stubs only, shellcheck not installed), garrison card/loot text on a phone, Node 22.13, humans. Known leaks left open and documented (BALANCE S6): loot/shipments can still overflow when stock arrives in flight, bounced backbone shipments lose overflow (reported to the sender). Garrison numbers (heldDefenceMult 2, shield 72 h) are first guesses from 2 seeds. Not claimed: balanced.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2MJaQ5LXAHzGnTDgceT14
2026-10-12 00:58:12 +02:00

42 KiB

Deploying UPSTREAM

UPSTREAM is one Node process: it serves the static client from public/, the REST API under /api/v1/ and a WebSocket push channel at /ws (docs/PROTOCOL.md). Unlike NETWALL it keeps state: one SQLite file, owned by that process. A restart loses nothing: overdue events are caught up in order before the server accepts traffic.

 phone / laptop --https--> nginx Proxy Manager (TLS, :443) --http (+ WebSocket)--> Debian 13 LXC: node :8080
                                                                                    /opt/upstream      (code, read-only)
                                                                                    /var/lib/upstream  (game.db, backups/)

Same shape as NETWALL (docs/DEPLOY.md there): same proxy, same hardening notes, same env vars for the network layer. The differences are the Node version (node:sqlite needs 22.13+), a state directory and backups.

What was and was not run while writing this (so you know what to check on the real box):

  • Run: systemd-analyze verify on the unit files (syntax and directives only; the two off-site units: only the expected "not executable" warning for a script path that does not exist on the dev machine), the CLI tools/admin.js and its tests on a dev machine (Node 24.21), a backup, verify and prune cycle, sh -n on deploy/upstream-admin. Re-checked on 2026-10-11 with the code as it is now (Node 24.21, scratch directory, no systemd): first boot with DB_PATH/PORT/HOST (migrations 001-004 applied, created world 1), /healthz, / headers, signup + build over the API, admin.js status / worlds / users / operators / inspect-town / events / replay-events / backup / verify-backup / ban / unban / grant / end-world (refusal without --yes), SIGTERM and restart (state, queue and password kept, no migrations re-applied; tokens revoked by ban stay revoked), SIGKILL then restart after a build fell due (caught up 1 overdue event(s), the build level was applied), and a restore by copying a snapshot to a fresh DB_PATH and starting on it (section 7: it logged caught up 1 overdue event(s)).
  • Run, ops scripts (dev machine, stubs, no systemd): deploy/update.sh and deploy/upstream-offsite and admin.js end-world through test/deploy-update.test.js, test/deploy-offsite.test.js, test/admin-endworld.test.js; sections 7 and 8 say exactly what each covers.
  • Taken from NETWALL's docs, not re-run for UPSTREAM: everything about nginx Proxy Manager (labels, header behaviour, "Client addresses"). NETWALL's guide records what was observed against NPM v2.16.0.
  • Never run: a Proxmox LXC, a started unit under systemd, the timer firing on its own, a Node install from the tarball or NodeSource on Debian 13, rrsync, Let's Encrypt, a restore on the real box. Treat those steps as a checklist to confirm, and fix this file when reality differs.

1. Requirements

  • Debian 13 (trixie) LXC with its own IPv4 (optionally IPv6) address the proxy can reach. 1 core, 512 MB RAM and 4 GB disk are plenty for a small world; the database grows with the event and notification history (measure it with upstream-admin status), and snapshots multiply it by the retention count.
  • Node 22.13 or newer. Debian 13 ships nodejs 20.19.2 (checked on tracker.debian.org: stable has 20.19.2, testing and unstable have 24.21). Node 20 has no node:sqlite, so apt install nodejs is not enough. package.json says "engines": {"node": ">=22.13"}; 22.5 to 22.12 only have node:sqlite behind --experimental-sqlite, which the unit does not pass. Development is on Node 24.21.
  • A DNS name for the game (for example upstream.example.com) pointing at the proxy.

Getting a suitable Node (pick one)

A. Official tarball (no third-party apt repo, nothing piped into a shell). Pick a current 24.x (Active LTS) or 22.x from https://nodejs.org/en/download and put its version in V:

apt update && apt install -y curl xz-utils ca-certificates
V=24.21.0                                  # example: use the current 24.x (or >= 22.13) you picked
ARCH=linux-x64                             # linux-arm64 on ARM hosts (uname -m: aarch64)
cd /tmp
curl -fsSLO https://nodejs.org/dist/v$V/node-v$V-$ARCH.tar.xz
curl -fsSLO https://nodejs.org/dist/v$V/SHASUMS256.txt
grep " node-v$V-$ARCH.tar.xz\$" SHASUMS256.txt | sha256sum -c -       # must print: ... OK
tar -C /opt -xJf node-v$V-$ARCH.tar.xz
ln -sfn /opt/node-v$V-$ARCH /opt/node
ln -sf /opt/node/bin/node /usr/local/bin/node
ln -sf /opt/node/bin/npm  /usr/local/bin/npm
ln -sf /opt/node/bin/npx  /usr/local/bin/npx

(For extra assurance also verify the signature of SHASUMS256.txt as described on nodejs.org.) An upgrade later is a new tarball and ln -sfn to it, then systemctl restart upstream. upstream.service starts /usr/bin/env node, and systemd's default PATH lists /usr/local/bin before /usr/bin, so the tarball wins over any distro Node.

B. NodeSource apt repository (follows their documentation; not run here). It puts Node in /usr/bin/node:

apt install -y ca-certificates curl gnupg
install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_24.x nodistro main" > /etc/apt/sources.list.d/nodesource.list
apt update && apt install -y nodejs        # includes npm; do not also install Debian's npm package

Verify either way (all three must succeed):

node --version                                        # v22.13.0 or newer
node -e "require('node:sqlite'); console.log('node:sqlite ok')"
which -a node                                         # the first one is the one the unit will use

On 22.x/24.x Node may print ExperimentalWarning: SQLite is an experimental feature to stderr; that is harmless (it shows up in the journal once per start). No such built-in module: node:sqlite means Node is too old (or it is the distro's 20.x earlier on the PATH). If node:sqlite ever misbehaves, DECISIONS.md D1 describes the one-file swap to better-sqlite3 in server/db.js; that would add a native dependency and a build toolchain to the LXC.

2. Create the LXC (Proxmox)

Container creation, the Proxmox UI way (CLI pct create works too):

  • Template debian-13-standard, unprivileged, hostname upstream, 1 core, 512 MB RAM (+256 MB swap), 4-8 GB disk.
  • Network: a static address (or a DHCP reservation) on a bridge the NPM host can reach. The proxy forwards to that address, so it must not change.
  • Features: Nesting is optional; tick it only if you want the namespace-based hardening block in the unit (section 3). Everything else works without it.
  • Start on boot: yes. Timezone does not matter: the game uses UTC ms, the backup timer uses the container's clock.
  • Proxmox's own CT backups (vzdump) are not a substitute for the snapshots in section 7: a running-container copy of a SQLite file is crash-consistent at best. Use them for the OS and let the verified snapshots be the data backup.

3. Install in the LXC

Run as root inside the container (Node from section 1 is already in place).

apt install -y rsync                               # for the copy below (or use git)

# a dedicated, login-less user that only runs the game
useradd --system --home-dir /var/lib/upstream --shell /usr/sbin/nologin upstream

# code: read-only for the service user, owned by root
mkdir -p /opt/upstream
#   from your workstation:
#   rsync -a --delete --exclude node_modules --exclude .git --exclude data --exclude e2e ./ root@<lxc>:/opt/upstream/
#   (needs server/, public/, tools/, deploy/, docs/, package.json, package-lock.json; `test/` is optional)
#   or, the way the live box is run, a git checkout that deploy/update.sh updates later (section 8):
#   git clone https://git.vanvikinternet.no/jvanvik/Upstream.git /opt/upstream   (plain HTTPS URL, no token in it)
cd /opt/upstream
npm ci --omit=dev                                   # only "ws"; follows the lockfile exactly
chown -R root:root /opt/upstream

# the admin helper on the PATH
install -m 0755 /opt/upstream/deploy/upstream-admin /usr/local/bin/upstream-admin

# optional settings / secrets file (read by the unit and by upstream-admin)
install -d -m 0750 -o root -g upstream /etc/upstream
install -m 0640 -o root -g upstream /dev/null /etc/upstream/upstream.env
#   edit it:  SIGNUP_CODE=choose-a-long-random-string
#             ALLOWED_ORIGINS=https://upstream.example.com

# the units
install -m 0644 /opt/upstream/deploy/upstream.service /opt/upstream/deploy/upstream-backup.service /opt/upstream/deploy/upstream-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now upstream upstream-backup.timer
systemctl status upstream
# off-box copy of the snapshots is optional and not installed here: section 7
journalctl -u upstream -f

The first start prints applied migrations: 001_init.sql, ..., created world 1 and the listen banner with database: /var/lib/upstream/game.db. Quick manual run as the service user instead of systemd (Ctrl-C to stop): runuser -u upstream -- env PORT=8080 DB_PATH=/var/lib/upstream/game.db node /opt/upstream/server/index.js.

What the units do

  • deploy/upstream.service: user upstream, WorkingDirectory=/opt/upstream, StateDirectory=upstream (systemd creates /var/lib/upstream owned by that user, mode 0700, so the game can write nowhere else), Restart=on-failure, EnvironmentFile=-/etc/upstream/upstream.env (overrides the Environment= lines), UMask=0077 (database and snapshots hold password hashes: owner-only).
  • Hardening that works in any container: NoNewPrivileges, empty capability set (port 8080 needs none), RestrictAddressFamilies, RestrictRealtime, RestrictSUIDSGID, LockPersonality, SystemCallArchitectures. No MemoryDenyWriteExecute (V8 would crash).
  • The namespace block (ProtectSystem=strict, ReadWritePaths=/var/lib/upstream, PrivateTmp, ...) is commented out. In an unprivileged LXC it fails with status=226/NAMESPACE unless Options > Features > Nesting is on. Because the app writes to disk, ProtectSystem=strict needs the ReadWritePaths= line that is already next to it in the unit; without it the database cannot be opened.
  • Shutdown: SIGTERM stops the scheduler timer, closes the listener and the database and exits 0 in about a second. Even a SIGKILL or power cut is survivable (WAL replay on the next open, then overdue events are caught up).
  • deploy/upstream-backup.service (oneshot) and .timer: see section 7. deploy/upstream-offsite.service and .timer (optional off-box copy) and deploy/update.sh (section 8) are not installed by the steps above.

4. Configuration

Environment variables read by node server/index.js. Put changes in /etc/upstream/upstream.env (KEY=value, one per line, no export) or in a systemctl edit upstream override, then systemctl restart upstream.

Variable Default (unit) Meaning
PORT 8080 TCP port. Unprivileged, so no capabilities needed. 0 picks a free one (banner prints it).
HOST :: Bind address. :: = one dual-stack socket for IPv4 and IPv6 (falls back to 0.0.0.0 if the kernel has no IPv6). 127.0.0.1 = this container only.
DB_PATH /var/lib/upstream/game.db SQLite file. The code default is ./data/game.db; the unit overrides it. Its directory is created if missing, it must be writable by the service user. -wal and -shm files appear next to it: always copy via a snapshot (section 7), never with cp while running.
TRUST_PROXY auto auto: believe X-Forwarded-For / X-Forwarded-Host only from loopback / private-range peers. 1: always (then firewall the port). 0: never. Used for the per-IP rate limits and the log.
TRUSTED_PROXIES empty Extra IPs / CIDRs auto believes: needed when the proxy reaches the LXC from a global IPv6, Tailscale / CGNAT 100.64.0.0/10 or another routable address.
ALLOWED_ORIGINS empty Extra origins for the WebSocket origin check, comma separated, with the port when it is not 80/443. The REST API uses bearer tokens (no cookies) and does not check origins.
MAX_CONN_PER_IP 40 Open WebSockets per client address (HTTP connections are not capped). 0 disables.
WORLD_SPEED unset (1) World speed multiplier, first boot only, when the database has no world yet.
WORLD_RADIUS unset (50) Map radius in tiles (coordinates -R..R, so 50 = 101 x 101), first boot only.
WORLD_NAME unset (World 1) Name of the first world, first boot only.
SIGNUP_CODE empty If set, POST /auth/signup needs "invite": "<this>" in the body or answers signup_closed (403). Empty = open signup. Keep it in /etc/upstream/upstream.env, not in the unit file.
UPSTREAM_DIR, UPSTREAM_ENV_FILE /opt/upstream, /etc/upstream/upstream.env Only for the upstream-admin wrapper.

Fixed in code (options of createServer()): per-IP and per-token rate limits (login and signup are stricter: about one per 10 s sustained, burst 8, per client address), 32 KB request bodies, idempotency keys pruned hourly.

First boot

An empty or missing database is created, all migrations are applied, and one world is created from WORLD_SPEED, WORLD_RADIUS, WORLD_NAME (seed random). After that those variables are ignored: changing them does not touch the existing world. More worlds, a different seed, ending or resetting a world are admin tasks (section 6). The first player to sign up becomes an ordinary operator; there is no special admin account, admin work is the CLI on the box.

5. nginx Proxy Manager

In NPM: Hosts > Proxy Hosts > Add Proxy Host.

Tab / field Value
Details: Domain Names upstream.example.com
Details: Scheme http
Details: Forward Hostname / IP the LXC's address (for example 192.168.1.51), not 127.0.0.1 when NPM runs elsewhere or in its own container
Details: Forward Port 8080
Details: Cache Assets off (static files are no-cache and revalidated by ETag; the game must see new versions)
Details: Block Common Exploits optional
Details: Websockets Support on. The REST game works without it, but /ws (live push) fails with HTTP 426 when this is off.
Custom Locations none needed
SSL Request a new Let's Encrypt certificate, Force SSL, HTTP/2 (HSTS optional)
Advanced (gear) nothing required; optional gzip below

Optional, Advanced box, to compress JS/JSON for phones (the Node server does not compress; NPM's own gzip on only covers HTML):

gzip on;
gzip_types text/css text/javascript application/javascript application/json application/manifest+json image/svg+xml;

Notes carried over from NETWALL's setup (observed there against NPM v2.16.0, not re-run for UPSTREAM):

  • NPM sends Host without the public port and no X-Forwarded-Host. On a public port other than 443 the browser Origin then differs from Host; list the origin with its port in ALLOWED_ORIGINS (matters for /ws).
  • A proxy on a non-private address (global IPv6, Tailscale) is not believed by TRUST_PROXY=auto: every player then shares one rate-limit budget and the log shows the proxy's address. Fix: TRUSTED_PROXIES=<its address or CIDR>.
  • Clients on your own LAN can pick their address through NPM's real_ip handling, so the per-IP limits protect against public abuse, not against LAN users.
  • Idle timeout: nginx's proxy_read_timeout is 60 s by default (NETWALL's notes say 90 s for NPM; not re-checked). NETWALL never idles because it streams 20 snapshots a second; UPSTREAM's socket is quiet for long stretches. server/push.js sends a protocol ping every 30 s (wsPingMs), below either value, which is reasoning, not a measurement behind a real NPM.

Firewall and dual stack: identical to NETWALL (docs/DEPLOY.md there, section 6): only the proxy needs to reach port 8080. curl -i http://127.0.0.1:8080/healthz and curl -gi 'http://[::1]:8080/healthz' check both families.

6. Operating: upstream-admin

tools/admin.js is the operator CLI (Node core only, uses the game's own db.js, state.js, rules.js). deploy/upstream-admin runs it as the service user against the service's database (it reads DB_PATH from /etc/upstream/upstream.env). Do not run node tools/admin.js as root: SQLite would create game.db-wal and -shm owned by root and the game could no longer open its database (symptom in section 10).

upstream-admin help
upstream-admin status                          # file, size, migrations, overdue events
upstream-admin worlds
upstream-admin create-world --name "Summer" --speed 2 --radius 60 --seed 1234 --end-in 90d
upstream-admin end-world 1 --yes                # same as the natural end: final ranking, escrow back, events cancelled
upstream-admin users [--banned]                # no hashes or tokens are ever printed
upstream-admin operators [--world 1]
upstream-admin ban mallory                     # sets banned, revokes all tokens: old tokens give 401, login 403
upstream-admin unban mallory
upstream-admin revoke-tokens ada               # log out everywhere
upstream-admin inspect-operator ada            # (--world N when the name exists in several worlds, --id N by operator id)
upstream-admin inspect-town 12                 # resources now (rules.resourcesAt), buildings, queue, units if the table exists
upstream-admin events --pending [--type buildDone --town 12 --limit 100]
upstream-admin replay-events --until +6h       # dry run: what the scheduler would process, in order
upstream-admin replay-events --until +6h --simulate   # + run the real handlers on a scratch copy, report outcome and follow-ups
upstream-admin backup /var/lib/upstream/backups --keep 14
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db
upstream-admin grant 12 --fiber 5000 --capital 2000 --yes          # testing only
upstream-admin reset-world 1 --yes-destroy --backup /var/lib/upstream/backups --new-seed
upstream-admin migrate                         # apply pending migrations without starting the server

All commands take --json (machine output) and --db FILE (otherwise $DB_PATH, otherwise ./data/game.db). Piping into head is fine: a closed pipe (EPIPE) stops the output quietly, no stack trace, and the exit code is kept. Exit codes: 0 ok, 1 failed, 2 usage error or a destructive command missing its flag, 3 not found.

Safety rules the CLI follows:

  • Destructive or test-only commands refuse without a flag: end-world and grant need --yes; reset-world needs --yes-destroy (without it, it prints what it would delete and exits 2). ban, unban, revoke-tokens are reversible and need none.
  • end-world <id> --yes ends a world exactly like the natural end: it runs the real worldEnd event handler (server/events/worldEnd.js, payload {"reason":"admin"}). So the final ranking snapshot is stored (GET /worlds/:id/results becomes available: true, winner null unless the backbone race had a winner already), open trade offers are closed and their escrow goes back to the POPs, every other pending event of that world is cancelled (builds in progress, training, movements on the road simply stop: the world is frozen), status becomes ended with end_at = now, and every operator gets a world_ended notification. Events of that world that were already due are processed first, in order, like the server's catch-up. Without --yes it refuses (exit 2) and says how many pending events and open offers it would cancel and close. A world that is already ended is left alone (exit 0). Other worlds are untouched. It works on a running or a stopped server; the CLI is a separate process, so a running server pushes nothing over WebSocket for the notification rows it wrote: open clients learn from their next request (world_ended error) or the next /me. A world that an older version of this command "ended" has no snapshot; the command says so and cannot repair it (reset the world or leave it).
  • reset-world deletes the world's game rows (everything with a world_id, plus tables pointing at them: towns, buildings, events, operators, alliances, messages, ... discovered from the schema, so tables added by later stages are covered) and keeps accounts and tokens; players must join the world again. It resets start_at, clears end_at and sets the world running. It rolls back if it would leave dangling references. Stop the game first (systemctl stop upstream): the server does not know the world was swept while it is running. Use --backup DIR.
  • Everything runs on a running server's database: writes are single short BEGIN IMMEDIATE transactions, the connection is always closed, and SQLite's 5 s busy timeout covers collisions with the game (database is locked from the CLI just means: try again).
  • grant adds to the town's stored amounts (valid at its res_at); the lazy model carries them forward. Amounts above the Rack hall cap are kept but stop growing, like a lowered cap.
  • inspect-town shows values as of now ignoring overdue events; it says so when there are any (server stopped).

7. Backups and restore

upstream-backup.timer runs upstream-backup.service daily at 04:15 (10 min jitter, Persistent=true). It runs tools/admin.js backup /var/lib/upstream/backups --keep 14 as the service user. A snapshot is:

  1. VACUUM INTO a temporary .partial-* file: a consistent copy as of one moment, safe while the game runs (WAL readers do not block the writer; the game keeps its own 5 s busy timeout);
  2. verified: opened read-only, PRAGMA integrity_check = ok, worlds and schema_migrations present;
  3. published under a dated name upstream-YYYYMMDDTHHMMSSZ.db (UTC) with a hard link, so an existing file is never replaced (a same-second collision becomes ...Z-1.db), mode 0600;
  4. only then older snapshots are pruned: --keep N keeps the newest N, --keep-days D keeps those younger than D days, both = survive if either keeps it, neither = no pruning. The newest snapshot and any file not matching the naming pattern are never deleted. A failed verification leaves nothing behind and prunes nothing, and the unit fails (systemctl status upstream-backup, journalctl -u upstream-backup).

Snapshots are plain SQLite files (journal mode delete), not -wal dependent: copy a single file.

systemctl list-timers upstream-backup.timer      # next run
systemctl start upstream-backup                  # one right now
journalctl -u upstream-backup -n 20
ls -l /var/lib/upstream/backups

Change the schedule or retention with systemctl edit upstream-backup.timer / upstream-backup.service (for every 6 hours: OnCalendar=*-*-* 00/6:15:00 and a larger --keep). Before an upgrade or a reset-world, take one by hand.

Off-box copy

Snapshots on the same disk die with the container. Two ways to get them off the box; use one. Both only ever copy files that the backup unit already verified, and neither deletes anything on the far side (retention over there is yours).

A. Pull from another machine (NAS, workstation, the Proxmox host), for example from cron or a timer there:

rsync -a --ignore-existing root@<lxc>:/var/lib/upstream/backups/ /srv/backups/upstream/

--ignore-existing makes it append-only on your side (nothing already stored is overwritten, and snapshots pruned on the LXC stay on the backup host until you decide otherwise). Better than a root key: a dedicated key restricted in authorized_keys to command="rrsync -ro /var/lib/upstream/backups",restrict ... (rrsync comes with the rsync package; not run here). The files are owner-only (upstream), so the reading user needs root or a group ACL.

B. Push from the box: upstream-offsite.service + .timer (deploy/). Disabled until you configure it; nothing here is installed by section 3. The script deploy/upstream-offsite picks the newest snapshot (by modification time), runs verify-backup on it (a snapshot that does not verify is never copied), copies it with rsync to a directory (scp is not supported: it cannot refuse to overwrite, OFFSITE_METHOD=scp is rejected with exit 2), never overwrites a file of the same name there, and re-reads the far copy and compares checksums. The timer fires daily at 05:00 (+ up to 10 min), after the backup timer (04:15). It runs as the upstream user and fails visibly (journalctl -u upstream-offsite).

Set it up (on the far side first, then the box):

# far side (backup host): an account that can only write into one directory, no shell, append-only by convention
useradd --system --create-home --shell /usr/sbin/nologin upstream-backup      # or a NAS user
install -d -o upstream-backup -m 0700 /srv/backups/upstream
#   ~upstream-backup/.ssh/authorized_keys, one line (the key generated below):
#   restrict,command="rrsync /srv/backups/upstream" ssh-ed25519 AAAA... upstream-offsite

# the box (as root): a key only for this, owned by the service user (ssh refuses keys readable by others)
install -d -m 0750 -o root -g upstream /etc/upstream
ssh-keygen -t ed25519 -N '' -C upstream-offsite -f /etc/upstream/offsite_key
chown upstream:upstream /etc/upstream/offsite_key /etc/upstream/offsite_key.pub; chmod 0600 /etc/upstream/offsite_key
#   the far side's host key, checked by eye: compare with `ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub` over there
ssh-keyscan -t ed25519 backup-host.example > /etc/upstream/offsite_known_hosts
ssh-keygen -lf /etc/upstream/offsite_known_hosts

cat >> /etc/upstream/upstream.env <<'EOT'
OFFSITE_DEST=upstream-backup@backup-host.example:/
OFFSITE_SSH_KEY=/etc/upstream/offsite_key
EOT
#   with rrsync the destination path is relative to its root, hence ":/" (or ":" ). A plain directory also works:
#   OFFSITE_DEST=/mnt/nas/upstream   (a mounted share; then no key or known_hosts is needed)

install -m 0644 /opt/upstream/deploy/upstream-offsite.service /opt/upstream/deploy/upstream-offsite.timer /etc/systemd/system/
systemctl daemon-reload
systemctl start upstream-offsite                  # one run now: journalctl -u upstream-offsite -n 20
systemctl enable --now upstream-offsite.timer

Settings (all in /etc/upstream/upstream.env): OFFSITE_DEST (required; without it every run is skipped quietly by the unit's ExecCondition, so installing the timer early is harmless), OFFSITE_METHOD (only rsync; a far side without rsync needs rrsync/a different tool, scp was removed because it would overwrite), OFFSITE_SSH_KEY, OFFSITE_KNOWN_HOSTS (default /etc/upstream/offsite_known_hosts), OFFSITE_SSH_OPTS (for example -p 2222). Host keys are always checked and ssh never prompts (BatchMode=yes). To copy right after each backup instead of on its own timer: systemctl edit upstream-backup and add [Unit] OnSuccess=upstream-offsite.service.

The key on the box can write to the far directory; make that directory the only thing it can reach (the restrict,command="rrsync ..." line) and prune old files over there with a job that lives there, so a compromised game box cannot delete the history. Test it once for real: check that the file arrived and that upstream-admin verify-backup accepts the far copy (a restore from it is section 7). upstream-offsite was exercised with a local directory as the destination (rsync, checksum compare, no-overwrite, refusing an unverified snapshot); the ssh transport, rrsync and the units under systemd were not run.

Restore

The restore is a file copy, and the time model makes it clean: the restored world simply fast-forwards. Every event that fell due since the snapshot is processed in order at start-up, each at its own due time. What is lost is what players did after the snapshot (builds started, signups, messages) and their tokens: players log in again.

systemctl stop upstream-backup.timer upstream
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db     # exit 0 or stop here
mkdir -p /var/lib/upstream/pre-restore
mv /var/lib/upstream/game.db* /var/lib/upstream/pre-restore/ 2>/dev/null                # keep the old one, incl. -wal/-shm
install -o upstream -g upstream -m 0600 /var/lib/upstream/backups/upstream-20261011T041500Z.db /var/lib/upstream/game.db
systemctl start upstream upstream-backup.timer
journalctl -u upstream -n 20          # look for "caught up N overdue event(s)"
upstream-admin status

Restoring onto another machine works the same way (install code and Node, copy the file to DB_PATH). Migrations in the snapshot are checksummed: if the code is newer than the snapshot, the missing migrations are applied at boot; run code that is older than the snapshot's last migration (upstream-admin status shows it) only if you know the schema still fits, the server does not check for that.

8. Upgrading

/opt/upstream is a root-owned git checkout of https://git.vanvikinternet.no/jvanvik/Upstream.git (plain HTTPS, no credentials in the URL or in .git/config; the script refuses an http(s):// origin URL that carries a user or token). Updates go through deploy/update.sh:

sudo sh /opt/upstream/deploy/update.sh --dry-run     # fetch, list the commits and new migrations, change nothing
sudo sh /opt/upstream/deploy/update.sh               # the real thing
sudo sh /opt/upstream/deploy/update.sh --test        # also run `npm test` on the new code before the restart

In order, stopping at the first failure (once the service is stopped, the script always starts it again):

  1. Tools: git, curl, npm, systemctl and the admin wrapper must be on the PATH (runuser, tar and mktemp too with --test). A missing one is exit 2 before anything is touched (without curl the health check could never pass and a good release would be rolled back).
  2. Refuses on a dirty tree (tracked changes or untracked files that are not git-ignored; node_modules/ and data/ are ignored), on a checkout that is not on main (detached HEAD included), on an http(s):// origin whose authority carries user:password@ or token@ (an ssh://git@host/... or git@host:path remote is fine, so is an @ later in the path), and while another update runs (lock directory /run/upstream-update.lock).
  3. git fetch origin main, no prompt. Not a fast-forward of the checkout (history rewritten, local commits): refuses, touches nothing. Nothing new: "already up to date", exit 0, no stop, no backup and no restart.
  4. Only with --test: git archive of the new rev into a temp directory, npm ci --omit=dev --ignore-scripts and npm test there as the service user (runuser -u upstream, HOME and the npm cache inside the temp directory; never as root, never in /opt/upstream). The live service keeps serving meanwhile. Red: exit 1, nothing was stopped or changed. (Fetched test code can still read what the service user can read, for example the database; --test is for your own commits.)
  5. systemctl stop upstream. Why before the merge: the running process require()s server modules lazily, so a process that keeps serving while the tree changes under it can load new files into old code. From here the tree and the process never disagree; the downtime is steps 5-8.
  6. Verified snapshot of the stopped (final) database: upstream-admin backup /var/lib/upstream/backups --keep 14, then verify-backup on the file it wrote. No verified snapshot, no update: the service is started again, exit 2, nothing changed. The snapshot path is printed at the end.
  7. git merge --ff-only.
  8. npm ci --omit=dev --ignore-scripts only if package.json / package-lock.json changed (or node_modules is missing). No package install script runs as root (ws needs none; if a future dependency needs one, run it by hand).
  9. systemctl start upstream, then wait for http://127.0.0.1:<PORT>/healthz (PORT from /etc/upstream/upstream.env, else 8080) for up to 30 s (HEALTH_TIMEOUT or --timeout N).
  10. Rollback on any failure from step 6 on: systemctl stop (a crash-looping new process must not run against the old tree), git reset --hard <old rev> (the tree was clean, so nothing is lost; the branch stays on main), npm ci again if the lock file had changed, systemctl start, wait for /healthz again.
  11. Prints old -> new revs and the commits that went out.

Exit codes: 0 updated or already current, 1 failed and the old version answers again (or --test was red), 2 refused or usage error (nothing changed), 3 failed and the old version does not answer either (the output names the snapshot; journalctl -u upstream -n 50).

Things the script cannot undo: migrations. A migration that ran at the new version's first start stays applied when the code is rolled back. When the update contains new files in server/migrations/ the script says so up front (NEW MIGRATIONS ...) and, after a rollback, prints the snapshot to restore if the old code misbehaves on the newer schema (section 7: stop, restore, start; players lose what they did since the snapshot). Migration 005_site_garrison.sql is called out specially: it adds site_stationed (units garrisoned in backbone landing sites). The previous code knows nothing of that table: it neither lists those units, nor charges their power, nor lets them go home. A code-only rollback is therefore clean only while nobody has garrisoned a site since the new code started; otherwise restore the snapshot. A rollback target is always the previous revision only, one step. An interrupt (Ctrl-C, SIGTERM) after the stop starts the service again (before the merge, on the untouched old tree) or rolls back like any other failure (after it); SIGKILL or a power cut mid-update cannot be caught: then git log -1, git status and systemctl status upstream tell where it stopped (if the service is down: systemctl start upstream; if the tree is already new but the service ran the old code, systemctl restart upstream). The stop/start drops open WebSockets; clients reconnect on their own.

Settings: UPSTREAM_DIR, UPSTREAM_UNIT, UPSTREAM_ENV_FILE, UPSTREAM_ADMIN, BACKUP_DIR, BACKUP_KEEP, UPSTREAM_REMOTE, UPSTREAM_SERVICE_USER (the user --test runs as, default upstream), UPSTREAM_BRANCH, HEALTH_URL, HEALTH_TIMEOUT, LOCK_DIR (sh update.sh --help lists them). It needs git, curl, npm, systemctl and upstream-admin (section 3) on the PATH of root (checked up front).

What was run: test/deploy-update.test.js drives the real script with sh against a scratch bare origin and checkout and stubs for systemctl, curl, npm, upstream-admin and sleep: up to date, update order (stop, snapshot, merge, start: the tree is old at the stop and new at the start), npm ci --ignore-scripts only when the lock changed, --test (isolated copy, via a stub runuser), missing tools, dirty tree, credentials in the URL (and the ssh / @-in-path URLs that are allowed), detached HEAD, non-fast-forward, concurrent run, --dry-run, failing or unverifiable backup, unhealthy new version (rollback, migrations warning, dependency restore), red tests, a rollback that does not recover (exit 3), PORT from the env file. sh -n, bash --posix -n and zsh -n pass; shellcheck and dash were not available on the dev machine. Not run: the script on the real box, against real systemd, a real npm ci / npm test, or a real /healthz.

Updating without git (rsync from a workstation, the way section 3 first installed it) still works; do the same by hand:

systemctl stop upstream                                           # 1. stop FIRST: the old process lazily loads modules from the tree
upstream-admin backup /var/lib/upstream/backups --keep 14         # 2. snapshot of the stopped database
rsync -a --delete --exclude node_modules --exclude .git --exclude data ./ root@<lxc>:/opt/upstream/    # 3. new code (from your workstation)
cd /opt/upstream && npm ci --omit=dev --ignore-scripts && chown -R root:root /opt/upstream             # 4. dependencies, ownership
systemctl start upstream                                          # 5. migrations apply at boot
journalctl -u upstream -n 30                                      # 6. "applied migrations: ..." then the listen banner
curl -i http://127.0.0.1:8080/healthz
  • Migrations are server/migrations/NNN_name.sql, applied in order inside a transaction each, recorded with a checksum and never edited after they shipped. If one that was applied has changed, the server refuses to start (migration 001_init.sql was edited after it was applied): restore that file from git and ship a new numbered migration instead. upstream-admin status lists applied and pending migrations; upstream-admin migrate applies them without starting the game.
  • Each migration runs in its own transaction: a failing one rolls back and the server exits, earlier ones of the same boot stay applied. Going back to the old code after a migration committed means restoring the pre-upgrade snapshot (section 7).
  • Updated unit files: install -m 0644 ..., systemctl daemon-reload, systemctl restart upstream.
  • Players reconnecting after the restart need no action; clients get serverTime in every response and refetch.

9. Health, logs, checks

curl -i https://upstream.example.com/healthz          # 200 ok (only says the process answers HTTP)
curl -I https://upstream.example.com/                 # 200 + Content-Security-Policy
upstream-admin status                                 # database, migrations, events pending / OVERDUE
journalctl -u upstream -f                             # live log; -p err for failures only
journalctl -u upstream --since "1 hour ago" | grep -E "failed|refused"

The log has: applied migrations, created world, caught-up events, refused rate_limited(...) ip=..., and event N (type) failed: ... for a handler that threw (that event is marked failed and the queue moves on; list them with upstream-admin events --status failed). Player IP addresses appear in the log; passwords and tokens never do. A healthy server has no overdue events in status: the timer processes events within milliseconds of their due time.

10. Troubleshooting

Symptom Cause and fix
Service dies at start: No such built-in module: node:sqlite Node is older than 22.13, or the distro's Node 20 is first on the PATH. node --version, which -a node, section 1.
ExperimentalWarning: SQLite is an experimental feature in the journal Harmless. Printed once per start on some Node versions.
SQLITE_CANTOPEN / unable to open database file DB_PATH's directory is not writable by upstream: wrong path in upstream.env, /var/lib/upstream created by hand with other ownership (chown -R upstream:upstream /var/lib/upstream), or ProtectSystem=strict enabled without ReadWritePaths=/var/lib/upstream.
attempt to write a readonly database / unable to open database file right after using the CLI admin.js was run as root and left root-owned game.db-wal / -shm. Stop the game, chown upstream:upstream /var/lib/upstream/game.db*, start. Use upstream-admin, which switches user.
migration NNN_x.sql was edited after it was applied An applied migration changed on disk (checksum mismatch). Restore it from git; changes go in a new migration.
UPSTREAM failed to start: ... with EADDRINUSE Another process on the port (ss -ltnp | grep 8080) or two copies. Change PORT and the NPM forward port together.
IPv6 is not available (...); falling back to 0.0.0.0 The kernel has no IPv6. IPv4 keeps working; set HOST=0.0.0.0 to silence it.
status=226/NAMESPACE The optional namespace hardening is on in an LXC without Nesting: enable Options > Features > Nesting and restart the CT, or comment the block out.
Cannot find module 'ws' npm ci --omit=dev was not run in /opt/upstream.
502 Bad Gateway from NPM Upstream unreachable: systemctl status upstream, curl -i http://<lxc>:8080/healthz from the NPM host, forward IP/port, firewall.
Page loads but a new deploy still shows the old client NPM Cache Assets is on, or a service layer ignores no-cache. Switch it off, hard-reload.
Signup answers 403 signup_closed SIGNUP_CODE is set. Send {"invite": "<code>"} in the signup body, or unset it and restart.
Login/signup answers 429 rate_limited for many players They all arrive from one address: the proxy is not trusted so the per-IP budget is shared (log says ip=<the proxy>). Set TRUSTED_PROXIES. Login and signup allow about one request per 10 s per address, burst 8; a whole classroom behind one NAT can hit that.
Game seems frozen: builds never finish upstream-admin status: OVERDUE events mean the scheduler is not running (server stopped or stuck) or handlers fail: upstream-admin events --status failed, journal event N ... failed. A restart catches up everything in order.
Banned user still plays They cannot: ban revokes every token and blocks login. Check the username spelling (upstream-admin users --banned).
database is locked from the CLI The game held a write lock longer than the 5 s busy timeout. Retry.
upstream-backup.service failed journalctl -u upstream-backup. Typical: disk full, or the integrity check of the new snapshot failed (then nothing was kept and nothing pruned; check the disk and the game.db itself).
Backups not appearing systemctl list-timers upstream-backup.timer; the timer must be enabled (systemctl enable --now upstream-backup.timer), not just installed.
World reset by mistake Restore the snapshot taken before (section 7), or the one reset-world --backup wrote.
WebSocket upgrade answered 426 / 403 / 429 426: Websockets Support off in NPM. 403: origin check, see ALLOWED_ORIGINS and the notes in section 5. 429: more than MAX_CONN_PER_IP sockets from one address.