Built: - Economy: raid loot capped to the room the raiders have at home (rest stays with the victim, "lootLeft" in the report); trade escrow reserves its room so refunds always fit; trade accept refuses with no_room_for_goods when the goods would not fit. Loot destroyed per 30-day run 31-61k -> ~1k, refunds clipped ~27k -> 0 (docs/BALANCE.md, 3 seeds). - End game: members of the holding alliance garrison landing sites (migration 005_site_garrison.sql), garrison counts in site defence, 72 h capture shield, withdraw route, POP view lists garrisoned units. Shipping capacity scales with world speed (3x world: winner 76-98 -> ~30-36 days after sites open). Bot runs show real contests (84-194 attacks on held sites at 1x). - Ops: deploy/update.sh (backup, stop, ff-merge, npm ci --ignore-scripts, health check, rollback), deploy/upstream-offsite (rsync only), admin end-world snapshots results, EPIPE fix, docs/workflows/ filled in. - e2e 62 -> 163 checks: units, raids/reports, alliance+messages, trade, end game, nb, PWA, reconnect, slow polling fallback (opt-in). Two app bugs found: message push ignored, "New" button stayed English. Tested: npm test 274/274, openapi check, npm run e2e 163/163 (137 s), bot runs 50x30d seeds 1-3 and 140-day/60-day speed-3 end-game runs INVARIANTS OK, adversarial review pass (findings fixed or documented). NOT tested: update.sh on a real box (systemd, real npm, runuser; stubs only, shellcheck not installed), garrison card/loot text on a phone, Node 22.13, humans. Known leaks left open and documented (BALANCE S6): loot/shipments can still overflow when stock arrives in flight, bounced backbone shipments lose overflow (reported to the sender). Garrison numbers (heldDefenceMult 2, shield 72 h) are first guesses from 2 seeds. Not claimed: balanced. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q2MJaQ5LXAHzGnTDgceT14
42 KiB
Deploying UPSTREAM
UPSTREAM is one Node process: it serves the static client from public/, the REST API under /api/v1/ and a
WebSocket push channel at /ws (docs/PROTOCOL.md). Unlike NETWALL it keeps state: one SQLite file, owned by that process. A restart loses
nothing: overdue events are caught up in order before the server accepts traffic.
phone / laptop --https--> nginx Proxy Manager (TLS, :443) --http (+ WebSocket)--> Debian 13 LXC: node :8080
/opt/upstream (code, read-only)
/var/lib/upstream (game.db, backups/)
Same shape as NETWALL (docs/DEPLOY.md there): same proxy, same hardening notes, same env vars for the network
layer. The differences are the Node version (node:sqlite needs 22.13+), a state directory and backups.
What was and was not run while writing this (so you know what to check on the real box):
- Run:
systemd-analyze verifyon the unit files (syntax and directives only; the two off-site units: only the expected "not executable" warning for a script path that does not exist on the dev machine), the CLItools/admin.jsand its tests on a dev machine (Node 24.21), a backup, verify and prune cycle,sh -nondeploy/upstream-admin. Re-checked on 2026-10-11 with the code as it is now (Node 24.21, scratch directory, no systemd): first boot withDB_PATH/PORT/HOST(migrations 001-004 applied,created world 1),/healthz,/headers, signup + build over the API,admin.jsstatus / worlds / users / operators / inspect-town / events / replay-events / backup / verify-backup / ban / unban / grant / end-world (refusal without--yes), SIGTERM and restart (state, queue and password kept, no migrations re-applied; tokens revoked bybanstay revoked), SIGKILL then restart after a build fell due (caught up 1 overdue event(s), the build level was applied), and a restore by copying a snapshot to a freshDB_PATHand starting on it (section 7: it loggedcaught up 1 overdue event(s)). - Run, ops scripts (dev machine, stubs, no systemd):
deploy/update.shanddeploy/upstream-offsiteandadmin.js end-worldthroughtest/deploy-update.test.js,test/deploy-offsite.test.js,test/admin-endworld.test.js; sections 7 and 8 say exactly what each covers. - Taken from NETWALL's docs, not re-run for UPSTREAM: everything about nginx Proxy Manager (labels, header behaviour, "Client addresses"). NETWALL's guide records what was observed against NPM v2.16.0.
- Never run: a Proxmox LXC, a started unit under systemd, the timer firing on its own, a Node install from the
tarball or NodeSource on Debian 13,
rrsync, Let's Encrypt, a restore on the real box. Treat those steps as a checklist to confirm, and fix this file when reality differs.
1. Requirements
- Debian 13 (trixie) LXC with its own IPv4 (optionally IPv6) address the proxy can reach. 1 core, 512 MB RAM and 4 GB
disk are plenty for a small world; the database grows with the event and notification history (measure it with
upstream-admin status), and snapshots multiply it by the retention count. - Node 22.13 or newer. Debian 13 ships
nodejs20.19.2 (checked on tracker.debian.org: stable has 20.19.2, testing and unstable have 24.21). Node 20 has nonode:sqlite, soapt install nodejsis not enough.package.jsonsays"engines": {"node": ">=22.13"}; 22.5 to 22.12 only havenode:sqlitebehind--experimental-sqlite, which the unit does not pass. Development is on Node 24.21. - A DNS name for the game (for example
upstream.example.com) pointing at the proxy.
Getting a suitable Node (pick one)
A. Official tarball (no third-party apt repo, nothing piped into a shell). Pick a current 24.x (Active LTS) or
22.x from https://nodejs.org/en/download and put its version in V:
apt update && apt install -y curl xz-utils ca-certificates
V=24.21.0 # example: use the current 24.x (or >= 22.13) you picked
ARCH=linux-x64 # linux-arm64 on ARM hosts (uname -m: aarch64)
cd /tmp
curl -fsSLO https://nodejs.org/dist/v$V/node-v$V-$ARCH.tar.xz
curl -fsSLO https://nodejs.org/dist/v$V/SHASUMS256.txt
grep " node-v$V-$ARCH.tar.xz\$" SHASUMS256.txt | sha256sum -c - # must print: ... OK
tar -C /opt -xJf node-v$V-$ARCH.tar.xz
ln -sfn /opt/node-v$V-$ARCH /opt/node
ln -sf /opt/node/bin/node /usr/local/bin/node
ln -sf /opt/node/bin/npm /usr/local/bin/npm
ln -sf /opt/node/bin/npx /usr/local/bin/npx
(For extra assurance also verify the signature of SHASUMS256.txt as described on nodejs.org.) An upgrade later is
a new tarball and ln -sfn to it, then systemctl restart upstream. upstream.service starts /usr/bin/env node, and
systemd's default PATH lists /usr/local/bin before /usr/bin, so the tarball wins over any distro Node.
B. NodeSource apt repository (follows their documentation; not run here). It puts Node in /usr/bin/node:
apt install -y ca-certificates curl gnupg
install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_24.x nodistro main" > /etc/apt/sources.list.d/nodesource.list
apt update && apt install -y nodejs # includes npm; do not also install Debian's npm package
Verify either way (all three must succeed):
node --version # v22.13.0 or newer
node -e "require('node:sqlite'); console.log('node:sqlite ok')"
which -a node # the first one is the one the unit will use
On 22.x/24.x Node may print ExperimentalWarning: SQLite is an experimental feature to stderr; that is harmless (it
shows up in the journal once per start). No such built-in module: node:sqlite means Node is too old (or it is the
distro's 20.x earlier on the PATH). If node:sqlite ever misbehaves, DECISIONS.md D1 describes the one-file swap
to better-sqlite3 in server/db.js; that would add a native dependency and a build toolchain to the LXC.
2. Create the LXC (Proxmox)
Container creation, the Proxmox UI way (CLI pct create works too):
- Template
debian-13-standard, unprivileged, hostnameupstream, 1 core, 512 MB RAM (+256 MB swap), 4-8 GB disk. - Network: a static address (or a DHCP reservation) on a bridge the NPM host can reach. The proxy forwards to that address, so it must not change.
- Features: Nesting is optional; tick it only if you want the namespace-based hardening block in the unit (section 3). Everything else works without it.
- Start on boot: yes. Timezone does not matter: the game uses UTC ms, the backup timer uses the container's clock.
- Proxmox's own CT backups (
vzdump) are not a substitute for the snapshots in section 7: a running-container copy of a SQLite file is crash-consistent at best. Use them for the OS and let the verified snapshots be the data backup.
3. Install in the LXC
Run as root inside the container (Node from section 1 is already in place).
apt install -y rsync # for the copy below (or use git)
# a dedicated, login-less user that only runs the game
useradd --system --home-dir /var/lib/upstream --shell /usr/sbin/nologin upstream
# code: read-only for the service user, owned by root
mkdir -p /opt/upstream
# from your workstation:
# rsync -a --delete --exclude node_modules --exclude .git --exclude data --exclude e2e ./ root@<lxc>:/opt/upstream/
# (needs server/, public/, tools/, deploy/, docs/, package.json, package-lock.json; `test/` is optional)
# or, the way the live box is run, a git checkout that deploy/update.sh updates later (section 8):
# git clone https://git.vanvikinternet.no/jvanvik/Upstream.git /opt/upstream (plain HTTPS URL, no token in it)
cd /opt/upstream
npm ci --omit=dev # only "ws"; follows the lockfile exactly
chown -R root:root /opt/upstream
# the admin helper on the PATH
install -m 0755 /opt/upstream/deploy/upstream-admin /usr/local/bin/upstream-admin
# optional settings / secrets file (read by the unit and by upstream-admin)
install -d -m 0750 -o root -g upstream /etc/upstream
install -m 0640 -o root -g upstream /dev/null /etc/upstream/upstream.env
# edit it: SIGNUP_CODE=choose-a-long-random-string
# ALLOWED_ORIGINS=https://upstream.example.com
# the units
install -m 0644 /opt/upstream/deploy/upstream.service /opt/upstream/deploy/upstream-backup.service /opt/upstream/deploy/upstream-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now upstream upstream-backup.timer
systemctl status upstream
# off-box copy of the snapshots is optional and not installed here: section 7
journalctl -u upstream -f
The first start prints applied migrations: 001_init.sql, ..., created world 1 and the listen banner with
database: /var/lib/upstream/game.db. Quick manual run as the service user instead of systemd (Ctrl-C to stop):
runuser -u upstream -- env PORT=8080 DB_PATH=/var/lib/upstream/game.db node /opt/upstream/server/index.js.
What the units do
deploy/upstream.service: userupstream,WorkingDirectory=/opt/upstream,StateDirectory=upstream(systemd creates/var/lib/upstreamowned by that user, mode 0700, so the game can write nowhere else),Restart=on-failure,EnvironmentFile=-/etc/upstream/upstream.env(overrides theEnvironment=lines),UMask=0077(database and snapshots hold password hashes: owner-only).- Hardening that works in any container:
NoNewPrivileges, empty capability set (port 8080 needs none),RestrictAddressFamilies,RestrictRealtime,RestrictSUIDSGID,LockPersonality,SystemCallArchitectures. NoMemoryDenyWriteExecute(V8 would crash). - The namespace block (
ProtectSystem=strict,ReadWritePaths=/var/lib/upstream,PrivateTmp, ...) is commented out. In an unprivileged LXC it fails withstatus=226/NAMESPACEunless Options > Features > Nesting is on. Because the app writes to disk,ProtectSystem=strictneeds theReadWritePaths=line that is already next to it in the unit; without it the database cannot be opened. - Shutdown: SIGTERM stops the scheduler timer, closes the listener and the database and exits 0 in about a second. Even a SIGKILL or power cut is survivable (WAL replay on the next open, then overdue events are caught up).
deploy/upstream-backup.service(oneshot) and.timer: see section 7.deploy/upstream-offsite.serviceand.timer(optional off-box copy) anddeploy/update.sh(section 8) are not installed by the steps above.
4. Configuration
Environment variables read by node server/index.js. Put changes in /etc/upstream/upstream.env (KEY=value, one per
line, no export) or in a systemctl edit upstream override, then systemctl restart upstream.
| Variable | Default (unit) | Meaning |
|---|---|---|
PORT |
8080 |
TCP port. Unprivileged, so no capabilities needed. 0 picks a free one (banner prints it). |
HOST |
:: |
Bind address. :: = one dual-stack socket for IPv4 and IPv6 (falls back to 0.0.0.0 if the kernel has no IPv6). 127.0.0.1 = this container only. |
DB_PATH |
/var/lib/upstream/game.db |
SQLite file. The code default is ./data/game.db; the unit overrides it. Its directory is created if missing, it must be writable by the service user. -wal and -shm files appear next to it: always copy via a snapshot (section 7), never with cp while running. |
TRUST_PROXY |
auto |
auto: believe X-Forwarded-For / X-Forwarded-Host only from loopback / private-range peers. 1: always (then firewall the port). 0: never. Used for the per-IP rate limits and the log. |
TRUSTED_PROXIES |
empty | Extra IPs / CIDRs auto believes: needed when the proxy reaches the LXC from a global IPv6, Tailscale / CGNAT 100.64.0.0/10 or another routable address. |
ALLOWED_ORIGINS |
empty | Extra origins for the WebSocket origin check, comma separated, with the port when it is not 80/443. The REST API uses bearer tokens (no cookies) and does not check origins. |
MAX_CONN_PER_IP |
40 |
Open WebSockets per client address (HTTP connections are not capped). 0 disables. |
WORLD_SPEED |
unset (1) | World speed multiplier, first boot only, when the database has no world yet. |
WORLD_RADIUS |
unset (50) | Map radius in tiles (coordinates -R..R, so 50 = 101 x 101), first boot only. |
WORLD_NAME |
unset (World 1) |
Name of the first world, first boot only. |
SIGNUP_CODE |
empty | If set, POST /auth/signup needs "invite": "<this>" in the body or answers signup_closed (403). Empty = open signup. Keep it in /etc/upstream/upstream.env, not in the unit file. |
UPSTREAM_DIR, UPSTREAM_ENV_FILE |
/opt/upstream, /etc/upstream/upstream.env |
Only for the upstream-admin wrapper. |
Fixed in code (options of createServer()): per-IP and per-token rate limits (login and signup are stricter: about one
per 10 s sustained, burst 8, per client address), 32 KB request bodies, idempotency keys pruned hourly.
First boot
An empty or missing database is created, all migrations are applied, and one world is created from WORLD_SPEED,
WORLD_RADIUS, WORLD_NAME (seed random). After that those variables are ignored: changing them does not touch the
existing world. More worlds, a different seed, ending or resetting a world are admin tasks (section 6). The first
player to sign up becomes an ordinary operator; there is no special admin account, admin work is the CLI on the box.
5. nginx Proxy Manager
In NPM: Hosts > Proxy Hosts > Add Proxy Host.
| Tab / field | Value |
|---|---|
| Details: Domain Names | upstream.example.com |
| Details: Scheme | http |
| Details: Forward Hostname / IP | the LXC's address (for example 192.168.1.51), not 127.0.0.1 when NPM runs elsewhere or in its own container |
| Details: Forward Port | 8080 |
| Details: Cache Assets | off (static files are no-cache and revalidated by ETag; the game must see new versions) |
| Details: Block Common Exploits | optional |
| Details: Websockets Support | on. The REST game works without it, but /ws (live push) fails with HTTP 426 when this is off. |
| Custom Locations | none needed |
| SSL | Request a new Let's Encrypt certificate, Force SSL, HTTP/2 (HSTS optional) |
| Advanced (gear) | nothing required; optional gzip below |
Optional, Advanced box, to compress JS/JSON for phones (the Node server does not compress; NPM's own gzip on only
covers HTML):
gzip on;
gzip_types text/css text/javascript application/javascript application/json application/manifest+json image/svg+xml;
Notes carried over from NETWALL's setup (observed there against NPM v2.16.0, not re-run for UPSTREAM):
- NPM sends
Hostwithout the public port and noX-Forwarded-Host. On a public port other than 443 the browserOriginthen differs fromHost; list the origin with its port inALLOWED_ORIGINS(matters for/ws). - A proxy on a non-private address (global IPv6, Tailscale) is not believed by
TRUST_PROXY=auto: every player then shares one rate-limit budget and the log shows the proxy's address. Fix:TRUSTED_PROXIES=<its address or CIDR>. - Clients on your own LAN can pick their address through NPM's
real_iphandling, so the per-IP limits protect against public abuse, not against LAN users. - Idle timeout: nginx's
proxy_read_timeoutis 60 s by default (NETWALL's notes say 90 s for NPM; not re-checked). NETWALL never idles because it streams 20 snapshots a second; UPSTREAM's socket is quiet for long stretches.server/push.jssends a protocol ping every 30 s (wsPingMs), below either value, which is reasoning, not a measurement behind a real NPM.
Firewall and dual stack: identical to NETWALL (docs/DEPLOY.md there, section 6): only the proxy needs to reach port
8080. curl -i http://127.0.0.1:8080/healthz and curl -gi 'http://[::1]:8080/healthz' check both families.
6. Operating: upstream-admin
tools/admin.js is the operator CLI (Node core only, uses the game's own db.js, state.js, rules.js).
deploy/upstream-admin runs it as the service user against the service's database (it reads DB_PATH from
/etc/upstream/upstream.env). Do not run node tools/admin.js as root: SQLite would create game.db-wal and
-shm owned by root and the game could no longer open its database (symptom in section 10).
upstream-admin help
upstream-admin status # file, size, migrations, overdue events
upstream-admin worlds
upstream-admin create-world --name "Summer" --speed 2 --radius 60 --seed 1234 --end-in 90d
upstream-admin end-world 1 --yes # same as the natural end: final ranking, escrow back, events cancelled
upstream-admin users [--banned] # no hashes or tokens are ever printed
upstream-admin operators [--world 1]
upstream-admin ban mallory # sets banned, revokes all tokens: old tokens give 401, login 403
upstream-admin unban mallory
upstream-admin revoke-tokens ada # log out everywhere
upstream-admin inspect-operator ada # (--world N when the name exists in several worlds, --id N by operator id)
upstream-admin inspect-town 12 # resources now (rules.resourcesAt), buildings, queue, units if the table exists
upstream-admin events --pending [--type buildDone --town 12 --limit 100]
upstream-admin replay-events --until +6h # dry run: what the scheduler would process, in order
upstream-admin replay-events --until +6h --simulate # + run the real handlers on a scratch copy, report outcome and follow-ups
upstream-admin backup /var/lib/upstream/backups --keep 14
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db
upstream-admin grant 12 --fiber 5000 --capital 2000 --yes # testing only
upstream-admin reset-world 1 --yes-destroy --backup /var/lib/upstream/backups --new-seed
upstream-admin migrate # apply pending migrations without starting the server
All commands take --json (machine output) and --db FILE (otherwise $DB_PATH, otherwise ./data/game.db). Piping into head is fine: a closed
pipe (EPIPE) stops the output quietly, no stack trace, and the exit code is kept.
Exit codes: 0 ok, 1 failed, 2 usage error or a destructive command missing its flag, 3 not found.
Safety rules the CLI follows:
- Destructive or test-only commands refuse without a flag:
end-worldandgrantneed--yes;reset-worldneeds--yes-destroy(without it, it prints what it would delete and exits 2).ban,unban,revoke-tokensare reversible and need none. end-world <id> --yesends a world exactly like the natural end: it runs the realworldEndevent handler (server/events/worldEnd.js, payload{"reason":"admin"}). So the final ranking snapshot is stored (GET /worlds/:id/resultsbecomesavailable: true,winnernull unless the backbone race had a winner already), open trade offers are closed and their escrow goes back to the POPs, every other pending event of that world is cancelled (builds in progress, training, movements on the road simply stop: the world is frozen), status becomesendedwithend_at= now, and every operator gets aworld_endednotification. Events of that world that were already due are processed first, in order, like the server's catch-up. Without--yesit refuses (exit 2) and says how many pending events and open offers it would cancel and close. A world that is already ended is left alone (exit 0). Other worlds are untouched. It works on a running or a stopped server; the CLI is a separate process, so a running server pushes nothing over WebSocket for the notification rows it wrote: open clients learn from their next request (world_endederror) or the next/me. A world that an older version of this command "ended" has no snapshot; the command says so and cannot repair it (reset the world or leave it).reset-worlddeletes the world's game rows (everything with aworld_id, plus tables pointing at them: towns, buildings, events, operators, alliances, messages, ... discovered from the schema, so tables added by later stages are covered) and keeps accounts and tokens; players must join the world again. It resetsstart_at, clearsend_atand sets the world running. It rolls back if it would leave dangling references. Stop the game first (systemctl stop upstream): the server does not know the world was swept while it is running. Use--backup DIR.- Everything runs on a running server's database: writes are single short
BEGIN IMMEDIATEtransactions, the connection is always closed, and SQLite's 5 s busy timeout covers collisions with the game (database is lockedfrom the CLI just means: try again). grantadds to the town's stored amounts (valid at itsres_at); the lazy model carries them forward. Amounts above the Rack hall cap are kept but stop growing, like a lowered cap.inspect-townshows values as of now ignoring overdue events; it says so when there are any (server stopped).
7. Backups and restore
upstream-backup.timer runs upstream-backup.service daily at 04:15 (10 min jitter, Persistent=true). It runs
tools/admin.js backup /var/lib/upstream/backups --keep 14 as the service user. A snapshot is:
VACUUM INTOa temporary.partial-*file: a consistent copy as of one moment, safe while the game runs (WAL readers do not block the writer; the game keeps its own 5 s busy timeout);- verified: opened read-only,
PRAGMA integrity_check=ok,worldsandschema_migrationspresent; - published under a dated name
upstream-YYYYMMDDTHHMMSSZ.db(UTC) with a hard link, so an existing file is never replaced (a same-second collision becomes...Z-1.db), mode 0600; - only then older snapshots are pruned:
--keep Nkeeps the newest N,--keep-days Dkeeps those younger than D days, both = survive if either keeps it, neither = no pruning. The newest snapshot and any file not matching the naming pattern are never deleted. A failed verification leaves nothing behind and prunes nothing, and the unit fails (systemctl status upstream-backup,journalctl -u upstream-backup).
Snapshots are plain SQLite files (journal mode delete), not -wal dependent: copy a single file.
systemctl list-timers upstream-backup.timer # next run
systemctl start upstream-backup # one right now
journalctl -u upstream-backup -n 20
ls -l /var/lib/upstream/backups
Change the schedule or retention with systemctl edit upstream-backup.timer / upstream-backup.service (for every
6 hours: OnCalendar=*-*-* 00/6:15:00 and a larger --keep). Before an upgrade or a reset-world, take one by hand.
Off-box copy
Snapshots on the same disk die with the container. Two ways to get them off the box; use one. Both only ever copy files that the backup unit already verified, and neither deletes anything on the far side (retention over there is yours).
A. Pull from another machine (NAS, workstation, the Proxmox host), for example from cron or a timer there:
rsync -a --ignore-existing root@<lxc>:/var/lib/upstream/backups/ /srv/backups/upstream/
--ignore-existing makes it append-only on your side (nothing already stored is overwritten, and snapshots pruned on
the LXC stay on the backup host until you decide otherwise). Better than a root key: a dedicated key restricted in
authorized_keys to command="rrsync -ro /var/lib/upstream/backups",restrict ... (rrsync comes with the rsync
package; not run here). The files are owner-only (upstream), so the reading user needs root or a group ACL.
B. Push from the box: upstream-offsite.service + .timer (deploy/). Disabled until you configure it; nothing here is installed by
section 3. The script deploy/upstream-offsite picks the newest snapshot (by modification time), runs verify-backup on it (a snapshot that
does not verify is never copied), copies it with rsync to a directory (scp is not supported: it cannot refuse to overwrite, OFFSITE_METHOD=scp is rejected with exit 2), never overwrites a file of the same
name there, and re-reads the far copy and compares checksums. The timer fires daily at 05:00 (+ up to 10 min), after the backup timer (04:15).
It runs as the upstream user and fails visibly (journalctl -u upstream-offsite).
Set it up (on the far side first, then the box):
# far side (backup host): an account that can only write into one directory, no shell, append-only by convention
useradd --system --create-home --shell /usr/sbin/nologin upstream-backup # or a NAS user
install -d -o upstream-backup -m 0700 /srv/backups/upstream
# ~upstream-backup/.ssh/authorized_keys, one line (the key generated below):
# restrict,command="rrsync /srv/backups/upstream" ssh-ed25519 AAAA... upstream-offsite
# the box (as root): a key only for this, owned by the service user (ssh refuses keys readable by others)
install -d -m 0750 -o root -g upstream /etc/upstream
ssh-keygen -t ed25519 -N '' -C upstream-offsite -f /etc/upstream/offsite_key
chown upstream:upstream /etc/upstream/offsite_key /etc/upstream/offsite_key.pub; chmod 0600 /etc/upstream/offsite_key
# the far side's host key, checked by eye: compare with `ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub` over there
ssh-keyscan -t ed25519 backup-host.example > /etc/upstream/offsite_known_hosts
ssh-keygen -lf /etc/upstream/offsite_known_hosts
cat >> /etc/upstream/upstream.env <<'EOT'
OFFSITE_DEST=upstream-backup@backup-host.example:/
OFFSITE_SSH_KEY=/etc/upstream/offsite_key
EOT
# with rrsync the destination path is relative to its root, hence ":/" (or ":" ). A plain directory also works:
# OFFSITE_DEST=/mnt/nas/upstream (a mounted share; then no key or known_hosts is needed)
install -m 0644 /opt/upstream/deploy/upstream-offsite.service /opt/upstream/deploy/upstream-offsite.timer /etc/systemd/system/
systemctl daemon-reload
systemctl start upstream-offsite # one run now: journalctl -u upstream-offsite -n 20
systemctl enable --now upstream-offsite.timer
Settings (all in /etc/upstream/upstream.env): OFFSITE_DEST (required; without it every run is skipped quietly by the unit's
ExecCondition, so installing the timer early is harmless), OFFSITE_METHOD (only rsync; a far side without rsync needs rrsync/a different tool, scp was removed because it would overwrite), OFFSITE_SSH_KEY, OFFSITE_KNOWN_HOSTS (default /etc/upstream/offsite_known_hosts),
OFFSITE_SSH_OPTS (for example -p 2222). Host keys are always checked and ssh never prompts (BatchMode=yes). To copy right after each
backup instead of on its own timer: systemctl edit upstream-backup and add [Unit] OnSuccess=upstream-offsite.service.
The key on the box can write to the far directory; make that directory the only thing it can reach (the restrict,command="rrsync ..."
line) and prune old files over there with a job that lives there, so a compromised game box cannot delete the history.
Test it once for real: check that the file arrived and that upstream-admin verify-backup accepts the far copy (a restore from it is section 7). upstream-offsite was exercised with a local directory as the destination (rsync, checksum
compare, no-overwrite, refusing an unverified snapshot); the ssh transport, rrsync and the units under systemd were not run.
Restore
The restore is a file copy, and the time model makes it clean: the restored world simply fast-forwards. Every event that fell due since the snapshot is processed in order at start-up, each at its own due time. What is lost is what players did after the snapshot (builds started, signups, messages) and their tokens: players log in again.
systemctl stop upstream-backup.timer upstream
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db # exit 0 or stop here
mkdir -p /var/lib/upstream/pre-restore
mv /var/lib/upstream/game.db* /var/lib/upstream/pre-restore/ 2>/dev/null # keep the old one, incl. -wal/-shm
install -o upstream -g upstream -m 0600 /var/lib/upstream/backups/upstream-20261011T041500Z.db /var/lib/upstream/game.db
systemctl start upstream upstream-backup.timer
journalctl -u upstream -n 20 # look for "caught up N overdue event(s)"
upstream-admin status
Restoring onto another machine works the same way (install code and Node, copy the file to DB_PATH). Migrations in
the snapshot are checksummed: if the code is newer than the snapshot, the missing migrations are applied at boot; run
code that is older than the snapshot's last migration (upstream-admin status shows it) only if you know the schema
still fits, the server does not check for that.
8. Upgrading
/opt/upstream is a root-owned git checkout of https://git.vanvikinternet.no/jvanvik/Upstream.git (plain HTTPS, no credentials in the URL
or in .git/config; the script refuses an http(s):// origin URL that carries a user or token). Updates go through deploy/update.sh:
sudo sh /opt/upstream/deploy/update.sh --dry-run # fetch, list the commits and new migrations, change nothing
sudo sh /opt/upstream/deploy/update.sh # the real thing
sudo sh /opt/upstream/deploy/update.sh --test # also run `npm test` on the new code before the restart
In order, stopping at the first failure (once the service is stopped, the script always starts it again):
- Tools:
git,curl,npm,systemctland the admin wrapper must be on the PATH (runuser,tarandmktemptoo with--test). A missing one is exit 2 before anything is touched (withoutcurlthe health check could never pass and a good release would be rolled back). - Refuses on a dirty tree (tracked changes or untracked files that are not git-ignored;
node_modules/anddata/are ignored), on a checkout that is not onmain(detached HEAD included), on anhttp(s)://origin whose authority carriesuser:password@ortoken@(anssh://git@host/...orgit@host:pathremote is fine, so is an@later in the path), and while another update runs (lock directory/run/upstream-update.lock). git fetch origin main, no prompt. Not a fast-forward of the checkout (history rewritten, local commits): refuses, touches nothing. Nothing new: "already up to date", exit 0, no stop, no backup and no restart.- Only with
--test:git archiveof the new rev into a temp directory,npm ci --omit=dev --ignore-scriptsandnpm testthere as the service user (runuser -u upstream,HOMEand the npm cache inside the temp directory; never as root, never in/opt/upstream). The live service keeps serving meanwhile. Red: exit 1, nothing was stopped or changed. (Fetched test code can still read what the service user can read, for example the database;--testis for your own commits.) systemctl stop upstream. Why before the merge: the running processrequire()s server modules lazily, so a process that keeps serving while the tree changes under it can load new files into old code. From here the tree and the process never disagree; the downtime is steps 5-8.- Verified snapshot of the stopped (final) database:
upstream-admin backup /var/lib/upstream/backups --keep 14, thenverify-backupon the file it wrote. No verified snapshot, no update: the service is started again, exit 2, nothing changed. The snapshot path is printed at the end. git merge --ff-only.npm ci --omit=dev --ignore-scriptsonly ifpackage.json/package-lock.jsonchanged (ornode_modulesis missing). No package install script runs as root (wsneeds none; if a future dependency needs one, run it by hand).systemctl start upstream, then wait forhttp://127.0.0.1:<PORT>/healthz(PORT from/etc/upstream/upstream.env, else 8080) for up to 30 s (HEALTH_TIMEOUTor--timeout N).- Rollback on any failure from step 6 on:
systemctl stop(a crash-looping new process must not run against the old tree),git reset --hard <old rev>(the tree was clean, so nothing is lost; the branch stays onmain),npm ciagain if the lock file had changed,systemctl start, wait for/healthzagain. - Prints
old -> newrevs and the commits that went out.
Exit codes: 0 updated or already current, 1 failed and the old version answers again (or --test was red), 2 refused or usage error
(nothing changed), 3 failed and the old version does not answer either (the output names the snapshot; journalctl -u upstream -n 50).
Things the script cannot undo: migrations. A migration that ran at the new version's first start stays applied when the code is rolled
back. When the update contains new files in server/migrations/ the script says so up front (NEW MIGRATIONS ...) and, after a rollback, prints
the snapshot to restore if the old code misbehaves on the newer schema (section 7: stop, restore, start; players lose what they did since the
snapshot). Migration 005_site_garrison.sql is called out specially: it adds site_stationed (units garrisoned in backbone landing sites).
The previous code knows nothing of that table: it neither lists those units, nor charges their power, nor lets them go home. A code-only rollback
is therefore clean only while nobody has garrisoned a site since the new code started; otherwise restore the snapshot. A rollback target is always
the previous revision only, one step. An interrupt (Ctrl-C, SIGTERM) after the stop starts the service again (before the merge, on the untouched old
tree) or rolls back like any other failure (after it); SIGKILL or a power cut mid-update cannot be caught: then git log -1, git status and
systemctl status upstream tell where it stopped (if the service is down: systemctl start upstream; if the tree is already new but the service
ran the old code, systemctl restart upstream). The stop/start drops open WebSockets; clients reconnect on their own.
Settings: UPSTREAM_DIR, UPSTREAM_UNIT, UPSTREAM_ENV_FILE, UPSTREAM_ADMIN, BACKUP_DIR, BACKUP_KEEP, UPSTREAM_REMOTE,
UPSTREAM_SERVICE_USER (the user --test runs as, default upstream), UPSTREAM_BRANCH, HEALTH_URL, HEALTH_TIMEOUT, LOCK_DIR (sh update.sh --help lists them). It needs git, curl, npm, systemctl and
upstream-admin (section 3) on the PATH of root (checked up front).
What was run: test/deploy-update.test.js drives the real script with sh against a scratch bare origin and checkout and stubs for
systemctl, curl, npm, upstream-admin and sleep: up to date, update order (stop, snapshot, merge, start: the tree is old at the stop and new at the start), npm ci --ignore-scripts only when the lock
changed, --test (isolated copy, via a stub runuser), missing tools, dirty tree, credentials in the URL (and the ssh / @-in-path URLs that are allowed), detached HEAD, non-fast-forward, concurrent run, --dry-run, failing or unverifiable
backup, unhealthy new version (rollback, migrations warning, dependency restore), red tests, a rollback that does not recover (exit 3), PORT from the
env file. sh -n, bash --posix -n and zsh -n pass; shellcheck and dash were not available on the dev machine. Not run: the script on
the real box, against real systemd, a real npm ci / npm test, or a real /healthz.
Updating without git (rsync from a workstation, the way section 3 first installed it) still works; do the same by hand:
systemctl stop upstream # 1. stop FIRST: the old process lazily loads modules from the tree
upstream-admin backup /var/lib/upstream/backups --keep 14 # 2. snapshot of the stopped database
rsync -a --delete --exclude node_modules --exclude .git --exclude data ./ root@<lxc>:/opt/upstream/ # 3. new code (from your workstation)
cd /opt/upstream && npm ci --omit=dev --ignore-scripts && chown -R root:root /opt/upstream # 4. dependencies, ownership
systemctl start upstream # 5. migrations apply at boot
journalctl -u upstream -n 30 # 6. "applied migrations: ..." then the listen banner
curl -i http://127.0.0.1:8080/healthz
- Migrations are
server/migrations/NNN_name.sql, applied in order inside a transaction each, recorded with a checksum and never edited after they shipped. If one that was applied has changed, the server refuses to start (migration 001_init.sql was edited after it was applied): restore that file from git and ship a new numbered migration instead.upstream-admin statuslists applied and pending migrations;upstream-admin migrateapplies them without starting the game. - Each migration runs in its own transaction: a failing one rolls back and the server exits, earlier ones of the same boot stay applied. Going back to the old code after a migration committed means restoring the pre-upgrade snapshot (section 7).
- Updated unit files:
install -m 0644 ...,systemctl daemon-reload,systemctl restart upstream. - Players reconnecting after the restart need no action; clients get
serverTimein every response and refetch.
9. Health, logs, checks
curl -i https://upstream.example.com/healthz # 200 ok (only says the process answers HTTP)
curl -I https://upstream.example.com/ # 200 + Content-Security-Policy
upstream-admin status # database, migrations, events pending / OVERDUE
journalctl -u upstream -f # live log; -p err for failures only
journalctl -u upstream --since "1 hour ago" | grep -E "failed|refused"
The log has: applied migrations, created world, caught-up events, refused rate_limited(...) ip=..., and
event N (type) failed: ... for a handler that threw (that event is marked failed and the queue moves on; list them
with upstream-admin events --status failed). Player IP addresses appear in the log; passwords and tokens never do.
A healthy server has no overdue events in status: the timer processes events within milliseconds of their due time.
10. Troubleshooting
| Symptom | Cause and fix |
|---|---|
Service dies at start: No such built-in module: node:sqlite |
Node is older than 22.13, or the distro's Node 20 is first on the PATH. node --version, which -a node, section 1. |
ExperimentalWarning: SQLite is an experimental feature in the journal |
Harmless. Printed once per start on some Node versions. |
SQLITE_CANTOPEN / unable to open database file |
DB_PATH's directory is not writable by upstream: wrong path in upstream.env, /var/lib/upstream created by hand with other ownership (chown -R upstream:upstream /var/lib/upstream), or ProtectSystem=strict enabled without ReadWritePaths=/var/lib/upstream. |
attempt to write a readonly database / unable to open database file right after using the CLI |
admin.js was run as root and left root-owned game.db-wal / -shm. Stop the game, chown upstream:upstream /var/lib/upstream/game.db*, start. Use upstream-admin, which switches user. |
migration NNN_x.sql was edited after it was applied |
An applied migration changed on disk (checksum mismatch). Restore it from git; changes go in a new migration. |
UPSTREAM failed to start: ... with EADDRINUSE |
Another process on the port (ss -ltnp | grep 8080) or two copies. Change PORT and the NPM forward port together. |
IPv6 is not available (...); falling back to 0.0.0.0 |
The kernel has no IPv6. IPv4 keeps working; set HOST=0.0.0.0 to silence it. |
status=226/NAMESPACE |
The optional namespace hardening is on in an LXC without Nesting: enable Options > Features > Nesting and restart the CT, or comment the block out. |
Cannot find module 'ws' |
npm ci --omit=dev was not run in /opt/upstream. |
| 502 Bad Gateway from NPM | Upstream unreachable: systemctl status upstream, curl -i http://<lxc>:8080/healthz from the NPM host, forward IP/port, firewall. |
| Page loads but a new deploy still shows the old client | NPM Cache Assets is on, or a service layer ignores no-cache. Switch it off, hard-reload. |
Signup answers 403 signup_closed |
SIGNUP_CODE is set. Send {"invite": "<code>"} in the signup body, or unset it and restart. |
Login/signup answers 429 rate_limited for many players |
They all arrive from one address: the proxy is not trusted so the per-IP budget is shared (log says ip=<the proxy>). Set TRUSTED_PROXIES. Login and signup allow about one request per 10 s per address, burst 8; a whole classroom behind one NAT can hit that. |
| Game seems frozen: builds never finish | upstream-admin status: OVERDUE events mean the scheduler is not running (server stopped or stuck) or handlers fail: upstream-admin events --status failed, journal event N ... failed. A restart catches up everything in order. |
| Banned user still plays | They cannot: ban revokes every token and blocks login. Check the username spelling (upstream-admin users --banned). |
database is locked from the CLI |
The game held a write lock longer than the 5 s busy timeout. Retry. |
upstream-backup.service failed |
journalctl -u upstream-backup. Typical: disk full, or the integrity check of the new snapshot failed (then nothing was kept and nothing pruned; check the disk and the game.db itself). |
| Backups not appearing | systemctl list-timers upstream-backup.timer; the timer must be enabled (systemctl enable --now upstream-backup.timer), not just installed. |
| World reset by mistake | Restore the snapshot taken before (section 7), or the one reset-world --backup wrote. |
| WebSocket upgrade answered 426 / 403 / 429 | 426: Websockets Support off in NPM. 403: origin check, see ALLOWED_ORIGINS and the notes in section 5. 429: more than MAX_CONN_PER_IP sockets from one address. |