Upstream/docs/DEPLOY.md
jvanvik d7a27db563 admin: reset-world --speed X (speed can only change on an emptied world)
Speed is stored scaled into rates and due times, so changing it under live rows would jump resources and mis-time events. reset-world now takes --speed to set the new world's speed in the same transaction.

Tested: test/admin-cli.test.js (speed asserted after reset), npm test green. NOT tested beyond that.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2MJaQ5LXAHzGnTDgceT14
2026-10-12 19:20:45 +02:00

531 lines
42 KiB
Markdown

# Deploying UPSTREAM
UPSTREAM is one Node process: it serves the static client from `public/`, the REST API under `/api/v1/` and a
WebSocket push channel at `/ws` (docs/PROTOCOL.md). Unlike NETWALL it **keeps state**: one SQLite file, owned by that process. A restart loses
nothing: overdue events are caught up in order before the server accepts traffic.
```
phone / laptop --https--> nginx Proxy Manager (TLS, :443) --http (+ WebSocket)--> Debian 13 LXC: node :8080
/opt/upstream (code, read-only)
/var/lib/upstream (game.db, backups/)
```
Same shape as NETWALL (`docs/DEPLOY.md` there): same proxy, same hardening notes, same env vars for the network
layer. The differences are the Node version (`node:sqlite` needs 22.13+), a state directory and backups.
**What was and was not run while writing this** (so you know what to check on the real box):
* **Run:** `systemd-analyze verify` on the unit files (syntax and directives only; the two off-site units: only the expected "not executable" warning for a script path that does not exist on the dev machine), the CLI `tools/admin.js`
and its tests on a dev machine (Node 24.21), a backup, verify and prune cycle, `sh -n` on `deploy/upstream-admin`.
Re-checked on 2026-10-11 with the code as it is now (Node 24.21, scratch directory, no systemd): first boot with `DB_PATH`/`PORT`/`HOST`
(migrations 001-004 applied, `created world 1`), `/healthz`, `/` headers, signup + build over the API, `admin.js` status / worlds / users /
operators / inspect-town / events / replay-events / backup / verify-backup / ban / unban / grant / end-world (refusal without `--yes`),
SIGTERM and restart (state, queue and password kept, no migrations re-applied; tokens revoked by `ban` stay revoked), SIGKILL then restart after a build fell due (`caught up 1 overdue event(s)`, the build level was applied), and a restore by copying a snapshot to a fresh `DB_PATH` and starting on it (section 7: it logged `caught up 1 overdue event(s)`).
* **Run, ops scripts (dev machine, stubs, no systemd):** `deploy/update.sh` and `deploy/upstream-offsite` and `admin.js end-world` through
`test/deploy-update.test.js`, `test/deploy-offsite.test.js`, `test/admin-endworld.test.js`; sections 7 and 8 say exactly what each covers.
* **Taken from NETWALL's docs, not re-run for UPSTREAM:** everything about nginx Proxy Manager (labels, header
behaviour, "Client addresses"). NETWALL's guide records what was observed against NPM v2.16.0.
* **Never run:** a Proxmox LXC, a started unit under systemd, the timer firing on its own, a Node install from the
tarball or NodeSource on Debian 13, `rrsync`, Let's Encrypt, a restore on the real box. Treat those steps as a
checklist to confirm, and fix this file when reality differs.
## 1. Requirements
* Debian 13 (trixie) LXC with its own IPv4 (optionally IPv6) address the proxy can reach. 1 core, 512 MB RAM and 4 GB
disk are plenty for a small world; the database grows with the event and notification history (measure it with
`upstream-admin status`), and snapshots multiply it by the retention count.
* **Node 22.13 or newer.** Debian 13 ships `nodejs` **20.19.2** (checked on tracker.debian.org: stable has 20.19.2,
testing and unstable have 24.21). Node 20 has no `node:sqlite`, so `apt install nodejs` is **not enough**. `package.json`
says `"engines": {"node": ">=22.13"}`; 22.5 to 22.12 only have `node:sqlite` behind `--experimental-sqlite`, which
the unit does not pass. Development is on Node 24.21.
* A DNS name for the game (for example `upstream.example.com`) pointing at the proxy.
### Getting a suitable Node (pick one)
**A. Official tarball (no third-party apt repo, nothing piped into a shell).** Pick a current 24.x (Active LTS) or
22.x from https://nodejs.org/en/download and put its version in `V`:
```sh
apt update && apt install -y curl xz-utils ca-certificates
V=24.21.0 # example: use the current 24.x (or >= 22.13) you picked
ARCH=linux-x64 # linux-arm64 on ARM hosts (uname -m: aarch64)
cd /tmp
curl -fsSLO https://nodejs.org/dist/v$V/node-v$V-$ARCH.tar.xz
curl -fsSLO https://nodejs.org/dist/v$V/SHASUMS256.txt
grep " node-v$V-$ARCH.tar.xz\$" SHASUMS256.txt | sha256sum -c - # must print: ... OK
tar -C /opt -xJf node-v$V-$ARCH.tar.xz
ln -sfn /opt/node-v$V-$ARCH /opt/node
ln -sf /opt/node/bin/node /usr/local/bin/node
ln -sf /opt/node/bin/npm /usr/local/bin/npm
ln -sf /opt/node/bin/npx /usr/local/bin/npx
```
(For extra assurance also verify the signature of `SHASUMS256.txt` as described on nodejs.org.) An upgrade later is
a new tarball and `ln -sfn` to it, then `systemctl restart upstream`. `upstream.service` starts `/usr/bin/env node`, and
systemd's default PATH lists `/usr/local/bin` before `/usr/bin`, so the tarball wins over any distro Node.
**B. NodeSource apt repository** (follows their documentation; not run here). It puts Node in `/usr/bin/node`:
```sh
apt install -y ca-certificates curl gnupg
install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_24.x nodistro main" > /etc/apt/sources.list.d/nodesource.list
apt update && apt install -y nodejs # includes npm; do not also install Debian's npm package
```
**Verify either way** (all three must succeed):
```sh
node --version # v22.13.0 or newer
node -e "require('node:sqlite'); console.log('node:sqlite ok')"
which -a node # the first one is the one the unit will use
```
On 22.x/24.x Node may print `ExperimentalWarning: SQLite is an experimental feature` to stderr; that is harmless (it
shows up in the journal once per start). `No such built-in module: node:sqlite` means Node is too old (or it is the
distro's 20.x earlier on the PATH). If `node:sqlite` ever misbehaves, `DECISIONS.md` D1 describes the one-file swap
to `better-sqlite3` in `server/db.js`; that would add a native dependency and a build toolchain to the LXC.
## 2. Create the LXC (Proxmox)
Container creation, the Proxmox UI way (CLI `pct create` works too):
* Template `debian-13-standard`, **unprivileged**, hostname `upstream`, 1 core, 512 MB RAM (+256 MB swap), 4-8 GB disk.
* Network: a static address (or a DHCP reservation) on a bridge the NPM host can reach. The proxy forwards to that
address, so it must not change.
* Features: **Nesting** is optional; tick it only if you want the namespace-based hardening block in the unit
(section 3). Everything else works without it.
* Start on boot: yes. Timezone does not matter: the game uses UTC ms, the backup timer uses the container's clock.
* Proxmox's own CT backups (`vzdump`) are not a substitute for the snapshots in section 7: a running-container copy of
a SQLite file is crash-consistent at best. Use them for the OS and let the verified snapshots be the data backup.
## 3. Install in the LXC
Run as root inside the container (Node from section 1 is already in place).
```sh
apt install -y rsync # for the copy below (or use git)
# a dedicated, login-less user that only runs the game
useradd --system --home-dir /var/lib/upstream --shell /usr/sbin/nologin upstream
# code: read-only for the service user, owned by root
mkdir -p /opt/upstream
# from your workstation:
# rsync -a --delete --exclude node_modules --exclude .git --exclude data --exclude e2e ./ root@<lxc>:/opt/upstream/
# (needs server/, public/, tools/, deploy/, docs/, package.json, package-lock.json; `test/` is optional)
# or, the way the live box is run, a git checkout that deploy/update.sh updates later (section 8):
# git clone https://git.vanvikinternet.no/jvanvik/Upstream.git /opt/upstream (plain HTTPS URL, no token in it)
cd /opt/upstream
npm ci --omit=dev # only "ws"; follows the lockfile exactly
chown -R root:root /opt/upstream
# the admin helper on the PATH
install -m 0755 /opt/upstream/deploy/upstream-admin /usr/local/bin/upstream-admin
# optional settings / secrets file (read by the unit and by upstream-admin)
install -d -m 0750 -o root -g upstream /etc/upstream
install -m 0640 -o root -g upstream /dev/null /etc/upstream/upstream.env
# edit it: SIGNUP_CODE=choose-a-long-random-string
# ALLOWED_ORIGINS=https://upstream.example.com
# the units
install -m 0644 /opt/upstream/deploy/upstream.service /opt/upstream/deploy/upstream-backup.service /opt/upstream/deploy/upstream-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now upstream upstream-backup.timer
systemctl status upstream
# off-box copy of the snapshots is optional and not installed here: section 7
journalctl -u upstream -f
```
The first start prints `applied migrations: 001_init.sql, ...`, `created world 1` and the listen banner with
`database: /var/lib/upstream/game.db`. Quick manual run as the service user instead of systemd (Ctrl-C to stop):
`runuser -u upstream -- env PORT=8080 DB_PATH=/var/lib/upstream/game.db node /opt/upstream/server/index.js`.
### What the units do
* `deploy/upstream.service`: user `upstream`, `WorkingDirectory=/opt/upstream`, `StateDirectory=upstream` (systemd creates
`/var/lib/upstream` owned by that user, mode 0700, so the game can write nowhere else), `Restart=on-failure`,
`EnvironmentFile=-/etc/upstream/upstream.env` (overrides the `Environment=` lines), `UMask=0077` (database and
snapshots hold password hashes: owner-only).
* Hardening that works in any container: `NoNewPrivileges`, empty capability set (port 8080 needs none),
`RestrictAddressFamilies`, `RestrictRealtime`, `RestrictSUIDSGID`, `LockPersonality`, `SystemCallArchitectures`.
No `MemoryDenyWriteExecute` (V8 would crash).
* The namespace block (`ProtectSystem=strict`, `ReadWritePaths=/var/lib/upstream`, `PrivateTmp`, ...) is commented out.
In an unprivileged LXC it fails with `status=226/NAMESPACE` unless **Options > Features > Nesting** is on. Because the
app writes to disk, `ProtectSystem=strict` needs the `ReadWritePaths=` line that is already next to it in the unit;
without it the database cannot be opened.
* Shutdown: SIGTERM stops the scheduler timer, closes the listener and the database and exits 0 in about a second.
Even a SIGKILL or power cut is survivable (WAL replay on the next open, then overdue events are caught up).
* `deploy/upstream-backup.service` (oneshot) and `.timer`: see section 7. `deploy/upstream-offsite.service` and `.timer` (optional off-box copy) and
`deploy/update.sh` (section 8) are not installed by the steps above.
## 4. Configuration
Environment variables read by `node server/index.js`. Put changes in `/etc/upstream/upstream.env` (`KEY=value`, one per
line, no `export`) or in a `systemctl edit upstream` override, then `systemctl restart upstream`.
| Variable | Default (unit) | Meaning |
|---|---|---|
| `PORT` | `8080` | TCP port. Unprivileged, so no capabilities needed. `0` picks a free one (banner prints it). |
| `HOST` | `::` | Bind address. `::` = one dual-stack socket for IPv4 and IPv6 (falls back to `0.0.0.0` if the kernel has no IPv6). `127.0.0.1` = this container only. |
| `DB_PATH` | `/var/lib/upstream/game.db` | SQLite file. The code default is `./data/game.db`; the unit overrides it. Its directory is created if missing, it must be writable by the service user. `-wal` and `-shm` files appear next to it: always copy via a snapshot (section 7), never with `cp` while running. |
| `TRUST_PROXY` | `auto` | `auto`: believe `X-Forwarded-For` / `X-Forwarded-Host` only from loopback / private-range peers. `1`: always (then firewall the port). `0`: never. Used for the per-IP rate limits and the log. |
| `TRUSTED_PROXIES` | empty | Extra IPs / CIDRs `auto` believes: needed when the proxy reaches the LXC from a global IPv6, Tailscale / CGNAT `100.64.0.0/10` or another routable address. |
| `ALLOWED_ORIGINS` | empty | Extra origins for the WebSocket origin check, comma separated, with the port when it is not 80/443. The REST API uses bearer tokens (no cookies) and does not check origins. |
| `MAX_CONN_PER_IP` | `40` | Open WebSockets per client address (HTTP connections are not capped). `0` disables. |
| `WORLD_SPEED` | unset (1) | World speed multiplier, **first boot only**, when the database has no world yet. |
| `WORLD_RADIUS` | unset (50) | Map radius in tiles (coordinates `-R..R`, so 50 = 101 x 101), **first boot only**. |
| `WORLD_NAME` | unset (`World 1`) | Name of the first world, **first boot only**. |
| `SIGNUP_CODE` | empty | If set, `POST /auth/signup` needs `"invite": "<this>"` in the body or answers `signup_closed` (403). Empty = open signup. Keep it in `/etc/upstream/upstream.env`, not in the unit file. |
| `UPSTREAM_DIR`, `UPSTREAM_ENV_FILE` | `/opt/upstream`, `/etc/upstream/upstream.env` | Only for the `upstream-admin` wrapper. |
Fixed in code (options of `createServer()`): per-IP and per-token rate limits (login and signup are stricter: about one
per 10 s sustained, burst 8, per client address), 32 KB request bodies, idempotency keys pruned hourly.
### First boot
An empty or missing database is created, all migrations are applied, and **one world is created** from `WORLD_SPEED`,
`WORLD_RADIUS`, `WORLD_NAME` (seed random). After that those variables are ignored: changing them does not touch the
existing world. More worlds, a different seed, ending or resetting a world are admin tasks (section 6). The first
player to sign up becomes an ordinary operator; there is no special admin account, admin work is the CLI on the box.
## 5. nginx Proxy Manager
In NPM: **Hosts > Proxy Hosts > Add Proxy Host**.
| Tab / field | Value |
|---|---|
| Details: Domain Names | `upstream.example.com` |
| Details: Scheme | `http` |
| Details: Forward Hostname / IP | the LXC's address (for example `192.168.1.51`), not `127.0.0.1` when NPM runs elsewhere or in its own container |
| Details: Forward Port | `8080` |
| Details: Cache Assets | **off** (static files are `no-cache` and revalidated by ETag; the game must see new versions) |
| Details: Block Common Exploits | optional |
| Details: **Websockets Support** | **on.** The REST game works without it, but `/ws` (live push) fails with HTTP 426 when this is off. |
| Custom Locations | none needed |
| SSL | Request a new Let's Encrypt certificate, **Force SSL**, **HTTP/2** (HSTS optional) |
| Advanced (gear) | nothing required; optional gzip below |
Optional, Advanced box, to compress JS/JSON for phones (the Node server does not compress; NPM's own `gzip on` only
covers HTML):
```nginx
gzip on;
gzip_types text/css text/javascript application/javascript application/json application/manifest+json image/svg+xml;
```
Notes carried over from NETWALL's setup (observed there against NPM v2.16.0, not re-run for UPSTREAM):
* NPM sends `Host` **without the public port** and no `X-Forwarded-Host`. On a public port other than 443 the browser
`Origin` then differs from `Host`; list the origin with its port in `ALLOWED_ORIGINS` (matters for `/ws`).
* A proxy on a non-private address (global IPv6, Tailscale) is not believed by `TRUST_PROXY=auto`: every player then
shares one rate-limit budget and the log shows the proxy's address. Fix: `TRUSTED_PROXIES=<its address or CIDR>`.
* Clients on your own LAN can pick their address through NPM's `real_ip` handling, so the per-IP limits protect against
public abuse, not against LAN users.
* **Idle timeout:** nginx's `proxy_read_timeout` is 60 s by default (NETWALL's notes say 90 s for NPM; not re-checked). NETWALL never
idles because it streams 20 snapshots a second; UPSTREAM's socket is quiet for long stretches. `server/push.js` sends a protocol ping
every 30 s (`wsPingMs`), below either value, which is reasoning, not a measurement behind a real NPM.
Firewall and dual stack: identical to NETWALL (`docs/DEPLOY.md` there, section 6): only the proxy needs to reach port
8080. `curl -i http://127.0.0.1:8080/healthz` and `curl -gi 'http://[::1]:8080/healthz'` check both families.
## 6. Operating: `upstream-admin`
`tools/admin.js` is the operator CLI (Node core only, uses the game's own `db.js`, `state.js`, `rules.js`).
`deploy/upstream-admin` runs it as the service user against the service's database (it reads `DB_PATH` from
`/etc/upstream/upstream.env`). **Do not run `node tools/admin.js` as root**: SQLite would create `game.db-wal` and
`-shm` owned by root and the game could no longer open its database (symptom in section 10).
```sh
upstream-admin help
upstream-admin status # file, size, migrations, overdue events
upstream-admin worlds
upstream-admin create-world --name "Summer" --speed 2 --radius 60 --seed 1234 --end-in 90d
upstream-admin end-world 1 --yes # same as the natural end: final ranking, escrow back, events cancelled
upstream-admin users [--banned] # no hashes or tokens are ever printed
upstream-admin operators [--world 1]
upstream-admin ban mallory # sets banned, revokes all tokens: old tokens give 401, login 403
upstream-admin unban mallory
upstream-admin revoke-tokens ada # log out everywhere
upstream-admin inspect-operator ada # (--world N when the name exists in several worlds, --id N by operator id)
upstream-admin inspect-town 12 # resources now (rules.resourcesAt), buildings, queue, units if the table exists
upstream-admin events --pending [--type buildDone --town 12 --limit 100]
upstream-admin replay-events --until +6h # dry run: what the scheduler would process, in order
upstream-admin replay-events --until +6h --simulate # + run the real handlers on a scratch copy, report outcome and follow-ups
upstream-admin backup /var/lib/upstream/backups --keep 14
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db
upstream-admin grant 12 --fiber 5000 --capital 2000 --yes # testing only
upstream-admin reset-world 1 --yes-destroy --backup /var/lib/upstream/backups --new-seed [--speed 3] # --speed is the only way to change a world's speed: it needs an emptied world
upstream-admin migrate # apply pending migrations without starting the server
```
All commands take `--json` (machine output) and `--db FILE` (otherwise `$DB_PATH`, otherwise `./data/game.db`). Piping into `head` is fine: a closed
pipe (EPIPE) stops the output quietly, no stack trace, and the exit code is kept.
Exit codes: `0` ok, `1` failed, `2` usage error **or a destructive command missing its flag**, `3` not found.
Safety rules the CLI follows:
* Destructive or test-only commands refuse without a flag: `end-world` and `grant` need `--yes`; `reset-world`
needs `--yes-destroy` (without it, it prints what it would delete and exits 2). `ban`, `unban`, `revoke-tokens` are
reversible and need none.
* `end-world <id> --yes` ends a world exactly like the natural end: it runs the real `worldEnd` event handler (`server/events/worldEnd.js`,
payload `{"reason":"admin"}`). So the final ranking snapshot is stored (`GET /worlds/:id/results` becomes `available: true`, `winner`
null unless the backbone race had a winner already), open trade offers are closed and their escrow goes back to the POPs, **every
other pending event of that world is cancelled** (builds in progress, training, movements on the road simply stop: the world is
frozen), status becomes `ended` with `end_at` = now, and every operator gets a `world_ended` notification. Events of that world that
were already due are processed first, in order, like the server's catch-up. Without `--yes` it refuses (exit 2) and says how many
pending events and open offers it would cancel and close. A world that is already ended is left alone (exit 0). Other worlds are untouched.
It works on a running or a stopped server; the CLI is a separate process, so a running server pushes nothing over WebSocket for the
notification rows it wrote: open clients learn from their next request (`world_ended` error) or the next `/me`. A world that an
older version of this command "ended" has no snapshot; the command says so and cannot repair it (reset the world or leave it).
* `reset-world` deletes the world's game rows (everything with a `world_id`, plus tables pointing at them: towns,
buildings, events, operators, alliances, messages, ... discovered from the schema, so tables added by later stages
are covered) and **keeps accounts and tokens**; players must join the world again. It resets `start_at`, clears
`end_at` and sets the world running. It rolls back if it would leave dangling references. Stop the game first
(`systemctl stop upstream`): the server does not know the world was swept while it is running. Use `--backup DIR`.
* Everything runs on a running server's database: writes are single short `BEGIN IMMEDIATE` transactions, the
connection is always closed, and SQLite's 5 s busy timeout covers collisions with the game (`database is locked`
from the CLI just means: try again).
* `grant` adds to the town's stored amounts (valid at its `res_at`); the lazy model carries them forward. Amounts above
the Rack hall cap are kept but stop growing, like a lowered cap.
* `inspect-town` shows values as of now **ignoring** overdue events; it says so when there are any (server stopped).
## 7. Backups and restore
`upstream-backup.timer` runs `upstream-backup.service` daily at 04:15 (10 min jitter, `Persistent=true`). It runs
`tools/admin.js backup /var/lib/upstream/backups --keep 14` as the service user. A snapshot is:
1. `VACUUM INTO` a temporary `.partial-*` file: a consistent copy as of one moment, safe while the game runs (WAL readers do
not block the writer; the game keeps its own 5 s busy timeout);
2. verified: opened read-only, `PRAGMA integrity_check` = `ok`, `worlds` and `schema_migrations` present;
3. published under a dated name `upstream-YYYYMMDDTHHMMSSZ.db` (UTC) with a hard link, so an existing file is never
replaced (a same-second collision becomes `...Z-1.db`), mode 0600;
4. only then older snapshots are pruned: `--keep N` keeps the newest N, `--keep-days D` keeps those younger than D days,
both = survive if either keeps it, neither = no pruning. The newest snapshot and any file not matching the naming
pattern are never deleted. A failed verification leaves nothing behind and prunes nothing, and the unit fails
(`systemctl status upstream-backup`, `journalctl -u upstream-backup`).
Snapshots are plain SQLite files (journal mode `delete`), not `-wal` dependent: copy a single file.
```sh
systemctl list-timers upstream-backup.timer # next run
systemctl start upstream-backup # one right now
journalctl -u upstream-backup -n 20
ls -l /var/lib/upstream/backups
```
Change the schedule or retention with `systemctl edit upstream-backup.timer` / `upstream-backup.service` (for every
6 hours: `OnCalendar=*-*-* 00/6:15:00` and a larger `--keep`). Before an upgrade or a `reset-world`, take one by hand.
### Off-box copy
Snapshots on the same disk die with the container. Two ways to get them off the box; use one. Both only ever copy files that the backup
unit already verified, and neither deletes anything on the far side (retention over there is yours).
**A. Pull from another machine** (NAS, workstation, the Proxmox host), for example from cron or a timer there:
```sh
rsync -a --ignore-existing root@<lxc>:/var/lib/upstream/backups/ /srv/backups/upstream/
```
`--ignore-existing` makes it append-only on your side (nothing already stored is overwritten, and snapshots pruned on
the LXC stay on the backup host until you decide otherwise). Better than a root key: a dedicated key restricted in
`authorized_keys` to `command="rrsync -ro /var/lib/upstream/backups",restrict ...` (`rrsync` comes with the `rsync`
package; not run here). The files are owner-only (`upstream`), so the reading user needs root or a group ACL.
**B. Push from the box: `upstream-offsite.service` + `.timer`** (`deploy/`). Disabled until you configure it; nothing here is installed by
section 3. The script `deploy/upstream-offsite` picks the newest snapshot (by modification time), runs `verify-backup` on it (a snapshot that
does not verify is never copied), copies it with `rsync` to a directory (`scp` is not supported: it cannot refuse to overwrite, `OFFSITE_METHOD=scp` is rejected with exit 2), **never overwrites** a file of the same
name there, and re-reads the far copy and compares checksums. The timer fires daily at 05:00 (+ up to 10 min), after the backup timer (04:15).
It runs as the `upstream` user and fails visibly (`journalctl -u upstream-offsite`).
Set it up (on the far side first, then the box):
```sh
# far side (backup host): an account that can only write into one directory, no shell, append-only by convention
useradd --system --create-home --shell /usr/sbin/nologin upstream-backup # or a NAS user
install -d -o upstream-backup -m 0700 /srv/backups/upstream
# ~upstream-backup/.ssh/authorized_keys, one line (the key generated below):
# restrict,command="rrsync /srv/backups/upstream" ssh-ed25519 AAAA... upstream-offsite
# the box (as root): a key only for this, owned by the service user (ssh refuses keys readable by others)
install -d -m 0750 -o root -g upstream /etc/upstream
ssh-keygen -t ed25519 -N '' -C upstream-offsite -f /etc/upstream/offsite_key
chown upstream:upstream /etc/upstream/offsite_key /etc/upstream/offsite_key.pub; chmod 0600 /etc/upstream/offsite_key
# the far side's host key, checked by eye: compare with `ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub` over there
ssh-keyscan -t ed25519 backup-host.example > /etc/upstream/offsite_known_hosts
ssh-keygen -lf /etc/upstream/offsite_known_hosts
cat >> /etc/upstream/upstream.env <<'EOT'
OFFSITE_DEST=upstream-backup@backup-host.example:/
OFFSITE_SSH_KEY=/etc/upstream/offsite_key
EOT
# with rrsync the destination path is relative to its root, hence ":/" (or ":" ). A plain directory also works:
# OFFSITE_DEST=/mnt/nas/upstream (a mounted share; then no key or known_hosts is needed)
install -m 0644 /opt/upstream/deploy/upstream-offsite.service /opt/upstream/deploy/upstream-offsite.timer /etc/systemd/system/
systemctl daemon-reload
systemctl start upstream-offsite # one run now: journalctl -u upstream-offsite -n 20
systemctl enable --now upstream-offsite.timer
```
Settings (all in `/etc/upstream/upstream.env`): `OFFSITE_DEST` (required; without it every run is skipped quietly by the unit's
`ExecCondition`, so installing the timer early is harmless), `OFFSITE_METHOD` (only `rsync`; a far side without rsync needs `rrsync`/a different tool, scp was removed because it would overwrite), `OFFSITE_SSH_KEY`, `OFFSITE_KNOWN_HOSTS` (default `/etc/upstream/offsite_known_hosts`),
`OFFSITE_SSH_OPTS` (for example `-p 2222`). Host keys are always checked and ssh never prompts (`BatchMode=yes`). To copy right after each
backup instead of on its own timer: `systemctl edit upstream-backup` and add `[Unit]` `OnSuccess=upstream-offsite.service`.
The key on the box can write to the far directory; make that directory the only thing it can reach (the `restrict,command="rrsync ..."`
line) and prune old files over there with a job that lives there, so a compromised game box cannot delete the history.
**Test it once for real**: check that the file arrived and that `upstream-admin verify-backup` accepts the far copy (a restore from it is section 7). `upstream-offsite` was exercised with a local directory as the destination (rsync, checksum
compare, no-overwrite, refusing an unverified snapshot); the ssh transport, `rrsync` and the units under systemd were **not** run.
### Restore
The restore is a file copy, and the time model makes it clean: the restored world simply fast-forwards. Every event that
fell due since the snapshot is processed in order at start-up, each at its own due time. What is lost is what players did after
the snapshot (builds started, signups, messages) and their tokens: players log in again.
```sh
systemctl stop upstream-backup.timer upstream
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db # exit 0 or stop here
mkdir -p /var/lib/upstream/pre-restore
mv /var/lib/upstream/game.db* /var/lib/upstream/pre-restore/ 2>/dev/null # keep the old one, incl. -wal/-shm
install -o upstream -g upstream -m 0600 /var/lib/upstream/backups/upstream-20261011T041500Z.db /var/lib/upstream/game.db
systemctl start upstream upstream-backup.timer
journalctl -u upstream -n 20 # look for "caught up N overdue event(s)"
upstream-admin status
```
Restoring onto another machine works the same way (install code and Node, copy the file to `DB_PATH`). Migrations in
the snapshot are checksummed: if the code is newer than the snapshot, the missing migrations are applied at boot; run
code that is **older** than the snapshot's last migration (`upstream-admin status` shows it) only if you know the schema
still fits, the server does not check for that.
## 8. Upgrading
`/opt/upstream` is a root-owned git checkout of `https://git.vanvikinternet.no/jvanvik/Upstream.git` (plain HTTPS, no credentials in the URL
or in `.git/config`; the script refuses an `http(s)://` origin URL that carries a user or token). Updates go through `deploy/update.sh`:
```sh
sudo sh /opt/upstream/deploy/update.sh --dry-run # fetch, list the commits and new migrations, change nothing
sudo sh /opt/upstream/deploy/update.sh # the real thing
sudo sh /opt/upstream/deploy/update.sh --test # also run `npm test` on the new code before the restart
```
In order, stopping at the first failure (once the service is stopped, the script always starts it again):
0. **Tools**: `git`, `curl`, `npm`, `systemctl` and the admin wrapper must be on the PATH (`runuser`, `tar` and `mktemp` too with `--test`).
A missing one is exit 2 before anything is touched (without `curl` the health check could never pass and a good release would be rolled back).
1. **Refuses** on a dirty tree (tracked changes or untracked files that are not git-ignored; `node_modules/` and `data/` are ignored), on a
checkout that is not on `main` (detached HEAD included), on an `http(s)://` origin whose authority carries `user:password@` or `token@`
(an `ssh://git@host/...` or `git@host:path` remote is fine, so is an `@` later in the path), and while another update runs (lock directory
`/run/upstream-update.lock`).
2. `git fetch origin main`, no prompt. Not a fast-forward of the checkout (history rewritten, local commits): refuses, touches nothing.
Nothing new: "already up to date", exit 0, **no stop, no backup and no restart**.
3. Only with `--test`: `git archive` of the **new** rev into a temp directory, `npm ci --omit=dev --ignore-scripts` and `npm test` there **as the
service user** (`runuser -u upstream`, `HOME` and the npm cache inside the temp directory; never as root, never in `/opt/upstream`). The live
service keeps serving meanwhile. Red: exit 1, nothing was stopped or changed. (Fetched test code can still read what the service user can read,
for example the database; `--test` is for your own commits.)
4. `systemctl stop upstream`. **Why before the merge:** the running process `require()`s server modules lazily, so a process that keeps serving
while the tree changes under it can load new files into old code. From here the tree and the process never disagree; the downtime is steps 5-8.
5. **Verified snapshot** of the stopped (final) database: `upstream-admin backup /var/lib/upstream/backups --keep 14`, then `verify-backup` on the
file it wrote. No verified snapshot, no update: the service is started again, exit 2, nothing changed. The snapshot path is printed at the end.
6. `git merge --ff-only`.
7. `npm ci --omit=dev --ignore-scripts` **only if** `package.json` / `package-lock.json` changed (or `node_modules` is missing). No package install
script runs as root (`ws` needs none; if a future dependency needs one, run it by hand).
8. `systemctl start upstream`, then wait for `http://127.0.0.1:<PORT>/healthz` (PORT from `/etc/upstream/upstream.env`, else 8080) for up to
30 s (`HEALTH_TIMEOUT` or `--timeout N`).
9. **Rollback** on any failure from step 6 on: `systemctl stop` (a crash-looping new process must not run against the old tree), `git reset --hard
<old rev>` (the tree was clean, so nothing is lost; the branch stays on `main`), `npm ci` again if the lock file had changed, `systemctl start`,
wait for `/healthz` again.
10. Prints `old -> new` revs and the commits that went out.
Exit codes: `0` updated or already current, `1` failed and the old version answers again (or `--test` was red), `2` refused or usage error
(nothing changed), `3` failed and the old version does **not** answer either (the output names the snapshot; `journalctl -u upstream -n 50`).
Things the script cannot undo: **migrations**. A migration that ran at the new version's first start stays applied when the code is rolled
back. When the update contains new files in `server/migrations/` the script says so up front (`NEW MIGRATIONS ...`) and, after a rollback, prints
the snapshot to restore if the old code misbehaves on the newer schema (section 7: stop, restore, start; players lose what they did since the
snapshot). **Migration `005_site_garrison.sql`** is called out specially: it adds `site_stationed` (units garrisoned in backbone landing sites).
The previous code knows nothing of that table: it neither lists those units, nor charges their power, nor lets them go home. A code-only rollback
is therefore clean only while nobody has garrisoned a site since the new code started; otherwise restore the snapshot. A rollback target is always
the previous revision only, one step. An interrupt (Ctrl-C, SIGTERM) after the stop starts the service again (before the merge, on the untouched old
tree) or rolls back like any other failure (after it); SIGKILL or a power cut mid-update cannot be caught: then `git log -1`, `git status` and
`systemctl status upstream` tell where it stopped (if the service is down: `systemctl start upstream`; if the tree is already new but the service
ran the old code, `systemctl restart upstream`). The stop/start drops open WebSockets; clients reconnect on their own.
Settings: `UPSTREAM_DIR`, `UPSTREAM_UNIT`, `UPSTREAM_ENV_FILE`, `UPSTREAM_ADMIN`, `BACKUP_DIR`, `BACKUP_KEEP`, `UPSTREAM_REMOTE`,
`UPSTREAM_SERVICE_USER` (the user `--test` runs as, default `upstream`), `UPSTREAM_BRANCH`, `HEALTH_URL`, `HEALTH_TIMEOUT`, `LOCK_DIR` (`sh update.sh --help` lists them). It needs `git`, `curl`, `npm`, `systemctl` and
`upstream-admin` (section 3) on the PATH of root (checked up front).
**What was run:** `test/deploy-update.test.js` drives the real script with `sh` against a scratch bare origin and checkout and stubs for
`systemctl`, `curl`, `npm`, `upstream-admin` and `sleep`: up to date, update order (stop, snapshot, merge, start: the tree is old at the stop and new at the start), `npm ci --ignore-scripts` only when the lock
changed, `--test` (isolated copy, via a stub `runuser`), missing tools, dirty tree, credentials in the URL (and the ssh / `@`-in-path URLs that are allowed), detached HEAD, non-fast-forward, concurrent run, `--dry-run`, failing or unverifiable
backup, unhealthy new version (rollback, migrations warning, dependency restore), red tests, a rollback that does not recover (exit 3), `PORT` from the
env file. `sh -n`, `bash --posix -n` and `zsh -n` pass; `shellcheck` and `dash` were not available on the dev machine. **Not run:** the script on
the real box, against real systemd, a real `npm ci` / `npm test`, or a real `/healthz`.
Updating **without git** (rsync from a workstation, the way section 3 first installed it) still works; do the same by hand:
```sh
systemctl stop upstream # 1. stop FIRST: the old process lazily loads modules from the tree
upstream-admin backup /var/lib/upstream/backups --keep 14 # 2. snapshot of the stopped database
rsync -a --delete --exclude node_modules --exclude .git --exclude data ./ root@<lxc>:/opt/upstream/ # 3. new code (from your workstation)
cd /opt/upstream && npm ci --omit=dev --ignore-scripts && chown -R root:root /opt/upstream # 4. dependencies, ownership
systemctl start upstream # 5. migrations apply at boot
journalctl -u upstream -n 30 # 6. "applied migrations: ..." then the listen banner
curl -i http://127.0.0.1:8080/healthz
```
* Migrations are `server/migrations/NNN_name.sql`, applied in order inside a transaction each, **recorded with a
checksum and never edited after they shipped**. If one that was applied has changed, the server refuses to start
(`migration 001_init.sql was edited after it was applied`): restore that file from git and ship a **new** numbered
migration instead. `upstream-admin status` lists applied and pending migrations; `upstream-admin migrate` applies
them without starting the game.
* Each migration runs in its own transaction: a failing one rolls back and the server exits, earlier ones of the same
boot stay applied. Going back to the old code after a migration committed means restoring the pre-upgrade snapshot
(section 7).
* Updated unit files: `install -m 0644 ...`, `systemctl daemon-reload`, `systemctl restart upstream`.
* Players reconnecting after the restart need no action; clients get `serverTime` in every response and refetch.
## 9. Health, logs, checks
```sh
curl -i https://upstream.example.com/healthz # 200 ok (only says the process answers HTTP)
curl -I https://upstream.example.com/ # 200 + Content-Security-Policy
upstream-admin status # database, migrations, events pending / OVERDUE
journalctl -u upstream -f # live log; -p err for failures only
journalctl -u upstream --since "1 hour ago" | grep -E "failed|refused"
```
The log has: applied migrations, created world, caught-up events, `refused rate_limited(...) ip=...`, and
`event N (type) failed: ...` for a handler that threw (that event is marked `failed` and the queue moves on; list them
with `upstream-admin events --status failed`). Player IP addresses appear in the log; passwords and tokens never do.
A healthy server has **no overdue events** in `status`: the timer processes events within milliseconds of their due time.
## 10. Troubleshooting
| Symptom | Cause and fix |
|---|---|
| Service dies at start: `No such built-in module: node:sqlite` | Node is older than 22.13, or the distro's Node 20 is first on the PATH. `node --version`, `which -a node`, section 1. |
| `ExperimentalWarning: SQLite is an experimental feature` in the journal | Harmless. Printed once per start on some Node versions. |
| `SQLITE_CANTOPEN` / `unable to open database file` | `DB_PATH`'s directory is not writable by `upstream`: wrong path in `upstream.env`, `/var/lib/upstream` created by hand with other ownership (`chown -R upstream:upstream /var/lib/upstream`), or `ProtectSystem=strict` enabled without `ReadWritePaths=/var/lib/upstream`. |
| `attempt to write a readonly database` / `unable to open database file` right after using the CLI | `admin.js` was run as root and left root-owned `game.db-wal` / `-shm`. Stop the game, `chown upstream:upstream /var/lib/upstream/game.db*`, start. Use `upstream-admin`, which switches user. |
| `migration NNN_x.sql was edited after it was applied` | An applied migration changed on disk (checksum mismatch). Restore it from git; changes go in a new migration. |
| `UPSTREAM failed to start: ...` with `EADDRINUSE` | Another process on the port (`ss -ltnp \| grep 8080`) or two copies. Change `PORT` and the NPM forward port together. |
| `IPv6 is not available (...); falling back to 0.0.0.0` | The kernel has no IPv6. IPv4 keeps working; set `HOST=0.0.0.0` to silence it. |
| `status=226/NAMESPACE` | The optional namespace hardening is on in an LXC without Nesting: enable **Options > Features > Nesting** and restart the CT, or comment the block out. |
| `Cannot find module 'ws'` | `npm ci --omit=dev` was not run in `/opt/upstream`. |
| 502 Bad Gateway from NPM | Upstream unreachable: `systemctl status upstream`, `curl -i http://<lxc>:8080/healthz` from the NPM host, forward IP/port, firewall. |
| Page loads but a new deploy still shows the old client | NPM *Cache Assets* is on, or a service layer ignores `no-cache`. Switch it off, hard-reload. |
| Signup answers 403 `signup_closed` | `SIGNUP_CODE` is set. Send `{"invite": "<code>"}` in the signup body, or unset it and restart. |
| Login/signup answers 429 `rate_limited` for many players | They all arrive from one address: the proxy is not trusted so the per-IP budget is shared (log says `ip=<the proxy>`). Set `TRUSTED_PROXIES`. Login and signup allow about one request per 10 s per address, burst 8; a whole classroom behind one NAT can hit that. |
| Game seems frozen: builds never finish | `upstream-admin status`: `OVERDUE` events mean the scheduler is not running (server stopped or stuck) or handlers fail: `upstream-admin events --status failed`, journal `event N ... failed`. A restart catches up everything in order. |
| Banned user still plays | They cannot: `ban` revokes every token and blocks login. Check the username spelling (`upstream-admin users --banned`). |
| `database is locked` from the CLI | The game held a write lock longer than the 5 s busy timeout. Retry. |
| `upstream-backup.service` failed | `journalctl -u upstream-backup`. Typical: disk full, or the integrity check of the new snapshot failed (then nothing was kept and nothing pruned; check the disk and the `game.db` itself). |
| Backups not appearing | `systemctl list-timers upstream-backup.timer`; the timer must be **enabled** (`systemctl enable --now upstream-backup.timer`), not just installed. |
| World reset by mistake | Restore the snapshot taken before (section 7), or the one `reset-world --backup` wrote. |
| WebSocket upgrade answered 426 / 403 / 429 | 426: **Websockets Support** off in NPM. 403: origin check, see `ALLOWED_ORIGINS` and the notes in section 5. 429: more than `MAX_CONN_PER_IP` sockets from one address. |