Stage 3-5 server (military, expansion, social), web client for stages 1-2, ops/deploy
Built: units/research/training, movements (scan, raid, attack, takeover, reinforce, settle, transport), deterministic combat resolved in the event queue, reports, power/offline units, founding + conquest + incumbent sites; alliances, roles, diplomacy (24h pact cancel via events), messages, rankings; vanilla-JS phone-first client (auth, POP, map, alerts, settings, en/nb); systemd units, backup timer, DEPLOY.md, tools/admin.js. Tested: npm test 96/96 (fake clock, in-memory db, a few real-HTTP tests); client checked with headless Chromium screenshots only. NOT done: marketplace/transport trade, client screens for military/social, WebSocket push, bots/balance, e2e suite. All unit stats and costs are unbalanced first guesses. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q2MJaQ5LXAHzGnTDgceT14
This commit is contained in:
parent
b56b13880e
commit
249818a7aa
111 changed files with 17472 additions and 13 deletions
397
docs/DEPLOY.md
Normal file
397
docs/DEPLOY.md
Normal file
|
|
@ -0,0 +1,397 @@
|
|||
# Deploying UPSTREAM
|
||||
|
||||
UPSTREAM is one Node process: it serves the static client from `public/`, the REST API under `/api/v1/` and (stage 6)
|
||||
a WebSocket at `/ws`. Unlike NETWALL it **keeps state**: one SQLite file, owned by that process. A restart loses
|
||||
nothing: overdue events are caught up in order before the server accepts traffic.
|
||||
|
||||
```
|
||||
phone / laptop --https--> nginx Proxy Manager (TLS, :443) --http (+ WebSocket)--> Debian 13 LXC: node :8080
|
||||
/opt/upstream (code, read-only)
|
||||
/var/lib/upstream (game.db, backups/)
|
||||
```
|
||||
|
||||
Same shape as NETWALL (`docs/DEPLOY.md` there): same proxy, same hardening notes, same env vars for the network
|
||||
layer. The differences are the Node version (`node:sqlite` needs 22.13+), a state directory and backups.
|
||||
|
||||
**What was and was not run while writing this** (so you know what to check on the real box):
|
||||
|
||||
* **Run:** `systemd-analyze verify` on the three unit files (syntax and directives only), the CLI `tools/admin.js`
|
||||
and its tests on a dev machine (Node 24.21), a backup, verify and prune cycle, `sh -n` on `deploy/upstream-admin`.
|
||||
* **Taken from NETWALL's docs, not re-run for UPSTREAM:** everything about nginx Proxy Manager (labels, header
|
||||
behaviour, "Client addresses"). NETWALL's guide records what was observed against NPM v2.16.0.
|
||||
* **Never run:** a Proxmox LXC, a started unit under systemd, the timer firing on its own, a Node install from the
|
||||
tarball or NodeSource on Debian 13, `rrsync`, Let's Encrypt, a restore on the real box. Treat those steps as a
|
||||
checklist to confirm, and fix this file when reality differs.
|
||||
|
||||
## 1. Requirements
|
||||
|
||||
* Debian 13 (trixie) LXC with its own IPv4 (optionally IPv6) address the proxy can reach. 1 core, 512 MB RAM and 4 GB
|
||||
disk are plenty for a small world; the database grows with the event and notification history (measure it with
|
||||
`upstream-admin status`), and snapshots multiply it by the retention count.
|
||||
* **Node 22.13 or newer.** Debian 13 ships `nodejs` **20.19.2** (checked on tracker.debian.org: stable has 20.19.2,
|
||||
testing and unstable have 24.21). Node 20 has no `node:sqlite`, so `apt install nodejs` is **not enough**. `package.json`
|
||||
says `"engines": {"node": ">=22.13"}`; 22.5 to 22.12 only have `node:sqlite` behind `--experimental-sqlite`, which
|
||||
the unit does not pass. Development is on Node 24.21.
|
||||
* A DNS name for the game (for example `upstream.example.com`) pointing at the proxy.
|
||||
|
||||
### Getting a suitable Node (pick one)
|
||||
|
||||
**A. Official tarball (no third-party apt repo, nothing piped into a shell).** Pick a current 24.x (Active LTS) or
|
||||
22.x from https://nodejs.org/en/download and put its version in `V`:
|
||||
|
||||
```sh
|
||||
apt update && apt install -y curl xz-utils ca-certificates
|
||||
V=24.21.0 # example: use the current 24.x (or >= 22.13) you picked
|
||||
ARCH=linux-x64 # linux-arm64 on ARM hosts (uname -m: aarch64)
|
||||
cd /tmp
|
||||
curl -fsSLO https://nodejs.org/dist/v$V/node-v$V-$ARCH.tar.xz
|
||||
curl -fsSLO https://nodejs.org/dist/v$V/SHASUMS256.txt
|
||||
grep " node-v$V-$ARCH.tar.xz\$" SHASUMS256.txt | sha256sum -c - # must print: ... OK
|
||||
tar -C /opt -xJf node-v$V-$ARCH.tar.xz
|
||||
ln -sfn /opt/node-v$V-$ARCH /opt/node
|
||||
ln -sf /opt/node/bin/node /usr/local/bin/node
|
||||
ln -sf /opt/node/bin/npm /usr/local/bin/npm
|
||||
ln -sf /opt/node/bin/npx /usr/local/bin/npx
|
||||
```
|
||||
|
||||
(For extra assurance also verify the signature of `SHASUMS256.txt` as described on nodejs.org.) An upgrade later is
|
||||
a new tarball and `ln -sfn` to it, then `systemctl restart upstream`. `upstream.service` starts `/usr/bin/env node`, and
|
||||
systemd's default PATH lists `/usr/local/bin` before `/usr/bin`, so the tarball wins over any distro Node.
|
||||
|
||||
**B. NodeSource apt repository** (follows their documentation; not run here). It puts Node in `/usr/bin/node`:
|
||||
|
||||
```sh
|
||||
apt install -y ca-certificates curl gnupg
|
||||
install -d -m 0755 /etc/apt/keyrings
|
||||
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
|
||||
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_24.x nodistro main" > /etc/apt/sources.list.d/nodesource.list
|
||||
apt update && apt install -y nodejs # includes npm; do not also install Debian's npm package
|
||||
```
|
||||
|
||||
**Verify either way** (all three must succeed):
|
||||
|
||||
```sh
|
||||
node --version # v22.13.0 or newer
|
||||
node -e "require('node:sqlite'); console.log('node:sqlite ok')"
|
||||
which -a node # the first one is the one the unit will use
|
||||
```
|
||||
|
||||
On 22.x/24.x Node may print `ExperimentalWarning: SQLite is an experimental feature` to stderr; that is harmless (it
|
||||
shows up in the journal once per start). `No such built-in module: node:sqlite` means Node is too old (or it is the
|
||||
distro's 20.x earlier on the PATH). If `node:sqlite` ever misbehaves, `DECISIONS.md` D1 describes the one-file swap
|
||||
to `better-sqlite3` in `server/db.js`; that would add a native dependency and a build toolchain to the LXC.
|
||||
|
||||
## 2. Create the LXC (Proxmox)
|
||||
|
||||
Container creation, the Proxmox UI way (CLI `pct create` works too):
|
||||
|
||||
* Template `debian-13-standard`, **unprivileged**, hostname `upstream`, 1 core, 512 MB RAM (+256 MB swap), 4-8 GB disk.
|
||||
* Network: a static address (or a DHCP reservation) on a bridge the NPM host can reach. The proxy forwards to that
|
||||
address, so it must not change.
|
||||
* Features: **Nesting** is optional; tick it only if you want the namespace-based hardening block in the unit
|
||||
(section 3). Everything else works without it.
|
||||
* Start on boot: yes. Timezone does not matter: the game uses UTC ms, the backup timer uses the container's clock.
|
||||
* Proxmox's own CT backups (`vzdump`) are not a substitute for the snapshots in section 7: a running-container copy of
|
||||
a SQLite file is crash-consistent at best. Use them for the OS and let the verified snapshots be the data backup.
|
||||
|
||||
## 3. Install in the LXC
|
||||
|
||||
Run as root inside the container (Node from section 1 is already in place).
|
||||
|
||||
```sh
|
||||
apt install -y rsync # for the copy below (or use git)
|
||||
|
||||
# a dedicated, login-less user that only runs the game
|
||||
useradd --system --home-dir /var/lib/upstream --shell /usr/sbin/nologin upstream
|
||||
|
||||
# code: read-only for the service user, owned by root
|
||||
mkdir -p /opt/upstream
|
||||
# from your workstation:
|
||||
# rsync -a --delete --exclude node_modules --exclude .git --exclude data --exclude e2e ./ root@<lxc>:/opt/upstream/
|
||||
# (needs server/, public/, tools/, deploy/, docs/, package.json, package-lock.json; `test/` is optional)
|
||||
cd /opt/upstream
|
||||
npm ci --omit=dev # only "ws"; follows the lockfile exactly
|
||||
chown -R root:root /opt/upstream
|
||||
|
||||
# the admin helper on the PATH
|
||||
install -m 0755 /opt/upstream/deploy/upstream-admin /usr/local/bin/upstream-admin
|
||||
|
||||
# optional settings / secrets file (read by the unit and by upstream-admin)
|
||||
install -d -m 0750 -o root -g upstream /etc/upstream
|
||||
install -m 0640 -o root -g upstream /dev/null /etc/upstream/upstream.env
|
||||
# edit it: SIGNUP_CODE=choose-a-long-random-string
|
||||
# ALLOWED_ORIGINS=https://upstream.example.com
|
||||
|
||||
# the units
|
||||
install -m 0644 /opt/upstream/deploy/upstream.service /opt/upstream/deploy/upstream-backup.service /opt/upstream/deploy/upstream-backup.timer /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now upstream upstream-backup.timer
|
||||
systemctl status upstream
|
||||
journalctl -u upstream -f
|
||||
```
|
||||
|
||||
The first start prints `applied migrations: 001_init.sql, ...`, `created world 1` and the listen banner with
|
||||
`database: /var/lib/upstream/game.db`. Quick manual run as the service user instead of systemd (Ctrl-C to stop):
|
||||
`runuser -u upstream -- env PORT=8080 DB_PATH=/var/lib/upstream/game.db node /opt/upstream/server/index.js`.
|
||||
|
||||
### What the units do
|
||||
|
||||
* `deploy/upstream.service`: user `upstream`, `WorkingDirectory=/opt/upstream`, `StateDirectory=upstream` (systemd creates
|
||||
`/var/lib/upstream` owned by that user, mode 0700, so the game can write nowhere else), `Restart=on-failure`,
|
||||
`EnvironmentFile=-/etc/upstream/upstream.env` (overrides the `Environment=` lines), `UMask=0077` (database and
|
||||
snapshots hold password hashes: owner-only).
|
||||
* Hardening that works in any container: `NoNewPrivileges`, empty capability set (port 8080 needs none),
|
||||
`RestrictAddressFamilies`, `RestrictRealtime`, `RestrictSUIDSGID`, `LockPersonality`, `SystemCallArchitectures`.
|
||||
No `MemoryDenyWriteExecute` (V8 would crash).
|
||||
* The namespace block (`ProtectSystem=strict`, `ReadWritePaths=/var/lib/upstream`, `PrivateTmp`, ...) is commented out.
|
||||
In an unprivileged LXC it fails with `status=226/NAMESPACE` unless **Options > Features > Nesting** is on. Because the
|
||||
app writes to disk, `ProtectSystem=strict` needs the `ReadWritePaths=` line that is already next to it in the unit;
|
||||
without it the database cannot be opened.
|
||||
* Shutdown: SIGTERM stops the scheduler timer, closes the listener and the database and exits 0 in about a second.
|
||||
Even a SIGKILL or power cut is survivable (WAL replay on the next open, then overdue events are caught up).
|
||||
* `deploy/upstream-backup.service` (oneshot) and `.timer`: see section 7.
|
||||
|
||||
## 4. Configuration
|
||||
|
||||
Environment variables read by `node server/index.js`. Put changes in `/etc/upstream/upstream.env` (`KEY=value`, one per
|
||||
line, no `export`) or in a `systemctl edit upstream` override, then `systemctl restart upstream`.
|
||||
|
||||
| Variable | Default (unit) | Meaning |
|
||||
|---|---|---|
|
||||
| `PORT` | `8080` | TCP port. Unprivileged, so no capabilities needed. `0` picks a free one (banner prints it). |
|
||||
| `HOST` | `::` | Bind address. `::` = one dual-stack socket for IPv4 and IPv6 (falls back to `0.0.0.0` if the kernel has no IPv6). `127.0.0.1` = this container only. |
|
||||
| `DB_PATH` | `/var/lib/upstream/game.db` | SQLite file. The code default is `./data/game.db`; the unit overrides it. Its directory is created if missing, it must be writable by the service user. `-wal` and `-shm` files appear next to it: always copy via a snapshot (section 7), never with `cp` while running. |
|
||||
| `TRUST_PROXY` | `auto` | `auto`: believe `X-Forwarded-For` / `X-Forwarded-Host` only from loopback / private-range peers. `1`: always (then firewall the port). `0`: never. Used for the per-IP rate limits and the log. |
|
||||
| `TRUSTED_PROXIES` | empty | Extra IPs / CIDRs `auto` believes: needed when the proxy reaches the LXC from a global IPv6, Tailscale / CGNAT `100.64.0.0/10` or another routable address. |
|
||||
| `ALLOWED_ORIGINS` | empty | Extra origins for the WebSocket origin check (stage 6), comma separated, with the port when it is not 80/443. The REST API uses bearer tokens (no cookies) and does not check origins. |
|
||||
| `MAX_CONN_PER_IP` | `40` | Open connections per client address (WebSocket cap, stage 6). `0` disables. |
|
||||
| `WORLD_SPEED` | unset (1) | World speed multiplier, **first boot only**, when the database has no world yet. |
|
||||
| `WORLD_RADIUS` | unset (50) | Map radius in tiles (coordinates `-R..R`, so 50 = 101 x 101), **first boot only**. |
|
||||
| `WORLD_NAME` | unset (`World 1`) | Name of the first world, **first boot only**. |
|
||||
| `SIGNUP_CODE` | empty | If set, `POST /auth/signup` needs `"invite": "<this>"` in the body or answers `signup_closed` (403). Empty = open signup. Keep it in `/etc/upstream/upstream.env`, not in the unit file. |
|
||||
| `UPSTREAM_DIR`, `UPSTREAM_ENV_FILE` | `/opt/upstream`, `/etc/upstream/upstream.env` | Only for the `upstream-admin` wrapper. |
|
||||
|
||||
Fixed in code (options of `createServer()`): per-IP and per-token rate limits (login and signup are stricter: about one
|
||||
per 10 s sustained, burst 8, per client address), 32 KB request bodies, idempotency keys pruned hourly.
|
||||
|
||||
### First boot
|
||||
|
||||
An empty or missing database is created, all migrations are applied, and **one world is created** from `WORLD_SPEED`,
|
||||
`WORLD_RADIUS`, `WORLD_NAME` (seed random). After that those variables are ignored: changing them does not touch the
|
||||
existing world. More worlds, a different seed, ending or resetting a world are admin tasks (section 6). The first
|
||||
player to sign up becomes an ordinary operator; there is no special admin account, admin work is the CLI on the box.
|
||||
|
||||
## 5. nginx Proxy Manager
|
||||
|
||||
In NPM: **Hosts > Proxy Hosts > Add Proxy Host**.
|
||||
|
||||
| Tab / field | Value |
|
||||
|---|---|
|
||||
| Details: Domain Names | `upstream.example.com` |
|
||||
| Details: Scheme | `http` |
|
||||
| Details: Forward Hostname / IP | the LXC's address (for example `192.168.1.51`), not `127.0.0.1` when NPM runs elsewhere or in its own container |
|
||||
| Details: Forward Port | `8080` |
|
||||
| Details: Cache Assets | **off** (static files are `no-cache` and revalidated by ETag; the game must see new versions) |
|
||||
| Details: Block Common Exploits | optional |
|
||||
| Details: **Websockets Support** | **on.** The REST game works without it, but stage 6 adds `/ws` and it fails with HTTP 426 when this is off. Switch it on now so you do not forget. |
|
||||
| Custom Locations | none needed |
|
||||
| SSL | Request a new Let's Encrypt certificate, **Force SSL**, **HTTP/2** (HSTS optional) |
|
||||
| Advanced (gear) | nothing required; optional gzip below |
|
||||
|
||||
Optional, Advanced box, to compress JS/JSON for phones (the Node server does not compress; NPM's own `gzip on` only
|
||||
covers HTML):
|
||||
|
||||
```nginx
|
||||
gzip on;
|
||||
gzip_types text/css text/javascript application/javascript application/json application/manifest+json image/svg+xml;
|
||||
```
|
||||
|
||||
Notes carried over from NETWALL's setup (observed there against NPM v2.16.0, not re-run for UPSTREAM):
|
||||
|
||||
* NPM sends `Host` **without the public port** and no `X-Forwarded-Host`. On a public port other than 443 the browser
|
||||
`Origin` then differs from `Host`; list the origin with its port in `ALLOWED_ORIGINS` (matters for `/ws`).
|
||||
* A proxy on a non-private address (global IPv6, Tailscale) is not believed by `TRUST_PROXY=auto`: every player then
|
||||
shares one rate-limit budget and the log shows the proxy's address. Fix: `TRUSTED_PROXIES=<its address or CIDR>`.
|
||||
* Clients on your own LAN can pick their address through NPM's `real_ip` handling, so the per-IP limits protect against
|
||||
public abuse, not against LAN users.
|
||||
* **Heads-up for stage 6:** NPM's default `proxy_read_timeout` is 90 s. NETWALL never idles because it streams 20
|
||||
snapshots a second; UPSTREAM's socket will be quiet for long stretches. The WebSocket server must send a ping
|
||||
well under 90 s (NETWALL's pings every 25 s), or the proxy drops idle players.
|
||||
|
||||
Firewall and dual stack: identical to NETWALL (`docs/DEPLOY.md` there, section 6): only the proxy needs to reach port
|
||||
8080. `curl -i http://127.0.0.1:8080/healthz` and `curl -gi 'http://[::1]:8080/healthz'` check both families.
|
||||
|
||||
## 6. Operating: `upstream-admin`
|
||||
|
||||
`tools/admin.js` is the operator CLI (Node core only, uses the game's own `db.js`, `state.js`, `rules.js`).
|
||||
`deploy/upstream-admin` runs it as the service user against the service's database (it reads `DB_PATH` from
|
||||
`/etc/upstream/upstream.env`). **Do not run `node tools/admin.js` as root**: SQLite would create `game.db-wal` and
|
||||
`-shm` owned by root and the game could no longer open its database (symptom in section 10).
|
||||
|
||||
```sh
|
||||
upstream-admin help
|
||||
upstream-admin status # file, size, migrations, overdue events
|
||||
upstream-admin worlds
|
||||
upstream-admin create-world --name "Summer" --speed 2 --radius 60 --seed 1234 --end-in 90d
|
||||
upstream-admin end-world 1 --yes
|
||||
upstream-admin users [--banned] # no hashes or tokens are ever printed
|
||||
upstream-admin operators [--world 1]
|
||||
upstream-admin ban mallory # sets banned, revokes all tokens: old tokens give 401, login 403
|
||||
upstream-admin unban mallory
|
||||
upstream-admin revoke-tokens ada # log out everywhere
|
||||
upstream-admin inspect-operator ada # (--world N when the name exists in several worlds, --id N by operator id)
|
||||
upstream-admin inspect-town 12 # resources now (rules.resourcesAt), buildings, queue, units if the table exists
|
||||
upstream-admin events --pending [--type buildDone --town 12 --limit 100]
|
||||
upstream-admin replay-events --until +6h # dry run: what the scheduler would process, in order
|
||||
upstream-admin replay-events --until +6h --simulate # + run the real handlers on a scratch copy, report outcome and follow-ups
|
||||
upstream-admin backup /var/lib/upstream/backups --keep 14
|
||||
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db
|
||||
upstream-admin grant 12 --fiber 5000 --capital 2000 --yes # testing only
|
||||
upstream-admin reset-world 1 --yes-destroy --backup /var/lib/upstream/backups --new-seed
|
||||
upstream-admin migrate # apply pending migrations without starting the server
|
||||
```
|
||||
|
||||
All commands take `--json` (machine output) and `--db FILE` (otherwise `$DB_PATH`, otherwise `./data/game.db`).
|
||||
Exit codes: `0` ok, `1` failed, `2` usage error **or a destructive command missing its flag**, `3` not found.
|
||||
|
||||
Safety rules the CLI follows:
|
||||
|
||||
* Destructive or test-only commands refuse without a flag: `end-world` and `grant` need `--yes`; `reset-world`
|
||||
needs `--yes-destroy` (without it, it prints what it would delete and exits 2). `ban`, `unban`, `revoke-tokens` are
|
||||
reversible and need none.
|
||||
* `reset-world` deletes the world's game rows (everything with a `world_id`, plus tables pointing at them: towns,
|
||||
buildings, events, operators, alliances, messages, ... discovered from the schema, so tables added by later stages
|
||||
are covered) and **keeps accounts and tokens**; players must join the world again. It resets `start_at`, clears
|
||||
`end_at` and sets the world running. It rolls back if it would leave dangling references. Stop the game first
|
||||
(`systemctl stop upstream`): the server does not know the world was swept while it is running. Use `--backup DIR`.
|
||||
* Everything runs on a running server's database: writes are single short `BEGIN IMMEDIATE` transactions, the
|
||||
connection is always closed, and SQLite's 5 s busy timeout covers collisions with the game (`database is locked`
|
||||
from the CLI just means: try again).
|
||||
* `grant` adds to the town's stored amounts (valid at its `res_at`); the lazy model carries them forward. Amounts above
|
||||
the Rack hall cap are kept but stop growing, like a lowered cap.
|
||||
* `inspect-town` shows values as of now **ignoring** overdue events; it says so when there are any (server stopped).
|
||||
|
||||
## 7. Backups and restore
|
||||
|
||||
`upstream-backup.timer` runs `upstream-backup.service` daily at 04:15 (10 min jitter, `Persistent=true`). It runs
|
||||
`tools/admin.js backup /var/lib/upstream/backups --keep 14` as the service user. A snapshot is:
|
||||
|
||||
1. `VACUUM INTO` a temporary `.partial-*` file: a consistent copy as of one moment, safe while the game runs (WAL readers do
|
||||
not block the writer; the game keeps its own 5 s busy timeout);
|
||||
2. verified: opened read-only, `PRAGMA integrity_check` = `ok`, `worlds` and `schema_migrations` present;
|
||||
3. published under a dated name `upstream-YYYYMMDDTHHMMSSZ.db` (UTC) with a hard link, so an existing file is never
|
||||
replaced (a same-second collision becomes `...Z-1.db`), mode 0600;
|
||||
4. only then older snapshots are pruned: `--keep N` keeps the newest N, `--keep-days D` keeps those younger than D days,
|
||||
both = survive if either keeps it, neither = no pruning. The newest snapshot and any file not matching the naming
|
||||
pattern are never deleted. A failed verification leaves nothing behind and prunes nothing, and the unit fails
|
||||
(`systemctl status upstream-backup`, `journalctl -u upstream-backup`).
|
||||
|
||||
Snapshots are plain SQLite files (journal mode `delete`), not `-wal` dependent: copy a single file.
|
||||
|
||||
```sh
|
||||
systemctl list-timers upstream-backup.timer # next run
|
||||
systemctl start upstream-backup # one right now
|
||||
journalctl -u upstream-backup -n 20
|
||||
ls -l /var/lib/upstream/backups
|
||||
```
|
||||
|
||||
Change the schedule or retention with `systemctl edit upstream-backup.timer` / `upstream-backup.service` (for every
|
||||
6 hours: `OnCalendar=*-*-* 00/6:15:00` and a larger `--keep`). Before an upgrade or a `reset-world`, take one by hand.
|
||||
|
||||
### Off-box copy
|
||||
|
||||
Snapshots on the same disk die with the container. Pull them from another machine (NAS, workstation, the Proxmox host),
|
||||
for example from cron or a timer there:
|
||||
|
||||
```sh
|
||||
rsync -a --ignore-existing root@<lxc>:/var/lib/upstream/backups/ /srv/backups/upstream/
|
||||
```
|
||||
|
||||
`--ignore-existing` makes it append-only on your side (nothing already stored is overwritten, and snapshots pruned on
|
||||
the LXC stay on the backup host until you decide otherwise). Better than a root key: a dedicated key restricted in
|
||||
`authorized_keys` to `command="rrsync -ro /var/lib/upstream/backups",restrict ...` (`rrsync` comes with the `rsync`
|
||||
package; not run here). The files are owner-only (`upstream`), so the reading user needs root or a group ACL.
|
||||
|
||||
### Restore
|
||||
|
||||
The restore is a file copy, and the time model makes it clean: the restored world simply fast-forwards. Every event that
|
||||
fell due since the snapshot is processed in order at start-up, each at its own due time. What is lost is what players did after
|
||||
the snapshot (builds started, signups, messages) and their tokens: players log in again.
|
||||
|
||||
```sh
|
||||
systemctl stop upstream-backup.timer upstream
|
||||
upstream-admin verify-backup /var/lib/upstream/backups/upstream-20261011T041500Z.db # exit 0 or stop here
|
||||
mkdir -p /var/lib/upstream/pre-restore
|
||||
mv /var/lib/upstream/game.db* /var/lib/upstream/pre-restore/ 2>/dev/null # keep the old one, incl. -wal/-shm
|
||||
install -o upstream -g upstream -m 0600 /var/lib/upstream/backups/upstream-20261011T041500Z.db /var/lib/upstream/game.db
|
||||
systemctl start upstream upstream-backup.timer
|
||||
journalctl -u upstream -n 20 # look for "caught up N overdue event(s)"
|
||||
upstream-admin status
|
||||
```
|
||||
|
||||
Restoring onto another machine works the same way (install code and Node, copy the file to `DB_PATH`). Migrations in
|
||||
the snapshot are checksummed: if the code is newer than the snapshot, the missing migrations are applied at boot; run
|
||||
code that is **older** than the snapshot's last migration (`upstream-admin status` shows it) only if you know the schema
|
||||
still fits, the server does not check for that.
|
||||
|
||||
## 8. Upgrading
|
||||
|
||||
```sh
|
||||
upstream-admin backup /var/lib/upstream/backups --keep 14 # 1. snapshot first
|
||||
rsync -a --delete --exclude node_modules --exclude .git --exclude data ./ root@<lxc>:/opt/upstream/ # 2. new code (from your workstation)
|
||||
cd /opt/upstream && npm ci --omit=dev && chown -R root:root /opt/upstream # 3. dependencies, ownership
|
||||
systemctl restart upstream # 4. migrations apply at boot
|
||||
journalctl -u upstream -n 30 # 5. "applied migrations: ..." then the listen banner
|
||||
curl -i http://127.0.0.1:8080/healthz
|
||||
```
|
||||
|
||||
* Migrations are `server/migrations/NNN_name.sql`, applied in order inside a transaction each, **recorded with a
|
||||
checksum and never edited after they shipped**. If one that was applied has changed, the server refuses to start
|
||||
(`migration 001_init.sql was edited after it was applied`): restore that file from git and ship a **new** numbered
|
||||
migration instead. `upstream-admin status` lists applied and pending migrations; `upstream-admin migrate` applies
|
||||
them without starting the game.
|
||||
* Each migration runs in its own transaction: a failing one rolls back and the server exits, earlier ones of the same
|
||||
boot stay applied. Going back to the old code after a migration committed means restoring the pre-upgrade snapshot
|
||||
(section 7).
|
||||
* Updated unit files: `install -m 0644 ...`, `systemctl daemon-reload`, `systemctl restart upstream`.
|
||||
* Players reconnecting after the restart need no action; clients get `serverTime` in every response and refetch.
|
||||
|
||||
## 9. Health, logs, checks
|
||||
|
||||
```sh
|
||||
curl -i https://upstream.example.com/healthz # 200 ok (only says the process answers HTTP)
|
||||
curl -I https://upstream.example.com/ # 200 + Content-Security-Policy
|
||||
upstream-admin status # database, migrations, events pending / OVERDUE
|
||||
journalctl -u upstream -f # live log; -p err for failures only
|
||||
journalctl -u upstream --since "1 hour ago" | grep -E "failed|refused"
|
||||
```
|
||||
|
||||
The log has: applied migrations, created world, caught-up events, `refused rate_limited(...) ip=...`, and
|
||||
`event N (type) failed: ...` for a handler that threw (that event is marked `failed` and the queue moves on; list them
|
||||
with `upstream-admin events --status failed`). Player IP addresses appear in the log; passwords and tokens never do.
|
||||
A healthy server has **no overdue events** in `status`: the timer processes events within milliseconds of their due time.
|
||||
|
||||
## 10. Troubleshooting
|
||||
|
||||
| Symptom | Cause and fix |
|
||||
|---|---|
|
||||
| Service dies at start: `No such built-in module: node:sqlite` | Node is older than 22.13, or the distro's Node 20 is first on the PATH. `node --version`, `which -a node`, section 1. |
|
||||
| `ExperimentalWarning: SQLite is an experimental feature` in the journal | Harmless. Printed once per start on some Node versions. |
|
||||
| `SQLITE_CANTOPEN` / `unable to open database file` | `DB_PATH`'s directory is not writable by `upstream`: wrong path in `upstream.env`, `/var/lib/upstream` created by hand with other ownership (`chown -R upstream:upstream /var/lib/upstream`), or `ProtectSystem=strict` enabled without `ReadWritePaths=/var/lib/upstream`. |
|
||||
| `attempt to write a readonly database` / `unable to open database file` right after using the CLI | `admin.js` was run as root and left root-owned `game.db-wal` / `-shm`. Stop the game, `chown upstream:upstream /var/lib/upstream/game.db*`, start. Use `upstream-admin`, which switches user. |
|
||||
| `migration NNN_x.sql was edited after it was applied` | An applied migration changed on disk (checksum mismatch). Restore it from git; changes go in a new migration. |
|
||||
| `UPSTREAM failed to start: ...` with `EADDRINUSE` | Another process on the port (`ss -ltnp \| grep 8080`) or two copies. Change `PORT` and the NPM forward port together. |
|
||||
| `IPv6 is not available (...); falling back to 0.0.0.0` | The kernel has no IPv6. IPv4 keeps working; set `HOST=0.0.0.0` to silence it. |
|
||||
| `status=226/NAMESPACE` | The optional namespace hardening is on in an LXC without Nesting: enable **Options > Features > Nesting** and restart the CT, or comment the block out. |
|
||||
| `Cannot find module 'ws'` | `npm ci --omit=dev` was not run in `/opt/upstream`. |
|
||||
| 502 Bad Gateway from NPM | Upstream unreachable: `systemctl status upstream`, `curl -i http://<lxc>:8080/healthz` from the NPM host, forward IP/port, firewall. |
|
||||
| Page loads but a new deploy still shows the old client | NPM *Cache Assets* is on, or a service layer ignores `no-cache`. Switch it off, hard-reload. |
|
||||
| Signup answers 403 `signup_closed` | `SIGNUP_CODE` is set. Send `{"invite": "<code>"}` in the signup body, or unset it and restart. |
|
||||
| Login/signup answers 429 `rate_limited` for many players | They all arrive from one address: the proxy is not trusted so the per-IP budget is shared (log says `ip=<the proxy>`). Set `TRUSTED_PROXIES`. Login and signup allow about one request per 10 s per address, burst 8; a whole classroom behind one NAT can hit that. |
|
||||
| Game seems frozen: builds never finish | `upstream-admin status`: `OVERDUE` events mean the scheduler is not running (server stopped or stuck) or handlers fail: `upstream-admin events --status failed`, journal `event N ... failed`. A restart catches up everything in order. |
|
||||
| Banned user still plays | They cannot: `ban` revokes every token and blocks login. Check the username spelling (`upstream-admin users --banned`). |
|
||||
| `database is locked` from the CLI | The game held a write lock longer than the 5 s busy timeout. Retry. |
|
||||
| `upstream-backup.service` failed | `journalctl -u upstream-backup`. Typical: disk full, or the integrity check of the new snapshot failed (then nothing was kept and nothing pruned; check the disk and the `game.db` itself). |
|
||||
| Backups not appearing | `systemctl list-timers upstream-backup.timer`; the timer must be **enabled** (`systemctl enable --now upstream-backup.timer`), not just installed. |
|
||||
| World reset by mistake | Restore the snapshot taken before (section 7), or the one `reset-world --backup` wrote. |
|
||||
| (stage 6) WebSocket upgrade answered 426 / 403 / 429 | 426: **Websockets Support** off in NPM. 403: origin check, see `ALLOWED_ORIGINS` and the notes in section 5. 429: more than `MAX_CONN_PER_IP` sockets from one address. |
|
||||
Loading…
Add table
Add a link
Reference in a new issue