Phase 4: close loot/refund leaks, landing-site garrisons, speed-scaled shipping, update/offsite tooling, broad e2e
Built: - Economy: raid loot capped to the room the raiders have at home (rest stays with the victim, "lootLeft" in the report); trade escrow reserves its room so refunds always fit; trade accept refuses with no_room_for_goods when the goods would not fit. Loot destroyed per 30-day run 31-61k -> ~1k, refunds clipped ~27k -> 0 (docs/BALANCE.md, 3 seeds). - End game: members of the holding alliance garrison landing sites (migration 005_site_garrison.sql), garrison counts in site defence, 72 h capture shield, withdraw route, POP view lists garrisoned units. Shipping capacity scales with world speed (3x world: winner 76-98 -> ~30-36 days after sites open). Bot runs show real contests (84-194 attacks on held sites at 1x). - Ops: deploy/update.sh (backup, stop, ff-merge, npm ci --ignore-scripts, health check, rollback), deploy/upstream-offsite (rsync only), admin end-world snapshots results, EPIPE fix, docs/workflows/ filled in. - e2e 62 -> 163 checks: units, raids/reports, alliance+messages, trade, end game, nb, PWA, reconnect, slow polling fallback (opt-in). Two app bugs found: message push ignored, "New" button stayed English. Tested: npm test 274/274, openapi check, npm run e2e 163/163 (137 s), bot runs 50x30d seeds 1-3 and 140-day/60-day speed-3 end-game runs INVARIANTS OK, adversarial review pass (findings fixed or documented). NOT tested: update.sh on a real box (systemd, real npm, runuser; stubs only, shellcheck not installed), garrison card/loot text on a phone, Node 22.13, humans. Known leaks left open and documented (BALANCE S6): loot/shipments can still overflow when stock arrives in flight, bounced backbone shipments lose overflow (reported to the sender). Garrison numbers (heldDefenceMult 2, shield 72 h) are first guesses from 2 seeds. Not claimed: balanced. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q2MJaQ5LXAHzGnTDgceT14
This commit is contained in:
parent
9104480463
commit
3c23c51902
92 changed files with 4887 additions and 300 deletions
155
docs/DEPLOY.md
155
docs/DEPLOY.md
|
|
@ -15,12 +15,14 @@ layer. The differences are the Node version (`node:sqlite` needs 22.13+), a stat
|
|||
|
||||
**What was and was not run while writing this** (so you know what to check on the real box):
|
||||
|
||||
* **Run:** `systemd-analyze verify` on the three unit files (syntax and directives only), the CLI `tools/admin.js`
|
||||
* **Run:** `systemd-analyze verify` on the unit files (syntax and directives only; the two off-site units: only the expected "not executable" warning for a script path that does not exist on the dev machine), the CLI `tools/admin.js`
|
||||
and its tests on a dev machine (Node 24.21), a backup, verify and prune cycle, `sh -n` on `deploy/upstream-admin`.
|
||||
Re-checked on 2026-10-11 with the code as it is now (Node 24.21, scratch directory, no systemd): first boot with `DB_PATH`/`PORT`/`HOST`
|
||||
(migrations 001-004 applied, `created world 1`), `/healthz`, `/` headers, signup + build over the API, `admin.js` status / worlds / users /
|
||||
operators / inspect-town / events / replay-events / backup / verify-backup / ban / unban / grant / end-world (refusal without `--yes`),
|
||||
SIGTERM and restart (state, queue and password kept, no migrations re-applied; tokens revoked by `ban` stay revoked), SIGKILL then restart after a build fell due (`caught up 1 overdue event(s)`, the build level was applied), and a restore by copying a snapshot to a fresh `DB_PATH` and starting on it (section 7: it logged `caught up 1 overdue event(s)`).
|
||||
* **Run, ops scripts (dev machine, stubs, no systemd):** `deploy/update.sh` and `deploy/upstream-offsite` and `admin.js end-world` through
|
||||
`test/deploy-update.test.js`, `test/deploy-offsite.test.js`, `test/admin-endworld.test.js`; sections 7 and 8 say exactly what each covers.
|
||||
* **Taken from NETWALL's docs, not re-run for UPSTREAM:** everything about nginx Proxy Manager (labels, header
|
||||
behaviour, "Client addresses"). NETWALL's guide records what was observed against NPM v2.16.0.
|
||||
* **Never run:** a Proxmox LXC, a started unit under systemd, the timer firing on its own, a Node install from the
|
||||
|
|
@ -113,6 +115,8 @@ mkdir -p /opt/upstream
|
|||
# from your workstation:
|
||||
# rsync -a --delete --exclude node_modules --exclude .git --exclude data --exclude e2e ./ root@<lxc>:/opt/upstream/
|
||||
# (needs server/, public/, tools/, deploy/, docs/, package.json, package-lock.json; `test/` is optional)
|
||||
# or, the way the live box is run, a git checkout that deploy/update.sh updates later (section 8):
|
||||
# git clone https://git.vanvikinternet.no/jvanvik/Upstream.git /opt/upstream (plain HTTPS URL, no token in it)
|
||||
cd /opt/upstream
|
||||
npm ci --omit=dev # only "ws"; follows the lockfile exactly
|
||||
chown -R root:root /opt/upstream
|
||||
|
|
@ -131,6 +135,7 @@ install -m 0644 /opt/upstream/deploy/upstream.service /opt/upstream/deploy/upstr
|
|||
systemctl daemon-reload
|
||||
systemctl enable --now upstream upstream-backup.timer
|
||||
systemctl status upstream
|
||||
# off-box copy of the snapshots is optional and not installed here: section 7
|
||||
journalctl -u upstream -f
|
||||
```
|
||||
|
||||
|
|
@ -153,7 +158,8 @@ The first start prints `applied migrations: 001_init.sql, ...`, `created world 1
|
|||
without it the database cannot be opened.
|
||||
* Shutdown: SIGTERM stops the scheduler timer, closes the listener and the database and exits 0 in about a second.
|
||||
Even a SIGKILL or power cut is survivable (WAL replay on the next open, then overdue events are caught up).
|
||||
* `deploy/upstream-backup.service` (oneshot) and `.timer`: see section 7.
|
||||
* `deploy/upstream-backup.service` (oneshot) and `.timer`: see section 7. `deploy/upstream-offsite.service` and `.timer` (optional off-box copy) and
|
||||
`deploy/update.sh` (section 8) are not installed by the steps above.
|
||||
|
||||
## 4. Configuration
|
||||
|
||||
|
|
@ -237,7 +243,7 @@ upstream-admin help
|
|||
upstream-admin status # file, size, migrations, overdue events
|
||||
upstream-admin worlds
|
||||
upstream-admin create-world --name "Summer" --speed 2 --radius 60 --seed 1234 --end-in 90d
|
||||
upstream-admin end-world 1 --yes # flag flip only, see below
|
||||
upstream-admin end-world 1 --yes # same as the natural end: final ranking, escrow back, events cancelled
|
||||
upstream-admin users [--banned] # no hashes or tokens are ever printed
|
||||
upstream-admin operators [--world 1]
|
||||
upstream-admin ban mallory # sets banned, revokes all tokens: old tokens give 401, login 403
|
||||
|
|
@ -255,7 +261,8 @@ upstream-admin reset-world 1 --yes-destroy --backup /var/lib/upstream/backups --
|
|||
upstream-admin migrate # apply pending migrations without starting the server
|
||||
```
|
||||
|
||||
All commands take `--json` (machine output) and `--db FILE` (otherwise `$DB_PATH`, otherwise `./data/game.db`).
|
||||
All commands take `--json` (machine output) and `--db FILE` (otherwise `$DB_PATH`, otherwise `./data/game.db`). Piping into `head` is fine: a closed
|
||||
pipe (EPIPE) stops the output quietly, no stack trace, and the exit code is kept.
|
||||
Exit codes: `0` ok, `1` failed, `2` usage error **or a destructive command missing its flag**, `3` not found.
|
||||
|
||||
Safety rules the CLI follows:
|
||||
|
|
@ -263,9 +270,16 @@ Safety rules the CLI follows:
|
|||
* Destructive or test-only commands refuse without a flag: `end-world` and `grant` need `--yes`; `reset-world`
|
||||
needs `--yes-destroy` (without it, it prints what it would delete and exits 2). `ban`, `unban`, `revoke-tokens` are
|
||||
reversible and need none.
|
||||
* `end-world` just sets the world's status to `ended` (and `end_at` to now): signup and join stop and every game command answers
|
||||
`world_ended`, but it writes no ranking snapshot (`GET /worlds/:id/results` stays `available: false`), cancels no pending event and
|
||||
closes no trade offer. The natural end (a backbone winner, then the `worldEnd` event two world days later) does all of that.
|
||||
* `end-world <id> --yes` ends a world exactly like the natural end: it runs the real `worldEnd` event handler (`server/events/worldEnd.js`,
|
||||
payload `{"reason":"admin"}`). So the final ranking snapshot is stored (`GET /worlds/:id/results` becomes `available: true`, `winner`
|
||||
null unless the backbone race had a winner already), open trade offers are closed and their escrow goes back to the POPs, **every
|
||||
other pending event of that world is cancelled** (builds in progress, training, movements on the road simply stop: the world is
|
||||
frozen), status becomes `ended` with `end_at` = now, and every operator gets a `world_ended` notification. Events of that world that
|
||||
were already due are processed first, in order, like the server's catch-up. Without `--yes` it refuses (exit 2) and says how many
|
||||
pending events and open offers it would cancel and close. A world that is already ended is left alone (exit 0). Other worlds are untouched.
|
||||
It works on a running or a stopped server; the CLI is a separate process, so a running server pushes nothing over WebSocket for the
|
||||
notification rows it wrote: open clients learn from their next request (`world_ended` error) or the next `/me`. A world that an
|
||||
older version of this command "ended" has no snapshot; the command says so and cannot repair it (reset the world or leave it).
|
||||
* `reset-world` deletes the world's game rows (everything with a `world_id`, plus tables pointing at them: towns,
|
||||
buildings, events, operators, alliances, messages, ... discovered from the schema, so tables added by later stages
|
||||
are covered) and **keeps accounts and tokens**; players must join the world again. It resets `start_at`, clears
|
||||
|
|
@ -307,8 +321,10 @@ Change the schedule or retention with `systemctl edit upstream-backup.timer` / `
|
|||
|
||||
### Off-box copy
|
||||
|
||||
Snapshots on the same disk die with the container. Pull them from another machine (NAS, workstation, the Proxmox host),
|
||||
for example from cron or a timer there:
|
||||
Snapshots on the same disk die with the container. Two ways to get them off the box; use one. Both only ever copy files that the backup
|
||||
unit already verified, and neither deletes anything on the far side (retention over there is yours).
|
||||
|
||||
**A. Pull from another machine** (NAS, workstation, the Proxmox host), for example from cron or a timer there:
|
||||
|
||||
```sh
|
||||
rsync -a --ignore-existing root@<lxc>:/var/lib/upstream/backups/ /srv/backups/upstream/
|
||||
|
|
@ -319,6 +335,52 @@ the LXC stay on the backup host until you decide otherwise). Better than a root
|
|||
`authorized_keys` to `command="rrsync -ro /var/lib/upstream/backups",restrict ...` (`rrsync` comes with the `rsync`
|
||||
package; not run here). The files are owner-only (`upstream`), so the reading user needs root or a group ACL.
|
||||
|
||||
**B. Push from the box: `upstream-offsite.service` + `.timer`** (`deploy/`). Disabled until you configure it; nothing here is installed by
|
||||
section 3. The script `deploy/upstream-offsite` picks the newest snapshot (by modification time), runs `verify-backup` on it (a snapshot that
|
||||
does not verify is never copied), copies it with `rsync` to a directory (`scp` is not supported: it cannot refuse to overwrite, `OFFSITE_METHOD=scp` is rejected with exit 2), **never overwrites** a file of the same
|
||||
name there, and re-reads the far copy and compares checksums. The timer fires daily at 05:00 (+ up to 10 min), after the backup timer (04:15).
|
||||
It runs as the `upstream` user and fails visibly (`journalctl -u upstream-offsite`).
|
||||
|
||||
Set it up (on the far side first, then the box):
|
||||
|
||||
```sh
|
||||
# far side (backup host): an account that can only write into one directory, no shell, append-only by convention
|
||||
useradd --system --create-home --shell /usr/sbin/nologin upstream-backup # or a NAS user
|
||||
install -d -o upstream-backup -m 0700 /srv/backups/upstream
|
||||
# ~upstream-backup/.ssh/authorized_keys, one line (the key generated below):
|
||||
# restrict,command="rrsync /srv/backups/upstream" ssh-ed25519 AAAA... upstream-offsite
|
||||
|
||||
# the box (as root): a key only for this, owned by the service user (ssh refuses keys readable by others)
|
||||
install -d -m 0750 -o root -g upstream /etc/upstream
|
||||
ssh-keygen -t ed25519 -N '' -C upstream-offsite -f /etc/upstream/offsite_key
|
||||
chown upstream:upstream /etc/upstream/offsite_key /etc/upstream/offsite_key.pub; chmod 0600 /etc/upstream/offsite_key
|
||||
# the far side's host key, checked by eye: compare with `ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub` over there
|
||||
ssh-keyscan -t ed25519 backup-host.example > /etc/upstream/offsite_known_hosts
|
||||
ssh-keygen -lf /etc/upstream/offsite_known_hosts
|
||||
|
||||
cat >> /etc/upstream/upstream.env <<'EOT'
|
||||
OFFSITE_DEST=upstream-backup@backup-host.example:/
|
||||
OFFSITE_SSH_KEY=/etc/upstream/offsite_key
|
||||
EOT
|
||||
# with rrsync the destination path is relative to its root, hence ":/" (or ":" ). A plain directory also works:
|
||||
# OFFSITE_DEST=/mnt/nas/upstream (a mounted share; then no key or known_hosts is needed)
|
||||
|
||||
install -m 0644 /opt/upstream/deploy/upstream-offsite.service /opt/upstream/deploy/upstream-offsite.timer /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl start upstream-offsite # one run now: journalctl -u upstream-offsite -n 20
|
||||
systemctl enable --now upstream-offsite.timer
|
||||
```
|
||||
|
||||
Settings (all in `/etc/upstream/upstream.env`): `OFFSITE_DEST` (required; without it every run is skipped quietly by the unit's
|
||||
`ExecCondition`, so installing the timer early is harmless), `OFFSITE_METHOD` (only `rsync`; a far side without rsync needs `rrsync`/a different tool, scp was removed because it would overwrite), `OFFSITE_SSH_KEY`, `OFFSITE_KNOWN_HOSTS` (default `/etc/upstream/offsite_known_hosts`),
|
||||
`OFFSITE_SSH_OPTS` (for example `-p 2222`). Host keys are always checked and ssh never prompts (`BatchMode=yes`). To copy right after each
|
||||
backup instead of on its own timer: `systemctl edit upstream-backup` and add `[Unit]` `OnSuccess=upstream-offsite.service`.
|
||||
|
||||
The key on the box can write to the far directory; make that directory the only thing it can reach (the `restrict,command="rrsync ..."`
|
||||
line) and prune old files over there with a job that lives there, so a compromised game box cannot delete the history.
|
||||
**Test it once for real**: check that the file arrived and that `upstream-admin verify-backup` accepts the far copy (a restore from it is section 7). `upstream-offsite` was exercised with a local directory as the destination (rsync, checksum
|
||||
compare, no-overwrite, refusing an unverified snapshot); the ssh transport, `rrsync` and the units under systemd were **not** run.
|
||||
|
||||
### Restore
|
||||
|
||||
The restore is a file copy, and the time model makes it clean: the restored world simply fast-forwards. Every event that
|
||||
|
|
@ -343,12 +405,77 @@ still fits, the server does not check for that.
|
|||
|
||||
## 8. Upgrading
|
||||
|
||||
`/opt/upstream` is a root-owned git checkout of `https://git.vanvikinternet.no/jvanvik/Upstream.git` (plain HTTPS, no credentials in the URL
|
||||
or in `.git/config`; the script refuses an `http(s)://` origin URL that carries a user or token). Updates go through `deploy/update.sh`:
|
||||
|
||||
```sh
|
||||
upstream-admin backup /var/lib/upstream/backups --keep 14 # 1. snapshot first
|
||||
rsync -a --delete --exclude node_modules --exclude .git --exclude data ./ root@<lxc>:/opt/upstream/ # 2. new code (from your workstation)
|
||||
cd /opt/upstream && npm ci --omit=dev && chown -R root:root /opt/upstream # 3. dependencies, ownership
|
||||
systemctl restart upstream # 4. migrations apply at boot
|
||||
journalctl -u upstream -n 30 # 5. "applied migrations: ..." then the listen banner
|
||||
sudo sh /opt/upstream/deploy/update.sh --dry-run # fetch, list the commits and new migrations, change nothing
|
||||
sudo sh /opt/upstream/deploy/update.sh # the real thing
|
||||
sudo sh /opt/upstream/deploy/update.sh --test # also run `npm test` on the new code before the restart
|
||||
```
|
||||
|
||||
In order, stopping at the first failure (once the service is stopped, the script always starts it again):
|
||||
|
||||
0. **Tools**: `git`, `curl`, `npm`, `systemctl` and the admin wrapper must be on the PATH (`runuser`, `tar` and `mktemp` too with `--test`).
|
||||
A missing one is exit 2 before anything is touched (without `curl` the health check could never pass and a good release would be rolled back).
|
||||
1. **Refuses** on a dirty tree (tracked changes or untracked files that are not git-ignored; `node_modules/` and `data/` are ignored), on a
|
||||
checkout that is not on `main` (detached HEAD included), on an `http(s)://` origin whose authority carries `user:password@` or `token@`
|
||||
(an `ssh://git@host/...` or `git@host:path` remote is fine, so is an `@` later in the path), and while another update runs (lock directory
|
||||
`/run/upstream-update.lock`).
|
||||
2. `git fetch origin main`, no prompt. Not a fast-forward of the checkout (history rewritten, local commits): refuses, touches nothing.
|
||||
Nothing new: "already up to date", exit 0, **no stop, no backup and no restart**.
|
||||
3. Only with `--test`: `git archive` of the **new** rev into a temp directory, `npm ci --omit=dev --ignore-scripts` and `npm test` there **as the
|
||||
service user** (`runuser -u upstream`, `HOME` and the npm cache inside the temp directory; never as root, never in `/opt/upstream`). The live
|
||||
service keeps serving meanwhile. Red: exit 1, nothing was stopped or changed. (Fetched test code can still read what the service user can read,
|
||||
for example the database; `--test` is for your own commits.)
|
||||
4. `systemctl stop upstream`. **Why before the merge:** the running process `require()`s server modules lazily, so a process that keeps serving
|
||||
while the tree changes under it can load new files into old code. From here the tree and the process never disagree; the downtime is steps 5-8.
|
||||
5. **Verified snapshot** of the stopped (final) database: `upstream-admin backup /var/lib/upstream/backups --keep 14`, then `verify-backup` on the
|
||||
file it wrote. No verified snapshot, no update: the service is started again, exit 2, nothing changed. The snapshot path is printed at the end.
|
||||
6. `git merge --ff-only`.
|
||||
7. `npm ci --omit=dev --ignore-scripts` **only if** `package.json` / `package-lock.json` changed (or `node_modules` is missing). No package install
|
||||
script runs as root (`ws` needs none; if a future dependency needs one, run it by hand).
|
||||
8. `systemctl start upstream`, then wait for `http://127.0.0.1:<PORT>/healthz` (PORT from `/etc/upstream/upstream.env`, else 8080) for up to
|
||||
30 s (`HEALTH_TIMEOUT` or `--timeout N`).
|
||||
9. **Rollback** on any failure from step 6 on: `systemctl stop` (a crash-looping new process must not run against the old tree), `git reset --hard
|
||||
<old rev>` (the tree was clean, so nothing is lost; the branch stays on `main`), `npm ci` again if the lock file had changed, `systemctl start`,
|
||||
wait for `/healthz` again.
|
||||
10. Prints `old -> new` revs and the commits that went out.
|
||||
|
||||
Exit codes: `0` updated or already current, `1` failed and the old version answers again (or `--test` was red), `2` refused or usage error
|
||||
(nothing changed), `3` failed and the old version does **not** answer either (the output names the snapshot; `journalctl -u upstream -n 50`).
|
||||
|
||||
Things the script cannot undo: **migrations**. A migration that ran at the new version's first start stays applied when the code is rolled
|
||||
back. When the update contains new files in `server/migrations/` the script says so up front (`NEW MIGRATIONS ...`) and, after a rollback, prints
|
||||
the snapshot to restore if the old code misbehaves on the newer schema (section 7: stop, restore, start; players lose what they did since the
|
||||
snapshot). **Migration `005_site_garrison.sql`** is called out specially: it adds `site_stationed` (units garrisoned in backbone landing sites).
|
||||
The previous code knows nothing of that table: it neither lists those units, nor charges their power, nor lets them go home. A code-only rollback
|
||||
is therefore clean only while nobody has garrisoned a site since the new code started; otherwise restore the snapshot. A rollback target is always
|
||||
the previous revision only, one step. An interrupt (Ctrl-C, SIGTERM) after the stop starts the service again (before the merge, on the untouched old
|
||||
tree) or rolls back like any other failure (after it); SIGKILL or a power cut mid-update cannot be caught: then `git log -1`, `git status` and
|
||||
`systemctl status upstream` tell where it stopped (if the service is down: `systemctl start upstream`; if the tree is already new but the service
|
||||
ran the old code, `systemctl restart upstream`). The stop/start drops open WebSockets; clients reconnect on their own.
|
||||
|
||||
Settings: `UPSTREAM_DIR`, `UPSTREAM_UNIT`, `UPSTREAM_ENV_FILE`, `UPSTREAM_ADMIN`, `BACKUP_DIR`, `BACKUP_KEEP`, `UPSTREAM_REMOTE`,
|
||||
`UPSTREAM_SERVICE_USER` (the user `--test` runs as, default `upstream`), `UPSTREAM_BRANCH`, `HEALTH_URL`, `HEALTH_TIMEOUT`, `LOCK_DIR` (`sh update.sh --help` lists them). It needs `git`, `curl`, `npm`, `systemctl` and
|
||||
`upstream-admin` (section 3) on the PATH of root (checked up front).
|
||||
|
||||
**What was run:** `test/deploy-update.test.js` drives the real script with `sh` against a scratch bare origin and checkout and stubs for
|
||||
`systemctl`, `curl`, `npm`, `upstream-admin` and `sleep`: up to date, update order (stop, snapshot, merge, start: the tree is old at the stop and new at the start), `npm ci --ignore-scripts` only when the lock
|
||||
changed, `--test` (isolated copy, via a stub `runuser`), missing tools, dirty tree, credentials in the URL (and the ssh / `@`-in-path URLs that are allowed), detached HEAD, non-fast-forward, concurrent run, `--dry-run`, failing or unverifiable
|
||||
backup, unhealthy new version (rollback, migrations warning, dependency restore), red tests, a rollback that does not recover (exit 3), `PORT` from the
|
||||
env file. `sh -n`, `bash --posix -n` and `zsh -n` pass; `shellcheck` and `dash` were not available on the dev machine. **Not run:** the script on
|
||||
the real box, against real systemd, a real `npm ci` / `npm test`, or a real `/healthz`.
|
||||
|
||||
Updating **without git** (rsync from a workstation, the way section 3 first installed it) still works; do the same by hand:
|
||||
|
||||
```sh
|
||||
systemctl stop upstream # 1. stop FIRST: the old process lazily loads modules from the tree
|
||||
upstream-admin backup /var/lib/upstream/backups --keep 14 # 2. snapshot of the stopped database
|
||||
rsync -a --delete --exclude node_modules --exclude .git --exclude data ./ root@<lxc>:/opt/upstream/ # 3. new code (from your workstation)
|
||||
cd /opt/upstream && npm ci --omit=dev --ignore-scripts && chown -R root:root /opt/upstream # 4. dependencies, ownership
|
||||
systemctl start upstream # 5. migrations apply at boot
|
||||
journalctl -u upstream -n 30 # 6. "applied migrations: ..." then the listen banner
|
||||
curl -i http://127.0.0.1:8080/healthz
|
||||
```
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue