Performance benchmark on a big network, posted on every PR

scripts/perf-bench.js times routing, a game day, an action, panels, the
NOC dashboard and the map on tests/fixtures/big-network.json (104 POPs,
950 PNIs, 832 caches), base and head in alternating turns, and says
whether routing results changed. perf.yml runs it on every PR against
the base; perf-comment.yml posts the table as one updated comment.
New performance skill, CLAUDE.md rule, PR template and docs.
This commit is contained in:
Frederic Jung 2026-10-26 06:39:23 +01:00
parent ad8b1bebcb
commit 308c3f7045
No known key found for this signature in database
13 changed files with 322 additions and 1 deletions

View file

@ -0,0 +1,61 @@
---
name: performance
description: Measure and demonstrate the performance of every Series of Tubes change on a very large network, compare it with main, and report the numbers.
---
# Performance is always demonstrated
Every change to the game (anything in `site/`) shows its performance on a very large network,
compared with `main`, before it is merged or released. A speed claim without numbers does not
count, and neither does "it feels fine on my small game". Most players who complain about lag
have huge late-game networks, and most of them are on phones.
## Measure
```bash
git show origin/main:site/index.html > /tmp/base.html
npm run perf -- --base /tmp/base.html # about a minute
```
`scripts/perf-bench.js` loads `tests/fixtures/big-network.json` (about 100 POPs, 150 links,
950 PNIs, 830 cache boxes, 110 IX ports, 170 customers) into the base and into this checkout, in
one process, and times them in alternating turns. It prints a Markdown table (median of 7 runs):
- Routing: `compute()`
- One game day: `dayTick()`
- An action: `after()`
- Side panel and left list
- NOC dashboard tabs
- Map at three zoom levels
and says whether both versions route the save's traffic identically.
Without `--base` it just times this checkout. `--runs N`, `--md FILE`, `--json FILE` and
`--summary FILE` are there for CI and for digging.
## Demonstrate
- **Every pull request**: the Performance workflow runs the same comparison and posts the table as a
comment on the PR (updated on every push). Read it before merging. Locally, paste the table into
the PR description or the commit summary you report.
- **Everything you hand back to Frederic** that touched `site/`: include the table, or at least the
rows that moved, in your summary. Say plainly when something got slower.
- **Before a release**: compare the release against the previous tag
(`git show v<last>:site/index.html > /tmp/base.html`) and mention notable changes.
## Read the table
- Below about 2 ms, differences are noise. The verdicts already ignore them.
- ⚠️ slower (15% or more) or 🔴 slower (50% or more) needs a reason in the PR, or a fix. A new
feature may cost time; a refactor or a fix should not.
- "Routing results differ" is expected when a change is meant to change gameplay (new cities,
balance, routing rules), and a bug when it only promises to be faster.
- It is headless (jsdom with a recording canvas): it measures our JavaScript, not the browser's
painting. For drawing changes, also look at the game on a phone.
## Keeping the fixture useful
The save loads through `migrate()` like any player's save, so new state gets its defaults. Only
regenerate it (`node scripts/perf-fixture.js`, several minutes) when the save format changes in a
way old saves cannot load, or the game grows new heavy systems the save does not exercise. Say in
the commit that the baseline changed, because numbers before and after are no longer comparable.

View file

@ -52,6 +52,8 @@ gh api repos/fjaeckel/seriesoftubes/releases/latest --jq '.tag_name+" "+.publish
- `package.json` → `"version"`
- `package-lock.json` → top-level `"version"` and `packages[""].version`
4. `npm test` once more.
Then compare performance with the previous release and mention anything notable to Frederic:
`git show v<last>:site/index.html > /tmp/base.html && npm run perf -- --base /tmp/base.html`.
5. Commit on `main` as Frederic, with no attribution trailers:
```bash
git -c user.name="Frederic Jung" -c user.email="frederic@jc-ix.net" commit \

15
.github/pull_request_template.md vendored Normal file
View file

@ -0,0 +1,15 @@
<!-- One small feature or one fix per pull request, please. See CONTRIBUTING.md. -->
## What and why
## How it was tested
<!-- Tests added or changed, and what you checked by hand (phone width, both orientations, Italian). -->
## Performance
<!-- Changes to site/ must show their performance on a big network. The Performance check posts the
numbers as a comment on this PR. Locally:
git show origin/main:site/index.html > /tmp/base.html
npm run perf -- --base /tmp/base.html
Explain anything marked slower. -->

40
.github/workflows/perf-comment.yml vendored Normal file
View file

@ -0,0 +1,40 @@
name: Performance comment
# Posts the table from the Performance workflow on its pull request, and updates the same comment on every push.
# Runs in this repository's context with write access, so it only reads the uploaded text and never runs PR code.
on:
workflow_run:
workflows: [Performance]
types: [completed]
permissions:
pull-requests: write
issues: write
actions: read
jobs:
comment:
if: github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success'
runs-on: ubuntu-latest
steps:
- uses: actions/download-artifact@v4
with:
name: perf
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Post or update the comment
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REPO: ${{ github.repository }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
run: |
pr=$(tr -cd '0-9' < pr-number.txt)
[ -n "$pr" ] || { echo "no PR number"; exit 0; }
# Trust the number only if that PR's head is the commit that was benchmarked.
head=$(gh api "repos/$REPO/pulls/$pr" --jq .head.sha)
[ "$head" = "$HEAD_SHA" ] || { echo "PR #$pr has moved on since this run"; exit 0; }
{ echo '<!-- perf-bench -->'; head -c 60000 perf.md; } > body.md
id=$(gh api "repos/$REPO/issues/$pr/comments" --paginate --jq '.[] | select(.user.login == "github-actions[bot]" and (.body | startswith("<!-- perf-bench -->"))) | .id' | head -n 1)
if [ -n "$id" ]; then gh api -X PATCH "repos/$REPO/issues/comments/$id" -F body=@body.md > /dev/null
else gh api -X POST "repos/$REPO/issues/$pr/comments" -F body=@body.md > /dev/null; fi
echo "Posted to #$pr"

50
.github/workflows/perf.yml vendored Normal file
View file

@ -0,0 +1,50 @@
name: Performance
# Times the PR against its base on a very large network and uploads the table. A separate workflow
# (perf-comment.yml) posts it on the PR: runs for pull requests from forks have no write access.
on:
pull_request:
paths:
- 'site/**'
- 'scripts/perf-*.js'
- 'tests/fixtures/**'
- 'tests/harness.js'
workflow_dispatch:
permissions:
contents: read
jobs:
bench:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: 24
cache: npm
- run: npm ci
- name: Base version of the game
if: github.event_name == 'pull_request'
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
git fetch --depth=1 origin "$BASE_SHA"
git show FETCH_HEAD:site/index.html > "$RUNNER_TEMP/base.html"
echo "BASE_ARG=--base $RUNNER_TEMP/base.html" >> "$GITHUB_ENV"
- name: Benchmark
run: node scripts/perf-bench.js $BASE_ARG --md perf.md --summary "$GITHUB_STEP_SUMMARY"
- name: Remember the PR number
if: github.event_name == 'pull_request'
env:
PR: ${{ github.event.pull_request.number }}
run: echo "$PR" > pr-number.txt
- uses: actions/upload-artifact@v4
if: github.event_name == 'pull_request'
with:
name: perf
path: |
perf.md
pr-number.txt
retention-days: 7

View file

@ -17,6 +17,7 @@ npm test # node:test + jsdom, all suites, must stay green
node scripts/i18n-check.js # must report "0 missing, 0 unused, 0 broken"
npm run build # builds _site/ (stamps version, bakes release notes)
npm run serve # build and serve locally
npm run perf -- --base /tmp/base.html # big-network benchmark vs another version (see `performance`)
```
To play during development, open `site/index.html` directly or run `npm run serve`.
@ -46,6 +47,10 @@ To play during development, open `site/index.html` directly or run `npm run serv
and bottom sheets. Check `isPhone()` layouts, not just desktop.
- **Saves are sacred.** Old saves must keep loading. Any new state field gets a default
in `newGame()` and a check/repair in `sanitize()`; renamed fields are migrated there.
- **Performance is always demonstrated.** Every change to `site/` is measured on the big-network
fixture against `main` (`npm run perf -- --base /tmp/base.html`, see the `performance` skill),
and the numbers go into the PR and into your summary. Every PR also gets them as a comment from
CI. Anything slower needs a reason or a fix.
- **Tests come with features.** Add or extend a test in `tests/` for every feature or
fix. Tests are seeded, so they must not depend on luck.
- **CHANGELOG style:** plain English, written for players, one bullet per change,

View file

@ -39,6 +39,10 @@ A tour of the code is in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
must not depend on luck.
- Add a bullet for players under `## [Unreleased]` in `CHANGELOG.md`, in plain English.
Releases are cut by the maintainer.
- Show the performance of changes to `site/` on a big network. The Performance check posts a table
comparing your PR with `main` as a comment; to run it yourself:
`git show origin/main:site/index.html > /tmp/base.html && npm run perf -- --base /tmp/base.html`.
Anything marked slower needs an explanation.
- Old saves must keep loading. New state needs a default for new games and a check in
`sanitize()`.
- Check your change on a phone-sized screen (about 360 px wide). Most players are on

View file

@ -38,6 +38,7 @@ Other files:
| `site/site.webmanifest`, icons | PWA install metadata |
| `scripts/build.sh` | Copies `site/` to `_site/`, stamps `__VERSION__`/`__BUILD_DATE__`, bakes release notes, writes `version.json` |
| `scripts/release-notes.js` | Extracts a version's CHANGELOG bullets |
| `scripts/perf-bench.js`, `scripts/perf-fixture.js` | Big-network benchmark (base vs head, posted on every PR by `perf.yml` + `perf-comment.yml`) and the generator of its save, `tests/fixtures/big-network.json` |
| `scripts/i18n-check.js` | Compares every `tr`/`trn`/`N_` literal and `data-i18n` text with each dictionary |
| `tests/harness.js` | Loads the page in jsdom with a seeded `Math.random` and a fake canvas |
| `.github/workflows/ci.yml` | Tests and build on every push and PR |

View file

@ -10,7 +10,8 @@
"test": "node --test --test-concurrency=1 tests/game.test.js tests/build.test.js tests/persist.test.js tests/services.test.js tests/peering.test.js tests/issues.test.js tests/regions.test.js tests/modes.test.js tests/levels.test.js tests/speed.test.js tests/i18n.test.js",
"build": "bash scripts/build.sh",
"i18n:check": "node scripts/i18n-check.js",
"serve": "npm run build && npx --yes serve@14.2.4 _site"
"serve": "npm run build && npx --yes serve@14.2.4 _site",
"perf": "node scripts/perf-bench.js"
},
"engines": { "node": ">=20" },
"devDependencies": {

92
scripts/perf-bench.js Normal file
View file

@ -0,0 +1,92 @@
#!/usr/bin/env node
// Performance benchmark on a very large network (tests/fixtures/big-network.json).
//
// node scripts/perf-bench.js time this checkout
// node scripts/perf-bench.js --base OLD.html compare OLD.html (e.g. main) with this checkout
// node scripts/perf-bench.js --base OLD.html --md out.md --summary "$GITHUB_STEP_SUMMARY"
//
// Both versions load the same save in one process and are timed in alternating turns, so a busy machine slows
// them equally. Each number is the median of several runs. It also says whether both versions route the save's
// traffic identically, which catches unintended gameplay changes in a "pure speed" PR.
// Headless (jsdom, a recording canvas): it measures our JavaScript, not the browser's painting.
const fs = require('node:fs');
const path = require('node:path');
const { load } = require('../tests/harness');
const args = process.argv.slice(2);
const opt = (k, d) => { const i = args.indexOf(k); return i >= 0 ? args[i + 1] : d; };
const ROOT = path.join(__dirname, '..');
const HEAD = opt('--head', path.join(ROOT, 'site', 'index.html'));
const BASE = opt('--base', null);
const RUNS = +opt('--runs', 7);
const SAVE = fs.readFileSync(opt('--save', path.join(ROOT, 'tests', 'fixtures', 'big-network.json')), 'utf8');
// What we time. Each body runs inside the game page; panels flip between two selections so nothing can be
// skipped as "unchanged".
const METRICS = [
['compute', 'Routing: compute()', 'compute()'],
['day', 'One game day: dayTick()', 'S.nextFail=S.day+99;S.nextEvent=S.day+99;dayTick()'],
['action', 'An action: after()', 'after()'],
['panels', 'Side panel and left list', "__flip=!__flip;UI.sel={type:'city',id:__flip?'fra':'ams'};UI.tab=__flip?'peer':'up';UI.left=__flip?'nets':'noc';UI.leftOpen=true;renderPanel();renderLeft()"],
['dash', 'NOC dashboard tabs', "$('dash').hidden=false;for(const k of ['overview','conn','traffic','infra','cust']){UI.dtab=k;renderDash();}$('dash').hidden=true"],
['map', 'Map at three zoom levels', "for(const z of [1,3.2,7]){cam.x=CITY.fra.x;cam.y=CITY.fra.y;cam.z=z;draw(0);}"],
];
function open(file) {
const { ev, errors } = load({ file, seed: 777 });
ev(`window.__flip=false;S=migrate(JSON.parse(${JSON.stringify(SAVE)}));UI.started=true;UI.pitched=true;UI.speed=0;closeModal();compute();renderAll();`);
const time = (body) => ev(`(()=>{const t=performance.now();${body};return performance.now()-t;})()`);
// What the network carries, to tell a pure speed-up from a change in how traffic is routed.
const outcome = () => ev(`(()=>{compute();const r=[R.served,R.demand,R.reach,R.peered,JSON.stringify(R.egT),...Object.keys(R.citySrv).sort().map(c=>R.citySrv[c]),...S.links.map(l=>R.load[l.id]||0)];
return r.map(x=>typeof x==='number'?x.toFixed(6):x).join('|');})()`);
const size = JSON.parse(ev("JSON.stringify({pops:Object.keys(S.pops).length,links:S.links.length,pni:S.pni.length,caches:S.caches.length,customers:S.biz.length,ix:Object.keys(S.ixp).length})"));
return { ev, errors, time, outcome, size };
}
const median = (a) => { const s = [...a].sort((x, y) => x - y); return s.length % 2 ? s[(s.length - 1) / 2] : (s[s.length / 2 - 1] + s[s.length / 2]) / 2; };
const ms = (x) => x >= 100 ? x.toFixed(0) : x >= 10 ? x.toFixed(1) : x.toFixed(2);
const head = open(HEAD), base = BASE ? open(BASE) : null;
const sameOutcome = base ? base.outcome() === head.outcome() : null;
const res = {};
for (const [k, , body] of METRICS) {
const h = [], b = [];
for (const v of [head, base].filter(Boolean)) { v.time(body); v.time(body); } // warm up
for (let i = 0; i < RUNS; i++) {
// Alternate who goes first, so drift over the run lands on both sides.
if (base && i % 2) { b.push(base.time(body)); h.push(head.time(body)); } else { h.push(head.time(body)); if (base) b.push(base.time(body)); }
}
res[k] = { head: median(h), base: base ? median(b) : null };
}
const errs = [...head.errors, ...(base ? base.errors : [])];
// Verdicts: small absolute differences are noise, whatever the percentage.
const verdict = (r) => {
if (r.base == null) return '';
const ratio = r.head / (r.base || 1e-9), diff = r.head - r.base;
if (Math.abs(diff) < 2) return '≈';
if (ratio >= 1.5) return '🔴 slower';
if (ratio >= 1.15) return '⚠️ slower';
if (ratio <= 0.85) return '🟢 faster';
return '≈';
};
const pct = (r) => r.base == null ? '' : `${r.head >= r.base ? '+' : ''}${(((r.head - r.base) / (r.base || 1e-9)) * 100).toFixed(0)}%`;
const s = head.size;
let md = `## Performance on a big network\n\n${s.pops} POPs · ${s.links} links · ${s.pni} PNIs · ${s.caches} cache boxes · ${s.ix} IX ports · ${s.customers} customers. Median of ${RUNS} runs, in milliseconds, headless.\n\n`;
md += base ? '| What | Base | This PR | Change | |\n|---|--:|--:|--:|---|\n' : '| What | Time |\n|---|--:|\n';
for (const [k, label] of METRICS) {
const r = res[k];
md += base ? `| ${label} | ${ms(r.base)} | ${ms(r.head)} | ${pct(r)} | ${verdict(r)} |\n` : `| ${label} | ${ms(r.head)} |\n`;
}
if (base) {
md += `\n${sameOutcome ? '✅ Routing results are identical to the base.' : '⚠️ Routing results differ from the base: traffic is carried differently. Expected if this PR changes gameplay; a bug if it only promises to be faster.'}\n`;
const slow = METRICS.filter(([k]) => /slower/.test(verdict(res[k])));
md += slow.length ? `\n**Slower:** ${slow.map(([, l]) => l).join(', ')}. Please explain or fix before merging.\n` : '\nNo slowdowns beyond noise.\n';
}
if (errs.length) md += `\n⚠️ Page errors while benchmarking:\n\n${errs.slice(0, 5).map((e) => '- `' + String(e).slice(0, 200).replace(/`/g, "'") + '`').join('\n')}\n`;
md += `\n<sub>scripts/perf-bench.js · jsdom with a recording canvas: our JavaScript, not the browser's painting. Below about 2 ms, differences are noise.</sub>\n`;
console.log(md);
const out = opt('--md', null); if (out) fs.writeFileSync(out, md);
const sum = opt('--summary', null); if (sum) fs.appendFileSync(sum, md + '\n');
const js = opt('--json', null); if (js) fs.writeFileSync(js, JSON.stringify({ size: s, runs: RUNS, results: res, sameOutcome, errors: errs }, null, 2));

39
scripts/perf-fixture.js Normal file
View file

@ -0,0 +1,39 @@
#!/usr/bin/env node
// Builds tests/fixtures/big-network.json: a deterministic, very large late-game network that the performance
// benchmark (scripts/perf-bench.js) loads into the base and the head of every pull request. Big on purpose: about
// a hundred POPs, a few hundred links and customers, and close to a thousand PNIs and cache boxes, like the saves
// that made the game lag (#89). Regenerate only when the save format changes in a way old saves cannot load:
// node scripts/perf-fixture.js
const fs = require('node:fs');
const path = require('node:path');
const { load } = require('../tests/harness');
const { ev, errors } = load({ seed: 4242 });
ev(`(()=>{UI.pitched=true;ACT.start({});S.cash=1e13;S.lvl=LEVELS.length;S.won=true;for(const g of GOALS)S.done[g.id]=0;S.tut=null;
const top=RG.findIndex(r=>r.lvl===LEVELS.length&&r.maxP===PORT.length-1);
const cs=ALLC.filter(c=>!CITY[c].ixOnly&&!CITY[c].cn);
for(const c of cs){if(!S.pops[c])ACT.buildPop({city:c});const p=S.pops[c];if(!p)continue;p.rgen=top>=0?top:RG.length-1;p.routers=RMAX;if(!p.bb)ACT.launchBB({city:c});if(p.bb)p.bb.d=CITY[c].P*0.65;if(Math.random()<0.3)ACT.addSite({city:c});}
const pops=Object.keys(S.pops);
// A backbone: every POP to its two nearest neighbours, plus long-haul shortcuts.
const L=(a,b,t)=>{if(a===b)return;UI.pending=null;UI.via=null;UI.lmode='wave';UI.route='diverse';try{openLinkModal(a,b);ACT.buildLink({tier:t});closeModal();}catch(e){closeModal();}};
for(const a of pops){const near=pops.filter(b=>b!==a).sort((x,y)=>kmBetween(a,x)-kmBetween(a,y)).slice(0,2);for(const b of near)if(!pairLinks(a,b).length)L(a,b,3);}
for(let i=0;i<60;i++){const a=pops[Math.floor(Math.random()*pops.length)],b=pops[Math.floor(Math.random()*pops.length)];L(a,b,4);}
// Transit in some cities, IX ports with route servers everywhere.
for(const c of pops.slice(0,30)){const ts=NETS.filter(n=>n.price&&n.pops.includes(c)&&(n.type==='t1'||n.type==='t2')).slice(0,2);for(const t of ts)ACT.buyTransit({city:c,net:t.id,tier:3});}
for(const c of pops)for(const x of ixIdsIn(c)){try{ACT.buyIx({city:x,tier:3});}catch(e){}if(S.dk&&S.dk.q){S.dk.q=null;}closeModal();if(S.ixp[x]!==undefined&&!S.rs[x])ACT.rs({city:x});}
compute();
// PNIs wherever a network is present, and cache boxes from every CDN.
for(const n of NETS)for(const c of pops)if(S.pni.length<950&&n.pops.includes(c)&&!n.tail)S.pni.push({id:'p'+n.id+'_'+c,net:n.id,city:c,tier:3,n:1,site:'a'});
for(const n of NETS.filter(n=>n.cache))for(const c of pops)for(const k of [1,2])if(S.caches.length<850)S.caches.push({id:'c'+k+n.id+'_'+c,city:c,net:n.id,site:'a'});
// Real customers: accept offers until there are a few hundred.
for(let i=0;i<6000&&S.biz.length<420;i++){S.offers=[];spawnOffer();const o=S.offers[0];if(o&&bHas(o))ACT.accept({id:o.id});}
S.offers=[];S.nextFail=S.day+30;S.nextEvent=S.day+30;S.inc=[];
for(let d=0;d<20;d++)dayTick();
})()`);
if (errors.length) { console.error(errors); process.exit(1); }
const save = ev('JSON.stringify(S)');
const n = JSON.parse(ev("JSON.stringify({pops:Object.keys(S.pops).length,links:S.links.length,pni:S.pni.length,caches:S.caches.length,biz:S.biz.length,transit:S.transit.length,ix:Object.keys(S.ixp).length,served:Math.round(R.served)})"));
const out = path.join(__dirname, '..', 'tests', 'fixtures', 'big-network.json');
fs.mkdirSync(path.dirname(out), { recursive: true });
fs.writeFileSync(out, save + '\n');
console.log(`wrote ${path.relative(process.cwd(), out)} (${Math.round(save.length / 1024)} KB):`, n);

1
tests/fixtures/big-network.json vendored Normal file

File diff suppressed because one or more lines are too long

View file

@ -92,3 +92,13 @@ test('in-app release notes stay short and link to the full notes on GitHub', ()
assert.equal(ev(`slimNote('${'x'.repeat(300)}')`).length <= 140, true);
assert.deepEqual(errors, []);
});
test('the big-network benchmark save loads cleanly and routes traffic', () => {
const fs = require('node:fs'), path = require('node:path');
const save = fs.readFileSync(path.join(__dirname, 'fixtures', 'big-network.json'), 'utf8');
const { ev, errors } = load();
ev(`S=migrate(JSON.parse(${JSON.stringify(save)}));UI.started=true;compute();renderAll()`);
assert.ok(ev('Object.keys(S.pops).length') >= 100 && ev('S.pni.length') >= 900 && ev('S.caches.length') >= 800, 'still a big network');
assert.ok(ev('R.served') > 10000, 'and it carries traffic');
assert.deepEqual(errors, []);
});