Benchmark in real Chromium too, and only call a row slower when the samples say so

npm run perf -- --browser times the same workloads in Chromium as a desktop
and as a phone (390x844 at 3x, CPU slowed 4x), one browser per version. Each
run collects garbage first and ends by forcing layout and the map canvas, so
deferred work counts. The jsdom DOM is about 30x slower than a browser's and
has no layout, so the interface rows need this.

Shared machines swing single runs by 30% on identical code, CPU time
included. A row now counts as slower or faster only when it moved 15% and a
Mann-Whitney U test gives p < 0.01 (15 runs a side by default). Identical
code: no flags in three full runs; a routing slowdown planted in a copy is
flagged in jsdom and on the phone.

The panel workload renders both selections every run instead of alternating,
which split its samples in two. CI drives the runner's Chrome through
playwright-core, no browser download.
This commit is contained in:
Frederic Jung 2026-10-26 12:04:57 +01:00
parent c88bad3eef
commit c76dd4d556
No known key found for this signature in database
11 changed files with 236 additions and 42 deletions

View file

@ -14,12 +14,28 @@ have huge late-game networks, and most of them are on phones.
```bash
git show origin/main:site/index.html > /tmp/base.html
npm run perf -- --base /tmp/base.html # about a minute
npm run perf -- --base /tmp/base.html # headless only, a few minutes
npm run perf -- --base /tmp/base.html --browser # plus real Chromium, desktop and phone (what CI runs)
```
`scripts/perf-bench.js` loads `tests/fixtures/big-network.json` (about 100 POPs, 150 links,
950 PNIs, 830 cache boxes, 110 IX ports, 170 customers) into the base and into this checkout, in
one process, and times them in alternating turns. It prints a Markdown table (median of 7 runs):
950 PNIs, 830 cache boxes, 110 IX ports, 170 customers) into the base and into this checkout and
times them in alternating turns, 15 runs each. It prints one table per environment:
- **Headless (jsdom):** our JavaScript. The most stable numbers for routing and simulation, but jsdom's
DOM is about 30× slower than a browser's and it has no layout, so its panel and dashboard rows say
little about what players feel.
- **Chromium, desktop** and **Chromium, phone** (`--browser`, `scripts/perf-browser.js`): the real DOM,
layout and canvas. The phone is 390×844 at 3× pixel density with the CPU slowed down 4×, about a
mid-range Android. Each version runs in its own browser. Use these rows for anything in the interface,
and to catch forced layouts (reading `scrollTop`, `offsetHeight` and friends right after changing the
DOM), which jsdom cannot see at all.
Browsers leave layout and canvas drawing for the next frame, so each timed run ends by forcing both;
without that, DOM and map work would look free. For the same reason, a profile that puts a lot of
time on a line reading `scrollTop` is usually showing layout that has to happen anyway, not waste:
check with the benchmark before "fixing" it.
Each table has these rows:
- Routing: `compute()`
- One game day: `dayTick()`
@ -28,7 +44,11 @@ one process, and times them in alternating turns. It prints a Markdown table (me
- NOC dashboard tabs
- Map at three zoom levels
and says whether both versions route the save's traffic identically.
and it says whether both versions route the save's traffic identically.
`--browser` needs Chromium: a local `playwright` with its browser works as is; otherwise
`playwright-core` (a dev dependency, installed by `npm ci`) drives the Chrome in `PERF_CHROME` or
`--chrome PATH`. CI uses the Chrome that ships with GitHub's Ubuntu runners.
Without `--base` it just times this checkout. `--runs N`, `--md FILE`, `--json FILE` and
`--summary FILE` are there for CI and for digging.
@ -57,15 +77,21 @@ incidents and failures).
## Read the table
- Below about 2 ms, differences are noise. The verdicts already ignore them.
- A busy machine can swing a single row by ±20% on identical code. When a verdict looks surprising
for what the change touches, rerun with `--runs 15` before reporting it, and report both runs.
- Shared machines (CI runners, cloud sandboxes) swing a single run by ±30% on identical code, and no
timing method gets around it: CPU time is as noisy as wall time there. So a row only gets a verdict
when it moved by 15% or more **and** the samples say it is real: p < 0.01 in a Mann–Whitney U test
(`scripts/perf-stats.js`), and more than 2 ms (8 ms on the phone). Otherwise it shows ≈, whatever
the percentage. The `p` column says how sure each row is.
- A big percentage with ≈ means "could be noise". If it matters for the change, run again (or with
`--runs 25`) and report both runs; do not report a single noisy number as a finding.
- ⚠️ slower (15% or more) or 🔴 slower (50% or more) needs a reason in the PR, or a fix. A new
feature may cost time; a refactor or a fix should not.
- "Routing results differ" is expected when a change is meant to change gameplay (new cities,
balance, routing rules), and a bug when it only promises to be faster.
- It is headless (jsdom with a recording canvas): it measures our JavaScript, not the browser's
painting. For drawing changes, also look at the game on a phone.
- Headless Chromium draws the canvas in software, real phones usually on the GPU, so map rows in the
Chromium tables overstate drawing cost. For drawing changes, also look at the game on a phone.
- Absolute numbers depend on the machine. Compare base and head from the same run, never numbers
from different runs or machines.
## Keeping the fixture useful

View file

@ -1,6 +1,7 @@
name: Performance
# Times the PR against its base on a very large network and uploads the table. A separate workflow
# Times the PR against its base on a very large network, headless and in Chrome (desktop and a throttled phone),
# and uploads the tables. A separate workflow
# (perf-comment.yml) posts it on the PR: runs for pull requests from forks have no write access.
on:
pull_request:
@ -17,7 +18,7 @@ permissions:
jobs:
bench:
runs-on: ubuntu-latest
timeout-minutes: 20
timeout-minutes: 40
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
@ -34,7 +35,11 @@ jobs:
git show FETCH_HEAD:site/index.html > "$RUNNER_TEMP/base.html"
echo "BASE_ARG=--base $RUNNER_TEMP/base.html" >> "$GITHUB_ENV"
- name: Benchmark
run: node scripts/perf-bench.js $BASE_ARG --md perf.md --summary "$GITHUB_STEP_SUMMARY"
# The runner image ships Google Chrome; playwright-core (a dev dependency) drives it, no browser download.
run: |
export PERF_CHROME="$(command -v google-chrome || command -v google-chrome-stable || command -v chromium)"
echo "Chrome: $PERF_CHROME ($("$PERF_CHROME" --version))"
node scripts/perf-bench.js $BASE_ARG --browser --md perf.md --summary "$GITHUB_STEP_SUMMARY"
- name: Remember the PR number
if: github.event_name == 'pull_request'
env:

View file

@ -18,6 +18,7 @@ node scripts/i18n-check.js # must report "0 missing, 0 unused, 0 broken"
npm run build # builds _site/ (stamps version, bakes release notes)
npm run serve # build and serve locally
npm run perf -- --base /tmp/base.html # big-network benchmark vs another version (see `performance`)
npm run perf -- --base /tmp/base.html --browser # the same, also in real Chromium (desktop and phone)
```
To play during development, open `site/index.html` directly or run `npm run serve`.

View file

@ -44,7 +44,8 @@ A tour of the code is in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
Releases are cut by the maintainer.
- Show the performance of changes to `site/` on a big network. The Performance check posts a table
comparing your PR with `main` as a comment; to run it yourself:
`git show origin/main:site/index.html > /tmp/base.html && npm run perf -- --base /tmp/base.html`.
`git show origin/main:site/index.html > /tmp/base.html && npm run perf -- --base /tmp/base.html`
(add `--browser` to also time it in Chromium as a desktop and a phone, like CI does).
Anything marked slower needs an explanation.
- Old saves must keep loading. New state needs a default for new games and a check in
`sanitize()`.

View file

@ -38,7 +38,7 @@ Other files:
| `site/site.webmanifest`, icons | PWA install metadata |
| `scripts/build.sh` | Copies `site/` to `_site/`, stamps `__VERSION__`/`__BUILD_DATE__`, bakes release notes, writes `version.json` |
| `scripts/release-notes.js` | Extracts a version's CHANGELOG bullets |
| `scripts/perf-bench.js`, `scripts/perf-fixture.js` | Big-network benchmark (base vs head, posted on every PR by `perf.yml` + `perf-comment.yml`) and the generator of its save, `tests/fixtures/big-network.json` |
| `scripts/perf-bench.js`, `scripts/perf-browser.js`, `scripts/perf-stats.js`, `scripts/perf-fixture.js` | Big-network benchmark, base vs head, headless and in Chromium as a desktop and a throttled phone, with a significance test per row (posted on every PR by `perf.yml` + `perf-comment.yml`), and the generator of its save, `tests/fixtures/big-network.json` |
| `scripts/i18n-check.js` | Compares every `tr`/`trn`/`N_` literal and `data-i18n` text with each dictionary |
| `tests/harness.js` | Loads the page in jsdom with a seeded `Math.random` and a fake canvas |
| `.github/workflows/ci.yml` | Tests and build on every push and PR |

20
package-lock.json generated
View file

@ -7,13 +7,14 @@
"": {
"name": "seriesoftubes",
"version": "1.29.0",
"license": "MIT",
"devDependencies": {
"jsdom": "24.1.3"
"jsdom": "24.1.3",
"playwright-core": "1.56.0"
},
"engines": {
"node": ">=20"
},
"license": "MIT"
}
},
"node_modules/@asamuzakjp/css-color": {
"version": "3.2.0",
@ -625,6 +626,19 @@
"url": "https://github.com/inikulin/parse5?sponsor=1"
}
},
"node_modules/playwright-core": {
"version": "1.56.0",
"resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.56.0.tgz",
"integrity": "sha512-1SXl7pMfemAMSDn5rkPeZljxOCYAmQnYLBTExuh6E8USHXGSX3dx6lYZN/xPpTz1vimXmPA9CDnILvmJaB8aSQ==",
"dev": true,
"license": "Apache-2.0",
"bin": {
"playwright-core": "cli.js"
},
"engines": {
"node": ">=18"
}
},
"node_modules/psl": {
"version": "1.15.0",
"resolved": "https://registry.npmjs.org/psl/-/psl-1.15.0.tgz",

View file

@ -5,16 +5,22 @@
"private": true,
"license": "MIT",
"homepage": "https://seriesoftub.es/",
"repository": { "type": "git", "url": "git+https://github.com/fjaeckel/seriesoftubes.git" },
"repository": {
"type": "git",
"url": "git+https://github.com/fjaeckel/seriesoftubes.git"
},
"scripts": {
"test": "node --test --test-concurrency=1 tests/game.test.js tests/build.test.js tests/persist.test.js tests/services.test.js tests/peering.test.js tests/issues.test.js tests/regions.test.js tests/modes.test.js tests/levels.test.js tests/speed.test.js tests/i18n.test.js tests/routing-reference.test.js",
"test": "node --test --test-concurrency=1 tests/game.test.js tests/build.test.js tests/persist.test.js tests/services.test.js tests/peering.test.js tests/issues.test.js tests/regions.test.js tests/modes.test.js tests/levels.test.js tests/speed.test.js tests/i18n.test.js tests/routing-reference.test.js tests/perf.test.js",
"build": "bash scripts/build.sh",
"i18n:check": "node scripts/i18n-check.js",
"serve": "npm run build && npx --yes serve@14.2.4 _site",
"perf": "node scripts/perf-bench.js"
},
"engines": { "node": ">=20" },
"engines": {
"node": ">=20"
},
"devDependencies": {
"jsdom": "24.1.3"
"jsdom": "24.1.3",
"playwright-core": "1.56.0"
}
}

View file

@ -5,11 +5,13 @@
// node scripts/perf-bench.js --base OLD.html compare OLD.html (e.g. main) with this checkout
// node scripts/perf-bench.js --base OLD.html --md out.md --summary "$GITHUB_STEP_SUMMARY"
// node scripts/perf-bench.js --save my-export.json --base OLD.html a player's save (plain or exported from the menu)
// node scripts/perf-bench.js --base OLD.html --browser also in real Chromium, as a desktop and as a phone
//
// Both versions load the same save in one process and are timed in alternating turns, so a busy machine slows
// them equally. Each number is the median of several runs. It also says whether both versions route the save's
// traffic identically, which catches unintended gameplay changes in a "pure speed" PR.
// Headless (jsdom, a recording canvas): it measures our JavaScript, not the browser's painting.
// Headless (jsdom, a recording canvas): it measures our JavaScript, not the browser's painting. --browser adds the
// same workloads in Chromium (scripts/perf-browser.js), where DOM work, layout and canvas drawing are real.
const fs = require('node:fs');
const path = require('node:path');
const { load } = require('../tests/harness');
@ -19,25 +21,25 @@ const opt = (k, d) => { const i = args.indexOf(k); return i >= 0 ? args[i + 1] :
const ROOT = path.join(__dirname, '..');
const HEAD = opt('--head', path.join(ROOT, 'site', 'index.html'));
const BASE = opt('--base', null);
const RUNS = +opt('--runs', 7);
const RUNS = +opt('--runs', 15);
// A plain save, or a file from the in-game "Export save", which wraps it as {app, version, exported, S}.
const RAW = JSON.parse(fs.readFileSync(opt('--save', path.join(ROOT, 'tests', 'fixtures', 'big-network.json')), 'utf8'));
const SAVE = JSON.stringify(RAW && RAW.app === 'series-of-tubes' && RAW.S ? RAW.S : RAW);
// What we time. Each body runs inside the game page; panels flip between two selections so nothing can be
// skipped as "unchanged".
// What we time. Each body runs inside the game page; panels render two selections in turn, so nothing can be
// skipped as "unchanged" and every run does the same work.
const METRICS = [
['compute', 'Routing: compute()', 'compute()'],
['day', 'One game day: dayTick()', 'S.nextFail=S.day+99;S.nextEvent=S.day+99;dayTick()'],
['action', 'An action: after()', 'after()'],
['panels', 'Side panel and left list', "__flip=!__flip;UI.sel={type:'city',id:__flip?'fra':'ams'};UI.tab=__flip?'peer':'up';UI.left=__flip?'nets':'noc';UI.leftOpen=true;renderPanel();renderLeft()"],
['panels', 'Side panel and left list', "for(const f of [true,false]){UI.sel={type:'city',id:f?'fra':'ams'};UI.tab=f?'peer':'up';UI.left=f?'nets':'noc';UI.leftOpen=true;renderPanel();renderLeft();}"],
['dash', 'NOC dashboard tabs', "$('dash').hidden=false;for(const k of ['overview','conn','traffic','infra','cust']){UI.dtab=k;renderDash();}$('dash').hidden=true"],
['map', 'Map at three zoom levels', "for(const z of [1,3.2,7]){cam.x=CITY.fra.x;cam.y=CITY.fra.y;cam.z=z;draw(0);}"],
];
function open(file) {
const { ev, errors } = load({ file, seed: 777 });
ev(`window.__flip=false;S=migrate(JSON.parse(${JSON.stringify(SAVE)}));UI.started=true;UI.pitched=true;UI.speed=0;closeModal();compute();renderAll();`);
ev(`S=migrate(JSON.parse(${JSON.stringify(SAVE)}));UI.started=true;UI.pitched=true;UI.speed=0;closeModal();compute();renderAll();`);
const time = (body) => ev(`(()=>{const t=performance.now();${body};return performance.now()-t;})()`);
// What the network carries, to tell a pure speed-up from a change in how traffic is routed.
const outcome = () => ev(`(()=>{compute();const r=[R.served,R.demand,R.reach,R.peered,JSON.stringify(R.egT),...Object.keys(R.citySrv).sort().map(c=>R.citySrv[c]),...S.links.map(l=>R.load[l.id]||0)];
@ -49,6 +51,7 @@ function open(file) {
const median = (a) => { const s = [...a].sort((x, y) => x - y); return s.length % 2 ? s[(s.length - 1) / 2] : (s[s.length / 2 - 1] + s[s.length / 2]) / 2; };
const ms = (x) => x >= 100 ? x.toFixed(0) : x >= 10 ? x.toFixed(1) : x.toFixed(2);
(async () => {
const head = open(HEAD), base = BASE ? open(BASE) : null;
const sameOutcome = base ? base.outcome() === head.outcome() : null;
const res = {};
@ -59,37 +62,49 @@ for (const [k, , body] of METRICS) {
// Alternate who goes first, so drift over the run lands on both sides.
if (base && i % 2) { b.push(base.time(body)); h.push(head.time(body)); } else { h.push(head.time(body)); if (base) b.push(base.time(body)); }
}
res[k] = { head: median(h), base: base ? median(b) : null };
res[k] = { head: median(h), base: base ? median(b) : null, samples: { head: h, base: base ? b : null } };
}
const errs = [...head.errors, ...(base ? base.errors : [])];
const browser = args.includes('--browser') ? await require('./perf-browser').runBrowser({ head: HEAD, base: BASE, save: SAVE, metrics: METRICS, runs: RUNS, exe: opt('--chrome', process.env.PERF_CHROME) }) : null;
// Verdicts: small absolute differences are noise, whatever the percentage.
const verdict = (r) => {
const { mannWhitneyP } = require('./perf-stats');
const pOf = (r) => (r.samples && r.samples.base ? mannWhitneyP(r.samples.head, r.samples.base) : 1);
const verdict = (r, noise = 2) => {
if (r.base == null) return '';
const ratio = r.head / (r.base || 1e-9), diff = r.head - r.base;
if (Math.abs(diff) < 2) return '≈';
const ratio = r.head / (r.base || 1e-9);
if (Math.abs(r.head - r.base) < noise || pOf(r) >= 0.01) return '≈';
if (ratio >= 1.5) return '🔴 slower';
if (ratio >= 1.15) return '⚠️ slower';
if (ratio <= 0.85) return '🟢 faster';
return '≈';
};
const pct = (r) => r.base == null ? '' : `${r.head >= r.base ? '+' : ''}${(((r.head - r.base) / (r.base || 1e-9)) * 100).toFixed(0)}%`;
const pTxt = (r) => { const p = pOf(r); return p < 0.001 ? '<0.001' : p.toFixed(p < 0.1 ? 3 : 2); };
// One table per environment, all the same shape. `noise` is the absolute floor below which nothing counts.
const table = (res, noise) => (base ? '| What | Base | This PR | Change | p | |\n|---|--:|--:|--:|--:|---|\n' : '| What | Time |\n|---|--:|\n') +
METRICS.map(([k, label]) => { const r = res[k]; return base ? `| ${label} | ${ms(r.base)} | ${ms(r.head)} | ${pct(r)} | ${pTxt(r)} | ${verdict(r, noise)} |` : `| ${label} | ${ms(r.head)} |`; }).join('\n') + '\n';
const slowIn = (res, noise) => METRICS.filter(([k]) => /slower/.test(verdict(res[k], noise))).map(([, l]) => l);
const s = head.size;
let md = `## Performance on a big network\n\n${s.pops} POPs · ${s.links} links · ${s.pni} PNIs · ${s.caches} cache boxes · ${s.ix} IX ports · ${s.customers} customers. Median of ${RUNS} runs, in milliseconds, headless.\n\n`;
md += base ? '| What | Base | This PR | Change | |\n|---|--:|--:|--:|---|\n' : '| What | Time |\n|---|--:|\n';
for (const [k, label] of METRICS) {
const r = res[k];
md += base ? `| ${label} | ${ms(r.base)} | ${ms(r.head)} | ${pct(r)} | ${verdict(r)} |\n` : `| ${label} | ${ms(r.head)} |\n`;
let md = `## Performance on a big network\n\n${s.pops} POPs · ${s.links} links · ${s.pni} PNIs · ${s.caches} cache boxes · ${s.ix} IX ports · ${s.customers} customers. Median of ${RUNS} runs, in milliseconds.\n\n`;
md += `### Headless (jsdom)\n\nOur JavaScript only: jsdom has no layout and its DOM is far slower than a browser's, so trust the routing rows most here.\n\n` + table(res, 2);
const slow = base ? slowIn(res, 2) : [];
const { PROFILES } = browser ? require('./perf-browser') : { PROFILES: [] };
for (const p of PROFILES) {
md += `\n### Chromium ${browser.version}: ${p.key === 'phone' ? 'phone' : 'desktop'}\n\n${p.key === 'phone' ? 'A 390×844 screen at 3× pixel density, with the CPU slowed down 4× like a mid-range Android.' : 'A 1280×800 window at full speed.'} Real DOM, layout and canvas.\n\n` + table(browser.profiles[p.key], 2 * p.rate);
if (base) slow.push(...slowIn(browser.profiles[p.key], 2 * p.rate).map((l) => `${l} (Chromium ${p.key})`));
}
if (base) {
md += `\n${sameOutcome ? '✅ Routing results are identical to the base.' : '⚠️ Routing results differ from the base: traffic is carried differently. Expected if this PR changes gameplay; a bug if it only promises to be faster.'}\n`;
const slow = METRICS.filter(([k]) => /slower/.test(verdict(res[k])));
md += slow.length ? `\n**Slower:** ${slow.map(([, l]) => l).join(', ')}. Please explain or fix before merging.\n` : '\nNo slowdowns beyond noise.\n';
md += slow.length ? `\n**Slower:** ${slow.join(', ')}. Please explain or fix before merging.\n` : '\nNo slowdowns beyond noise.\n';
}
if (errs.length) md += `\n⚠️ Page errors while benchmarking:\n\n${errs.slice(0, 5).map((e) => '- `' + String(e).slice(0, 200).replace(/`/g, "'") + '`').join('\n')}\n`;
md += `\n<sub>scripts/perf-bench.js · jsdom with a recording canvas: our JavaScript, not the browser's painting. Below about 2 ms, differences are noise.</sub>\n`;
for (const [where, list] of [['', errs], [' in Chromium', browser ? browser.errors : []]]) {
if (list.length) md += `\n⚠️ Page errors${where} while benchmarking:\n\n${list.slice(0, 5).map((e) => '- `' + String(e).slice(0, 200).replace(/`/g, "'") + '`').join('\n')}\n`;
}
md += `\n<sub>scripts/perf-bench.js · A row only counts as slower or faster when it moved by 15% or more and p &lt; 0.01 (Mann–Whitney U), and by more than 2 ms (8 ms on the phone). Headless Chromium draws the canvas in software; phones usually use the GPU.</sub>\n`;
console.log(md);
const out = opt('--md', null); if (out) fs.writeFileSync(out, md);
const sum = opt('--summary', null); if (sum) fs.appendFileSync(sum, md + '\n');
const js = opt('--json', null); if (js) fs.writeFileSync(js, JSON.stringify({ size: s, runs: RUNS, results: res, sameOutcome, errors: errs }, null, 2));
const js = opt('--json', null); if (js) fs.writeFileSync(js, JSON.stringify({ size: s, runs: RUNS, results: res, sameOutcome, errors: errs, browser }, null, 2));
})().catch((e) => { console.error(e); process.exit(1); });

72
scripts/perf-browser.js Normal file
View file

@ -0,0 +1,72 @@
// The perf-bench workloads in real Chromium, for what jsdom cannot show: real DOM speed (jsdom is about 30×
// slower at it), layout (jsdom has none, so it cannot see a forced reflow) and canvas drawing. Two profiles: a
// desktop at full speed, and a phone (390×844, 3× pixel density, CPU slowed 4× like a mid-range Android).
// Used by scripts/perf-bench.js --browser. Needs Chromium: a local `playwright` with its browser, or
// `playwright-core` (a dev dependency) plus a Chrome binary in PERF_CHROME or --chrome (CI uses the runner's Chrome).
const { pathToFileURL } = require('node:url');
const PROFILES = [
{ key: 'desktop', label: 'Desktop', rate: 1, ctx: { viewport: { width: 1280, height: 800 }, deviceScaleFactor: 1 } },
{ key: 'phone', label: 'Phone (4× slower CPU)', rate: 4, ctx: { viewport: { width: 390, height: 844 }, deviceScaleFactor: 3, isMobile: true, hasTouch: true } },
];
function chromium(exe) {
// With a Chrome binary given, playwright-core is enough; without one, a full playwright brings its own browser.
for (const m of exe ? ['playwright-core', 'playwright'] : ['playwright', 'playwright-core']) {
try { return require(m).chromium; } catch (e) { /* try the next one */ }
}
throw new Error('--browser needs playwright-core (npm ci) and a Chrome binary in PERF_CHROME, or a local playwright');
}
const median = (a) => { const s = [...a].sort((x, y) => x - y); return s.length % 2 ? s[(s.length - 1) / 2] : (s[s.length / 2 - 1] + s[s.length / 2]) / 2; };
async function openPage(browser, profile, file, save, errors) {
const ctx = await browser.newContext(profile.ctx);
const page = await ctx.newPage();
// Nothing from the network (fonts, stats): both versions see exactly the same page, and no request adds noise.
await page.route('**/*', (r) => (r.request().url().startsWith('file:') ? r.continue() : r.abort()));
// Same as the tests: no animation loop or update checks running between measurements, and seeded randomness.
await page.addInitScript(() => { window.__TUBES_TEST__ = true; let r = 777; Math.random = () => { r = (r * 16807) % 2147483647; return r / 2147483647; }; });
page.on('pageerror', (e) => errors.push(String(e.message || e)));
await page.goto(pathToFileURL(file).href, { waitUntil: 'load' });
await page.evaluate((s) => { S = migrate(JSON.parse(s)); UI.started = true; UI.pitched = true; UI.speed = 0; closeModal(); compute(); renderAll(); }, save);
const cdp = await ctx.newCDPSession(page);
if (profile.rate > 1) await cdp.send('Emulation.setCPUThrottlingRate', { rate: profile.rate });
await cdp.send('HeapProfiler.enable');
// One run of a workload, timed inside the page so the round trip to Node does not count. Garbage is collected
// first, so a run never pays for the one before it (its own garbage still costs it, as in the game).
// The browser leaves layout and canvas drawing for the next frame, so code that changes the DOM or draws the map
// would look free: each run ends by forcing both (reading a size, and one pixel back from the map canvas).
const time = async (k, body) => { await cdp.send('HeapProfiler.collectGarbage'); return page.evaluate(([k, body]) => {
const F = window.__fx || (window.__fx = {}); const f = F[k] || (F[k] = new Function(body)); const map = document.getElementById('map').getContext('2d');
const t = performance.now(); f(); void document.documentElement.offsetHeight; map.getImageData(0, 0, 1, 1); return performance.now() - t; }, [k, body]); };
return { ctx, time };
}
// Returns {version, profiles: {desktop: {compute: {head, base}, ...}, phone: {...}}, errors}.
async function runBrowser({ head, base, save, metrics, runs, exe }) {
// One browser per version: pages of one browser can share a renderer process and its heap, and then one
// version's garbage is collected while the other one is being timed.
const launch = () => chromium(exe).launch(exe ? { executablePath: exe } : {});
const hb = await launch(), bb = base ? await launch() : null;
const errors = [], out = { version: hb.version(), profiles: {}, errors };
try {
for (const p of PROFILES) {
const h = await openPage(hb, p, head, save, errors), b = base ? await openPage(bb, p, base, save, errors) : null;
const res = out.profiles[p.key] = {};
for (const [k, , body] of metrics) {
const H = [], B = [];
for (const v of [h, b].filter(Boolean)) { await v.time(k, body); await v.time(k, body); } // warm up
for (let i = 0; i < runs; i++) {
// Alternate who goes first, like the jsdom runs.
if (b && i % 2) { B.push(await b.time(k, body)); H.push(await h.time(k, body)); } else { H.push(await h.time(k, body)); if (b) B.push(await b.time(k, body)); }
}
res[k] = { head: median(H), base: b ? median(B) : null, samples: { head: H.map((x) => +x.toFixed(1)), base: b ? B.map((x) => +x.toFixed(1)) : null } };
}
await h.ctx.close(); if (b) await b.ctx.close();
}
} finally { await hb.close(); if (bb) await bb.close(); }
return out;
}
module.exports = { runBrowser, PROFILES };

19
scripts/perf-stats.js Normal file
View file

@ -0,0 +1,19 @@
// Statistics for scripts/perf-bench.js.
// Is the head really slower or faster, or is it the machine? Shared CI runners swing a single run by ±30% on
// identical code, so a difference only counts when it is big (15%) AND the samples say it is real: a two-sided
// Mann–Whitney U test (no assumption about how the timings are spread), p below 0.01.
function mannWhitneyP(a, b) {
const all = [...a.map((v) => [v, 0]), ...b.map((v) => [v, 1])].sort((x, y) => x[0] - y[0]);
const n1 = a.length, n2 = b.length, N = n1 + n2, rank = new Array(N);
let ties = 0;
for (let i = 0; i < N;) { let j = i; while (j + 1 < N && all[j + 1][0] === all[i][0]) j++; const r = (i + j) / 2 + 1, t = j - i + 1; for (let k = i; k <= j; k++) rank[k] = r; ties += t * t * t - t; i = j + 1; }
const r1 = all.reduce((sum, [, g], i) => sum + (g === 0 ? rank[i] : 0), 0), U = r1 - n1 * (n1 + 1) / 2;
const sd = Math.sqrt(n1 * n2 / 12 * ((N + 1) - ties / (N * (N - 1))));
if (!sd) return 1;
const z = (Math.abs(U - n1 * n2 / 2) - 0.5) / sd;
// Two-sided p from the normal approximation (fine from about 8 samples a side).
const t = 1 / (1 + 0.2316419 * z), d = 0.3989423 * Math.exp(-z * z / 2);
return Math.min(1, 2 * d * t * (0.3193815 + t * (-0.3565638 + t * (1.781478 + t * (-1.821256 + t * 1.330274)))));
}
module.exports = { mannWhitneyP };

35
tests/perf.test.js Normal file
View file

@ -0,0 +1,35 @@
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');
const { execFileSync } = require('node:child_process');
const { load } = require('./harness');
const { mannWhitneyP } = require('../scripts/perf-stats');
test('perf bench: the Mann–Whitney p-values match SciPy', () => {
// scipy.stats.mannwhitneyu(a, b, alternative='two-sided', method='asymptotic', use_continuity=True)
const close = (a, b) => assert.ok(Math.abs(a - b) <= 1e-6 + 1e-3 * b, `${a} vs ${b}`);
close(mannWhitneyP([1, 2, 3, 4, 5, 6, 7, 8], [9, 10, 11, 12, 13, 14, 15, 16]), 0.0009391056991171899);
close(mannWhitneyP([5, 1, 4, 2, 3, 6, 8, 7], [2, 6, 1, 7, 3, 8, 4, 5]), 1);
close(mannWhitneyP([10, 10, 12, 12, 14, 15, 15, 16, 18, 20], [11, 12, 12, 13, 15, 15, 17, 19, 21, 22]), 0.42367207100291315);
close(mannWhitneyP([100, 104, 98, 110, 95, 120, 101, 99, 103, 97, 105, 140, 96, 102, 108],
[118, 122, 115, 130, 112, 125, 119, 121, 117, 135, 114, 160, 113, 120, 128]), 0.0001739079497761901);
assert.equal(mannWhitneyP([3, 3, 3], [3, 3, 3]), 1, 'all ties');
});
test('perf bench: runs end to end on a small save and compares a version with itself', () => {
const { ev } = load();
ev("ACT.start({});S.cash=1e9;for(const g of GOALS)S.done[g.id]=0;ACT.launchBB({city:'fra'});ACT.buildPop({city:'ams'})");
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-'));
const save = path.join(dir, 'save.json'), js = path.join(dir, 'out.json');
fs.writeFileSync(save, JSON.stringify({ app: 'series-of-tubes', version: 'test', S: JSON.parse(ev('JSON.stringify(S)')) }));
const md = execFileSync(process.execPath, [path.join(__dirname, '..', 'scripts', 'perf-bench.js'), '--save', save,
'--base', path.join(__dirname, '..', 'site', 'index.html'), '--runs', '3', '--json', js], { encoding: 'utf8' });
assert.match(md, /2 POPs/, 'reads an exported save');
assert.match(md, /Routing results are identical to the base/);
assert.match(md, /\| Routing: compute\(\) \|/);
const out = JSON.parse(fs.readFileSync(js, 'utf8'));
assert.equal(out.results.compute.samples.head.length, 3);
assert.deepEqual(out.errors, []);
});