diff --git a/.claude/skills/performance/SKILL.md b/.claude/skills/performance/SKILL.md
index 2fb10a2..1f1e598 100644
--- a/.claude/skills/performance/SKILL.md
+++ b/.claude/skills/performance/SKILL.md
@@ -14,12 +14,28 @@ have huge late-game networks, and most of them are on phones.
```bash
git show origin/main:site/index.html > /tmp/base.html
-npm run perf -- --base /tmp/base.html # about a minute
+npm run perf -- --base /tmp/base.html # headless only, a few minutes
+npm run perf -- --base /tmp/base.html --browser # plus real Chromium, desktop and phone (what CI runs)
```
`scripts/perf-bench.js` loads `tests/fixtures/big-network.json` (about 100 POPs, 150 links,
-950 PNIs, 830 cache boxes, 110 IX ports, 170 customers) into the base and into this checkout, in
-one process, and times them in alternating turns. It prints a Markdown table (median of 7 runs):
+950 PNIs, 830 cache boxes, 110 IX ports, 170 customers) into the base and into this checkout and
+times them in alternating turns, 15 runs each. It prints one table per environment:
+
+- **Headless (jsdom):** our JavaScript. The most stable numbers for routing and simulation, but jsdom's
+ DOM is about 30× slower than a browser's and it has no layout, so its panel and dashboard rows say
+ little about what players feel.
+- **Chromium, desktop** and **Chromium, phone** (`--browser`, `scripts/perf-browser.js`): the real DOM,
+ layout and canvas. The phone is 390×844 at 3× pixel density with the CPU slowed down 4×, about a
+ mid-range Android. Each version runs in its own browser. Use these rows for anything in the interface,
+ and to catch forced layouts (reading `scrollTop`, `offsetHeight` and friends right after changing the
+ DOM), which jsdom cannot see at all.
+ Browsers leave layout and canvas drawing for the next frame, so each timed run ends by forcing both;
+ without that, DOM and map work would look free. For the same reason, a profile that puts a lot of
+ time on a line reading `scrollTop` is usually showing layout that has to happen anyway, not waste:
+ check with the benchmark before "fixing" it.
+
+Each table has these rows:
- Routing: `compute()`
- One game day: `dayTick()`
@@ -28,7 +44,11 @@ one process, and times them in alternating turns. It prints a Markdown table (me
- NOC dashboard tabs
- Map at three zoom levels
-and says whether both versions route the save's traffic identically.
+and it says whether both versions route the save's traffic identically.
+
+`--browser` needs Chromium: a local `playwright` with its browser works as is; otherwise
+`playwright-core` (a dev dependency, installed by `npm ci`) drives the Chrome in `PERF_CHROME` or
+`--chrome PATH`. CI uses the Chrome that ships with GitHub's Ubuntu runners.
Without `--base` it just times this checkout. `--runs N`, `--md FILE`, `--json FILE` and
`--summary FILE` are there for CI and for digging.
@@ -57,15 +77,21 @@ incidents and failures).
## Read the table
-- Below about 2 ms, differences are noise. The verdicts already ignore them.
-- A busy machine can swing a single row by ±20% on identical code. When a verdict looks surprising
- for what the change touches, rerun with `--runs 15` before reporting it, and report both runs.
+- Shared machines (CI runners, cloud sandboxes) swing a single run by ±30% on identical code, and no
+ timing method gets around it: CPU time is as noisy as wall time there. So a row only gets a verdict
+ when it moved by 15% or more **and** the samples say it is real: p < 0.01 in a Mann–Whitney U test
+ (`scripts/perf-stats.js`), and more than 2 ms (8 ms on the phone). Otherwise it shows ≈, whatever
+ the percentage. The `p` column says how sure each row is.
+- A big percentage with ≈ means "could be noise". If it matters for the change, run again (or with
+ `--runs 25`) and report both runs; do not report a single noisy number as a finding.
- ⚠️ slower (15% or more) or 🔴 slower (50% or more) needs a reason in the PR, or a fix. A new
feature may cost time; a refactor or a fix should not.
- "Routing results differ" is expected when a change is meant to change gameplay (new cities,
balance, routing rules), and a bug when it only promises to be faster.
-- It is headless (jsdom with a recording canvas): it measures our JavaScript, not the browser's
- painting. For drawing changes, also look at the game on a phone.
+- Headless Chromium draws the canvas in software, real phones usually on the GPU, so map rows in the
+ Chromium tables overstate drawing cost. For drawing changes, also look at the game on a phone.
+- Absolute numbers depend on the machine. Compare base and head from the same run, never numbers
+ from different runs or machines.
## Keeping the fixture useful
diff --git a/.github/workflows/perf.yml b/.github/workflows/perf.yml
index fad41c6..98dbd90 100644
--- a/.github/workflows/perf.yml
+++ b/.github/workflows/perf.yml
@@ -1,6 +1,7 @@
name: Performance
-# Times the PR against its base on a very large network and uploads the table. A separate workflow
+# Times the PR against its base on a very large network, headless and in Chrome (desktop and a throttled phone),
+# and uploads the tables. A separate workflow
# (perf-comment.yml) posts it on the PR: runs for pull requests from forks have no write access.
on:
pull_request:
@@ -17,7 +18,7 @@ permissions:
jobs:
bench:
runs-on: ubuntu-latest
- timeout-minutes: 20
+ timeout-minutes: 40
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
@@ -34,7 +35,11 @@ jobs:
git show FETCH_HEAD:site/index.html > "$RUNNER_TEMP/base.html"
echo "BASE_ARG=--base $RUNNER_TEMP/base.html" >> "$GITHUB_ENV"
- name: Benchmark
- run: node scripts/perf-bench.js $BASE_ARG --md perf.md --summary "$GITHUB_STEP_SUMMARY"
+ # The runner image ships Google Chrome; playwright-core (a dev dependency) drives it, no browser download.
+ run: |
+ export PERF_CHROME="$(command -v google-chrome || command -v google-chrome-stable || command -v chromium)"
+ echo "Chrome: $PERF_CHROME ($("$PERF_CHROME" --version))"
+ node scripts/perf-bench.js $BASE_ARG --browser --md perf.md --summary "$GITHUB_STEP_SUMMARY"
- name: Remember the PR number
if: github.event_name == 'pull_request'
env:
diff --git a/CLAUDE.md b/CLAUDE.md
index b5c39a9..9eaf5e8 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -18,6 +18,7 @@ node scripts/i18n-check.js # must report "0 missing, 0 unused, 0 broken"
npm run build # builds _site/ (stamps version, bakes release notes)
npm run serve # build and serve locally
npm run perf -- --base /tmp/base.html # big-network benchmark vs another version (see `performance`)
+npm run perf -- --base /tmp/base.html --browser # the same, also in real Chromium (desktop and phone)
```
To play during development, open `site/index.html` directly or run `npm run serve`.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index a62b32d..d945d44 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -44,7 +44,8 @@ A tour of the code is in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
Releases are cut by the maintainer.
- Show the performance of changes to `site/` on a big network. The Performance check posts a table
comparing your PR with `main` as a comment; to run it yourself:
- `git show origin/main:site/index.html > /tmp/base.html && npm run perf -- --base /tmp/base.html`.
+ `git show origin/main:site/index.html > /tmp/base.html && npm run perf -- --base /tmp/base.html`
+ (add `--browser` to also time it in Chromium as a desktop and a phone, like CI does).
Anything marked slower needs an explanation.
- Old saves must keep loading. New state needs a default for new games and a check in
`sanitize()`.
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index a42ed32..82797ad 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -38,7 +38,7 @@ Other files:
| `site/site.webmanifest`, icons | PWA install metadata |
| `scripts/build.sh` | Copies `site/` to `_site/`, stamps `__VERSION__`/`__BUILD_DATE__`, bakes release notes, writes `version.json` |
| `scripts/release-notes.js` | Extracts a version's CHANGELOG bullets |
-| `scripts/perf-bench.js`, `scripts/perf-fixture.js` | Big-network benchmark (base vs head, posted on every PR by `perf.yml` + `perf-comment.yml`) and the generator of its save, `tests/fixtures/big-network.json` |
+| `scripts/perf-bench.js`, `scripts/perf-browser.js`, `scripts/perf-stats.js`, `scripts/perf-fixture.js` | Big-network benchmark, base vs head, headless and in Chromium as a desktop and a throttled phone, with a significance test per row (posted on every PR by `perf.yml` + `perf-comment.yml`), and the generator of its save, `tests/fixtures/big-network.json` |
| `scripts/i18n-check.js` | Compares every `tr`/`trn`/`N_` literal and `data-i18n` text with each dictionary |
| `tests/harness.js` | Loads the page in jsdom with a seeded `Math.random` and a fake canvas |
| `.github/workflows/ci.yml` | Tests and build on every push and PR |
diff --git a/package-lock.json b/package-lock.json
index 066c1b9..a6b488b 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -7,13 +7,14 @@
"": {
"name": "seriesoftubes",
"version": "1.29.0",
+ "license": "MIT",
"devDependencies": {
- "jsdom": "24.1.3"
+ "jsdom": "24.1.3",
+ "playwright-core": "1.56.0"
},
"engines": {
"node": ">=20"
- },
- "license": "MIT"
+ }
},
"node_modules/@asamuzakjp/css-color": {
"version": "3.2.0",
@@ -625,6 +626,19 @@
"url": "https://github.com/inikulin/parse5?sponsor=1"
}
},
+ "node_modules/playwright-core": {
+ "version": "1.56.0",
+ "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.56.0.tgz",
+ "integrity": "sha512-1SXl7pMfemAMSDn5rkPeZljxOCYAmQnYLBTExuh6E8USHXGSX3dx6lYZN/xPpTz1vimXmPA9CDnILvmJaB8aSQ==",
+ "dev": true,
+ "license": "Apache-2.0",
+ "bin": {
+ "playwright-core": "cli.js"
+ },
+ "engines": {
+ "node": ">=18"
+ }
+ },
"node_modules/psl": {
"version": "1.15.0",
"resolved": "https://registry.npmjs.org/psl/-/psl-1.15.0.tgz",
diff --git a/package.json b/package.json
index e0c110a..c2ad103 100644
--- a/package.json
+++ b/package.json
@@ -5,16 +5,22 @@
"private": true,
"license": "MIT",
"homepage": "https://seriesoftub.es/",
- "repository": { "type": "git", "url": "git+https://github.com/fjaeckel/seriesoftubes.git" },
+ "repository": {
+ "type": "git",
+ "url": "git+https://github.com/fjaeckel/seriesoftubes.git"
+ },
"scripts": {
- "test": "node --test --test-concurrency=1 tests/game.test.js tests/build.test.js tests/persist.test.js tests/services.test.js tests/peering.test.js tests/issues.test.js tests/regions.test.js tests/modes.test.js tests/levels.test.js tests/speed.test.js tests/i18n.test.js tests/routing-reference.test.js",
+ "test": "node --test --test-concurrency=1 tests/game.test.js tests/build.test.js tests/persist.test.js tests/services.test.js tests/peering.test.js tests/issues.test.js tests/regions.test.js tests/modes.test.js tests/levels.test.js tests/speed.test.js tests/i18n.test.js tests/routing-reference.test.js tests/perf.test.js",
"build": "bash scripts/build.sh",
"i18n:check": "node scripts/i18n-check.js",
"serve": "npm run build && npx --yes serve@14.2.4 _site",
"perf": "node scripts/perf-bench.js"
},
- "engines": { "node": ">=20" },
+ "engines": {
+ "node": ">=20"
+ },
"devDependencies": {
- "jsdom": "24.1.3"
+ "jsdom": "24.1.3",
+ "playwright-core": "1.56.0"
}
}
diff --git a/scripts/perf-bench.js b/scripts/perf-bench.js
index ff30811..aec8948 100644
--- a/scripts/perf-bench.js
+++ b/scripts/perf-bench.js
@@ -5,11 +5,13 @@
// node scripts/perf-bench.js --base OLD.html compare OLD.html (e.g. main) with this checkout
// node scripts/perf-bench.js --base OLD.html --md out.md --summary "$GITHUB_STEP_SUMMARY"
// node scripts/perf-bench.js --save my-export.json --base OLD.html a player's save (plain or exported from the menu)
+// node scripts/perf-bench.js --base OLD.html --browser also in real Chromium, as a desktop and as a phone
//
// Both versions load the same save in one process and are timed in alternating turns, so a busy machine slows
// them equally. Each number is the median of several runs. It also says whether both versions route the save's
// traffic identically, which catches unintended gameplay changes in a "pure speed" PR.
-// Headless (jsdom, a recording canvas): it measures our JavaScript, not the browser's painting.
+// Headless (jsdom, a recording canvas): it measures our JavaScript, not the browser's painting. --browser adds the
+// same workloads in Chromium (scripts/perf-browser.js), where DOM work, layout and canvas drawing are real.
const fs = require('node:fs');
const path = require('node:path');
const { load } = require('../tests/harness');
@@ -19,25 +21,25 @@ const opt = (k, d) => { const i = args.indexOf(k); return i >= 0 ? args[i + 1] :
const ROOT = path.join(__dirname, '..');
const HEAD = opt('--head', path.join(ROOT, 'site', 'index.html'));
const BASE = opt('--base', null);
-const RUNS = +opt('--runs', 7);
+const RUNS = +opt('--runs', 15);
// A plain save, or a file from the in-game "Export save", which wraps it as {app, version, exported, S}.
const RAW = JSON.parse(fs.readFileSync(opt('--save', path.join(ROOT, 'tests', 'fixtures', 'big-network.json')), 'utf8'));
const SAVE = JSON.stringify(RAW && RAW.app === 'series-of-tubes' && RAW.S ? RAW.S : RAW);
-// What we time. Each body runs inside the game page; panels flip between two selections so nothing can be
-// skipped as "unchanged".
+// What we time. Each body runs inside the game page; panels render two selections in turn, so nothing can be
+// skipped as "unchanged" and every run does the same work.
const METRICS = [
['compute', 'Routing: compute()', 'compute()'],
['day', 'One game day: dayTick()', 'S.nextFail=S.day+99;S.nextEvent=S.day+99;dayTick()'],
['action', 'An action: after()', 'after()'],
- ['panels', 'Side panel and left list', "__flip=!__flip;UI.sel={type:'city',id:__flip?'fra':'ams'};UI.tab=__flip?'peer':'up';UI.left=__flip?'nets':'noc';UI.leftOpen=true;renderPanel();renderLeft()"],
+ ['panels', 'Side panel and left list', "for(const f of [true,false]){UI.sel={type:'city',id:f?'fra':'ams'};UI.tab=f?'peer':'up';UI.left=f?'nets':'noc';UI.leftOpen=true;renderPanel();renderLeft();}"],
['dash', 'NOC dashboard tabs', "$('dash').hidden=false;for(const k of ['overview','conn','traffic','infra','cust']){UI.dtab=k;renderDash();}$('dash').hidden=true"],
['map', 'Map at three zoom levels', "for(const z of [1,3.2,7]){cam.x=CITY.fra.x;cam.y=CITY.fra.y;cam.z=z;draw(0);}"],
];
function open(file) {
const { ev, errors } = load({ file, seed: 777 });
- ev(`window.__flip=false;S=migrate(JSON.parse(${JSON.stringify(SAVE)}));UI.started=true;UI.pitched=true;UI.speed=0;closeModal();compute();renderAll();`);
+ ev(`S=migrate(JSON.parse(${JSON.stringify(SAVE)}));UI.started=true;UI.pitched=true;UI.speed=0;closeModal();compute();renderAll();`);
const time = (body) => ev(`(()=>{const t=performance.now();${body};return performance.now()-t;})()`);
// What the network carries, to tell a pure speed-up from a change in how traffic is routed.
const outcome = () => ev(`(()=>{compute();const r=[R.served,R.demand,R.reach,R.peered,JSON.stringify(R.egT),...Object.keys(R.citySrv).sort().map(c=>R.citySrv[c]),...S.links.map(l=>R.load[l.id]||0)];
@@ -49,6 +51,7 @@ function open(file) {
const median = (a) => { const s = [...a].sort((x, y) => x - y); return s.length % 2 ? s[(s.length - 1) / 2] : (s[s.length / 2 - 1] + s[s.length / 2]) / 2; };
const ms = (x) => x >= 100 ? x.toFixed(0) : x >= 10 ? x.toFixed(1) : x.toFixed(2);
+(async () => {
const head = open(HEAD), base = BASE ? open(BASE) : null;
const sameOutcome = base ? base.outcome() === head.outcome() : null;
const res = {};
@@ -59,37 +62,49 @@ for (const [k, , body] of METRICS) {
// Alternate who goes first, so drift over the run lands on both sides.
if (base && i % 2) { b.push(base.time(body)); h.push(head.time(body)); } else { h.push(head.time(body)); if (base) b.push(base.time(body)); }
}
- res[k] = { head: median(h), base: base ? median(b) : null };
+ res[k] = { head: median(h), base: base ? median(b) : null, samples: { head: h, base: base ? b : null } };
}
const errs = [...head.errors, ...(base ? base.errors : [])];
+const browser = args.includes('--browser') ? await require('./perf-browser').runBrowser({ head: HEAD, base: BASE, save: SAVE, metrics: METRICS, runs: RUNS, exe: opt('--chrome', process.env.PERF_CHROME) }) : null;
-// Verdicts: small absolute differences are noise, whatever the percentage.
-const verdict = (r) => {
+const { mannWhitneyP } = require('./perf-stats');
+const pOf = (r) => (r.samples && r.samples.base ? mannWhitneyP(r.samples.head, r.samples.base) : 1);
+const verdict = (r, noise = 2) => {
if (r.base == null) return '';
- const ratio = r.head / (r.base || 1e-9), diff = r.head - r.base;
- if (Math.abs(diff) < 2) return '≈';
+ const ratio = r.head / (r.base || 1e-9);
+ if (Math.abs(r.head - r.base) < noise || pOf(r) >= 0.01) return '≈';
if (ratio >= 1.5) return '🔴 slower';
if (ratio >= 1.15) return '⚠️ slower';
if (ratio <= 0.85) return '🟢 faster';
return '≈';
};
const pct = (r) => r.base == null ? '' : `${r.head >= r.base ? '+' : ''}${(((r.head - r.base) / (r.base || 1e-9)) * 100).toFixed(0)}%`;
+const pTxt = (r) => { const p = pOf(r); return p < 0.001 ? '<0.001' : p.toFixed(p < 0.1 ? 3 : 2); };
+// One table per environment, all the same shape. `noise` is the absolute floor below which nothing counts.
+const table = (res, noise) => (base ? '| What | Base | This PR | Change | p | |\n|---|--:|--:|--:|--:|---|\n' : '| What | Time |\n|---|--:|\n') +
+ METRICS.map(([k, label]) => { const r = res[k]; return base ? `| ${label} | ${ms(r.base)} | ${ms(r.head)} | ${pct(r)} | ${pTxt(r)} | ${verdict(r, noise)} |` : `| ${label} | ${ms(r.head)} |`; }).join('\n') + '\n';
+const slowIn = (res, noise) => METRICS.filter(([k]) => /slower/.test(verdict(res[k], noise))).map(([, l]) => l);
+
const s = head.size;
-let md = `## Performance on a big network\n\n${s.pops} POPs · ${s.links} links · ${s.pni} PNIs · ${s.caches} cache boxes · ${s.ix} IX ports · ${s.customers} customers. Median of ${RUNS} runs, in milliseconds, headless.\n\n`;
-md += base ? '| What | Base | This PR | Change | |\n|---|--:|--:|--:|---|\n' : '| What | Time |\n|---|--:|\n';
-for (const [k, label] of METRICS) {
- const r = res[k];
- md += base ? `| ${label} | ${ms(r.base)} | ${ms(r.head)} | ${pct(r)} | ${verdict(r)} |\n` : `| ${label} | ${ms(r.head)} |\n`;
+let md = `## Performance on a big network\n\n${s.pops} POPs · ${s.links} links · ${s.pni} PNIs · ${s.caches} cache boxes · ${s.ix} IX ports · ${s.customers} customers. Median of ${RUNS} runs, in milliseconds.\n\n`;
+md += `### Headless (jsdom)\n\nOur JavaScript only: jsdom has no layout and its DOM is far slower than a browser's, so trust the routing rows most here.\n\n` + table(res, 2);
+const slow = base ? slowIn(res, 2) : [];
+const { PROFILES } = browser ? require('./perf-browser') : { PROFILES: [] };
+for (const p of PROFILES) {
+ md += `\n### Chromium ${browser.version}: ${p.key === 'phone' ? 'phone' : 'desktop'}\n\n${p.key === 'phone' ? 'A 390×844 screen at 3× pixel density, with the CPU slowed down 4× like a mid-range Android.' : 'A 1280×800 window at full speed.'} Real DOM, layout and canvas.\n\n` + table(browser.profiles[p.key], 2 * p.rate);
+ if (base) slow.push(...slowIn(browser.profiles[p.key], 2 * p.rate).map((l) => `${l} (Chromium ${p.key})`));
}
if (base) {
md += `\n${sameOutcome ? '✅ Routing results are identical to the base.' : '⚠️ Routing results differ from the base: traffic is carried differently. Expected if this PR changes gameplay; a bug if it only promises to be faster.'}\n`;
- const slow = METRICS.filter(([k]) => /slower/.test(verdict(res[k])));
- md += slow.length ? `\n**Slower:** ${slow.map(([, l]) => l).join(', ')}. Please explain or fix before merging.\n` : '\nNo slowdowns beyond noise.\n';
+ md += slow.length ? `\n**Slower:** ${slow.join(', ')}. Please explain or fix before merging.\n` : '\nNo slowdowns beyond noise.\n';
}
-if (errs.length) md += `\n⚠️ Page errors while benchmarking:\n\n${errs.slice(0, 5).map((e) => '- `' + String(e).slice(0, 200).replace(/`/g, "'") + '`').join('\n')}\n`;
-md += `\nscripts/perf-bench.js · jsdom with a recording canvas: our JavaScript, not the browser's painting. Below about 2 ms, differences are noise.\n`;
+for (const [where, list] of [['', errs], [' in Chromium', browser ? browser.errors : []]]) {
+ if (list.length) md += `\n⚠️ Page errors${where} while benchmarking:\n\n${list.slice(0, 5).map((e) => '- `' + String(e).slice(0, 200).replace(/`/g, "'") + '`').join('\n')}\n`;
+}
+md += `\nscripts/perf-bench.js · A row only counts as slower or faster when it moved by 15% or more and p < 0.01 (Mann–Whitney U), and by more than 2 ms (8 ms on the phone). Headless Chromium draws the canvas in software; phones usually use the GPU.\n`;
console.log(md);
const out = opt('--md', null); if (out) fs.writeFileSync(out, md);
const sum = opt('--summary', null); if (sum) fs.appendFileSync(sum, md + '\n');
-const js = opt('--json', null); if (js) fs.writeFileSync(js, JSON.stringify({ size: s, runs: RUNS, results: res, sameOutcome, errors: errs }, null, 2));
+const js = opt('--json', null); if (js) fs.writeFileSync(js, JSON.stringify({ size: s, runs: RUNS, results: res, sameOutcome, errors: errs, browser }, null, 2));
+})().catch((e) => { console.error(e); process.exit(1); });
diff --git a/scripts/perf-browser.js b/scripts/perf-browser.js
new file mode 100644
index 0000000..a8a7123
--- /dev/null
+++ b/scripts/perf-browser.js
@@ -0,0 +1,72 @@
+// The perf-bench workloads in real Chromium, for what jsdom cannot show: real DOM speed (jsdom is about 30×
+// slower at it), layout (jsdom has none, so it cannot see a forced reflow) and canvas drawing. Two profiles: a
+// desktop at full speed, and a phone (390×844, 3× pixel density, CPU slowed 4× like a mid-range Android).
+// Used by scripts/perf-bench.js --browser. Needs Chromium: a local `playwright` with its browser, or
+// `playwright-core` (a dev dependency) plus a Chrome binary in PERF_CHROME or --chrome (CI uses the runner's Chrome).
+const { pathToFileURL } = require('node:url');
+
+const PROFILES = [
+ { key: 'desktop', label: 'Desktop', rate: 1, ctx: { viewport: { width: 1280, height: 800 }, deviceScaleFactor: 1 } },
+ { key: 'phone', label: 'Phone (4× slower CPU)', rate: 4, ctx: { viewport: { width: 390, height: 844 }, deviceScaleFactor: 3, isMobile: true, hasTouch: true } },
+];
+
+function chromium(exe) {
+ // With a Chrome binary given, playwright-core is enough; without one, a full playwright brings its own browser.
+ for (const m of exe ? ['playwright-core', 'playwright'] : ['playwright', 'playwright-core']) {
+ try { return require(m).chromium; } catch (e) { /* try the next one */ }
+ }
+ throw new Error('--browser needs playwright-core (npm ci) and a Chrome binary in PERF_CHROME, or a local playwright');
+}
+
+const median = (a) => { const s = [...a].sort((x, y) => x - y); return s.length % 2 ? s[(s.length - 1) / 2] : (s[s.length / 2 - 1] + s[s.length / 2]) / 2; };
+
+async function openPage(browser, profile, file, save, errors) {
+ const ctx = await browser.newContext(profile.ctx);
+ const page = await ctx.newPage();
+ // Nothing from the network (fonts, stats): both versions see exactly the same page, and no request adds noise.
+ await page.route('**/*', (r) => (r.request().url().startsWith('file:') ? r.continue() : r.abort()));
+ // Same as the tests: no animation loop or update checks running between measurements, and seeded randomness.
+ await page.addInitScript(() => { window.__TUBES_TEST__ = true; let r = 777; Math.random = () => { r = (r * 16807) % 2147483647; return r / 2147483647; }; });
+ page.on('pageerror', (e) => errors.push(String(e.message || e)));
+ await page.goto(pathToFileURL(file).href, { waitUntil: 'load' });
+ await page.evaluate((s) => { S = migrate(JSON.parse(s)); UI.started = true; UI.pitched = true; UI.speed = 0; closeModal(); compute(); renderAll(); }, save);
+ const cdp = await ctx.newCDPSession(page);
+ if (profile.rate > 1) await cdp.send('Emulation.setCPUThrottlingRate', { rate: profile.rate });
+ await cdp.send('HeapProfiler.enable');
+ // One run of a workload, timed inside the page so the round trip to Node does not count. Garbage is collected
+ // first, so a run never pays for the one before it (its own garbage still costs it, as in the game).
+ // The browser leaves layout and canvas drawing for the next frame, so code that changes the DOM or draws the map
+ // would look free: each run ends by forcing both (reading a size, and one pixel back from the map canvas).
+ const time = async (k, body) => { await cdp.send('HeapProfiler.collectGarbage'); return page.evaluate(([k, body]) => {
+ const F = window.__fx || (window.__fx = {}); const f = F[k] || (F[k] = new Function(body)); const map = document.getElementById('map').getContext('2d');
+ const t = performance.now(); f(); void document.documentElement.offsetHeight; map.getImageData(0, 0, 1, 1); return performance.now() - t; }, [k, body]); };
+ return { ctx, time };
+}
+
+// Returns {version, profiles: {desktop: {compute: {head, base}, ...}, phone: {...}}, errors}.
+async function runBrowser({ head, base, save, metrics, runs, exe }) {
+ // One browser per version: pages of one browser can share a renderer process and its heap, and then one
+ // version's garbage is collected while the other one is being timed.
+ const launch = () => chromium(exe).launch(exe ? { executablePath: exe } : {});
+ const hb = await launch(), bb = base ? await launch() : null;
+ const errors = [], out = { version: hb.version(), profiles: {}, errors };
+ try {
+ for (const p of PROFILES) {
+ const h = await openPage(hb, p, head, save, errors), b = base ? await openPage(bb, p, base, save, errors) : null;
+ const res = out.profiles[p.key] = {};
+ for (const [k, , body] of metrics) {
+ const H = [], B = [];
+ for (const v of [h, b].filter(Boolean)) { await v.time(k, body); await v.time(k, body); } // warm up
+ for (let i = 0; i < runs; i++) {
+ // Alternate who goes first, like the jsdom runs.
+ if (b && i % 2) { B.push(await b.time(k, body)); H.push(await h.time(k, body)); } else { H.push(await h.time(k, body)); if (b) B.push(await b.time(k, body)); }
+ }
+ res[k] = { head: median(H), base: b ? median(B) : null, samples: { head: H.map((x) => +x.toFixed(1)), base: b ? B.map((x) => +x.toFixed(1)) : null } };
+ }
+ await h.ctx.close(); if (b) await b.ctx.close();
+ }
+ } finally { await hb.close(); if (bb) await bb.close(); }
+ return out;
+}
+
+module.exports = { runBrowser, PROFILES };
diff --git a/scripts/perf-stats.js b/scripts/perf-stats.js
new file mode 100644
index 0000000..1904e3e
--- /dev/null
+++ b/scripts/perf-stats.js
@@ -0,0 +1,19 @@
+// Statistics for scripts/perf-bench.js.
+// Is the head really slower or faster, or is it the machine? Shared CI runners swing a single run by ±30% on
+// identical code, so a difference only counts when it is big (15%) AND the samples say it is real: a two-sided
+// Mann–Whitney U test (no assumption about how the timings are spread), p below 0.01.
+function mannWhitneyP(a, b) {
+ const all = [...a.map((v) => [v, 0]), ...b.map((v) => [v, 1])].sort((x, y) => x[0] - y[0]);
+ const n1 = a.length, n2 = b.length, N = n1 + n2, rank = new Array(N);
+ let ties = 0;
+ for (let i = 0; i < N;) { let j = i; while (j + 1 < N && all[j + 1][0] === all[i][0]) j++; const r = (i + j) / 2 + 1, t = j - i + 1; for (let k = i; k <= j; k++) rank[k] = r; ties += t * t * t - t; i = j + 1; }
+ const r1 = all.reduce((sum, [, g], i) => sum + (g === 0 ? rank[i] : 0), 0), U = r1 - n1 * (n1 + 1) / 2;
+ const sd = Math.sqrt(n1 * n2 / 12 * ((N + 1) - ties / (N * (N - 1))));
+ if (!sd) return 1;
+ const z = (Math.abs(U - n1 * n2 / 2) - 0.5) / sd;
+ // Two-sided p from the normal approximation (fine from about 8 samples a side).
+ const t = 1 / (1 + 0.2316419 * z), d = 0.3989423 * Math.exp(-z * z / 2);
+ return Math.min(1, 2 * d * t * (0.3193815 + t * (-0.3565638 + t * (1.781478 + t * (-1.821256 + t * 1.330274)))));
+}
+
+module.exports = { mannWhitneyP };
diff --git a/tests/perf.test.js b/tests/perf.test.js
new file mode 100644
index 0000000..fe7ac39
--- /dev/null
+++ b/tests/perf.test.js
@@ -0,0 +1,35 @@
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const { execFileSync } = require('node:child_process');
+const { load } = require('./harness');
+const { mannWhitneyP } = require('../scripts/perf-stats');
+
+test('perf bench: the Mann–Whitney p-values match SciPy', () => {
+ // scipy.stats.mannwhitneyu(a, b, alternative='two-sided', method='asymptotic', use_continuity=True)
+ const close = (a, b) => assert.ok(Math.abs(a - b) <= 1e-6 + 1e-3 * b, `${a} vs ${b}`);
+ close(mannWhitneyP([1, 2, 3, 4, 5, 6, 7, 8], [9, 10, 11, 12, 13, 14, 15, 16]), 0.0009391056991171899);
+ close(mannWhitneyP([5, 1, 4, 2, 3, 6, 8, 7], [2, 6, 1, 7, 3, 8, 4, 5]), 1);
+ close(mannWhitneyP([10, 10, 12, 12, 14, 15, 15, 16, 18, 20], [11, 12, 12, 13, 15, 15, 17, 19, 21, 22]), 0.42367207100291315);
+ close(mannWhitneyP([100, 104, 98, 110, 95, 120, 101, 99, 103, 97, 105, 140, 96, 102, 108],
+ [118, 122, 115, 130, 112, 125, 119, 121, 117, 135, 114, 160, 113, 120, 128]), 0.0001739079497761901);
+ assert.equal(mannWhitneyP([3, 3, 3], [3, 3, 3]), 1, 'all ties');
+});
+
+test('perf bench: runs end to end on a small save and compares a version with itself', () => {
+ const { ev } = load();
+ ev("ACT.start({});S.cash=1e9;for(const g of GOALS)S.done[g.id]=0;ACT.launchBB({city:'fra'});ACT.buildPop({city:'ams'})");
+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-'));
+ const save = path.join(dir, 'save.json'), js = path.join(dir, 'out.json');
+ fs.writeFileSync(save, JSON.stringify({ app: 'series-of-tubes', version: 'test', S: JSON.parse(ev('JSON.stringify(S)')) }));
+ const md = execFileSync(process.execPath, [path.join(__dirname, '..', 'scripts', 'perf-bench.js'), '--save', save,
+ '--base', path.join(__dirname, '..', 'site', 'index.html'), '--runs', '3', '--json', js], { encoding: 'utf8' });
+ assert.match(md, /2 POPs/, 'reads an exported save');
+ assert.match(md, /Routing results are identical to the base/);
+ assert.match(md, /\| Routing: compute\(\) \|/);
+ const out = JSON.parse(fs.readFileSync(js, 'utf8'));
+ assert.equal(out.results.compute.samples.head.length, 3);
+ assert.deepEqual(out.errors, []);
+});