Performance benchmark on a big network, posted on every PR
scripts/perf-bench.js times routing, a game day, an action, panels, the NOC dashboard and the map on tests/fixtures/big-network.json (104 POPs, 950 PNIs, 832 caches), base and head in alternating turns, and says whether routing results changed. perf.yml runs it on every PR against the base; perf-comment.yml posts the table as one updated comment. New performance skill, CLAUDE.md rule, PR template and docs.
This commit is contained in:
parent
ad8b1bebcb
commit
308c3f7045
13 changed files with 322 additions and 1 deletions
61
.claude/skills/performance/SKILL.md
Normal file
61
.claude/skills/performance/SKILL.md
Normal file
|
|
@ -0,0 +1,61 @@
|
||||||
|
---
|
||||||
|
name: performance
|
||||||
|
description: Measure and demonstrate the performance of every Series of Tubes change on a very large network, compare it with main, and report the numbers.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Performance is always demonstrated
|
||||||
|
|
||||||
|
Every change to the game (anything in `site/`) shows its performance on a very large network,
|
||||||
|
compared with `main`, before it is merged or released. A speed claim without numbers does not
|
||||||
|
count, and neither does "it feels fine on my small game". Most players who complain about lag
|
||||||
|
have huge late-game networks, and most of them are on phones.
|
||||||
|
|
||||||
|
## Measure
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git show origin/main:site/index.html > /tmp/base.html
|
||||||
|
npm run perf -- --base /tmp/base.html # about a minute
|
||||||
|
```
|
||||||
|
|
||||||
|
`scripts/perf-bench.js` loads `tests/fixtures/big-network.json` (about 100 POPs, 150 links,
|
||||||
|
950 PNIs, 830 cache boxes, 110 IX ports, 170 customers) into the base and into this checkout, in
|
||||||
|
one process, and times them in alternating turns. It prints a Markdown table (median of 7 runs):
|
||||||
|
|
||||||
|
- Routing: `compute()`
|
||||||
|
- One game day: `dayTick()`
|
||||||
|
- An action: `after()`
|
||||||
|
- Side panel and left list
|
||||||
|
- NOC dashboard tabs
|
||||||
|
- Map at three zoom levels
|
||||||
|
|
||||||
|
and says whether both versions route the save's traffic identically.
|
||||||
|
|
||||||
|
Without `--base` it just times this checkout. `--runs N`, `--md FILE`, `--json FILE` and
|
||||||
|
`--summary FILE` are there for CI and for digging.
|
||||||
|
|
||||||
|
## Demonstrate
|
||||||
|
|
||||||
|
- **Every pull request**: the Performance workflow runs the same comparison and posts the table as a
|
||||||
|
comment on the PR (updated on every push). Read it before merging. Locally, paste the table into
|
||||||
|
the PR description or the commit summary you report.
|
||||||
|
- **Everything you hand back to Frederic** that touched `site/`: include the table, or at least the
|
||||||
|
rows that moved, in your summary. Say plainly when something got slower.
|
||||||
|
- **Before a release**: compare the release against the previous tag
|
||||||
|
(`git show v<last>:site/index.html > /tmp/base.html`) and mention notable changes.
|
||||||
|
|
||||||
|
## Read the table
|
||||||
|
|
||||||
|
- Below about 2 ms, differences are noise. The verdicts already ignore them.
|
||||||
|
- ⚠️ slower (15% or more) or 🔴 slower (50% or more) needs a reason in the PR, or a fix. A new
|
||||||
|
feature may cost time; a refactor or a fix should not.
|
||||||
|
- "Routing results differ" is expected when a change is meant to change gameplay (new cities,
|
||||||
|
balance, routing rules), and a bug when it only promises to be faster.
|
||||||
|
- It is headless (jsdom with a recording canvas): it measures our JavaScript, not the browser's
|
||||||
|
painting. For drawing changes, also look at the game on a phone.
|
||||||
|
|
||||||
|
## Keeping the fixture useful
|
||||||
|
|
||||||
|
The save loads through `migrate()` like any player's save, so new state gets its defaults. Only
|
||||||
|
regenerate it (`node scripts/perf-fixture.js`, several minutes) when the save format changes in a
|
||||||
|
way old saves cannot load, or the game grows new heavy systems the save does not exercise. Say in
|
||||||
|
the commit that the baseline changed, because numbers before and after are no longer comparable.
|
||||||
|
|
@ -52,6 +52,8 @@ gh api repos/fjaeckel/seriesoftubes/releases/latest --jq '.tag_name+" "+.publish
|
||||||
- `package.json` → `"version"`
|
- `package.json` → `"version"`
|
||||||
- `package-lock.json` → top-level `"version"` and `packages[""].version`
|
- `package-lock.json` → top-level `"version"` and `packages[""].version`
|
||||||
4. `npm test` once more.
|
4. `npm test` once more.
|
||||||
|
Then compare performance with the previous release and mention anything notable to Frederic:
|
||||||
|
`git show v<last>:site/index.html > /tmp/base.html && npm run perf -- --base /tmp/base.html`.
|
||||||
5. Commit on `main` as Frederic, with no attribution trailers:
|
5. Commit on `main` as Frederic, with no attribution trailers:
|
||||||
```bash
|
```bash
|
||||||
git -c user.name="Frederic Jung" -c user.email="frederic@jc-ix.net" commit \
|
git -c user.name="Frederic Jung" -c user.email="frederic@jc-ix.net" commit \
|
||||||
|
|
|
||||||
15
.github/pull_request_template.md
vendored
Normal file
15
.github/pull_request_template.md
vendored
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
<!-- One small feature or one fix per pull request, please. See CONTRIBUTING.md. -->
|
||||||
|
|
||||||
|
## What and why
|
||||||
|
|
||||||
|
## How it was tested
|
||||||
|
|
||||||
|
<!-- Tests added or changed, and what you checked by hand (phone width, both orientations, Italian). -->
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
<!-- Changes to site/ must show their performance on a big network. The Performance check posts the
|
||||||
|
numbers as a comment on this PR. Locally:
|
||||||
|
git show origin/main:site/index.html > /tmp/base.html
|
||||||
|
npm run perf -- --base /tmp/base.html
|
||||||
|
Explain anything marked slower. -->
|
||||||
40
.github/workflows/perf-comment.yml
vendored
Normal file
40
.github/workflows/perf-comment.yml
vendored
Normal file
|
|
@ -0,0 +1,40 @@
|
||||||
|
name: Performance comment
|
||||||
|
|
||||||
|
# Posts the table from the Performance workflow on its pull request, and updates the same comment on every push.
|
||||||
|
# Runs in this repository's context with write access, so it only reads the uploaded text and never runs PR code.
|
||||||
|
on:
|
||||||
|
workflow_run:
|
||||||
|
workflows: [Performance]
|
||||||
|
types: [completed]
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
pull-requests: write
|
||||||
|
issues: write
|
||||||
|
actions: read
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
comment:
|
||||||
|
if: github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success'
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/download-artifact@v4
|
||||||
|
with:
|
||||||
|
name: perf
|
||||||
|
run-id: ${{ github.event.workflow_run.id }}
|
||||||
|
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||||
|
- name: Post or update the comment
|
||||||
|
env:
|
||||||
|
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||||
|
REPO: ${{ github.repository }}
|
||||||
|
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
|
||||||
|
run: |
|
||||||
|
pr=$(tr -cd '0-9' < pr-number.txt)
|
||||||
|
[ -n "$pr" ] || { echo "no PR number"; exit 0; }
|
||||||
|
# Trust the number only if that PR's head is the commit that was benchmarked.
|
||||||
|
head=$(gh api "repos/$REPO/pulls/$pr" --jq .head.sha)
|
||||||
|
[ "$head" = "$HEAD_SHA" ] || { echo "PR #$pr has moved on since this run"; exit 0; }
|
||||||
|
{ echo '<!-- perf-bench -->'; head -c 60000 perf.md; } > body.md
|
||||||
|
id=$(gh api "repos/$REPO/issues/$pr/comments" --paginate --jq '.[] | select(.user.login == "github-actions[bot]" and (.body | startswith("<!-- perf-bench -->"))) | .id' | head -n 1)
|
||||||
|
if [ -n "$id" ]; then gh api -X PATCH "repos/$REPO/issues/comments/$id" -F body=@body.md > /dev/null
|
||||||
|
else gh api -X POST "repos/$REPO/issues/$pr/comments" -F body=@body.md > /dev/null; fi
|
||||||
|
echo "Posted to #$pr"
|
||||||
50
.github/workflows/perf.yml
vendored
Normal file
50
.github/workflows/perf.yml
vendored
Normal file
|
|
@ -0,0 +1,50 @@
|
||||||
|
name: Performance
|
||||||
|
|
||||||
|
# Times the PR against its base on a very large network and uploads the table. A separate workflow
|
||||||
|
# (perf-comment.yml) posts it on the PR: runs for pull requests from forks have no write access.
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
paths:
|
||||||
|
- 'site/**'
|
||||||
|
- 'scripts/perf-*.js'
|
||||||
|
- 'tests/fixtures/**'
|
||||||
|
- 'tests/harness.js'
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
bench:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 20
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v7
|
||||||
|
- uses: actions/setup-node@v7
|
||||||
|
with:
|
||||||
|
node-version: 24
|
||||||
|
cache: npm
|
||||||
|
- run: npm ci
|
||||||
|
- name: Base version of the game
|
||||||
|
if: github.event_name == 'pull_request'
|
||||||
|
env:
|
||||||
|
BASE_SHA: ${{ github.event.pull_request.base.sha }}
|
||||||
|
run: |
|
||||||
|
git fetch --depth=1 origin "$BASE_SHA"
|
||||||
|
git show FETCH_HEAD:site/index.html > "$RUNNER_TEMP/base.html"
|
||||||
|
echo "BASE_ARG=--base $RUNNER_TEMP/base.html" >> "$GITHUB_ENV"
|
||||||
|
- name: Benchmark
|
||||||
|
run: node scripts/perf-bench.js $BASE_ARG --md perf.md --summary "$GITHUB_STEP_SUMMARY"
|
||||||
|
- name: Remember the PR number
|
||||||
|
if: github.event_name == 'pull_request'
|
||||||
|
env:
|
||||||
|
PR: ${{ github.event.pull_request.number }}
|
||||||
|
run: echo "$PR" > pr-number.txt
|
||||||
|
- uses: actions/upload-artifact@v4
|
||||||
|
if: github.event_name == 'pull_request'
|
||||||
|
with:
|
||||||
|
name: perf
|
||||||
|
path: |
|
||||||
|
perf.md
|
||||||
|
pr-number.txt
|
||||||
|
retention-days: 7
|
||||||
|
|
@ -17,6 +17,7 @@ npm test # node:test + jsdom, all suites, must stay green
|
||||||
node scripts/i18n-check.js # must report "0 missing, 0 unused, 0 broken"
|
node scripts/i18n-check.js # must report "0 missing, 0 unused, 0 broken"
|
||||||
npm run build # builds _site/ (stamps version, bakes release notes)
|
npm run build # builds _site/ (stamps version, bakes release notes)
|
||||||
npm run serve # build and serve locally
|
npm run serve # build and serve locally
|
||||||
|
npm run perf -- --base /tmp/base.html # big-network benchmark vs another version (see `performance`)
|
||||||
```
|
```
|
||||||
|
|
||||||
To play during development, open `site/index.html` directly or run `npm run serve`.
|
To play during development, open `site/index.html` directly or run `npm run serve`.
|
||||||
|
|
@ -46,6 +47,10 @@ To play during development, open `site/index.html` directly or run `npm run serv
|
||||||
and bottom sheets. Check `isPhone()` layouts, not just desktop.
|
and bottom sheets. Check `isPhone()` layouts, not just desktop.
|
||||||
- **Saves are sacred.** Old saves must keep loading. Any new state field gets a default
|
- **Saves are sacred.** Old saves must keep loading. Any new state field gets a default
|
||||||
in `newGame()` and a check/repair in `sanitize()`; renamed fields are migrated there.
|
in `newGame()` and a check/repair in `sanitize()`; renamed fields are migrated there.
|
||||||
|
- **Performance is always demonstrated.** Every change to `site/` is measured on the big-network
|
||||||
|
fixture against `main` (`npm run perf -- --base /tmp/base.html`, see the `performance` skill),
|
||||||
|
and the numbers go into the PR and into your summary. Every PR also gets them as a comment from
|
||||||
|
CI. Anything slower needs a reason or a fix.
|
||||||
- **Tests come with features.** Add or extend a test in `tests/` for every feature or
|
- **Tests come with features.** Add or extend a test in `tests/` for every feature or
|
||||||
fix. Tests are seeded, so they must not depend on luck.
|
fix. Tests are seeded, so they must not depend on luck.
|
||||||
- **CHANGELOG style:** plain English, written for players, one bullet per change,
|
- **CHANGELOG style:** plain English, written for players, one bullet per change,
|
||||||
|
|
|
||||||
|
|
@ -39,6 +39,10 @@ A tour of the code is in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
|
||||||
must not depend on luck.
|
must not depend on luck.
|
||||||
- Add a bullet for players under `## [Unreleased]` in `CHANGELOG.md`, in plain English.
|
- Add a bullet for players under `## [Unreleased]` in `CHANGELOG.md`, in plain English.
|
||||||
Releases are cut by the maintainer.
|
Releases are cut by the maintainer.
|
||||||
|
- Show the performance of changes to `site/` on a big network. The Performance check posts a table
|
||||||
|
comparing your PR with `main` as a comment; to run it yourself:
|
||||||
|
`git show origin/main:site/index.html > /tmp/base.html && npm run perf -- --base /tmp/base.html`.
|
||||||
|
Anything marked slower needs an explanation.
|
||||||
- Old saves must keep loading. New state needs a default for new games and a check in
|
- Old saves must keep loading. New state needs a default for new games and a check in
|
||||||
`sanitize()`.
|
`sanitize()`.
|
||||||
- Check your change on a phone-sized screen (about 360 px wide). Most players are on
|
- Check your change on a phone-sized screen (about 360 px wide). Most players are on
|
||||||
|
|
|
||||||
|
|
@ -38,6 +38,7 @@ Other files:
|
||||||
| `site/site.webmanifest`, icons | PWA install metadata |
|
| `site/site.webmanifest`, icons | PWA install metadata |
|
||||||
| `scripts/build.sh` | Copies `site/` to `_site/`, stamps `__VERSION__`/`__BUILD_DATE__`, bakes release notes, writes `version.json` |
|
| `scripts/build.sh` | Copies `site/` to `_site/`, stamps `__VERSION__`/`__BUILD_DATE__`, bakes release notes, writes `version.json` |
|
||||||
| `scripts/release-notes.js` | Extracts a version's CHANGELOG bullets |
|
| `scripts/release-notes.js` | Extracts a version's CHANGELOG bullets |
|
||||||
|
| `scripts/perf-bench.js`, `scripts/perf-fixture.js` | Big-network benchmark (base vs head, posted on every PR by `perf.yml` + `perf-comment.yml`) and the generator of its save, `tests/fixtures/big-network.json` |
|
||||||
| `scripts/i18n-check.js` | Compares every `tr`/`trn`/`N_` literal and `data-i18n` text with each dictionary |
|
| `scripts/i18n-check.js` | Compares every `tr`/`trn`/`N_` literal and `data-i18n` text with each dictionary |
|
||||||
| `tests/harness.js` | Loads the page in jsdom with a seeded `Math.random` and a fake canvas |
|
| `tests/harness.js` | Loads the page in jsdom with a seeded `Math.random` and a fake canvas |
|
||||||
| `.github/workflows/ci.yml` | Tests and build on every push and PR |
|
| `.github/workflows/ci.yml` | Tests and build on every push and PR |
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,8 @@
|
||||||
"test": "node --test --test-concurrency=1 tests/game.test.js tests/build.test.js tests/persist.test.js tests/services.test.js tests/peering.test.js tests/issues.test.js tests/regions.test.js tests/modes.test.js tests/levels.test.js tests/speed.test.js tests/i18n.test.js",
|
"test": "node --test --test-concurrency=1 tests/game.test.js tests/build.test.js tests/persist.test.js tests/services.test.js tests/peering.test.js tests/issues.test.js tests/regions.test.js tests/modes.test.js tests/levels.test.js tests/speed.test.js tests/i18n.test.js",
|
||||||
"build": "bash scripts/build.sh",
|
"build": "bash scripts/build.sh",
|
||||||
"i18n:check": "node scripts/i18n-check.js",
|
"i18n:check": "node scripts/i18n-check.js",
|
||||||
"serve": "npm run build && npx --yes serve@14.2.4 _site"
|
"serve": "npm run build && npx --yes serve@14.2.4 _site",
|
||||||
|
"perf": "node scripts/perf-bench.js"
|
||||||
},
|
},
|
||||||
"engines": { "node": ">=20" },
|
"engines": { "node": ">=20" },
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
|
|
|
||||||
92
scripts/perf-bench.js
Normal file
92
scripts/perf-bench.js
Normal file
|
|
@ -0,0 +1,92 @@
|
||||||
|
#!/usr/bin/env node
|
||||||
|
// Performance benchmark on a very large network (tests/fixtures/big-network.json).
|
||||||
|
//
|
||||||
|
// node scripts/perf-bench.js time this checkout
|
||||||
|
// node scripts/perf-bench.js --base OLD.html compare OLD.html (e.g. main) with this checkout
|
||||||
|
// node scripts/perf-bench.js --base OLD.html --md out.md --summary "$GITHUB_STEP_SUMMARY"
|
||||||
|
//
|
||||||
|
// Both versions load the same save in one process and are timed in alternating turns, so a busy machine slows
|
||||||
|
// them equally. Each number is the median of several runs. It also says whether both versions route the save's
|
||||||
|
// traffic identically, which catches unintended gameplay changes in a "pure speed" PR.
|
||||||
|
// Headless (jsdom, a recording canvas): it measures our JavaScript, not the browser's painting.
|
||||||
|
const fs = require('node:fs');
|
||||||
|
const path = require('node:path');
|
||||||
|
const { load } = require('../tests/harness');
|
||||||
|
|
||||||
|
const args = process.argv.slice(2);
|
||||||
|
const opt = (k, d) => { const i = args.indexOf(k); return i >= 0 ? args[i + 1] : d; };
|
||||||
|
const ROOT = path.join(__dirname, '..');
|
||||||
|
const HEAD = opt('--head', path.join(ROOT, 'site', 'index.html'));
|
||||||
|
const BASE = opt('--base', null);
|
||||||
|
const RUNS = +opt('--runs', 7);
|
||||||
|
const SAVE = fs.readFileSync(opt('--save', path.join(ROOT, 'tests', 'fixtures', 'big-network.json')), 'utf8');
|
||||||
|
|
||||||
|
// What we time. Each body runs inside the game page; panels flip between two selections so nothing can be
|
||||||
|
// skipped as "unchanged".
|
||||||
|
const METRICS = [
|
||||||
|
['compute', 'Routing: compute()', 'compute()'],
|
||||||
|
['day', 'One game day: dayTick()', 'S.nextFail=S.day+99;S.nextEvent=S.day+99;dayTick()'],
|
||||||
|
['action', 'An action: after()', 'after()'],
|
||||||
|
['panels', 'Side panel and left list', "__flip=!__flip;UI.sel={type:'city',id:__flip?'fra':'ams'};UI.tab=__flip?'peer':'up';UI.left=__flip?'nets':'noc';UI.leftOpen=true;renderPanel();renderLeft()"],
|
||||||
|
['dash', 'NOC dashboard tabs', "$('dash').hidden=false;for(const k of ['overview','conn','traffic','infra','cust']){UI.dtab=k;renderDash();}$('dash').hidden=true"],
|
||||||
|
['map', 'Map at three zoom levels', "for(const z of [1,3.2,7]){cam.x=CITY.fra.x;cam.y=CITY.fra.y;cam.z=z;draw(0);}"],
|
||||||
|
];
|
||||||
|
|
||||||
|
function open(file) {
|
||||||
|
const { ev, errors } = load({ file, seed: 777 });
|
||||||
|
ev(`window.__flip=false;S=migrate(JSON.parse(${JSON.stringify(SAVE)}));UI.started=true;UI.pitched=true;UI.speed=0;closeModal();compute();renderAll();`);
|
||||||
|
const time = (body) => ev(`(()=>{const t=performance.now();${body};return performance.now()-t;})()`);
|
||||||
|
// What the network carries, to tell a pure speed-up from a change in how traffic is routed.
|
||||||
|
const outcome = () => ev(`(()=>{compute();const r=[R.served,R.demand,R.reach,R.peered,JSON.stringify(R.egT),...Object.keys(R.citySrv).sort().map(c=>R.citySrv[c]),...S.links.map(l=>R.load[l.id]||0)];
|
||||||
|
return r.map(x=>typeof x==='number'?x.toFixed(6):x).join('|');})()`);
|
||||||
|
const size = JSON.parse(ev("JSON.stringify({pops:Object.keys(S.pops).length,links:S.links.length,pni:S.pni.length,caches:S.caches.length,customers:S.biz.length,ix:Object.keys(S.ixp).length})"));
|
||||||
|
return { ev, errors, time, outcome, size };
|
||||||
|
}
|
||||||
|
|
||||||
|
const median = (a) => { const s = [...a].sort((x, y) => x - y); return s.length % 2 ? s[(s.length - 1) / 2] : (s[s.length / 2 - 1] + s[s.length / 2]) / 2; };
|
||||||
|
const ms = (x) => x >= 100 ? x.toFixed(0) : x >= 10 ? x.toFixed(1) : x.toFixed(2);
|
||||||
|
|
||||||
|
const head = open(HEAD), base = BASE ? open(BASE) : null;
|
||||||
|
const sameOutcome = base ? base.outcome() === head.outcome() : null;
|
||||||
|
const res = {};
|
||||||
|
for (const [k, , body] of METRICS) {
|
||||||
|
const h = [], b = [];
|
||||||
|
for (const v of [head, base].filter(Boolean)) { v.time(body); v.time(body); } // warm up
|
||||||
|
for (let i = 0; i < RUNS; i++) {
|
||||||
|
// Alternate who goes first, so drift over the run lands on both sides.
|
||||||
|
if (base && i % 2) { b.push(base.time(body)); h.push(head.time(body)); } else { h.push(head.time(body)); if (base) b.push(base.time(body)); }
|
||||||
|
}
|
||||||
|
res[k] = { head: median(h), base: base ? median(b) : null };
|
||||||
|
}
|
||||||
|
const errs = [...head.errors, ...(base ? base.errors : [])];
|
||||||
|
|
||||||
|
// Verdicts: small absolute differences are noise, whatever the percentage.
|
||||||
|
const verdict = (r) => {
|
||||||
|
if (r.base == null) return '';
|
||||||
|
const ratio = r.head / (r.base || 1e-9), diff = r.head - r.base;
|
||||||
|
if (Math.abs(diff) < 2) return '≈';
|
||||||
|
if (ratio >= 1.5) return '🔴 slower';
|
||||||
|
if (ratio >= 1.15) return '⚠️ slower';
|
||||||
|
if (ratio <= 0.85) return '🟢 faster';
|
||||||
|
return '≈';
|
||||||
|
};
|
||||||
|
const pct = (r) => r.base == null ? '' : `${r.head >= r.base ? '+' : ''}${(((r.head - r.base) / (r.base || 1e-9)) * 100).toFixed(0)}%`;
|
||||||
|
const s = head.size;
|
||||||
|
let md = `## Performance on a big network\n\n${s.pops} POPs · ${s.links} links · ${s.pni} PNIs · ${s.caches} cache boxes · ${s.ix} IX ports · ${s.customers} customers. Median of ${RUNS} runs, in milliseconds, headless.\n\n`;
|
||||||
|
md += base ? '| What | Base | This PR | Change | |\n|---|--:|--:|--:|---|\n' : '| What | Time |\n|---|--:|\n';
|
||||||
|
for (const [k, label] of METRICS) {
|
||||||
|
const r = res[k];
|
||||||
|
md += base ? `| ${label} | ${ms(r.base)} | ${ms(r.head)} | ${pct(r)} | ${verdict(r)} |\n` : `| ${label} | ${ms(r.head)} |\n`;
|
||||||
|
}
|
||||||
|
if (base) {
|
||||||
|
md += `\n${sameOutcome ? '✅ Routing results are identical to the base.' : '⚠️ Routing results differ from the base: traffic is carried differently. Expected if this PR changes gameplay; a bug if it only promises to be faster.'}\n`;
|
||||||
|
const slow = METRICS.filter(([k]) => /slower/.test(verdict(res[k])));
|
||||||
|
md += slow.length ? `\n**Slower:** ${slow.map(([, l]) => l).join(', ')}. Please explain or fix before merging.\n` : '\nNo slowdowns beyond noise.\n';
|
||||||
|
}
|
||||||
|
if (errs.length) md += `\n⚠️ Page errors while benchmarking:\n\n${errs.slice(0, 5).map((e) => '- `' + String(e).slice(0, 200).replace(/`/g, "'") + '`').join('\n')}\n`;
|
||||||
|
md += `\n<sub>scripts/perf-bench.js · jsdom with a recording canvas: our JavaScript, not the browser's painting. Below about 2 ms, differences are noise.</sub>\n`;
|
||||||
|
|
||||||
|
console.log(md);
|
||||||
|
const out = opt('--md', null); if (out) fs.writeFileSync(out, md);
|
||||||
|
const sum = opt('--summary', null); if (sum) fs.appendFileSync(sum, md + '\n');
|
||||||
|
const js = opt('--json', null); if (js) fs.writeFileSync(js, JSON.stringify({ size: s, runs: RUNS, results: res, sameOutcome, errors: errs }, null, 2));
|
||||||
39
scripts/perf-fixture.js
Normal file
39
scripts/perf-fixture.js
Normal file
|
|
@ -0,0 +1,39 @@
|
||||||
|
#!/usr/bin/env node
|
||||||
|
// Builds tests/fixtures/big-network.json: a deterministic, very large late-game network that the performance
|
||||||
|
// benchmark (scripts/perf-bench.js) loads into the base and the head of every pull request. Big on purpose: about
|
||||||
|
// a hundred POPs, a few hundred links and customers, and close to a thousand PNIs and cache boxes, like the saves
|
||||||
|
// that made the game lag (#89). Regenerate only when the save format changes in a way old saves cannot load:
|
||||||
|
// node scripts/perf-fixture.js
|
||||||
|
const fs = require('node:fs');
|
||||||
|
const path = require('node:path');
|
||||||
|
const { load } = require('../tests/harness');
|
||||||
|
|
||||||
|
const { ev, errors } = load({ seed: 4242 });
|
||||||
|
ev(`(()=>{UI.pitched=true;ACT.start({});S.cash=1e13;S.lvl=LEVELS.length;S.won=true;for(const g of GOALS)S.done[g.id]=0;S.tut=null;
|
||||||
|
const top=RG.findIndex(r=>r.lvl===LEVELS.length&&r.maxP===PORT.length-1);
|
||||||
|
const cs=ALLC.filter(c=>!CITY[c].ixOnly&&!CITY[c].cn);
|
||||||
|
for(const c of cs){if(!S.pops[c])ACT.buildPop({city:c});const p=S.pops[c];if(!p)continue;p.rgen=top>=0?top:RG.length-1;p.routers=RMAX;if(!p.bb)ACT.launchBB({city:c});if(p.bb)p.bb.d=CITY[c].P*0.65;if(Math.random()<0.3)ACT.addSite({city:c});}
|
||||||
|
const pops=Object.keys(S.pops);
|
||||||
|
// A backbone: every POP to its two nearest neighbours, plus long-haul shortcuts.
|
||||||
|
const L=(a,b,t)=>{if(a===b)return;UI.pending=null;UI.via=null;UI.lmode='wave';UI.route='diverse';try{openLinkModal(a,b);ACT.buildLink({tier:t});closeModal();}catch(e){closeModal();}};
|
||||||
|
for(const a of pops){const near=pops.filter(b=>b!==a).sort((x,y)=>kmBetween(a,x)-kmBetween(a,y)).slice(0,2);for(const b of near)if(!pairLinks(a,b).length)L(a,b,3);}
|
||||||
|
for(let i=0;i<60;i++){const a=pops[Math.floor(Math.random()*pops.length)],b=pops[Math.floor(Math.random()*pops.length)];L(a,b,4);}
|
||||||
|
// Transit in some cities, IX ports with route servers everywhere.
|
||||||
|
for(const c of pops.slice(0,30)){const ts=NETS.filter(n=>n.price&&n.pops.includes(c)&&(n.type==='t1'||n.type==='t2')).slice(0,2);for(const t of ts)ACT.buyTransit({city:c,net:t.id,tier:3});}
|
||||||
|
for(const c of pops)for(const x of ixIdsIn(c)){try{ACT.buyIx({city:x,tier:3});}catch(e){}if(S.dk&&S.dk.q){S.dk.q=null;}closeModal();if(S.ixp[x]!==undefined&&!S.rs[x])ACT.rs({city:x});}
|
||||||
|
compute();
|
||||||
|
// PNIs wherever a network is present, and cache boxes from every CDN.
|
||||||
|
for(const n of NETS)for(const c of pops)if(S.pni.length<950&&n.pops.includes(c)&&!n.tail)S.pni.push({id:'p'+n.id+'_'+c,net:n.id,city:c,tier:3,n:1,site:'a'});
|
||||||
|
for(const n of NETS.filter(n=>n.cache))for(const c of pops)for(const k of [1,2])if(S.caches.length<850)S.caches.push({id:'c'+k+n.id+'_'+c,city:c,net:n.id,site:'a'});
|
||||||
|
// Real customers: accept offers until there are a few hundred.
|
||||||
|
for(let i=0;i<6000&&S.biz.length<420;i++){S.offers=[];spawnOffer();const o=S.offers[0];if(o&&bHas(o))ACT.accept({id:o.id});}
|
||||||
|
S.offers=[];S.nextFail=S.day+30;S.nextEvent=S.day+30;S.inc=[];
|
||||||
|
for(let d=0;d<20;d++)dayTick();
|
||||||
|
})()`);
|
||||||
|
if (errors.length) { console.error(errors); process.exit(1); }
|
||||||
|
const save = ev('JSON.stringify(S)');
|
||||||
|
const n = JSON.parse(ev("JSON.stringify({pops:Object.keys(S.pops).length,links:S.links.length,pni:S.pni.length,caches:S.caches.length,biz:S.biz.length,transit:S.transit.length,ix:Object.keys(S.ixp).length,served:Math.round(R.served)})"));
|
||||||
|
const out = path.join(__dirname, '..', 'tests', 'fixtures', 'big-network.json');
|
||||||
|
fs.mkdirSync(path.dirname(out), { recursive: true });
|
||||||
|
fs.writeFileSync(out, save + '\n');
|
||||||
|
console.log(`wrote ${path.relative(process.cwd(), out)} (${Math.round(save.length / 1024)} KB):`, n);
|
||||||
1
tests/fixtures/big-network.json
vendored
Normal file
1
tests/fixtures/big-network.json
vendored
Normal file
File diff suppressed because one or more lines are too long
|
|
@ -92,3 +92,13 @@ test('in-app release notes stay short and link to the full notes on GitHub', ()
|
||||||
assert.equal(ev(`slimNote('${'x'.repeat(300)}')`).length <= 140, true);
|
assert.equal(ev(`slimNote('${'x'.repeat(300)}')`).length <= 140, true);
|
||||||
assert.deepEqual(errors, []);
|
assert.deepEqual(errors, []);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('the big-network benchmark save loads cleanly and routes traffic', () => {
|
||||||
|
const fs = require('node:fs'), path = require('node:path');
|
||||||
|
const save = fs.readFileSync(path.join(__dirname, 'fixtures', 'big-network.json'), 'utf8');
|
||||||
|
const { ev, errors } = load();
|
||||||
|
ev(`S=migrate(JSON.parse(${JSON.stringify(save)}));UI.started=true;compute();renderAll()`);
|
||||||
|
assert.ok(ev('Object.keys(S.pops).length') >= 100 && ev('S.pni.length') >= 900 && ev('S.caches.length') >= 800, 'still a big network');
|
||||||
|
assert.ok(ev('R.served') > 10000, 'and it carries traffic');
|
||||||
|
assert.deepEqual(errors, []);
|
||||||
|
});
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue