From 065c922287b0575654edcd97d403d3fa165259ac Mon Sep 17 00:00:00 2001 From: jvanvik Date: Sat, 7 Nov 2026 11:21:23 +0100 Subject: [PATCH 1/7] Server-run play, part 2: the engine The engine that runs every network of a server session (server-sim design, section 4): each network in its own vm realm, realms in a small pool of worker threads, a promise API for the session host. Nothing uses it yet; the plain build and the game are unchanged (no file under src/). - scripts/lib/mp-sim-realm.js: one network in a DONT_CONTEXTIFY realm running the plain page's game (never the MP page's client folder), with recording DOM stand-ins, no-op timers that count, and the srv... calls: load from a checkpoint (route memory seeded, then routed), begin a new network, marks and restores, the day as a page runs it, the action pipeline with drafts, arming, the sanitize gate and rollback, what-ifs that put S, R, the dice and uidN back, pulls with top-level key diffs, slim R and flow records on demand, fx captured and bounded, saves, cards. - scripts/lib/mp-sim-worker.js: realms per session in one worker, days one network at a time with other messages between them, loads and unloads waiting for a session's day, time limits; a throw goes back to the mark in place, a timeout gets a fresh realm (new epoch) carrying the old R, a day is retried once and the network broken after a second failure. - scripts/lib/mp-sim-engine.js: createEngine() with the fake engine's API, placement by realm count, every pending call rejected when its worker exits, a worker of its own after two exits in an hour; snapWith() joins the host's clock, roster and since to a realm's snapshot. - Contract: a pulled snapshot is the realm's half (the host adds clock, roster, since); haptic fx carry milliseconds; the fake engine follows, with close() and snapWith(). The button crawl moved to tests/lib. - Tests: realm vs jsdom byte for byte over 60 scenic big-network days with actions; restore; retry after a throw or a timeout; realm order; a timer audit and the gate over a sweep of sent buttons (SIM_LONG=1: a year and every button); captures, rollbacks, asks, selection reports, ordering, bounds, pollution, worker exits; the act tables against the game. The gate found two keys the game writes in a shape a reload repairs (buyTE's S.te, mkLeave's S.monkey): game fixes for the integrator, kept visible as a todo test and a list in the audit. Rebased onto the 3.3.0 merge (61e31c5): the engine itself needed no change. A realm still routes in the game's own code (no route worker, no saves), the route cache that now keeps list orders from day to day leaves a checkpoint and a retry the same, and the capacity market and moved customers play in a realm as on a page, under the strict gate. The tests needed three things: an old save's first days earn the new traffic badges, which the unwatched network's snapshot keeps; the restore test compares JSON text with sorted keys instead of a deep-equal of two networks (a diff of gigabytes when they differ); and the crawl opens the windows that move a customer, with a new test playing the market's and the path moves' acts, 40 days of sales and a checkpoint against the page. --- scripts/lib/mp-sim-engine.js | 182 +++++++++++++ scripts/lib/mp-sim-realm.js | 444 +++++++++++++++++++++++++++++++ scripts/lib/mp-sim-worker.js | 290 ++++++++++++++++++++ tests/durations.json | 3 + tests/lib/mp-sim-crawl.js | 28 ++ tests/lib/mp-sim-fake-engine.js | 13 +- tests/lib/mp-sim-play.js | 85 ++++++ tests/lib/mp-sim-shape.js | 22 +- tests/mp-sim-acts.test.js | 220 +++++++++++++++ tests/mp-sim-contract.test.js | 26 +- tests/mp-sim-determinism.test.js | 314 ++++++++++++++++++++++ tests/mp-sim-engine.test.js | 308 +++++++++++++++++++++ 12 files changed, 1906 insertions(+), 29 deletions(-) create mode 100644 scripts/lib/mp-sim-engine.js create mode 100644 scripts/lib/mp-sim-realm.js create mode 100644 scripts/lib/mp-sim-worker.js create mode 100644 tests/lib/mp-sim-crawl.js create mode 100644 tests/lib/mp-sim-play.js create mode 100644 tests/mp-sim-acts.test.js create mode 100644 tests/mp-sim-determinism.test.js create mode 100644 tests/mp-sim-engine.test.js diff --git a/scripts/lib/mp-sim-engine.js b/scripts/lib/mp-sim-engine.js new file mode 100644 index 0000000..3887e76 --- /dev/null +++ b/scripts/lib/mp-sim-engine.js @@ -0,0 +1,182 @@ +// The server-run engine (the server-sim design, sections 4.2-4.3): a small pool of worker threads (mp-sim-worker.js), +// each holding the vm realms (mp-sim-realm.js) of the sessions placed on it, behind a promise API. The session host +// (scripts/lib/mp-sim.js) keeps the clock, presence and votes, and asks this for days, actions, what-ifs and pulls. +// tests/lib/mp-sim-fake-engine.js has the same API with canned answers. +// +// const engine = createEngine({workers: 2, heapMb: 1536, root, log}); +// engine.load(code, [{nid, env?, begin?, lang, prefs, rng?}]) → [{t:'loaded', nid, epoch, day, lost, saved?} | +// {t:'loaded', nid, err}], one per network, in order. env: the checkpoint envelope {s, uidN, rt, last, +// day} (s: the save text); begin: {name, home, preset, asn, mp, tut} builds a new network, checkpointed +// at once (saved). A network whose save does not load (or that cannot begin) answers err: broken (4.7). +// engine.day(code, {d, speed, last, alone:[nid], nets:[nid], mute}) → {t:'dayDone', code, d, nets:{[nid]: {day, lost, +// pause?, err?, retried?, ms}}}. Each network in sorted order; one that failed twice answers err and +// its day as it was, and is broken from then on (no more days; actions answer 'stopped'). +// engine.act(code, nid, msg) → {t:'actDone', nid, epoch, rid, outcome, why?, v, upd} (upd: the update with the result, +// version v; null when stopped) | {t:'err', code, nid, rid, what:'timeout'|'throw', msg} +// engine.ask(code, nid, msg) → {t:'askDone', nid, rid, modal?, ui?, err?} | {t:'err', …} +// engine.pull(code, nid, {snap, views}) → {t:'pulled', nid, epoch, v, upd?, snap?, subs?}. snap is the realm's half +// of a snapshot: snapWith(snap, {clock, roster, since}) makes the stream event. +// engine.sub(code, nid, {views}) → {t:'subbed', nid, epoch, v, subs}; engine.greet(code, nid) → a pulled +// engine.prefs(code, nid, {prefs}) → undefined; engine.save(code, nid) → {t:'saved', nid, txt, uidN, rt, day, lost} +// engine.card(code, nid) → {t:'card', nid, card}; engine.unload(code, nid?) → {t:'unloaded', code, nid?} +// engine.stats() → {workers, realms, sessions} (not a promise); engine.close() → stops the workers +// engine.simId the build id this engine's game belongs to (SIM_ID of the multiplayer page from the same src/) +// +// Rejections: a call for a network that is not loaded (code 'NO_REALM'), a malformed call, and every call pending on +// a worker that exits (out of memory, a crash): Error('worker exited'), code 'EXIT'; that worker's sessions are gone and +// the host loads them again (4.7). A session whose worker exited twice within an hour gets a worker of its own; when that +// one exits too, its loads reject with code 'GAVE_UP' until the engine is made again (the server restarts). +// Epochs: every realm load mints one (a load, a begin, the fresh realm after a timeout); loaded, pulled, actDone and +// subbed carry the realm's current one. Versions are the realm's: only a pull (or an action's own pull) makes one. +// +// Tests only, with createEngine({test: {seed, now, step} | true}): engine.eval(code, nid, src) runs code in a realm +// ({value} | {error} | {timeout, epoch}); engine.mem() reports each worker's heap; engine.kill(i) ends worker i. +'use strict'; +const path = require('node:path'); +const vm = require('node:vm'); +const { Worker } = require('node:worker_threads'); + +const FILE = path.join(__dirname, 'mp-sim-worker.js'); +const HOUR = 3600e3; + +/* The stream event of a snapshot: the realm's half (S, r, keep, epoch, v, now) with the host's (clock, roster, + since), joined as text: the realm's 270 KB are never parsed again. */ +function snapWith(snap, { clock, roster, since }) { + if (typeof snap !== 'string' || !snap.endsWith('}')) throw new Error('snapWith: not a snapshot'); + return snap.slice(0, -1) + ',"clock":' + JSON.stringify(clock) + ',"roster":' + JSON.stringify(roster) + ',"since":' + JSON.stringify(since) + '}'; +} + +function createEngine(opts = {}) { + if (!vm.constants || vm.constants.DONT_CONTEXTIFY === undefined) { + throw new Error(`server-run play needs Node 22.8 or newer (vm.constants.DONT_CONTEXTIFY); this is ${process.version}`); + } + const N = Math.max(1, Math.floor(+opts.workers || 2)); + const heapMb = Math.max(64, Math.floor(+opts.heapMb || 1536)); + const log = typeof opts.log === 'function' ? opts.log : () => {}; + const test = opts.test || null; + const W = { root: opts.root, page: opts.page, test, strict: opts.strict !== undefined ? !!opts.strict : process.env.SIM_STRICT === '1', timeouts: opts.timeouts }; + let seq = 0, closed = false; + const pool = [], own = new Map(), placed = new Map(), exits = new Map(), gaveUp = new Set(); + // A session's realms, counted where they live (placement by realm count). + const count = (w) => [...w.sessions.values()].reduce((a, b) => a + b.size, 0); + + function spawn(dedicated = false) { + const w = { worker: null, pending: new Map(), sessions: new Map(), dedicated, dead: false }; + w.worker = new Worker(FILE, { workerData: { mpSim: W }, resourceLimits: { maxOldGenerationSizeMb: heapMb } }); + w.worker.on('message', (m) => { + if (m.log) { log(m.log[0], m.log[1]); return; } + if (m.fatal) { log('error', 'mp-sim worker could not start: ' + m.fatal); return; } + if (m.ready) return; + const p = w.pending.get(m.id); + if (!p) return; + w.pending.delete(m.id); + // An idle worker does not keep the process alive; one with calls pending does. + if (!w.pending.size) w.worker.unref(); + if (m.ok) p.ok(m.reply); + else p.no(Object.assign(new Error(m.error.message), { code: m.error.code })); + }); + w.worker.on('error', (e) => log('error', 'mp-sim worker failed: ' + (e && e.message))); + w.worker.on('exit', (c) => died(w, c)); + w.worker.unref(); + return w; + } + function died(w, c) { + if (w.dead) return; + w.dead = true; + const e = Object.assign(new Error('worker exited'), { code: 'EXIT', exitCode: c }); + for (const p of w.pending.values()) p.no(e); + w.pending.clear(); + const t = Date.now(); + for (const code of w.sessions.keys()) { + placed.delete(code); + if (w.dedicated) { gaveUp.add(code); own.delete(code); log('error', `mp-sim: session ${code} took down its own worker too; not loading it again until a restart`); continue; } + exits.set(code, (exits.get(code) || []).filter((x) => t - x < HOUR).concat(t)); + } + w.sessions.clear(); + if (!closed) { + if (!closed) log('warn', `mp-sim: a worker exited (code ${c}); its sessions load again from their checkpoints`); + const i = pool.indexOf(w); + if (i >= 0) pool[i] = spawn(); + } + } + for (let i = 0; i < N; i++) pool.push(spawn()); + + // The worker of a session: where it is, else the pool's least loaded one (or one of its own, after two exits). + function workerOf(code, create) { + let w = placed.get(code); + if (w && !w.dead) return w; + if (!create) return null; + if (gaveUp.has(code)) throw Object.assign(new Error(`session ${code} kept failing its worker`), { code: 'GAVE_UP' }); + if ((exits.get(code) || []).filter((x) => Date.now() - x < HOUR).length >= 2) { + w = spawn(true); own.set(code, w); + } else w = pool.reduce((a, b) => (count(b) < count(a) ? b : a)); + placed.set(code, w); + return w; + } + function send(w, m) { + if (closed) return Promise.reject(Object.assign(new Error('engine closed'), { code: 'CLOSED' })); + if (!w || w.dead) return Promise.reject(Object.assign(new Error('worker exited'), { code: 'EXIT' })); + const id = ++seq; + return new Promise((ok, no) => { if (!w.pending.size) w.worker.ref(); w.pending.set(id, { ok, no }); w.worker.postMessage({ id, ...m }); }); + } + const noRealm = (code, nid) => Promise.reject(Object.assign(new Error(`no realm ${code}/${nid}`), { code: 'NO_REALM' })); + const to = (code, nid, m) => { const w = workerOf(code, false); return w ? send(w, { code, nid, ...m }) : noRealm(code, nid); }; + const mine = (w, code) => { let s = w.sessions.get(code); if (!s) w.sessions.set(code, s = new Set()); return s; }; + + const engine = { + simId: null, + load(code, nets) { + if (!Array.isArray(nets) || !nets.length) return Promise.reject(new Error('load: give the networks to load')); + let w; + try { w = workerOf(code, true); } catch (e) { return Promise.reject(e); } + return send(w, { t: 'load', code, nets }).then((r) => { + const s = mine(w, code); + for (const l of r) if (!l.err) s.add(l.nid); else s.delete(l.nid); + return r; + }); + }, + day(code, { d, speed = 1, last = 1, alone = [], nets = [], mute = false } = {}) { + const w = workerOf(code, false); + if (!w) return Promise.reject(Object.assign(new Error(`no session ${code} here`), { code: 'NO_REALM' })); + return send(w, { t: 'day', code, d, speed, last, alone, nets, mute }); + }, + act(code, nid, msg) { return to(code, nid, { t: 'act', msg }); }, + ask(code, nid, msg) { return to(code, nid, { t: 'ask', msg }); }, + pull(code, nid, { snap = false, views } = {}) { return to(code, nid, { t: 'pull', snap: !!snap, ...(views ? { views } : {}) }); }, + sub(code, nid, { views = [] } = {}) { return to(code, nid, { t: 'sub', views }); }, + prefs(code, nid, { prefs } = {}) { return to(code, nid, { t: 'prefs', prefs }).then(() => undefined); }, + greet(code, nid) { return to(code, nid, { t: 'greet' }); }, + save(code, nid) { return to(code, nid, { t: 'save' }); }, + card(code, nid) { return to(code, nid, { t: 'card' }); }, + unload(code, nid) { + const w = workerOf(code, false); + if (!w) return Promise.resolve({ t: 'unloaded', code, ...(nid !== undefined ? { nid } : {}) }); + return send(w, { t: 'unload', code, ...(nid !== undefined ? { nid } : {}) }).then((r) => { + const s = w.sessions.get(code); + if (s) { if (nid === undefined) s.clear(); else s.delete(nid); if (!s.size) { w.sessions.delete(code); placed.delete(code); } } + if (w.dedicated && !w.sessions.size) { own.delete(code); w.dead = true; w.worker.terminate(); } + return r; + }); + }, + stats() { + const ws = [...pool, ...own.values()].filter((w) => !w.dead); + return { workers: ws.length, realms: ws.reduce((a, w) => a + count(w), 0), sessions: ws.reduce((a, w) => a + w.sessions.size, 0) }; + }, + close() { + closed = true; + const ws = [...pool, ...own.values()]; + return Promise.all(ws.map((w) => (w.dead ? null : w.worker.terminate()))).then(() => undefined); + }, + snapWith, + }; + if (test) { + engine.eval = (code, nid, src, ms) => to(code, nid, { t: 'eval', src: String(src), ...(ms ? { ms } : {}) }); + engine.mem = () => Promise.all(pool.filter((w) => !w.dead).map((w) => send(w, { t: 'mem' }))); + engine.kill = (i = 0) => { const w = pool[i]; if (w && !w.dead) w.worker.terminate(); }; + engine.workerOf = (code) => pool.indexOf(placed.get(code)); + } + { const page = require('./page'); engine.simId = page.simId(page.assemble({ src: path.join(opts.root || page.ROOT, 'src'), mp: true }).html); } + return engine; +} + +module.exports = { createEngine, snapWith }; diff --git a/scripts/lib/mp-sim-realm.js b/scripts/lib/mp-sim-realm.js new file mode 100644 index 0000000..d5a0a19 --- /dev/null +++ b/scripts/lib/mp-sim-realm.js @@ -0,0 +1,444 @@ +// One network of a server session, run by the game's own code in a vm realm (the server-sim design, section 4.1). +// The realm is a vm context made with DONT_CONTEXTIFY, so it has its own intrinsics (a prototype polluted in one realm +// stays there) and runs at full speed. Into it go, in order: the prelude below (DOM stand-ins that record what the +// game writes and reads back, no-op timers that count, the network's language), the network's dictionary +// (site/i18n/.js, not English), the game script, and the epilogue (the srv… functions: what the server asks of a +// network, and the rebinds that turn toasts, sounds, windows and the like into effects, "fx", for the page). +// +// The worker (mp-sim-worker.js) calls into a realm only through call()/raw(): the input goes in as one JSON string +// (globalThis.__in), the answer comes back as one string, and no object crosses between host and realm. An answer is +// a JSON header, optionally followed by more parts, each on its own line (JSON text never holds a raw newline): +// srvPull answers `{"v":3,"parts":["upd","snap"]}\n\n`, ready to write to a stream. +// A call that throws inside the realm answers {"srvErr": stack}; a call that runs over its time limit throws +// ERR_SCRIPT_EXECUTION_TIMEOUT here, and the realm must not be used for play again (a timeout skips finally blocks). +// +// gameSource({root, page}) the game script (the largest inline