Server-run engine: a restored network plays on exactly as before
A network loaded from its checkpoint drifted from the one that never stopped:
after a checkpoint on day 1, 2 or 5 its R differed at once (bizDem at Ashburn
6426 against 26, then cityDem, citySrv, tNet), and from the next day S too
(cash, eyeball growth). Three causes, each shown by a probe:
- R after a day is the R of the day's last routing, made before the day's end
changed S (customers who left or signed, moods, bills, transit history) and
routed from the memory of the R before it. srvLoad routed the saved S again
from R's own memory: another R, which the next day reads (sfOf for eyeball
growth, the peering share). Every compute() in the realm now keeps, for the
R it makes, S as it routed it (one JSON per top-level key, 0.8 ms big) and
the memory it started from. rt carries that routing point (pt: the memory,
the keys that changed since, the keys S did not have then), and srvLoad
routes that S again from it: the same R, byte for byte. The old restore test
checkpointed after day 10, where a failure or event routed again at the end
of the day and nothing changed after it: it passed by luck.
- The sales desk's tried offers (UI.slTried, an offer that did not fit waits a
week) live outside S. A fresh realm forgot them and the desk tried other
offers. rt carries them (sl); a day mark keeps them and the in-place restore
after a throw puts them back (before, a parsed S dropped them).
- A save and a reload give some keys back in another key order (a customer's
defaults go last, the intern's week tally in its own order), and the week's
report lists the tally in that order. At a checkpoint rt carries the live
text of each such key (ord, srvGate's round trip, about 8 ms), taken on load
only where it holds what sanitize() made of the save. A fresh realm after a
timeout takes the mark's own text the same way (live).
The plain game drifts across a save and a reload too (jsdom: from the first
day after it; the page routes a reloaded S with an empty R and keeps no desk
memory): a game property, so the engine carries what is missing.
The envelope keeps its shape {s, uidN, rt, last, day}; rt keeps R's own route
memory, so an older checkpoint loads as before (close, not exact). Big network:
the checkpoint grows from about 42-49 KB gzip to 54-79 KB (rt 7-29 KB; the
whole R would be 375 KB), srvSave from 2-2.5 ms to 7-8 ms, a load by one
routing. A day with its mark costs the same (big 39.8 -> 37.8 ms, busy
90.0 -> 88.8 ms, within noise: the day mark joins per-key JSON now, which pays
for the routing point).
Tests: the restore test checkpoints after days 1, 2, 5 and 10 of the scenic big
network and of the same network busy (intern on every order, the desk signing
on its own, failures, events, an attack), those after days 2 and 10 right after
an action and a what-if (sent to the copies too), each compared byte for byte
at once and 10 days later (SIM_LONG: after every day to 30, 60 checkpoints). It
fails on the old engine, and without any one of the three parts. The retry test
plays busy networks (fails without the desk memory in srvRestore). The market
checkpoint compares bytes too; canon() is gone.
This commit is contained in:
parent
065c922287
commit
c5c7ea9508
5 changed files with 169 additions and 72 deletions
|
|
@ -9,7 +9,7 @@
|
|||
// begin(nid, o) a new network's begin entry for engine.load()
|
||||
// RSIG R with its lazy parts read out, attacks by id (as tests/route-worker.test.js reads it)
|
||||
// checked(r) r, after checkWorker (and checkEvent of the upd/snap inside) found nothing
|
||||
// differs(a, b), canon(s) the first differing top-level keys of two JSON texts; a JSON text with its keys sorted
|
||||
// differs(a, b) the first differing top-level keys of two JSON texts (compare texts, never a deep-equal)
|
||||
'use strict';
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
|
@ -77,9 +77,4 @@ function differs(a, b) {
|
|||
.slice(0, 4).map((k) => `${k}: ${String(JSON.stringify(A2[k])).slice(0, 300)} ≠ ${String(JSON.stringify(B2[k])).slice(0, 300)}`).join('\n');
|
||||
}
|
||||
|
||||
// JSON text with every object's keys sorted (arrays keep their order): two networks that differ only in key order are
|
||||
// the same text. Compare these strings, never the parsed objects with a deep-equal.
|
||||
const sortK = (v) => (Array.isArray(v) ? v.map(sortK) : v && typeof v === 'object' ? Object.keys(v).sort().reduce((o, k) => ((o[k] = sortK(v[k])), o), {}) : v);
|
||||
const canon = (s) => JSON.stringify(sortK(JSON.parse(s)));
|
||||
|
||||
module.exports = { CODE, PID, DEV, MP, T0, STEP, TEST, ROOMY, sim, page, bigScenic, msg, ctxOf, begin, RSIG, checked, differs, canon };
|
||||
module.exports = { CODE, PID, DEV, MP, T0, STEP, TEST, ROOMY, sim, page, bigScenic, msg, ctxOf, begin, RSIG, checked, differs };
|
||||
|
|
|
|||
|
|
@ -2,23 +2,25 @@
|
|||
// - Realm vs jsdom: the big network with its L2 customers on scenic routes, 60 days with actions in between, the
|
||||
// page's arm clicking as a player does (each click ending in after()) and rendering the HUD after every day: S and R
|
||||
// byte for byte the same.
|
||||
// - Restore: 20 days straight, or 10, a checkpoint, a fresh realm and 10 more: the same network.
|
||||
// - Retry: a day that fails once (a throw, and a timeout, both after the day's DDoS step with an attack on) leaves the
|
||||
// same network as no failure.
|
||||
// - Restore: a checkpoint after any day, loaded into a fresh realm, plays on byte for byte as the network that kept
|
||||
// going, the scenic big network and the same one busy (intern, sales desk, failures, events, an attack).
|
||||
// - Retry: a day that fails once (a throw, and a timeout, both after the day's DDoS step with an attack on, the networks
|
||||
// busy) leaves the same network as no failure.
|
||||
// - Order: three networks of a session, in two different realm orders, each the same.
|
||||
// - Market and moved customers (3.2.0, 3.3.0): the market's settings and a customer moved to another fiber or pinned
|
||||
// to a path, then days of sales, against the page and under the strict gate; a checkpoint of it plays on the same.
|
||||
// - Timer audit and gate: days with failures, events, the intern on every order and the sales desk on auto, then every
|
||||
// sent button a rich crawl finds: no timer is ever scheduled on these paths, nothing changes after them, and no
|
||||
// action is rolled back by the sanitize gate (which also runs after every day here, SIM_STRICT).
|
||||
// SIM_LONG=1 plays the audit for a year, as section 11 asks (a few minutes); by default it plays 60 days.
|
||||
// SIM_LONG=1 plays the audit for a year, as section 11 asks, and checkpoints after every day to 30 (a few minutes); by
|
||||
// default the audit plays 60 days.
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { createEngine } = require('../scripts/lib/mp-sim-engine');
|
||||
const { crawl } = require('./lib/mp-sim-crawl');
|
||||
const A = require('../scripts/lib/mp-sim-acts');
|
||||
const { load } = require('./harness');
|
||||
const { CODE, MP, TEST, ROOMY, T0, STEP, page, bigScenic, msg, ctxOf, begin, RSIG, checked, differs, canon } = require('./lib/mp-sim-play');
|
||||
const { CODE, MP, TEST, ROOMY, page, bigScenic, msg, ctxOf, begin, RSIG, checked, differs } = require('./lib/mp-sim-play');
|
||||
|
||||
const LONG = process.env.SIM_LONG === '1';
|
||||
/* Acts the gate refuses because the game leaves a key in a shape sanitize() repairs: game fixes for the integrator (the
|
||||
|
|
@ -27,6 +29,10 @@ const LONG = process.env.SIM_LONG === '1';
|
|||
const PENDING = { buyTE: 'te', mkBanana: 'monkey' };
|
||||
const E = createEngine({ workers: 2, test: TEST, timeouts: ROOMY });
|
||||
test.after(() => E.close());
|
||||
// The big network with everything that keeps state between days on: the intern on every order, the sales desk (rep)
|
||||
// signing on its own, a failure and an event due.
|
||||
const busy = (rep) => `S.lvl=Math.max(S.lvl,6);S.rules=makeRules('bgp');ACT.opsHire({id:'kevin'});for(const k in S.ops.on)S.ops.on[k]=true;S.ops.floor=0;closeModal();
|
||||
ACT.salesHire({id:'${rep}'});ACT.salesAuto();closeModal();S.nextFail=S.day+2;S.nextEvent=S.day+3;compute();`;
|
||||
const on = (X) => ({
|
||||
day: (code, d, nets, o = {}) => X.day(code, { d, speed: 1, last: 1, alone: [], nets, ...o }).then(checked),
|
||||
realmS: async (code, nid) => (await X.eval(code, nid, 'return JSON.stringify(S)')).value,
|
||||
|
|
@ -90,26 +96,64 @@ test('realm vs jsdom: 60 days of the scenic big network with actions in between,
|
|||
assert.deepEqual(P.errors, []);
|
||||
});
|
||||
|
||||
test('restore: 20 days straight, or 10, a checkpoint, a fresh realm and 10 more, is the same network', async () => {
|
||||
const code = 'R3ST0R', a = 'rsta0001', b = 'rstb0002', c = 'rstc0003';
|
||||
await loadBig(code, a); await loadBig(code, b);
|
||||
for (let d = 1; d <= 10; d++) await day(code, d, [a, b]);
|
||||
// b's checkpoint: the envelope a server writes (4.6), into a fresh realm; the network's clock goes on.
|
||||
const sv = checked(await E.save(code, b));
|
||||
const env = { s: sv.txt, uidN: sv.uidN, rt: sv.rt, last: 10, day: sv.day };
|
||||
await E.unload(code, b);
|
||||
checked((await E.load(code, [{ nid: c, env: JSON.parse(JSON.stringify(env)), lang: 'en', prefs: {} }]))[0]);
|
||||
await E.eval(code, c, `__srvClock(${T0 + 10 * STEP})`);
|
||||
for (let d = 11; d <= 20; d++) await day(code, d, [a, c]);
|
||||
// The same play: every value the same. (Only the order of a customer's keys can differ: the save drops the defaults
|
||||
// that sanitize() puts back, at the end. A page takes a snapshot of a reloaded realm anyway, a new epoch.)
|
||||
// Compared as text, keys sorted: a deep-equal of two big networks that differ builds a diff of gigabytes.
|
||||
const sa = canon(await realmS(code, a)), sc = canon(await realmS(code, c));
|
||||
assert.ok(sc === sa, `S after 20 days:\n${sc === sa ? '' : differs(sa, sc)}`);
|
||||
const ra = canon(await realmR(code, a)), rc = canon(await realmR(code, c));
|
||||
assert.ok(rc === ra, `R after 20 days:\n${rc === ra ? '' : differs(ra, rc)}`);
|
||||
assert.ok(JSON.parse(sa).biz.some((x) => x.route === 'scenic'), 'with scenic circuits');
|
||||
await E.unload(code);
|
||||
/* A restart or an idle unload is invisible (4.6, 4.7, 11): a network checkpointed after any day and loaded into a fresh
|
||||
realm is at once, and stays, the network that kept going, S and R byte for byte. Two of them: the scenic big network,
|
||||
and the same busy (busy(), and an attack on). They need everything rt carries besides the save: R's routing point (R
|
||||
after a day is the R of the day's last routing, made before the day's end changed S, so the saved S routes to another
|
||||
R, which the next day reads; a checkpoint after day 10 passed by luck: a failure or event routed at that day's end), the
|
||||
sales desk's tried offers, and the live order of keys a reload reorders (the intern's weekly report lists its tally
|
||||
in it). Checkpoints after days 1, 2, 5 and 10 of each (2 and 10 right after an action), each played 10 days on;
|
||||
SIM_LONG: after every day up to 30. Compared as text: a deep-equal of two big networks that differ builds a diff of
|
||||
gigabytes. */
|
||||
test('restore: a checkpoint after any day, in a fresh realm, plays on byte for byte as the network that kept going', { timeout: LONG ? 1800e3 : 300e3 }, async (t) => {
|
||||
const X = createEngine({ workers: 1, test: TEST, timeouts: ROOMY, heapMb: LONG ? 4096 : 1536 });
|
||||
t.after(() => X.close());
|
||||
const { day, realmS, realmR } = on(X);
|
||||
const code = 'R3ST0R', H = 10, KS = LONG ? Array.from({ length: 30 }, (_, i) => i + 1) : [1, 2, 5, 10], N = Math.max(...KS) + H;
|
||||
const bases = ['busy0000', 'scen0000'];
|
||||
for (const nid of bases) checked((await X.load(code, [{ nid, env: { s: bigScenic(), uidN: 1000, rt: null }, lang: 'en', prefs: {} }]))[0]);
|
||||
assert.equal((await X.eval(code, bases[0], busy('eng') + 'const a=ddosAttack(true);a.until=S.day+6;return slP();')).value, 'eng');
|
||||
const same = async (what, a, b) => {
|
||||
const sa = await realmS(code, a), sb = await realmS(code, b);
|
||||
assert.ok(sa === sb, `${what}: S\n${sa === sb ? '' : differs(sa, sb)}`);
|
||||
const ra = await realmR(code, a), rb = await realmR(code, b);
|
||||
assert.ok(ra === rb, `${what}: R\n${ra === rb ? '' : differs(ra, rb)}`);
|
||||
};
|
||||
let copies = [];
|
||||
const carried = { point: 0, desk: 0, order: 0 };
|
||||
for (let d = 1; d <= N; d++) {
|
||||
await day(code, d, [...bases, ...copies.map((c) => c.nid)]);
|
||||
for (const c of copies.filter((x) => x.k + H === d)) { await same(`${c.nid}, ${H} days after its checkpoint`, c.base, c.nid); await X.unload(code, c.nid); }
|
||||
copies = copies.filter((x) => x.k + H > d);
|
||||
if (!KS.includes(d)) continue;
|
||||
// Every other one right after an action and a what-if (a server also checkpoints a network after an action), sent
|
||||
// to the network and to its copies alike.
|
||||
if (d % 2 === 0) for (const b of bases) {
|
||||
const city = (await X.eval(code, b, "return Object.keys(CITY).filter(c=>!S.pops[c]&&homeOk(c)).sort((x,y)=>kmBetween('fra',x)-kmBetween('fra',y))[0]")).value;
|
||||
const lid = (await X.eval(code, b, 'return S.links[0].id')).value;
|
||||
for (const nid of [b, ...copies.filter((c) => c.base === b).map((c) => c.nid)]) {
|
||||
assert.equal(checked(await X.act(code, nid, msg('buildPop', { city }))).outcome, 'applied', `${nid}: a POP in ${city}`);
|
||||
assert.ok(checked(await X.ask(code, nid, msg('linkCheck', { id: lid }))).modal, `${nid}: the what-if`);
|
||||
}
|
||||
}
|
||||
for (const b of bases) {
|
||||
// The envelope a server writes (4.6), through text, into a fresh realm; the network's clock goes on.
|
||||
const sv = checked(await X.save(code, b)), nid = b.slice(0, 4) + String(d).padStart(4, '0');
|
||||
const env = JSON.parse(JSON.stringify({ s: sv.txt, uidN: sv.uidN, rt: sv.rt, last: d, day: sv.day }));
|
||||
checked((await X.load(code, [{ nid, env, lang: 'en', prefs: {} }]))[0]);
|
||||
await X.eval(code, nid, `__srvClock(${(await X.eval(code, b, 'return __srvClock()')).value});return 1;`);
|
||||
await same(`${nid}, loaded from its checkpoint after day ${d}`, b, nid);
|
||||
copies.push({ nid, base: b, k: d });
|
||||
const rt = env.rt;
|
||||
if (rt.pt && rt.pt.at.biz) carried.point++;
|
||||
if (rt.sl.length) carried.desk++;
|
||||
if (rt.ord && rt.ord.ops) carried.order++;
|
||||
}
|
||||
}
|
||||
// What it played: scenic circuits, and checkpoints that needed each part of rt.
|
||||
assert.ok(JSON.parse(await realmS(code, bases[1])).biz.some((x) => x.route === 'scenic'), 'with scenic circuits');
|
||||
assert.ok(carried.point && carried.desk && carried.order, `rt carried each part: ${JSON.stringify(carried)}`);
|
||||
await X.unload(code);
|
||||
});
|
||||
|
||||
test('retry: a day that throws, or times out, once after the DDoS step with an attack on, leaves the same network', async (t) => {
|
||||
|
|
@ -120,6 +164,8 @@ test('retry: a day that throws, or times out, once after the DDoS step with an a
|
|||
const code = 'R3TRYX', clean = 'rtya0001', thrown = 'rtyb0002', slow = 'rtyc0003', nets = [clean, thrown, slow];
|
||||
const L = await E.load(code, nets.map((nid) => ({ nid, env: { s: bigScenic(), uidN: 1000, rt: null }, lang: 'en', prefs: {} })));
|
||||
L.forEach(checked);
|
||||
// Busy: what keeps state between days goes back to the mark too (the sales desk's tried offers live outside S).
|
||||
for (const nid of nets) await E.eval(code, nid, busy('eng') + 'return 1;');
|
||||
for (let d = 1; d <= 3; d++) await day(code, d, nets);
|
||||
// An attack that is still on during day 4 (the same one in each).
|
||||
for (const nid of nets) assert.ok((await E.eval(code, nid, 'const a=ddosAttack(true);a.until=S.day+3;return ddosAll().length')).value > 0, 'an attack is on');
|
||||
|
|
@ -243,14 +289,14 @@ test('market and moved customers: realm and page the same, the strict gate passe
|
|||
assert.ok(U.fx.some(([k, p]) => k === 'toast' && p.k === 'good'), 'the sales were told');
|
||||
const sold = ev('S.stats.cmSold');
|
||||
assert.ok(sold > 20 && ev("S.biz.find(b=>b.id==='k').pin.join()") === via, `the market sold (${sold}) and the circuit stayed pinned`);
|
||||
// A checkpoint into a fresh realm (route memory, no route cache) plays on as the realm that kept going.
|
||||
// A checkpoint into a fresh realm (its rt, no route cache) plays on byte for byte as the realm that kept going.
|
||||
const sv = checked(await S2.save(code, nid));
|
||||
checked((await S2.load(code, [{ nid: nid2, env: { s: sv.txt, uidN: sv.uidN, rt: sv.rt }, lang: 'en', prefs: {} }]))[0]);
|
||||
await realm(nid2, `__srvClock(${await realm(nid, 'return __srvClock()')});return 1;`);
|
||||
await days(15, [nid, nid2], false);
|
||||
const s1 = canon(await realm(nid, 'return JSON.stringify(S)')), s2 = canon(await realm(nid2, 'return JSON.stringify(S)'));
|
||||
const s1 = await realm(nid, 'return JSON.stringify(S)'), s2 = await realm(nid2, 'return JSON.stringify(S)');
|
||||
assert.ok(s1 === s2, `S after the checkpoint:\n${s1 === s2 ? '' : differs(s1, s2)}`);
|
||||
assert.ok(canon(await realm(nid, `return ${RSIG}`)) === canon(await realm(nid2, `return ${RSIG}`)), 'R after the checkpoint');
|
||||
assert.ok(await realm(nid, `return ${RSIG}`) === await realm(nid2, `return ${RSIG}`), 'R after the checkpoint');
|
||||
assert.ok(JSON.parse(s1).stats.cmSold > sold, 'it went on selling');
|
||||
assert.deepEqual(await realm(nid, 'return [__SRV_T.n,__SRV_T.at]'), [0, []], 'no timer');
|
||||
assert.deepEqual(P.errors, []);
|
||||
|
|
@ -260,8 +306,7 @@ test(`timer audit and gate: ${LONG ? 365 : 60} days with everything on, then eve
|
|||
const S2 = createEngine({ workers: 1, test: TEST, strict: true, timeouts: ROOMY });
|
||||
t.after(() => S2.close());
|
||||
const code = 'AVD1T5', nid = 'audt0001';
|
||||
const SETUP = `S.lvl=Math.max(S.lvl,6);S.rules=makeRules('bgp');ACT.opsHire({id:'kevin'});for(const k in S.ops.on)S.ops.on[k]=true;S.ops.floor=0;closeModal();
|
||||
ACT.salesHire({id:'shark'});ACT.salesAuto();closeModal();S.nextFail=S.day+2;S.nextEvent=S.day+3;compute();`;
|
||||
const SETUP = busy('shark');
|
||||
checked((await S2.load(code, [{ nid, env: { s: bigScenic(), uidN: 1000, rt: null }, lang: 'en', prefs: {} }]))[0]);
|
||||
await S2.eval(code, nid, SETUP + 'return 1;');
|
||||
const timers = async () => (await S2.eval(code, nid, 'return [__SRV_T.n,__SRV_T.at]')).value;
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue