Server-run engine: a restored network plays on exactly as before
A network loaded from its checkpoint drifted from the one that never stopped:
after a checkpoint on day 1, 2 or 5 its R differed at once (bizDem at Ashburn
6426 against 26, then cityDem, citySrv, tNet), and from the next day S too
(cash, eyeball growth). Three causes, each shown by a probe:
- R after a day is the R of the day's last routing, made before the day's end
changed S (customers who left or signed, moods, bills, transit history) and
routed from the memory of the R before it. srvLoad routed the saved S again
from R's own memory: another R, which the next day reads (sfOf for eyeball
growth, the peering share). Every compute() in the realm now keeps, for the
R it makes, S as it routed it (one JSON per top-level key, 0.8 ms big) and
the memory it started from. rt carries that routing point (pt: the memory,
the keys that changed since, the keys S did not have then), and srvLoad
routes that S again from it: the same R, byte for byte. The old restore test
checkpointed after day 10, where a failure or event routed again at the end
of the day and nothing changed after it: it passed by luck.
- The sales desk's tried offers (UI.slTried, an offer that did not fit waits a
week) live outside S. A fresh realm forgot them and the desk tried other
offers. rt carries them (sl); a day mark keeps them and the in-place restore
after a throw puts them back (before, a parsed S dropped them).
- A save and a reload give some keys back in another key order (a customer's
defaults go last, the intern's week tally in its own order), and the week's
report lists the tally in that order. At a checkpoint rt carries the live
text of each such key (ord, srvGate's round trip, about 8 ms), taken on load
only where it holds what sanitize() made of the save. A fresh realm after a
timeout takes the mark's own text the same way (live).
The plain game drifts across a save and a reload too (jsdom: from the first
day after it; the page routes a reloaded S with an empty R and keeps no desk
memory): a game property, so the engine carries what is missing.
The envelope keeps its shape {s, uidN, rt, last, day}; rt keeps R's own route
memory, so an older checkpoint loads as before (close, not exact). Big network:
the checkpoint grows from about 42-49 KB gzip to 54-79 KB (rt 7-29 KB; the
whole R would be 375 KB), srvSave from 2-2.5 ms to 7-8 ms, a load by one
routing. A day with its mark costs the same (big 39.8 -> 37.8 ms, busy
90.0 -> 88.8 ms, within noise: the day mark joins per-key JSON now, which pays
for the routing point).
Tests: the restore test checkpoints after days 1, 2, 5 and 10 of the scenic big
network and of the same network busy (intern on every order, the desk signing
on its own, failures, events, an attack), those after days 2 and 10 right after
an action and a what-if (sent to the copies too), each compared byte for byte
at once and 10 days later (SIM_LONG: after every day to 30, 60 checkpoints). It
fails on the old engine, and without any one of the three parts. The retry test
plays busy networks (fails without the desk memory in srvRestore). The market
checkpoint compares bytes too; canon() is gone.
This commit is contained in:
parent
065c922287
commit
c5c7ea9508
5 changed files with 169 additions and 72 deletions
|
|
@ -7,10 +7,11 @@
|
|||
// unloads of a session wait for its day; actions for a network that is still loading wait for it.
|
||||
//
|
||||
// Failures. Every realm call has a time limit (a day 2 s, an action 1 s, a what-if 3 s, a pull or a save 1 s). Before
|
||||
// a day or an action the realm is marked (its S as text, uidN, the dice, and for a day yesterday's route memory), and the
|
||||
// worker keeps the mark. A call that throws has run its finally blocks: the realm goes back to the mark in place. A call
|
||||
// that times out has not (vm skips them), so that realm is never used again: a fresh one is built from the mark, with a
|
||||
// new epoch. A failed day runs once more; failing again, the network is broken at its state before the day.
|
||||
// a day or an action the realm is marked (its S as text, uidN, the dice, and for a day its rt: route memory and routing
|
||||
// point), and the worker keeps the mark. A call that throws has run its finally blocks: the realm goes back to the mark
|
||||
// in place. A call that times out has not (vm skips them), so that realm is never used again: a fresh one is built from
|
||||
// the mark, with a new epoch. A failed day runs once more; failing again, the network is broken at its state before
|
||||
// the day.
|
||||
//
|
||||
// The worker → host replies are the shapes of tests/lib/mp-sim-shape.js checkWorker(), sent as {id, ok, reply} (or
|
||||
// {id, ok:false, error}) for the engine to match its promises; logs go up as {log:[level, text]}.
|
||||
|
|
@ -44,27 +45,28 @@ function createCore(opts = {}) {
|
|||
}
|
||||
// A realm call: [header, ...parts]; a throw inside the realm comes back as {srvErr}, a timeout throws.
|
||||
const call = (e, name, input, ms) => e.realm.call(name, input, ms);
|
||||
/* A fresh realm in place of one that timed out, from {s, uidN, rt}: same network, new epoch, versions from 0. R comes
|
||||
from the old realm, which still answers after a timeout: the marked R (mark) or its current one, packed whole. Only
|
||||
when it cannot is R routed again from the route memory, which gives a close R, not always the same. */
|
||||
/* A fresh realm in place of one that timed out, from {s, uidN, rt, live}: same network, new epoch, versions from 0. R
|
||||
comes from the old realm, which still answers after a timeout: the marked R (mark) or its current one, packed whole.
|
||||
When it cannot, R is routed again from rt's routing point. live: s is a mark, the realm's own S as text. */
|
||||
function rebuild(e, from, mark = true) {
|
||||
let rpack = null;
|
||||
try { const s = e.realm && e.realm.raw('srvRPack', { mark }, T.load); if (typeof s === 'string' && s.startsWith('{"o"')) rpack = s; } catch (x) { /* gone */ }
|
||||
const r = build(e.nid, e.lang), epoch = epochOf();
|
||||
const [h] = r.call('srvLoad', { save: from.s, uidN: from.uidN, rt: from.rt, rpack, prefs: e.prefs, epoch, strict: !!opts.strict, clock: from.clock }, T.load);
|
||||
const [h] = r.call('srvLoad', { save: from.s, uidN: from.uidN, rt: from.rt, rpack, live: !!from.live, prefs: e.prefs, epoch, strict: !!opts.strict, clock: from.clock }, T.load);
|
||||
if (h.srvErr) throw new Error('rebuilding the realm failed: ' + h.srvErr);
|
||||
e.realm = r; e.epoch = epoch; e.v = 0; e.day = h.day; e.lost = h.lost;
|
||||
log('warn', `[${e.nid}] realm rebuilt after a timeout (epoch ${epoch}${rpack ? '' : ', R routed again'})`);
|
||||
}
|
||||
// The route memory of the old realm's mark, if it still answers; else the last day's.
|
||||
// The rt of the old realm's mark, if it still answers; else the last day's, whose routing point belongs to that day's
|
||||
// S: with another mark only its route memory counts (a close R).
|
||||
function markRt(e) {
|
||||
try { const [h] = call(e, 'srvRt', { mark: true }, 200); if (h && !h.srvErr && h.load) return h; } catch (x) { /* gone */ }
|
||||
return e.dayMark ? e.dayMark.rt : null;
|
||||
return e.dayMark ? (e.dayMark === e.mark ? e.dayMark.rt : { ...e.dayMark.rt, pt: undefined }) : null;
|
||||
}
|
||||
function mark(e, input) {
|
||||
const [h, s] = call(e, 'srvMark', input, T.mark);
|
||||
if (h.srvErr) throw new Error('mark failed: ' + h.srvErr);
|
||||
const m = { s, uidN: h.uidN, rt: h.rt || null, day: h.day, clock: h.clock };
|
||||
const m = { s, uidN: h.uidN, rt: h.rt || null, day: h.day, clock: h.clock, live: true };
|
||||
if (input.day) e.dayMark = m;
|
||||
e.mark = m;
|
||||
return m;
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue