463877773c
Reverse-engineering + open reimplementation workspace for Eushully's AGE/SYS4 engine (first target: Himegari). The repo root is age-reimpl/; the original game install and the extracted ALF data are siblings outside the repo and are never tracked. build/ (derived corpora) is gitignored and regenerated by the tools. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
183 lines
13 KiB
Markdown
183 lines
13 KiB
Markdown
# Phase A — Vertical Slice Plan (the first build step)
|
||
|
||
Concrete execution plan for Phase A of `remake-architecture-and-roadmap.md`. Decided over the
|
||
alternative (fully decoding `SCJUMP.BIN`) after recon showed SCJUMP is not the gating unknown.
|
||
|
||
## Why the slice, and why headless-first
|
||
|
||
**SCJUMP recon (2026-07-06):** `SCJUMP.BIN` is a 29,796-instruction **progression state machine**,
|
||
not the `call-script` registry. Top level switches on `global 0x3234` (mode 1–9 → big blocks); each
|
||
block is nested `eq`/`ne`/`and`/`jcc` on flags, ending in `mov`s to output globals. Almost no
|
||
`call-script`. So it decides *what scene/branch comes next* via state, and does **not** resolve
|
||
`call-script id → code`. Consequence: the id→code registry stays engine-level (deferred), **but the
|
||
slice can stub `call-script`** — it is not gating for running one scene's dialogue.
|
||
|
||
**Correctness bootstrap (roadmap §5) drives the ordering:** the VM must be *validated-correct* before
|
||
it is trustworthy. Our strongest oracle is `build/text/dialogue.jsonl` (the `show-text` lines per
|
||
script). So the very first slice is **headless and text-only, validated by that oracle** — no Godot,
|
||
no AGF, no audio, no dispatch registry. Only once the VM reproduces dialogue do we add rendering.
|
||
|
||
Phase A therefore splits:
|
||
- **A0 — headless VM, dialogue-validated (Python prototype).** ← immediate, executable now.
|
||
- **A1 — port the validated model to C#** (the runtime's VM core).
|
||
- **A2 — Godot ADV backend** (render one scene with visuals + voice).
|
||
|
||
---
|
||
|
||
## A0 — Headless VM validated by the dialogue oracle
|
||
|
||
**Goal:** a Python interpreter that executes one ADV scene's bytecode and emits its `show-text`
|
||
sequence; that sequence is a coherent, in-order subsequence of the script's static `dialogue.jsonl`
|
||
lines. This proves the execution model — control flow, operand/pointer semantics, string handling,
|
||
and the no-op-marker assumptions — *before* any C#/Godot investment. Reuses `tools/sys4load.py` for
|
||
all parsing/decoding (no new parser).
|
||
|
||
### Execution model to implement
|
||
- **Memory:** one flat **global bank** = `dict[int,int]` (globals are raw offsets into one space;
|
||
`global-int A` ⇒ `G[A]`, default 0). Per-call **local frame** with typed banks sized by header
|
||
F0–F5 (`local_int[F0]`, `local_float[F1]`, `local_string[F2]`, …).
|
||
- **PC / control flow:** build `offset→instruction-index` map from `sys4load` instructions (each has
|
||
`.offset` = dword index; jump targets are dword indices). `jmp t` → pc = map[t]. `jcc(cond, A, B)`
|
||
→ cond truthy ? goto A : goto B, where `0xffffffff` = fall through (confirmed model from RECOVER).
|
||
- **Operand resolution by type:** imm→value; global-int→`G[value]`; local-int→`frame.int[value]`;
|
||
string(2)→decoded string at dword offset; float/global-string/etc. analogous.
|
||
- **⚠ Pointer/lvalue semantics — the key modeling task.** RECOVER proves `-ptr` operands are
|
||
*lvalues*: `lookup-array(dst_ptr, base, idx)` yields a *reference* to `G[base+idx]`; `mov` through a
|
||
ptr writes to the referenced cell; reading a ptr rvalue dereferences it. Model a ptr slot as holding
|
||
an address into the global bank; nail this so the RECOVER array-copy produces correct results (unit
|
||
test it directly).
|
||
- **Opcode handlers (~52 named ops):**
|
||
- arithmetic/bit `add sub mul div mod and or sar shl` → `p1 = p2 ⊙ p3`.
|
||
- compares `eq ne lt lte gr gre` → 0/1.
|
||
- `mov` (incl. through ptr), `lookup-array` (`p1=mem[base+idx]`), `lookup-array-2d`
|
||
(`p1=mem[base + i*stride + col]`), `copy-to-global`, `set-array-to`, `bit-set/reset`, `check-bit`.
|
||
- control `jmp call jcc ret exit exit-script`.
|
||
- string `set-string concat strlen toString`.
|
||
- **ADV capture:** `show-text` → append (arg text) to the emitted list; `end-text-line`,
|
||
`wait-for-input`, `set-font`, `comment` → capture/skip (no visible state).
|
||
- **Markers → no-op (this TESTS the classification):** `0x1f4 0x1f5 0x1d5 0x1bc 0x1bf` skip;
|
||
tentative `0x21b 0x1d2 0x258` skip — if dialogue stays correct, the no-op assumption is validated.
|
||
- **`call-script` → STUB:** log `(id)`, return immediately. (Its dialogue belongs to other scripts;
|
||
stubbing keeps the emitted set = this script's own lines.)
|
||
- **Effectful (draw/texture/audio/ui/input) → STUB:** log and ignore.
|
||
- **Unknown/other opcodes → log + no-op**, so a rare op doesn't halt the run (record coverage).
|
||
|
||
### Oracle & scene choice
|
||
- **Oracle:** with calls stubbed and default state, every emitted `show-text` line must be a real
|
||
decoded string from the script's pool, and the sequence must be an **in-order subsequence** of that
|
||
script's `dialogue.jsonl` lines (≈ equality for a linear scene). Catches: garbage strings (bad
|
||
operand/ptr handling), impossible ordering (bad control flow), missing/extra lines.
|
||
- **Scene pick:** choose a **short, mostly-linear ADV scene** — high `show-text` count, low `jcc`
|
||
density, few `call-script`. Selection step: rank `SC####`/`SP####` by
|
||
`(show-text count) / (jcc + call-script count)`, small size. Known-good fallback: `SC0030.BIN`
|
||
(dialogue verified). Also run a **RECOVER unit test** to validate pointer/array semantics independent
|
||
of dialogue.
|
||
|
||
### Steps
|
||
1. `tools/vm0.py`: load a script via `sys4load`, build offset→index map, frame + global bank.
|
||
2. Implement operand resolution + the arithmetic/compare/mov/lookup/control handlers; unit-test on
|
||
`RECOVER.BIN` (array copy + both loops must produce correct global writes).
|
||
3. Add ADV capture + markers-as-noop + call/effectful stubs; add opcode-coverage logging.
|
||
4. Run on the chosen linear scene; diff emitted `show-text` vs `dialogue.jsonl` (subsequence check);
|
||
eyeball the first ~15 lines for coherence.
|
||
5. Iterate until several scenes pass; record which ops/markers were exercised and any surprises
|
||
(esp. whether the tentative-no-op markers hold).
|
||
|
||
### Success criteria (A0 done)
|
||
- RECOVER unit test passes (pointer/array model correct).
|
||
- ≥3 ADV scenes: emitted `show-text` is a coherent in-order subsequence of their `dialogue.jsonl`,
|
||
no garbage strings.
|
||
- Coverage report of which opcodes actually executed (drives A1/A2 priorities).
|
||
- The no-op-marker assumption is confirmed or corrected with evidence.
|
||
|
||
---
|
||
|
||
### A0 result (2026-07-06) — execution model VALIDATED
|
||
|
||
`tools/vm0.py` built (reuses `sys4load`; ~250 lines). Results:
|
||
- **RECOVER unit test PASSES** — all 7 checks (block-1 3-field copy, block-2 restore + flag, both
|
||
skip-guards). The pointer/lvalue model, 2D stride indexing, both loops, and two-way `jcc` all
|
||
execute correctly. **The core execution model is proven.**
|
||
- **Full SC/SP oracle sweep (`vm0.py --sweep`): 282 / 294 scenes DIALOGUE-VALID = 95.9%.** Every
|
||
emitted `show-text` line is checked (by string offset) as an in-order subsequence of the script's
|
||
static `dialogue.jsonl` lines. **Zero STRAY and zero ORDER violations across all 294 scenes** — the
|
||
model never emits a garbage string and never emits dialogue out of order. 279 CLEAN (valid + natural
|
||
`exit`); 3 OK/LOOP (valid subsequence, halted by the loop-guard); 12 EMPTY; 3 skipped (no static
|
||
show-text). SC0000 = 326 static / clean; SP0062 = 220/220 CLEAN.
|
||
|
||
**A0-remainder work done (2026-07-06, session 2):**
|
||
- **Loop-guard added** (`EMIT_CAP=2`): halt a run once any single line is re-emitted a 3rd time —
|
||
a semantic guard tied to the oracle (vs. a blind step limit), and it *classifies* the scene LOOPED
|
||
instead of spewing garbage. The 3 zero-state spinners (SC0010/SC0600/SC0200) now terminate cleanly
|
||
in <12k steps and their emitted lines are all valid.
|
||
- **SP0062 "stray" was a measurement artifact**, not a bug — the precise offset-based oracle shows it
|
||
CLEAN (220/220, natural exit). Offset-match ⟹ text-match (VM decodes each string at the same offset
|
||
the extractor did), so CLEAN is trustworthy.
|
||
- **`0x71` (label-def) folded into the no-op marker set** — structural, no runtime effect.
|
||
- **Sweep + single-scene diff harness** added to `vm0.py`: `--sweep [N]` (coverage table over all
|
||
SC/SP), `--scene NAME` (detailed diff for one script), plus `load_oracle`/`subsequence_status`.
|
||
|
||
**op 0x90 investigated in depth — it is input chrome, NOT a correctness hole** (full evidence:
|
||
`vm-map/himegari-opcode-notes.md` §F). Kelebek left it "ukn"; corpus analysis resolves it:
|
||
`0x90 x y w h tgt_a tgt_b tgt_c` (argc 7) is a **cursor/input hotspot hit-test** that branches per
|
||
interaction outcome and **falls through to pc+1 when nothing matches** (design-confirmed: enc.len 15
|
||
lands the next instr on the fall-through statement). It occurs ONLY in a shared ADV-chrome subroutine
|
||
that is byte-identical in all 301 ADV scripts — **exactly 8 sites each** (5 immediate-rect buttons at
|
||
`(684..772, 572)` toggling `G[0x6c9..0x6cd]` + 3 local-operand keyed forms), **zero scene-specific
|
||
use**. Headless (no cursor/input) ⇒ fall through ⇒ **vm0's stub is already correct**, proven safe by
|
||
all 279 CLEAN scenes (which contain these same 8 sites). `op 0x97` (argc 5, no targets) is its
|
||
companion register-hotspot call. **So 0x90 stays as fall-through in A1 with confidence; it is modelled
|
||
as a live hotspot test only in A2** (Godot input backend), confirming target→state mapping via Frida.
|
||
|
||
**The 12 EMPTY scenes — state-gated interactive screens, not a model failure.** Traced SC0830: it
|
||
exits early because `G[0xaba5c]==1` gates the content; past that gate the dialogue sits behind the ADV
|
||
input-wait loop (the hotspot-polling chrome above), so with no seeded state and no input the scene
|
||
exits or spins before reaching text. Unlocking them = seed per-scene state + supply input →
|
||
**Phase A2/B**, not an A0 model fix.
|
||
|
||
**⚠ Honest scope of the 95.9%:** the subsequence oracle proves **no-garbage / in-order**, not a
|
||
*complete* path — inherent to a subsequence oracle run headlessly (interactive/state-gated branches
|
||
take the no-input path by design). That anti-garbage guarantee is exactly what A0 set out to prove.
|
||
|
||
**Confirmed by this run:** the classified no-op markers (`0x1f4/0x1f5/0x1d5/0x1bc/0x1bf` + tentative
|
||
`0x21b/0x1d2/0x258`, now + `0x71`) are safe as no-ops for ADV flow; `call-script` is stubbable;
|
||
effectful ops (`draw-texture`/`create-texture`/`play-voice`/`0x1f7`/`0x202`/`0x203`/…) stub cleanly.
|
||
|
||
**✅ A0 COMPLETE.** Success criteria met: RECOVER unit test green (pointer/array/control-flow model
|
||
proven); 282 ADV scenes emit clean in-order subsequences with zero garbage; coverage number recorded;
|
||
no-op-marker assumption confirmed at scale; `op 0x90` (the last big control-flow unknown) resolved as
|
||
input chrome whose fall-through stub is correct headless. Next = **A1** — port the model to the C# VM
|
||
core, differential-test against `vm0.py`. 0x90/0x97 stay stubbed (correct headless); the interactive
|
||
input path + per-scene state seeding land in **A2** (Godot backend) alongside the real hotspot model.
|
||
|
||
## A1 — Port the validated model to C#
|
||
Reimplement the A0 execution model as the runtime VM core in C# (the language decision from the
|
||
roadmap; GDScript is too slow for the loop). A0 is the reference: differential-test C# against the
|
||
Python prototype's traces on the same scenes. Port the container parser too (or load via a shared
|
||
spec). Deliverable: headless C# VM reproducing A0's results.
|
||
|
||
## A2 — Godot ADV backend (one scene, with visuals)
|
||
Wire the C# VM's effectful ops to Godot: `show-text`/message window (+ furigana via `display-furigana`),
|
||
`set-font`, `wait-for-input`, choices, `play-voice`/`play-bgm`, and `create-texture`/`set-texture`/
|
||
`draw-texture`/`draw-string` for the background + sprites. Convert the scene's AGF art with the
|
||
on-disk `AGF2BMP2AGF.exe`. Resolve just-enough `call-script`/state so the scene's setup runs (or
|
||
hand-set the preconditions). Deliverable: **the chosen scene playable in Godot** — bg + dialogue +
|
||
a choice + voice — matching A0's text.
|
||
|
||
---
|
||
|
||
## Risks / open questions for A0
|
||
- **Pointer/lvalue semantics** — the main modeling risk; RECOVER is the litmus test.
|
||
- **Initial global state** — a scene may assume preconditions from earlier flow (`SCJUMP`/prior
|
||
scenes). Mitigation: default-zero globals + set the few a scene reads early; the subsequence oracle
|
||
tolerates a shortened path.
|
||
- **Runtime vs static dialogue order** — static `dialogue.jsonl` is file-order (all lines); runtime is
|
||
execution-order (branch taken). Hence *subsequence*, not equality; pick linear scenes to tighten it.
|
||
- **Hidden effect in a "stub"** — a stubbed effectful op that actually gates control flow could skew
|
||
output. Watch for divergence; promote a stub to a real handler if a scene needs it.
|
||
|
||
## Immediate next action
|
||
Build `tools/vm0.py` and get the **RECOVER unit test** green (pointer/array/control-flow correctness),
|
||
then run the first linear ADV scene against the dialogue oracle. That single result tells us whether
|
||
the whole VM approach executes correctly — the load-bearing question behind option 3.
|