diff --git a/docs/opcode-reference.md b/docs/opcode-reference.md index 3128162..6049844 100644 --- a/docs/opcode-reference.md +++ b/docs/opcode-reference.md @@ -213,6 +213,11 @@ op 0x90 (u0041BEB0, argc 7): `0x90 x y w h tgt_a tgt_b tgt_c`. Kelebek left it " - **summary:** 2 imm; runs in a chain right after script-entry 0x259, enumerating ids — prologue declaration/registration? - **grounding:** source=harness, confidence=low, noop_headless=True +### 0x259 `script-entry` (u00416410, argc 0) +- **summary:** zero-arg; the first instruction of a script (offset 0), opens the decl chain that 0x258 continues — script/prologue entry marker, structural +- **grounding:** source=harness, confidence=low, noop_headless=True +- **evidence:** SC0000 offset 0x0 = op 0x259 (argc 0); 0x258's summary names it 'script-entry 0x259'; VM treats it as no-op (default stub) across all 279 CLEAN A0 scenes + ## structural ### 0x71 `label-def` (u0041A7B0, argc 1) @@ -1045,10 +1050,6 @@ op 0x90 (u0041BEB0, argc 7): `0x90 x y w h tgt_a tgt_b tgt_c`. Kelebek left it " - **summary:** — - **grounding:** source=kelebek, confidence=low -### 0x259 `u00416410` (u00416410, argc 0) -- **summary:** — -- **grounding:** source=kelebek, confidence=low - ### 0x2bd `u00423100` (u00423100, argc 1) - **summary:** — - **grounding:** source=kelebek, confidence=low diff --git a/docs/phase-a-slice-plan.md b/docs/phase-a-slice-plan.md index ca43200..b4ffefa 100644 --- a/docs/phase-a-slice-plan.md +++ b/docs/phase-a-slice-plan.md @@ -308,6 +308,36 @@ Next visual chunk = **alpha/blend + effect fading** (`0x202/0x203` already store compositing. NOTE the two-boot gap: our Phase-B `--boot` runs *data* `*INIT` scripts; this added the *system* boot — a "full boot" should run both. +### A2b — Scene completeness gauge (opcode coverage tracker, 2026-07-07) + +To stop guessing how "done" a rendered scene is, `tools/scene_opcode_coverage.py` histograms a scene's +static opcodes and classifies each against the C# VM: **impl** (VM has a handler arm — real or a deliberate +no-op like `set-font`), **safe-noop** (no arm, but `opcodes.toml` marks it `noop_headless` — a statement / +block marker, correct to skip), or **GAP** (no arm and effectful → the VM silently `pc+1`s past it). The +implemented set is parsed from `VirtualMachine.cs`'s `case` arms (single source of truth, no drift); output is +`build/scene-opcode-coverage/.md`. This makes a half-rendered scene legible: *"N ops still stubbed"*, +not *"something's wrong and we thought everything ran."* + +**SC0000 baseline:** 129 distinct opcodes / 16257 instrs. Instruction-weighted the VM already covers **~94.8%** +(impl 12368 + safe-noop 3053); the holes are **68 GAP opcodes / 836 instrs (5.1%)**. The GAP list clusters into +concrete backlog buckets (drives the rendering roadmap below): +- **ADV on-screen text** — `draw-string`(0x204)×205 + `0x7a` text-param×205 (1:1 paired). Dialogue text is + currently surfaced via `IHost.ShowText` → Godot `Label`; the engine's *native* glyph/window draw path is + unmodeled (cosmetic for now, but it owns text layout/speed). +- **Unmodeled gfx-range cluster** — `0x21c–0x243` + `0x2bd/0x2bf` (e.g. 0x220×66, 0x22f×34, 0x228×33, 0x21e×25): + siblings of the `0x212–0x21a` command-buffer family we implemented, **not yet reversed** → the biggest single + rendering unknown (likely sprite/effect/blend geometry). RE these next before more compositor work. +- **Timing** — `sleep`(0xc8)×20: animation pacing; fades/effects can't *animate* (only snap) until this exists. +- **Audio/SFX** — `play-sound-effect`(0xb4)×24 + `0xb5/0xb6/0xc2/0xd9` (channel/volume/stop control) — stubbed. +- **Scene coroutine** — `0x7b`×6 / `0x7c`×2 / `0x140`×1: the scene-coroutine framework backlog (multi-object + scene setup routes through it; see the "SECOND latent gap" note in the status memory). +- **Misc VM-support ops** — a long tail (`0x75-0x77`, `0x85/0x88/0x8b`, `0x93/0x94`, `0x197-0x1a4`, `0x1c7-0x1cf`, + `0x1fd`, `0x20a/0x20c/0x20e`, …), 1–2 sites each; mixed markers vs effectful — triage per-op as the VM reaches them. + +Re-run per scene (`scene_opcode_coverage.py SC0240 …`) to gauge any target. The tracker also cross-checks +`opcodes.toml` metadata against VM behavior — it already surfaced `0x259` (script-entry marker) missing its +`noop_headless` flag (now reconciled). + --- ## Risks / open questions for A0 diff --git a/docs/tools-reference.md b/docs/tools-reference.md index 49d4c2e..d11f105 100644 --- a/docs/tools-reference.md +++ b/docs/tools-reference.md @@ -62,6 +62,7 @@ All opcode knowledge (ABI, semantics, provenance, `depends_on`) is hand-edited * | Tool | Purpose | Run | Reads → Writes | |---|---|---|---| | `vm0.py` | Headless Python bytecode VM (Phase A0 execution-model prototype; reuses `sys4load`). | `--test` (RECOVER unit test) · `--sweep [N]` (oracle coverage) · `--scene NAME` · `--settex NAME` (set-texture resId trace + exec trace) · `` | corpus → stdout; `build/vm0-trace.json`; `build/settex-.json` | +| `scene_opcode_coverage.py` | Per-scene opcode completeness gauge: histograms a scene's static opcodes and classifies each **impl** / **safe-noop** / **GAP** (effectful op the VM silently stubs). Implemented set parsed from `VirtualMachine.cs` `case` arms; metadata from `opcodes.json`. Surfaces the concrete rendering/feature holes so a half-drawn scene reads as "N ops still stubbed", not "mystery". | `scene_opcode_coverage.py [SCENE …]` (default SC0000) | corpus, `build/opcodes.json`, `engine/…/VirtualMachine.cs`, `build/callscript-names.json` → ⚙ `build/scene-opcode-coverage/.md` + stdout | | `correlate_scope.py` | Align the VM's `set-texture(resId)` trace with the game's Frida load order → tag each load's DATA2 package, flag package transitions, dump the significant ops in each transition span (the **scope selector** hunt). | `correlate_scope.py ` | `build/settex-.json` + `build/frida-load-order-result.json` + index → stdout | ## Engine (C#) — VM core, CLI, Godot frontend diff --git a/tools/age_opcodes_himegari.py b/tools/age_opcodes_himegari.py index c0845dd..30f265e 100644 --- a/tools/age_opcodes_himegari.py +++ b/tools/age_opcodes_himegari.py @@ -17,4 +17,5 @@ INFERRED: dict[int, dict] = { 0x1f5: dict(name='stmt-end', category='marker', noop=True, confidence='high', source='investigation', summary='zero-arg; precedes exit/next-stmt, pairs with 0x1f4'), 0x21b: dict(name='line-id?', category='marker', noop=True, confidence='med', source='harness', summary='1 imm; mov->0x21b->stmt-end; near save/load-messkip — likely line/stmt id, verify not msg-control'), 0x258: dict(name='decl?', category='marker', noop=True, confidence='low', source='harness', summary='2 imm; runs in a chain right after script-entry 0x259, enumerating ids — prologue declaration/registration?'), + 0x259: dict(name='script-entry', category='marker', noop=True, confidence='low', source='harness', summary='zero-arg; the first instruction of a script (offset 0), opens the decl chain that 0x258 continues — script/prologue entry marker, structural'), } diff --git a/tools/scene_opcode_coverage.py b/tools/scene_opcode_coverage.py new file mode 100644 index 0000000..2936dbb --- /dev/null +++ b/tools/scene_opcode_coverage.py @@ -0,0 +1,235 @@ +#!/usr/bin/env py -3.11 -X utf8 +"""Per-scene opcode coverage / completeness tracker. + +Answers "how complete SHOULD this scene be?" by histogramming the opcodes a +scene's bytecode actually contains and classifying each against what the C# VM +implements. A half-rendered scene is far less alarming once we can see it still +has effectful ops sitting un-implemented (a GAP) versus everything being handled. + +Three tiers per opcode: + impl - the VM has an explicit handler arm (real effect OR deliberate + no-op like set-font/comment). Correct treatment. + safe-noop - no VM arm, but opcodes.toml marks it noop_headless (statement / + block markers, 0x258/0x259/0x1f4...). Skipping it is correct. + GAP - no VM arm AND not a safe no-op -> an effectful op the VM silently + skips (default stub, pc+1). THESE are the completeness holes. + +Single sources of truth (no hand-maintained duplicate list): + - implemented labels <- parsed from engine/Age.Engine/Vm/VirtualMachine.cs + (every `case "label":` arm) + - opcode metadata <- build/opcodes.json (label/name/category/noop_headless) + - scene opcodes <- sys4load.load(.BIN).instructions (canonical loader) + +Usage: + py -3.11 -X utf8 tools/scene_opcode_coverage.py # SC0000 (default) + py -3.11 -X utf8 tools/scene_opcode_coverage.py SC0240 SP0062 # specific scenes + py -3.11 -X utf8 tools/scene_opcode_coverage.py --build # regen the SC0000 report doc + +Writes build/scene-opcode-coverage/.md and prints a summary to stdout. +""" +from __future__ import annotations +import sys +import re +import json +import collections +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import paths +import sys4load + +VM_SRC = paths.REPO / "engine" / "Age.Engine" / "Vm" / "VirtualMachine.cs" +OPCODES = paths.BUILD / "opcodes.json" +CALLNAME = paths.BUILD / "callscript-names.json" +OUT_DIR = paths.BUILD / "scene-opcode-coverage" + + +def implemented_labels() -> set[str]: + """Every opcode label the VM dispatches with an explicit `case` arm.""" + src = VM_SRC.read_text(encoding="utf8") + return set(re.findall(r'case\s+"([^"]+)"\s*:', src)) + + +def opcode_meta() -> dict[int, dict]: + """op(int) -> {label,name,category,noop_headless,confidence} from opcodes.json.""" + doc = json.loads(OPCODES.read_text(encoding="utf8")) + out: dict[int, dict] = {} + for e in doc["opcodes"]: + op = int(e["op"], 16) + sem = e.get("semantics", {}) + out[op] = { + "label": e.get("label", f"op_{op:#x}"), + "name": sem.get("name", ""), + "category": sem.get("category", "unknown"), + "noop_headless": bool(sem.get("noop_headless", False)), + "confidence": sem.get("confidence", "low"), + "summary": sem.get("summary", ""), + } + return out + + +def callscript_names() -> dict[int, str]: + if not CALLNAME.exists(): + return {} + doc = json.loads(CALLNAME.read_text(encoding="utf8")) + # file maps str(id) -> name + return {int(k): v for k, v in doc.items()} + + +def classify(op: int, meta: dict, impl: set[str]) -> str: + label = meta.get(op, {}).get("label", "") + if label in impl: + return "impl" + if meta.get(op, {}).get("noop_headless", False): + return "safe-noop" + return "GAP" + + +def analyze(scene: str, meta: dict[int, dict], impl: set[str], cnames: dict[int, str]): + scripts = paths.scripts() + key = scene.upper() + if not key.endswith(".BIN"): + key += ".BIN" + if key not in scripts: + raise SystemExit(f"scene not found in corpus: {key}") + scr = sys4load.load(scripts[key]) + + hist = collections.Counter() + callscript_ids = collections.Counter() + for ins in scr.instructions: + hist[ins.opcode] += 1 + m = meta.get(ins.opcode, {}) + if m.get("label") == "call-script" and ins.args: + a0 = ins.args[0] + # only resolve immediate ids (arg type 0); dynamic ids left unresolved + if getattr(a0, "type", None) == 0: + callscript_ids[a0.value] += 1 + + rows = [] + for op, n in hist.items(): + m = meta.get(op, {"label": f"op_{op:#x}", "name": "", "category": "unknown", + "confidence": "low", "summary": ""}) + rows.append({ + "op": op, "count": n, "label": m["label"], "name": m["name"], + "category": m["category"], "confidence": m["confidence"], + "summary": m["summary"], "status": classify(op, meta, impl), + }) + + total_ins = sum(hist.values()) + by_status = collections.Counter(r["status"] for r in rows) + ins_by_status = collections.Counter() + for r in rows: + ins_by_status[r["status"]] += r["count"] + + return { + "scene": key, "rows": rows, "total_ins": total_ins, + "distinct": len(rows), "by_status": by_status, + "ins_by_status": ins_by_status, "callscript_ids": callscript_ids, + "cnames": cnames, + } + + +STATUS_ORDER = {"GAP": 0, "impl": 1, "safe-noop": 2} + + +def render_md(a: dict) -> str: + L = [] + L.append(f"# Opcode coverage — {a['scene']}") + L.append("") + L.append("> Generated by `tools/scene_opcode_coverage.py` — do NOT hand-edit.") + L.append("> Static histogram of the scene's bytecode vs the C# VM's implemented ops.") + L.append("") + d, t = a["distinct"], a["total_ins"] + bs, ib = a["by_status"], a["ins_by_status"] + + def pct(x, whole): + return f"{100.0*x/whole:.1f}%" if whole else "—" + + L.append("## Summary") + L.append("") + L.append(f"- **{d} distinct opcodes**, {t} instructions total.") + L.append(f"- **impl**: {bs['impl']} ops ({pct(bs['impl'], d)}) / " + f"{ib['impl']} instrs ({pct(ib['impl'], t)}) — VM has a handler.") + L.append(f"- **safe-noop**: {bs['safe-noop']} ops ({pct(bs['safe-noop'], d)}) / " + f"{ib['safe-noop']} instrs ({pct(ib['safe-noop'], t)}) — markers, correct to skip.") + L.append(f"- **GAP**: {bs['GAP']} ops ({pct(bs['GAP'], d)}) / " + f"{ib['GAP']} instrs ({pct(ib['GAP'], t)}) — effectful, silently stubbed.") + handled = bs["impl"] + bs["safe-noop"] + L.append(f"- **Correctly handled (impl + safe-noop): {handled}/{d} ops " + f"= {pct(handled, d)}**; remaining {bs['GAP']} are the completeness holes below.") + L.append("") + + def table(title, rows): + L.append(f"## {title}") + L.append("") + if not rows: + L.append("_(none)_") + L.append("") + return + L.append("| op | count | label | name | category | conf | summary |") + L.append("|----|------:|-------|------|----------|------|---------|") + for r in rows: + summ = r["summary"].replace("|", "\\|") + L.append(f"| `{r['op']:#x}` | {r['count']} | `{r['label']}` | " + f"{r['name']} | {r['category']} | {r['confidence']} | " + f"{summ} |") + L.append("") + + rows = sorted(a["rows"], key=lambda r: (STATUS_ORDER[r["status"]], -r["count"])) + gaps = [r for r in rows if r["status"] == "GAP"] + impl = [r for r in rows if r["status"] == "impl"] + noop = [r for r in rows if r["status"] == "safe-noop"] + + table("GAP — effectful ops the VM silently skips (the completeness holes)", gaps) + table("impl — handled by the VM", impl) + table("safe-noop — markers / structural (correct to skip headless)", noop) + + if a["callscript_ids"]: + L.append("## call-script targets (immediate ids in this scene)") + L.append("") + L.append("Subroutines this scene invokes statically — their own opcodes are NOT") + L.append("counted above (run the tracker on them too for full-depth coverage).") + L.append("") + L.append("| id | name | sites |") + L.append("|----|------|------:|") + for cid, n in sorted(a["callscript_ids"].items(), key=lambda kv: -kv[1]): + nm = a["cnames"].get(cid, "?") + L.append(f"| `{cid:#x}` | {nm} | {n} |") + L.append("") + + return "\n".join(L) + + +def print_summary(a: dict): + bs, ib, d, t = a["by_status"], a["ins_by_status"], a["distinct"], a["total_ins"] + handled = bs["impl"] + bs["safe-noop"] + print(f"\n{a['scene']}: {d} distinct opcodes, {t} instructions") + print(f" impl {bs['impl']:3d} ops / {ib['impl']:6d} instrs") + print(f" safe-noop {bs['safe-noop']:3d} ops / {ib['safe-noop']:6d} instrs") + print(f" GAP {bs['GAP']:3d} ops / {ib['GAP']:6d} instrs <- effectful, stubbed") + print(f" handled {handled}/{d} ops ({100.0*handled/d:.1f}%)") + if bs["GAP"]: + gaps = sorted((r for r in a["rows"] if r["status"] == "GAP"), + key=lambda r: -r["count"]) + print(" GAP ops:", ", ".join( + f"{r['label']}({r['op']:#x})×{r['count']}" for r in gaps)) + + +def main(argv: list[str]): + scenes = [x for x in argv if not x.startswith("--")] + if not scenes: + scenes = ["SC0000"] + meta = opcode_meta() + impl = implemented_labels() + cnames = callscript_names() + OUT_DIR.mkdir(parents=True, exist_ok=True) + for scene in scenes: + a = analyze(scene, meta, impl, cnames) + out = OUT_DIR / f"{a['scene'].replace('.BIN', '')}.md" + out.write_text(render_md(a), encoding="utf8") + print_summary(a) + print(f" -> {out.relative_to(paths.REPO)}") + + +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/vm-map/opcodes.toml b/vm-map/opcodes.toml index 5410c75..166ed96 100644 --- a/vm-map/opcodes.toml +++ b/vm-map/opcodes.toml @@ -6644,14 +6644,14 @@ argc = 0 abi_source = "kelebek+decode-validated" [opcode.semantics] -name = "u00416410" -category = "unknown" -summary = "" -noop_headless = false -source = "kelebek" +name = "script-entry" +category = "marker" +summary = "zero-arg; the first instruction of a script (offset 0), opens the decl chain that 0x258 continues — script/prologue entry marker, structural" +noop_headless = true +source = "harness" confidence = "low" depends_on = [] -evidence = "" +evidence = "SC0000 offset 0x0 = op 0x259 (argc 0); 0x258's summary names it 'script-entry 0x259'; VM treats it as no-op (default stub) across all 279 CLEAN A0 scenes" [[opcode]] op = 0x2bd