re: record differential oracle outcome + first divergence
engine-re.md: the offset-path diff method, the working capture (operand hook 0x41b940; tick 0x410fb0 sees ecx!=ctx), the two capture caveats (hook-before-load gating + operand-mode argc>=1 filter), SC0000 codebase id, and the first divergence found (op 0xa0 jcc @ 0x8d on uninitialized G[0x6c1], the ADV-chrome enable — a two-boot-gap state hole, not the predicted coroutine yield). tools-reference.md: trace_engine_ops.py, diff_optrace.py, --trace-json. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
gamer147
@@ -752,6 +752,49 @@ Diagnosed with the new `--gfx-log` compositor/op trace (docs/tools-reference.md)
|
||||
NOT a compositor/z-order/blend bug. Diagnostics: `AGE_DIAG_SETTEX=1` env → VM logs each `set-texture`
|
||||
slot operand + `query-gfx-object?` result.
|
||||
|
||||
### Differential offset-path oracle — engine-vs-VM control-flow diff (2026-07-09)
|
||||
|
||||
**Method (lever #3 of the RE-front-loading program).** Run the same scene in the real engine and our C#
|
||||
VM and diff the **executed script-offset path**. Both run the same bytecode, so the opcode at each offset
|
||||
is static (from disasm); the first place the two offset sequences differ is exactly the branch/opcode/state
|
||||
we modeled wrong — cheaper and higher-signal than diffing effects, and precisely where the render-drift
|
||||
walk-backs lived. Tools: `tools/frida/trace_engine_ops.py` (engine capture) · `Age.Cli trace <SCENE>
|
||||
--boot --trace-json` (VM capture) · `tools/diff_optrace.py` (align + first-divergence). Spec/plan:
|
||||
`docs/superpowers/{specs,plans}/2026-07-09-differential-oracle*`.
|
||||
|
||||
**Capture method that WORKS = the operand hook `vm_operand_fetch@0x41b940`** (thiscall `ecx=ctx`; per op,
|
||||
`offset=(pc−codebase)/4` from `cur_ctx_index@0x53d14` / `frame_pc@0x53d2c` / `frame_codebase@0x53d28`,
|
||||
per-context stride `0x78`). Validation: **100% of captured offsets land on valid SC0000 instruction
|
||||
starts** — proves both the ctx-field offsets and the `(pc−codebase)/4` math. **The tick hook
|
||||
`adv_interpreter_tick@0x410fb0` does NOT work** — plain-JS `Interceptor.attach` there sees `ecx≠ctx`
|
||||
(0 entries), so the recon gate's *tick* path is closed too, not just the CModule path noted under Frame
|
||||
cadence. Two capture caveats, both handled:
|
||||
- **Frida must hook BEFORE the scene loads.** The scene-entry setup (decl preamble + first CG load at
|
||||
`0x802 mov G[0x62424]=0x23; call label_12649`) runs in a µs burst at load; a capture started mid-scene
|
||||
floors at ~`0x80d` and misses it. `trace_engine_ops.py` writes `build/tracer-live.flag` once the hook is
|
||||
installed → launch it in the background and gate the New-Game trigger on that flag.
|
||||
- **Operand mode skips zero-operand ops** (stmt-begin/end markers, script-entry `0x259`) — they never
|
||||
trigger an operand fetch. `diff_optrace.py` filters the VM trace to argc≥1 ops (same subsequence);
|
||||
control flow is preserved (markers don't branch). `--full` disables it for a hypothetical tick capture.
|
||||
|
||||
**Codebase identification.** The engine trace tags each op with its script's `codebase` pointer (a per-run
|
||||
heap address). `pick_scene_codebase` picks the codebase whose offset sequence shares the longest common
|
||||
prefix with the VM trace. From a boot→opening capture (7 codebases / 265k ops), SC0000 = `0x09c1afe8`
|
||||
(13252 ops, 100% valid, first offset `1` = offset 0 filtered).
|
||||
|
||||
**FIRST DIVERGENCE FOUND (the tool's first catch + self-test).** On the SC0000 opening the VM and engine
|
||||
agree for **27 ops** (including the coroutine op `0x7b` @ `0x79`, which matched), then **diverge at offset
|
||||
`0x8d` = op `0xa0` (jcc) on `global-int G[0x6c1]`:** the engine falls through to `0x94` (the op-`0x90`
|
||||
hotspot-chrome registration block) ⇒ `G[0x6c1]≠0`; the VM jumps to `label_df` (`0xdf`) ⇒ `G[0x6c1]==0`.
|
||||
Because both ran `1..0x8d` identically, `G[0x6c1]` is set **before** SC0000 — by pre-scene *system* boot
|
||||
the VM's cold `--boot` (INITCONFIG/INIT2/INIT) doesn't replicate (the **two-boot gap**). `G[0x6c1]` is an
|
||||
unlabeled but heavily-used scalar (766 uses) in the **same cluster as the op-`0x90` hotspot flags
|
||||
`G[0x6c9..0x6cd]`** = ADV-chrome/input state. Headlessly benign (no input) but a genuine VM-side state gap
|
||||
— and a clean demonstration that the oracle localizes a mis-modeled branch to a single instruction. It is
|
||||
NOT the predicted coroutine yield (`0x140`/`~0x50f`); the oracle reports whatever diverges *first*, and it
|
||||
surfaced an earlier state hole. **This is now the repeatable way to localize a mis-modeled op/state.**
|
||||
Phase-2 extension (deferred): effects-diff (global-bank / gfx-registry writes) for branchy scenes.
|
||||
|
||||
---
|
||||
|
||||
## Native walls backlog (targets for this loop)
|
||||
|
||||
@@ -64,6 +64,7 @@ All opcode knowledge (ABI, semantics, provenance, `depends_on`) is hand-edited *
|
||||
| `vm0.py` | Headless Python bytecode VM (Phase A0 execution-model prototype; reuses `sys4load`). | `--test` (RECOVER unit test) · `--sweep [N]` (oracle coverage) · `--scene NAME` · `--settex NAME` (set-texture resId trace + exec trace) · `<file.BIN>` | corpus → stdout; `build/vm0-trace.json`; `build/settex-<NAME>.json` |
|
||||
| `scene_opcode_coverage.py` | Per-scene opcode completeness gauge: histograms a scene's static opcodes and classifies each **impl** / **safe-noop** / **GAP** (effectful op the VM silently stubs). Implemented set parsed from `VirtualMachine.cs` `case` arms; metadata from `opcodes.json`. Surfaces the concrete rendering/feature holes so a half-drawn scene reads as "N ops still stubbed", not "mystery". | `scene_opcode_coverage.py [SCENE …]` (default SC0000) | corpus, `build/opcodes.json`, `engine/…/VirtualMachine.cs`, `build/callscript-names.json` → ⚙ `build/scene-opcode-coverage/<SCENE>.md` + stdout |
|
||||
| `correlate_scope.py` | Align the VM's `set-texture(resId)` trace with the game's Frida load order → tag each load's DATA2 package, flag package transitions, dump the significant ops in each transition span (the **scope selector** hunt). | `correlate_scope.py <SCENE>` | `build/settex-<SCENE>.json` + `build/frida-load-order-result.json` + index → stdout |
|
||||
| `diff_optrace.py` | **Differential offset-path oracle** (`docs/engine-re.md`): diff the engine's executed offset path (`trace_engine_ops.py`) against the VM's (`Age.Cli trace --trace-json`) → first divergence = the mis-modeled branch/op/state, with opcode + ±3 ops of context. Identifies the scene's codebase by longest-common-prefix; filters the VM trace to argc≥1 (operand-capture parity). Pure core unit-tested (`test_diff_optrace.py`). | `py -3.11 -X utf8 tools/diff_optrace.py SC0000 [--full]` | `build/engine-optrace.jsonl` + `build/vm-optrace.json` + disasm → stdout |
|
||||
|
||||
## Engine (C#) — VM core, CLI, Godot frontend
|
||||
|
||||
@@ -81,6 +82,7 @@ subsystem oracles. Test scenes are **synthesized** via `Age.Engine/Sys4/ScriptAs
|
||||
|---|---|---|
|
||||
| `run <file.BIN>` | Execute a script; print steps, show-text count, **call-script dispatch count**, the first 30 lines (each tagged with its source script), and the distinct source scripts. | `CaptureHost` (headless); **executes call-script**. |
|
||||
| `trace <out.json>` | Trace every SC/SP scene → offsets + halt + steps. **Provider-less** (call-script stubbed) = a base-ISA offset dump. | writes JSON. (Was the vm0 differential oracle; vm0 is retired from oracle duty — `TraceDiffTests` removed.) |
|
||||
| `trace <SCENE.BIN> [--boot] --trace-json <out>` | ★ Emit the **full per-op executed-offset path** of one scene (not just show-text), filtered to the scene's own frame — the VM side of the differential offset-path oracle (`diff_optrace.py`). `--boot` runs the SYSTEM4 state prefix first (matches the engine). | `JsonOffsetTraceSink` (observe-only, parity held) → `{scene, offsets:[…]}` JSON. |
|
||||
| `audio <SCENE.BIN> [0xADDR=VAL…]` | Dump executed `play-bgm`/`play-voice` in order + resolved file. | optional seeds. provider-less (stub) for now. |
|
||||
| `gfx [--boot] <SCENE.BIN> [0xADDR=VAL…]` | Dump executed `set-texture`/`get-texture-size`/`draw-texture` (resolved file + computed geometry) **plus the per-object gfx slots** — the headless geometry oracle. **`--boot`** runs SYSTEM4's state prefix (`INITCONFIG/INIT2/INIT`) via `GameSession` first (so INIT2's gfx handle array is present) and runs the target with call-script on; without it, seeds-only + provider-less. | gfx ops now execute against `GfxState`. |
|
||||
| `play [--boot] [--state <f>] [--save-state <f>] <SCENE.BIN…> [0xADDR=VAL…]` | ★ Cross-scene **state runner**: run a scene sequence carrying persistent globals. `--boot` first runs the 9 `*INIT` data scripts (real skill/item/unit/map/stage state). `--state`/`--save-state` load/persist a JSON snapshot. | `GameSession`; **executes call-script**. |
|
||||
@@ -160,6 +162,7 @@ texture ops (no GPU context) — run windowed for real scenes. User args (after
|
||||
| `tools/frida/capture_gfx_objects.py` | Capture the native gfx object-manager state: grab engine ctx (`esi` via operand-fetch `ecx`), poll the object-record array `[esi+0x53d64]` (20×120B; `field[0]=0xffffffff`=free, cmd-type at rec+0x24). **⚠ Its "0 CG records ⇒ drift is state-divergence" reading was DISPROVEN** (Ghidra: op 0x215 read settles the drift as a native command-buffer op — `docs/engine-re.md`; the poll observed the record array, not the lookup map that drives the branch, and cmd-buffer records are transient). Kept as a runtime-observation tool. | `py -3.11 -u -X utf8 tools/frida/capture_gfx_objects.py [pid] [secs]` | running game → `build/gfx-objects.jsonl` |
|
||||
| `tools/frida/probe_frame_cadence.py` | **Frame-cadence probe** (`docs/engine-re.md` "Frame cadence — live measurement"): plain-JS hook on operand-fetch `0x41b940` (grab ctx + count exec rate) + system-DLL message/timing hooks; auto-buckets by Ctrl/skip-bit. Measured: exec **rate-limited** ~1788 ops/sec normal, ~4× fast-forward. **Read-only/import-only — never CModule-hook the hot interpreter (crashes the game).** Play actively during capture; hold Ctrl the back half. | `py -3.11 -u -X utf8 tools/frida/probe_frame_cadence.py [secs] [proc]` | running game → `build/frida-frame-cadence.jsonl` + stdout report |
|
||||
| `tools/frida/probe_present.py` | **Present-rate probe:** grab ctx, scan it for the D3D9 device (d3d9-vtable object with a full ~119-method table), hook `IDirect3DDevice9::Present`/`EndScene` (+ GDI-blit fallback). Found: **D3D9, UNCAPPED** (`Present` ~1908/sec, no vsync; no `ddraw`; 2D StretchRect compositor) ⇒ no fixed frame rate. Click 2–3× at start to grab ctx. | `py -3.11 -u -X utf8 tools/frida/probe_present.py [secs]` | running game → `build/frida-present.jsonl` + stdout report |
|
||||
| `tools/frida/trace_engine_ops.py` | **Engine op-path tracer** for the differential oracle (`docs/engine-re.md` "Differential offset-path oracle"): per executed op, read `cur_ctx_index@0x53d14`/`frame_pc@0x53d2c`/`frame_codebase@0x53d28` → emit `(codebase, offset=(pc−codebase)/4)`. **Use `--hook operand` (0x41b940, proven-safe)** — `--hook tick` (0x410fb0) sees `ecx≠ctx` (0 entries). Writes `build/tracer-live.flag` when the hook is installed → launch in the background, gate the New-Game trigger on the flag (else the scene-entry burst is missed). | `py -3.11 -u -X utf8 tools/frida/trace_engine_ops.py [--hook operand\|tick] [secs]` | running game → `build/engine-optrace.jsonl` |
|
||||
|
||||
*(Static disassembly of `build/engine-dump/range_00400000.bin` uses **capstone** — `py -3.11 -m pip install capstone`; VA `X` → file offset `X−0x400000`.)*
|
||||
|
||||
|
||||
Reference in New Issue
Block a user