From 5366fe2171d69919faba28b028ced28b1574e671 Mon Sep 17 00:00:00 2001 From: gamer147 Date: Wed, 8 Jul 2026 17:26:27 -0400 Subject: [PATCH] docs+godot: finish frame-cadence RE notes + headless shot null-guard Prior-session WIP: RE findings on the engine frame cadence (engine-re.md, phase-a-slice-plan.md, tools-reference.md) and a null-guard so headless --shot-sequence advances without a rendered viewport texture. Co-Authored-By: Claude Opus 4.8 --- docs/engine-re.md | 76 ++++++++++++++++++++++++++++++++++++++ docs/phase-a-slice-plan.md | 34 +++++++++++++++++ docs/tools-reference.md | 2 + godot/Main.cs | 6 ++- 4 files changed, 116 insertions(+), 2 deletions(-) diff --git a/docs/engine-re.md b/docs/engine-re.md index 510735a..7733f9a 100644 --- a/docs/engine-re.md +++ b/docs/engine-re.md @@ -409,6 +409,82 @@ corrected). Profile the real Godot run (`--trace-histogram`) to find it. no-op (`noop_headless=true`); the Kelebek label `u00416200` was VA-drift. This corrects the earlier open item ("no per-frame present") above — present is host-implicit; only `sleep` timing was missing. +### Frame cadence — the interpreter tick, and why our port "speeds through" (2026-07-08) + +Answers the open question the `sleep` section above left ("what advances the rapid opening burst is still +unknown"). The pace is an **engine-level execution cadence**, not any bytecode primitive. Corroborated in-game +by Ctrl fast-forwarding ADV (a speed governor). Motivated by the user's observation that our port visibly +speeds through the opening — which contradicted, and correctly overturned, an earlier same-day overclaim that +"there is no missing pacer" (that was inferred from headless op-counts, which cannot render). + +**Confirmed from the engine image (annotated in Ghidra):** + +- **The interpreter is a cooperative one-op-per-tick step, not a run-to-completion loop.** + `adv_interpreter_tick`@`0x410fb0` (renamed from `FUN_00410fb0`) executes **exactly one opcode** per call: + `op = **(ctx+0x53d2c + curCtx*0x78)`; if `0 ≤ op ≤ 0x3ff` it dispatches `(*(ctx+0x9b24c+op*4))()` (the + handler table = `ctx[0x26c93+op]`) then advances `PC += *(ctx+0x53d88+curCtx*0x78) * 4` (decoded cmd size), + else the default handler `FUN_004162b0`. It also runs the **message-skip / click / auto-advance** logic each + tick (`s_set_CancelMesSkipOnClick`, `s_message_ReadTextSkip`, skip bit `ctx+0xa0ce4 & 0x8000000`) — i.e. the + **Ctrl fast-forward governor lives at the per-op level**, and a click can reposition the PC (skip-to-next). +- **Script contexts are coroutine records.** `curCtx = *(ctx+0x53d14)` indexes `0x78`-byte records at + `ctx+0x53d60`/`ctx+0x53d2c` (PC, codebase, cmd-size). The engine multiplexes script "threads." Init/reset = + `scene_context_init_reset`@`0x40b3b0` (zeroes `0x53d14` + `0xa0ce4`, allocs surfaces `ctx+0x52bd4[1000]`). +- **Advancement is gated by an interpreter run-state flags word `ctx+0xa0ce4`** (bit1 = sleeping, plus wait/ + skip/etc.), read+written by ~40 state functions. `sleep_op_0xc8` sets bit1 + arms the ms timer and returns — + it does not block. So the outer loop consults `0xa0ce4` to decide whether to step the script this frame. +- **Effects are frame-stepped.** Screen transitions `FUN_0043cdb0` (12 wipe/slide modes) render **one frame per + step** and take a step-count parameter (the natural place a speed multiplier applies); `present-frame` (0x20c) + and the anim clock (0x238) advance per frame. A CG transition therefore spreads over many real frames. +- **Timing source** = the ms-clock function pointer `*DAT_0056f3d4` (`timeGetTime`-class), used throughout. + +**Model:** single-threaded, vsync-timed frame loop; each frame it steps opcodes until the context **yields** +(`sleep` armed / `wait-for-input` 0x72 / active frame-stepped transition/anim / present), renders +(`gfx_render_frame`), waits on the clock, continues. Back-to-back draws inside one page compose into a single +frame (fine); the opening's CG-to-CG advances are gated by frame-stepped transitions + sleeps, which spread +them over real time. + +**⚠ Not statically resolvable (honest boundary):** the **outer frame loop itself** is not readable from this +dump. `adv_interpreter_tick` is invoked through a **runtime-set mode function pointer** (heap/vtable slot) — it +has zero static xrefs, and its address bytes (`b0 10 41 00`) appear nowhere in `range_00400000` (0x400000– +0x65ffff). The functions touching the scheduler state (`0x53d14`, `0xa0ce4`) are init/reset, save +(`context_state_serialize`@`0x40d320`), and op-handlers — never the loop. The "run-until-yield then render" +statement above is a **reconstruction** from those pieces, not a line read from the loop; pinning the actual +loop + its exact per-frame step budget / vsync wait needs a **live-debugger break** (attach + break in the +frame loop), or a wider memory dump that includes the mode object. + +**Port relevance (the speed-through root cause).** Our Godot VM runs on a **free-running background thread** +(`Task.Run(() => vm.Run())` in `Main.cs`) with no frame binding — it executes an entire page's ops in +microseconds; only `WaitForInput` and `Sleep` pause it, and the compositor merely samples `GfxState` at 60fps. +So every no-`sleep` CG/state advance collapses to its end state → the speed-through. **Fix shape:** throttle +the VM to a bounded wall-clock op rate (see the measured numbers below) via a per-opcode host yield; retire the +free-running thread. Spec: `docs/superpowers/specs/2026-07-08-frame-stepped-vm-design.md`. + +### Frame cadence — live measurement (2026-07-08, Frida read-only) + +The static pass couldn't reach the outer loop, so we measured the running game. **Read-only / import-only +only** (`tools/frida/probe_frame_cadence.py`, `probe_present.py`): a plain-JS hook on the proven operand-fetch +`0x41b940` (grab ctx + count exec rate) + system-DLL hooks; no engine-code patching. **Lesson learned the hard +way:** a first attempt with a **CModule** hook on the hyper-hot `adv_interpreter_tick` crashed the game +instantly (bad native callback into the hottest path — *not* anti-tamper; our other scripts hook engine code +via plain JS and survive). Use plain-JS hooks on proven addresses + memory polling. + +Findings: +- **Execution is rate-limited, not free-running.** Normal active rate ≈ **1,788 operand-fetches/sec** (peak + ~5,796) — far below an unthrottled interpreter (millions/sec), so the engine paces itself. Execution is + bursty (parked at `wait-for-input` prompts, then a bounded burst), confirming per-iteration op-budgeting. +- **Fast-forward (Ctrl) scales the rate ~4×** (≈7,738/sec avg, peak ~15,572), gated by the engine skip bit + **`ctx+0xa0ce4 & 0x8000000`** (set only while fast-forwarding). It runs *more ops per unit time* — it does + not skip content. (Ctrl is **ADV-scoped**; it does not speed up gameplay/menus.) +- **Rendering = Direct3D 9, UNCAPPED.** `ddraw.dll` is not loaded; the game uses `d3d9.dll` (+ `nvd3dum.dll`). + `IDirect3DDevice9::Present` (device vtable slot 17, found by scanning ctx for a d3d9-vtable object with a + full ~119-method table) fires ~**1,908/sec** with **no vsync**; `BeginScene`/`EndScene` never fire → a **2D + StretchRect-style compositor**, not a 3D scene. So there is **no fixed display-frame rate**; `Present` rate + ≈ op rate (~1 op per present). ⇒ the pacing quantity is the **wall-clock op rate**, not a per-frame budget. +- **Implication for the port:** throttle our VM to ~**1,800 ops/sec** wall-clock (≈30 ops per 60 fps Godot + `_Process`, tunable), ~4× under a future Ctrl multiplier; Godot's 60 fps compositor + wall-clock tweens then + show the smoothly-advancing state. This *measured* mechanism replaces the earlier present-driven guess + (present is rare in our path and uncapped natively). + ### The render drift's SECOND half: missing system-boot state (2026-07-07, resolved) Implementing the gfx ops (above) was necessary but not sufficient — a cold single-scene run of SC0000 still diff --git a/docs/phase-a-slice-plan.md b/docs/phase-a-slice-plan.md index 565e3d3..d6dbc5a 100644 --- a/docs/phase-a-slice-plan.md +++ b/docs/phase-a-slice-plan.md @@ -528,3 +528,37 @@ DRAWOBJ, CALCREVISE, LOOK) live in the real runtime; SC0000 renders the opening provider-less); **engine-level diagnostics** (the next pivot — the engine, not the frontend, should surface script/scene execution + call-script dispatch); `decision→scene` (scene chaining) rides this same loader once the SCJUMP decision→scene-id native hop is reversed. + +## A2b — opening speed-through: the pacing lead is ENGINE CADENCE (2026-07-08) + +> ⛔ **Correction.** An earlier draft of this section claimed "there is no missing pacer — all opening +> pacing primitives are already shipped." That was **retracted**: it was inferred from *headless op-counts* +> (which cannot render) and it contradicts the direct eyes-on observation that the opening **visibly speeds +> through**. Ground truth = it speeds through; the pacer is real and the cause is still open. What survives +> below is only the mechanically-verified part plus the corrected hypothesis. + +**Verified (mechanical).** +1. The "rapid burst" seen in *plow* traces is a headless fiction (input-plow). The real back-to-back + `set-texture→draw-texture` swaps are the **per-page compositor** `label_1235a` (instruction indices + `0x3958–0x3973` = dword `0x123e8–0x12497`): a loop over ≤8 gfx object slots that per slot queries state + (`0x215`), erases/releases (`0x1f7`/`0x1fa`), and draws with alpha (`0x203`). +2. Op `0xcd get-input-type` is **unmodeled** in the VM (no `case`; only `0x72 wait-for-input` is handled — + two input mechanisms, one modeled). `INPUTNAME.BIN:0x1c1` name-entry is a `get-input-type→jcc→sleep 1→jmp` + poll that spins unfed. This is a **real but separate** gap (interactive blocker), **not** the speed-through. + +**The corrected hypothesis (engine cadence, not bytecode).** Many opening draws are **back-to-back with no +`sleep` between them in the bytecode** (e.g. resId `0x29` twice at `0xc45`/`0xc52`; `0x34`→`0x35` at +`0x1467`/`0x1479`), yet the real game paces them. So the pace comes from the **engine's execution cadence**, +not a script primitive. Corroboration: holding **Ctrl fast-forwards ADV** in the real game (faster, not +instant) — a global engine speed governor over interpreter advancement. **Suspected cause in our port:** the +Godot VM runs on a **free-running background thread** that is not synced to the 60fps compositor, so it blasts +an entire page's draws in microseconds and the main-thread `Recomposite` only ever samples the final gfx-state +→ the intermediate CGs collapse. The native engine is a cooperatively-scheduled main loop where the interpreter +yields per frame under vsync (consistent with `sleep` already being RE'd as a *non-blocking, main-loop-polled* +timer — which only makes sense if the interpreter yields back to that loop). + +**Next (open).** (1) Determine what actually landed re 60fps/vsync and the VM threading model in `godot/`. +(2) RE the native engine main loop in Ghidra: how it ticks the interpreter per frame, the frame cap/vsync, and +the Ctrl speed governor. (3) Frame-lock our VM to the render loop (a per-frame step budget or a per-frame +yield/sync point) instead of the free-running thread. Confirm any fix against **pixels** (windowed +`--shot-sequence` → real PNGs), not op-counts. diff --git a/docs/tools-reference.md b/docs/tools-reference.md index 9d5a92e..50c43ec 100644 --- a/docs/tools-reference.md +++ b/docs/tools-reference.md @@ -156,6 +156,8 @@ texture ops (no GPU context) — run windowed for real scenes. User args (after | `tools/frida/dump_engine.py` | ★ **Dump the UNPACKED engine code** from the live process for offline static RE (native handlers). `AGE.EXE` unpacks in-place at `0x400000`; Kelebek VAs map `VA−0x400000` = file-off. Validated via the AGF-decoder landmark `+0x74f1f`. | `py -3.11 -u -X utf8 tools/frida/dump_engine.py [pid]` | running game → `build/engine-dump/{manifest.json,range_.bin}` | | `tools/frida/probe_handlers.py` | Probe which region the interpreter executes from (module vs heap). Confirmed: **operand-fetch `+0x1b940` fires ~8500/s ⇒ interpreter runs from the module `0x400000`** (handlers hookable by dump address). | `py -3.11 -u -X utf8 tools/frida/probe_handlers.py [pid]` | running game → stdout (per-hook fire counts) | | `tools/frida/capture_gfx_objects.py` | Capture the native gfx object-manager state: grab engine ctx (`esi` via operand-fetch `ecx`), poll the object-record array `[esi+0x53d64]` (20×120B; `field[0]=0xffffffff`=free, cmd-type at rec+0x24). **⚠ Its "0 CG records ⇒ drift is state-divergence" reading was DISPROVEN** (Ghidra: op 0x215 read settles the drift as a native command-buffer op — `docs/engine-re.md`; the poll observed the record array, not the lookup map that drives the branch, and cmd-buffer records are transient). Kept as a runtime-observation tool. | `py -3.11 -u -X utf8 tools/frida/capture_gfx_objects.py [pid] [secs]` | running game → `build/gfx-objects.jsonl` | +| `tools/frida/probe_frame_cadence.py` | **Frame-cadence probe** (`docs/engine-re.md` "Frame cadence — live measurement"): plain-JS hook on operand-fetch `0x41b940` (grab ctx + count exec rate) + system-DLL message/timing hooks; auto-buckets by Ctrl/skip-bit. Measured: exec **rate-limited** ~1788 ops/sec normal, ~4× fast-forward. **Read-only/import-only — never CModule-hook the hot interpreter (crashes the game).** Play actively during capture; hold Ctrl the back half. | `py -3.11 -u -X utf8 tools/frida/probe_frame_cadence.py [secs] [proc]` | running game → `build/frida-frame-cadence.jsonl` + stdout report | +| `tools/frida/probe_present.py` | **Present-rate probe:** grab ctx, scan it for the D3D9 device (d3d9-vtable object with a full ~119-method table), hook `IDirect3DDevice9::Present`/`EndScene` (+ GDI-blit fallback). Found: **D3D9, UNCAPPED** (`Present` ~1908/sec, no vsync; no `ddraw`; 2D StretchRect compositor) ⇒ no fixed frame rate. Click 2–3× at start to grab ctx. | `py -3.11 -u -X utf8 tools/frida/probe_present.py [secs]` | running game → `build/frida-present.jsonl` + stdout report | *(Static disassembly of `build/engine-dump/range_00400000.bin` uses **capstone** — `py -3.11 -m pip install capstone`; VA `X` → file offset `X−0x400000`.)* diff --git a/godot/Main.cs b/godot/Main.cs index 242bc1b..7846a39 100644 --- a/godot/Main.cs +++ b/godot/Main.cs @@ -162,8 +162,10 @@ public partial class Main : Godot.Control if (_seqDir != null && _seqIdx < _seqFrames && !_done) { System.IO.Directory.CreateDirectory(_seqDir); - var fimg = GetViewport().GetTexture().GetImage(); - fimg.SavePng($"{_seqDir}/frame_{_seqIdx:0000}.png"); + // Headless has no rendered viewport texture (GetImage() is null). Still advance/count/quit so the + // real-run trace-histogram can profile the live path without a display; only the PNG grab is skipped. + var fimg = GetViewport().GetTexture()?.GetImage(); + fimg?.SavePng($"{_seqDir}/frame_{_seqIdx:0000}.png"); _seqIdx++; if (_seqIdx >= _seqFrames) { GD.Print($"SEQ saved {_seqIdx} frames -> {_seqDir}"); GetTree().Quit(0); } return;