docs: archive EngineCtx design and implementation plan
This commit is contained in:
451
docs/superpowers/plans/2026-07-09-engine-ctx-struct.md
Normal file
451
docs/superpowers/plans/2026-07-09-engine-ctx-struct.md
Normal file
@@ -0,0 +1,451 @@
|
||||
# EngineCtx Struct — Implementation Plan
|
||||
|
||||
**Status:** completed 2026-07-09 in `3826064` and `0ecd776`. This is the archived execution plan;
|
||||
checkboxes below are preserved as originally authored.
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Type the engine context as a Ghidra `EngineCtx` struct sourced from a canonical `vm-map/engine-ctx.toml`, and retype the dispatch handlers' `this` so they decompile `ctx->field` instead of magic offsets.
|
||||
|
||||
**Architecture:** Single source of truth (`engine-ctx.toml`) → generator (`engine_ctx_build.py`) emits `build/engine-ctx.json` + `docs/engine-ctx-reference.md` → a `run_script_inline` pass builds the `EngineCtx` struct and retypes handler `this` params on `/v2`. Pure model/lint/emit logic is unit-tested; the applied result is validated by decompiling two handlers.
|
||||
|
||||
**Tech Stack:** Python 3.11 (`py -3.11 -X utf8`, `tomllib`), ghidra-mcp `run_script_inline` (Java; `GHIDRA_MCP_ALLOW_SCRIPTS=1`).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-07-09-engine-ctx-struct-design.md`.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Run Python as `py -3.11 -X utf8 tools/<name>.py …`. Tools compute paths via `paths.py`; `build/` is disposable.
|
||||
- **Never hand-edit generated files** (`build/engine-ctx.json`, `docs/engine-ctx-reference.md`): edit `engine-ctx.toml`, re-run `--build`.
|
||||
- Scope = engine `ctx` (esi) fields ONLY; the VM global bank `G[…]` is a separate space (`globals.toml`), out of scope.
|
||||
- **Ghidra:** confirm the active program is `/v2/range_00400000.bin` (base `0x400000`, 4400+ fns — the two-program gotcha) before any script; end mutation with `save_program`.
|
||||
- Seed fields are 4-byte scalars (`int`/`uint`/`void*`) — names the offset without span-management; arrays/sub-structs are a later refinement.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Canonical source + generator (`engine-ctx.toml` → JSON + doc)
|
||||
|
||||
**Files:**
|
||||
- Create: `vm-map/engine-ctx.toml` (canonical source)
|
||||
- Create: `tools/engine_ctx_build.py` (model + lint + emit + CLI)
|
||||
- Create: `tools/test_engine_ctx.py` (unit tests for lint + emit)
|
||||
- Create: `build/engine-ctx.json` (generated), `docs/engine-ctx-reference.md` (generated)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `load(toml_text) -> dict` (`{"meta": {...}, "fields": [{"offset":int,"name","type","note"}...]}`); `lint(model) -> list[str]` (errors); `emit_json(model) -> dict`; `emit_reference_md(model) -> str`.
|
||||
- `TYPE_SIZES = {"int":4, "uint":4, "void*":4}` (all seed fields 4 bytes).
|
||||
|
||||
- [ ] **Step 1: Create the canonical source `vm-map/engine-ctx.toml`**
|
||||
|
||||
```toml
|
||||
# vm-map/engine-ctx.toml -- CANONICAL source for the EngineCtx struct (hand-edited).
|
||||
# Generated: build/engine-ctx.json + docs/engine-ctx-reference.md via tools/engine_ctx_build.py --build.
|
||||
# Applied to the Ghidra /v2 image via run_script_inline (see the plan). ctx = engine context (esi/thiscall this).
|
||||
# The VM global bank G[...] is a SEPARATE space (vm-map/globals.toml), NOT ctx offsets.
|
||||
[meta]
|
||||
struct_name = "EngineCtx"
|
||||
size = 0xa1000
|
||||
|
||||
[[field]]
|
||||
offset = 0x3028
|
||||
name = "alt_pack_table"
|
||||
type = "int"
|
||||
note = "call-script high-byte alternate pack table (unused by corpus)"
|
||||
[[field]]
|
||||
offset = 0x408
|
||||
name = "gfx_obj_registry"
|
||||
type = "int"
|
||||
note = "retained gfx-object map (std::map handle->object); geometry/draw get-or-create, 0x215 returns obj+4 source slot, 0x1f7 erases"
|
||||
[[field]]
|
||||
offset = 0x40c
|
||||
name = "sys4ini_count"
|
||||
type = "int"
|
||||
note = "SYS4INI record count"
|
||||
[[field]]
|
||||
offset = 0x410
|
||||
name = "archive_name_table"
|
||||
type = "void*"
|
||||
note = "archive-name table base (arc_id*0x100 indexes it)"
|
||||
[[field]]
|
||||
offset = 0x414
|
||||
name = "sys4ini_records"
|
||||
type = "void*"
|
||||
note = "SYS4INI 80-byte record base {name[64],arc_id,file_number,offset,size}; record = base + id*0x50"
|
||||
[[field]]
|
||||
offset = 0xb558
|
||||
name = "gfx_dirty_a"
|
||||
type = "int"
|
||||
note = "gfx dirty flag (anim set raises)"
|
||||
[[field]]
|
||||
offset = 0xb560
|
||||
name = "gfx_dirty_b"
|
||||
type = "int"
|
||||
note = "gfx dirty flag"
|
||||
[[field]]
|
||||
offset = 0x14d54
|
||||
name = "gfx_obj_ptr_table"
|
||||
type = "void*"
|
||||
note = "per-object pointer table (ops 0x212/0x213 write obj+0x64/0x68/0x6c)"
|
||||
[[field]]
|
||||
offset = 0x14f45
|
||||
name = "script_frame_index"
|
||||
type = "int"
|
||||
note = "call-script frame index (0x1e-dword frames)"
|
||||
[[field]]
|
||||
offset = 0x46d14
|
||||
name = "query_table_46d14"
|
||||
type = "void*"
|
||||
note = "stride-0x14 table read by op 0x216"
|
||||
[[field]]
|
||||
offset = 0x51b64
|
||||
name = "frame_timer"
|
||||
type = "int"
|
||||
note = "frame timer (present updates 0x51b64/0x51b68)"
|
||||
[[field]]
|
||||
offset = 0x51b78
|
||||
name = "anim_clock_elapsed"
|
||||
type = "int"
|
||||
note = "global anim clock elapsed (op 0x238 zeroes)"
|
||||
[[field]]
|
||||
offset = 0x51b7c
|
||||
name = "anim_clock_duration"
|
||||
type = "int"
|
||||
note = "global anim clock total duration (op 0x238 sets)"
|
||||
[[field]]
|
||||
offset = 0x52bd4
|
||||
name = "surfaces"
|
||||
type = "void*"
|
||||
note = "surface array base [~1000 slots]; create/set-texture (0x1f8/0x1f9) allocate"
|
||||
[[field]]
|
||||
offset = 0x53d14
|
||||
name = "cur_ctx_index"
|
||||
type = "uint"
|
||||
note = "current gfx-object / script-context index (curCtx); indexes 0x78-byte records"
|
||||
[[field]]
|
||||
offset = 0x53d28
|
||||
name = "frame_codebase"
|
||||
type = "void*"
|
||||
note = "current frame codebase (PC = codebase + off*4)"
|
||||
[[field]]
|
||||
offset = 0x53d2c
|
||||
name = "frame_pc"
|
||||
type = "int"
|
||||
note = "current frame PC column (op = *(0x53d2c + curCtx*0x78))"
|
||||
[[field]]
|
||||
offset = 0x53d60
|
||||
name = "ctx_record_base"
|
||||
type = "void*"
|
||||
note = "0x78-byte context-record array base (coroutine/script contexts)"
|
||||
[[field]]
|
||||
offset = 0x53d64
|
||||
name = "gfx_obj_record_array"
|
||||
type = "void*"
|
||||
note = "gfx object-record array (field[0]=0xffffffff free; cmd-type at rec+0x24)"
|
||||
[[field]]
|
||||
offset = 0x53d88
|
||||
name = "cmd_type_table"
|
||||
type = "int"
|
||||
note = "per-object cmd-type column base (write *(0x53d88 + curCtx*0x78))"
|
||||
[[field]]
|
||||
offset = 0x55120
|
||||
name = "anti_tamper_a"
|
||||
type = "int"
|
||||
note = "anti-tamper checksum operand"
|
||||
[[field]]
|
||||
offset = 0x55124
|
||||
name = "anti_tamper_b"
|
||||
type = "int"
|
||||
note = "anti-tamper checksum operand"
|
||||
[[field]]
|
||||
offset = 0x5512c
|
||||
name = "anti_tamper_fp"
|
||||
type = "int"
|
||||
note = "anti-tamper (import fn ptr / result)"
|
||||
[[field]]
|
||||
offset = 0x55248
|
||||
name = "ret_stack_a"
|
||||
type = "void*"
|
||||
note = "per-frame return stack (op 0x8f call pushes)"
|
||||
[[field]]
|
||||
offset = 0x552e8
|
||||
name = "ret_stack_b"
|
||||
type = "void*"
|
||||
note = "per-frame return stack (companion)"
|
||||
[[field]]
|
||||
offset = 0x5f304
|
||||
name = "sleep_timer"
|
||||
type = "int"
|
||||
note = "sleep timer object (op 0xc8; +8 active, +0x14 start-ms, +0x18 duration)"
|
||||
[[field]]
|
||||
offset = 0x6da88
|
||||
name = "coroutine_yield_a"
|
||||
type = "void*"
|
||||
note = "op 0x7b yield-state save (op1 -> +ctxidx*4)"
|
||||
[[field]]
|
||||
offset = 0x6db28
|
||||
name = "coroutine_yield_b"
|
||||
type = "void*"
|
||||
note = "op 0x7b yield-state save (op2 -> +ctxidx*4)"
|
||||
[[field]]
|
||||
offset = 0x6dbc8
|
||||
name = "coroutine_runstate"
|
||||
type = "int"
|
||||
note = "op 0x7c resume gate (run-state bit 0x2000000)"
|
||||
[[field]]
|
||||
offset = 0x6dbcc
|
||||
name = "coroutine_resume_off"
|
||||
type = "int"
|
||||
note = "op 0x7c resume PC offset"
|
||||
[[field]]
|
||||
offset = 0x9b24c
|
||||
name = "dispatch_table"
|
||||
type = "void*"
|
||||
note = "opcode->handler table base [0x400]; handler(op) = *(0x9b24c + op*4)"
|
||||
[[field]]
|
||||
offset = 0xa0cc0
|
||||
name = "screen_w"
|
||||
type = "int"
|
||||
note = "screen width (640)"
|
||||
[[field]]
|
||||
offset = 0xa0cc4
|
||||
name = "screen_h"
|
||||
type = "int"
|
||||
note = "screen height (480)"
|
||||
[[field]]
|
||||
offset = 0xa0cc8
|
||||
name = "screen_bpp"
|
||||
type = "int"
|
||||
note = "screen bpp (8)"
|
||||
[[field]]
|
||||
offset = 0xa0ce4
|
||||
name = "run_state_flags"
|
||||
type = "uint"
|
||||
note = "interpreter run-state flags (bit1 sleeping; 0x8000000 skip/fast-forward)"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write the failing tests**
|
||||
|
||||
```python
|
||||
# tools/test_engine_ctx.py (plain runner, no pytest)
|
||||
import sys
|
||||
from engine_ctx_build import load, lint, emit_json
|
||||
|
||||
FAILS = []
|
||||
def check(c, m):
|
||||
(FAILS.append(m) or print("FAIL:", m)) if not c else print("ok:", m)
|
||||
|
||||
GOOD = '[meta]\nstruct_name="EngineCtx"\nsize=0x1000\n' \
|
||||
'[[field]]\noffset=0x10\nname="a"\ntype="int"\nnote="x"\n' \
|
||||
'[[field]]\noffset=0x20\nname="b"\ntype="void*"\nnote="y"\n'
|
||||
|
||||
def test_load_and_emit():
|
||||
m = load(GOOD)
|
||||
j = emit_json(m)
|
||||
check(j["fields"]["0x10"]["name"] == "a" and j["meta"]["struct_name"] == "EngineCtx",
|
||||
"emit_json keys fields by hex offset + carries meta")
|
||||
|
||||
def test_lint_clean():
|
||||
check(lint(load(GOOD)) == [], "clean model lints with no errors")
|
||||
|
||||
def test_lint_catches_overlap():
|
||||
bad = '[meta]\nstruct_name="E"\nsize=0x1000\n' \
|
||||
'[[field]]\noffset=0x10\nname="a"\ntype="int"\nnote=""\n' \
|
||||
'[[field]]\noffset=0x12\nname="b"\ntype="int"\nnote=""\n' # 0x10+4 > 0x12 -> overlap
|
||||
check(any("overlap" in e.lower() for e in lint(load(bad))), "lint flags overlapping fields")
|
||||
|
||||
def test_lint_catches_oob_and_dupname():
|
||||
bad = '[meta]\nstruct_name="E"\nsize=0x14\n' \
|
||||
'[[field]]\noffset=0x10\nname="a"\ntype="int"\nnote=""\n' \
|
||||
'[[field]]\noffset=0x40\nname="a"\ntype="int"\nnote=""\n' # 0x40 > size AND dup name
|
||||
errs = lint(load(bad))
|
||||
check(any("out of bounds" in e.lower() for e in errs) and any("duplicate" in e.lower() for e in errs),
|
||||
"lint flags out-of-bounds offset and duplicate name")
|
||||
|
||||
def main():
|
||||
test_load_and_emit(); test_lint_clean(); test_lint_catches_overlap(); test_lint_catches_oob_and_dupname()
|
||||
print("FAILURES:", len(FAILS)); return 1 if FAILS else 0
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run tests to verify they fail**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/test_engine_ctx.py`
|
||||
Expected: FAIL — `ModuleNotFoundError: No module named 'engine_ctx_build'`.
|
||||
|
||||
- [ ] **Step 4: Implement `tools/engine_ctx_build.py`**
|
||||
|
||||
```python
|
||||
"""Build the EngineCtx struct artifacts from vm-map/engine-ctx.toml (single source of truth).
|
||||
|
||||
py -3.11 -X utf8 tools/engine_ctx_build.py --build # -> build/engine-ctx.json + docs/engine-ctx-reference.md
|
||||
py -3.11 -X utf8 tools/engine_ctx_build.py --lint # checks only
|
||||
|
||||
Apply to the Ghidra /v2 image via run_script_inline (see docs/superpowers/plans/2026-07-09-engine-ctx-struct.md).
|
||||
"""
|
||||
import json, sys, tomllib
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
TYPE_SIZES = {"int": 4, "uint": 4, "void*": 4}
|
||||
|
||||
|
||||
def load(toml_text):
|
||||
d = tomllib.loads(toml_text)
|
||||
fields = [{"offset": int(f["offset"]), "name": f["name"], "type": f["type"], "note": f.get("note", "")}
|
||||
for f in d.get("field", [])]
|
||||
fields.sort(key=lambda f: f["offset"])
|
||||
return {"meta": d["meta"], "fields": fields}
|
||||
|
||||
|
||||
def lint(model):
|
||||
errs, size = [], int(model["meta"]["size"])
|
||||
seen_names, prev = {}, None
|
||||
for f in model["fields"]:
|
||||
sz = TYPE_SIZES.get(f["type"])
|
||||
if sz is None:
|
||||
errs.append(f"unknown type {f['type']!r} for {f['name']}")
|
||||
sz = 4
|
||||
if f["offset"] + sz > size:
|
||||
errs.append(f"field {f['name']} @0x{f['offset']:x} out of bounds (size 0x{size:x})")
|
||||
if f["name"] in seen_names:
|
||||
errs.append(f"duplicate name {f['name']!r}")
|
||||
seen_names[f["name"]] = True
|
||||
if prev is not None and f["offset"] < prev["end"]:
|
||||
errs.append(f"overlap: {f['name']} @0x{f['offset']:x} into {prev['name']} (ends 0x{prev['end']:x})")
|
||||
prev = {"name": f["name"], "end": f["offset"] + sz}
|
||||
return errs
|
||||
|
||||
|
||||
def emit_json(model):
|
||||
return {"meta": model["meta"],
|
||||
"fields": {hex(f["offset"]): {"name": f["name"], "type": f["type"]} for f in model["fields"]}}
|
||||
|
||||
|
||||
def emit_reference_md(model):
|
||||
lines = ["# EngineCtx field reference", "",
|
||||
"> Generated from `vm-map/engine-ctx.toml` by `tools/engine_ctx_build.py --build`. Do not edit.",
|
||||
"", f"Struct `{model['meta']['struct_name']}`, size `0x{int(model['meta']['size']):x}`.", "",
|
||||
"| offset | name | type | note |", "|---|---|---|---|"]
|
||||
for f in model["fields"]:
|
||||
lines.append(f"| `0x{f['offset']:x}` | `{f['name']}` | `{f['type']}` | {f['note']} |")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def main():
|
||||
text = (REPO / "vm-map" / "engine-ctx.toml").read_text(encoding="utf-8")
|
||||
model = load(text)
|
||||
errs = lint(model)
|
||||
if errs:
|
||||
print("LINT ERRORS:")
|
||||
for e in errs:
|
||||
print(" " + e)
|
||||
return 1
|
||||
if "--lint" in sys.argv[1:]:
|
||||
print(f"lint clean: {len(model['fields'])} fields")
|
||||
return 0
|
||||
if "--build" in sys.argv[1:]:
|
||||
(REPO / "build" / "engine-ctx.json").write_text(json.dumps(emit_json(model), indent=2) + "\n", encoding="utf-8")
|
||||
(REPO / "docs" / "engine-ctx-reference.md").write_text(emit_reference_md(model), encoding="utf-8")
|
||||
print(f"built {len(model['fields'])} fields -> build/engine-ctx.json + docs/engine-ctx-reference.md")
|
||||
return 0
|
||||
print(__doc__)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run tests to verify they pass**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/test_engine_ctx.py`
|
||||
Expected: PASS (4 tests, `FAILURES: 0`).
|
||||
|
||||
- [ ] **Step 6: Build the real artifacts and lint-check the seed**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/engine_ctx_build.py --build`
|
||||
Expected: `built 35 fields -> build/engine-ctx.json + docs/engine-ctx-reference.md`, no lint errors. Eyeball `docs/engine-ctx-reference.md` for the field table.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add vm-map/engine-ctx.toml tools/engine_ctx_build.py tools/test_engine_ctx.py docs/engine-ctx-reference.md
|
||||
git commit -m "re: EngineCtx field registry (engine-ctx.toml) + builder"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Apply the struct + retype handlers in Ghidra, validate
|
||||
|
||||
**Precondition:** `build/engine-ctx.json` + `build/op-handler-map.json` exist; active program `/v2`.
|
||||
|
||||
**Files:** none in-repo — driven via `run_script_inline` (Java).
|
||||
|
||||
- [ ] **Step 1: Confirm the correct program + capture BEFORE snippet**
|
||||
|
||||
Via MCP `get_current_program_info` — assert base `0x400000`, 4400+ fns (else `open_program`/`switch_program /v2/range_00400000.bin`). Then `decompile_function 0x44cff0` (`sleep_timer_arm`) and note the current `(*DAT_...)` / `param_1 + 0x5f304` rendering for the before/after.
|
||||
Expected: confirmed program + a BEFORE snippet showing raw offsets.
|
||||
|
||||
- [ ] **Step 2: Create the `EngineCtx` struct from `build/engine-ctx.json`**
|
||||
|
||||
`run_script_inline` (Java): read the JSON; get/create a `StructureDataType("EngineCtx", size)` in the program's data type manager; for each `offset → {name,type}` map `type` to a Ghidra `DataType` (`int`/`uint` → `IntegerDataType`/`UnsignedIntegerDataType`; `void*` → `PointerDataType`) and `struct.replaceAtOffset(offset, dt, dt.getLength(), name, note)`; add via `dataTypeManager.addDataType(struct, REPLACE_HANDLER)`. One transaction. Print field count placed.
|
||||
|
||||
```java
|
||||
import java.nio.file.*; import ghidra.program.model.data.*;
|
||||
String txt = new String(Files.readAllBytes(Paths.get("S:/Game Hacking/Eushully/Himegari/age-reimpl/build/engine-ctx.json")));
|
||||
// parse meta.size + each "0xoff":{"name":..,"type":..} with a regex; build the struct.
|
||||
// map type: "void*" -> new PointerDataType(); "uint" -> UnsignedIntegerDataType.dataType; else IntegerDataType.dataType
|
||||
// struct.replaceAtOffset(off, dt, 4, name, note);
|
||||
```
|
||||
Expected: `EngineCtx` struct created with 35 fields.
|
||||
|
||||
- [ ] **Step 3: Retype dispatch-handler `this` to `EngineCtx *`**
|
||||
|
||||
`run_script_inline` (Java): read `build/op-handler-map.json`; resolve `EngineCtx *` (`new PointerDataType(struct)`); for each handler VA, get the `Function`; if it has a `this`/first param whose usage is `ctx` (thiscall or first param), set that param's type to `EngineCtx *` (`func.getParameter(0).setDataType(ptr, SourceType.USER_DEFINED)`, or `func.setCallingConvention("__thiscall")` + this-type where needed). Count retyped; collect + print the VAs skipped (no first param / not ctx-shaped) for review. One transaction; `save_program`.
|
||||
Expected: most handlers retyped; a small skip list.
|
||||
|
||||
- [ ] **Step 4: Validate — AFTER snippet**
|
||||
|
||||
`decompile_function 0x44cff0` (`sleep_timer_arm`) and `0x42a0b0` (`gfx_op_0x215_query_source_slot`).
|
||||
Expected: `sleep_timer_arm` renders `ctx->sleep_timer` (was `param_1 + 0x5f304`); the 0x215 handler renders `ctx->cur_ctx_index` / `ctx->cmd_type_table` at `0x53d14`/`0x53d88`. Record before/after. If fields don't render, diagnose the this-type application (Step 3) before proceeding.
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Docs + memory + close
|
||||
|
||||
**Files:**
|
||||
- Modify: `CLAUDE.md` (canonical-documents map + single-source-of-truth table: add the `engine-ctx.toml` row)
|
||||
- Modify: `docs/tools-reference.md` (add `engine_ctx_build.py`)
|
||||
- Modify: `docs/engine-re.md` (link `docs/engine-ctx-reference.md` from the ctx-offsets discussion)
|
||||
- Modify: `~/.claude/…/memory/himegari-port-status.md` (milestone)
|
||||
|
||||
- [ ] **Step 1: Update CLAUDE.md**
|
||||
|
||||
Add to the canonical-documents map: `| Engine ctx struct fields | age-reimpl/vm-map/engine-ctx.toml (generated → build/engine-ctx.json, docs/engine-ctx-reference.md) |`. Add to the single-source table: `| vm-map/engine-ctx.toml | py -3.11 -X utf8 tools/engine_ctx_build.py --build | build/engine-ctx.json, docs/engine-ctx-reference.md |`.
|
||||
|
||||
- [ ] **Step 2: Update tools-reference.md + engine-re.md**
|
||||
|
||||
`tools-reference.md`: add an `engine_ctx_build.py` row (source → JSON+doc; `--build`/`--lint`; applied to Ghidra via run_script_inline). `engine-re.md`: in the dispatch/ctx-offsets area, add a line pointing to `docs/engine-ctx-reference.md` as the field map, noting the struct is applied to the `/v2` image.
|
||||
|
||||
- [ ] **Step 3: Update the status memory**
|
||||
|
||||
Record: EngineCtx struct DONE — N fields, applied to /v2 (M handlers retyped), before/after validated; source `engine-ctx.toml`; grows incrementally.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add CLAUDE.md docs/tools-reference.md docs/engine-re.md
|
||||
git commit -m "re: apply EngineCtx struct to /v2 handlers; doc + canonical-map wiring"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
**Spec coverage:** source `engine-ctx.toml` → Task 1 Step 1; builder + lint + JSON + doc → Task 1 Steps 4/6; struct creation → Task 2 Step 2; handler `this` retype → Task 2 Step 3; validation decompiles → Task 2 Step 4; CLAUDE.md/tools-reference/memory → Task 3. Covered.
|
||||
|
||||
**Placeholder scan:** Task 2 Java bodies are described with the key API calls (`replaceAtOffset`, `setDataType`, `PointerDataType`) rather than full source — intentional: the exact DataTypeManager calls are pinned at implementation against the live API (same as Task A's iterative script bring-up), and the logic is fully specified. No TBDs in the Python (Task 1 is complete code).
|
||||
|
||||
**Type consistency:** `load` returns `{"meta","fields":[{offset,name,type,note}]}`; `lint`/`emit_json`/`emit_reference_md` all consume that shape; `emit_json` outputs `{meta, fields:{hex:{name,type}}}` which the Task 2 Step 2 Ghidra parser reads. `TYPE_SIZES` keys (`int`/`uint`/`void*`) match the toml `type` values and the Task 2 Ghidra type mapping. Consistent.
|
||||
112
docs/superpowers/specs/2026-07-09-engine-ctx-struct-design.md
Normal file
112
docs/superpowers/specs/2026-07-09-engine-ctx-struct-design.md
Normal file
@@ -0,0 +1,112 @@
|
||||
# EngineCtx struct — typing the engine context — design
|
||||
|
||||
**Date:** 2026-07-09
|
||||
**Status:** implemented 2026-07-09 (`3826064`, `0ecd776`)
|
||||
**Home in the canonical map:** a NEW canonical source (`vm-map/engine-ctx.toml`) + a generated
|
||||
reference (`docs/engine-ctx-reference.md`); add a row to the CLAUDE.md canonical-documents map and the
|
||||
single-source-of-truth table. RE narrative stays in `docs/engine-re.md` (which already documents the
|
||||
offsets); this struct is the machine-readable + applied form of that knowledge.
|
||||
|
||||
## Motivation
|
||||
|
||||
The recurring RE friction that survives the handler-labeling and import-map work is **re-decoding the
|
||||
same `ctx` (esi) offsets by hand** — every handler RE means reading `*(int*)(param_1 + 0x53d14)`,
|
||||
`+0x53d88`, `+0x9b24c`, `+0x408`, `+0x52bd4`, `+0x5f304`, `+0xa0ce4`… and recalling what each means. We
|
||||
have already documented ~30 of these in `docs/engine-re.md`. Defining them **once** as a Ghidra struct
|
||||
and applying it makes every handler decompile with named fields (`ctx->cur_gfx_idx`), front-loading the
|
||||
"what's this offset" lookup so it never recurs. This is the same shape of win as Task A (handler
|
||||
labeling): a one-time pass that pays out on every future look at the decomp.
|
||||
|
||||
Scope note: this is strictly the **engine context** struct (`esi`/thiscall `this`). The VM **global bank**
|
||||
`G[…]` is a *separate* address space (not `ctx` offsets) and is out of scope here (it has its own map,
|
||||
`vm-map/globals.toml`).
|
||||
|
||||
## Architecture
|
||||
|
||||
Single source of truth → generated machine view + doc → applied to the Ghidra image. Mirrors the
|
||||
`opcodes.toml`/`globals.toml` pattern exactly.
|
||||
|
||||
### 1. `vm-map/engine-ctx.toml` (canonical source — hand-edited)
|
||||
|
||||
```toml
|
||||
[meta]
|
||||
struct_name = "EngineCtx"
|
||||
size = 0xa1000 # >= max field end; sparse (gaps = undefined). Grows as needed.
|
||||
|
||||
[[field]]
|
||||
offset = 0x53d14
|
||||
name = "cur_ctx_index"
|
||||
type = "uint"
|
||||
note = "current gfx-object / script-context index (curCtx); indexes the 0x78-byte context records"
|
||||
source = "native-RE"
|
||||
confidence = "high"
|
||||
# … one [[field]] per documented offset …
|
||||
```
|
||||
|
||||
Seeded with the ~30 fields already documented in `engine-re.md`, e.g.: `0x408` retained gfx-object map
|
||||
(`std::map`; geometry/draw get-or-create, `0x215` queries obj+4 source slot, `0x1f7` erases), `0x40c`
|
||||
SYS4INI record count, `0x410` archive-name table, `0x414` SYS4INI 80-byte record
|
||||
base, `0x14d54` obj-pointer table, `0x14f45` script-frame index, `0x46d14` stride-0x14 query table,
|
||||
`0x51b78/0x51b7c` anim clock elapsed/duration, `0x52bd4` surfaces[1000], `0x53d14` cur-ctx-index,
|
||||
`0x53d28/0x53d2c` frame codebase/PC, `0x53d60` context-record base (stride 0x78), `0x53d64` gfx
|
||||
object-record array, `0x53d88` per-object cmd-type table (stride 0x78), `0x55120/0x55124/0x5512c`
|
||||
anti-tamper checksum, `0x55248/0x552e8` per-frame return stack, `0x5f304` sleep-timer object, `0x6da88/
|
||||
0x6db28` coroutine yield-state, `0x6dbc8/0x6dbcc` coroutine resume-state, `0x9b24c` dispatch handler
|
||||
table[0x400], `0xa0cc0/0xa0cc4/0xa0cc8` screen w/h/bpp, `0xa0ce4` run-state flags, `0xb558/0xb560` gfx
|
||||
dirty flags.
|
||||
|
||||
**Types kept simple:** `int`/`uint`/`void*`; arrays where clearly arrays (`dispatch[0x400]` as
|
||||
`void*[0x400]`, `surfaces[1000]`); a field may be a plain scalar even if it's the head of a table (the
|
||||
struct only needs the offset named — the array modelling is optional polish).
|
||||
|
||||
### 2. `tools/engine_ctx_build.py` (generator + linter)
|
||||
|
||||
- `--build` → `build/engine-ctx.json` (`{ "0x53d14": {"name": "...", "type": "uint"}, ... }` + meta) and
|
||||
regenerates `docs/engine-ctx-reference.md` (offset / name / type / note table, grouped).
|
||||
- `--lint` → fail on: overlapping fields (offset+size collisions), a field whose end exceeds `meta.size`,
|
||||
an unknown type, or a duplicate name. Run in `--build`.
|
||||
- Model + lint in the tool (or a small `engine_ctx_model.py` if it grows); unit-tested (`test_engine_ctx.py`).
|
||||
|
||||
### 3. Ghidra apply (`run_script_inline`, Java — the Task A pattern)
|
||||
|
||||
Confirm the active program is `/v2/range_00400000.bin` (base `0x400000`, 4400+ fns — the two-program
|
||||
gotcha). Then:
|
||||
1. Read `build/engine-ctx.json`; create (or replace) a struct `EngineCtx` of `meta.size` bytes with each
|
||||
field placed at its offset (`struct.replaceAtOffset` / `insertAtOffset` per the Ghidra API), gaps left
|
||||
undefined.
|
||||
2. Read `build/op-handler-map.json`; for each dispatch handler, set its `this`/first-parameter type to
|
||||
`EngineCtx *` (via the thiscall `this` type, or parameter 0 where the convention lacks a `this`).
|
||||
Skip functions that don't take `ctx` as first arg (report them).
|
||||
3. One transaction; `save_program`.
|
||||
|
||||
## Data flow
|
||||
|
||||
`vm-map/engine-ctx.toml` ──`--build`──▶ `build/engine-ctx.json` + `docs/engine-ctx-reference.md`
|
||||
`build/engine-ctx.json` + `build/op-handler-map.json` ──`run_script_inline`──▶ `EngineCtx` struct +
|
||||
retyped handler `this` on `/v2` ──▶ handlers decompile `ctx->field`.
|
||||
|
||||
## Validation
|
||||
|
||||
- `--lint` passes (no overlaps/OOB/dup).
|
||||
- Decompile **`sleep_timer_arm`** — expect `ctx->sleep_timer` (offset `0x5f304`) instead of
|
||||
`param_1 + 0x5f304`; decompile **`gfx_op_0x215_query_source_slot`** — expect `ctx->cur_ctx_index` /
|
||||
`ctx->cmd_type_table[...]` style rendering at `0x53d14`/`0x53d88`. Record a before/after snippet.
|
||||
- Handler count retyped reported; functions skipped (no `ctx` first-arg) listed for review.
|
||||
- `save_program` succeeds.
|
||||
|
||||
## Scope & boundaries
|
||||
|
||||
- **In:** the ~30 documented `ctx` fields; struct creation; retyping dispatch-handler `this` params.
|
||||
- **Out:** the VM global bank (`globals.toml`); worker functions (`FUN_0047xxxx`) — retype later as we
|
||||
touch them; exhaustive field discovery (grows incrementally via the toml, not a big-bang sweep);
|
||||
nested sub-structs (e.g. modelling the 0x78-byte context record as its own type — a later refinement).
|
||||
- **Regenerable:** `build/engine-ctx.json` + `docs/engine-ctx-reference.md` are generated — never
|
||||
hand-edit; edit `engine-ctx.toml` and re-run `--build`.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `vm-map/engine-ctx.toml` holds the ~30 seed fields; `--build` regenerates the JSON + doc; `--lint` clean.
|
||||
- `EngineCtx` struct exists in the `/v2` image; dispatch-handler `this` params retyped (count reported).
|
||||
- The two validation decompiles render named `ctx->…` fields; `save_program` OK.
|
||||
- CLAUDE.md canonical-map + single-source table gain the `engine-ctx.toml` row; `tools-reference.md`
|
||||
gains `engine_ctx_build.py`; status memory records the milestone.
|
||||
Reference in New Issue
Block a user