From 5cd5abb00a9eb267ede5f2149d4514a1b8f27f55 Mon Sep 17 00:00:00 2001 From: gamer147 Date: Thu, 9 Jul 2026 10:38:29 -0400 Subject: [PATCH] re: hot-helper naming (lever #2, pivoted from STL/CRT auto-naming) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Function ID recon = WEAK (0 matches; bundled FidDbs don't cover the VC9 static runtime) and our library workers are already named. Pivoted to naming the hottest documented-but-unnamed helpers by call-count: named 5 (~2400 call sites) in Ghidra — vm_operand_fetch@0x41b940 (1021 refs), vm_operand_write@0x425fb0, vm_operand_lvalue@0x415f30, __security_check_cookie@0x54f981 (692), operator_new@0x5502be (533). Validated: gfx_op_0x215 reads near source-level. Spec/plan + engine-re.md. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/engine-re.md | 20 ++- .../plans/2026-07-09-stl-crt-naming.md | 156 ++++++++++++++++++ .../specs/2026-07-09-stl-crt-naming-design.md | 84 ++++++++++ 3 files changed, 257 insertions(+), 3 deletions(-) create mode 100644 docs/superpowers/plans/2026-07-09-stl-crt-naming.md create mode 100644 docs/superpowers/specs/2026-07-09-stl-crt-naming-design.md diff --git a/docs/engine-re.md b/docs/engine-re.md index 02b7299..c445b76 100644 --- a/docs/engine-re.md +++ b/docs/engine-re.md @@ -116,9 +116,23 @@ decode when you reverse one (the generic name is a floor, not a final). > GUI's active program, so a wrong-program script would mutate/measure garbage. **Other confirmed engine-context offsets** (`ctx`/`esi`): `+0x53d14` = current gfx-object index; -`+0x53d88` = per-object cmd-type table (stride `0x78` = 120 bytes); operand-fetch helper = `call -0x41b940` (thiscall, `ecx=ctx`, arg = operand index → returns the operand value); `FUN_00415f30(i)` = -a companion operand accessor. +`+0x53d88` = per-object cmd-type table (stride `0x78` = 120 bytes); operand-fetch helper = +**`vm_operand_fetch`@`0x41b940`** (thiscall, `ecx=ctx`, arg = operand index → returns the operand value); +**`vm_operand_write`@`0x425fb0`** = the counterpart store; **`vm_operand_lvalue`@`0x415f30`** = the +companion index/pointer accessor. + +**Hot-helper naming pass (2026-07-09, lever #2).** Ghidra's Function ID analyzer names **0** functions on +this image (the bundled FidDbs don't cover the VC9/VS2008 static runtime; ~3,660 of 4,428 funcs stay +`FUN_`), and the library workers we actually touch were already hand-named (`gfx_registry_map_find`, +etc.). So "STL/CRT auto-naming" had little to add — but the recon (rank unnamed funcs by call-count) +surfaced the real win: ubiquitous **documented-but-unnamed helpers**. Named the top 5 (~2,400 call +sites): `vm_operand_fetch`@`0x41b940` (1021 refs), `vm_operand_write`@`0x425fb0` (188), +`vm_operand_lvalue`@`0x415f30`, plus two CRT primitives identified by behavior — +`__security_check_cookie`@`0x54f981` (692; compares `__security_cookie`=`DAT_005c28c0`) and +`operator_new`@`0x5502be` (533; `_malloc`+`__callnewh`+throw `bad_alloc`). ⇒ every handler now reads +e.g. `vm_operand_fetch(2)` not `FUN_0041b940(2)`. No FidDb generation (out of scope, low ROI). Rename +hot unnamed funcs by call-count when de-noising further; there is no registry file for these — the +Ghidra name is the record. **These `ctx` offsets are now a typed struct (2026-07-09).** The canonical field map is `vm-map/engine-ctx.toml` → generated `docs/engine-ctx-reference.md`; a `run_script_inline` pass created diff --git a/docs/superpowers/plans/2026-07-09-stl-crt-naming.md b/docs/superpowers/plans/2026-07-09-stl-crt-naming.md new file mode 100644 index 0000000..496e1d2 --- /dev/null +++ b/docs/superpowers/plans/2026-07-09-stl-crt-naming.md @@ -0,0 +1,156 @@ +# STL/CRT Naming — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** De-noise the library layer of the `/v2` image — take whatever Ghidra Function ID names for free, then curate the specific CRT/STL functions we've already RE'd so handlers stop burying real logic in unnamed `FUN_` calls. + +**Architecture:** Recon-gated. Task 1 forces a Function ID pass and measures; if it names a meaningful number, that's the win. If weak (expected), Task 2 builds a small canonical `vm-map/lib-functions.toml` (same pattern as `engine-ctx.toml`) of known library functions and applies it. Deliberately light — no custom FidDb generation. + +**Tech Stack:** ghidra-mcp `run_analysis` + `run_script_inline` (Java; `GHIDRA_MCP_ALLOW_SCRIPTS=1`); Python 3.11 (`tomllib`) for the registry builder. + +**Spec:** `docs/superpowers/specs/2026-07-09-stl-crt-naming-design.md`. + +## Global Constraints + +- Confirm the active program is `/v2/range_00400000.bin` (base `0x400000`, ~4415 fns) before any script — the two-program gotcha. End image mutation with `save_program`. +- **Never clobber a `USER_DEFINED` symbol** (our handler/worker names). Function ID applies `ANALYSIS`-source labels (won't clobber); the curated rename skips any `USER_DEFINED`. +- Baseline (measured this session): **3,660 of 4,415 functions default-named** (`FUN_`/`LAB_`). +- `build/` is disposable/gitignored; never hand-edit generated files. +- Keep it light: **no custom VC9/MSVCR90 FidDb generation** (out of scope). + +--- + +## Task 1: Function ID recon (THE GATE) + +**Files:** none in-repo — driven via MCP. + +- [ ] **Step 1: Confirm program + record baseline** + +MCP `get_current_program_info` → assert base `0x400000`. Then a counting `run_script_inline` (Java): iterate functions, count `default (FUN_/LAB_)` vs `user`. Record the baseline (expected ~3660 default). +Expected: baseline default count confirmed. + +- [ ] **Step 2: Force a Function ID pass with "Always Apply FID Labels"** + +`run_script_inline` (Java): enable the option, then re-run analysis so Function ID re-evaluates: +```java +import ghidra.app.plugin.core.analysis.AutoAnalysisManager; +ghidra.framework.options.Options opts = currentProgram.getOptions("Analyzers"); +opts.setBoolean("Function ID.Always Apply FID Labels", true); +AutoAnalysisManager mgr = AutoAnalysisManager.getAnalysisManager(currentProgram); +int tx = currentProgram.startTransaction("re-run Function ID"); +try { mgr.reAnalyzeAll(currentProgram.getMemory()); mgr.startAnalysis(monitor); } +finally { currentProgram.endTransaction(tx, true); } +println("re-analysis triggered"); +``` +If `reAnalyzeAll`/`startAnalysis` is impractical here (timeout / no monitor), fall back to MCP `run_analysis` after setting the option. Either way, allow it to finish. +Expected: analysis completes (may take a minute). + +- [ ] **Step 3: Measure the delta + gate** + +Re-run the Step 1 counting script. Compute `newlyNamed = baselineDefault - afterDefault`. Spot-check: does `FUN_0047f280` (our `std::map::find`) or any obvious CRT function now carry a library name (`get_function_by_address`)? +- **GATE — STRONG (`newlyNamed >= 100`):** Function ID is the win. `save_program`. Skip Task 2; go to Task 3 (document the count). +- **GATE — WEAK (`newlyNamed` small / ~0):** the bundled FidDbs don't cover this runtime. Revert the option if desired, and proceed to Task 2. Record the weak result. +Expected: an explicit STRONG/WEAK verdict with the number. + +- [ ] **Step 4: If STRONG, save + commit note** + +`save_program`. (No repo files change; the doc commit is Task 3.) If WEAK, nothing to save from this task. + +--- + +## Task 2: Curated library-function registry (ONLY if Task 1 is WEAK) + +**Files:** +- Create: `vm-map/lib-functions.toml` (canonical source) +- Create: `tools/lib_functions_build.py` (model + lint + emit; mirrors `engine_ctx_build.py`) +- Create: `tools/test_lib_functions.py` (unit tests) +- Create: `build/lib-functions.json`, `docs/lib-functions-reference.md` (generated) + +**Interfaces:** +- `load(toml_text) -> {"funcs":[{"address":int,"name","note"}...]}`; `lint(model) -> list[str]` (dup address, dup name, bad name chars); `emit_json(model) -> {hex_addr: name}`. + +- [ ] **Step 1: Seed `vm-map/lib-functions.toml` with the known library functions** + +Include the library functions we've identified in RE (grep `engine-re.md` for `std::map`, `FUN_0047f280`, `FUN_0042cf70`, hash insert/find, `operator new`, etc.). Start conservative — only addresses we're confident about: +```toml +# vm-map/lib-functions.toml -- CANONICAL registry of identified statically-linked library functions. +# Generated: build/lib-functions.json + docs/lib-functions-reference.md via tools/lib_functions_build.py --build. +# Applied to /v2 via run_script_inline (rename; never clobbers USER_DEFINED). Grows as we identify more. +[[func]] +address = 0x47f280 +name = "std_map_find" +note = "std::map::find over the gfx object registry (op 0x215 handler calls it; returns value or 0xffffffff)" +source = "native-RE" +confidence = "high" +[[func]] +address = 0x42cf70 +name = "gfx_registry_hash_insert" +note = "open-addressing hash insert into the op-0x215 query registry (op 0x1a2 handler calls it)" +source = "native-RE" +confidence = "high" +``` +*(Add more only where confident — verify each address in `engine-re.md`/the image first. A small seed is fine; the registry's value is that it grows.)* + +- [ ] **Step 2: Write failing tests `tools/test_lib_functions.py`** + +Mirror `test_engine_ctx.py`: a `load`/`emit_json` shape test and a `lint` dup-address + dup-name test (plain runner, no pytest). Run → FAIL (`ModuleNotFoundError: lib_functions_build`). + +- [ ] **Step 3: Implement `tools/lib_functions_build.py`** + +Mirror `engine_ctx_build.py`: `load` (tomllib → `{"funcs":[...]}`), `lint` (duplicate address, duplicate name, name not `[A-Za-z0-9_]+`), `emit_json` (`{hex_addr: name}`), `emit_reference_md` (address/name/note table), `--build`/`--lint` CLI writing `build/lib-functions.json` + `docs/lib-functions-reference.md`. Run tests → PASS. + +- [ ] **Step 4: Build + lint** + +Run: `py -3.11 -X utf8 tools/lib_functions_build.py --build` +Expected: `built N funcs -> build/lib-functions.json + docs/lib-functions-reference.md`, lint clean. + +- [ ] **Step 5: Apply to `/v2` (rename, skip USER_DEFINED)** + +`run_script_inline` (Java): read `build/lib-functions.json`; for each `addr → name`, get the function; if its symbol is `USER_DEFINED` skip (already named), else `rename` to `name` (`SourceType.USER_DEFINED`). Count renamed/skipped. One transaction; `save_program`. +Expected: N renamed (or skipped if we'd already named them), 0 clobbers. + +- [ ] **Step 6: Validate + commit** + +`decompile_function 0x42a0b0` (`gfx_op_0x215_register_query`) — its registry-find call should read `std_map_find(...)`. +```bash +git add vm-map/lib-functions.toml tools/lib_functions_build.py tools/test_lib_functions.py docs/lib-functions-reference.md +git commit -m "re: curated library-function registry (lib-functions.toml) + apply" +``` + +--- + +## Task 3: Docs + memory + close + +**Files:** +- Modify: `docs/engine-re.md` (record the outcome — Function ID result, and/or the curated registry) +- Modify (iff Task 2 ran): `CLAUDE.md` (canonical map + single-source table: `lib-functions.toml` row), `docs/tools-reference.md` (`lib_functions_build.py`) +- Modify: `~/.claude/…/memory/himegari-port-status.md` (milestone; lever #2 done, #3 next) + +- [ ] **Step 1: Update engine-re.md** + +Add a short note: Function ID recon result (STRONG count / WEAK), and — if built — the curated `lib-functions.toml` registry as the home for identified library functions. Link `docs/lib-functions-reference.md` if it exists. + +- [ ] **Step 2: Update CLAUDE.md + tools-reference.md (iff a registry was built)** + +CLAUDE.md canonical map row: `| Identified library (CRT/STL) functions | age-reimpl/vm-map/lib-functions.toml (generated → build/lib-functions.json, docs/lib-functions-reference.md) |`. Single-source table row + `engine_ctx`-style update trigger. tools-reference.md: `lib_functions_build.py` row. + +- [ ] **Step 3: Update the status memory** + +Record: STL/CRT naming DONE — Function ID result (N named or weak); curated registry seeded with M funcs (if built); lever #2 of 3 complete, #3 (differential oracle) next. + +- [ ] **Step 4: Commit docs** + +```bash +git add docs/engine-re.md docs/tools-reference.md +git commit -m "re: record STL/CRT naming outcome (Function ID + curated registry)" +``` + +--- + +## Self-Review + +**Spec coverage:** recon gate → Task 1 (explicit STRONG/WEAK); curated fallback → Task 2 (conditional); no custom FidDb → Global Constraints + Task 2 note; validation spot-checks → Task 1 Step 3 + Task 2 Step 6; docs/memory → Task 3. Covered. + +**Placeholder scan:** Task 2 Steps 2–3 say "mirror `engine_ctx_build.py`/`test_engine_ctx.py`" rather than repeating the full code — justified: those files exist in-repo as the exact template (load/lint/emit + plain-runner tests), and the shapes are specified in the Interfaces block. The toml seed is concrete. Task 1 Step 2 has a fallback path stated (run_analysis) for the uncertain re-analysis API. No hidden TODOs. + +**Type consistency:** `load` → `{"funcs":[{address,name,note}]}` consumed by `lint`/`emit_json`; `emit_json` → `{hex_addr:name}` consumed by the Task 2 Step 5 Ghidra rename parser. Matches the `engine_ctx_build.py` contract it mirrors. diff --git a/docs/superpowers/specs/2026-07-09-stl-crt-naming-design.md b/docs/superpowers/specs/2026-07-09-stl-crt-naming-design.md new file mode 100644 index 0000000..973a064 --- /dev/null +++ b/docs/superpowers/specs/2026-07-09-stl-crt-naming-design.md @@ -0,0 +1,84 @@ +# STL/CRT auto-naming — de-noising the library layer — design + +**Date:** 2026-07-09 +**Status:** approved (design), plan pending +**Lever:** #2 of 3 in the RE-front-loading program (after handler-labeling, import-map, ctx-struct; +before the differential oracle). See the status memory. +**Home in the canonical map:** results in `docs/engine-re.md`; if a curated registry is built, it becomes +a NEW canonical source (`vm-map/lib-functions.toml`) with a row in the CLAUDE.md maps. + +## Motivation + +3,660 of 4,415 functions in the `/v2` image are still default-named (`FUN_`/`LAB_`), a large slice of +which is statically-linked CRT/STL. We keep re-recognizing the same library functions by behavior +(`std::map::find` = `FUN_0047f280`, the open-addressing hash insert = `FUN_0042cf70`, `operator new`, +string/`memcpy` ops). Naming them lets the decompile stop burying real engine logic in unnamed library +noise — the last cheap static front-loader before the dynamic oracle. This is explicitly a *de-noising +polish*, lower-leverage than the first three levers, so it is scoped to stay light. + +## Approach (recon-first, decided in brainstorming) + +Ghidra's **Function ID** analyzer is enabled but has named little so far (the bundled FidDbs may not +cover this VC9/VS2008-era static runtime). So the recon decides the shape. + +### Task 1 — Function ID recon (THE GATE) + +Re-run the Function ID analyzer against `/v2`, including a pass with **"Always Apply FID Labels"** +enabled, and measure the delta: how many `FUN_`/`LAB_` functions become named (source `ANALYSIS`/library +names). Record the before (3,660 default) and after counts. + +- **GATE — STRONG** (names a meaningful number, say ≥ 100 new library names): Function ID *is* the win. + Keep the labels, `save_program`, document the count, done. No curated registry needed. +- **GATE — WEAK** (~0 new): the bundled FidDbs don't match this runtime → fall back to the curated + hot-set (Task 2). Do **not** invest in generating a custom VC9 FidDb (out of scope — heavy, low return). + +Function ID applies labels with `ANALYSIS` source, which never clobbers our `USER_DEFINED` names — so a +strong result composes cleanly with the handler/ctx work. + +### Task 2 — curated hot-set (ONLY if Task 1 is WEAK) + +A small canonical registry, same pattern as `engine-ctx.toml`: + +`vm-map/lib-functions.toml` — `[[func]]` entries `{ address, name, note, source, confidence }` for the +library functions we have already identified in our RE (seeded from `engine-re.md`): e.g. +`0x47f280 → std_map_find`, `0x42cf70 → hashmap_insert`, plus `operator new`/`delete`, `memcpy`, and any +string ops we've pinned. `tools/lib_functions_build.py` (`--build`/`--lint`) → `build/lib-functions.json` ++ `docs/lib-functions-reference.md`; a `run_script_inline` pass renames each function (skip if already +`USER_DEFINED`). Grows one `[[func]]` at a time as we identify more — the durable home for "this worker +is a known library function." **Small by design** (a handful to low-dozens), not a comprehensive sweep. + +## Validation + +- Task 1: report the newly-named count; spot-check 2–3 (e.g. does `FUN_0047f280` now carry a + `std::map`-family name? does a known `operator new` site read named?). +- Task 2 (if taken): decompile `gfx_op_0x215_register_query` — its `gfx_registry_map_find(...)` / + `FUN_0047f280` call should read as the curated `std_map_find`; `--lint` clean; 0 clobbers. +- `save_program` succeeds. + +## Scope & boundaries + +- **In:** run Function ID + apply; (conditionally) a small curated registry of known library functions. +- **Out:** generating a custom VC9/MSVCR90 FidDb (heavy, low marginal return); BSim; comprehensively + naming all 3,660 (we take what Function ID gives + what we've RE'd, not a big-bang). +- **Regenerable:** `build/lib-functions.json` + `docs/lib-functions-reference.md` are generated. + +## Acceptance criteria + +- Task 1 recon run; before/after default-function counts recorded; gate verdict explicit. +- If STRONG: labels applied, saved, count documented. If WEAK: `vm-map/lib-functions.toml` seeded with the + known hot functions, built (`--lint` clean), applied to `/v2` (0 clobbers), saved. +- ≥2 validation spot-checks read as named library calls. +- `docs/engine-re.md` records the outcome; `tools-reference.md` + CLAUDE.md maps updated iff a registry + was built; status memory records the milestone (and marks lever #2 done, #3 next). + +## Outcome (2026-07-09) — PIVOTED + +Task 1 recon: Function ID = **WEAK** (0 new names; bundled FidDbs don't cover the VC9 runtime). Also +discovered the library functions we actually touch are already `USER_DEFINED` (annotate-as-you-go). So +the curated CRT/STL registry (Task 2 as specced) would have renamed already-named functions — near-zero +value. **Pivoted** (user-approved) to naming the *hot documented-but-unnamed helpers* the recon surfaced +by call-count: `vm_operand_fetch`@`0x41b940` (1021 refs), `vm_operand_write`@`0x425fb0`, +`vm_operand_lvalue`@`0x415f30`, `__security_check_cookie`@`0x54f981` (692), `operator_new`@`0x5502be` +(533) — ~2,400 call sites de-noised. **No `lib-functions.toml`/builder built** (the Ghidra name is the +record for these; no tool consumes them), so CLAUDE.md/tools-reference were not touched. Full account: +`docs/engine-re.md` "Hot-helper naming pass".