re: hot-helper naming (lever #2, pivoted from STL/CRT auto-naming)

Function ID recon = WEAK (0 matches; bundled FidDbs don't cover the VC9
static runtime) and our library workers are already named. Pivoted to
naming the hottest documented-but-unnamed helpers by call-count: named 5
(~2400 call sites) in Ghidra — vm_operand_fetch@0x41b940 (1021 refs),
vm_operand_write@0x425fb0, vm_operand_lvalue@0x415f30,
__security_check_cookie@0x54f981 (692), operator_new@0x5502be (533).
Validated: gfx_op_0x215 reads near source-level. Spec/plan + engine-re.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
gamer147
2026-07-09 10:38:29 -04:00
parent 1b151f6e17
commit 5cd5abb00a
3 changed files with 257 additions and 3 deletions

View File

@@ -116,9 +116,23 @@ decode when you reverse one (the generic name is a floor, not a final).
> GUI's active program, so a wrong-program script would mutate/measure garbage. > GUI's active program, so a wrong-program script would mutate/measure garbage.
**Other confirmed engine-context offsets** (`ctx`/`esi`): `+0x53d14` = current gfx-object index; **Other confirmed engine-context offsets** (`ctx`/`esi`): `+0x53d14` = current gfx-object index;
`+0x53d88` = per-object cmd-type table (stride `0x78` = 120 bytes); operand-fetch helper = `call `+0x53d88` = per-object cmd-type table (stride `0x78` = 120 bytes); operand-fetch helper =
0x41b940` (thiscall, `ecx=ctx`, arg = operand index → returns the operand value); `FUN_00415f30(i)` = **`vm_operand_fetch`@`0x41b940`** (thiscall, `ecx=ctx`, arg = operand index → returns the operand value);
a companion operand accessor. **`vm_operand_write`@`0x425fb0`** = the counterpart store; **`vm_operand_lvalue`@`0x415f30`** = the
companion index/pointer accessor.
**Hot-helper naming pass (2026-07-09, lever #2).** Ghidra's Function ID analyzer names **0** functions on
this image (the bundled FidDbs don't cover the VC9/VS2008 static runtime; ~3,660 of 4,428 funcs stay
`FUN_`), and the library workers we actually touch were already hand-named (`gfx_registry_map_find`,
etc.). So "STL/CRT auto-naming" had little to add — but the recon (rank unnamed funcs by call-count)
surfaced the real win: ubiquitous **documented-but-unnamed helpers**. Named the top 5 (~2,400 call
sites): `vm_operand_fetch`@`0x41b940` (1021 refs), `vm_operand_write`@`0x425fb0` (188),
`vm_operand_lvalue`@`0x415f30`, plus two CRT primitives identified by behavior —
`__security_check_cookie`@`0x54f981` (692; compares `__security_cookie`=`DAT_005c28c0`) and
`operator_new`@`0x5502be` (533; `_malloc`+`__callnewh`+throw `bad_alloc`). ⇒ every handler now reads
e.g. `vm_operand_fetch(2)` not `FUN_0041b940(2)`. No FidDb generation (out of scope, low ROI). Rename
hot unnamed funcs by call-count when de-noising further; there is no registry file for these — the
Ghidra name is the record.
**These `ctx` offsets are now a typed struct (2026-07-09).** The canonical field map is **These `ctx` offsets are now a typed struct (2026-07-09).** The canonical field map is
`vm-map/engine-ctx.toml` → generated `docs/engine-ctx-reference.md`; a `run_script_inline` pass created `vm-map/engine-ctx.toml` → generated `docs/engine-ctx-reference.md`; a `run_script_inline` pass created

View File

@@ -0,0 +1,156 @@
# STL/CRT Naming — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** De-noise the library layer of the `/v2` image — take whatever Ghidra Function ID names for free, then curate the specific CRT/STL functions we've already RE'd so handlers stop burying real logic in unnamed `FUN_` calls.
**Architecture:** Recon-gated. Task 1 forces a Function ID pass and measures; if it names a meaningful number, that's the win. If weak (expected), Task 2 builds a small canonical `vm-map/lib-functions.toml` (same pattern as `engine-ctx.toml`) of known library functions and applies it. Deliberately light — no custom FidDb generation.
**Tech Stack:** ghidra-mcp `run_analysis` + `run_script_inline` (Java; `GHIDRA_MCP_ALLOW_SCRIPTS=1`); Python 3.11 (`tomllib`) for the registry builder.
**Spec:** `docs/superpowers/specs/2026-07-09-stl-crt-naming-design.md`.
## Global Constraints
- Confirm the active program is `/v2/range_00400000.bin` (base `0x400000`, ~4415 fns) before any script — the two-program gotcha. End image mutation with `save_program`.
- **Never clobber a `USER_DEFINED` symbol** (our handler/worker names). Function ID applies `ANALYSIS`-source labels (won't clobber); the curated rename skips any `USER_DEFINED`.
- Baseline (measured this session): **3,660 of 4,415 functions default-named** (`FUN_`/`LAB_`).
- `build/` is disposable/gitignored; never hand-edit generated files.
- Keep it light: **no custom VC9/MSVCR90 FidDb generation** (out of scope).
---
## Task 1: Function ID recon (THE GATE)
**Files:** none in-repo — driven via MCP.
- [ ] **Step 1: Confirm program + record baseline**
MCP `get_current_program_info` → assert base `0x400000`. Then a counting `run_script_inline` (Java): iterate functions, count `default (FUN_/LAB_)` vs `user`. Record the baseline (expected ~3660 default).
Expected: baseline default count confirmed.
- [ ] **Step 2: Force a Function ID pass with "Always Apply FID Labels"**
`run_script_inline` (Java): enable the option, then re-run analysis so Function ID re-evaluates:
```java
import ghidra.app.plugin.core.analysis.AutoAnalysisManager;
ghidra.framework.options.Options opts = currentProgram.getOptions("Analyzers");
opts.setBoolean("Function ID.Always Apply FID Labels", true);
AutoAnalysisManager mgr = AutoAnalysisManager.getAnalysisManager(currentProgram);
int tx = currentProgram.startTransaction("re-run Function ID");
try { mgr.reAnalyzeAll(currentProgram.getMemory()); mgr.startAnalysis(monitor); }
finally { currentProgram.endTransaction(tx, true); }
println("re-analysis triggered");
```
If `reAnalyzeAll`/`startAnalysis` is impractical here (timeout / no monitor), fall back to MCP `run_analysis` after setting the option. Either way, allow it to finish.
Expected: analysis completes (may take a minute).
- [ ] **Step 3: Measure the delta + gate**
Re-run the Step 1 counting script. Compute `newlyNamed = baselineDefault - afterDefault`. Spot-check: does `FUN_0047f280` (our `std::map::find`) or any obvious CRT function now carry a library name (`get_function_by_address`)?
- **GATE — STRONG (`newlyNamed >= 100`):** Function ID is the win. `save_program`. Skip Task 2; go to Task 3 (document the count).
- **GATE — WEAK (`newlyNamed` small / ~0):** the bundled FidDbs don't cover this runtime. Revert the option if desired, and proceed to Task 2. Record the weak result.
Expected: an explicit STRONG/WEAK verdict with the number.
- [ ] **Step 4: If STRONG, save + commit note**
`save_program`. (No repo files change; the doc commit is Task 3.) If WEAK, nothing to save from this task.
---
## Task 2: Curated library-function registry (ONLY if Task 1 is WEAK)
**Files:**
- Create: `vm-map/lib-functions.toml` (canonical source)
- Create: `tools/lib_functions_build.py` (model + lint + emit; mirrors `engine_ctx_build.py`)
- Create: `tools/test_lib_functions.py` (unit tests)
- Create: `build/lib-functions.json`, `docs/lib-functions-reference.md` (generated)
**Interfaces:**
- `load(toml_text) -> {"funcs":[{"address":int,"name","note"}...]}`; `lint(model) -> list[str]` (dup address, dup name, bad name chars); `emit_json(model) -> {hex_addr: name}`.
- [ ] **Step 1: Seed `vm-map/lib-functions.toml` with the known library functions**
Include the library functions we've identified in RE (grep `engine-re.md` for `std::map`, `FUN_0047f280`, `FUN_0042cf70`, hash insert/find, `operator new`, etc.). Start conservative — only addresses we're confident about:
```toml
# vm-map/lib-functions.toml -- CANONICAL registry of identified statically-linked library functions.
# Generated: build/lib-functions.json + docs/lib-functions-reference.md via tools/lib_functions_build.py --build.
# Applied to /v2 via run_script_inline (rename; never clobbers USER_DEFINED). Grows as we identify more.
[[func]]
address = 0x47f280
name = "std_map_find"
note = "std::map::find over the gfx object registry (op 0x215 handler calls it; returns value or 0xffffffff)"
source = "native-RE"
confidence = "high"
[[func]]
address = 0x42cf70
name = "gfx_registry_hash_insert"
note = "open-addressing hash insert into the op-0x215 query registry (op 0x1a2 handler calls it)"
source = "native-RE"
confidence = "high"
```
*(Add more only where confident — verify each address in `engine-re.md`/the image first. A small seed is fine; the registry's value is that it grows.)*
- [ ] **Step 2: Write failing tests `tools/test_lib_functions.py`**
Mirror `test_engine_ctx.py`: a `load`/`emit_json` shape test and a `lint` dup-address + dup-name test (plain runner, no pytest). Run → FAIL (`ModuleNotFoundError: lib_functions_build`).
- [ ] **Step 3: Implement `tools/lib_functions_build.py`**
Mirror `engine_ctx_build.py`: `load` (tomllib → `{"funcs":[...]}`), `lint` (duplicate address, duplicate name, name not `[A-Za-z0-9_]+`), `emit_json` (`{hex_addr: name}`), `emit_reference_md` (address/name/note table), `--build`/`--lint` CLI writing `build/lib-functions.json` + `docs/lib-functions-reference.md`. Run tests → PASS.
- [ ] **Step 4: Build + lint**
Run: `py -3.11 -X utf8 tools/lib_functions_build.py --build`
Expected: `built N funcs -> build/lib-functions.json + docs/lib-functions-reference.md`, lint clean.
- [ ] **Step 5: Apply to `/v2` (rename, skip USER_DEFINED)**
`run_script_inline` (Java): read `build/lib-functions.json`; for each `addr → name`, get the function; if its symbol is `USER_DEFINED` skip (already named), else `rename` to `name` (`SourceType.USER_DEFINED`). Count renamed/skipped. One transaction; `save_program`.
Expected: N renamed (or skipped if we'd already named them), 0 clobbers.
- [ ] **Step 6: Validate + commit**
`decompile_function 0x42a0b0` (`gfx_op_0x215_register_query`) — its registry-find call should read `std_map_find(...)`.
```bash
git add vm-map/lib-functions.toml tools/lib_functions_build.py tools/test_lib_functions.py docs/lib-functions-reference.md
git commit -m "re: curated library-function registry (lib-functions.toml) + apply"
```
---
## Task 3: Docs + memory + close
**Files:**
- Modify: `docs/engine-re.md` (record the outcome — Function ID result, and/or the curated registry)
- Modify (iff Task 2 ran): `CLAUDE.md` (canonical map + single-source table: `lib-functions.toml` row), `docs/tools-reference.md` (`lib_functions_build.py`)
- Modify: `~/.claude/…/memory/himegari-port-status.md` (milestone; lever #2 done, #3 next)
- [ ] **Step 1: Update engine-re.md**
Add a short note: Function ID recon result (STRONG count / WEAK), and — if built — the curated `lib-functions.toml` registry as the home for identified library functions. Link `docs/lib-functions-reference.md` if it exists.
- [ ] **Step 2: Update CLAUDE.md + tools-reference.md (iff a registry was built)**
CLAUDE.md canonical map row: `| Identified library (CRT/STL) functions | age-reimpl/vm-map/lib-functions.toml (generated → build/lib-functions.json, docs/lib-functions-reference.md) |`. Single-source table row + `engine_ctx`-style update trigger. tools-reference.md: `lib_functions_build.py` row.
- [ ] **Step 3: Update the status memory**
Record: STL/CRT naming DONE — Function ID result (N named or weak); curated registry seeded with M funcs (if built); lever #2 of 3 complete, #3 (differential oracle) next.
- [ ] **Step 4: Commit docs**
```bash
git add docs/engine-re.md docs/tools-reference.md
git commit -m "re: record STL/CRT naming outcome (Function ID + curated registry)"
```
---
## Self-Review
**Spec coverage:** recon gate → Task 1 (explicit STRONG/WEAK); curated fallback → Task 2 (conditional); no custom FidDb → Global Constraints + Task 2 note; validation spot-checks → Task 1 Step 3 + Task 2 Step 6; docs/memory → Task 3. Covered.
**Placeholder scan:** Task 2 Steps 23 say "mirror `engine_ctx_build.py`/`test_engine_ctx.py`" rather than repeating the full code — justified: those files exist in-repo as the exact template (load/lint/emit + plain-runner tests), and the shapes are specified in the Interfaces block. The toml seed is concrete. Task 1 Step 2 has a fallback path stated (run_analysis) for the uncertain re-analysis API. No hidden TODOs.
**Type consistency:** `load``{"funcs":[{address,name,note}]}` consumed by `lint`/`emit_json`; `emit_json``{hex_addr:name}` consumed by the Task 2 Step 5 Ghidra rename parser. Matches the `engine_ctx_build.py` contract it mirrors.

View File

@@ -0,0 +1,84 @@
# STL/CRT auto-naming — de-noising the library layer — design
**Date:** 2026-07-09
**Status:** approved (design), plan pending
**Lever:** #2 of 3 in the RE-front-loading program (after handler-labeling, import-map, ctx-struct;
before the differential oracle). See the status memory.
**Home in the canonical map:** results in `docs/engine-re.md`; if a curated registry is built, it becomes
a NEW canonical source (`vm-map/lib-functions.toml`) with a row in the CLAUDE.md maps.
## Motivation
3,660 of 4,415 functions in the `/v2` image are still default-named (`FUN_`/`LAB_`), a large slice of
which is statically-linked CRT/STL. We keep re-recognizing the same library functions by behavior
(`std::map::find` = `FUN_0047f280`, the open-addressing hash insert = `FUN_0042cf70`, `operator new`,
string/`memcpy` ops). Naming them lets the decompile stop burying real engine logic in unnamed library
noise — the last cheap static front-loader before the dynamic oracle. This is explicitly a *de-noising
polish*, lower-leverage than the first three levers, so it is scoped to stay light.
## Approach (recon-first, decided in brainstorming)
Ghidra's **Function ID** analyzer is enabled but has named little so far (the bundled FidDbs may not
cover this VC9/VS2008-era static runtime). So the recon decides the shape.
### Task 1 — Function ID recon (THE GATE)
Re-run the Function ID analyzer against `/v2`, including a pass with **"Always Apply FID Labels"**
enabled, and measure the delta: how many `FUN_`/`LAB_` functions become named (source `ANALYSIS`/library
names). Record the before (3,660 default) and after counts.
- **GATE — STRONG** (names a meaningful number, say ≥ 100 new library names): Function ID *is* the win.
Keep the labels, `save_program`, document the count, done. No curated registry needed.
- **GATE — WEAK** (~0 new): the bundled FidDbs don't match this runtime → fall back to the curated
hot-set (Task 2). Do **not** invest in generating a custom VC9 FidDb (out of scope — heavy, low return).
Function ID applies labels with `ANALYSIS` source, which never clobbers our `USER_DEFINED` names — so a
strong result composes cleanly with the handler/ctx work.
### Task 2 — curated hot-set (ONLY if Task 1 is WEAK)
A small canonical registry, same pattern as `engine-ctx.toml`:
`vm-map/lib-functions.toml``[[func]]` entries `{ address, name, note, source, confidence }` for the
library functions we have already identified in our RE (seeded from `engine-re.md`): e.g.
`0x47f280 → std_map_find`, `0x42cf70 → hashmap_insert`, plus `operator new`/`delete`, `memcpy`, and any
string ops we've pinned. `tools/lib_functions_build.py` (`--build`/`--lint`) → `build/lib-functions.json`
+ `docs/lib-functions-reference.md`; a `run_script_inline` pass renames each function (skip if already
`USER_DEFINED`). Grows one `[[func]]` at a time as we identify more — the durable home for "this worker
is a known library function." **Small by design** (a handful to low-dozens), not a comprehensive sweep.
## Validation
- Task 1: report the newly-named count; spot-check 23 (e.g. does `FUN_0047f280` now carry a
`std::map`-family name? does a known `operator new` site read named?).
- Task 2 (if taken): decompile `gfx_op_0x215_register_query` — its `gfx_registry_map_find(...)` /
`FUN_0047f280` call should read as the curated `std_map_find`; `--lint` clean; 0 clobbers.
- `save_program` succeeds.
## Scope & boundaries
- **In:** run Function ID + apply; (conditionally) a small curated registry of known library functions.
- **Out:** generating a custom VC9/MSVCR90 FidDb (heavy, low marginal return); BSim; comprehensively
naming all 3,660 (we take what Function ID gives + what we've RE'd, not a big-bang).
- **Regenerable:** `build/lib-functions.json` + `docs/lib-functions-reference.md` are generated.
## Acceptance criteria
- Task 1 recon run; before/after default-function counts recorded; gate verdict explicit.
- If STRONG: labels applied, saved, count documented. If WEAK: `vm-map/lib-functions.toml` seeded with the
known hot functions, built (`--lint` clean), applied to `/v2` (0 clobbers), saved.
- ≥2 validation spot-checks read as named library calls.
- `docs/engine-re.md` records the outcome; `tools-reference.md` + CLAUDE.md maps updated iff a registry
was built; status memory records the milestone (and marks lever #2 done, #3 next).
## Outcome (2026-07-09) — PIVOTED
Task 1 recon: Function ID = **WEAK** (0 new names; bundled FidDbs don't cover the VC9 runtime). Also
discovered the library functions we actually touch are already `USER_DEFINED` (annotate-as-you-go). So
the curated CRT/STL registry (Task 2 as specced) would have renamed already-named functions — near-zero
value. **Pivoted** (user-approved) to naming the *hot documented-but-unnamed helpers* the recon surfaced
by call-count: `vm_operand_fetch`@`0x41b940` (1021 refs), `vm_operand_write`@`0x425fb0`,
`vm_operand_lvalue`@`0x415f30`, `__security_check_cookie`@`0x54f981` (692), `operator_new`@`0x5502be`
(533) — ~2,400 call sites de-noised. **No `lib-functions.toml`/builder built** (the Ghidra name is the
record for these; no tool consumes them), so CLAUDE.md/tools-reference were not touched. Full account:
`docs/engine-re.md` "Hot-helper naming pass".