conco/OpenMaidEngine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

feat(assets): solve asset resolution (SYS4INI per-scene section manifest)

Browse Source 
resId -> files[section_base(scene) + resId]. SYS4INI's file list is
sectioned, one per scene (SCxxxx.BIN + its cross-archive asset manifest);
file_number is the index within the section. Unified for set-texture,
play-bgm, play-voice. Fully static/general -> no per-scene capture.

- tools/parse_sys4ini.py: SYS4INI (S4IC422, LZSS) -> build/asset-index.json
- tools/resolve_asset.py: sections + (scene,resId) resolver -> build/asset-sections.json
- validated: 97% structural, SC0000 17/17 vs Frida, 586/595 captured loads
- opcodes.toml: set-texture/create/draw-texture, play-bgm/voice enriched (frida-grounded)
- Frida tooling (capture_load_order all-archive, correlate_scope, ...) + vm0 --settex
- docs: asset-resolution-re (step2 SOLVED), global-memory-re (shelved), tools-reference

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
gamer147
2026-07-06 20:48:22 -04:00
parent b92e815850
commit a61c0c9abd
18 changed files with 1887 additions and 87 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docs/PROJECT-STRUCTURE.md
View File

@@ -54,7 +54,10 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
│ └── opcode-leads.json, small-script-listings.md
│
├── docs/ all documentation
│ ├── PROJECT-STRUCTURE.md this file
│ ├── PROJECT-STRUCTURE.md this file (where things live)
│ ├── tools-reference.md every tool: purpose, usage, I/O (operational companion)
│ ├── asset-resolution-re.md resId→file RE (graphics/audio); asset-index steering
│ ├── global-memory-re.md runtime global observation RE (SHELVED; future starting point)
│ ├── remake-architecture-and-roadmap.md THE direction doc (phases A–E)
│ ├── phase-a-slice-plan.md the current slice (A0/A1/A2)
│ ├── vm-mapping-plan.md the phased decode plan

65
docs/asset-resolution-re.md
View File

@@ -31,8 +31,9 @@ highest-risk area of the port. This doc is the steering state; it feeds the A2b
- **The resolution chain is opaque statically.** `CGINIT` (`build/data/CGINIT.json`) is a
925-column *numeric* record table (row-major, sparse) — **not** an id→filename map.
**`SYS4INI.BIN` (magic `S4IC422`) is the authoritative asset index** the game + `BinExtractALF`
use (name ↔ archive ↔ offset ↔ size), but filenames are **not stored as plain ASCII** (an
`EV001AA` search misses), so it needs S4IC-format RE to parse.
use (name ↔ archive ↔ offset ↔ size). Filenames aren't plain ASCII because the whole directory
is **LZSS-compressed** (not encrypted). **DONE (2026-07-06):** `tools/parse_sys4ini.py` parses it
→ `build/asset-index.json` (13206 entries). See step 1 below.
- **Frida file-I/O is noisy.** `ReadFile` hooks on `DATA2.ALF` capture reads during the opening, but
the offsets/spans don't line up with extracted AGF sizes → the game likely **memory-maps** the
archives (so `ReadFile` offsets are OS paging, not clean per-asset loads) and/or uses async reads.
@@ -40,16 +41,46 @@ highest-risk area of the port. This doc is the steering state; it feeds the A2b
## The RE plan (ordered)
1. **Parse `SYS4INI` (S4IC422) → an asset index** `{name, archive, offset, size}`. *Reusable and
bounded* — it names every asset in every DATA*.ALF, gives archive-offset→name (to rescue Frida
offsets), and is the **answer key** for step 2. Deliverable: `tools/parse_sys4ini.py` +
`build/asset-index.json`. (Format reference: asmodean's `exs4alf`, which `BinExtractALF` is based on.)
2. **Crack `resId → filename`.** With SYS4INI as the answer key, either (a) **order-correlate**: run
SC0000 in our engine to get the `set-texture(resId)` sequence, capture the real game's asset-load
order via a *reliable* Frida hook, and align them; or (b) **hook the internal load-by-id
function** directly (find via the opcode dispatch for `0x1f9`) to read `resId → name` at the
source. Likely underlying rule: `resId → CGINIT/table → name`. Deliverable: the mechanism +
`vm-map/resources.json` (or a generated map) seeding at least SC0000's slideshow.
1. **Parse `SYS4INI` (S4IC422) → an asset index** `{name, archive, offset, size}`. **✅ DONE
(2026-07-06).** `tools/parse_sys4ini.py` → `build/asset-index.json`: 5 archives (DATA1–5),
13206 real entries (2 `@` placeholders skipped). **Format:** `uint32 packed_size @0x134`, then an
LZSS stream at `0x138` running to EOF (GARbro-style: 0x1000 zero-filled ring buffer, init pos
0xFEE, control bits LSB→MSB, 1=literal / 0=two-byte backref `off=(hi&0xf0)<<4|lo`, `len=3+(hi&0xf)`).
Decompresses to `uint32 arc_count`, `arc_count × char[256]` archive names, `uint32 file_count`,
then `file_count ×` 80-byte records `{char name[64]; u32 arc_id, file_number, offset, size}`.
**Validated:** decompressed length (1058783) equals the stored size dword at `0x12c`; per-archive
counts match the `extracted/` ground truth exactly (DATA2=985, DATA3=39, DATA4=9733, DATA5=210);
all 13206 `offset+size` fit inside their real `.ALF`; 837 name-matched files → 0 size mismatches.
`files[]` preserves directory order (feeds step 2's order-correlation). Re-run:
`py -3.11 -X utf8 tools/parse_sys4ini.py --check`. (Ref: asmodean's `exs4alf` / GARbro Eushully `ArcALF.cs`.)
2. **Resolve `resId → asset file`.** **✅ SOLVED (2026-07-06) — fully static & general; NO runtime capture.**
**The rule:** SYS4INI's file list is organized into **SECTIONS, one per scene** — each is a
`SCxxxx.BIN` script entry followed by that scene's **asset MANIFEST**: every asset it references,
across *all* archives and types (EV/BG/CS/AE graphics **and** OGG/WAV audio), interleaved in usage
order. `file_number` is the **0-based index within the section**. So:
> **`resId → files[ section_base(scene) + resId ]`**, where `section_base` = the start of the SYS4INI
> section containing the scene's `SCxxxx.BIN`.
Unified for `set-texture(resId)`, `play-bgm(id)`, `play-voice(id)` — one manifest. **Tool:**
`tools/resolve_asset.py --build` → `build/asset-sections.json` (359 sections, 136 scenes);
`resolve_asset.py <SCENE> [resId]` resolves. **Validated:** `file_number == position − section_base`
for 12848/13206 files (97%); SC0000 resolves 17/17 across archives vs the Frida capture (`0x25→EV052CA`,
`0x36→BG030A` background, `0x6c→EM* effect`, `play-bgm 5→BGM006`); 586/595 distinct captured loads
(all sections) satisfy `files[base+fn]==name`. This is the derivable rule that generalizes to any
AGE game with the same container — **the "scope" was just which SYS4INI section the scene lives in.**
*How we got here (condensed):* first confirmed `resId == file_number` via Frida load-order correlation
for SC0000's opening, but `file_number` is not globally unique so a per-scene "scope" was needed. A long
hunt for the selector (thought it was native scene state; even tried reading `G[0x62424]` live — the
VM global memory is structured/packed, see `docs/global-memory-re.md`) missed the real structure until a
**full multi-archive capture** (user domain tip: DATA1 holds BG/CS/CB/CA/CP graphics by name prefix, not
just DATA2 EV CGs) revealed `file_number == SYS4INI position` inside per-scene sections. Superseded tools:
`tools/correlate_scope.py`, `vm0.py --settex` (VM set-texture trace; still useful, but vm0 diverges on
branchy non-opening scenes — use the C# VM to trace those). Runtime note for future work: the game is
**packed** (main VM logic in a per-run heap `r-x` region) and streams archives through a heap block-cache
via `ReadFile` (not mmap); the stable AGF decoder is `AGE.EXE+0x74f1f`.
3. **Wire the backend** (already designed — A2b-background plan Tasks 3–5): `ResourceMap` resolver +
Godot `TextureRect` compositing; render only resolved full-screen slots. Mechanical once (1)+(2) land.
4. **Audio** (parallel, same shape): resolve `play-voice`/`play-bgm` `id → OGG` via SYS4INI + a
@@ -67,7 +98,9 @@ rendering what the executed bytecode + the map produce (never a hardcoded image)
## Status
A2b-background: **machinery landed** (texture ops engine-driven, tools, findings). The **render is
blocked on asset resolution** (steps 1–2), which is promoted to its own foundational effort. Next:
either start step 1 (`SYS4INI` parser) or bank momentum with the Frida-free **choices** sub-slice
(static-RE opcode hunt) while resolution waits its scheduled turn.
A2b-background: **machinery landed**; **steps 1 & 2 SOLVED (static, general).** Step 1 =
`build/asset-index.json`. Step 2 = **`resId → files[section_base(scene) + resId]`** via SYS4INI
per-scene sections (`tools/resolve_asset.py` + `build/asset-sections.json`) — no runtime capture, works
across all archives/types and for audio too. Remaining for the render (step 3): wire a `ResourceMap`
(scene → section_base; resId → asset via the index) + Godot `TextureRect` compositing (A2b plan Tasks 3–5,
now purely mechanical). Audio (step 4) uses the *same* resolver (`play-bgm/play-voice id → files[base+id]`).

128
docs/global-memory-re.md Normal file
View File

@@ -0,0 +1,128 @@
# Runtime Global Observation — RE state & starting point
**Goal.** Read the running game's live VM global variables by their bytecode address (e.g.
`G[0x62424]`, `global-string 0x279`). This is the **VM-validation cornerstone**: with it we can
compare our VM's global state against the real game's at any point, verify effectful opcodes that
have no machine oracle (battle math, flag logic, stat updates), and *name* the ~200 still-unclassified
globals by watching them change. The roadmap flags it as the thing that replaces most Frida/Unicorn
guesswork once the VM is validated-correct.
**Status: SHELVED (2026-07-06), deliberately.** We learned the shape of the problem but did not
achieve a general "read any global by address" capability. This doc records what we tried, what we
proved, the concrete landmarks found, and the *right* way to resume. It is **not** on the critical
path for anything else, and — importantly — it does **not** crack the asset-resolution scope selector
(see "What this does NOT solve").
---
## The core finding: our flat-address model ≠ physical layout
Our VM models globals as one flat address space (`G[addr]`), and that is **execution-correct**
(byte-exact against the dialogue-trace oracle). But it is an *abstraction*. Physically the game's
state is **structured and multi-store**:
- **Multiple stores.** VM int-globals, string-globals, and float-globals are almost certainly
separate arrays (their address magnitudes differ wildly: `global-string 0x279` = 633 vs
`global-int 0x62424` = 403,492). On top of that, values get **copied into native C++ objects**.
- **Structured records, not flat int32.** Entity data (units, party) is stored as records with
**inline fixed-size string slots + mixed int fields + native heap pointers** — not a packed int
array. A `base + addr*4` scan therefore finds nothing (confirmed: no stride in {2,4,8,12,16}
reproduces a known consecutive-address `*INIT` value run).
- **Transient vs. stable globals.** A global is a real variable, but *how it's used* varies:
- **Transient** (e.g. `G[0x62424]`, the CG resId): a scratch/argument register — `mov G[0x62424]
= resId; call load` — set right before a call, overwritten right after. Never holds a stable
value. Non-atomic value scans **race** against it and miss it.
- **Stable** (e.g. `global-string 0x279`, the player name): persists across play. *These* are
findable by value+stability scans; transient ones are not.
- **Multiple resId globals.** Backgrounds vs. foreground portraits/sprites appear to use different
resId globals / churn `G[0x62424]` between visible background changes (observed live).
- **Packed process.** Main VM logic runs from a **per-run heap `r-x` region** (~30 MB, nonstable
base) — so the interpreter can't be hooked at a fixed `AGE.EXE+off`. Archives are streamed via a
heap block-cache through `ReadFile` (not memory-mapped). See `docs/asset-resolution-re.md`.
**Why the naive scan failed.** The first attempt scanned for a flat `base+addr*4` int32 array using a
signature of `*INIT` constants. It found 0 matches because (a) the layout isn't flat int32, and (b)
`*INIT` entity data lives in structured records. The individual values exist in memory but only
coincidentally adjacent.
---
## What we tried (chronological, with outcomes)
1. **Flat-int32 signature scan** — `tools/frida/find_globals_base.py` (+ `build/globals-signature.json`,
15,481 distinctive `(addr,value)` pairs from `*INIT` `mov (global-int A) IMM`; 215-dword contiguous
anchor @`0x631a9`). **Result: 0 hits** at any anchor length, in the opening AND in the first dungeon
(ruling out load-timing). Diagnostic: the individual values occur (coincidentally); no stride in
{2,4,8,12,16} reproduces the consecutive-address sequence ⇒ **not flat int32**.
2. **"Lily" (player name) anchor** — searched for the entered name. Found as ASCII `Lily\0`. **But the
context proved it's the wrong anchor:** every copy is embedded in a **native unit-record** (inline
string slot + stats + heap pointers), i.e. downstream copies, not the VM string-global store.
3. **Differential value scan on `G[0x62424]`** — `tools/frida/find_global_by_sequence.py`. Self-driven
by the ReadFile→file_number(=resId) signal: scan for resId at load 1, keep those that become the
next resId, etc. **Converged to 3 addresses that tracked 35→37→39→43→46 perfectly — but all were
STACK slots** (region `0x18f000`, self-referential + `0x76xxxxxx` return addresses); their values
are garbage between loads. They're the per-call argument copies, not the global.
4. **Stability filter** (re-read survivors ~1.4 s later, during the pause) — proved the point: 2084
locations held resId 35 *stably*, but **0** of them became 37 ⇒ **`G[0x62424]` is transient**, never
stable at a value, so neither value-scans nor stability-filters can pin it.
---
## Concrete landmarks (for a future run — but note ASLR: these are per-process)
- **Player name** = `global-string 0x279`; default `"リリィ"` (set in `INPUTNAME.BIN` via
`set-string (global-string 0x279)`, then the input op `0x1aa`/`u00425920` writes the entered name to
`0x279`, then copied to `local-string 0x135`). **Stable global** — the best future anchor.
- **Native unit records** holding the name: ASCII `Lily\0` in 16-byte-ish slots, followed by int stats
(level/…, e.g. `0c 0f 04 0f 13 1f`) and native pointers; arrays with stride ~`0x1c`. Region example
`0x6e9c000 +0x82000` (rw-, no module). These are **native objects**, not the VM store.
- **Transient-arg stack region** where load-arg resId copies appear: ~`0x18f000 +0x11000` (68 KB rw-).
- **Large heap regions** (candidate VM stores): 51.8 MB @ `0x2f64000`, plus ~10/9.6/8/6 MB regions.
- **Unpacked code (packer)**: 30.3 MB **r-x** @ `0x62411000` (main VM logic; nonstable base per run).
- Element size / mapping: **unknown**; it is *not* uniform `base+addr*4` int32.
## Tools built (kept for resumption)
All under `tools/frida/` (see `tools/frida/README.md` and `docs/tools-reference.md`):
- `find_globals_base.py` — `--build-sig` builds `build/globals-signature.json`; scan mode does the
flat-int32 signature scan (robust prefix ladder + verify). *Currently finds nothing → layout isn't flat.*
- `find_global_by_sequence.py` — differential resId scan with stability filter. *Finds stack proxies;
transient globals elude it.*
- `capture_load_order.py`, `locate_resource_load.py`, `capture_resid_args.py` — asset-resolution
captures (context: how the loader/decoder chain was found).
---
## What this does NOT solve (avoid the trap we fell into)
Reading `G[0x62424]` live would **not** reveal the asset-resolution **scope selector** (why resId 37 →
`EV052CA` and not one of the other 8 fn-37 files). The resId is already free from the ReadFile
`file_number`; the scope is *separate native scene state*. So global observation and the scope selector
are independent problems — don't chase globals expecting to crack resolution.
---
## How to resume properly (recommended plan)
Heuristic value-scans only ever find *specific, stable* globals one at a time; they give no general
`address → memory` mapping. A **guaranteed, general** capability needs the **interpreter's
address-resolution logic**. Recommended order:
1. **Anchor on a STABLE global**, not a transient one. Best: `global-string 0x279` (the name). Find its
*VM-store* copy (distinguish from native unit copies: the VM store won't be wrapped in heap
pointers). A distinctive name makes the value scan collapse fast.
2. **Find the interpreter's global-access function.** Set a hardware/`MemoryAccessMonitor` watchpoint on
that stable global's physical location, trigger a bytecode read (e.g. open a menu that draws the
name), and **backtrace into the heap interpreter**. That function's address computation *is* the
`bytecode-address → physical` mapping — read it rather than guessing the layout.
3. **Generalize + verify.** Derive the mapping (likely per-store / per-region), build `read_global(addr)`,
and verify against our VM's known state (e.g. read a stable flag whose value our VM predicts).
4. **Then** build a runtime global-watch tool for VM validation and effectful-op naming.
Alternative bootstraps if watchpoints are awkward: snapshot-diff at **quiescent** points (change a
known stable global via gameplay, diff memory); or a hardware **write** watchpoint on a stable global's
location to catch the interpreter's write path.
**Bottom line for the next session:** the pieces (signature, tools, landmarks, the name anchor) are in
place. Resume from a *stable* anchor and target the *interpreter*, not heuristic scans of transient
globals — and only when runtime observation is actually the priority (it isn't blocking other work).

45
docs/opcode-reference.md
View File

@@ -17,6 +17,16 @@
- **grounding:** source=inference, confidence=low
- **evidence:** confirm via frida
### 0xbf `play-bgm` (play-bgm, argc 1)
- **summary:** Play background music by id; id resolves via the SYS4INI section manifest -> files[section_base(scene)+id] (OGG in DATA3). Same resolution as set-texture.
- **grounding:** source=frida, confidence=high
- **evidence:** Frida capture: `play-bgm 0x5` in SC0000 (section base 0) loaded BGM006.OGG = files[5]. Unified with set-texture resolution rule.
### 0xc4 `play-voice` (play-voice, argc 1)
- **summary:** Play a voice clip by id; id resolves via the SYS4INI section manifest -> files[section_base(scene)+id] (voice OGG in DATA1/DATA4). Same rule as set-texture/play-bgm.
- **grounding:** source=investigation, confidence=med
- **evidence:** Section-manifest resolution validated across archives incl. DATA4 voice OGGs (586/595 captured loads); per-clip id->OGG not individually Frida-pinned yet.
## compute
### 0x1a2 `resolve-handle?` (u00428010, argc 1)
@@ -36,11 +46,26 @@
- **grounding:** source=inference, confidence=med
- **evidence:** confirm via frida
### 0x1f8 `create-texture` (create-texture, argc 4)
- **summary:** Allocate/prepare a texture slot: (slot, width, height, flag). e.g. `create-texture 0xd 0x190 0x1e 0x0` = slot 13, 400x30.
- **grounding:** source=investigation, confidence=med
- **evidence:** SC0000 CG/UI-draw path disasm; slot/w/h roles read off the operands (400x30 text bars, etc.).
### 0x1f9 `set-texture` (set-texture, argc 3)
- **summary:** Load asset #resId into texture slot: (resId, slot, flag=-1). resId resolves via the SYS4INI per-scene section manifest: files[section_base(scene)+resId] (same rule for play-bgm/play-voice). See docs/asset-resolution-re.md.
- **grounding:** source=frida, confidence=high
- **evidence:** SC0000 Frida-confirmed 17/17 (0x25->EV052CA, 0x2e->EV052DB, 0x36->BG030A background); resolution rule validated on 586/595 captured loads. Traced in CG-load subroutine label_12649 as `set-texture G[0x62424] <slot> -1`.
### 0x1fa `ui-clear?` (u00420480, argc 1)
- **summary:** 1 arg (element id); follows 0x1f7 — show/hide/clear UI element by id
- **grounding:** source=inference, confidence=med
- **evidence:** confirm via frida
### 0x1fb `draw-texture` (draw-texture, argc 8)
- **summary:** Blit a texture slot to screen. Observed 8 args: (handle, slot, srcx, srcy, w, h, dstx, dsty). e.g. `draw-texture 0xcf08 0x3 0 0 0x320 0x258 0 0` = full-screen (800x600) slot 3 at (0,0).
- **grounding:** source=investigation, confidence=med
- **evidence:** SC0000 CG-load subroutine label_12649: `draw-texture (ptr) (slot) 0 0 (w) (h) (dstx) (dsty)`; full-screen slot-3 draws use 0x320x0x258 (800x600).
### 0x1ff `draw?` (u00420770, argc 4)
- **summary:** 4 args (global+imms); follows 0x217, then call
- **grounding:** source=inference, confidence=low
@@ -403,10 +428,6 @@ op 0x90 (u0041BEB0, argc 7): `0x90 x y w h tgt_a tgt_b tgt_c`. Kelebek left it "
- **summary:** —
- **grounding:** source=kelebek, confidence=low
### 0xbf `play-bgm` (play-bgm, argc 1)
- **summary:** —
- **grounding:** source=kelebek, confidence=med
### 0xc0 `u00415620` (u00415620, argc 1)
- **summary:** —
- **grounding:** source=kelebek, confidence=low
@@ -415,10 +436,6 @@ op 0x90 (u0041BEB0, argc 7): `0x90 x y w h tgt_a tgt_b tgt_c`. Kelebek left it "
- **summary:** —
- **grounding:** source=kelebek, confidence=low
### 0xc4 `play-voice` (play-voice, argc 1)
- **summary:** —
- **grounding:** source=kelebek, confidence=med
### 0xc5 `u0041D4A0` (u0041D4A0, argc 2)
- **summary:** —
- **grounding:** source=kelebek, confidence=low
@@ -803,18 +820,6 @@ op 0x90 (u0041BEB0, argc 7): `0x90 x y w h tgt_a tgt_b tgt_c`. Kelebek left it "
- **summary:** —
- **grounding:** source=kelebek, confidence=low
### 0x1f8 `create-texture` (create-texture, argc 4)
- **summary:** —
- **grounding:** source=kelebek, confidence=med
### 0x1f9 `set-texture` (set-texture, argc 3)
- **summary:** —
- **grounding:** source=kelebek, confidence=med
### 0x1fb `draw-texture` (draw-texture, argc 8)
- **summary:** —
- **grounding:** source=kelebek, confidence=med
### 0x1fd `u00420620` (u00420620, argc 4)
- **summary:** —
- **grounding:** source=kelebek, confidence=low

87
docs/tools-reference.md Normal file
View File

@@ -0,0 +1,87 @@
# Tools Reference
Living catalogue of every script in `tools/` — **what it does, how to run it, and what it
reads/writes**. This is the operational companion to `docs/PROJECT-STRUCTURE.md` (which is the
*where-things-live* map); when they overlap, PROJECT-STRUCTURE owns layout, this file owns
usage + I/O. Keep it current: **add a row here whenever you add a tool, and update the row
whenever a tool's inputs/outputs change.**
## Conventions (apply to every tool)
- **Run with** `py -3.11 -X utf8 tools/<name>.py …` — the `-X utf8` is required on Windows so
cp932/Shift-JIS source text renders (and generated files stay UTF-8).
- **Paths are never hard-coded.** Every tool imports `tools/paths.py` for `GAME_DIR` /
`EXTRACTED` / `DATA1` / `BUILD` / `VM_MAP` / `BIN`. Relocate the tree by editing only that file.
- **Generated files are never hand-edited** (they're marked ⚙ below). Edit the source, re-run
the generator.
- **`build/` and `extracted/` are disposable** — everything under them regenerates from a tool.
## Path anchor
| Tool | Purpose | I/O |
|---|---|---|
| `paths.py` | ★ Single path anchor — derives all workspace dirs from its own location; `paths.scripts()` returns the override-aware `{NAME.BIN → path}` corpus map (loose game-folder patches shadow `extracted/DATA1`). | *Imported, not run.* |
## Container parse / disassemble
| Tool | Purpose | Run | Reads → Writes |
|---|---|---|---|
| `sys4load.py` | Loader + opcode-decoding disassembler for SYS4 `.BIN` scripts (the container-format core every other tool builds on). | `sys4load.py <file.BIN>` · `--summary` · `--strings` · `--json` · `sys4load.py <dir> --validate` (corpus check) | `.BIN` + `age_opcodes*.py` + `build/global-var-map.json` → stdout listing, or `build/scripts-json/` with `--json` |
| `age_opcodes.py` | 548-entry Kelebek AGE opcode/arg-type table. **PRISTINE upstream data — never edit.** | *Imported.* | — |
## Opcode reference toolchain — single source of truth = `vm-map/opcodes.toml`
All opcode knowledge (ABI, semantics, provenance, `depends_on`) is hand-edited **only** in
`vm-map/opcodes.toml`. Everything else is generated from it.
| Tool | Purpose | Run | Reads → Writes |
|---|---|---|---|
| `opcodes_build.py` | Generator + linter for the opcode reference. | `--build` · `--lint` · `--bootstrap` | `vm-map/opcodes.toml` → ⚙ `tools/age_opcodes_himegari.py`, ⚙ `build/opcodes.json`, ⚙ `docs/opcode-reference.md`, ⚙ `build/opcode-coverage.md` |
| `opcodes_model.py` | In-memory model + loader + linter (dangling-ref / confidence-ceiling / vocabulary / dependents). | *Imported by `opcodes_build.py`.* | `vm-map/opcodes.toml` → — |
| `test_opcodes.py` | Unit tests for the opcode tooling. | `test_opcodes.py` | — |
| `opcode_context.py` | Read-only evidence gatherer for classifying unnamed opcodes (frequency, argc, operand-type signature, neighbours, disasm snippets, Kelebek comment). | `--top 20` · `opcode_context.py 0x1f4 0x71 …` | corpus → stdout |
| `validate_opcode_table.py` | Definitive decode-coverage validator (replicates Kelebek's `data_array_end` code/data split). | `validate_opcode_table.py` | corpus → stdout |
| `validate_opcode_table_naive.py` | Naïve variant of the above (baseline comparison). | `validate_opcode_table_naive.py` | corpus → stdout |
| `age_opcodes_himegari.py` | ⚙ Inferred Himegari opcode semantics — **generated; do not hand-edit.** | *Imported by `sys4load.py`.* | — |
## Extraction / data corpora
| Tool | Purpose | Run | Reads → Writes |
|---|---|---|---|
| `extract_phase2.py` | Batch: disassembly + text corpora for every script. | `extract_phase2.py` | corpus → `build/disasm/*.asm`, `build/text/{dialogue.jsonl,strings.jsonl,*.strings.txt}`, `build/manifest.json` |
| `extract_init.py` | Parse a `*INIT` data table (auto-detects name / numeric / footer shape). | `extract_init.py <TABLE> [OUTNAME] [--mode …]` | `<TABLE>.BIN` → `build/data/<OUTNAME>.json` |
| `global_map.py` | Build the partial global-variable name map from static evidence. | `global_map.py` | corpus + `build/data/` → `build/global-var-map.{json,md}` |
## VM
| Tool | Purpose | Run | Reads → Writes |
|---|---|---|---|
| `vm0.py` | Headless Python bytecode VM (Phase A0 execution-model prototype; reuses `sys4load`). | `--test` (RECOVER unit test) · `--sweep [N]` (oracle coverage) · `--scene NAME` · `--settex NAME` (set-texture resId trace + exec trace) · `<file.BIN>` | corpus → stdout; `build/vm0-trace.json`; `build/settex-<NAME>.json` |
| `correlate_scope.py` | Align the VM's `set-texture(resId)` trace with the game's Frida load order → tag each load's DATA2 package, flag package transitions, dump the significant ops in each transition span (the **scope selector** hunt). | `correlate_scope.py <SCENE>` | `build/settex-<SCENE>.json` + `build/frida-load-order-result.json` + index → stdout |
## Asset resolution / graphics
| Tool | Purpose | Run | Reads → Writes |
|---|---|---|---|
| `parse_sys4ini.py` | Parse `SYS4INI.BIN` (S4IC422, LZSS-compressed) into the authoritative asset index — name ↔ archive ↔ offset ↔ size for all DATA*.ALF (the `resId→file` answer key). | `parse_sys4ini.py [--check]` (`--check` validates vs `extracted/` + `.ALF` sizes) | `姫狩り…/SYS4INI.BIN` → `build/asset-index.json` |
| `resolve_asset.py` | ★ **The static asset resolver.** SYS4INI is sectioned (one per scene: `SCxxxx.BIN` + its cross-archive manifest; `file_number` = index within section). Resolves `resId → files[section_base(scene) + resId]` for graphics AND audio, no capture. | `resolve_asset.py --build` · `resolve_asset.py <SCENE> [resId]` | `build/asset-index.json` → `build/asset-sections.json`; resolves any (scene, resId) |
| `resolve_frida_reads.py` | Rescue noisy Frida archive-read offsets → asset names via the index (per-archive range search; drops 0x20000 paging reads); recovers the per-scene asset load order. | `resolve_frida_reads.py [reads.log] [-o out.json]` | `build/frida-reads.log` + `build/asset-index.json` → `build/frida-asset-loads.json` |
| `convert_agf.py` | Convert named AGF stills to BMP via `AGF2BMP2AGF.exe`. | `convert_agf.py EV001AA.AGF …` | `extracted/DATA2|DATA5/*.AGF` → `build/textures/*.BMP` |
## Runtime capture (Frida)
| Tool | Purpose | Run | Reads → Writes |
|---|---|---|---|
| `tools/frida/capture_graphics.py` | Attach Frida to the running game; log archive reads/opens (ground-truth for asset resolution). See `tools/frida/README.md`. | `py -3.11 -u -X utf8 tools/frida/capture_graphics.py [AGE.EXE]` | running game → `build/frida-reads.log`, `build/frida-opens.log` |
| `tools/frida/capture_load_order.py` | **Primary asset-resolution capture:** recover a scene's per-asset load order from exact-start `ReadFile` reads → names via the index; confirms `resId==file_number`. Attach; replay scene; `--analyze`. | `py -3.11 -u -X utf8 tools/frida/capture_load_order.py [pid]` · `--analyze` | running game + index → `build/frida-load-order.jsonl`, `…-result.json` |
| `tools/frida/locate_resource_load.py` | Phase-1 locator: back-traces asset-opens to find the native AGF load chain (`0x16d5d7→0x74f1f`). | `py -3.11 -u -X utf8 tools/frida/locate_resource_load.py [pid]` · `--aggregate` | running game → `build/frida-resource-bt.jsonl` |
| `tools/frida/capture_resid_args.py` | Phase-2 probe: dumps the decoder's args / context / caller frame (established the loader carries only offsets, not names). | `py -3.11 -u -X utf8 tools/frida/capture_resid_args.py [pid]` · `--analyze` | running game → `build/frida-resid-args.jsonl` |
| `tools/frida/find_globals_base.py` | Runtime-global RE (SHELVED — see `docs/global-memory-re.md`): flat-int32 signature scan for the VM global array. Finds nothing → layout isn't flat. | `--build-sig` · `py -3.11 -u -X utf8 tools/frida/find_globals_base.py [pid]` | `*INIT` → `build/globals-signature.json`; scans running game |
| `tools/frida/find_global_by_sequence.py` | Runtime-global RE (SHELVED): differential resId value-scan + stability filter. Finds stack proxies; proved `G[0x62424]` is a transient arg-register. | `py -3.11 -u -X utf8 tools/frida/find_global_by_sequence.py [pid]` | running game + index → stdout |
## Historical / one-off
| Tool | Purpose |
|---|---|
| `probe_*.py` (`probe_header`, `probe_leads`, `probe_refs`, `probe_tables`, `probe_tags`, `probe_types`, `probe_xref`) | Container/opcode format-RE probes used to reverse the format originally. Kept for reproducibility; not part of the normal workflow. |
| `pack_check.py` | Checks whether `AGE.EXE` is packed (it is). No longer a blocker — we run our own VM. |