chore: initialize age-reimpl repo
Reverse-engineering + open reimplementation workspace for Eushully's AGE/SYS4 engine (first target: Himegari). The repo root is age-reimpl/; the original game install and the extracted ALF data are siblings outside the repo and are never tracked. build/ (derived corpora) is gitignored and regenerated by the tools. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
gamer147
96
docs/PROJECT-STRUCTURE.md
Normal file
96
docs/PROJECT-STRUCTURE.md
Normal file
@@ -0,0 +1,96 @@
|
||||
# Project Structure
|
||||
|
||||
Layout for the *姫狩りダンジョンマイスター* (Himegari) → open AGE-engine reimplementation.
|
||||
The guiding rule is **source vs. derived vs. our work**: the shipped game is read-only
|
||||
input, the extracted archives and everything our tools generate are disposable/reproducible,
|
||||
and our code + docs live entirely apart from the game install. Nothing we produce is ever
|
||||
written back into the game folder.
|
||||
|
||||
Workspace root: `S:\Game Hacking\Eushully\Himegari\`
|
||||
|
||||
```
|
||||
S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
|
||||
│
|
||||
├── 姫狩りダンジョンマイスター/ ← SOURCE — pristine game install (read-only)
|
||||
│ │ Never edit, move, or add to this folder. It holds ORIGINALS ONLY.
|
||||
│ ├── AGE.EXE, AGERC.DLL, *.dll shipped engine (packed). Stays intact and
|
||||
│ │ runnable in place — Frida launches it if needed.
|
||||
│ ├── DATA1-5.ALF, APPEND01.ALF/.AAI shipped archives (~2.3 GB).
|
||||
│ ├── *.BIN 52 loose patch-override scripts (v1.03) —
|
||||
│ │ AUTHORITATIVE over their DATA1 copies. Plus
|
||||
│ │ non-script indices (SYS4INI=S4IC, SYS4AB=S4AB).
|
||||
│ └── *.exe (uninstallers), SAS0099.OGG … other shipped files.
|
||||
│
|
||||
├── extracted/ ← DERIVED (game-side) — extracted ALF contents,
|
||||
│ │ ~3.9 GB, regenerable via age-reimpl/bin/BinExtractALF.
|
||||
│ └── DATA1/ … DATA5/ DATA1 = 481 .BIN scripts (the corpus we parse)
|
||||
│ + AGF/BMP/WAV in the others.
|
||||
│
|
||||
└── age-reimpl/ ← OUR WORK (everything we made lives here)
|
||||
│
|
||||
├── tools/ Python tooling (parser/disassembler + extractors + VM)
|
||||
│ ├── paths.py ★ central path anchor — the ONLY place that knows
|
||||
│ │ where the game / extracted / build dirs are. All
|
||||
│ │ tools import it; relocatable with no other edits.
|
||||
│ ├── sys4load.py loader + disassembler (opcode-decoding)
|
||||
│ ├── age_opcodes.py 548-entry AGE opcode/arg-type table (pristine)
|
||||
│ ├── age_opcodes_himegari.py inferred Himegari opcode-name overlay
|
||||
│ ├── vm0.py headless Python VM (Phase A0); `--test` = RECOVER unit test
|
||||
│ ├── extract_phase2.py batch: disasm + text + data extraction
|
||||
│ ├── extract_init.py, global_map.py … *INIT parsers, global-var map builder
|
||||
│ ├── validate_opcode_table*.py decode-coverage validators
|
||||
│ └── probe_*.py format reverse-engineering probes (historical)
|
||||
│
|
||||
├── bin/ 3rd-party binaries we use (not ours, not the game's)
|
||||
│ ├── BinExtractALF.exe ALF archive extractor → produces extracted/
|
||||
│ └── LzssCpp.dll its LZSS codec dependency
|
||||
│
|
||||
├── vm-map/ VM / reverse-engineering reference artifacts
|
||||
│ ├── opcodes-himegari.json validated opcode table (what this game uses)
|
||||
│ ├── kelebek1-age-shared.cpp / -disassembler.cpp upstream opcode-table source
|
||||
│ └── opcode-leads.json, small-script-listings.md, himegari-opcode-notes.md
|
||||
│
|
||||
├── docs/ all documentation
|
||||
│ ├── PROJECT-STRUCTURE.md this file
|
||||
│ ├── remake-architecture-and-roadmap.md THE direction doc (phases A–E)
|
||||
│ ├── phase-a-slice-plan.md the current slice (A0/A1/A2)
|
||||
│ ├── vm-mapping-plan.md the phased decode plan
|
||||
│ ├── himegari-port-reference.md master reference + engine background
|
||||
│ ├── name-resolution.md call-script + global-var name recovery
|
||||
│ ├── sys4-format-notes.md byte-level container format
|
||||
│ └── script-inventory.md what the 481 scripts are
|
||||
│
|
||||
├── build/ DERIVED (our-work-side) — generated by tools/; disposable
|
||||
│ ├── disasm/ <NAME>.asm — human-readable disassembly, one per script
|
||||
│ ├── text/ extracted text:
|
||||
│ │ ├── <NAME>.strings.txt all inline strings in a script
|
||||
│ │ ├── dialogue.jsonl show-text lines only (the translation corpus)
|
||||
│ │ └── strings.jsonl every string, tagged by source opcode
|
||||
│ ├── data/ parsed data tables (*INIT → JSON)
|
||||
│ ├── scripts-json/ machine-readable full dumps (on demand via --json)
|
||||
│ ├── global-var-map.{json,md} partial global-variable name map
|
||||
│ └── manifest.json, opcode-coverage.md
|
||||
│
|
||||
└── godot/ DELIVERABLE — the Godot/C# engine project (built in Phase A2+)
|
||||
```
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Three-way separation.** `姫狩りダンジョンマイスター/` = untouched originals; `extracted/` =
|
||||
game-derived data (regenerable, game-side); `age-reimpl/` = everything we authored. The first two
|
||||
are consumed, never modified.
|
||||
- **Tools never hard-code paths.** `tools/paths.py` derives `GAME_DIR`, `EXTRACTED`, `DATA1`,
|
||||
`BUILD`, etc. from its own location. To point the tools at a different install, edit that one file.
|
||||
The whole tree can be relocated without touching any other tool.
|
||||
- **Path references in docs** are `age-reimpl/`-relative (e.g. `tools/sys4load.py`,
|
||||
`build/text/dialogue.jsonl`) unless they name a game/extracted path explicitly.
|
||||
- **Authoritative script copies:** where a script exists both as a loose `.BIN` in the game folder
|
||||
and under `extracted/DATA1/`, the game-folder copy (patch v1.03) wins. `paths.scripts()` resolves
|
||||
this automatically (overrides win).
|
||||
- **`build/` and `extracted/` are disposable.** `build/` regenerates via `tools/extract_phase2.py`
|
||||
(or `sys4load.py`); `extracted/` regenerates via `bin/BinExtractALF.exe` on the `.ALF` files.
|
||||
Safe to delete and rebuild; do not hand-edit.
|
||||
- **Encoding:** all generated text is UTF-8 (source strings are cp932/Shift-JIS, decoded on
|
||||
extraction). Run Python as `py -3.11 -X utf8`.
|
||||
- **The game install is a runnable unit** — do not relocate `AGE.EXE`/`*.ALF`/DLLs relative to each
|
||||
other, or the game (and any Frida work) breaks.
|
||||
180
docs/himegari-port-reference.md
Normal file
180
docs/himegari-port-reference.md
Normal file
@@ -0,0 +1,180 @@
|
||||
# Princess Hunting Dungeon Meister — Godot Port Reference
|
||||
|
||||
Working reference for porting *姫狩りダンジョンマイスター* (Eushully, 2009) to Godot.
|
||||
|
||||
Source install: `C:\Program Files (x86)\Eushully\姫狩りダンジョンマイスター\`
|
||||
Workspace: `S:\Game Hacking\Eushully\Himegari\` — three siblings: `姫狩りダンジョンマイスター\` (pristine game), `extracted\` (extracted ALF data), `age-reimpl\` (our work). See `docs/PROJECT-STRUCTURE.md`.
|
||||
|
||||
---
|
||||
|
||||
## Engine background
|
||||
|
||||
Eushully built their own engine, called **AGE**, with three major generations distinguished by their config/index files and archive magic bytes:
|
||||
|
||||
| Generation | Index file | ALF magic | Era (rough) |
|
||||
|---|---|---|---|
|
||||
| SYS3 | `sys3ini.bin` | `S3IC` / `S3IN` | mid-2000s |
|
||||
| SYS4 | `sys4ini.bin` / `.AAI` | `S4AC` / `S4IC` | late-2000s — early-2010s |
|
||||
| SYS5 | `sys5ini.bin` | `S5IN` / `S5IC` / `S5A` | ~2013+ |
|
||||
|
||||
**Confirmed SYS4** for this game — `extracted\DATA1\SYSTEM4.BIN` is the smoking gun, and the `APPEND01.AAI` sidecar is the SYS4 signature pattern.
|
||||
|
||||
Install dir contents: `AGE.EXE`, `DATA1-5.ALF`, `APPEND01.ALF`, `APPEND01.AAI`, plus `BinExtractALF.exe` and `AGE Patch.exe`.
|
||||
|
||||
---
|
||||
|
||||
## Tier 1 — Primary references
|
||||
|
||||
### [Kelebek1/Eushully-Decompiler](https://github.com/Kelebek1/Eushully-Decompiler)
|
||||
C++ decompiler + recompiler for `.bin` scripts, with `extract_alf.py` for archive extraction.
|
||||
|
||||
- Targets **SYS5** specifically (its extractor scans for `SYS5INI.BIN` and `S5*` magic). Will need adaptation for SYS4, but the structure is parallel and the opcode work is gold as a starting framework.
|
||||
- Most actively maintained (latest release Aug 2024). 22 stars but the most serious decompiler effort in the wild.
|
||||
- Known limitation: opcode tables are hand-built per engine version — community has open requests to add more opcodes.
|
||||
- See [extract_alf.py](https://github.com/Kelebek1/Eushully-Decompiler/blob/master/extract_alf.py) for the cleanest reference implementation of ALF parsing.
|
||||
|
||||
### [morkt/GARbro](https://github.com/morkt/GARbro)
|
||||
Swiss-army VN asset extractor. Has [`ArcFormats/Eushully/ArcALF.cs`](https://github.com/morkt/GARbro/blob/master/ArcFormats/Eushully/ArcALF.cs) covering **all three engine generations** (SYS3/4/5), plus contained formats AGF (graphics), AOG (audio), SCR.
|
||||
|
||||
- The more directly applicable extractor for this game — use it as the authoritative reference for the SYS4 ALF format and LZSS decompression of the index.
|
||||
- C# / .NET. Source is readable and well-organized per format.
|
||||
|
||||
## Tier 2 — Supporting tools
|
||||
|
||||
### [marcussacana/EushullyEditor](https://github.com/marcussacana/EushullyEditor)
|
||||
C# library for string-level edits to `.bin` scripts (translation-focused, not full decompilation).
|
||||
|
||||
- Pre-configured for *Kamidori Alchemy Meister* and *Kami no Rhapsody*.
|
||||
- Useful as a second reference for the script string-table structure, especially since Kamidori is the same SYS4 era as Dungeon Meister.
|
||||
|
||||
### Existing tools already on disk
|
||||
- `BinExtractALF.exe` — pre-built ALF extractor someone already ran here. Worth confirming origin (likely from a Japanese tools site or HongFire / Mikocon thread).
|
||||
- `extracted\DATA1\AGF2BMP2AGF.exe` + `LzssCpp.dll` — bidirectional AGF↔BMP converter for the proprietary graphics format. Original by **asmodean** ([asmodean.reverse.net](http://asmodean.reverse.net/) — canonical source of older Japanese game tooling, worth bookmarking).
|
||||
- `AGE Patch.exe` — likely an English localization or no-DVD patch, not a tool per se.
|
||||
|
||||
---
|
||||
|
||||
## Current extraction state
|
||||
|
||||
| Archive | Files | Size | Contents |
|
||||
|---|---|---|---|
|
||||
| DATA1 | 3,749 | 2.2 GB | **Mixed core data**: 1511 AGF + 1508 BMP (graphics, ~all already converted), **481 .BIN scripts**, 238 WAV (system sfx), 9 cursors, the AGF2BMP2AGF tool + LzssCpp.dll |
|
||||
| DATA2 | 985 | 791 MB | Event CGs (`.AGF` only) — pure graphics archive |
|
||||
| DATA3 | 39 | 124 MB | BGM tracks (`.OGG` only) |
|
||||
| DATA4 | 9,733 | 471 MB | Voice files (`.OGG` only) — character lines |
|
||||
| DATA5 | 210 | 291 MB | More `.AGF` graphics — likely appendix/extra content |
|
||||
|
||||
### Notable scripts already extracted (DATA1)
|
||||
- `SYSTEM4.BIN` — engine config/setup (the SYS4 index)
|
||||
- `HISTORY.BIN`, `MENU.BIN`, `HIDEWIN.BIN` — UI/system scripts
|
||||
- `SC0000.BIN` and presumably hundreds of `SC####.BIN` — scene scripts, the actual game logic
|
||||
|
||||
### Status by category
|
||||
- **Graphics**: 1508/1511 in DATA1 already converted to BMP. ~1,195 AGFs remaining in DATA2 and DATA5.
|
||||
- **Audio**: Fully extracted as standard OGG Vorbis (9,772 files). Godot ingests natively — no further work.
|
||||
- **Scripts**: 481 `.BIN` files extracted but still in AGE bytecode form. This is the real porting work.
|
||||
|
||||
---
|
||||
|
||||
## Next steps
|
||||
|
||||
### Where things stand (read this first)
|
||||
Container format is **fully reversed and machine-verified** (header, 4 sections, 3
|
||||
typed pointer tables, inline string encoding — all 481 scripts parse clean). Tooling
|
||||
exists: `tools/sys4load.py` (loader + disassembler-ish dumper + `--validate`) and the
|
||||
`tools/probe_*.py` analysis scripts. Companion docs: [script-inventory.md](script-inventory.md)
|
||||
(what the 481 scripts are) and [sys4-format-notes.md](sys4-format-notes.md) (the byte format).
|
||||
|
||||
**The one blocker for everything downstream is opcode semantics** — the code stream is
|
||||
a tagged-dword format whose instruction meanings are unknown. That needs the VM dispatch
|
||||
loop in `AGE.EXE`, which is the Ghidra task below. Graphics conversion (DATA2/5 AGFs) and
|
||||
save-format work remain deferred.
|
||||
|
||||
### Immediate (no tools needed beyond what's on disk)
|
||||
1. ~~**Relocate the `Output\` tree**~~ **DONE** — workspace now at `S:\Game Hacking\Eushully\Himegari\姫狩りダンジョンマイスター\`.
|
||||
2. **Convert remaining AGFs** in DATA2 (985 files) and DATA5 (210 files) with `AGF2BMP2AGF.exe`. *(Deferred — graphics not needed yet.)* The 3-file DATA1 gap is `CHAPTER.AGF`, `LOGO.AGF`, `TEST.AGF`.
|
||||
3. ~~**Inventory the script files**~~ **DONE** — see [script-inventory.md](script-inventory.md). Key findings: all 481 scripts share magic `SYS4422 `; 52 loose root-dir `.BIN` files are patch overrides that shadow DATA1 copies (use those as authoritative); heavy game logic (damage calc, dungeon loop, battle flow) lives in bytecode, favoring a VM re-implementation in Godot.
|
||||
1. ~~**Relocate the `Output\` tree**~~ **DONE** — workspace now at `S:\Game Hacking\Eushully\Himegari\姫狩りダンジョンマイスター\`.
|
||||
2. **Convert remaining AGFs** in DATA2 (985 files) and DATA5 (210 files) with `AGF2BMP2AGF.exe`. *(Deferred — graphics not needed yet.)* The 3-file DATA1 gap is `CHAPTER.AGF`, `LOGO.AGF`, `TEST.AGF`.
|
||||
3. ~~**Inventory the script files**~~ **DONE** — see [script-inventory.md](script-inventory.md). Key findings: all 481 scripts share magic `SYS4422 `; 52 loose root-dir `.BIN` files are patch overrides that shadow DATA1 copies (use those as authoritative); heavy game logic (damage calc, dungeon loop, battle flow) lives in bytecode, favoring a VM re-implementation in Godot.
|
||||
|
||||
### Header structure — DONE (hex-first, pre-Ghidra)
|
||||
See [sys4-format-notes.md](sys4-format-notes.md). Confirmed across all 481 files: 60-byte header (magic `SYS4422 ` + 13 u32 fields, all offsets in dwords), body split into CODE + 3 typed pointer tables (tags 0x71/0x03/0x8F, 1 dword each, 100% pure) + inline strings. Strings are XOR-0xFF cp932, referenced by a `0x02 <dword-offset>` tagged operand — verified by decoding real dialogue out of `SC0030.BIN`. Remaining unknowns (opcode dispatch, flag fields F0/F2/F3/F5) need the VM.
|
||||
|
||||
### NEXT ACTION — port the opcode table (see [vm-mapping-plan.md](vm-mapping-plan.md))
|
||||
**The full phased playbook lives in [vm-mapping-plan.md](vm-mapping-plan.md)** — start there.
|
||||
|
||||
> ✅ **BREAKTHROUGH (verified 2026-07-05): the opcode set is already solved.**
|
||||
> Kelebek1's decompiler (`age-shared.cpp`) ships an AGE opcode table that decodes
|
||||
> this game directly — **476/476 scripts decode 100% clean, 1.46M instructions, 0
|
||||
> unknown opcodes, 37,392/0 string args resolved.** Model: code = instructions of
|
||||
> `<opcode> + argc*(<type><value>)`, length `1+2*argc`; stop code at the first inline
|
||||
> string offset. Himegari uses 248 opcodes, 52 named (see `vm-map/opcodes-himegari.json`).
|
||||
> Header fields F0–F5 are now known = local-variable counts (Kelebek's `BinaryHeader`).
|
||||
> **Unpacking `AGE.EXE` is no longer the blocker** — it's demoted to optional Phase 3
|
||||
> enrichment (prefer Frida hooking). Reproduce: `tools/validate_opcode_table.py`.
|
||||
>
|
||||
> ⚠️ **Coverage nuance (measured 2026-07-06):** "solved" means every instruction *decodes*
|
||||
> (structure/length known, 481/481 clean). It does **not** mean every instruction is
|
||||
> *understood*: the 52 named ops are only **72.6% of instruction volume**; the unnamed
|
||||
> `u004xxxx` 27.4% is concentrated in the highest-frequency opcodes and can't be fully
|
||||
> deferred before the Godot VM. Also the 52 semantic *labels* come from a later AGE title —
|
||||
> numbers+argc are validated for Himegari, semantics are not (text ops confirmed by the
|
||||
> dialogue corpus; effectful ops need Frida confirmation). See `vm-mapping-plan.md` Phase 3.
|
||||
|
||||
> ⚠️ **Note on `AGE.EXE`:** still packed (max-entropy sections, IAT RVA 0). Only
|
||||
> relevant if you later need to name the 196 unnamed opcodes statically — see the
|
||||
> plan's appendix. `SYS4AB.BIN` (magic `S4AB`) is a 2nd encrypted engine image.
|
||||
|
||||
Original Ghidra sketch (superseded; kept only for the appendix unpack route):
|
||||
|
||||
4. **Find the dispatch loop.** Look for where `AGE.EXE` reads a script's first body
|
||||
dword (`0x259` in 301/481 files — likely the prologue/entry opcode) and switches on
|
||||
dword tag values. Expect a large switch or jump table. Diff against Kelebek1's SYS5
|
||||
opcode-handler addresses to map SYS4 equivalents (same engine family, parallel structure).
|
||||
5. **Seed the opcode map from known anchors** (from `sys4load.py` dumps, already observed):
|
||||
- `0x02` = string-pointer operand tag (confirmed).
|
||||
- `0x1A7` / `0x1A5` = opcodes that immediately precede string refs in `MENU.BIN` →
|
||||
candidate text/message-display instructions. Start here; they're the easiest to confirm.
|
||||
- `0x8F` / `0x71` / `0x03` = the tags at T3-line / T1-label / T2-data table targets.
|
||||
- `0x55` = most frequent code lead (likely statement/expr separator); `0x09` = recurring
|
||||
operand-type prefix (register/var ref?).
|
||||
Cross-reference with marcussacana's Kamidori (same SYS4 era) as a second opcode source.
|
||||
6. **Encode the opcode table into `sys4load.py`.** As each opcode's length + operand
|
||||
grammar is confirmed in Ghidra, add it so the dumper decodes real instructions instead
|
||||
of chunking by the T3 line-index. `MENU.BIN` (3 strings, small control flow) is the
|
||||
validation target — decode it fully first, then a mid-size `SC####` scene end-to-end.
|
||||
7. **(Optional sanity check)** Locate the ALF mount/decrypt code to confirm GARbro's parser
|
||||
matches this build. Low priority — extraction already succeeded, so this is only if an
|
||||
archive anomaly shows up.
|
||||
|
||||
### Data-table extraction (unblocks in parallel once a few opcodes are known)
|
||||
8. The `*INIT` giants (`STINIT` 579 KB stages, `EBINIT` 338 KB enemies, `MPINIT` 330 KB
|
||||
maps, `ITINIT` items, `SKINIT` skills, `CGINIT` gallery) are static data tables. Once
|
||||
the T2/data-entry grammar is understood they can be dumped to JSON/CSV **without** a
|
||||
complete opcode set — an early, high-value win for the game database.
|
||||
9. **Bulk-extract all dialogue** — `sys4load.py --strings` already pulls clean cp932 text
|
||||
from every scene today. A batch run over all `SC####`/`SP####` yields the full script
|
||||
corpus for translation, independent of the VM work.
|
||||
|
||||
### Deferred tracks
|
||||
10. **Save-file format** — reverse `SAVE.BIN` / the save layout only if the port must read
|
||||
existing saves. Likely a small struct dump; low priority until gameplay runs.
|
||||
11. **Godot representation** — decide: re-implement the AGE VM in GDScript/C#, or transpile
|
||||
`.BIN` → native Godot scenes. The inventory already tilts toward **re-implementing the
|
||||
VM** (damage calc, dungeon loop, battle flow all live in bytecode, so a transpiler would
|
||||
have to cover nearly the whole opcode set anyway). Ghidra's view of how much logic sits
|
||||
in `AGE.EXE` vs. bytecode makes the final call.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [Kelebek1/Eushully-Decompiler](https://github.com/Kelebek1/Eushully-Decompiler)
|
||||
- [Kelebek1/Eushully-Decompiler/extract_alf.py](https://github.com/Kelebek1/Eushully-Decompiler/blob/master/extract_alf.py)
|
||||
- [Kelebek1issue #2 — AGE engine version discussion](https://github.com/Kelebek1/Eushully-Decompiler/issues/2)
|
||||
- [marcussacana/EushullyEditor](https://github.com/marcussacana/EushullyEditor)
|
||||
- [morkt/GARbro](https://github.com/morkt/GARbro)
|
||||
- [GARbro ArcFormats/Eushully/ArcALF.cs](https://github.com/morkt/GARbro/blob/master/ArcFormats/Eushully/ArcALF.cs)
|
||||
- [Eushully Fandom Wiki — Princess Hunting Dungeon Meister notes](https://eushully.fandom.com/wiki/Princess_Hunting_Dungeon_Meister:Notes)
|
||||
- [asmodean.reverse.net](http://asmodean.reverse.net/) — canonical home of older Japanese game tooling
|
||||
130
docs/name-resolution.md
Normal file
130
docs/name-resolution.md
Normal file
@@ -0,0 +1,130 @@
|
||||
# Name resolution — recovering what the compiler stripped
|
||||
|
||||
The disassembler reads the SYS4 bytecode's **operations and control flow** cleanly (see any
|
||||
`build/disasm/*.asm`). What it can't show is the two kinds of *names* the AGE compiler
|
||||
discarded: **which function a call targets** (#1) and **what a global variable means** (#2).
|
||||
Both are data-labeling problems, not decoding problems. This note records what each is, what
|
||||
we found, and how tractable it is.
|
||||
|
||||
Motivating example: `RECOVER.BIN` translates to correct pseudocode today, but reads as
|
||||
`call-script 0x329d` (#1) and `C[unit][s] = E[unit][s]` over raw addresses (#2). Naming those
|
||||
would make it read like source.
|
||||
|
||||
---
|
||||
|
||||
## #1 — `call-script` target resolution (naming the call graph)
|
||||
|
||||
**What it is.** `call-script N` (Kelebek opcode 0x03) carries a bare number — `0x329d`,
|
||||
`0x2ade` — the id of an engine entry point. To render `call RECOVER` instead of
|
||||
`call-script 0x329d` you need a table `id → (script, entry)`.
|
||||
|
||||
**Findings (inspected 2026-07-06):**
|
||||
- `SYSTEM4.BIN` is **not** an index — it's a small SYS4 script (375 instrs) titled
|
||||
"SYSTEM4 INIT", the engine boot/init routine (ADV mode, fonts, error text).
|
||||
- `SYS4INI.BIN` (`S4IC422`) is the **ALF asset index** — archive filenames for extraction
|
||||
(`SYSTEM4.BIN`, `M002.OGG`, `EV049A.AGF`…), not a script-call registry.
|
||||
- The ids are large and sparse (`0x329d` = 12,957 ≫ 481 scripts), so the number is an index
|
||||
into a global **entry-point registry** the engine builds, not a script-file index.
|
||||
- Even Kelebek's reference decompiler leaves these numeric (its comment only says "param =
|
||||
SYSTEM4.bin index"). So this is genuinely **unresolved upstream**, not merely unfinished.
|
||||
|
||||
**Why it's engine-level (harder than a file lookup).** There is no `id → name` table sitting
|
||||
on disk to read. Resolving it needs one of:
|
||||
- ~~Decode `SCJUMP.BIN`~~ **RULED OUT as the registry (recon 2026-07-06).** `SCJUMP.BIN` (29,796
|
||||
instrs) is a **progression state machine**, not an id→code table: it switches on `global 0x3234`
|
||||
(mode 1–9) then nested `eq`/`ne`/`and`/`jcc` on flags, ending in `mov`s to output globals. It
|
||||
decides *what comes next* via state; it barely uses `call-script`. Useful for game-flow logic, not
|
||||
for resolving `call-script` ids. So the id→code registry is genuinely engine-level.
|
||||
- **Watch the engine resolve one (Frida)** — breakpoint the `call-script` handler in the running
|
||||
game, log `id → resolved address/script`. Ground truth; Phase-3 (live-tools) work.
|
||||
- **Find the registration path** — if a boot script assigns ids to entry points, extract it
|
||||
statically (SYSTEM4.BIN is far too small to hold ~13k, so it's cumulative or lives in AGE.EXE).
|
||||
|
||||
**Status: deferred.** Not the quick win first assumed. Belongs with the engine/dispatch work
|
||||
(Phase 3), or a dedicated `SCJUMP.BIN` reverse. Until then `call-script` stays numeric.
|
||||
|
||||
---
|
||||
|
||||
## #2 — The global-variable map (naming the data)
|
||||
|
||||
**What it is.** The VM has one flat **global memory bank**; the bytecode addresses it by raw
|
||||
offset (`global-int 0x152616`, `global-int 0x52383`). Each offset is a specific piece of game
|
||||
state (a unit's HP, the current-unit index, a stat table). The map we want is
|
||||
`offset → (name, type, structure)`.
|
||||
|
||||
**Why it's opaque.** No symbol table exists anywhere; meaning lives in how AGE.EXE and the
|
||||
scripts *use* each global. Nothing declares "0x152616 is the current unit."
|
||||
|
||||
**Why a big chunk is recoverable statically (the tractable one).** Unlike #1, #2 has strong
|
||||
free handholds — several of which we've already built:
|
||||
|
||||
1. **The `*INIT` scripts are the writers, and we already extracted them.** `EBINIT`/`ITINIT`/
|
||||
`SKINIT`/`CGINIT`/`MPINIT` populate global arrays with names and data (`build/data/*.json`).
|
||||
The base address `EBINIT` writes 277 unit names into *is* the unit-name table. Each JSON's
|
||||
`name_array_base`, `desc_array_bases`, and `field_columns` are literally global addresses we
|
||||
can label by which table wrote them.
|
||||
2. **Strings anchor the string side for free.** `set-string` writes skill names to
|
||||
`global-string 0x23a3…` → that array is the skill-name table. `*MES` tables likewise.
|
||||
3. **Access shape reveals structure without names.** A global read as `base[unit*stride + col]`
|
||||
exposes a per-unit record and its width (RECOVER showed 14-, 3-, 30-column tables). A global
|
||||
used as the loop-invariant row index everywhere (`0x152616`) is a "current X" pointer.
|
||||
Constants-compared → mode/flag; only-incremented → counter.
|
||||
4. **Frida for the ambiguous ones (heavy, ground truth).** Do a known action in-game (take
|
||||
damage, gain a level), watch which global changes → definitive labels. Reserve for leftovers.
|
||||
|
||||
**Feasibility.** A *partial* map — enough to make most gameplay scripts readable — is achievable
|
||||
now, statically, from methods 1–3. A *complete* map needs Frida for the tail. It's incremental:
|
||||
label the ~dozen hottest globals first (biggest readability payoff), grow the rest on demand.
|
||||
|
||||
**Partial map — BUILT (v1, 2026-07-06).** `tools/global_map.py` → `build/global-var-map.json`
|
||||
(all evidence) + `build/global-var-map.md` (labelled subset). It ingests `build/data/*.json`
|
||||
(name/desc/field bases), scans the 481-script corpus for each global's **access shape**
|
||||
(2D-table base + stride, 1D-array base, row-index, scalar), and ranks "current entity" index
|
||||
pointers by purity. **First result: 16,354 of 49,435 distinct globals labelled** —
|
||||
|
||||
| kind | count | example |
|
||||
|---|---|---|
|
||||
| string tables (names/descs/messages) | 3,199 | `0x23a3` = skill-name table |
|
||||
| per-entity data-field arrays (from *INIT) | 12,700 | dense = shared fields, `?` = sparse per-entity |
|
||||
| row-major record tables (from access shape) | 122 | `0x52383` = record-table[stride 30] |
|
||||
| 1D arrays | 307 | |
|
||||
| index / "current entity" pointers | 26 | `0x152616` (purity 0.51), `0xeff75` (0.95) |
|
||||
|
||||
**Validated against `RECOVER`:** the map independently reproduces its hand-traced layout —
|
||||
`0x4e11b`→stride 14, `0x52383`→stride 30, `0xaacb4`→1D array, `0x152616`→current-entity index.
|
||||
|
||||
**Wired into the disassembler.** `sys4load` annotates global operands with the map's high/medium
|
||||
-confidence labels (low-confidence tail omitted for readability), e.g. RECOVER now renders
|
||||
`lookup-array-2d p0 (global-int 0x4e11b =rec[s14]) (global-int 0x152616 =current-entity-index?) …`.
|
||||
Labels are prefixed `=` to mark them as inferred aliases. Regenerate the `.asm` corpus with
|
||||
`tools/extract_phase2.py` after refreshing the map. Turn it off by deleting/renaming
|
||||
`build/global-var-map.json` (the loader degrades gracefully).
|
||||
|
||||
Confidence is marked per entry; labels ending `?` are low-confidence guesses.
|
||||
|
||||
### Future step — growing the map (planned, not yet done)
|
||||
|
||||
The v1 map labels *shapes and tables*; the next increments add *meaning*, cheapest first:
|
||||
|
||||
1. **Fold in the `*MES` message-table writers** (`ITMES`, `SKMES`, `VIMES`, …) and any other
|
||||
`set-string`/`copy-to-global` writers not covered by the `*INIT` set — pure static win,
|
||||
extends the string/data labels. (Also: most name-table bases are *read* rarely — reads
|
||||
likely go through `*MES`/an indirection; tracing that would connect names to their readers.)
|
||||
2. **Label 2D record tables by their readers** — cross-reference which scripts read each
|
||||
`rec[sN]` table and infer purpose from context (e.g. RECOVER's 30-wide tables ↔ a
|
||||
status/recovery system). Static, medium effort.
|
||||
3. **Name *which stat* each field is (Frida).** The one step needing live tools: change a
|
||||
known value in-game (take damage, gain XP), watch which global moves → definitive
|
||||
`field@X = "HP"`. Reserve for the fields that matter; this is the last mile.
|
||||
|
||||
Re-run `tools/global_map.py` after each increment; `sys4load` picks up the new labels
|
||||
automatically (it reads `build/global-var-map.json` at load).
|
||||
|
||||
---
|
||||
|
||||
## How the two relate
|
||||
#1 names **functions** (the call graph); #2 names **data** (game state). In `RECOVER`, #1 turns
|
||||
`call-script 0x329d` into a name; #2 turns `C[unit][s] = E[unit][s]` into `unit.hp[s] =
|
||||
unit.maxHp[s]`. Priority reversal from the first guess: **#2 is the tractable readability lever
|
||||
now** (static handholds already half-built via the `*INIT` extraction); **#1 needs the engine**
|
||||
(dispatch reverse or Frida) and is deferred.
|
||||
182
docs/phase-a-slice-plan.md
Normal file
182
docs/phase-a-slice-plan.md
Normal file
@@ -0,0 +1,182 @@
|
||||
# Phase A — Vertical Slice Plan (the first build step)
|
||||
|
||||
Concrete execution plan for Phase A of `remake-architecture-and-roadmap.md`. Decided over the
|
||||
alternative (fully decoding `SCJUMP.BIN`) after recon showed SCJUMP is not the gating unknown.
|
||||
|
||||
## Why the slice, and why headless-first
|
||||
|
||||
**SCJUMP recon (2026-07-06):** `SCJUMP.BIN` is a 29,796-instruction **progression state machine**,
|
||||
not the `call-script` registry. Top level switches on `global 0x3234` (mode 1–9 → big blocks); each
|
||||
block is nested `eq`/`ne`/`and`/`jcc` on flags, ending in `mov`s to output globals. Almost no
|
||||
`call-script`. So it decides *what scene/branch comes next* via state, and does **not** resolve
|
||||
`call-script id → code`. Consequence: the id→code registry stays engine-level (deferred), **but the
|
||||
slice can stub `call-script`** — it is not gating for running one scene's dialogue.
|
||||
|
||||
**Correctness bootstrap (roadmap §5) drives the ordering:** the VM must be *validated-correct* before
|
||||
it is trustworthy. Our strongest oracle is `build/text/dialogue.jsonl` (the `show-text` lines per
|
||||
script). So the very first slice is **headless and text-only, validated by that oracle** — no Godot,
|
||||
no AGF, no audio, no dispatch registry. Only once the VM reproduces dialogue do we add rendering.
|
||||
|
||||
Phase A therefore splits:
|
||||
- **A0 — headless VM, dialogue-validated (Python prototype).** ← immediate, executable now.
|
||||
- **A1 — port the validated model to C#** (the runtime's VM core).
|
||||
- **A2 — Godot ADV backend** (render one scene with visuals + voice).
|
||||
|
||||
---
|
||||
|
||||
## A0 — Headless VM validated by the dialogue oracle
|
||||
|
||||
**Goal:** a Python interpreter that executes one ADV scene's bytecode and emits its `show-text`
|
||||
sequence; that sequence is a coherent, in-order subsequence of the script's static `dialogue.jsonl`
|
||||
lines. This proves the execution model — control flow, operand/pointer semantics, string handling,
|
||||
and the no-op-marker assumptions — *before* any C#/Godot investment. Reuses `tools/sys4load.py` for
|
||||
all parsing/decoding (no new parser).
|
||||
|
||||
### Execution model to implement
|
||||
- **Memory:** one flat **global bank** = `dict[int,int]` (globals are raw offsets into one space;
|
||||
`global-int A` ⇒ `G[A]`, default 0). Per-call **local frame** with typed banks sized by header
|
||||
F0–F5 (`local_int[F0]`, `local_float[F1]`, `local_string[F2]`, …).
|
||||
- **PC / control flow:** build `offset→instruction-index` map from `sys4load` instructions (each has
|
||||
`.offset` = dword index; jump targets are dword indices). `jmp t` → pc = map[t]. `jcc(cond, A, B)`
|
||||
→ cond truthy ? goto A : goto B, where `0xffffffff` = fall through (confirmed model from RECOVER).
|
||||
- **Operand resolution by type:** imm→value; global-int→`G[value]`; local-int→`frame.int[value]`;
|
||||
string(2)→decoded string at dword offset; float/global-string/etc. analogous.
|
||||
- **⚠ Pointer/lvalue semantics — the key modeling task.** RECOVER proves `-ptr` operands are
|
||||
*lvalues*: `lookup-array(dst_ptr, base, idx)` yields a *reference* to `G[base+idx]`; `mov` through a
|
||||
ptr writes to the referenced cell; reading a ptr rvalue dereferences it. Model a ptr slot as holding
|
||||
an address into the global bank; nail this so the RECOVER array-copy produces correct results (unit
|
||||
test it directly).
|
||||
- **Opcode handlers (~52 named ops):**
|
||||
- arithmetic/bit `add sub mul div mod and or sar shl` → `p1 = p2 ⊙ p3`.
|
||||
- compares `eq ne lt lte gr gre` → 0/1.
|
||||
- `mov` (incl. through ptr), `lookup-array` (`p1=mem[base+idx]`), `lookup-array-2d`
|
||||
(`p1=mem[base + i*stride + col]`), `copy-to-global`, `set-array-to`, `bit-set/reset`, `check-bit`.
|
||||
- control `jmp call jcc ret exit exit-script`.
|
||||
- string `set-string concat strlen toString`.
|
||||
- **ADV capture:** `show-text` → append (arg text) to the emitted list; `end-text-line`,
|
||||
`wait-for-input`, `set-font`, `comment` → capture/skip (no visible state).
|
||||
- **Markers → no-op (this TESTS the classification):** `0x1f4 0x1f5 0x1d5 0x1bc 0x1bf` skip;
|
||||
tentative `0x21b 0x1d2 0x258` skip — if dialogue stays correct, the no-op assumption is validated.
|
||||
- **`call-script` → STUB:** log `(id)`, return immediately. (Its dialogue belongs to other scripts;
|
||||
stubbing keeps the emitted set = this script's own lines.)
|
||||
- **Effectful (draw/texture/audio/ui/input) → STUB:** log and ignore.
|
||||
- **Unknown/other opcodes → log + no-op**, so a rare op doesn't halt the run (record coverage).
|
||||
|
||||
### Oracle & scene choice
|
||||
- **Oracle:** with calls stubbed and default state, every emitted `show-text` line must be a real
|
||||
decoded string from the script's pool, and the sequence must be an **in-order subsequence** of that
|
||||
script's `dialogue.jsonl` lines (≈ equality for a linear scene). Catches: garbage strings (bad
|
||||
operand/ptr handling), impossible ordering (bad control flow), missing/extra lines.
|
||||
- **Scene pick:** choose a **short, mostly-linear ADV scene** — high `show-text` count, low `jcc`
|
||||
density, few `call-script`. Selection step: rank `SC####`/`SP####` by
|
||||
`(show-text count) / (jcc + call-script count)`, small size. Known-good fallback: `SC0030.BIN`
|
||||
(dialogue verified). Also run a **RECOVER unit test** to validate pointer/array semantics independent
|
||||
of dialogue.
|
||||
|
||||
### Steps
|
||||
1. `tools/vm0.py`: load a script via `sys4load`, build offset→index map, frame + global bank.
|
||||
2. Implement operand resolution + the arithmetic/compare/mov/lookup/control handlers; unit-test on
|
||||
`RECOVER.BIN` (array copy + both loops must produce correct global writes).
|
||||
3. Add ADV capture + markers-as-noop + call/effectful stubs; add opcode-coverage logging.
|
||||
4. Run on the chosen linear scene; diff emitted `show-text` vs `dialogue.jsonl` (subsequence check);
|
||||
eyeball the first ~15 lines for coherence.
|
||||
5. Iterate until several scenes pass; record which ops/markers were exercised and any surprises
|
||||
(esp. whether the tentative-no-op markers hold).
|
||||
|
||||
### Success criteria (A0 done)
|
||||
- RECOVER unit test passes (pointer/array model correct).
|
||||
- ≥3 ADV scenes: emitted `show-text` is a coherent in-order subsequence of their `dialogue.jsonl`,
|
||||
no garbage strings.
|
||||
- Coverage report of which opcodes actually executed (drives A1/A2 priorities).
|
||||
- The no-op-marker assumption is confirmed or corrected with evidence.
|
||||
|
||||
---
|
||||
|
||||
### A0 result (2026-07-06) — execution model VALIDATED
|
||||
|
||||
`tools/vm0.py` built (reuses `sys4load`; ~250 lines). Results:
|
||||
- **RECOVER unit test PASSES** — all 7 checks (block-1 3-field copy, block-2 restore + flag, both
|
||||
skip-guards). The pointer/lvalue model, 2D stride indexing, both loops, and two-way `jcc` all
|
||||
execute correctly. **The core execution model is proven.**
|
||||
- **Full SC/SP oracle sweep (`vm0.py --sweep`): 282 / 294 scenes DIALOGUE-VALID = 95.9%.** Every
|
||||
emitted `show-text` line is checked (by string offset) as an in-order subsequence of the script's
|
||||
static `dialogue.jsonl` lines. **Zero STRAY and zero ORDER violations across all 294 scenes** — the
|
||||
model never emits a garbage string and never emits dialogue out of order. 279 CLEAN (valid + natural
|
||||
`exit`); 3 OK/LOOP (valid subsequence, halted by the loop-guard); 12 EMPTY; 3 skipped (no static
|
||||
show-text). SC0000 = 326 static / clean; SP0062 = 220/220 CLEAN.
|
||||
|
||||
**A0-remainder work done (2026-07-06, session 2):**
|
||||
- **Loop-guard added** (`EMIT_CAP=2`): halt a run once any single line is re-emitted a 3rd time —
|
||||
a semantic guard tied to the oracle (vs. a blind step limit), and it *classifies* the scene LOOPED
|
||||
instead of spewing garbage. The 3 zero-state spinners (SC0010/SC0600/SC0200) now terminate cleanly
|
||||
in <12k steps and their emitted lines are all valid.
|
||||
- **SP0062 "stray" was a measurement artifact**, not a bug — the precise offset-based oracle shows it
|
||||
CLEAN (220/220, natural exit). Offset-match ⟹ text-match (VM decodes each string at the same offset
|
||||
the extractor did), so CLEAN is trustworthy.
|
||||
- **`0x71` (label-def) folded into the no-op marker set** — structural, no runtime effect.
|
||||
- **Sweep + single-scene diff harness** added to `vm0.py`: `--sweep [N]` (coverage table over all
|
||||
SC/SP), `--scene NAME` (detailed diff for one script), plus `load_oracle`/`subsequence_status`.
|
||||
|
||||
**op 0x90 investigated in depth — it is input chrome, NOT a correctness hole** (full evidence:
|
||||
`vm-map/himegari-opcode-notes.md` §F). Kelebek left it "ukn"; corpus analysis resolves it:
|
||||
`0x90 x y w h tgt_a tgt_b tgt_c` (argc 7) is a **cursor/input hotspot hit-test** that branches per
|
||||
interaction outcome and **falls through to pc+1 when nothing matches** (design-confirmed: enc.len 15
|
||||
lands the next instr on the fall-through statement). It occurs ONLY in a shared ADV-chrome subroutine
|
||||
that is byte-identical in all 301 ADV scripts — **exactly 8 sites each** (5 immediate-rect buttons at
|
||||
`(684..772, 572)` toggling `G[0x6c9..0x6cd]` + 3 local-operand keyed forms), **zero scene-specific
|
||||
use**. Headless (no cursor/input) ⇒ fall through ⇒ **vm0's stub is already correct**, proven safe by
|
||||
all 279 CLEAN scenes (which contain these same 8 sites). `op 0x97` (argc 5, no targets) is its
|
||||
companion register-hotspot call. **So 0x90 stays as fall-through in A1 with confidence; it is modelled
|
||||
as a live hotspot test only in A2** (Godot input backend), confirming target→state mapping via Frida.
|
||||
|
||||
**The 12 EMPTY scenes — state-gated interactive screens, not a model failure.** Traced SC0830: it
|
||||
exits early because `G[0xaba5c]==1` gates the content; past that gate the dialogue sits behind the ADV
|
||||
input-wait loop (the hotspot-polling chrome above), so with no seeded state and no input the scene
|
||||
exits or spins before reaching text. Unlocking them = seed per-scene state + supply input →
|
||||
**Phase A2/B**, not an A0 model fix.
|
||||
|
||||
**⚠ Honest scope of the 95.9%:** the subsequence oracle proves **no-garbage / in-order**, not a
|
||||
*complete* path — inherent to a subsequence oracle run headlessly (interactive/state-gated branches
|
||||
take the no-input path by design). That anti-garbage guarantee is exactly what A0 set out to prove.
|
||||
|
||||
**Confirmed by this run:** the classified no-op markers (`0x1f4/0x1f5/0x1d5/0x1bc/0x1bf` + tentative
|
||||
`0x21b/0x1d2/0x258`, now + `0x71`) are safe as no-ops for ADV flow; `call-script` is stubbable;
|
||||
effectful ops (`draw-texture`/`create-texture`/`play-voice`/`0x1f7`/`0x202`/`0x203`/…) stub cleanly.
|
||||
|
||||
**✅ A0 COMPLETE.** Success criteria met: RECOVER unit test green (pointer/array/control-flow model
|
||||
proven); 282 ADV scenes emit clean in-order subsequences with zero garbage; coverage number recorded;
|
||||
no-op-marker assumption confirmed at scale; `op 0x90` (the last big control-flow unknown) resolved as
|
||||
input chrome whose fall-through stub is correct headless. Next = **A1** — port the model to the C# VM
|
||||
core, differential-test against `vm0.py`. 0x90/0x97 stay stubbed (correct headless); the interactive
|
||||
input path + per-scene state seeding land in **A2** (Godot backend) alongside the real hotspot model.
|
||||
|
||||
## A1 — Port the validated model to C#
|
||||
Reimplement the A0 execution model as the runtime VM core in C# (the language decision from the
|
||||
roadmap; GDScript is too slow for the loop). A0 is the reference: differential-test C# against the
|
||||
Python prototype's traces on the same scenes. Port the container parser too (or load via a shared
|
||||
spec). Deliverable: headless C# VM reproducing A0's results.
|
||||
|
||||
## A2 — Godot ADV backend (one scene, with visuals)
|
||||
Wire the C# VM's effectful ops to Godot: `show-text`/message window (+ furigana via `display-furigana`),
|
||||
`set-font`, `wait-for-input`, choices, `play-voice`/`play-bgm`, and `create-texture`/`set-texture`/
|
||||
`draw-texture`/`draw-string` for the background + sprites. Convert the scene's AGF art with the
|
||||
on-disk `AGF2BMP2AGF.exe`. Resolve just-enough `call-script`/state so the scene's setup runs (or
|
||||
hand-set the preconditions). Deliverable: **the chosen scene playable in Godot** — bg + dialogue +
|
||||
a choice + voice — matching A0's text.
|
||||
|
||||
---
|
||||
|
||||
## Risks / open questions for A0
|
||||
- **Pointer/lvalue semantics** — the main modeling risk; RECOVER is the litmus test.
|
||||
- **Initial global state** — a scene may assume preconditions from earlier flow (`SCJUMP`/prior
|
||||
scenes). Mitigation: default-zero globals + set the few a scene reads early; the subsequence oracle
|
||||
tolerates a shortened path.
|
||||
- **Runtime vs static dialogue order** — static `dialogue.jsonl` is file-order (all lines); runtime is
|
||||
execution-order (branch taken). Hence *subsequence*, not equality; pick linear scenes to tighten it.
|
||||
- **Hidden effect in a "stub"** — a stubbed effectful op that actually gates control flow could skew
|
||||
output. Watch for divergence; promote a stub to a real handler if a scene needs it.
|
||||
|
||||
## Immediate next action
|
||||
Build `tools/vm0.py` and get the **RECOVER unit test** green (pointer/array/control-flow correctness),
|
||||
then run the first linear ADV scene against the dialogue oracle. That single result tells us whether
|
||||
the whole VM approach executes correctly — the load-bearing question behind option 3.
|
||||
267
docs/remake-architecture-and-roadmap.md
Normal file
267
docs/remake-architecture-and-roadmap.md
Normal file
@@ -0,0 +1,267 @@
|
||||
# AGE Remake — Architecture & Roadmap (option 3: remake / enhance / mod)
|
||||
|
||||
**Decided goal (2026-07-06):** not a translation patch (a translated build is already playable),
|
||||
not a bare cross-platform port — but an **open reimplementation of the AGE engine that runs the
|
||||
original games and makes modding a first-class feature.** Himegari (SYS4) is the first target;
|
||||
the design must extend to other AGE games and engine versions (SYS3/SYS5).
|
||||
|
||||
The right mental model is **ScummVM / OpenMW for Eushully's AGE engine**: we ship an *engine*;
|
||||
the user supplies the *original game data* they own; mods layer on top. Those projects prove this
|
||||
shape is feasible — and that it's a large, long-lived effort. Our decoding work (done) is the
|
||||
foundation; the runtime + backends + mod system is the bulk of the remaining work.
|
||||
|
||||
---
|
||||
|
||||
## 1. Guiding principles
|
||||
|
||||
1. **Bytecode-faithful logic, pragmatic presentation.** Run the original `.BIN` scripts on a
|
||||
reimplemented VM — you get every gameplay rule (damage, dungeon, battle, recovery) correct by
|
||||
construction, without re-deriving them. Reimplement the *effectful* ops (draw/text/audio/input)
|
||||
against a clean modern backend that looks right, not byte-identical to D3D9.
|
||||
2. **One engine, many profiles.** Do NOT fork per game or per version. A shared VM core, with
|
||||
*version front-ends* (parser/codec/opcode table) and *per-game profiles* (maps, data schemas,
|
||||
asset rules) selected by a manifest.
|
||||
3. **Modding is architecture, not an afterthought.** The data model, content loading, and script
|
||||
dispatch are designed so mods can override assets, edit data, patch scripts, and inject
|
||||
host-language hooks. The engine already hints at this: 52 loose root `.BIN` files natively
|
||||
shadow their archived copies — a built-in override mechanism we generalize.
|
||||
4. **The original owns the content; we own the engine.** Users provide their AGE install; the
|
||||
runtime imports/loads it. This keeps us on the right side of distribution and mirrors ScummVM.
|
||||
5. **De-risk with vertical slices.** Prove "run one scene end-to-end" before breadth. Nothing is
|
||||
real until a script executes and renders.
|
||||
|
||||
---
|
||||
|
||||
## 2. Target architecture — the split
|
||||
|
||||
Three layers, cleanly separated:
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ RUNTIME (Godot + C#) — ships to players & modders │
|
||||
│ ├─ AGE VM core (C#) fetch/execute, typed var banks, control flow │
|
||||
│ ├─ Version front-ends SYS3 / SYS4 / SYS5: header, string codec, │
|
||||
│ │ opcode table, archive format │
|
||||
│ ├─ Backend adapters render(Godot 2D) · text/msg · audio · input │
|
||||
│ │ — the effectful opcodes call into these │
|
||||
│ ├─ Content loader ALF archives + loose overrides + mod folders │
|
||||
│ ├─ Data layer game data from moddable files (bootstrapped │
|
||||
│ │ from *INIT extraction) │
|
||||
│ └─ Mod system override resolution · hook API · patch loader │
|
||||
├──────────────────────────────────────────────────────────────────────┤
|
||||
│ PROFILE / MANIFEST (per game) — data, not code │
|
||||
│ engine_version, archives, string_codec, opcode_table_ref, │
|
||||
│ global_var_map, callscript_map, data_schemas, boot_entry, │
|
||||
│ asset_conversion rules │
|
||||
├──────────────────────────────────────────────────────────────────────┤
|
||||
│ TOOLCHAIN (Python — what we've already built) — offline, for modders │
|
||||
│ disassembler · assembler · extractors · global-map builder · │
|
||||
│ data exporters · mod packaging │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
- **Runtime language.** VM core in **C#** (Godot's C# support) — a 1.5M-instruction fetch/execute
|
||||
loop is too hot for GDScript. Presentation, UI, mod tooling, and export targets use Godot. This
|
||||
is why **Godot now fits**: under the earlier "faithful port" framing it was overkill (you'd use
|
||||
~10% of it); under *remake/enhance/mod* its editor, UI toolkit, asset pipeline, GDScript modding,
|
||||
and multi-platform export all earn their keep.
|
||||
- **Toolchain vs runtime.** The runtime owns the canonical parser+VM (C#). The Python tools remain
|
||||
the offline analysis/authoring chain; they were the reference implementation and stay useful for
|
||||
modders. They share the *format spec* (documented), not code — acceptable for a small, stable
|
||||
container format.
|
||||
- **Profile = manifest.** Adding a game = a new profile + its maps. Adding an engine version = a new
|
||||
front-end plugin + profiles that reference it. See §5.
|
||||
|
||||
---
|
||||
|
||||
## 3. How modding works with a bytecode VM
|
||||
|
||||
Modding is tiered from trivial to deep. The first two tiers cover the large majority of "proper
|
||||
modding" and need **no decompilation**.
|
||||
|
||||
**Tier 1 — assets & data (easy, no tools needed beyond a text/image editor).**
|
||||
- *Asset overrides:* drop replacement textures/CGs/voices/BGM into a mod folder; the content loader
|
||||
resolves mod → loose-override → archive (generalizing the engine's native override behavior).
|
||||
- *Data edits:* game data (skills/items/units/maps/stages) is **externalized to editable files**
|
||||
(JSON) that the runtime loads, bootstrapped from our `*INIT` extraction. Rebalancing, new items,
|
||||
new skills = editing JSON. No bytecode involved.
|
||||
|
||||
**Tier 2 — logic (medium; asm-level or host-language).**
|
||||
- *Script patches:* disassemble → edit the `.age-asm` → reassemble to `.BIN` (Kelebek's project has
|
||||
a reassembler to adapt). Mods ship patched/replacement scripts; the VM runs them unmodified.
|
||||
- *Host hooks:* a mod API lets mods register callbacks in **GDScript/C#** — fire before/after a
|
||||
script, intercept an opcode, replace a script by id, react to events, add UI. Original scripts run
|
||||
as-is; mods augment. (This is the BepInEx/script-extender model and avoids a bytecode compiler for
|
||||
most behavioral mods.)
|
||||
|
||||
**Tier 3 — a friendly modding language (stretch, later).**
|
||||
- A high-level decompiled DSL + a compiler back to bytecode, so mods are written in readable source.
|
||||
This is a real compiler project and its quality is bounded by how complete the global-var map and
|
||||
call-script resolution are. Realistic as a *later* milestone, not near-term.
|
||||
|
||||
**Readability, concretely:** annotated disassembly is achievable today (opcode names + global-var
|
||||
aliases + eventually call-script names). Pseudo-decompilation for *reading* is feasible (demonstrated
|
||||
on RECOVER). Clean round-trippable *source* is Tier 3. So near-term "how readable" = well-annotated
|
||||
assembly + external data/assets + host hooks; the read-like-C dream is a stretch goal.
|
||||
|
||||
**Two enablers become load-bearing under this goal** (they were "polish" for a port):
|
||||
- **Global-var map** — modders must know what game state a global is to touch it safely.
|
||||
- **Call-script resolution** — needed both to *run* scripts and to *add/replace* scenes. This is now
|
||||
on the critical path, not deferred.
|
||||
|
||||
---
|
||||
|
||||
## 4. Roadmap — high-level progression
|
||||
|
||||
Each phase ends with something demonstrable. The documented side-tasks map into these phases (noted).
|
||||
|
||||
### Phase A — Prove the VM (vertical slice) ⟵ the immediate priority
|
||||
Goal: **run one ADV scene end-to-end** in the new runtime — background image + dialogue + a choice +
|
||||
a voice line — and match its `show-text` sequence to `build/text/dialogue.jsonl`.
|
||||
Forces, and thereby de-risks, every core unknown at once:
|
||||
- Port the container parser + VM core to C#.
|
||||
- Implement the ADV effectful ops against Godot: `show-text`, `end-text-line`, `wait-for-input`,
|
||||
`set-font`, `play-voice`, `play-bgm`, `draw-texture`/`create-texture`/`draw-string`, choices.
|
||||
- Resolve **just enough `call-script`** to enter/leave a scene (side-task: `SCJUMP.BIN` decode or a
|
||||
targeted Frida capture — now critical-path).
|
||||
- **AGF → texture** for the one scene's art (side-task; `AGF2BMP2AGF.exe` already on disk).
|
||||
- Treat the classified no-op markers as skips; validate the tentative-no-op ops via the dialogue diff.
|
||||
|
||||
### Phase B — Broaden coverage (playable ADV, then systems)
|
||||
- Implement the remaining effectful ops; Frida sessions for the opaque ones (the shortlist in
|
||||
`build/opcode-coverage.md`); Unicorn for `0x215`-style computational ops.
|
||||
- Grow the **global-var map** (side-task 2.5: `*MES` writers → record-table readers → Frida field
|
||||
naming) — now a core enabler, not polish.
|
||||
- Get a full chapter of ADV playable; then the dungeon/battle/menu systems (they run as bytecode —
|
||||
we implement the ops they use, not the rules).
|
||||
- Side-tasks absorbed here: `STINIT` parser, remaining data schemas, save-file format (needed for a
|
||||
real playthrough — reversible struct work).
|
||||
|
||||
### Phase C — Externalize & modding foundation
|
||||
- Move game data from bytecode-embedded tables to **editable external files** the runtime loads.
|
||||
- Generalize the **override/mod-loading** (mod folders, load order) from the engine's native
|
||||
loose-file mechanism.
|
||||
- Asset pipeline: AGF↔PNG, audio, packaging. → Tier-1 modding works.
|
||||
|
||||
### Phase D — Logic modding
|
||||
- Integrate the **assembler** (Tier-2 bytecode-patch mods) and ship the **host hook API**
|
||||
(GDScript/C#). → Tier-2 modding works.
|
||||
- Optionally invest in decompiler quality toward Tier 3.
|
||||
|
||||
### Phase E — Enhance, polish, productize
|
||||
- Enhancements the VM unlocks: higher/wide resolution, faster text, QoL, save-anywhere, new-content
|
||||
mods. Modding docs + tools. Save/UX polish.
|
||||
|
||||
---
|
||||
|
||||
## 5. The VM as our analysis instrument — and the correctness bootstrap
|
||||
|
||||
Beyond being the runtime, the VM is the best analysis tool we can build — **but only once it is
|
||||
validated-correct, and that ordering is load-bearing.**
|
||||
|
||||
**Upside: owning the VM turns static RE into dynamic observation.** Instrumenting the original
|
||||
packed `AGE.EXE` means fighting an anti-debug binary with Frida; instrumenting *our* interpreter is
|
||||
one line in a handler. So a class of documented side-tasks become built-in debugger views instead of
|
||||
separate investigations:
|
||||
- **Live named-global watch** — with the global-var map, watch state change in real time; take damage
|
||||
→ see which global moved → *that is* the "name which stat" step (`name-resolution.md` → Future),
|
||||
now a debugger feature, not a Frida session.
|
||||
- **Call-graph / dispatch trace** — watch `call-script` resolve live, helping crack the entry-point
|
||||
registry dynamically.
|
||||
- **Breakpoints, single-step, var-bank inspection, opcode/script/global coverage**; and because VM
|
||||
state is just variable banks + a program counter, cheap **snapshot / rewind** (time-travel
|
||||
debugging nearly falls out of the design).
|
||||
|
||||
**The hard caveat: the instrument is only as trustworthy as the VM is correct.** A VM that executes
|
||||
*wrong* produces *wrong* observations — and circularly so: you would "learn" false facts about game
|
||||
state from a broken interpreter and bake them into the global map, the dispatch model, and everything
|
||||
downstream. **The analysis power is unlocked by correctness, not a substitute for achieving it.** You
|
||||
cannot debug the unknown with an instrument you have not first validated. Until the VM is running
|
||||
mostly correctly, using it for analysis is meaningless.
|
||||
|
||||
**Therefore the bootstrap order matters:**
|
||||
1. Build the VM.
|
||||
2. **Validate it against ground truth we already hold, by *independent* means** — chiefly the dialogue
|
||||
oracle (`build/text/dialogue.jsonl`: the VM's `show-text` sequence per scene must match), plus
|
||||
differential checks against known-correct behavior. This *external* oracle certifies the instrument.
|
||||
3. Only then use the validated core's observability to understand the *adjacent unknown* — which
|
||||
globals mean what, dispatch, effectful-op behavior. **Correctness propagates outward from validated
|
||||
anchors.**
|
||||
|
||||
**Two refinements that bound the trust:**
|
||||
- **Trust is per-subsystem, and only as strong as the oracle covering it.** The dialogue diff strongly
|
||||
validates the ADV/text layer. But computational/battle logic has *weaker* oracles — a wrong damage
|
||||
number can look plausible and pass unnoticed. So "mostly working" must mean *demonstrated correct per
|
||||
subsystem*; where oracles are weak (battle math), the VM-as-instrument is correspondingly less
|
||||
trustworthy, and **Frida/Unicorn cross-checks retain their value there** (exactly why Unicorn stayed
|
||||
on the list for computational ops). A green ADV oracle does not imply the battle math is right.
|
||||
- **Coverage-limited, plus a chicken-and-egg.** Runtime tools only observe what a playthrough
|
||||
exercises (rare branches stay dark), so they complement rather than replace static analysis. And
|
||||
bootstrapping the VM needs *just enough* `call-script` dispatch to run before observation can help
|
||||
refine dispatch — hence Phase A hardcodes a minimal dispatch first.
|
||||
|
||||
**Net effect on sequencing:** the runtime debugger makes *most* of the Frida/Unicorn side-tasks
|
||||
cheaper or obsolete — but only after the VM earns trust on a validated core. So the priority is to
|
||||
reach *validated* correctness on the ADV layer first (via the dialogue oracle); that trusted anchor is
|
||||
what makes the instrument usable for everything else. The debugger is a force multiplier on a correct
|
||||
VM and dead weight on an incorrect one.
|
||||
|
||||
---
|
||||
|
||||
## 6. Extending to other AGE games and versions
|
||||
|
||||
### Other AGE games, same version (e.g. Kamidori, also SYS4)
|
||||
**Reused for free:** container parser, VM core, backend adapters, the opcode table (the engine ABI
|
||||
is shared across the family), disassembler/assembler, the whole extraction methodology.
|
||||
**Per-game (inherent content work):** the **global-var map** (globals are game-specific), the
|
||||
**call-script registry**, the **data-table layouts** (each game's `*INIT` differs), assets, and any
|
||||
game-specific effectful behavior. Process: point the toolchain at the new game's archives, re-run
|
||||
extraction, rebuild its global map, resolve its call-script registry, author a profile. The long
|
||||
poles are exactly the two enablers (global map + call-script). **This is the core payoff of the VM
|
||||
approach:** the *engine* cost amortizes across all AGE games; only content-mapping recurs — far less
|
||||
than re-coding each game's logic bespoke.
|
||||
|
||||
### Other engine versions (SYS3 / SYS5) — one app, not many
|
||||
Versions differ in: header (SYS4 `0x3C` vs SYS5 `0x44`), string codec (SYS4 cp932^0xFF vs SYS5
|
||||
UTF-16^0xFFFF), opcode set (overlapping, version-specific; Kelebek's table already spans the family
|
||||
and notes version-gated ops), operand types (SYS5 adds `0x8003+`), and archive magic (`S3IC`/`S4IC`/
|
||||
`S5IN`). **Architecture answer: version-parameterize, don't fork.** One runtime with:
|
||||
- a **version-detection** step (magic → SYS3/4/5),
|
||||
- **pluggable front-ends** (header parser, string decoder, opcode table, archive reader per version),
|
||||
- the **shared VM core** (the execution model — opcode+typed operands, var banks, control flow — is
|
||||
the same engine evolving),
|
||||
- **per-game profiles** that name the version + game-specific maps.
|
||||
|
||||
So: **not a separate application per version — a manifest/profile selects the front-end + game data.**
|
||||
SYS5 is well-covered (it's Kelebek's target); SYS3 is older and less documented and would need more
|
||||
front-end work (Kelebek's parser only does SYS4/SYS5), but the plug-in shape accommodates it. Result:
|
||||
one "AGE Engine" app that, given a profile, runs Himegari, Kamidori, a SYS5 title, etc., each moddable
|
||||
through the same system.
|
||||
|
||||
---
|
||||
|
||||
## 7. Feasibility, risks, open questions
|
||||
|
||||
**Feasible? Yes — but it is the largest phase of the whole effort**, on the scale of a small ScummVM
|
||||
target. The decoding groundwork substantially de-risks it (we understand the format, 97% of opcodes,
|
||||
the data, a partial global map). Biggest risks, with mitigations:
|
||||
- **`call-script` dispatch entangled in the packed AGE.EXE** → try `SCJUMP.BIN` static decode first;
|
||||
fall back to a targeted Frida capture. Gating for *anything* running.
|
||||
- **Effectful-op surface is large and quirk-laden** (esp. SRPG battle/dungeon UI) → ADV-first; defer
|
||||
SRPG; lean on the Frida shortlist.
|
||||
- **AGF graphics** → low risk; `AGF2BMP2AGF.exe` (asmodean) already present.
|
||||
- **Save format** → reversible struct work; needed before a full playthrough.
|
||||
- **Toolchain (Python) vs runtime (C#) drift** → share the documented format spec; runtime is
|
||||
canonical.
|
||||
|
||||
**Open questions to resolve early:** exact `call-script` mechanism; how scenes register (needed to
|
||||
*add* content); how much the SRPG layer's rendering diverges from ADV; save layout.
|
||||
|
||||
---
|
||||
|
||||
## 8. Immediate next step
|
||||
Start **Phase A, the vertical slice** — it converts all of the above from architecture into evidence
|
||||
and tells us fast whether option 3 is as feasible as it looks. Concretely: pick one small ADV scene,
|
||||
stand up the C# VM core + Godot ADV backend, resolve just-enough `call-script`, convert that scene's
|
||||
AGF art, and get its dialogue rendering and matching `build/text/dialogue.jsonl`. Everything else in
|
||||
this roadmap is sequenced behind that proof.
|
||||
110
docs/script-inventory.md
Normal file
110
docs/script-inventory.md
Normal file
@@ -0,0 +1,110 @@
|
||||
# DATA1 Script Inventory — 481 `.BIN` files
|
||||
|
||||
All scripts share the magic header `SYS4422 ` (8 bytes), confirming a uniform SYS4
|
||||
bytecode format (engine version 4.4.2.2) across the entire set. Header is followed
|
||||
by what appear to be little-endian u32 fields (version/section table — to be mapped
|
||||
during disassembler work).
|
||||
|
||||
Source: `extracted\DATA1\` (extracted from `DATA1.ALF`).
|
||||
|
||||
**Patch overrides:** 52 loose `.BIN` files sit in the game root directory and shadow
|
||||
their DATA1 counterparts at runtime (sizes differ slightly — e.g. `FIELD.BIN` root
|
||||
200,536 vs archive 200,224). These are the v1.03 / append-patch versions and should be
|
||||
treated as **authoritative** over the archive copies. Two engine files exist only in
|
||||
the root: `SYS4INI.BIN` (272 KB) and `SYS4AB.BIN` (1.08 MB).
|
||||
|
||||
---
|
||||
|
||||
## Breakdown by series
|
||||
|
||||
| Series | Count | Total size | Role (inferred) |
|
||||
|---|---|---|---|
|
||||
| `SC####` | 136 | 15.6 MB | Scenario/event scripts (numbered 0000–1690, step 10) |
|
||||
| `SP####` | 163 | 10.6 MB | Secondary scene series (0010–1369; incl. `SP0051A/B` split) — likely character/H-events |
|
||||
| Named scripts | 175 | ~7.5 MB | Engine subsystems, data tables, UI, battle logic |
|
||||
| `DEBUG*` | 5 | 1.4 MB | Debug tools (`DEBUGADV.BIN` alone is 1.4 MB — a scene viewer/jump menu) |
|
||||
| `RTN_M###` / `RTN_B###` | 26 | ~70 KB | Small routine scripts (map routines M001–M061, battle routines B001–B004) |
|
||||
|
||||
### SC series notes
|
||||
- Numbered `SC0000`–`SC0880` (main chapters, largest files — up to 700 KB) and
|
||||
`SC1000`–`SC1690` (smaller; likely sub-events, endings, appendix content).
|
||||
- `SCJUMP.BIN` (778 KB) is almost certainly the master scene-dispatch/jump table.
|
||||
- `SCINIT.BIN` (88 KB) initializes scenario state.
|
||||
|
||||
---
|
||||
|
||||
## Named scripts by subsystem (inferred from names)
|
||||
|
||||
### Engine core / boot flow
|
||||
`SYSTEM4.BIN` (config root), `INIT.BIN` (64 bytes — smallest script, ideal first
|
||||
disassembly target), `INITCONFIG`, `LOADCONFIG`, `CONFIG`, `TUNE`, `LOGO`, `OP`,
|
||||
`ED`, `TITLE`, `GAMESTART`, `GAMECLEAR`, `STAGECLEAR`
|
||||
|
||||
### Data-table INIT scripts (likely static game data, not logic)
|
||||
Large, table-like scripts — prime candidates for data extraction:
|
||||
- `STINIT` (579 KB) — stages/scenario tables
|
||||
- `EBINIT` (338 KB) — enemy battle data
|
||||
- `MPINIT` (330 KB) — maps
|
||||
- `SCINIT` (88 KB), `CGINIT` (79 KB — CG gallery), `ITINIT` (70 KB — items),
|
||||
`RTINIT` (67 KB), `CCINIT` (41 KB), `SKINIT` (37 KB — skills), `CDINIT` (31 KB),
|
||||
`BTANINIT` (105 KB — battle animations)
|
||||
- Smaller: `AFINIT`, `ALINIT`, `CIINIT`, `CNINIT`, `CTINIT`, `CVINIT`, `ILINIT`,
|
||||
`LAINIT`, `MAINIT`, `OBINIT`, `SPINIT`, `TRINIT`, `VIINIT`
|
||||
|
||||
### Message/string tables (`*MES`)
|
||||
`ITMES` (64 KB — item text), `VIMES` (43 KB), `EIMES` (37 KB), `SKMES` (31 KB — skill
|
||||
text), `CIMES` (15 KB), `MAMES`, `INFOMES`, `MES` — where most translatable text
|
||||
outside scenes lives.
|
||||
|
||||
### Battle system
|
||||
`BTL` (61 KB — main battle loop), `BTRTN`, `ROUND`, `AIM`, `ATSEEK`, `MVSEEK`,
|
||||
`MVRTN`, `MAGIC`, `USEMAGIC`, `SUMMON`, `EXILE`, `DISARM`, `RECOVER`, `COUNTUNIT`,
|
||||
`SETOCC`, and the `CALC*` family: `CALCBTPARAM`, `CALCDMG` (16 KB — damage formula!),
|
||||
`CALCSCOPE`, `CALCOCC`, `CALCREVISE`, `CALCCC`, `CALCILL`, `CALCARR`
|
||||
|
||||
### Dungeon/map engine
|
||||
`FIELD` (200 KB — the core dungeon-crawl loop), `DRAWMAP`, `RENDERMAP`,
|
||||
`DRAWMINIMAP`, `DRAWCH`/`DRAWCHP`/`DRAWENP`/`DRAWOBJ`/`DRAWTIP`/`DRAWVOL`,
|
||||
`SETCH`/`SETEN`/`SETLAND`/`SETOBJ`/`SETROUTE`/`SETMVWORK`,
|
||||
`DELCH`/`DELEN`/`DELENMASS`/`DELLAND`, `RESETLAND`, `WARPU`/`WARPD`, `LOOK`,
|
||||
`READICON`
|
||||
|
||||
### Unit/party management
|
||||
`ADDEXP`, `ADDSKILL`, `ADDEN`, `ADDITEM`, `ADDRANDOMITEM`, `LOSTRANDOMITEM`,
|
||||
`ADDILL`/`ADDILLSUB`, `EVOLVE`, `IMPROVE` (37 KB), `TRAIN`, `STUDY`, `UNITECH`,
|
||||
`REMOVECH`, `SHOWGROW`, `STATUS`, `USEITEM`, `SETCH`
|
||||
|
||||
### Base/facility gameplay
|
||||
`CAMP`, `ROOM`, `FORT`, `ALCHEMY` (40 KB), `SALLY` (40 KB — sortie/deployment),
|
||||
`READY` (39 KB — pre-battle prep), `SELSTAGE` (34 KB), `SELACT` (30 KB)
|
||||
|
||||
### Menus / UI / meta
|
||||
`MENU`, `CHMENU` (69 KB — character menu), `INFO`/`INFOAF`/`INFOCH`/`INFOEN`/
|
||||
`INFOIT`/`INFOVO` (info panels: characters, enemies, items, voices), `SAVE` (40 KB),
|
||||
`HISTORY`, `HIDEWIN`, `CLOSE`, `INPUTNAME` (28 KB), `CGMODE` (24 KB — gallery),
|
||||
`MMODE` (music mode), `HMODE` (22 KB — scene replay)
|
||||
|
||||
### Flow control / branching
|
||||
`BUNKI` (分岐 = branch, 15 KB), `SBUNKI`, `BUNKIMOVE`, `SBUNKIMOVE`, `SCJUMP`
|
||||
|
||||
### Callbacks (engine → script hooks)
|
||||
`CALLBACK_LOAD`, `CALLBACK_LOST`, `CALLBACK_SETTING`, `CALLBACK_WINDOW`
|
||||
|
||||
### Debug
|
||||
`DEBUG`, `DEBUGADV` (1.4 MB), `DEBUGANIME`, `DEBUGBTL`, `DEBUGMAP` (+2 numbered)
|
||||
|
||||
---
|
||||
|
||||
## Implications for the port
|
||||
|
||||
1. **Much more game logic lives in bytecode than expected.** Damage formulas
|
||||
(`CALCDMG`), the dungeon loop (`FIELD`), battle flow (`BTL`, `ROUND`), and unit
|
||||
progression (`ADDEXP`, `EVOLVE`) are all scripts — `AGE.EXE` is closer to a pure
|
||||
VM/renderer. This strengthens the case for **re-implementing the AGE VM in Godot**
|
||||
rather than transpiling every script by hand (the doc's open question #12).
|
||||
2. **Disassembler bootstrapping order** (small → large, system → scene):
|
||||
`INIT.BIN` (64 B) → `ED.BIN`/`OP.BIN`/`LOGO.BIN` (~230 B) → `CALLBACK_LOST`
|
||||
(176 B) → `MENU.BIN` (3 KB) → `CALCDMG` → a mid-size `SC####`.
|
||||
3. **The `*INIT` giants are likely data tables**, decodable early even with a
|
||||
partial opcode map — instant win for extracting item/skill/enemy/stage databases.
|
||||
4. **Use root-directory overrides, not archive copies**, for the 52 patched scripts.
|
||||
911
docs/superpowers/plans/2026-07-06-opcode-reference.md
Normal file
911
docs/superpowers/plans/2026-07-06-opcode-reference.md
Normal file
@@ -0,0 +1,911 @@
|
||||
# Living Opcode Reference Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Make `vm-map/opcodes.toml` the single hand-edited source of truth for opcode knowledge (ABI + semantics + provenance + dependencies), from which we generate the Python shim the tooling imports, a machine JSON, a human Markdown reference, and coverage.
|
||||
|
||||
**Architecture:** One canonical TOML file. A generator/linter (`tools/opcodes_build.py`) reads it (stdlib `tomllib`) through a small data model (`tools/opcodes_model.py`) and emits four artifacts. Bootstrap seeds all 248 used opcodes from the pristine Kelebek table (`tools/age_opcodes.py`) by *appending* skeleton text (no TOML writer dependency). Emitters are pure `(model) -> str` functions so the real files are only rewritten in the final migration task.
|
||||
|
||||
**Tech Stack:** Python 3.11 (`py -3.11 -X utf8`), stdlib only (`tomllib`, `json`, `dataclasses`). Reuses `tools/sys4load.py` + `tools/paths.py`. No new dependencies.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Run all Python as `py -3.11 -X utf8` (Shift-JIS strings need utf8 mode on Windows).
|
||||
- Stdlib only — do NOT add `pyyaml`/`tomli_w`/`pytest`. Tests are plain scripts run with `py -3.11`.
|
||||
- `tools/age_opcodes.py` (Kelebek table) is PRISTINE — never edit it.
|
||||
- `tools/paths.py` is the only place that knows filesystem locations; import paths from it, never hardcode.
|
||||
- The generated `tools/age_opcodes_himegari.py` MUST keep exposing `INFERRED: dict[int, dict]` where each entry has a `name` key (the only field `sys4load` reads: `sys4load.py:84`). Do not change `sys4load.py`.
|
||||
- **This workspace is not a git repo.** Treat every **Checkpoint** step as: if `git` is initialized, run the shown `git add/commit`; otherwise just confirm the named outputs exist and continue. Do not run `git init` unless the user asks.
|
||||
- Controlled vocabularies (the linter enforces these):
|
||||
- `category ∈ {marker, structural, control, adv, draw, audio, input, compute, unknown}`
|
||||
- `source ∈ {kelebek, harness, investigation, frida, unicorn, inference}`
|
||||
- `confidence ∈ {low, med, high}` (ordered low<med<high)
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
- Create `tools/opcodes_model.py` — dataclasses + `load()` + `lint()` + `dependents()`. Pure, importable, testable.
|
||||
- Create `tools/opcodes_build.py` — CLI: `--bootstrap`, `--build`, `--lint`. Corpus scan + pure emitters + file wiring.
|
||||
- Create `tools/test_opcodes.py` — standalone test script (no pytest); `py -3.11 -X utf8 tools/test_opcodes.py` prints results and exits nonzero on failure.
|
||||
- Create `vm-map/opcodes.toml` — canonical file (bootstrapped skeletons, then hand-migrated).
|
||||
- Generated (written only in Task 6): `tools/age_opcodes_himegari.py` (overwrites the hand-version), `build/opcodes.json`, `docs/opcode-reference.md`, `build/opcode-coverage.md`.
|
||||
- Retire in Task 7: `vm-map/opcodes-himegari.json`, `vm-map/himegari-opcode-notes.md`, hand-maintained `build/opcode-coverage.md`.
|
||||
- Modify in Task 7: `docs/PROJECT-STRUCTURE.md`, memory (`himegari-port-status.md`, `MEMORY.md`).
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Data model + loader (`opcodes_model.py`)
|
||||
|
||||
**Files:**
|
||||
- Create: `tools/opcodes_model.py`
|
||||
- Test: `tools/test_opcodes.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `load(path) -> Model`; `Model(meta: dict, opcodes: dict[int, Opcode])`; `Opcode(op, label, argc, code_target_args, abi_source, abi_note, semantics)`; `Semantics(name, category, summary, noop_headless, source, confidence, depends_on: list[int], evidence, details, confirm_by, args: list[dict])`; `dependents(model) -> dict[int, list[int]]`; constants `CATEGORIES`, `SOURCES`, `CONFIDENCE`.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Create `tools/test_opcodes.py`:
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""Standalone tests for the opcode reference tooling. Run: py -3.11 -X utf8 tools/test_opcodes.py"""
|
||||
import os, sys, tempfile
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import opcodes_model as M
|
||||
|
||||
FAILS = []
|
||||
def check(cond, msg):
|
||||
print((" ok " if cond else " FAIL ") + msg)
|
||||
if not cond: FAILS.append(msg)
|
||||
|
||||
FIXTURE = '''
|
||||
[meta]
|
||||
opcodes_used_by_himegari = 2
|
||||
[[opcode]]
|
||||
op = 0x90
|
||||
label = "u0041BEB0"
|
||||
argc = 7
|
||||
code_target_args = [5, 6, 7]
|
||||
[opcode.semantics]
|
||||
name = "hotspot-branch"
|
||||
category = "input"
|
||||
summary = "cursor hotspot hit-test"
|
||||
noop_headless = true
|
||||
source = "investigation"
|
||||
confidence = "high"
|
||||
depends_on = [0x1f4]
|
||||
evidence = "301/301 uniform"
|
||||
[[opcode.semantics.args]]
|
||||
i = 1
|
||||
role = "x"
|
||||
observed_types = ["imm"]
|
||||
[[opcode]]
|
||||
op = 0x1f4
|
||||
label = "u004160D0"
|
||||
argc = 0
|
||||
[opcode.semantics]
|
||||
name = "stmt-begin"
|
||||
category = "marker"
|
||||
source = "investigation"
|
||||
confidence = "high"
|
||||
'''
|
||||
|
||||
def write_tmp(text):
|
||||
fd, p = tempfile.mkstemp(suffix=".toml"); os.close(fd)
|
||||
open(p, "w", encoding="utf-8").write(text)
|
||||
return p
|
||||
|
||||
def test_load():
|
||||
m = M.load(write_tmp(FIXTURE))
|
||||
check(set(m.opcodes) == {0x90, 0x1f4}, "loads both opcodes keyed by int")
|
||||
o = m.opcodes[0x90]
|
||||
check(o.argc == 7, "0x90 argc == 7")
|
||||
check(o.code_target_args == [5, 6, 7], "0x90 code_target_args parsed")
|
||||
check(o.semantics.name == "hotspot-branch", "0x90 semantics.name")
|
||||
check(o.semantics.depends_on == [0x1f4], "depends_on parsed as int list")
|
||||
check(o.semantics.args[0]["role"] == "x", "arg role parsed")
|
||||
rev = M.dependents(m)
|
||||
check(rev.get(0x1f4) == [0x90], "dependents: 0x1f4 depended on by 0x90")
|
||||
|
||||
def main():
|
||||
test_load()
|
||||
print("FAILURES:", len(FAILS))
|
||||
return 1 if FAILS else 0
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/test_opcodes.py`
|
||||
Expected: FAIL — `ModuleNotFoundError: No module named 'opcodes_model'`.
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
Create `tools/opcodes_model.py`:
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""In-memory model + loader + linter for vm-map/opcodes.toml (the canonical opcode reference).
|
||||
Read-only: uses stdlib tomllib. See docs/superpowers/specs/2026-07-06-opcode-reference-design.md."""
|
||||
from __future__ import annotations
|
||||
import tomllib
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
CATEGORIES = {"marker", "structural", "control", "adv", "draw", "audio", "input", "compute", "unknown"}
|
||||
SOURCES = {"kelebek", "harness", "investigation", "frida", "unicorn", "inference"}
|
||||
CONFIDENCE = {"low": 1, "med": 2, "high": 3}
|
||||
|
||||
@dataclass
|
||||
class Semantics:
|
||||
name: str
|
||||
category: str = "unknown"
|
||||
summary: str = ""
|
||||
noop_headless: bool = False
|
||||
source: str = "kelebek"
|
||||
confidence: str = "low"
|
||||
depends_on: list[int] = field(default_factory=list)
|
||||
evidence: str = ""
|
||||
details: str = ""
|
||||
confirm_by: str = ""
|
||||
args: list[dict] = field(default_factory=list)
|
||||
|
||||
@dataclass
|
||||
class Opcode:
|
||||
op: int
|
||||
label: str
|
||||
argc: int
|
||||
code_target_args: list[int] = field(default_factory=list)
|
||||
abi_source: str = "kelebek+decode-validated"
|
||||
abi_note: str = ""
|
||||
semantics: Semantics | None = None
|
||||
|
||||
@dataclass
|
||||
class Model:
|
||||
meta: dict
|
||||
opcodes: dict[int, Opcode]
|
||||
|
||||
def load(path) -> Model:
|
||||
data = tomllib.loads(Path(path).read_text(encoding="utf-8"))
|
||||
ops: dict[int, Opcode] = {}
|
||||
for e in data.get("opcode", []):
|
||||
sem = None
|
||||
s = e.get("semantics")
|
||||
if s is not None:
|
||||
sem = Semantics(
|
||||
name=s.get("name", e.get("label", "")),
|
||||
category=s.get("category", "unknown"),
|
||||
summary=s.get("summary", ""),
|
||||
noop_headless=bool(s.get("noop_headless", False)),
|
||||
source=s.get("source", "kelebek"),
|
||||
confidence=s.get("confidence", "low"),
|
||||
depends_on=[int(x) for x in s.get("depends_on", [])],
|
||||
evidence=s.get("evidence", ""),
|
||||
details=s.get("details", ""),
|
||||
confirm_by=s.get("confirm_by", ""),
|
||||
args=list(s.get("args", [])),
|
||||
)
|
||||
ops[int(e["op"])] = Opcode(
|
||||
op=int(e["op"]), label=e.get("label", ""), argc=int(e["argc"]),
|
||||
code_target_args=[int(x) for x in e.get("code_target_args", [])],
|
||||
abi_source=e.get("abi_source", "kelebek+decode-validated"),
|
||||
abi_note=e.get("abi_note", ""), semantics=sem,
|
||||
)
|
||||
return Model(meta=data.get("meta", {}), opcodes=ops)
|
||||
|
||||
def dependents(model: Model) -> dict[int, list[int]]:
|
||||
"""Reverse of depends_on: op -> [ops whose semantics depend on it]."""
|
||||
rev: dict[int, list[int]] = {op: [] for op in model.opcodes}
|
||||
for op, oc in model.opcodes.items():
|
||||
if oc.semantics:
|
||||
for dep in oc.semantics.depends_on:
|
||||
rev.setdefault(dep, []).append(op)
|
||||
for k in rev:
|
||||
rev[k].sort()
|
||||
return rev
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/test_opcodes.py`
|
||||
Expected: all `test_load` lines `ok`, `FAILURES: 0`, exit 0.
|
||||
|
||||
- [ ] **Step 5: Checkpoint**
|
||||
|
||||
If git initialized: `git add tools/opcodes_model.py tools/test_opcodes.py && git commit -m "feat(opcodes): data model + loader for opcodes.toml"`
|
||||
Else: confirm `tools/opcodes_model.py` and `tools/test_opcodes.py` exist; continue.
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Linter (`lint` in `opcodes_model.py`)
|
||||
|
||||
**Files:**
|
||||
- Modify: `tools/opcodes_model.py` (add `lint`)
|
||||
- Test: `tools/test_opcodes.py` (add `test_lint`)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `lint(model) -> tuple[list[str], list[str]]` returning `(errors, warnings)`.
|
||||
- Rules: (1) `category`/`source`/`confidence` must be in the controlled vocab — else **error**. (2) every `depends_on` id must exist — else **error** (dangling-ref). (3) an entry's confidence may not exceed the min confidence among its dependencies — else **warning** (confidence-ceiling).
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add to `tools/test_opcodes.py` (call `test_lint()` from `main` before the summary):
|
||||
|
||||
```python
|
||||
DANGLING = '''
|
||||
[[opcode]]
|
||||
op = 0x10
|
||||
label = "x"
|
||||
argc = 0
|
||||
[opcode.semantics]
|
||||
name = "a"
|
||||
category = "compute"
|
||||
source = "inference"
|
||||
confidence = "low"
|
||||
depends_on = [0x99]
|
||||
'''
|
||||
|
||||
CEILING = '''
|
||||
[[opcode]]
|
||||
op = 0x10
|
||||
label = "x"
|
||||
argc = 0
|
||||
[opcode.semantics]
|
||||
name = "low-op"
|
||||
category = "compute"
|
||||
source = "kelebek"
|
||||
confidence = "low"
|
||||
[[opcode]]
|
||||
op = 0x11
|
||||
label = "y"
|
||||
argc = 0
|
||||
[opcode.semantics]
|
||||
name = "high-op"
|
||||
category = "compute"
|
||||
source = "inference"
|
||||
confidence = "high"
|
||||
depends_on = [0x10]
|
||||
'''
|
||||
|
||||
BADVOCAB = '''
|
||||
[[opcode]]
|
||||
op = 0x10
|
||||
label = "x"
|
||||
argc = 0
|
||||
[opcode.semantics]
|
||||
name = "a"
|
||||
category = "bogus"
|
||||
source = "inference"
|
||||
confidence = "low"
|
||||
'''
|
||||
|
||||
def test_lint():
|
||||
e, w = M.lint(M.load(write_tmp(DANGLING)))
|
||||
check(any("0x99" in m for m in e), "dangling depends_on is an error")
|
||||
e, w = M.lint(M.load(write_tmp(CEILING)))
|
||||
check(any("0x11" in m for m in w), "confidence-ceiling violation is a warning")
|
||||
check(e == [], "confidence-ceiling case has no errors")
|
||||
e, w = M.lint(M.load(write_tmp(BADVOCAB)))
|
||||
check(any("category" in m for m in e), "unknown category is an error")
|
||||
e, w = M.lint(M.load(write_tmp(FIXTURE)))
|
||||
check(e == [], "clean fixture has no lint errors")
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/test_opcodes.py`
|
||||
Expected: FAIL — `AttributeError: module 'opcodes_model' has no attribute 'lint'`.
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
Add to `tools/opcodes_model.py`:
|
||||
|
||||
```python
|
||||
def lint(model: Model) -> tuple[list[str], list[str]]:
|
||||
errors: list[str] = []
|
||||
warnings: list[str] = []
|
||||
ops = model.opcodes
|
||||
for op, oc in sorted(ops.items()):
|
||||
s = oc.semantics
|
||||
if not s:
|
||||
continue
|
||||
tag = f"0x{op:x}"
|
||||
if s.category not in CATEGORIES:
|
||||
errors.append(f"{tag}: bad category {s.category!r}")
|
||||
if s.source not in SOURCES:
|
||||
errors.append(f"{tag}: bad source {s.source!r}")
|
||||
if s.confidence not in CONFIDENCE:
|
||||
errors.append(f"{tag}: bad confidence {s.confidence!r}")
|
||||
for dep in s.depends_on:
|
||||
if dep not in ops:
|
||||
errors.append(f"{tag}: depends_on missing opcode 0x{dep:x}")
|
||||
if s.confidence in CONFIDENCE:
|
||||
dep_confs = [CONFIDENCE[ops[d].semantics.confidence]
|
||||
for d in s.depends_on
|
||||
if d in ops and ops[d].semantics
|
||||
and ops[d].semantics.confidence in CONFIDENCE]
|
||||
if dep_confs and CONFIDENCE[s.confidence] > min(dep_confs):
|
||||
warnings.append(f"{tag}: confidence {s.confidence!r} exceeds dependency ceiling")
|
||||
return errors, warnings
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/test_opcodes.py`
|
||||
Expected: `test_load` + `test_lint` all `ok`, `FAILURES: 0`.
|
||||
|
||||
- [ ] **Step 5: Checkpoint**
|
||||
|
||||
If git: `git add tools/opcodes_model.py tools/test_opcodes.py && git commit -m "feat(opcodes): linter (dangling-ref, confidence-ceiling, vocabulary)"`
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Bootstrap (`opcodes_build.py --bootstrap`)
|
||||
|
||||
**Files:**
|
||||
- Create: `tools/opcodes_build.py`
|
||||
- Test: `tools/test_opcodes.py` (add `test_bootstrap`)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `paths.scripts()`, `sys4load.load`, `age_opcodes.OPCODES`, `opcodes_model`.
|
||||
- Produces: `scan_corpus() -> (used: Counter, argtypes: dict[int, dict[int, set[int]]])`; `skeleton_toml(op, label, argc, argtypes_for_op) -> str`; `bootstrap(toml_path: Path) -> None` (creates file with `[meta]` on first run, then appends a skeleton block for each used opcode not already present). CLI: `py -3.11 -X utf8 tools/opcodes_build.py --bootstrap [--toml PATH]`.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add to `tools/test_opcodes.py`:
|
||||
|
||||
```python
|
||||
def test_bootstrap():
|
||||
import opcodes_build as B
|
||||
fd, p = tempfile.mkstemp(suffix=".toml"); os.close(fd); os.remove(p)
|
||||
from pathlib import Path
|
||||
tp = Path(p)
|
||||
B.bootstrap(tp) # first run: meta + all skeletons
|
||||
m = M.load(tp)
|
||||
check(len(m.opcodes) >= 240, f"bootstrap seeded ~248 opcodes (got {len(m.opcodes)})")
|
||||
check(0x90 in m.opcodes and m.opcodes[0x90].argc == 7, "0x90 seeded with argc 7")
|
||||
n1 = len(m.opcodes)
|
||||
B.bootstrap(tp) # idempotent: appends nothing new
|
||||
check(len(M.load(tp).opcodes) == n1, "second bootstrap adds no duplicates")
|
||||
e, w = M.lint(m)
|
||||
check(e == [], f"bootstrapped file lints clean (errors: {e[:3]})")
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/test_opcodes.py`
|
||||
Expected: FAIL — `ModuleNotFoundError: No module named 'opcodes_build'`.
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
Create `tools/opcodes_build.py`:
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""Generator + linter for the living opcode reference (vm-map/opcodes.toml).
|
||||
--bootstrap seed skeletons for every used opcode (append-only; preserves hand edits)
|
||||
--build emit age_opcodes_himegari.py + build/opcodes.json + docs/opcode-reference.md + build/opcode-coverage.md
|
||||
--lint run the linter, print errors/warnings, exit nonzero on errors
|
||||
See docs/superpowers/specs/2026-07-06-opcode-reference-design.md."""
|
||||
from __future__ import annotations
|
||||
import os, sys, json, argparse, collections
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import paths
|
||||
import sys4load
|
||||
import opcodes_model as M
|
||||
from age_opcodes import OPCODES
|
||||
|
||||
TOML_DEFAULT = paths.VM_MAP / "opcodes.toml"
|
||||
|
||||
TYPE_NAMES = {0x0: "imm", 0x1: "float", 0x2: "string", 0x3: "g-int", 0x4: "g-float",
|
||||
0x5: "g-str", 0x6: "g-ptr", 0x8: "g-str-ptr", 0x9: "l-int", 0xa: "l-float",
|
||||
0xb: "l-str", 0xc: "l-ptr", 0xd: "l-float-ptr", 0xe: "l-str-ptr"}
|
||||
|
||||
META_TOML = '''# vm-map/opcodes.toml -- CANONICAL living opcode reference (hand-edited).
|
||||
# Generated artifacts (age_opcodes_himegari.py, build/opcodes.json, docs/opcode-reference.md,
|
||||
# build/opcode-coverage.md) come from this file via tools/opcodes_build.py --build. Do not edit those.
|
||||
# Skeletons are appended by --bootstrap; enrich each [opcode.semantics] as we investigate.
|
||||
[meta]
|
||||
instruction_model = "code = seq of <opcode:u32> then argc*(<argtype:u32><value:u32>); len_dwords = 1 + 2*argc"
|
||||
opcodes_used_by_himegari = 248
|
||||
|
||||
[meta.arg_types]
|
||||
"0x0" = "immediate"
|
||||
"0x1" = "float"
|
||||
"0x2" = "string"
|
||||
"0x3" = "global-int"
|
||||
"0x4" = "global-float"
|
||||
"0x5" = "global-string"
|
||||
"0x6" = "global-ptr"
|
||||
"0x8" = "global-string-ptr"
|
||||
"0x9" = "local-int"
|
||||
"0xa" = "local-float"
|
||||
"0xb" = "local-string"
|
||||
"0xc" = "local-ptr"
|
||||
"0xd" = "local-float-ptr"
|
||||
"0xe" = "local-string-ptr"
|
||||
|
||||
[meta.header_fields]
|
||||
"F0" = "local_integer_1"
|
||||
"F1" = "local_floats"
|
||||
"F2" = "local_strings_1"
|
||||
"F3" = "local_integer_2"
|
||||
"F4" = "unknown_data"
|
||||
"F5" = "local_strings_2"
|
||||
"F6" = "sub_header_length(=0x1C)"
|
||||
"F7" = "table_1_length"
|
||||
"F8" = "table_1_offset(=code end)"
|
||||
"F9" = "table_2_length"
|
||||
"F10" = "table_2_offset"
|
||||
"F11" = "table_3_length"
|
||||
"F12" = "table_3_offset"
|
||||
'''
|
||||
|
||||
def scan_corpus():
|
||||
"""used[op] = count; argtypes[op][arg_index] = set(type-codes) across the corpus."""
|
||||
used = collections.Counter()
|
||||
argtypes: dict[int, dict[int, set]] = collections.defaultdict(lambda: collections.defaultdict(set))
|
||||
for name, path in paths.scripts().items():
|
||||
try:
|
||||
scr = sys4load.load(path)
|
||||
except Exception:
|
||||
continue
|
||||
for ins in scr.instructions:
|
||||
used[ins.opcode] += 1
|
||||
for i, (t, v) in enumerate(ins.args):
|
||||
argtypes[ins.opcode][i].add(t)
|
||||
return used, argtypes
|
||||
|
||||
def _is_named(label: str) -> bool:
|
||||
return not (label.startswith("u00") or label == "dev_ukn" or label.startswith("?"))
|
||||
|
||||
def skeleton_toml(op: int, label: str, argc: int, argtypes_for_op: dict) -> str:
|
||||
conf = "med" if _is_named(label) else "low"
|
||||
lines = ["[[opcode]]", f"op = 0x{op:x}", f'label = "{label}"', f"argc = {argc}",
|
||||
'abi_source = "kelebek+decode-validated"', "", "[opcode.semantics]",
|
||||
f'name = "{label}"', 'category = "unknown"', 'summary = ""',
|
||||
"noop_headless = false", 'source = "kelebek"', f'confidence = "{conf}"',
|
||||
"depends_on = []", 'evidence = ""']
|
||||
for i in range(argc):
|
||||
tnames = [TYPE_NAMES.get(t, "t%#x" % t) for t in sorted(argtypes_for_op.get(i, ()))]
|
||||
obs = ", ".join('"%s"' % n for n in tnames)
|
||||
lines += ["", "[[opcode.semantics.args]]", f"i = {i + 1}", 'role = ""',
|
||||
f"observed_types = [{obs}]"]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
def bootstrap(toml_path: Path) -> None:
|
||||
used, argtypes = scan_corpus()
|
||||
present = set(M.load(toml_path).opcodes) if toml_path.exists() else set()
|
||||
blocks = []
|
||||
for op in sorted(used):
|
||||
if op in present:
|
||||
continue
|
||||
label, argc = OPCODES.get(op, ("0x%x" % op, 0))
|
||||
blocks.append(skeleton_toml(op, label, argc, argtypes[op]))
|
||||
if not toml_path.exists():
|
||||
toml_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
toml_path.write_text(META_TOML + "\n", encoding="utf-8")
|
||||
with toml_path.open("a", encoding="utf-8") as f:
|
||||
f.write("\n".join(blocks))
|
||||
print(f"bootstrap: {len(used)} used opcodes; appended {len(blocks)} new skeletons -> {toml_path}")
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--bootstrap", action="store_true")
|
||||
ap.add_argument("--build", action="store_true")
|
||||
ap.add_argument("--lint", action="store_true")
|
||||
ap.add_argument("--toml", default=str(TOML_DEFAULT))
|
||||
args = ap.parse_args(argv)
|
||||
tp = Path(args.toml)
|
||||
if args.bootstrap:
|
||||
bootstrap(tp)
|
||||
return 0
|
||||
ap.error("no action (expected --bootstrap/--build/--lint)")
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/test_opcodes.py`
|
||||
Expected: `test_bootstrap` lines `ok`, `FAILURES: 0`.
|
||||
|
||||
- [ ] **Step 5: Checkpoint**
|
||||
|
||||
If git: `git add tools/opcodes_build.py tools/test_opcodes.py && git commit -m "feat(opcodes): bootstrap seeds 248 skeletons from Kelebek + corpus arg-types"`
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Emit the Python shim (`--build` → `age_opcodes_himegari.py`)
|
||||
|
||||
**Files:**
|
||||
- Modify: `tools/opcodes_build.py` (add `emit_inferred_py`, wire `--build`)
|
||||
- Test: `tools/test_opcodes.py` (add `test_emit_inferred`)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `emit_inferred_py(model) -> str`. Pure. Emits `INFERRED: dict[int, dict]` containing an entry ONLY for opcodes whose `semantics.name != label` (i.e., ops we've given a distinct mnemonic) — this reproduces the current `sys4load` behavior exactly (bare Kelebek skeletons add nothing, so they are omitted and untouched unnamed ops keep rendering from `OPCODES`). Each entry carries `name` (required by sys4load) plus `category/noop/confidence/source/summary`.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add to `tools/test_opcodes.py`:
|
||||
|
||||
```python
|
||||
def test_emit_inferred():
|
||||
import opcodes_build as B
|
||||
src = B.emit_inferred_py(M.load(write_tmp(FIXTURE)))
|
||||
check("INFERRED" in src and "hotspot-branch" in src, "shim contains INFERRED + our mnemonic")
|
||||
ns = {}
|
||||
exec(compile(src, "<gen>", "exec"), ns)
|
||||
inf = ns["INFERRED"]
|
||||
check(0x90 in inf and inf[0x90]["name"] == "hotspot-branch", "generated INFERRED[0x90]['name'] correct")
|
||||
check(0x1f4 in inf, "named marker 0x1f4 (name != label) included")
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/test_opcodes.py`
|
||||
Expected: FAIL — `AttributeError: module 'opcodes_build' has no attribute 'emit_inferred_py'`.
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
Add to `tools/opcodes_build.py` (above `main`):
|
||||
|
||||
```python
|
||||
GEN_HEADER = "# DO NOT EDIT -- generated from vm-map/opcodes.toml by tools/opcodes_build.py --build\n"
|
||||
|
||||
def emit_inferred_py(model: M.Model) -> str:
|
||||
lines = [GEN_HEADER, '"""Inferred Himegari opcode semantics (generated). sys4load reads INFERRED[op][\'name\']."""',
|
||||
"from __future__ import annotations", "", "INFERRED: dict[int, dict] = {"]
|
||||
for op, oc in sorted(model.opcodes.items()):
|
||||
s = oc.semantics
|
||||
if not s or s.name == oc.label: # only ops we've given a distinct mnemonic
|
||||
continue
|
||||
lines.append(" 0x%x: dict(name=%r, category=%r, noop=%r, confidence=%r, source=%r, summary=%r),"
|
||||
% (op, s.name, s.category, s.noop_headless, s.confidence, s.source, s.summary))
|
||||
lines.append("}")
|
||||
return "\n".join(lines) + "\n"
|
||||
```
|
||||
|
||||
And wire `--build` in `main` (replace the final `ap.error(...)` line):
|
||||
|
||||
```python
|
||||
if args.build:
|
||||
model = M.load(tp)
|
||||
errors, warnings = M.lint(model)
|
||||
for m in warnings:
|
||||
print("warn:", m)
|
||||
if errors:
|
||||
for m in errors:
|
||||
print("error:", m)
|
||||
return 1
|
||||
(paths.REPO / "tools" / "age_opcodes_himegari.py").write_text(emit_inferred_py(model), encoding="utf-8")
|
||||
print("build: wrote tools/age_opcodes_himegari.py")
|
||||
return 0
|
||||
if args.lint:
|
||||
errors, warnings = M.lint(M.load(tp))
|
||||
for m in warnings:
|
||||
print("warn:", m)
|
||||
for m in errors:
|
||||
print("error:", m)
|
||||
print(f"lint: {len(errors)} errors, {len(warnings)} warnings")
|
||||
return 1 if errors else 0
|
||||
ap.error("no action (expected --bootstrap/--build/--lint)")
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/test_opcodes.py`
|
||||
Expected: `test_emit_inferred` lines `ok`, `FAILURES: 0`.
|
||||
|
||||
- [ ] **Step 5: Checkpoint**
|
||||
|
||||
If git: `git add tools/opcodes_build.py tools/test_opcodes.py && git commit -m "feat(opcodes): emit drop-in age_opcodes_himegari.py shim; wire --build/--lint"`
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Emit JSON + Markdown reference + coverage (`--build`)
|
||||
|
||||
**Files:**
|
||||
- Modify: `tools/opcodes_build.py` (add `emit_json`, `emit_reference_md`, `emit_coverage_md`; wire into `--build`)
|
||||
- Test: `tools/test_opcodes.py` (add `test_emit_views`)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `emit_json(model) -> str` (includes a `dependents` map), `emit_reference_md(model) -> str` (per-opcode section with a "depended on by" line), `emit_coverage_md(model) -> str` (counts by source/confidence/category). All pure.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add to `tools/test_opcodes.py`:
|
||||
|
||||
```python
|
||||
def test_emit_views():
|
||||
import opcodes_build as B, json as _json
|
||||
m = M.load(write_tmp(FIXTURE))
|
||||
j = _json.loads(B.emit_json(m))
|
||||
check(j["dependents"]["0x1f4"] == ["0x90"], "json dependents index correct")
|
||||
check(any(o["op"] == "0x90" for o in j["opcodes"]), "json lists opcode 0x90")
|
||||
md = B.emit_reference_md(m)
|
||||
check("hotspot-branch" in md and "depended on by" in md.lower(), "reference md has entry + dependents line")
|
||||
cov = B.emit_coverage_md(m)
|
||||
check("investigation" in cov, "coverage md breaks down by source")
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/test_opcodes.py`
|
||||
Expected: FAIL — `AttributeError: ... 'emit_json'`.
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
Add to `tools/opcodes_build.py`:
|
||||
|
||||
```python
|
||||
def emit_json(model: M.Model) -> str:
|
||||
rev = M.dependents(model)
|
||||
out = {"meta": model.meta, "opcodes": [],
|
||||
"dependents": {"0x%x" % k: ["0x%x" % d for d in v] for k, v in rev.items() if v}}
|
||||
for op, oc in sorted(model.opcodes.items()):
|
||||
e = {"op": "0x%x" % op, "label": oc.label, "argc": oc.argc,
|
||||
"code_target_args": oc.code_target_args, "abi_source": oc.abi_source}
|
||||
s = oc.semantics
|
||||
if s:
|
||||
e["semantics"] = {"name": s.name, "category": s.category, "summary": s.summary,
|
||||
"noop_headless": s.noop_headless, "source": s.source,
|
||||
"confidence": s.confidence, "depends_on": ["0x%x" % d for d in s.depends_on],
|
||||
"evidence": s.evidence, "details": s.details, "args": s.args}
|
||||
out["opcodes"].append(e)
|
||||
return json.dumps(out, ensure_ascii=False, indent=2) + "\n"
|
||||
|
||||
def emit_reference_md(model: M.Model) -> str:
|
||||
rev = M.dependents(model)
|
||||
L = ["<!-- DO NOT EDIT -- generated from vm-map/opcodes.toml by tools/opcodes_build.py --build -->",
|
||||
"# Opcode Reference (generated)", "",
|
||||
f"{len(model.opcodes)} opcodes used by Himegari. Source of truth: `vm-map/opcodes.toml`.", ""]
|
||||
by_cat = collections.defaultdict(list)
|
||||
for op, oc in model.opcodes.items():
|
||||
cat = oc.semantics.category if oc.semantics else "unknown"
|
||||
by_cat[cat].append(op)
|
||||
for cat in sorted(by_cat):
|
||||
L += [f"## {cat}", ""]
|
||||
for op in sorted(by_cat[cat]):
|
||||
oc = model.opcodes[op]
|
||||
s = oc.semantics
|
||||
name = s.name if s else oc.label
|
||||
L.append(f"### 0x{op:x} `{name}` ({oc.label}, argc {oc.argc})")
|
||||
if s:
|
||||
L.append(f"- **summary:** {s.summary}" if s.summary else "- **summary:** —")
|
||||
L.append(f"- **grounding:** source={s.source}, confidence={s.confidence}"
|
||||
+ (f", noop_headless={s.noop_headless}" if s.noop_headless else ""))
|
||||
if s.depends_on:
|
||||
L.append("- **depends on:** " + ", ".join("0x%x" % d for d in s.depends_on))
|
||||
if rev.get(op):
|
||||
L.append("- **depended on by:** " + ", ".join("0x%x" % d for d in rev[op]))
|
||||
if s.evidence:
|
||||
L.append(f"- **evidence:** {s.evidence}")
|
||||
if s.details:
|
||||
L += ["", s.details]
|
||||
L.append("")
|
||||
return "\n".join(L) + "\n"
|
||||
|
||||
def emit_coverage_md(model: M.Model) -> str:
|
||||
by_src = collections.Counter()
|
||||
by_conf = collections.Counter()
|
||||
by_cat = collections.Counter()
|
||||
named = 0
|
||||
for oc in model.opcodes.values():
|
||||
s = oc.semantics
|
||||
if s:
|
||||
by_src[s.source] += 1
|
||||
by_conf[s.confidence] += 1
|
||||
by_cat[s.category] += 1
|
||||
if s.name != oc.label:
|
||||
named += 1
|
||||
L = ["<!-- DO NOT EDIT -- generated from vm-map/opcodes.toml -->", "# Opcode Coverage (generated)", "",
|
||||
f"- opcodes: {len(model.opcodes)}", f"- given a distinct mnemonic: {named}", "",
|
||||
"## by source", ""]
|
||||
L += [f"- {k}: {v}" for k, v in sorted(by_src.items())]
|
||||
L += ["", "## by confidence", ""] + [f"- {k}: {by_conf[k]}" for k in ("high", "med", "low")]
|
||||
L += ["", "## by category", ""] + [f"- {k}: {v}" for k, v in sorted(by_cat.items())]
|
||||
return "\n".join(L) + "\n"
|
||||
```
|
||||
|
||||
Extend the `--build` block in `main` (after writing the shim, before `return 0`):
|
||||
|
||||
```python
|
||||
(paths.BUILD).mkdir(parents=True, exist_ok=True)
|
||||
(paths.BUILD / "opcodes.json").write_text(emit_json(model), encoding="utf-8")
|
||||
(paths.REPO / "docs" / "opcode-reference.md").write_text(emit_reference_md(model), encoding="utf-8")
|
||||
(paths.BUILD / "opcode-coverage.md").write_text(emit_coverage_md(model), encoding="utf-8")
|
||||
print("build: wrote build/opcodes.json, docs/opcode-reference.md, build/opcode-coverage.md")
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/test_opcodes.py`
|
||||
Expected: `test_emit_views` lines `ok`, `FAILURES: 0`.
|
||||
|
||||
- [ ] **Step 5: Checkpoint**
|
||||
|
||||
If git: `git add tools/opcodes_build.py tools/test_opcodes.py && git commit -m "feat(opcodes): emit opcodes.json, opcode-reference.md, coverage"`
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Real bootstrap + migrate legacy inferences + differential-verify
|
||||
|
||||
**Files:**
|
||||
- Create: `vm-map/opcodes.toml` (via bootstrap, then hand-edit)
|
||||
- Regenerate: `tools/age_opcodes_himegari.py`, `build/opcodes.json`, `docs/opcode-reference.md`, `build/opcode-coverage.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: everything above. No new code except a one-time migration helper (shown below; not committed as tooling).
|
||||
|
||||
- [ ] **Step 1: Snapshot current disassembly (regression baseline)**
|
||||
|
||||
Run (captures pre-change mnemonics for two representative scripts):
|
||||
```bash
|
||||
py -3.11 -X utf8 tools/sys4load.py ../../extracted/DATA1/SC0830.BIN > /tmp/sc0830.before.asm
|
||||
py -3.11 -X utf8 tools/sys4load.py ../../extracted/DATA1/MENU.BIN > /tmp/menu.before.asm
|
||||
```
|
||||
Expected: two files written (they contain `hotspot-branch`, `stmt-begin`, etc. from the current hand-written overlay).
|
||||
|
||||
- [ ] **Step 2: Bootstrap the real canonical file**
|
||||
|
||||
Run: `py -3.11 -X utf8 tools/opcodes_build.py --bootstrap`
|
||||
Expected: `bootstrap: 248 used opcodes; appended 248 new skeletons -> ...opcodes.toml`. Confirm `vm-map/opcodes.toml` exists with a `[meta]` block and 248 `[[opcode]]` blocks.
|
||||
|
||||
- [ ] **Step 3: Generate migration suggestions from the legacy overlay**
|
||||
|
||||
Run this one-time helper (reads the CURRENT hand-written `age_opcodes_himegari.py` before it gets overwritten, and prints TOML `[opcode.semantics]` blocks to paste):
|
||||
```bash
|
||||
py -3.11 -X utf8 - <<'PY'
|
||||
import sys, os
|
||||
sys.path.insert(0, "tools")
|
||||
from age_opcodes_himegari import INFERRED
|
||||
MAP = {"structure": "investigation", "context": "inference", "harness": "harness",
|
||||
"frida": "frida", "unicorn": "unicorn"}
|
||||
for op, e in sorted(INFERRED.items()):
|
||||
src = MAP.get(e.get("method", ""), "investigation")
|
||||
print(f"# --- 0x{op:x}: replace the seeded [opcode.semantics] with: ---")
|
||||
print("[opcode.semantics]")
|
||||
print(f'name = {e["name"]!r}')
|
||||
print(f'category = {e.get("category","unknown")!r}')
|
||||
print(f'summary = {e.get("note","")!r}')
|
||||
print(f'noop_headless = {str(bool(e.get("noop", False))).lower()}')
|
||||
print(f'source = {src!r}')
|
||||
print(f'confidence = {e.get("confidence","low")!r}')
|
||||
print("depends_on = [] # FILL: opcodes this reading rests on")
|
||||
print(f'evidence = {e.get("note","")!r}')
|
||||
print()
|
||||
PY
|
||||
```
|
||||
Expected: ~26 TOML blocks printed (0x71, 0x7a, 0x90, 0x97, 0xb6, 0x1a2, 0x1bc, 0x1bf, 0x1d2, 0x1d5, 0x1f4, 0x1f5, 0x1f7, 0x1fa, 0x1ff, 0x202, 0x203, 0x215, 0x217, 0x218, 0x21a, 0x21b, 0x258).
|
||||
|
||||
- [ ] **Step 4: Hand-migrate into `vm-map/opcodes.toml`**
|
||||
|
||||
For each printed block, find that opcode's `[opcode.semantics]` in `vm-map/opcodes.toml` and replace the seeded fields with the printed ones. Then add `depends_on`.
|
||||
|
||||
**`depends_on` principle:** it tracks *inference-on-inference* chains — list an op here ONLY when our reading rests on another op whose meaning is itself uncertain (our inference), so a later correction cascades. Reliance on a **validated core op** (e.g. `jcc 0xa0`, `call 0x8f`, `mov 0x55` — Kelebek-named and harness/RECOVER-proven) is a solid root: put that reasoning in `evidence` text, NOT in `depends_on` (adding it would also trip a spurious confidence-ceiling warning, since core ops seed at `med`). Applying this:
|
||||
- `0x90` (`hotspot-branch`): `depends_on = [0x1f4, 0x1f5]` (rests on our *inferred* stmt markers); set `details` to the multi-line evidence from `vm-map/himegari-opcode-notes.md` §F (paste the section body into a TOML `details = """ ... """`).
|
||||
- `0x97` (`hotspot-reg?`): `depends_on = [0x90]` (its role was inferred from interleaving with our inferred 0x90).
|
||||
- `0x1d5`/`0x1bc`/`0x1bf` (markers inferred from following `jcc`/`call`): `depends_on = []`; put "always follows jcc 0xa0" / "call 0x8f → 0x1bf" in `evidence` (jcc/call are validated roots).
|
||||
- Leave `depends_on = []` for ops grounded directly (`0x1f4`/`0x1f5`/`0x71` structural, harness-confirmed ADV ops).
|
||||
|
||||
Also fold the cross-cutting evidence from `himegari-opcode-notes.md` (bucket intros, the coverage narrative) that you want to keep into the relevant entries' `details` or the `[meta]` block — everything that must survive the retirement of that file in Task 7.
|
||||
|
||||
- [ ] **Step 5: Lint, then build**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
py -3.11 -X utf8 tools/opcodes_build.py --lint
|
||||
py -3.11 -X utf8 tools/opcodes_build.py --build
|
||||
```
|
||||
Expected: lint prints `0 errors` (confidence-ceiling warnings are acceptable — review each; downgrade confidence or fix a dependency if a warning is legitimate). Build writes all four artifacts.
|
||||
|
||||
- [ ] **Step 6: Differential verification (the proof it's a faithful drop-in)**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
py -3.11 -X utf8 tools/vm0.py --test # RECOVER unit test
|
||||
py -3.11 -X utf8 tools/vm0.py --sweep | tail -2 # coverage number
|
||||
py -3.11 -X utf8 tools/sys4load.py ../../extracted/DATA1/SC0830.BIN > /tmp/sc0830.after.asm
|
||||
py -3.11 -X utf8 tools/sys4load.py ../../extracted/DATA1/MENU.BIN > /tmp/menu.after.asm
|
||||
diff /tmp/sc0830.before.asm /tmp/sc0830.after.asm && echo "SC0830 identical"
|
||||
diff /tmp/menu.before.asm /tmp/menu.after.asm && echo "MENU identical"
|
||||
```
|
||||
Expected: `RECOVER unit test: PASS`; sweep still `282/294 = 95.9%`; both `diff`s empty (`... identical`). If a diff is non-empty, an opcode's `name` was migrated wrong — fix that entry in `opcodes.toml`, rebuild, re-diff.
|
||||
|
||||
- [ ] **Step 7: Checkpoint**
|
||||
|
||||
If git: `git add vm-map/opcodes.toml tools/age_opcodes_himegari.py build/opcodes.json docs/opcode-reference.md build/opcode-coverage.md && git commit -m "feat(opcodes): migrate to opcodes.toml as single source of truth; regenerate artifacts"`
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Retire superseded files + update docs/memory
|
||||
|
||||
**Files:**
|
||||
- Delete: `vm-map/opcodes-himegari.json`, `vm-map/himegari-opcode-notes.md`
|
||||
- Modify: `docs/PROJECT-STRUCTURE.md`, `C:\Users\m\.claude\projects\S--Game-Hacking-Eushully-Himegari\memory\himegari-port-status.md`, `...\memory\MEMORY.md`
|
||||
|
||||
**Interfaces:** none (documentation).
|
||||
|
||||
- [ ] **Step 1: Confirm content is preserved before deleting**
|
||||
|
||||
Verify the retiring files' load-bearing content now lives in `vm-map/opcodes.toml` / `docs/opcode-reference.md`:
|
||||
```bash
|
||||
grep -c "hotspot-branch" docs/opcode-reference.md # >=1
|
||||
grep -c "instruction_model" vm-map/opcodes.toml # ==1 (meta migrated from opcodes-himegari.json)
|
||||
grep -ci "hotspot" vm-map/opcodes.toml # 0x90/0x97 details migrated from notes §F
|
||||
```
|
||||
Expected: all nonzero. Only proceed if the §F evidence and the JSON's meta really made it into `opcodes.toml`.
|
||||
|
||||
- [ ] **Step 2: Delete the superseded files**
|
||||
|
||||
```bash
|
||||
rm vm-map/opcodes-himegari.json vm-map/himegari-opcode-notes.md
|
||||
```
|
||||
(The hand-maintained `build/opcode-coverage.md` is now overwritten by `--build`, so no delete needed — it's generated.)
|
||||
|
||||
- [ ] **Step 3: Update `docs/PROJECT-STRUCTURE.md`**
|
||||
|
||||
In the `vm-map/` and `tools/` sections, replace mentions of `opcodes-himegari.json` / `himegari-opcode-notes.md` and the hand-written `age_opcodes_himegari.py` with the new model:
|
||||
```
|
||||
├── vm-map/
|
||||
│ ├── opcodes.toml ★ CANONICAL opcode reference (hand-edited: ABI + semantics
|
||||
│ │ + provenance + depends_on). Source of truth for the opcode layer.
|
||||
│ ├── kelebek1-age-shared.cpp upstream opcode-table source
|
||||
│ └── opcode-leads.json, small-script-listings.md
|
||||
├── tools/
|
||||
│ ├── opcodes_build.py generator/linter: opcodes.toml -> {age_opcodes_himegari.py,
|
||||
│ │ build/opcodes.json, docs/opcode-reference.md, build/opcode-coverage.md}
|
||||
│ ├── opcodes_model.py load + lint (dangling-ref, confidence-ceiling, vocab) + dependents
|
||||
│ ├── age_opcodes.py Kelebek table, PRISTINE (ABI baseline; never edit)
|
||||
│ ├── age_opcodes_himegari.py GENERATED from opcodes.toml (do not hand-edit)
|
||||
```
|
||||
Add a bullet under Conventions: *"Opcode knowledge is edited ONLY in `vm-map/opcodes.toml`; run `tools/opcodes_build.py --build` to regenerate the shim/JSON/reference/coverage. `docs/opcode-reference.md` and `build/opcodes.json` are generated."*
|
||||
|
||||
- [ ] **Step 4: Update memory**
|
||||
|
||||
In `himegari-port-status.md`, update the tooling/opcode paragraph: opcode work is now a single source of truth at `vm-map/opcodes.toml` (+ `opcodes_build.py`/`opcodes_model.py`), generating `age_opcodes_himegari.py` + `build/opcodes.json` + `docs/opcode-reference.md` + coverage; `himegari-opcode-notes.md` and `opcodes-himegari.json` retired (content folded in). In `MEMORY.md`, adjust the `[SYS4 script format]` / status hooks that referenced those files.
|
||||
|
||||
- [ ] **Step 5: Final verification**
|
||||
|
||||
```bash
|
||||
py -3.11 -X utf8 tools/test_opcodes.py # FAILURES: 0
|
||||
py -3.11 -X utf8 tools/opcodes_build.py --lint # 0 errors
|
||||
py -3.11 -X utf8 tools/vm0.py --test # PASS
|
||||
```
|
||||
Expected: all green. No remaining references to the deleted files in `tools/` or `docs/`:
|
||||
```bash
|
||||
grep -rl "opcodes-himegari.json\|himegari-opcode-notes" tools docs || echo "no stale references"
|
||||
```
|
||||
Expected: `no stale references`.
|
||||
|
||||
- [ ] **Step 6: Checkpoint**
|
||||
|
||||
If git: `git add -A && git commit -m "docs(opcodes): retire superseded opcode files; update structure + memory"`
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
**Spec coverage** (each spec section → task):
|
||||
- Single source of truth / data flow → Tasks 3–6 (bootstrap, build, migrate). ✓
|
||||
- `[meta]` + `[[opcode]]` schema (ABI vs semantics) → Task 1 model + Task 3 skeleton. ✓
|
||||
- Source vocabulary + confidence → Task 2 lint (vocabulary) + Task 6 migration mapping. ✓
|
||||
- Generator subcommands (`--bootstrap/--build/--lint`) → Tasks 3, 4, 5. ✓
|
||||
- Four generated artifacts → Task 4 (shim) + Task 5 (json/md/coverage). ✓
|
||||
- Three+ lint checks (dangling-ref, dependents index, confidence-ceiling, vocab) → Task 2 (+ dependents in Task 1, rendered in Task 5). ✓
|
||||
- Bootstrap auto-fills observed_types from corpus → Task 3 `scan_corpus`/`skeleton_toml`. ✓
|
||||
- Migration of ~26 inferences + notes evidence → Task 6. ✓
|
||||
- Retire 3 files; keep Kelebek pristine → Task 7 (+ Global Constraint). ✓
|
||||
- Zero disruption to sys4load/vm0 → Task 4 emit rule (name != label) + Task 6 diff regression. ✓
|
||||
- Testing: regression (disasm diff, --test, --sweep), lint fixtures, round-trip-ish load → Tasks 1,2,6. ✓
|
||||
|
||||
**Placeholder scan:** no "TBD/handle edge cases"; the only intentionally-manual step is Task 6 Step 4 (paste migration blocks + assign `depends_on`), which is inherent to a human judgement task and is spelled out per-opcode.
|
||||
|
||||
**Type consistency:** `Model`/`Opcode`/`Semantics` fields are used identically across `load`, `lint`, `dependents`, and every `emit_*`. `emit_inferred_py` writes `dict(name=...)` → `INFERRED[op]["name"]`, matching `sys4load.py:84`. CLI flags `--bootstrap/--build/--lint/--toml` consistent between Task 3 and Tasks 4–5.
|
||||
181
docs/superpowers/specs/2026-07-06-opcode-reference-design.md
Normal file
181
docs/superpowers/specs/2026-07-06-opcode-reference-design.md
Normal file
@@ -0,0 +1,181 @@
|
||||
# Design: Living Opcode Reference (single-source-of-truth + provenance)
|
||||
|
||||
Status: **approved (design)** · Date: 2026-07-06 · Author: session work
|
||||
Related: `docs/phase-a-slice-plan.md`, `vm-map/himegari-opcode-notes.md`, `docs/PROJECT-STRUCTURE.md`
|
||||
|
||||
## Problem
|
||||
|
||||
Opcode knowledge for the AGE/SYS4 VM is currently spread across five artifacts:
|
||||
|
||||
| File | Role today | Consumed by |
|
||||
|------|-----------|-------------|
|
||||
| `tools/age_opcodes.py` | Kelebek table verbatim: `{op: (label, argc)}` + arg-types | sys4load, vm0 |
|
||||
| `tools/age_opcodes_himegari.py` | our `INFERRED` dict (name/category/noop/confidence/method/note) | sys4load |
|
||||
| `vm-map/opcodes-himegari.json` | data snapshot (arg-types, header fields, per-op array) | — (reference) |
|
||||
| `vm-map/himegari-opcode-notes.md` | prose evidence, buckets A–F | — (humans) |
|
||||
| `build/opcode-coverage.md` | coverage tiers | — (humans) |
|
||||
|
||||
The same fact lives in several places, so they drift (this session already found stale "6/8 scenes"
|
||||
wording, and `age_opcodes_himegari.py`'s `method` field duplicates prose in the notes doc). None of
|
||||
them record **why** we believe a claim or **what other claims it rests on** — so when a reversal proves
|
||||
one opcode wrong, there is no way to find the downstream inferences that depended on it.
|
||||
|
||||
## Goals
|
||||
|
||||
1. **One canonical, hand-edited file** as the single source of truth; every other opcode artifact is
|
||||
generated from it and can never disagree with it or with the VM.
|
||||
2. **Provenance per claim** — record the source (Kelebek / our harness / our investigation / Frida /
|
||||
Unicorn) and confidence, with the near-certain ABI facts kept separate from the fallible semantics.
|
||||
3. **Dependency tracking** — a semantic claim can declare the opcodes whose interpretation it rests on,
|
||||
so a correction's blast radius is visible and mechanically checkable.
|
||||
4. **Zero disruption to working tooling** — sys4load/vm0 keep importing the same Python interface,
|
||||
now generated.
|
||||
5. **Complete map** — an entry for every one of the 248 opcodes Himegari uses, so coverage is measurable
|
||||
and any opcode can be named as a dependency.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Not touching `tools/age_opcodes.py` (the Kelebek table): it stays pristine as the ABI baseline and the
|
||||
bootstrap input.
|
||||
- No per-call-site type modelling (arg types are per-instruction in the bytecode, not fixed per opcode).
|
||||
- No live-capture work (Frida/Unicorn) here — this is the reference structure; those remain sources a
|
||||
future entry can cite.
|
||||
- Not covering the ~300 Kelebek opcodes Himegari never uses.
|
||||
|
||||
## Architecture
|
||||
|
||||
One canonical file; everything else is generated:
|
||||
|
||||
```
|
||||
vm-map/opcodes.toml ← CANONICAL, hand-edited. [meta] + 248 [[opcode]] tables.
|
||||
│
|
||||
▼ tools/opcodes_build.py (generator + linter; stdlib tomllib read, no new dep)
|
||||
├─ tools/age_opcodes_himegari.py GENERATED — exposes INFERRED (sys4load imports unchanged)
|
||||
├─ build/opcodes.json GENERATED — language-neutral machine view (future C# VM)
|
||||
├─ docs/opcode-reference.md GENERATED — human reference incl. reverse dependents index
|
||||
└─ build/opcode-coverage.md GENERATED — counts by source / confidence / category
|
||||
```
|
||||
|
||||
- `tools/age_opcodes.py` (Kelebek) **stays**: ABI baseline + bootstrap input, never hand-edited.
|
||||
- **Retired** (content migrates into `opcodes.toml`, then re-emitted): hand-maintained
|
||||
`age_opcodes_himegari.py` (now generated), `vm-map/opcodes-himegari.json` (→ `build/opcodes.json`),
|
||||
`vm-map/himegari-opcode-notes.md` (per-op evidence → each entry's `details`; rendered into
|
||||
`docs/opcode-reference.md`), and hand-maintained `build/opcode-coverage.md`.
|
||||
- Editing loop: edit `opcodes.toml` → run `opcodes_build.py` → tooling, machine view, human doc, and
|
||||
coverage all update together.
|
||||
|
||||
## Canonical file format (`vm-map/opcodes.toml`)
|
||||
|
||||
### `[meta]` (migrated from `opcodes-himegari.json`)
|
||||
- `instruction_model` — the `1 + 2*argc` decode description (string).
|
||||
- `[meta.arg_types]` — type-code → name (`0x0`→immediate, `0x2`→string, `0x9`→local-int, …).
|
||||
- `[meta.header_fields]` — F0–F12 meanings.
|
||||
- `[meta.sources]`, `[meta.confidence]`, `[meta.categories]` — controlled vocabularies (documented
|
||||
below), so the linter can reject unknown values.
|
||||
|
||||
### `[[opcode]]` — ABI block (high trust)
|
||||
| field | meaning |
|
||||
|-------|---------|
|
||||
| `op` | opcode number (TOML `0x..`) |
|
||||
| `label` | Kelebek engine address (e.g. `"u0041BEB0"`) |
|
||||
| `argc` | argument count — **validated by our 481/481 clean decode** (`len = 1 + 2*argc`) |
|
||||
| `code_target_args` | 1-based arg indices that are jump targets (from Kelebek's notes; optional) |
|
||||
| `abi_source` | e.g. `"kelebek+decode-validated"` |
|
||||
| `abi_note` | optional caveat |
|
||||
|
||||
### `[opcode.semantics]` — our fallible layer (where cascades live)
|
||||
| field | meaning |
|
||||
|-------|---------|
|
||||
| `name` | short mnemonic for the disassembler (**the one field sys4load requires**) |
|
||||
| `category` | `marker\|structural\|control\|adv\|draw\|audio\|input\|compute\|unknown` |
|
||||
| `summary` | one-line description |
|
||||
| `noop_headless` | bool — safe for the headless VM v1 to skip/fall-through (no state/visible effect) |
|
||||
| `source` | `kelebek\|harness\|investigation\|frida\|unicorn\|inference` |
|
||||
| `confidence` | `high\|med\|low` |
|
||||
| `depends_on` | list of op ids whose interpretation this claim rests on (default `[]`) |
|
||||
| `evidence` | concise grounding (one line) |
|
||||
| `details` | optional multi-line long-form evidence (migrated from the notes doc) |
|
||||
| `confirm_by` | optional — what would raise confidence or falsify this |
|
||||
|
||||
### `[[opcode.semantics.args]]` — per-arg roles (array of tables, optional)
|
||||
| field | meaning |
|
||||
|-------|---------|
|
||||
| `i` | 1-based arg index |
|
||||
| `role` | semantic role (`"x"`, `"y"`, `"target:click"`, `"count"`, …) |
|
||||
| `observed_types` | list of type names actually seen at this position (**auto-filled from corpus**) |
|
||||
| `note` | optional |
|
||||
|
||||
### Source vocabulary (grounding roots vs. fallible)
|
||||
- `kelebek` — from the Kelebek table (ABI trusted; *semantics* from a later AGE title → treat as a lead,
|
||||
not truth). A root.
|
||||
- `harness` — confirmed by our dialogue-oracle diff (`build/text/dialogue.jsonl`). Empirically grounded root.
|
||||
- `investigation` — our own static/corpus analysis (may `depends_on` other ops).
|
||||
- `frida` / `unicorn` — live runtime capture / micro-execution. Roots (direct observation).
|
||||
- `inference` — reasoned from neighbours/context; weakest, usually has `depends_on`.
|
||||
|
||||
## Generator + linter (`tools/opcodes_build.py`)
|
||||
|
||||
Subcommands:
|
||||
- `--bootstrap` — create `opcodes.toml`: the used-opcode set is computed by scanning the corpus with
|
||||
`sys4load` (authoritative, reproducible); seed each with `op/label/argc` from `age_opcodes.py`,
|
||||
`abi_source="kelebek+decode-validated"` (argc is validated for all 248 by our 481/481 decode),
|
||||
`semantics.source="kelebek"`, `confidence` = `med` if Kelebek names it else `low` (bare
|
||||
`u004xxx`/`dev_ukn` label → `category="unknown"`); **auto-fill `observed_types` per arg position**
|
||||
from the corpus scan. Idempotent:
|
||||
re-running preserves hand-edited entries (only fills missing).
|
||||
- `--build` (default) — read `opcodes.toml`, run lint, then emit the four generated artifacts. The
|
||||
generated `age_opcodes_himegari.py` exposes `INFERRED: dict[int, dict]` with at least `name` (plus
|
||||
category/noop/confidence/source for future consumers) — a drop-in for the current interface.
|
||||
- `--lint` — run the three checks; nonzero exit on any error.
|
||||
|
||||
Generated-file guard: each emitted file carries a `DO NOT EDIT — generated from vm-map/opcodes.toml`
|
||||
header.
|
||||
|
||||
## Lint checks (the cascade mechanism)
|
||||
|
||||
1. **Dangling-ref (error):** every id in a `depends_on` must be an existing `op`. No orphan dependencies.
|
||||
2. **Reverse index / dependents (report):** build `A → [ops whose semantics depend on A]`; render it in
|
||||
`docs/opcode-reference.md` as a "depended on by: 0x…" line per opcode, so a revision's blast radius is
|
||||
visible. Also emitted to the lint output.
|
||||
3. **Confidence-ceiling (warning):** an entry's `confidence` may not exceed the minimum confidence among
|
||||
its `depends_on` targets (can't be `high` while resting on a `low`). Surfaces shaky foundations.
|
||||
4. **Vocabulary (error):** `category`/`source`/`confidence` must be in the `[meta]` controlled lists.
|
||||
|
||||
Chains terminate at a `kelebek`/`harness`/`frida`/`unicorn` root or at direct `evidence`.
|
||||
|
||||
## Bootstrap / migration procedure
|
||||
|
||||
1. Implement `opcodes_build.py`; run `--bootstrap` → `opcodes.toml` with 248 seeded skeletons + observed
|
||||
arg-type histograms.
|
||||
2. Hand-migrate our real inferences into their entries: the ~26 in `age_opcodes_himegari.py` plus the
|
||||
evidence in `himegari-opcode-notes.md` (buckets A–F, incl. the `0x90`/`0x97` deep-dive), each with
|
||||
proper `source`, `depends_on`, `evidence`, `details`.
|
||||
3. Run `--build`; confirm generated `age_opcodes_himegari.py` reproduces the current names, and
|
||||
`sys4load … --validate` + `vm0.py --test` + `vm0.py --sweep` are unchanged (differential check).
|
||||
4. Delete the retired hand-maintained files; update `docs/PROJECT-STRUCTURE.md` and the memory index.
|
||||
|
||||
## Testing / verification
|
||||
|
||||
- **Regression (must be byte-for-byte where it matters):** after migration, `sys4load` disassembly of a
|
||||
few scripts (MENU, SC0830) shows the same mnemonics; `vm0.py --test` PASS and `--sweep` still
|
||||
282/294. This proves the generated shim is a true drop-in.
|
||||
- **Generator round-trip:** `--build` is deterministic; re-running produces no diff.
|
||||
- **Lint unit checks:** craft a tiny fixture TOML exercising dangling-ref (error), a confidence-ceiling
|
||||
violation (warning), and an unknown category (error).
|
||||
- **Coverage sanity:** generated `opcode-coverage.md` totals reconcile with 248 used opcodes.
|
||||
|
||||
## Risks / open questions
|
||||
|
||||
- **TOML verbosity for 248 entries** — acceptable; entries are mostly skeletons and diffs read cleanly.
|
||||
- **`observed_types` staleness** — it's derived; re-run `--bootstrap --refresh-observed` (fills only that
|
||||
field) if the corpus set changes. Not load-bearing (evidence, not truth).
|
||||
- **Deciding `confidence` for Kelebek-named ops** — Kelebek names are ABI-labels from another title;
|
||||
default seed = `med` for named, and we downgrade/confirm as the VM exercises them (harness).
|
||||
- **git** — workspace isn't a git repo, so the design/spec aren't committed; offer `git init` later.
|
||||
|
||||
## Out of scope (future)
|
||||
|
||||
- Emitting a C#-native binding from `build/opcodes.json` (A1 will consume the JSON directly).
|
||||
- A GUI/queryable browser over the reference.
|
||||
- Automated "review-needed" flags on git-diff of `opcodes.toml` (the dependents index already gives the
|
||||
manual signal).
|
||||
158
docs/sys4-format-notes.md
Normal file
158
docs/sys4-format-notes.md
Normal file
@@ -0,0 +1,158 @@
|
||||
# SYS4 Script Format — Reverse-Engineering Notes (hex-first)
|
||||
|
||||
Derived purely from byte-pattern analysis of the 481 DATA1 `.BIN` scripts, before
|
||||
any Ghidra work on `AGE.EXE`. Confidence levels flagged per finding. Probe scripts
|
||||
live in `tools/probe_*.py`.
|
||||
|
||||
> **UPDATE 2026-07-05 — opcode set solved via Kelebek1's table.** Everything below
|
||||
> under "Instruction stream — PARTIAL" is now resolved: code = instructions of
|
||||
> `<opcode:u32> + argc*(<argtype:u32><value:u32>)`, length `1+2*argc`; inline strings
|
||||
> live after code inside `[0,F8)`, so stop decoding at the first type-2/`0x64` arg
|
||||
> offset. 476/476 scripts decode clean (0 unknown opcodes). Header fields F0–F5 are
|
||||
> **local-variable counts** (F0=local_integer_1, F1=local_floats, F2=local_strings_1,
|
||||
> F3=local_integer_2, F4=unknown, F5=local_strings_2). See `vm-mapping-plan.md` and
|
||||
> `vm-map/opcodes-himegari.json`. The tag values below (0x71/0x03/0x8F etc.) are the
|
||||
> *opcodes at table targets*: 0x8F=`call`, 0x03=`call-script`, 0x71=`u0041A7B0`.
|
||||
|
||||
## Header — CONFIRMED
|
||||
|
||||
Fixed 60-byte (0x3C) header: 8-byte magic + thirteen little-endian u32 fields.
|
||||
Verified across all 481 files.
|
||||
|
||||
```
|
||||
off field meaning evidence
|
||||
0x00 magic "SYS4422 " (0x53 59 53 34 34 32 32 20) 481/481 identical
|
||||
0x08 F0 scenario/script id or flags 98 distinct; 0x5 dominant (252×)
|
||||
0x0C F1 = 1 always 481/481 == 1
|
||||
0x10 F2 = 1 (457×) or 2 (20×); 4 outliers format/feature flag
|
||||
0x14 F3 0x06 (268×),0x04,0x01,0x05... 15 distinct — minor version?
|
||||
0x18 F4 = 1 always 481/481 == 1
|
||||
0x1C F5 = 2 (319×) or 1 (144×); rarely 3/5 flag
|
||||
0x20 F6 = 0x1C always 481/481 == 0x1C (header-size marker)
|
||||
0x24 F7 table-1 entry count see below
|
||||
0x28 F8 table-1 offset == code-section length ordering F8<=F10<=F12<=EOF
|
||||
0x2C F9 table-2 entry count
|
||||
0x30 F10 table-2 offset
|
||||
0x34 F11 table-3 entry count
|
||||
0x38 F12 table-3 offset
|
||||
0x3C body dword stream (code + 3 tables + strings)
|
||||
```
|
||||
|
||||
**All offsets/counts are in DWORDS (×4 bytes), relative to body start (0x3C).**
|
||||
The `0x1C` in F6 is the only *byte* count — it's the offset from F6's own position
|
||||
(0x20) back-referenced, i.e. a self-describing "28 bytes of descriptor follow" marker
|
||||
consistent with the SYS4/SYS5 family.
|
||||
|
||||
### Section layout (CONFIRMED — 0 ordering violations, 481/481)
|
||||
|
||||
```
|
||||
body[0 .. F8) CODE bytecode instruction stream
|
||||
body[F8 .. F10) TABLE-1 (F7 entries, 1 dword each) -> targets of type 0x71
|
||||
body[F10 .. F12) TABLE-2 (F9 entries, 1 dword each) -> targets of type 0x03
|
||||
body[F12 .. EOF) TABLE-3 (F11 entries, 1 dword each) -> targets of type 0x8F
|
||||
```
|
||||
|
||||
Every table entry is exactly **1 dword** — a pointer (dword index into body).
|
||||
Solved algebraically across the whole corpus: `(F10-F8)/F7 == (F12-F10)/F9 ==
|
||||
(EOF-F12)/F11 == 1` with zero non-integer results.
|
||||
|
||||
### The three tables are typed pointer indexes (CONFIRMED)
|
||||
|
||||
Each table points at body locations, and the dword *at* every target is a constant
|
||||
tag identifying the pointed-to construct:
|
||||
|
||||
| Table | count/off | Target dword tag | Hits | Meaning (inferred) |
|
||||
|---|---|---|---|---|
|
||||
| T1 | F7 / F8 | **0x71** | 26,445/26,445 | labels / call targets (operand at +2 is small: mostly 1) |
|
||||
| T2 | F9 / F10 | **0x03** | 3,018/3,018 | data/variable entries (operand at +2 large, e.g. addresses) |
|
||||
| T3 | F11 / F12| **0x8F** | 72,941/72,941 | instruction/line entries (largest table; operand at +2 huge) |
|
||||
|
||||
100% type purity — not a single target had a different tag. T3 is the big one
|
||||
(~73k entries corpus-wide), consistent with it being a per-instruction or
|
||||
per-source-line index (a debug/line table). T1 ≈ labels, T2 ≈ a smaller symbol set.
|
||||
|
||||
## Instruction stream — PARTIAL
|
||||
|
||||
The code section is a flat dword stream. Recurring "type/opcode" dwords observed:
|
||||
`0x03, 0x55, 0x6E, 0x6F, 0x71, 0x72, 0x8F`. These read as **operand-type tags**
|
||||
in a tagged-operand VM rather than raw opcodes, e.g. the repeating shape:
|
||||
|
||||
```
|
||||
... <TAG> <value> ... tag 0x02 => string pointer (see below)
|
||||
... 0x71 0x00 0x01 0x55 ... label marker + following instruction
|
||||
```
|
||||
|
||||
- **First body dword is 0x259 (601) in 301/481 files** — likely a standard
|
||||
"script entry" / prologue opcode. Second-most-common openers are small ints.
|
||||
- `0x55` appears pervasively as an instruction lead — probably the most common
|
||||
opcode (statement / expression separator).
|
||||
|
||||
*Full opcode semantics need the VM dispatch loop in `AGE.EXE` — that's the Ghidra
|
||||
task. These tags give a head start on labeling the disassembly.*
|
||||
|
||||
## Strings — CONFIRMED
|
||||
|
||||
- Stored inline in the body as **byte-complement (XOR 0xFF) cp932 / Shift-JIS**,
|
||||
packed 4 bytes per dword, NUL-terminated (a `\0` byte, i.e. `0xFF` after XOR,
|
||||
ends the string), then padded to the next dword.
|
||||
- **Referenced by a tagged operand: the dword `0x02` immediately followed by the
|
||||
dword-offset of the string.** Confirmed directly by xref:
|
||||
- `MENU.BIN`: `...2 30b...` @0x2A9 → offset 0x30B = `"ＭＳ 明朝"`; `...2 30e...` → `"loadmesskip menu"`
|
||||
- `SC0030.BIN`: `...2 ef80...` → `"▼G0030 ２章マップ０２１クリア"`; `...2 ef89...` → `"「よし、素晴らしい成果だな」"`
|
||||
- Decoder (validated — pulls clean Japanese dialogue):
|
||||
```python
|
||||
raw = bytes(b ^ 0xFF for b in body[off*4:]) # until a 0x00 appears
|
||||
text = raw.split(b"\0")[0].decode("cp932")
|
||||
```
|
||||
- Scene scripts hold the full dialogue; e.g. `SC0030.BIN` decodes to readable
|
||||
story text, choice-branch labels ("本来の分岐", "チェック用"), font names
|
||||
("ＭＳ 明朝"), and engine directives ("loadmesskip advset", "ＡＤＶパート").
|
||||
- Non-scene scripts (`MENU`, `ADDEXP`) contain only a handful of control strings —
|
||||
consistent with the inventory's subsystem/data-table categorization.
|
||||
|
||||
## Patch-override caveat (re-confirmed)
|
||||
|
||||
52 loose `.BIN` in the game root shadow their DATA1 copies at runtime and differ
|
||||
slightly in size. The disassembler should target the **root** copies where present.
|
||||
The header format is identical (same magic/layout) so tooling is copy-agnostic.
|
||||
|
||||
## What's solid vs. what needs Ghidra
|
||||
|
||||
**Solid (byte-verified, build a loader now):**
|
||||
- 60-byte header, all 13 fields, dword units, section boundaries
|
||||
- 3 typed pointer tables (0x71 / 0x03 / 0x8F), 1 dword each, 100% pure
|
||||
- String encoding (XOR-0xFF cp932) + reference mechanism (tag 0x02 + offset)
|
||||
|
||||
**Needs the VM (Ghidra on `AGE.EXE`):**
|
||||
- Opcode dispatch — confirm tagged-operand model, enumerate opcodes
|
||||
- Meaning of F0/F2/F3/F5 flag fields
|
||||
- Exact operand grammar per instruction (how many dwords each opcode consumes)
|
||||
- Semantics of T1/T2/T3 beyond "label/data/line" guesses
|
||||
|
||||
## Loader — DONE
|
||||
|
||||
`tools/sys4load.py` parses the header, splits the 4 sections, resolves the 3
|
||||
tables, decodes inline strings, and emits an assembly-ish listing with strings
|
||||
inlined at their `2 <off>` refs (opcodes not yet named — code is chunked by the
|
||||
T3 line-index). Importable API (`load()` → `Sys4Script`) plus CLI:
|
||||
|
||||
```
|
||||
sys4load.py <file.BIN> full listing
|
||||
sys4load.py <file.BIN> --summary header + section sizes + table/string counts
|
||||
sys4load.py <file.BIN> --strings decoded string pool
|
||||
sys4load.py <file.BIN> --json machine-readable structure
|
||||
sys4load.py <dir> --validate re-check invariants across a folder
|
||||
```
|
||||
|
||||
`--validate` over all 481 DATA1 scripts: **481 parsed clean, 0 failures, 0 impure
|
||||
table tags** — the format spec above is fully machine-verified. This listing is the
|
||||
artifact to diff against Ghidra output once the VM dispatch loop is mapped.
|
||||
|
||||
### Observations surfaced by the listing (leads for the VM work)
|
||||
- String-display sites look like `<opcode> 0x02 <str-off>` — e.g. opcodes `0x1A7`
|
||||
and `0x1A5` immediately precede string refs in `MENU.BIN`. Candidate text/message ops.
|
||||
- T3 entries are 3-dword records `[0x8F, 0x00, value]`; T1 labels are `[0x71, 0x00,
|
||||
value]`. T3 acts as a per-statement line index (editor metadata) — note scene
|
||||
scripts even carry editor annotation strings like `"LABEL"`, `"ループ開始"` (loop start).
|
||||
- `0x55` is the most frequent code lead (likely statement/expr separator); `0x09`
|
||||
recurs as an operand-type prefix (register/var reference?).
|
||||
153
docs/vm-mapping-plan.md
Normal file
153
docs/vm-mapping-plan.md
Normal file
@@ -0,0 +1,153 @@
|
||||
# SYS4 VM Mapping — Plan of Action
|
||||
|
||||
> **For the executing agent:** This is a reverse-engineering playbook. Work it phase-by-phase; each phase ends with a concrete, checkable deliverable. Verify claims against bytes before recording them. Validated seed data lives in `vm-map/`.
|
||||
|
||||
**Goal:** Decode the SYS4 bytecode into named instructions so the game logic can be re-implemented in Godot.
|
||||
|
||||
---
|
||||
|
||||
## ✅ BREAKTHROUGH (2026-07-05): the opcode set is already solved
|
||||
|
||||
**The prior "unpack AGE.EXE in Ghidra" critical path is no longer needed to disassemble scripts.** Kelebek1's decompiler ships a complete AGE opcode table that decodes this game directly.
|
||||
|
||||
**What was verified this session** (see `tools/validate_opcode_table.py`, run it to reproduce):
|
||||
|
||||
- Kelebek1/Eushully-Decompiler's `age-shared.cpp` contains an opcode table (`{op_code, label, argument_count}`) and a header parser that **explicitly handles the SYS4 signature** (`"SYS4"`, header length `0x3C`, cp932 XOR-0xFF strings) — this exact game's format.
|
||||
- The instruction model: **code = a flat sequence of instructions; each instruction = `<opcode:u32>` followed by `argument_count` arguments, where every argument is a `<type:u32><value:u32>` pair. Instruction length in dwords = `1 + 2*argc`.** Inline strings sit *after* the code inside the `[0,F8)` region; stop decoding at the lowest string offset referenced (a type-2 arg, or op `0x64` arg 1).
|
||||
- Applying that table to Himegari's scripts: **476 of 476 parseable scripts decode 100% clean — 1,463,788 instructions, 0 unknown opcodes, and all 37,392 inline-string arguments resolve to valid decoded strings.** (The 7 non-decoding `.BIN` are container-level non-scripts like `SYS4AB`/`SYS4INI`, different magic.)
|
||||
- Himegari uses **248 distinct opcodes; 52 have semantic names** (in `vm-map/opcodes-himegari.json`). The other 196 decode perfectly (known length) but have engine-internal names only (`u004xxxx`). **Caveat (measured 2026-07-06):** the named 52 are the dialogue/ADV core but cover only **72.6% of instruction volume**, not "the entire core" — the unnamed 27.4% is concentrated in the highest-frequency opcodes and must be partly addressed before Phase 4. See Phase 3's coverage correction.
|
||||
|
||||
**This resolves the header unknowns too.** Kelebek's `BinaryHeader` struct maps my F0–F12 exactly: `F0`=local_integer_1, `F1`=local_floats, `F2`=local_strings_1, `F3`=local_integer_2, `F4`=unknown, `F5`=local_strings_2, `F6`=sub_header_length(0x1C), then the three (length, offset) table pairs. The "flag fields" were **local-variable counts**. Arg `type` codes: 0=immediate, 1=float, 2=string, 3=global-int, 4=global-float, 5=global-string, 6=global-ptr, 8=global-string-ptr, 9=local-int, A=local-float, B=local-string, C=local-ptr, D=local-float-ptr, E=local-string-ptr.
|
||||
|
||||
**Consequence:** Unpacking `AGE.EXE` (still packed — see appendix) drops from *the blocker* to an *optional enrichment* used only to name the 196 unnamed opcodes' fine semantics, and even that has a cheaper dynamic alternative.
|
||||
|
||||
**Provenance / sources in `vm-map/`:** `kelebek1-age-shared.cpp` (the opcode table), `kelebek1-disassembler.cpp` (the parser), `opcodes-himegari.json` (validated table filtered to what this game uses), `opcode-leads.json` + `small-script-listings.md` (this session's static analysis, now confirmed).
|
||||
|
||||
---
|
||||
|
||||
## Global constraints
|
||||
|
||||
- **Python:** `py -3.11 -X utf8 …` always (Shift-JIS output needs utf8 mode on Windows).
|
||||
- **Authoritative copies:** the 52 loose game-folder `.BIN` shadow their `extracted/DATA1/` copies at runtime — target the game-folder copy where both exist. `sys4load.load()` is copy-agnostic; `paths.scripts()` resolves the override.
|
||||
- **Units:** all script offsets/counts are DWORDS (×4 bytes), relative to body start `0x3C`.
|
||||
- **Instruction rule:** `len_dwords = 1 + 2*argc`; args are `(type,value)`; **stop code decode at the first inline-string/array offset**, not blindly at `F8`.
|
||||
- Record confidence per finding (confirmed-by-bytes / confirmed-by-runtime / hypothesis).
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Port the opcode table into `sys4load.py`, disassemble everything *(✅ DONE 2026-07-05)*
|
||||
|
||||
**Deliverable:** `sys4load.py` emits real named instructions; every script disassembles with zero unknown opcodes. **Achieved: 481/481 DATA1 scripts decode fully clean; MENU.BIN and SC0030.BIN verified by hand.**
|
||||
|
||||
- [x] **1.1 — Embed the opcode table.** Full Kelebek table (548 entries) transcribed to `tools/age_opcodes.py` (`OPCODES`, `ARG_TYPES`, `CONTROL_FLOW`, `is_label_argument`), generated from `vm-map/kelebek1-age-shared.cpp`.
|
||||
- [x] **1.2 — Replace the T3-chunking stub.** `sys4load.py` now has `decode_code()` (the `1+2*argc` walker with shrinking `code_end`) and a rewritten `render_listing()` that prints mnemonics, typed operands, inline strings, and `label_xxxx:` control-flow anchors.
|
||||
- [x] **1.3 — Validate.** `tools/sys4load.py ../../extracted/DATA1 --validate` → **481/481 parsed clean, 481/481 opcode-decode clean.** MENU.BIN: 148 instrs, `set-font "ＭＳ 明朝"` + `comment` strings correct. SC0030.BIN: 11,951 instrs, `show-text` shows dialogue inline. (`tools/validate_opcode_table.py` still reproduces the standalone 476/476 over the merged root+DATA1 set.)
|
||||
- [x] **1.4 — Regression-guard:** container `--validate` still reports 481 clean, 0 failures, 0 impure tags.
|
||||
- [x] **1.5 — Disassembler is the artifact.** `sys4load.py <file>` prints the full listing; `--json` now includes decoded `code` (with `--json` + `to_dict(with_code=True)`), instruction counts, and decode-clean flag.
|
||||
|
||||
**Note:** 7 root `.BIN` are non-script engine indices with different magic (`SYS4INI` = `S4IC422`, `SYS4AB` = `S4AB`, etc.) — correctly rejected by the container parser, not scripts.
|
||||
|
||||
## Phase 2 — Extract data tables + dialogue *(✅ mostly DONE 2026-07-06)*
|
||||
|
||||
**Deliverable:** game database as JSON + full translatable dialogue corpus. Structure spec: `docs/PROJECT-STRUCTURE.md`. Extractors: `tools/extract_phase2.py`, `tools/extract_init.py`.
|
||||
|
||||
- [x] **2.0 — Project structure.** Established `docs/`, `build/{disasm,text,data,scripts-json}/`, `godot/`; game install stays read-only in place. Also relaxed the loader magic check to the `SYS4` family (`SYS4424` patch scripts now parse — was silently skipping 5 scripts).
|
||||
- [x] **2.1 — Text corpora.** `tools/extract_phase2.py` → 481/481 scripts: full disassembly (`build/disasm/*.asm`), per-script strings, `build/text/dialogue.jsonl` (**30,057 show-text lines** — the translation corpus), `build/text/strings.jsonl` (38,449 strings tagged by source opcode), `build/manifest.json`.
|
||||
- [x] **2.2 — `*INIT` data tables → JSON.** `tools/extract_init.py` auto-detects table shape (`name`/`numeric`/`footer`) → **SKINIT (129 skills), ITINIT (189 items), EBINIT (277 units)** [name: name+desc+fields], **CGINIT (379 CG entries)** [numeric: index-keyed columns], **MPINIT (1472 map records)** [footer: 50-value arrays from the file footer]. Validated; see `build/data/README.md`. Column addresses are raw engine globals — naming them (attack/cost/…) needs the global-var map (Phase 3-adjacent).
|
||||
- [ ] **2.3 — `STINIT` (74 stages) needs a bespoke parser.** Heterogeneous per-stage `copy-to-global` param blocks + a variable number of clear-condition strings per stage — fits none of the three auto modes. Its clear-condition text is already in `build/text/STINIT.strings.txt`; only the per-stage numeric params await a custom extractor. Low priority (stages are also encoded in the SC-scene scripts).
|
||||
- [x] **2.4 — Partial global-var map BUILT + wired into the disassembler.** `tools/global_map.py` → `build/global-var-map.{json,md}` (16,354/49,435 globals labelled: string tables, `*INIT` field arrays, 122 record tables w/ strides, current-entity index pointers). `sys4load` renders the labels inline (`=rec[s30]`, `=current-entity-index?`). See `docs/name-resolution.md`.
|
||||
- [ ] **2.5 — Grow the global-var map (future, incremental).** Static first: fold in `*MES` writers; label 2D record tables by their reader scripts. Then Frida to name *which stat* each field is. Full detail: `docs/name-resolution.md` → "Future step — growing the map". Also deferred: `call-script` id→name resolution (engine-level — SCJUMP.BIN decode or Frida; see `docs/name-resolution.md` #1).
|
||||
|
||||
## Phase 3 — Name the unnamed opcodes *(top ~20 BEFORE Phase 4; the rest on demand)*
|
||||
|
||||
> **⚠️ Coverage correction (measured 2026-07-06).** The earlier framing — "52 named ops
|
||||
> cover the entire core, name the other 196 lazily" — is **overstated**. Across the full
|
||||
> corpus (1,503,166 instructions, all 481 scripts), **named opcodes are only 72.6% of
|
||||
> instructions; the 195 unnamed `u004xxxx` ops are 27.4%** — and that 27% is front-loaded
|
||||
> into the *most common* opcodes, not a deferrable long tail. The top unnamed ops by
|
||||
> frequency: `0x1f4`/`0x1f5` (**60,297 each** — equal counts → a begin/end or push/pop
|
||||
> pair, both zero-arg), `0x1d5` (34k), `0x1bc` (27k), `0x71` (26,445 — *exactly* the
|
||||
> corpus T1 label-table entry count, so it's the **label-definition pseudo-op**, nameable
|
||||
> by structure for free), `0x1a2` (18k), `0x7a` (17k, argc 3, follows arithmetic →
|
||||
> computational), `0x1d2` (17k). **A Godot VM hits these in the first few instructions of
|
||||
> any script.** So naming the top ~20 is a *prerequisite* for Phase 4, not a lazy
|
||||
> follow-on. Only the genuine long tail (rare ops) is deferrable. Reproduce the measurement
|
||||
> by iterating `sys4load.load` over the corpus and bucketing `ins.opcode` against
|
||||
> `age_opcodes.OPCODES` (label starting `u00`/`dev_ukn` = unnamed).
|
||||
|
||||
> **⚠️ Named labels are from a *different* AGE title.** The 52 semantic labels are
|
||||
> transcribed from Kelebek's table for a *later* AGE game. The opcode **number + argc** are
|
||||
> validated for Himegari (481/481 clean decode proves structure), but the **semantics are
|
||||
> not independently verified**. The ADV/text core is empirically safe — the 30,057-line
|
||||
> `build/text/dialogue.jsonl` is proof that `show-text`/`end-text-line`/the string
|
||||
> mechanism are right, and arithmetic/control-flow labels are corroborated by operand-type
|
||||
> and jump-target consistency. The exposure is the **effectful named ops you can't see in
|
||||
> text output** (`play-voice 0xc4`, `draw-texture 0x1fb`, sound/UI/draw ops) — Frida-confirm
|
||||
> those against Himegari before the VM relies on them; don't assume them.
|
||||
|
||||
**Do this before Phase 4:** name/classify the ~20 highest-frequency unnamed opcodes.
|
||||
Most fall to free inference (3.0); a few opaque effectful ones want a Frida session;
|
||||
computational ones suit Unicorn. Everything below still applies — it's the *ordering* that
|
||||
changes, not the toolkit. The genuine rare tail stays lazy (name on demand).
|
||||
|
||||
- [x] **3.0 — Inference pass DONE (2026-07-06).** Classified the top 21 unnamed opcodes →
|
||||
**instruction coverage 72.62% (named) → 96.94% (classified)**; ~90.5% is VM-handleable by
|
||||
inference alone. Tooling: `tools/opcode_context.py` (evidence gatherer). Results:
|
||||
`vm-map/himegari-opcode-notes.md` (per-op evidence), `tools/age_opcodes_himegari.py`
|
||||
(`INFERRED` dict consumed by the disassembler + future VM), `build/opcode-coverage.md`
|
||||
(tiers + Frida/Unicorn shortlist). `sys4load` now renders inferred names (verified: MENU's
|
||||
`label-def 0x71` land exactly on its T1 targets). Key findings: `0x1f4`/`0x1f5` = stmt
|
||||
begin/end brackets, `0x1d5`/`0x1bc`/`0x1bf` = block markers (all zero-arg no-ops); `0x71`
|
||||
= label-def (count == T1 size); `0x21b`/`0x1d2`/`0x258` = tentative-no-op statement metadata
|
||||
(harness-verify); `0x7a` = ADV text param, `0x202/0x203/0x1f7/0x1fa/0x217/0x218/0x21a/0x1ff`
|
||||
= draw/UI, `0xb6` = audio, `0x215` = count/search — the effectful/computational Frida/Unicorn
|
||||
shortlist. Reserve live tools for those; rare tail (3%) stays lazy.
|
||||
|
||||
### 3.1 — Frida: dynamic observation *(primary tool for effectful opcodes)*
|
||||
|
||||
Frida injects a JS engine into the **running** game and hooks functions live. It sidesteps the packer (memory is already decrypted by the time you attach), gives ground-truth behavior, and lets you correlate an opcode with its on-screen/audible effect — the only reliable way to name rendering/audio/input/save/UI handlers. Two stages:
|
||||
|
||||
- [ ] **3.1a — Locate the dispatch loop.** Kelebek's `u004xxxx` addresses are from a *different* AGE title and will NOT match Himegari's `AGE.EXE`, so find Himegari's dispatch first. Best anchor: search process memory for a known script's opening opcode sequence (you have every script decoded), set a **hardware read breakpoint / `MemoryAccessMonitor` guard page** on its first opcode dword; when the VM fetches it, the instruction pointer is inside the dispatch fetch. Alternate anchors: breakpoint a winmm/DirectSound call and trigger `play-voice` (0xC4), then walk the stack back; or pattern-scan for the bounds-check + `call [table + opcode*4]`. **Payoff:** read the jump-table base → you get the handler address for all 548 opcodes in Himegari at once.
|
||||
- [ ] **3.1b — Instrument + correlate.** `Interceptor.attach` the dispatch (or a specific handler); log opcode + operand `(type,value)` pairs (read from the bytecode pointer — layout known) + effect. Three correlation techniques: **API** (hook a basket of D3D9/winmm/user32/file APIs; see which an unknown handler calls), **behavioral** (trigger one in-game action, diff the opcode trace vs. baseline to attribute ops to subsystems), **memory** (log which global/local var-bank slots — sized by header F0–F5 — the handler reads/writes).
|
||||
- **Setup / gotchas:** `pip install frida-tools`; **attach to the already-running game** (`frida AGE.EXE`) after the title screen rather than spawning — this skips the packer's startup anti-debug. 32-bit x86 target. Japanese locale required to run. Eushully's protector *may* detect Frida's injected thread; if it trips, quiet it (ScyllaHide-style hooks or `frida-gadget`).
|
||||
|
||||
### 3.2 — Unicorn: microexecution *(complement for computational opcodes)*
|
||||
|
||||
Unicorn is a bare CPU emulator (no OS). It is the **better** tool for the *pure-computation* handlers — arithmetic/bit/string/array helpers and especially the `CALC*` damage/stat formulas — where you want the *exact* operation, not a label. It is **blind** to effectful handlers: the instant one calls D3D9/winmm/file APIs it runs into unmapped code and stubbing tells you nothing (the effect *is* the meaning). Do not use it as a Frida replacement.
|
||||
|
||||
- [ ] **3.2a — Microexecute a handler.** Map the handler's code + a synthetic VM state (variable bank + operand), run from entry to `ret`, read back what changed; sweep inputs to recover the formula deterministically, offline.
|
||||
- [ ] **3.2b — Preferred combo: Frida-snapshot → Unicorn-replay.** Use Frida (3.1a) to find handler addresses and dump the relevant memory (code + var banks + globals) at a known-good moment (e.g. mid-battle); load that snapshot into Unicorn and microexecute individual handlers with input sweeps. Gets Frida's context-setup for free + Unicorn's determinism. **Caveat:** microexecution only recovers behavior that's a pure function of the captured state — if a handler reads a global you didn't snapshot, results are wrong silently. Fine for pure ops; a rabbit hole for stateful ones (leave those to live Frida). Needs a decrypted image to feed (a dump, or bytes pulled via Frida) since `AGE.EXE` is packed.
|
||||
|
||||
### 3.3 — Cross-reference siblings *(free, do alongside 3.0)*
|
||||
- [ ] Kelebek's labels come from a later AGE title; marcussacana/EushullyEditor targets *Kamidori* (same SYS4 era). Diff their handler notes for the specific opcodes you need.
|
||||
|
||||
### 3.4 — Static unpack + Ghidra *(last resort)*
|
||||
- [ ] Only if the above stall. See appendix — dump the decrypted image, load in Ghidra, read the handler at its address. High effort; reserve for genuinely opaque ops that Frida/Unicorn can't pin down.
|
||||
|
||||
## Phase 4 — Godot re-implementation
|
||||
|
||||
**Deliverable:** the AGE VM running Himegari scripts in Godot.
|
||||
|
||||
> **Prerequisite:** don't start Phase 4 against a blank opcode set — the top ~20 unnamed ops
|
||||
> (Phase 3 preamble) are hit in the first few instructions of any script. Do that thin
|
||||
> naming slice first, or bring-up stalls immediately on `0x1f4`/`0x71`/etc.
|
||||
|
||||
- [ ] **4.0 — Stand up the validation harness *first* (before writing VM opcodes).** The
|
||||
strongest correctness oracle already exists in `build/`: for a given scene script, the VM's
|
||||
emitted `show-text` sequence must match that file's lines in `build/text/dialogue.jsonl`.
|
||||
Wire this as an automated diff (drive one `SC####` script → collect show-text → compare to
|
||||
the 30k-line corpus filtered by `file`). This turns "is the VM right?" into a per-scene
|
||||
regression test and catches control-flow/branch bugs (wrong jcc → wrong dialogue order)
|
||||
early. Extend later to assert extracted-table reads (SKINIT/ITINIT/EBINIT JSON) once
|
||||
data-driven opcodes come online.
|
||||
- [ ] **4.1 — Re-implement the VM** (GDScript/C#): a dword-fetch loop, the core opcodes (arithmetic, comparisons, `jmp`/`call`/`jcc`, `mov`, string ops) **plus the high-frequency unnamed ops named in Phase 3**, the global/local variable banks (sized by the header's F0–F5 counts), and the ADV layer (`show-text`/`end-text-line`/`wait-for-input`/`set-font`/`play-voice`/`draw-*`). Note the named set is only ~73% of instruction volume — budget for the unnamed remainder. The bytecode-heavy design (damage calc, dungeon loop, battle flow are all scripts) makes re-implementation the right call over transpilation.
|
||||
- [ ] **4.2 — Fill opcodes on demand** from Phase 3's genuine long tail as scripts exercise them (the top ~20 are already done as a Phase 3/4 prerequisite).
|
||||
- [ ] **4.3 — Deferred:** save-file format (reverse `SAVE.BIN` only if the port must read existing saves).
|
||||
|
||||
---
|
||||
|
||||
## Appendix — `AGE.EXE` is packed (relevant to Phase 3.4, and as the image source for 3.2)
|
||||
|
||||
Verified this session (`tools/pack_check.py`): 32-bit PE, code sections at max entropy (8.00), blank section names, IAT RVA 0, no plaintext anchors. `SYS4AB.BIN` (magic `S4AB`, entropy 7.94) is a second encrypted engine image whose header stores `AGE.EXE`'s exact size (`0x0010E000`) — likely the patched VM the loader maps.
|
||||
|
||||
Static analysis therefore requires a **runtime dump first** — you're dumping for *analysis* not redistribution, so don't chase OEP: launch to the title screen (Japanese locale required), then dump the decrypted image and load it in Ghidra. Tools: **PE-sieve** (CLI, agent-drivable: `pe-sieve.exe /pid <PID> /imp 3`) or **x32dbg + Scylla + ScyllaHide** (GUI, handles the anti-debug). Validate the dump by confirming `SYS4422`/`.BIN`/`DATA1` now appear in plaintext. **But prefer Frida dynamic hooking (Phase 3.1) — it avoids the unpack entirely.**
|
||||
Reference in New Issue
Block a user