chore: initialize age-reimpl repo

Reverse-engineering + open reimplementation workspace for Eushully's AGE/SYS4 engine (first target: Himegari). The repo root is age-reimpl/; the original game install and the extracted ALF data are siblings outside the repo and are never tracked. build/ (derived corpora) is gitignored and regenerated by the tools. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 12:58:30 -04:00
commit aa3da00bfa
36 changed files with 9829 additions and 0 deletions
--- a/docs/himegari-port-reference.md
+++ b/docs/himegari-port-reference.md
@@ -0,0 +1,180 @@
+# Princess Hunting Dungeon Meister — Godot Port Reference
+
+Working reference for porting *姫狩りダンジョンマイスター* (Eushully, 2009) to Godot.
+
+Source install: `C:\Program Files (x86)\Eushully\姫狩りダンジョンマイスター\`
+Workspace: `S:\Game Hacking\Eushully\Himegari\` — three siblings: `姫狩りダンジョンマイスター\` (pristine game), `extracted\` (extracted ALF data), `age-reimpl\` (our work). See `docs/PROJECT-STRUCTURE.md`.
+
+---
+
+## Engine background
+
+Eushully built their own engine, called **AGE**, with three major generations distinguished by their config/index files and archive magic bytes:
+
+| Generation | Index file | ALF magic | Era (rough) |
+|---|---|---|---|
+| SYS3 | `sys3ini.bin` | `S3IC` / `S3IN` | mid-2000s |
+| SYS4 | `sys4ini.bin` / `.AAI` | `S4AC` / `S4IC` | late-2000s — early-2010s |
+| SYS5 | `sys5ini.bin` | `S5IN` / `S5IC` / `S5A` | ~2013+ |
+
+**Confirmed SYS4** for this game — `extracted\DATA1\SYSTEM4.BIN` is the smoking gun, and the `APPEND01.AAI` sidecar is the SYS4 signature pattern.
+
+Install dir contents: `AGE.EXE`, `DATA1-5.ALF`, `APPEND01.ALF`, `APPEND01.AAI`, plus `BinExtractALF.exe` and `AGE Patch.exe`.
+
+---
+
+## Tier 1 — Primary references
+
+### [Kelebek1/Eushully-Decompiler](https://github.com/Kelebek1/Eushully-Decompiler)
+C++ decompiler + recompiler for `.bin` scripts, with `extract_alf.py` for archive extraction.
+
+- Targets **SYS5** specifically (its extractor scans for `SYS5INI.BIN` and `S5*` magic). Will need adaptation for SYS4, but the structure is parallel and the opcode work is gold as a starting framework.
+- Most actively maintained (latest release Aug 2024). 22 stars but the most serious decompiler effort in the wild.
+- Known limitation: opcode tables are hand-built per engine version — community has open requests to add more opcodes.
+- See [extract_alf.py](https://github.com/Kelebek1/Eushully-Decompiler/blob/master/extract_alf.py) for the cleanest reference implementation of ALF parsing.
+
+### [morkt/GARbro](https://github.com/morkt/GARbro)
+Swiss-army VN asset extractor. Has [`ArcFormats/Eushully/ArcALF.cs`](https://github.com/morkt/GARbro/blob/master/ArcFormats/Eushully/ArcALF.cs) covering **all three engine generations** (SYS3/4/5), plus contained formats AGF (graphics), AOG (audio), SCR.
+
+- The more directly applicable extractor for this game — use it as the authoritative reference for the SYS4 ALF format and LZSS decompression of the index.
+- C# / .NET. Source is readable and well-organized per format.
+
+## Tier 2 — Supporting tools
+
+### [marcussacana/EushullyEditor](https://github.com/marcussacana/EushullyEditor)
+C# library for string-level edits to `.bin` scripts (translation-focused, not full decompilation).
+
+- Pre-configured for *Kamidori Alchemy Meister* and *Kami no Rhapsody*.
+- Useful as a second reference for the script string-table structure, especially since Kamidori is the same SYS4 era as Dungeon Meister.
+
+### Existing tools already on disk
+- `BinExtractALF.exe` — pre-built ALF extractor someone already ran here. Worth confirming origin (likely from a Japanese tools site or HongFire / Mikocon thread).
+- `extracted\DATA1\AGF2BMP2AGF.exe` + `LzssCpp.dll` — bidirectional AGF↔BMP converter for the proprietary graphics format. Original by **asmodean** ([asmodean.reverse.net](http://asmodean.reverse.net/) — canonical source of older Japanese game tooling, worth bookmarking).
+- `AGE Patch.exe` — likely an English localization or no-DVD patch, not a tool per se.
+
+---
+
+## Current extraction state
+
+| Archive | Files | Size | Contents |
+|---|---|---|---|
+| DATA1 | 3,749 | 2.2 GB | **Mixed core data**: 1511 AGF + 1508 BMP (graphics, ~all already converted), **481 .BIN scripts**, 238 WAV (system sfx), 9 cursors, the AGF2BMP2AGF tool + LzssCpp.dll |
+| DATA2 | 985 | 791 MB | Event CGs (`.AGF` only) — pure graphics archive |
+| DATA3 | 39 | 124 MB | BGM tracks (`.OGG` only) |
+| DATA4 | 9,733 | 471 MB | Voice files (`.OGG` only) — character lines |
+| DATA5 | 210 | 291 MB | More `.AGF` graphics — likely appendix/extra content |
+
+### Notable scripts already extracted (DATA1)
+- `SYSTEM4.BIN` — engine config/setup (the SYS4 index)
+- `HISTORY.BIN`, `MENU.BIN`, `HIDEWIN.BIN` — UI/system scripts
+- `SC0000.BIN` and presumably hundreds of `SC####.BIN` — scene scripts, the actual game logic
+
+### Status by category
+- **Graphics**: 1508/1511 in DATA1 already converted to BMP. ~1,195 AGFs remaining in DATA2 and DATA5.
+- **Audio**: Fully extracted as standard OGG Vorbis (9,772 files). Godot ingests natively — no further work.
+- **Scripts**: 481 `.BIN` files extracted but still in AGE bytecode form. This is the real porting work.
+
+---
+
+## Next steps
+
+### Where things stand (read this first)
+Container format is **fully reversed and machine-verified** (header, 4 sections, 3
+typed pointer tables, inline string encoding — all 481 scripts parse clean). Tooling
+exists: `tools/sys4load.py` (loader + disassembler-ish dumper + `--validate`) and the
+`tools/probe_*.py` analysis scripts. Companion docs: [script-inventory.md](script-inventory.md)
+(what the 481 scripts are) and [sys4-format-notes.md](sys4-format-notes.md) (the byte format).
+
+**The one blocker for everything downstream is opcode semantics** — the code stream is
+a tagged-dword format whose instruction meanings are unknown. That needs the VM dispatch
+loop in `AGE.EXE`, which is the Ghidra task below. Graphics conversion (DATA2/5 AGFs) and
+save-format work remain deferred.
+
+### Immediate (no tools needed beyond what's on disk)
+1. ~~**Relocate the `Output\` tree**~~ **DONE** — workspace now at `S:\Game Hacking\Eushully\Himegari\姫狩りダンジョンマイスター\`.
+2. **Convert remaining AGFs** in DATA2 (985 files) and DATA5 (210 files) with `AGF2BMP2AGF.exe`. *(Deferred — graphics not needed yet.)* The 3-file DATA1 gap is `CHAPTER.AGF`, `LOGO.AGF`, `TEST.AGF`.
+3. ~~**Inventory the script files**~~ **DONE** — see [script-inventory.md](script-inventory.md). Key findings: all 481 scripts share magic `SYS4422 `; 52 loose root-dir `.BIN` files are patch overrides that shadow DATA1 copies (use those as authoritative); heavy game logic (damage calc, dungeon loop, battle flow) lives in bytecode, favoring a VM re-implementation in Godot.
+1. ~~**Relocate the `Output\` tree**~~ **DONE** — workspace now at `S:\Game Hacking\Eushully\Himegari\姫狩りダンジョンマイスター\`.
+2. **Convert remaining AGFs** in DATA2 (985 files) and DATA5 (210 files) with `AGF2BMP2AGF.exe`. *(Deferred — graphics not needed yet.)* The 3-file DATA1 gap is `CHAPTER.AGF`, `LOGO.AGF`, `TEST.AGF`.
+3. ~~**Inventory the script files**~~ **DONE** — see [script-inventory.md](script-inventory.md). Key findings: all 481 scripts share magic `SYS4422 `; 52 loose root-dir `.BIN` files are patch overrides that shadow DATA1 copies (use those as authoritative); heavy game logic (damage calc, dungeon loop, battle flow) lives in bytecode, favoring a VM re-implementation in Godot.
+
+### Header structure — DONE (hex-first, pre-Ghidra)
+See [sys4-format-notes.md](sys4-format-notes.md). Confirmed across all 481 files: 60-byte header (magic `SYS4422 ` + 13 u32 fields, all offsets in dwords), body split into CODE + 3 typed pointer tables (tags 0x71/0x03/0x8F, 1 dword each, 100% pure) + inline strings. Strings are XOR-0xFF cp932, referenced by a `0x02 <dword-offset>` tagged operand — verified by decoding real dialogue out of `SC0030.BIN`. Remaining unknowns (opcode dispatch, flag fields F0/F2/F3/F5) need the VM.
+
+### NEXT ACTION — port the opcode table (see [vm-mapping-plan.md](vm-mapping-plan.md))
+**The full phased playbook lives in [vm-mapping-plan.md](vm-mapping-plan.md)** — start there.
+
+> ✅ **BREAKTHROUGH (verified 2026-07-05): the opcode set is already solved.**
+> Kelebek1's decompiler (`age-shared.cpp`) ships an AGE opcode table that decodes
+> this game directly — **476/476 scripts decode 100% clean, 1.46M instructions, 0
+> unknown opcodes, 37,392/0 string args resolved.** Model: code = instructions of
+> `<opcode> + argc*(<type><value>)`, length `1+2*argc`; stop code at the first inline
+> string offset. Himegari uses 248 opcodes, 52 named (see `vm-map/opcodes-himegari.json`).
+> Header fields F0–F5 are now known = local-variable counts (Kelebek's `BinaryHeader`).
+> **Unpacking `AGE.EXE` is no longer the blocker** — it's demoted to optional Phase 3
+> enrichment (prefer Frida hooking). Reproduce: `tools/validate_opcode_table.py`.
+>
+> ⚠️ **Coverage nuance (measured 2026-07-06):** "solved" means every instruction *decodes*
+> (structure/length known, 481/481 clean). It does **not** mean every instruction is
+> *understood*: the 52 named ops are only **72.6% of instruction volume**; the unnamed
+> `u004xxxx` 27.4% is concentrated in the highest-frequency opcodes and can't be fully
+> deferred before the Godot VM. Also the 52 semantic *labels* come from a later AGE title —
+> numbers+argc are validated for Himegari, semantics are not (text ops confirmed by the
+> dialogue corpus; effectful ops need Frida confirmation). See `vm-mapping-plan.md` Phase 3.
+
+> ⚠️ **Note on `AGE.EXE`:** still packed (max-entropy sections, IAT RVA 0). Only
+> relevant if you later need to name the 196 unnamed opcodes statically — see the
+> plan's appendix. `SYS4AB.BIN` (magic `S4AB`) is a 2nd encrypted engine image.
+
+Original Ghidra sketch (superseded; kept only for the appendix unpack route):
+
+4. **Find the dispatch loop.** Look for where `AGE.EXE` reads a script's first body
+   dword (`0x259` in 301/481 files — likely the prologue/entry opcode) and switches on
+   dword tag values. Expect a large switch or jump table. Diff against Kelebek1's SYS5
+   opcode-handler addresses to map SYS4 equivalents (same engine family, parallel structure).
+5. **Seed the opcode map from known anchors** (from `sys4load.py` dumps, already observed):
+   - `0x02` = string-pointer operand tag (confirmed).
+   - `0x1A7` / `0x1A5` = opcodes that immediately precede string refs in `MENU.BIN` →
+     candidate text/message-display instructions. Start here; they're the easiest to confirm.
+   - `0x8F` / `0x71` / `0x03` = the tags at T3-line / T1-label / T2-data table targets.
+   - `0x55` = most frequent code lead (likely statement/expr separator); `0x09` = recurring
+     operand-type prefix (register/var ref?).
+   Cross-reference with marcussacana's Kamidori (same SYS4 era) as a second opcode source.
+6. **Encode the opcode table into `sys4load.py`.** As each opcode's length + operand
+   grammar is confirmed in Ghidra, add it so the dumper decodes real instructions instead
+   of chunking by the T3 line-index. `MENU.BIN` (3 strings, small control flow) is the
+   validation target — decode it fully first, then a mid-size `SC####` scene end-to-end.
+7. **(Optional sanity check)** Locate the ALF mount/decrypt code to confirm GARbro's parser
+   matches this build. Low priority — extraction already succeeded, so this is only if an
+   archive anomaly shows up.
+
+### Data-table extraction (unblocks in parallel once a few opcodes are known)
+8. The `*INIT` giants (`STINIT` 579 KB stages, `EBINIT` 338 KB enemies, `MPINIT` 330 KB
+   maps, `ITINIT` items, `SKINIT` skills, `CGINIT` gallery) are static data tables. Once
+   the T2/data-entry grammar is understood they can be dumped to JSON/CSV **without** a
+   complete opcode set — an early, high-value win for the game database.
+9. **Bulk-extract all dialogue** — `sys4load.py --strings` already pulls clean cp932 text
+   from every scene today. A batch run over all `SC####`/`SP####` yields the full script
+   corpus for translation, independent of the VM work.
+
+### Deferred tracks
+10. **Save-file format** — reverse `SAVE.BIN` / the save layout only if the port must read
+    existing saves. Likely a small struct dump; low priority until gameplay runs.
+11. **Godot representation** — decide: re-implement the AGE VM in GDScript/C#, or transpile
+    `.BIN` → native Godot scenes. The inventory already tilts toward **re-implementing the
+    VM** (damage calc, dungeon loop, battle flow all live in bytecode, so a transpiler would
+    have to cover nearly the whole opcode set anyway). Ghidra's view of how much logic sits
+    in `AGE.EXE` vs. bytecode makes the final call.
+
+---
+
+## Sources
+
+- [Kelebek1/Eushully-Decompiler](https://github.com/Kelebek1/Eushully-Decompiler)
+- [Kelebek1/Eushully-Decompiler/extract_alf.py](https://github.com/Kelebek1/Eushully-Decompiler/blob/master/extract_alf.py)
+- [Kelebek1issue #2 — AGE engine version discussion](https://github.com/Kelebek1/Eushully-Decompiler/issues/2)
+- [marcussacana/EushullyEditor](https://github.com/marcussacana/EushullyEditor)
+- [morkt/GARbro](https://github.com/morkt/GARbro)
+- [GARbro ArcFormats/Eushully/ArcALF.cs](https://github.com/morkt/GARbro/blob/master/ArcFormats/Eushully/ArcALF.cs)
+- [Eushully Fandom Wiki — Princess Hunting Dungeon Meister notes](https://eushully.fandom.com/wiki/Princess_Hunting_Dungeon_Meister:Notes)
+- [asmodean.reverse.net](http://asmodean.reverse.net/) — canonical home of older Japanese game tooling