docs(opcodes): retire superseded opcode files; update structure + doc pointers
- rm vm-map/opcodes-himegari.json, vm-map/himegari-opcode-notes.md (folded into opcodes.toml; in history) - PROJECT-STRUCTURE.md: opcodes.toml single-source-of-truth + generated artifacts + convention - repoint dead references in vm-mapping-plan/sys4-format-notes/himegari-port-reference/phase-a-slice-plan Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
gamer147
@@ -33,8 +33,10 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
|
||||
│ │ where the game / extracted / build dirs are. All
|
||||
│ │ tools import it; relocatable with no other edits.
|
||||
│ ├── sys4load.py loader + disassembler (opcode-decoding)
|
||||
│ ├── age_opcodes.py 548-entry AGE opcode/arg-type table (pristine)
|
||||
│ ├── age_opcodes_himegari.py inferred Himegari opcode-name overlay
|
||||
│ ├── age_opcodes.py 548-entry Kelebek AGE opcode/arg-type table (PRISTINE; never edit)
|
||||
│ ├── opcodes_build.py generator/linter: vm-map/opcodes.toml -> the 4 artifacts below
|
||||
│ ├── opcodes_model.py load + lint (dangling-ref, confidence-ceiling, vocab) + dependents
|
||||
│ ├── age_opcodes_himegari.py GENERATED from opcodes.toml (do not hand-edit)
|
||||
│ ├── vm0.py headless Python VM (Phase A0); `--test` = RECOVER unit test
|
||||
│ ├── extract_phase2.py batch: disasm + text + data extraction
|
||||
│ ├── extract_init.py, global_map.py … *INIT parsers, global-var map builder
|
||||
@@ -46,9 +48,10 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
|
||||
│ └── LzssCpp.dll its LZSS codec dependency
|
||||
│
|
||||
├── vm-map/ VM / reverse-engineering reference artifacts
|
||||
│ ├── opcodes-himegari.json validated opcode table (what this game uses)
|
||||
│ ├── opcodes.toml ★ CANONICAL opcode reference (hand-edited: ABI + semantics
|
||||
│ │ + provenance + depends_on). Single source of truth for opcodes.
|
||||
│ ├── kelebek1-age-shared.cpp / -disassembler.cpp upstream opcode-table source
|
||||
│ └── opcode-leads.json, small-script-listings.md, himegari-opcode-notes.md
|
||||
│ └── opcode-leads.json, small-script-listings.md
|
||||
│
|
||||
├── docs/ all documentation
|
||||
│ ├── PROJECT-STRUCTURE.md this file
|
||||
@@ -58,7 +61,8 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
|
||||
│ ├── himegari-port-reference.md master reference + engine background
|
||||
│ ├── name-resolution.md call-script + global-var name recovery
|
||||
│ ├── sys4-format-notes.md byte-level container format
|
||||
│ └── script-inventory.md what the 481 scripts are
|
||||
│ ├── script-inventory.md what the 481 scripts are
|
||||
│ └── opcode-reference.md GENERATED from opcodes.toml (human-readable opcode reference)
|
||||
│
|
||||
├── build/ DERIVED (our-work-side) — generated by tools/; disposable
|
||||
│ ├── disasm/ <NAME>.asm — human-readable disassembly, one per script
|
||||
@@ -69,7 +73,8 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
|
||||
│ ├── data/ parsed data tables (*INIT → JSON)
|
||||
│ ├── scripts-json/ machine-readable full dumps (on demand via --json)
|
||||
│ ├── global-var-map.{json,md} partial global-variable name map
|
||||
│ └── manifest.json, opcode-coverage.md
|
||||
│ ├── opcodes.json GENERATED from opcodes.toml (machine view for the C# VM)
|
||||
│ └── manifest.json, opcode-coverage.md (opcode-coverage.md GENERATED from opcodes.toml)
|
||||
│
|
||||
└── godot/ DELIVERABLE — the Godot/C# engine project (built in Phase A2+)
|
||||
```
|
||||
@@ -90,6 +95,10 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
|
||||
- **`build/` and `extracted/` are disposable.** `build/` regenerates via `tools/extract_phase2.py`
|
||||
(or `sys4load.py`); `extracted/` regenerates via `bin/BinExtractALF.exe` on the `.ALF` files.
|
||||
Safe to delete and rebuild; do not hand-edit.
|
||||
- **Opcode knowledge is edited ONLY in `vm-map/opcodes.toml`** (ABI + semantics + provenance +
|
||||
`depends_on`). Run `tools/opcodes_build.py --build` to regenerate the shim (`tools/age_opcodes_himegari.py`),
|
||||
machine JSON (`build/opcodes.json`), reference (`docs/opcode-reference.md`), and coverage. `--lint`
|
||||
checks dangling deps / confidence-ceiling / vocabulary. Kelebek's `tools/age_opcodes.py` stays pristine.
|
||||
- **Encoding:** all generated text is UTF-8 (source strings are cp932/Shift-JIS, decoded on
|
||||
extraction). Run Python as `py -3.11 -X utf8`.
|
||||
- **The game install is a runnable unit** — do not relocate `AGE.EXE`/`*.ALF`/DLLs relative to each
|
||||
|
||||
@@ -109,7 +109,7 @@ See [sys4-format-notes.md](sys4-format-notes.md). Confirmed across all 481 files
|
||||
> this game directly — **476/476 scripts decode 100% clean, 1.46M instructions, 0
|
||||
> unknown opcodes, 37,392/0 string args resolved.** Model: code = instructions of
|
||||
> `<opcode> + argc*(<type><value>)`, length `1+2*argc`; stop code at the first inline
|
||||
> string offset. Himegari uses 248 opcodes, 52 named (see `vm-map/opcodes-himegari.json`).
|
||||
> string offset. Himegari uses 248 opcodes, 52 named (see `vm-map/opcodes.toml`).
|
||||
> Header fields F0–F5 are now known = local-variable counts (Kelebek's `BinaryHeader`).
|
||||
> **Unpacking `AGE.EXE` is no longer the blocker** — it's demoted to optional Phase 3
|
||||
> enrichment (prefer Frida hooking). Reproduce: `tools/validate_opcode_table.py`.
|
||||
|
||||
@@ -118,7 +118,7 @@ all parsing/decoding (no new parser).
|
||||
SC/SP), `--scene NAME` (detailed diff for one script), plus `load_oracle`/`subsequence_status`.
|
||||
|
||||
**op 0x90 investigated in depth — it is input chrome, NOT a correctness hole** (full evidence:
|
||||
`vm-map/himegari-opcode-notes.md` §F). Kelebek left it "ukn"; corpus analysis resolves it:
|
||||
`vm-map/opcodes.toml` op 0x90 `details`). Kelebek left it "ukn"; corpus analysis resolves it:
|
||||
`0x90 x y w h tgt_a tgt_b tgt_c` (argc 7) is a **cursor/input hotspot hit-test** that branches per
|
||||
interaction outcome and **falls through to pc+1 when nothing matches** (design-confirmed: enc.len 15
|
||||
lands the next instr on the fall-through statement). It occurs ONLY in a shared ADV-chrome subroutine
|
||||
|
||||
@@ -11,7 +11,7 @@ live in `tools/probe_*.py`.
|
||||
> offset. 476/476 scripts decode clean (0 unknown opcodes). Header fields F0–F5 are
|
||||
> **local-variable counts** (F0=local_integer_1, F1=local_floats, F2=local_strings_1,
|
||||
> F3=local_integer_2, F4=unknown, F5=local_strings_2). See `vm-mapping-plan.md` and
|
||||
> `vm-map/opcodes-himegari.json`. The tag values below (0x71/0x03/0x8F etc.) are the
|
||||
> `vm-map/opcodes.toml`. The tag values below (0x71/0x03/0x8F etc.) are the
|
||||
> *opcodes at table targets*: 0x8F=`call`, 0x03=`call-script`, 0x71=`u0041A7B0`.
|
||||
|
||||
## Header — CONFIRMED
|
||||
|
||||
@@ -15,13 +15,13 @@
|
||||
- Kelebek1/Eushully-Decompiler's `age-shared.cpp` contains an opcode table (`{op_code, label, argument_count}`) and a header parser that **explicitly handles the SYS4 signature** (`"SYS4"`, header length `0x3C`, cp932 XOR-0xFF strings) — this exact game's format.
|
||||
- The instruction model: **code = a flat sequence of instructions; each instruction = `<opcode:u32>` followed by `argument_count` arguments, where every argument is a `<type:u32><value:u32>` pair. Instruction length in dwords = `1 + 2*argc`.** Inline strings sit *after* the code inside the `[0,F8)` region; stop decoding at the lowest string offset referenced (a type-2 arg, or op `0x64` arg 1).
|
||||
- Applying that table to Himegari's scripts: **476 of 476 parseable scripts decode 100% clean — 1,463,788 instructions, 0 unknown opcodes, and all 37,392 inline-string arguments resolve to valid decoded strings.** (The 7 non-decoding `.BIN` are container-level non-scripts like `SYS4AB`/`SYS4INI`, different magic.)
|
||||
- Himegari uses **248 distinct opcodes; 52 have semantic names** (in `vm-map/opcodes-himegari.json`). The other 196 decode perfectly (known length) but have engine-internal names only (`u004xxxx`). **Caveat (measured 2026-07-06):** the named 52 are the dialogue/ADV core but cover only **72.6% of instruction volume**, not "the entire core" — the unnamed 27.4% is concentrated in the highest-frequency opcodes and must be partly addressed before Phase 4. See Phase 3's coverage correction.
|
||||
- Himegari uses **248 distinct opcodes; 52 have semantic names** (in `vm-map/opcodes.toml`). The other 196 decode perfectly (known length) but have engine-internal names only (`u004xxxx`). **Caveat (measured 2026-07-06):** the named 52 are the dialogue/ADV core but cover only **72.6% of instruction volume**, not "the entire core" — the unnamed 27.4% is concentrated in the highest-frequency opcodes and must be partly addressed before Phase 4. See Phase 3's coverage correction.
|
||||
|
||||
**This resolves the header unknowns too.** Kelebek's `BinaryHeader` struct maps my F0–F12 exactly: `F0`=local_integer_1, `F1`=local_floats, `F2`=local_strings_1, `F3`=local_integer_2, `F4`=unknown, `F5`=local_strings_2, `F6`=sub_header_length(0x1C), then the three (length, offset) table pairs. The "flag fields" were **local-variable counts**. Arg `type` codes: 0=immediate, 1=float, 2=string, 3=global-int, 4=global-float, 5=global-string, 6=global-ptr, 8=global-string-ptr, 9=local-int, A=local-float, B=local-string, C=local-ptr, D=local-float-ptr, E=local-string-ptr.
|
||||
|
||||
**Consequence:** Unpacking `AGE.EXE` (still packed — see appendix) drops from *the blocker* to an *optional enrichment* used only to name the 196 unnamed opcodes' fine semantics, and even that has a cheaper dynamic alternative.
|
||||
|
||||
**Provenance / sources in `vm-map/`:** `kelebek1-age-shared.cpp` (the opcode table), `kelebek1-disassembler.cpp` (the parser), `opcodes-himegari.json` (validated table filtered to what this game uses), `opcode-leads.json` + `small-script-listings.md` (this session's static analysis, now confirmed).
|
||||
**Provenance / sources in `vm-map/`:** `kelebek1-age-shared.cpp` (the opcode table), `kelebek1-disassembler.cpp` (the parser), `opcodes.toml` (validated table filtered to what this game uses), `opcode-leads.json` + `small-script-listings.md` (this session's static analysis, now confirmed).
|
||||
|
||||
---
|
||||
|
||||
@@ -93,7 +93,7 @@ changes, not the toolkit. The genuine rare tail stays lazy (name on demand).
|
||||
- [x] **3.0 — Inference pass DONE (2026-07-06).** Classified the top 21 unnamed opcodes →
|
||||
**instruction coverage 72.62% (named) → 96.94% (classified)**; ~90.5% is VM-handleable by
|
||||
inference alone. Tooling: `tools/opcode_context.py` (evidence gatherer). Results:
|
||||
`vm-map/himegari-opcode-notes.md` (per-op evidence), `tools/age_opcodes_himegari.py`
|
||||
`vm-map/opcodes.toml` (per-op evidence + provenance), `tools/age_opcodes_himegari.py`
|
||||
(`INFERRED` dict consumed by the disassembler + future VM), `build/opcode-coverage.md`
|
||||
(tiers + Frida/Unicorn shortlist). `sys4load` now renders inferred names (verified: MENU's
|
||||
`label-def 0x71` land exactly on its T1 targets). Key findings: `0x1f4`/`0x1f5` = stmt
|
||||
|
||||
Reference in New Issue
Block a user