docs(opcodes): retire superseded opcode files; update structure + doc pointers
- rm vm-map/opcodes-himegari.json, vm-map/himegari-opcode-notes.md (folded into opcodes.toml; in history) - PROJECT-STRUCTURE.md: opcodes.toml single-source-of-truth + generated artifacts + convention - repoint dead references in vm-mapping-plan/sys4-format-notes/himegari-port-reference/phase-a-slice-plan Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
gamer147
@@ -33,8 +33,10 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
|
||||
│ │ where the game / extracted / build dirs are. All
|
||||
│ │ tools import it; relocatable with no other edits.
|
||||
│ ├── sys4load.py loader + disassembler (opcode-decoding)
|
||||
│ ├── age_opcodes.py 548-entry AGE opcode/arg-type table (pristine)
|
||||
│ ├── age_opcodes_himegari.py inferred Himegari opcode-name overlay
|
||||
│ ├── age_opcodes.py 548-entry Kelebek AGE opcode/arg-type table (PRISTINE; never edit)
|
||||
│ ├── opcodes_build.py generator/linter: vm-map/opcodes.toml -> the 4 artifacts below
|
||||
│ ├── opcodes_model.py load + lint (dangling-ref, confidence-ceiling, vocab) + dependents
|
||||
│ ├── age_opcodes_himegari.py GENERATED from opcodes.toml (do not hand-edit)
|
||||
│ ├── vm0.py headless Python VM (Phase A0); `--test` = RECOVER unit test
|
||||
│ ├── extract_phase2.py batch: disasm + text + data extraction
|
||||
│ ├── extract_init.py, global_map.py … *INIT parsers, global-var map builder
|
||||
@@ -46,9 +48,10 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
|
||||
│ └── LzssCpp.dll its LZSS codec dependency
|
||||
│
|
||||
├── vm-map/ VM / reverse-engineering reference artifacts
|
||||
│ ├── opcodes-himegari.json validated opcode table (what this game uses)
|
||||
│ ├── opcodes.toml ★ CANONICAL opcode reference (hand-edited: ABI + semantics
|
||||
│ │ + provenance + depends_on). Single source of truth for opcodes.
|
||||
│ ├── kelebek1-age-shared.cpp / -disassembler.cpp upstream opcode-table source
|
||||
│ └── opcode-leads.json, small-script-listings.md, himegari-opcode-notes.md
|
||||
│ └── opcode-leads.json, small-script-listings.md
|
||||
│
|
||||
├── docs/ all documentation
|
||||
│ ├── PROJECT-STRUCTURE.md this file
|
||||
@@ -58,7 +61,8 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
|
||||
│ ├── himegari-port-reference.md master reference + engine background
|
||||
│ ├── name-resolution.md call-script + global-var name recovery
|
||||
│ ├── sys4-format-notes.md byte-level container format
|
||||
│ └── script-inventory.md what the 481 scripts are
|
||||
│ ├── script-inventory.md what the 481 scripts are
|
||||
│ └── opcode-reference.md GENERATED from opcodes.toml (human-readable opcode reference)
|
||||
│
|
||||
├── build/ DERIVED (our-work-side) — generated by tools/; disposable
|
||||
│ ├── disasm/ <NAME>.asm — human-readable disassembly, one per script
|
||||
@@ -69,7 +73,8 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
|
||||
│ ├── data/ parsed data tables (*INIT → JSON)
|
||||
│ ├── scripts-json/ machine-readable full dumps (on demand via --json)
|
||||
│ ├── global-var-map.{json,md} partial global-variable name map
|
||||
│ └── manifest.json, opcode-coverage.md
|
||||
│ ├── opcodes.json GENERATED from opcodes.toml (machine view for the C# VM)
|
||||
│ └── manifest.json, opcode-coverage.md (opcode-coverage.md GENERATED from opcodes.toml)
|
||||
│
|
||||
└── godot/ DELIVERABLE — the Godot/C# engine project (built in Phase A2+)
|
||||
```
|
||||
@@ -90,6 +95,10 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
|
||||
- **`build/` and `extracted/` are disposable.** `build/` regenerates via `tools/extract_phase2.py`
|
||||
(or `sys4load.py`); `extracted/` regenerates via `bin/BinExtractALF.exe` on the `.ALF` files.
|
||||
Safe to delete and rebuild; do not hand-edit.
|
||||
- **Opcode knowledge is edited ONLY in `vm-map/opcodes.toml`** (ABI + semantics + provenance +
|
||||
`depends_on`). Run `tools/opcodes_build.py --build` to regenerate the shim (`tools/age_opcodes_himegari.py`),
|
||||
machine JSON (`build/opcodes.json`), reference (`docs/opcode-reference.md`), and coverage. `--lint`
|
||||
checks dangling deps / confidence-ceiling / vocabulary. Kelebek's `tools/age_opcodes.py` stays pristine.
|
||||
- **Encoding:** all generated text is UTF-8 (source strings are cp932/Shift-JIS, decoded on
|
||||
extraction). Run Python as `py -3.11 -X utf8`.
|
||||
- **The game install is a runnable unit** — do not relocate `AGE.EXE`/`*.ALF`/DLLs relative to each
|
||||
|
||||
@@ -109,7 +109,7 @@ See [sys4-format-notes.md](sys4-format-notes.md). Confirmed across all 481 files
|
||||
> this game directly — **476/476 scripts decode 100% clean, 1.46M instructions, 0
|
||||
> unknown opcodes, 37,392/0 string args resolved.** Model: code = instructions of
|
||||
> `<opcode> + argc*(<type><value>)`, length `1+2*argc`; stop code at the first inline
|
||||
> string offset. Himegari uses 248 opcodes, 52 named (see `vm-map/opcodes-himegari.json`).
|
||||
> string offset. Himegari uses 248 opcodes, 52 named (see `vm-map/opcodes.toml`).
|
||||
> Header fields F0–F5 are now known = local-variable counts (Kelebek's `BinaryHeader`).
|
||||
> **Unpacking `AGE.EXE` is no longer the blocker** — it's demoted to optional Phase 3
|
||||
> enrichment (prefer Frida hooking). Reproduce: `tools/validate_opcode_table.py`.
|
||||
|
||||
@@ -118,7 +118,7 @@ all parsing/decoding (no new parser).
|
||||
SC/SP), `--scene NAME` (detailed diff for one script), plus `load_oracle`/`subsequence_status`.
|
||||
|
||||
**op 0x90 investigated in depth — it is input chrome, NOT a correctness hole** (full evidence:
|
||||
`vm-map/himegari-opcode-notes.md` §F). Kelebek left it "ukn"; corpus analysis resolves it:
|
||||
`vm-map/opcodes.toml` op 0x90 `details`). Kelebek left it "ukn"; corpus analysis resolves it:
|
||||
`0x90 x y w h tgt_a tgt_b tgt_c` (argc 7) is a **cursor/input hotspot hit-test** that branches per
|
||||
interaction outcome and **falls through to pc+1 when nothing matches** (design-confirmed: enc.len 15
|
||||
lands the next instr on the fall-through statement). It occurs ONLY in a shared ADV-chrome subroutine
|
||||
|
||||
@@ -11,7 +11,7 @@ live in `tools/probe_*.py`.
|
||||
> offset. 476/476 scripts decode clean (0 unknown opcodes). Header fields F0–F5 are
|
||||
> **local-variable counts** (F0=local_integer_1, F1=local_floats, F2=local_strings_1,
|
||||
> F3=local_integer_2, F4=unknown, F5=local_strings_2). See `vm-mapping-plan.md` and
|
||||
> `vm-map/opcodes-himegari.json`. The tag values below (0x71/0x03/0x8F etc.) are the
|
||||
> `vm-map/opcodes.toml`. The tag values below (0x71/0x03/0x8F etc.) are the
|
||||
> *opcodes at table targets*: 0x8F=`call`, 0x03=`call-script`, 0x71=`u0041A7B0`.
|
||||
|
||||
## Header — CONFIRMED
|
||||
|
||||
@@ -15,13 +15,13 @@
|
||||
- Kelebek1/Eushully-Decompiler's `age-shared.cpp` contains an opcode table (`{op_code, label, argument_count}`) and a header parser that **explicitly handles the SYS4 signature** (`"SYS4"`, header length `0x3C`, cp932 XOR-0xFF strings) — this exact game's format.
|
||||
- The instruction model: **code = a flat sequence of instructions; each instruction = `<opcode:u32>` followed by `argument_count` arguments, where every argument is a `<type:u32><value:u32>` pair. Instruction length in dwords = `1 + 2*argc`.** Inline strings sit *after* the code inside the `[0,F8)` region; stop decoding at the lowest string offset referenced (a type-2 arg, or op `0x64` arg 1).
|
||||
- Applying that table to Himegari's scripts: **476 of 476 parseable scripts decode 100% clean — 1,463,788 instructions, 0 unknown opcodes, and all 37,392 inline-string arguments resolve to valid decoded strings.** (The 7 non-decoding `.BIN` are container-level non-scripts like `SYS4AB`/`SYS4INI`, different magic.)
|
||||
- Himegari uses **248 distinct opcodes; 52 have semantic names** (in `vm-map/opcodes-himegari.json`). The other 196 decode perfectly (known length) but have engine-internal names only (`u004xxxx`). **Caveat (measured 2026-07-06):** the named 52 are the dialogue/ADV core but cover only **72.6% of instruction volume**, not "the entire core" — the unnamed 27.4% is concentrated in the highest-frequency opcodes and must be partly addressed before Phase 4. See Phase 3's coverage correction.
|
||||
- Himegari uses **248 distinct opcodes; 52 have semantic names** (in `vm-map/opcodes.toml`). The other 196 decode perfectly (known length) but have engine-internal names only (`u004xxxx`). **Caveat (measured 2026-07-06):** the named 52 are the dialogue/ADV core but cover only **72.6% of instruction volume**, not "the entire core" — the unnamed 27.4% is concentrated in the highest-frequency opcodes and must be partly addressed before Phase 4. See Phase 3's coverage correction.
|
||||
|
||||
**This resolves the header unknowns too.** Kelebek's `BinaryHeader` struct maps my F0–F12 exactly: `F0`=local_integer_1, `F1`=local_floats, `F2`=local_strings_1, `F3`=local_integer_2, `F4`=unknown, `F5`=local_strings_2, `F6`=sub_header_length(0x1C), then the three (length, offset) table pairs. The "flag fields" were **local-variable counts**. Arg `type` codes: 0=immediate, 1=float, 2=string, 3=global-int, 4=global-float, 5=global-string, 6=global-ptr, 8=global-string-ptr, 9=local-int, A=local-float, B=local-string, C=local-ptr, D=local-float-ptr, E=local-string-ptr.
|
||||
|
||||
**Consequence:** Unpacking `AGE.EXE` (still packed — see appendix) drops from *the blocker* to an *optional enrichment* used only to name the 196 unnamed opcodes' fine semantics, and even that has a cheaper dynamic alternative.
|
||||
|
||||
**Provenance / sources in `vm-map/`:** `kelebek1-age-shared.cpp` (the opcode table), `kelebek1-disassembler.cpp` (the parser), `opcodes-himegari.json` (validated table filtered to what this game uses), `opcode-leads.json` + `small-script-listings.md` (this session's static analysis, now confirmed).
|
||||
**Provenance / sources in `vm-map/`:** `kelebek1-age-shared.cpp` (the opcode table), `kelebek1-disassembler.cpp` (the parser), `opcodes.toml` (validated table filtered to what this game uses), `opcode-leads.json` + `small-script-listings.md` (this session's static analysis, now confirmed).
|
||||
|
||||
---
|
||||
|
||||
@@ -93,7 +93,7 @@ changes, not the toolkit. The genuine rare tail stays lazy (name on demand).
|
||||
- [x] **3.0 — Inference pass DONE (2026-07-06).** Classified the top 21 unnamed opcodes →
|
||||
**instruction coverage 72.62% (named) → 96.94% (classified)**; ~90.5% is VM-handleable by
|
||||
inference alone. Tooling: `tools/opcode_context.py` (evidence gatherer). Results:
|
||||
`vm-map/himegari-opcode-notes.md` (per-op evidence), `tools/age_opcodes_himegari.py`
|
||||
`vm-map/opcodes.toml` (per-op evidence + provenance), `tools/age_opcodes_himegari.py`
|
||||
(`INFERRED` dict consumed by the disassembler + future VM), `build/opcode-coverage.md`
|
||||
(tiers + Frida/Unicorn shortlist). `sys4load` now renders inferred names (verified: MENU's
|
||||
`label-def 0x71` land exactly on its T1 targets). Key findings: `0x1f4`/`0x1f5` = stmt
|
||||
|
||||
@@ -1,111 +0,0 @@
|
||||
# Himegari opcode inference notes
|
||||
|
||||
Classification of the top ~20 unnamed opcodes (Phase 3.0, inference-only, 2026-07-06).
|
||||
Machine-readable form: `tools/age_opcodes_himegari.py` (`INFERRED`). Evidence gathered with
|
||||
`tools/opcode_context.py` over the 481-script `Output/DATA1` corpus (1,502,986 instructions).
|
||||
|
||||
Baseline: named opcodes = **72.62%** of instruction volume. Classifying the top 20 unnamed
|
||||
ops covers to **~96.7%** cumulative. Kelebek labels (`u004xxxx`) are engine addresses from a
|
||||
*later* AGE title — opcode number + argc are validated for Himegari (481/481 clean decode);
|
||||
the meanings below are inferred and flagged by confidence + confirmation method.
|
||||
|
||||
## Buckets
|
||||
|
||||
### A. Statement/scope scaffolding — zero-arg, no operands, safe VM no-ops (HIGH)
|
||||
The compiler brackets statements/blocks with operand-less markers. They carry no data and
|
||||
sit at structural boundaries, so the VM v1 skips them; the Phase-4 dialogue harness confirms.
|
||||
|
||||
- **0x1f4 `stmt-begin`** (60,297; 4.01%) — opens scripts (first instr of ADDEN/ADDEXP/…), and
|
||||
`stmt-end 0x1f5 → 0x1f4` pairs 25,899×. Successor is the statement's first real op (eq/mov/call).
|
||||
- **0x1f5 `stmt-end`** (60,297; 4.01%) — equal count to 0x1f4; precedes `exit`/next statement.
|
||||
`... → 0x1f5 → stmt-begin/lt/show-text`.
|
||||
- **0x1d5 `cond-block`** (34,238; 2.28%) — **always** preceded by `jcc`; marks the fall-through
|
||||
(condition-true) block entry. `jcc cond -1 label; 0x1d5; <body>`.
|
||||
- **0x1bc `block-mark`** (26,865; 1.79%) — `jcc → 0x1bc → mov` (26,413×): another block boundary.
|
||||
- **0x1bf `call-end`** (8,752; 0.58%) — `call → 0x1bf → stmt-end` (8,600×): end-of-call statement.
|
||||
|
||||
### B. Statement metadata with an id — tentative no-op, confirm via harness (MED/LOW)
|
||||
Same scaffolding family but carrying an immediate; likely line numbers / statement descriptors.
|
||||
Treated as no-ops in v1 but flagged `method=harness` because a wrong no-op here could drop state.
|
||||
|
||||
- **0x21b `line-id?`** (27,016; 1.80%) — `mov → 0x21b <imm> → stmt-end`. Arg is 0/1/small. Sits
|
||||
near `savemesskip`/`loadmesskip` dev-comments (MENU/BUNKI) — **verify it isn't a message-control
|
||||
toggle** before trusting the no-op.
|
||||
- **0x1d2 `stmt-desc?`** (17,323; 1.15%) — always immediately after `stmt-begin 0x1f4`; 2 imm args
|
||||
(e.g. `0x2 0x5`) then a `mov`. Reads like a per-statement descriptor.
|
||||
- **0x258 `decl?`** (7,224; 0.48%) — runs in a chain right after script-entry `0x259`, enumerating
|
||||
ids (`0x4 0x1`, `0x5 0x1`, `0x6 0x1`, …). Prologue declaration/registration table.
|
||||
|
||||
### C. Structural (HIGH)
|
||||
- **0x71 `label-def`** (26,445; 1.76%) — its corpus count **exactly equals the T1 table size**, so
|
||||
this is the instruction T1 (tag 0x71) indexes: a label/anchor definition. Arg is a small id.
|
||||
Preceded by `end-text-line`/`call`. VM v1 = no-op; **revisit if menu/callback dispatch resolves
|
||||
entries by this id** (T1 may be a named-entry-point table).
|
||||
|
||||
### D. Computational — exact behaviour via Unicorn/inference (MED/LOW)
|
||||
- **0x215 `count?`** (5,675; 0.38%) — writes its first operand (a global) which is then tested
|
||||
`gre …, 0` / `lt` and branched on. Signature (global,global). Looks like a count / search-returns
|
||||
-index helper. Micro-exec candidate.
|
||||
- **0x1a2 `resolve-handle?`** (17,585; 1.17%) — takes a `local-ptr` just filled by `lookup-array`,
|
||||
then `create-texture` follows. Resolves a looked-up resource/handle. Borderline effectful → Frida.
|
||||
|
||||
### E. Effectful — draw / UI / audio / ADV; need Frida to confirm (see shortlist)
|
||||
All in the 0x420–0x421 graphics family or 0x41D audio family, clustered with named draw/audio ops.
|
||||
Not semantically no-ops; VM must implement (or knowingly stub) them.
|
||||
|
||||
- **0x7a `text-param?`** (17,324; 1.15%) — `sub` computes a value, then `0x7a <imm> <computed> <imm>`,
|
||||
then `show-text`. A text speed / auto-wait / window parameter for the following line. (The review
|
||||
guessed "computational"; context says ADV-text-effectful.)
|
||||
- **0x202 `draw-blit?`** (9,223) / **0x203 `draw?`** (9,149) — 4–5 coord/size args, preceded by
|
||||
coordinate arithmetic, chained with `draw-texture`/`u00420950`. Blit/positioned-draw.
|
||||
- **0x1f7 `ui-elem?`** (7,735) / **0x1fa `ui-clear?`** (6,126) — paired (`0x1f7 → 0x1fa` 5,014×);
|
||||
create/begin a UI element then show/hide/clear it by id. From MENU: `0x1f7 0x11170 0xc80`, `0x1fa 0x49/0x4a/0x4b`.
|
||||
- **0x217 / 0x218 / 0x21a `gfx-geom?`** (~5,000 each) — a chain each taking 4 global-ints
|
||||
(`0x21a → 0x218`, `0x217 → 0x1ff`); rect/coordinate/geometry transforms in the draw library.
|
||||
- **0x1ff `draw?`** (3,926) — 4 args then `call`; follows `0x217`.
|
||||
- **0xb6 `snd-ctrl?`** (4,805) — 0x41D family, self-chains, near `play-sound-effect`/`0xb5`; sound
|
||||
channel / volume / stop control.
|
||||
|
||||
### F. Input / UI hotspot branch — op 0x90 & 0x97 (A0-slice deep-dive, 2026-07-06 s2, HIGH)
|
||||
Solved while investigating the A0 EMPTY scenes. Kelebek: `{0x90,"u0041BEB0",0x7} // args 5,6,7 =
|
||||
code locations` (they left it "ukn"). Corpus evidence (`scratchpad/analyze_0x90.py` over all scripts):
|
||||
|
||||
- **op 0x90 = pointer/input hotspot hit-test branch.** Encoding `0x90 x y w h tgt_a tgt_b tgt_c`
|
||||
(argc 7; targets `0xffffffff` = unused). Branches to one of 3 targets per interaction outcome and
|
||||
**falls through to pc+1 when nothing matches** — design-confirmed: a 0x90 is 15 dwords, and the
|
||||
instruction after the last one lands exactly on the fall-through statement (`0xd0+15 = 0xdf`).
|
||||
- **Two forms, both only in one shared subroutine:**
|
||||
- *Mode A* (1505 = 5×301): all-immediate, `w=h=20`. The five on-screen ADV buttons — identical in
|
||||
all 301 scripts at `(x,y) = (684|706|728|750|772, 572)`, each setting one of `G[0x6c9..0x6cd]` to
|
||||
1 (target a) / 0 (target b) / 0+run-action (target c). Reads as hover-enter / hover-leave / click.
|
||||
- *Mode B* (903 = 3×301): local-int operands, `w=h=1`, only target c real — a keyed/degenerate
|
||||
2-way input test (fall through, or jump to c). Operands are `0-1000`-based sentinels (key codes?).
|
||||
- **Distribution is unanimous: every one of the 301 ADV scripts has EXACTLY 8 sites (5 A + 3 B).**
|
||||
Zero scene-specific use — it is boilerplate ADV chrome (the system button row + input polling), not
|
||||
gameplay branching.
|
||||
- **VM handling:** headless (no cursor/input) ⇒ fall through ⇒ **our stub is already correct**; proven
|
||||
safe by all 279 CLEAN dialogue scenes (which contain these same 8 sites). So 0x90 is NOT a headless
|
||||
correctness hole. It must be modelled for real in **A2** (Godot input backend) as a live hotspot test;
|
||||
confirm the exact target→interaction-state mapping via input capture / Frida then.
|
||||
- **op 0x97** (`u0041C150`, argc 5: `v1 v2 1 1 <action-id>`, NO code targets) interleaves with 0x90 in
|
||||
the same subroutine → companion *register-hotspot / set-widget-action* call (trailing imm = action id
|
||||
0x0/0x7/0x8). Not control flow; part of the `0x90/0x91/0x92/0x95/0x97` widget cluster.
|
||||
|
||||
**Consequence for A0/A1:** the 12 EMPTY scenes are gated by **state + interactive input-wait loops**
|
||||
(this chrome polls hotspots until a click), NOT by unmodelled 0x90 semantics. A1 can keep 0x90 as
|
||||
fall-through with confidence; unlocking the EMPTY family is a Phase-A2/B task (seed state + input).
|
||||
|
||||
## Coverage after classification
|
||||
See `build/opcode-coverage.md`. Roughly: named 72.6% + no-op scaffolding & label-def (A+C, high
|
||||
conf) ≈ 14.4% + tentative-no-op metadata (B) ≈ 3.4% → **~90% handleable without live tools**;
|
||||
the effectful/computational remainder (D+E) ≈ 6–7% is classified but needs Frida/Unicorn for
|
||||
exact behaviour.
|
||||
|
||||
## Frida / Unicorn shortlist (feeds Phase 3.1/3.2)
|
||||
- **Frida (effectful):** 0x7a (adv text param), 0x202/0x203/0x1f7/0x1fa/0x217/0x218/0x21a/0x1ff
|
||||
(draw/UI), 0xb6 (audio), 0x1a2 (resource resolve). Also **Frida-confirm already-named effectful
|
||||
ops the VM will rely on**: `play-voice 0xc4`, `play-bgm 0xbf`, `play-sound-effect 0xb4`,
|
||||
`draw-texture 0x1fb`, `draw-string 0x204`, `create-texture 0x1f8`, `set-texture 0x1f9`.
|
||||
- **Unicorn (computational):** 0x215 (count/search).
|
||||
- **Harness (no live tools):** 0x21b / 0x1d2 / 0x258 — confirm no-op assumption by diffing the
|
||||
VM's `show-text` sequence per scene against `build/text/dialogue.jsonl`.
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user