Files

gamer147 8e0f769a6e Implement base SYS4 asset store

2026-07-11 09:14:14 -04:00

19 KiB

Raw Blame History

SYS4 VM Mapping — Plan of Action

For the executing agent: This is a reverse-engineering playbook. Work it phase-by-phase; each phase ends with a concrete, checkable deliverable. Verify claims against bytes before recording them. Validated seed data lives in vm-map/.

Goal: Decode the SYS4 bytecode into named instructions so the game logic can be re-implemented in Godot.

✅ BREAKTHROUGH (2026-07-05): the opcode set is already solved

The prior "unpack AGE.EXE in Ghidra" critical path is no longer needed to disassemble scripts. Kelebek1's decompiler ships a complete AGE opcode table that decodes this game directly.

What was verified this session (see tools/validate_opcode_table.py, run it to reproduce):

Kelebek1/Eushully-Decompiler's age-shared.cpp contains an opcode table ({op_code, label, argument_count}) and a header parser that explicitly handles the SYS4 signature ("SYS4", header length 0x3C, cp932 XOR-0xFF strings) — this exact game's format.
The instruction model: code = a flat sequence of instructions; each instruction = <opcode:u32> followed by argument_count arguments, where every argument is a <type:u32><value:u32> pair. Instruction length in dwords = 1 + 2*argc. Inline strings sit after the code inside the [0,F8) region; stop decoding at the lowest string offset referenced (a type-2 arg, or op 0x64 arg 1).
Applying that table to Himegari's scripts: 476 of 476 parseable scripts decode 100% clean — 1,463,788 instructions, 0 unknown opcodes, and all 37,392 inline-string arguments resolve to valid decoded strings. (The 7 non-decoding .BIN are container-level non-scripts like SYS4AB/SYS4INI, different magic.)
Himegari uses 248 distinct opcodes; 52 have semantic names (in vm-map/opcodes.toml). The other 196 decode perfectly (known length) but have engine-internal names only (u004xxxx). Caveat (measured 2026-07-06): the named 52 are the dialogue/ADV core but cover only 72.6% of instruction volume, not "the entire core" — the unnamed 27.4% is concentrated in the highest-frequency opcodes and must be partly addressed before Phase 4. See Phase 3's coverage correction.

This resolves the header unknowns too. Kelebek's BinaryHeader struct maps my F0–F12 exactly: F0=local_integer_1, F1=local_floats, F2=local_strings_1, F3=local_integer_2, F4=unknown, F5=local_strings_2, F6=sub_header_length(0x1C), then the three (length, offset) table pairs. The "flag fields" were local-variable counts. Arg type codes: 0=immediate, 1=float, 2=string, 3=global-int, 4=global-float, 5=global-string, 6=global-ptr, 8=global-string-ptr, 9=local-int, A=local-float, B=local-string, C=local-ptr, D=local-float-ptr, E=local-string-ptr.

Consequence: Unpacking AGE.EXE (still packed — see appendix) drops from the blocker to an optional enrichment used only to name the 196 unnamed opcodes' fine semantics, and even that has a cheaper dynamic alternative.

Provenance / sources in vm-map/: kelebek1-age-shared.cpp (the opcode table), kelebek1-disassembler.cpp (the parser), opcodes.toml (validated table filtered to what this game uses), opcode-leads.json + small-script-listings.md (this session's static analysis, now confirmed).

Global constraints

Python: py -3.11 -X utf8 … always (Shift-JIS output needs utf8 mode on Windows).
Authoritative copies: 49 loose game-folder script .BIN files shadow extracted/DATA1/ copies at runtime; two engine BINs are root-only. Target the game-folder copy where both exist. sys4load.load() is copy-agnostic; paths.scripts() resolves the override.
Units: all script offsets/counts are DWORDS (×4 bytes), relative to body start 0x3C.
Instruction rule: len_dwords = 1 + 2*argc; args are (type,value); stop code decode at the first inline-string/array offset, not blindly at F8.
Record confidence per finding (confirmed-by-bytes / confirmed-by-runtime / hypothesis).

Phase 1 — Port the opcode table into `sys4load.py`, disassemble everything (✅ DONE 2026-07-05)

Deliverable: sys4load.py emits real named instructions; every script disassembles with zero unknown opcodes. Achieved: 481/481 DATA1 scripts decode fully clean; MENU.BIN and SC0030.BIN verified by hand.

1.1 — Embed the opcode table. Full Kelebek table (548 entries) transcribed to tools/age_opcodes.py (OPCODES, ARG_TYPES, CONTROL_FLOW, is_label_argument), generated from vm-map/kelebek1-age-shared.cpp.
1.2 — Replace the T3-chunking stub. sys4load.py now has decode_code() (the 1+2*argc walker with shrinking code_end) and a rewritten render_listing() that prints mnemonics, typed operands, inline strings, and label_xxxx: control-flow anchors.
1.3 — Validate. tools/sys4load.py ../../extracted/DATA1 --validate → 481/481 parsed clean, 481/481 opcode-decode clean. MENU.BIN: 148 instrs, set-font "ＭＳ明朝" + comment strings correct. SC0030.BIN: 11,951 instrs, show-text shows dialogue inline. (tools/validate_opcode_table.py still reproduces the standalone 476/476 over the merged root+DATA1 set.)
1.4 — Regression-guard: container --validate still reports 481 clean, 0 failures, 0 impure tags.
1.5 — Disassembler is the artifact. sys4load.py <file> prints the full listing; --json now includes decoded code (with --json + to_dict(with_code=True)), instruction counts, and decode-clean flag.

Note: 7 root .BIN are non-script engine indices with different magic (SYS4INI = S4IC422, SYS4AB = S4AB, etc.) — correctly rejected by the container parser, not scripts.

Phase 2 — Extract data tables + dialogue (✅ mostly DONE 2026-07-06)

Deliverable: game database as JSON + full translatable dialogue corpus. Structure spec: docs/PROJECT-STRUCTURE.md. Extractors: tools/extract_phase2.py, tools/extract_init.py.

2.0 — Project structure. Established docs/, build/{disasm,text,data,scripts-json}/, godot/; game install stays read-only in place. Also relaxed the loader magic check to the SYS4 family (SYS4424 patch scripts now parse — was silently skipping 5 scripts).
2.1 — Text corpora. tools/extract_phase2.py → 481/481 scripts: full disassembly (build/disasm/*.asm), per-script strings, build/text/dialogue.jsonl (30,057 show-text lines — the translation corpus), build/text/strings.jsonl (38,449 strings tagged by source opcode), build/manifest.json.
2.2 — *INIT data tables → JSON. tools/extract_init.py auto-detects table shape (name/numeric/footer) → SKINIT (129 skills), ITINIT (189 items), EBINIT (277 units) [name: name+desc+fields], CGINIT (379 CG entries) [numeric: index-keyed columns], MPINIT (1472 map records) [footer: 50-value arrays from the file footer]. Validated; see build/data/README.md. Column addresses are raw engine globals — naming them (attack/cost/…) needs the global-var map (Phase 3-adjacent).
2.3 — STINIT (74 stages) needs a bespoke parser. Heterogeneous per-stage copy-to-global param blocks + a variable number of clear-condition strings per stage — fits none of the three auto modes. Its clear-condition text is already in build/text/STINIT.strings.txt; only the per-stage numeric params await a custom extractor. Low priority (stages are also encoded in the SC-scene scripts).
2.4 — Partial global-var map BUILT + wired into the disassembler. tools/global_map.py → build/global-var-map.{json,md} (16,354/49,435 globals labelled: string tables, *INIT field arrays, 122 record tables w/ strides, current-entity index pointers). sys4load renders the labels inline (=rec[s30], =current-entity-index?). See docs/name-resolution.md.
2.5 — Grow the global-var map (future, incremental). Static first: fold in *MES writers; label 2D record tables by their reader scripts. Then Frida to name which stat each field is. Full detail: docs/name-resolution.md → "Future step — growing the map". Also deferred: call-script id→name resolution (engine-level — SCJUMP.BIN decode or Frida; see docs/name-resolution.md #1).

Phase 3 — Name the unnamed opcodes (top ~20 BEFORE Phase 4; the rest on demand)

⚠️ Coverage correction (measured 2026-07-06). The earlier framing — "52 named ops cover the entire core, name the other 196 lazily" — is overstated. Across the full corpus (1,503,166 instructions, all 481 scripts), named opcodes are only 72.6% of instructions; the 195 unnamed u004xxxx ops are 27.4% — and that 27% is front-loaded into the most common opcodes, not a deferrable long tail. The top unnamed ops by frequency: 0x1f4/0x1f5 (60,297 each — equal counts → a begin/end or push/pop pair, both zero-arg), 0x1d5 (34k), 0x1bc (27k), 0x71 (26,445 — exactly the corpus T1 label-table entry count, so it's the label-definition pseudo-op, nameable by structure for free), 0x1a2 (18k), 0x7a (17k, argc 3, follows arithmetic → computational), 0x1d2 (17k). A Godot VM hits these in the first few instructions of any script. So naming the top ~20 is a prerequisite for Phase 4, not a lazy follow-on. Only the genuine long tail (rare ops) is deferrable. Reproduce the measurement by iterating sys4load.load over the corpus and bucketing ins.opcode against age_opcodes.OPCODES (label starting u00/dev_ukn = unnamed).

⚠️ Named labels are from a different AGE title. The 52 semantic labels are transcribed from Kelebek's table for a later AGE game. The opcode number + argc are validated for Himegari (481/481 clean decode proves structure), but the semantics are not independently verified. The ADV/text core is empirically safe — the 30,057-line build/text/dialogue.jsonl is proof that show-text/end-text-line/the string mechanism are right, and arithmetic/control-flow labels are corroborated by operand-type and jump-target consistency. The exposure is the effectful named ops you can't see in text output (play-voice 0xc4, draw-texture 0x1fb, sound/UI/draw ops) — Frida-confirm those against Himegari before the VM relies on them; don't assume them.

Do this before Phase 4: name/classify the ~20 highest-frequency unnamed opcodes. Most fall to free inference (3.0); a few opaque effectful ones want a Frida session; computational ones suit Unicorn. Everything below still applies — it's the ordering that changes, not the toolkit. The genuine rare tail stays lazy (name on demand).

3.0 — Inference pass DONE (2026-07-06). Classified the top 21 unnamed opcodes → instruction coverage 72.62% (named) → 96.94% (classified); ~90.5% is VM-handleable by inference alone. Tooling: tools/opcode_context.py (evidence gatherer). Results: vm-map/opcodes.toml (per-op evidence + provenance), tools/age_opcodes_himegari.py (INFERRED dict consumed by the disassembler + future VM), build/opcode-coverage.md (tiers + Frida/Unicorn shortlist). sys4load now renders inferred names (verified: MENU's label-def 0x71 land exactly on its T1 targets). Key findings: 0x1f4/0x1f5 = stmt begin/end brackets, 0x1d5/0x1bc/0x1bf = block markers (all zero-arg no-ops); 0x71 = label-def (count == T1 size); 0x21b/0x1d2/0x258 = tentative-no-op statement metadata (harness-verify); 0x7a = ADV text param, 0x202/0x203/0x1f7/0x1fa/0x217/0x218/0x21a/0x1ff = draw/UI, 0xb6 = audio, 0x215 = count/search — the effectful/computational Frida/Unicorn shortlist. Reserve live tools for those; rare tail (3%) stays lazy.

3.1 — Frida: dynamic observation (primary tool for effectful opcodes)

Frida injects a JS engine into the running game and hooks functions live. It sidesteps the packer (memory is already decrypted by the time you attach), gives ground-truth behavior, and lets you correlate an opcode with its on-screen/audible effect — the only reliable way to name rendering/audio/input/save/UI handlers. Two stages:

3.1a — Locate the dispatch loop. Kelebek's u004xxxx addresses are from a different AGE title and will NOT match Himegari's AGE.EXE, so find Himegari's dispatch first. Best anchor: search process memory for a known script's opening opcode sequence (you have every script decoded), set a hardware read breakpoint / MemoryAccessMonitor guard page on its first opcode dword; when the VM fetches it, the instruction pointer is inside the dispatch fetch. Alternate anchors: breakpoint a winmm/DirectSound call and trigger play-voice (0xC4), then walk the stack back; or pattern-scan for the bounds-check + call [table + opcode*4]. Payoff: read the jump-table base → you get the handler address for all 548 opcodes in Himegari at once.
3.1b — Instrument + correlate. Interceptor.attach the dispatch (or a specific handler); log opcode + operand (type,value) pairs (read from the bytecode pointer — layout known) + effect. Three correlation techniques: API (hook a basket of D3D9/winmm/user32/file APIs; see which an unknown handler calls), behavioral (trigger one in-game action, diff the opcode trace vs. baseline to attribute ops to subsystems), memory (log which global/local var-bank slots — sized by header F0–F5 — the handler reads/writes).
Setup / gotchas: pip install frida-tools; attach to the already-running game (frida AGE.EXE) after the title screen rather than spawning — this skips the packer's startup anti-debug. 32-bit x86 target. Japanese locale required to run. Eushully's protector may detect Frida's injected thread; if it trips, quiet it (ScyllaHide-style hooks or frida-gadget).

3.2 — Unicorn: microexecution (complement for computational opcodes)

Unicorn is a bare CPU emulator (no OS). It is the better tool for the pure-computation handlers — arithmetic/bit/string/array helpers and especially the CALC* damage/stat formulas — where you want the exact operation, not a label. It is blind to effectful handlers: the instant one calls D3D9/winmm/file APIs it runs into unmapped code and stubbing tells you nothing (the effect is the meaning). Do not use it as a Frida replacement.

3.2a — Microexecute a handler. Map the handler's code + a synthetic VM state (variable bank + operand), run from entry to ret, read back what changed; sweep inputs to recover the formula deterministically, offline.
3.2b — Preferred combo: Frida-snapshot → Unicorn-replay. Use Frida (3.1a) to find handler addresses and dump the relevant memory (code + var banks + globals) at a known-good moment (e.g. mid-battle); load that snapshot into Unicorn and microexecute individual handlers with input sweeps. Gets Frida's context-setup for free + Unicorn's determinism. Caveat: microexecution only recovers behavior that's a pure function of the captured state — if a handler reads a global you didn't snapshot, results are wrong silently. Fine for pure ops; a rabbit hole for stateful ones (leave those to live Frida). Needs a decrypted image to feed (a dump, or bytes pulled via Frida) since AGE.EXE is packed.

3.3 — Cross-reference siblings (free, do alongside 3.0)

Kelebek's labels come from a later AGE title; marcussacana/EushullyEditor targets Kamidori (same SYS4 era). Diff their handler notes for the specific opcodes you need.

3.4 — Static unpack + Ghidra (last resort)

Only if the above stall. See appendix — dump the decrypted image, load in Ghidra, read the handler at its address. High effort; reserve for genuinely opaque ops that Frida/Unicorn can't pin down.

Phase 4 — Godot re-implementation

Deliverable: the AGE VM running Himegari scripts in Godot.

Prerequisite: don't start Phase 4 against a blank opcode set — the top ~20 unnamed ops (Phase 3 preamble) are hit in the first few instructions of any script. Do that thin naming slice first, or bring-up stalls immediately on 0x1f4/0x71/etc.

4.0 — Stand up the validation harness first (before writing VM opcodes). The strongest correctness oracle already exists in build/: for a given scene script, the VM's emitted show-text sequence must match that file's lines in build/text/dialogue.jsonl. Wire this as an automated diff (drive one SC#### script → collect show-text → compare to the 30k-line corpus filtered by file). This turns "is the VM right?" into a per-scene regression test and catches control-flow/branch bugs (wrong jcc → wrong dialogue order) early. Extend later to assert extracted-table reads (SKINIT/ITINIT/EBINIT JSON) once data-driven opcodes come online.
4.1 — Re-implement the VM (GDScript/C#): a dword-fetch loop, the core opcodes (arithmetic, comparisons, jmp/call/jcc, mov, string ops) plus the high-frequency unnamed ops named in Phase 3, the global/local variable banks (sized by the header's F0–F5 counts), and the ADV layer (show-text/end-text-line/wait-for-input/set-font/play-voice/draw-*). Note the named set is only ~73% of instruction volume — budget for the unnamed remainder. The bytecode-heavy design (damage calc, dungeon loop, battle flow are all scripts) makes re-implementation the right call over transpilation.
4.2 — Fill opcodes on demand from Phase 3's genuine long tail as scripts exercise them (the top ~20 are already done as a Phase 3/4 prerequisite).
4.3 — Deferred: save-file format (reverse SAVE.BIN only if the port must read existing saves).

Appendix — `AGE.EXE` is packed (relevant to Phase 3.4, and as the image source for 3.2)

Verified this session (tools/pack_check.py): 32-bit PE, code sections at max entropy (8.00), blank section names, IAT RVA 0, no plaintext anchors. SYS4AB.BIN (magic S4AB) is a dead end for static analysis — it decrypts to AGE.EXE byte-for-byte (2026-07-06): 0x2c-byte header ("S4AB" + version + 0x0010E000 size dword ×3 + an 8-byte key/hash field), then the payload is a trivial XOR-0xFF of the same packed AGE.EXE (bytes(x^0xFF for x in payload) == AGE.EXE, exact). So it is not a patched/unpacked VM — both on-disk engine images are the identical packed binary, and VA 0x421160 (any handler) is entropy-8.00 garbage in both. The real handler code exists only unpacked in the runtime heap (~30 MB r-x @ 0x62411000, nonstable base per run — see docs/global-memory-re.md). Static Unicorn/Ghidra therefore requires a runtime dump of that region, or hook it live.

Static analysis therefore requires a runtime dump first — you're dumping for analysis not redistribution, so don't chase OEP: launch to the title screen (Japanese locale required), then dump the decrypted image and load it in Ghidra. Tools: PE-sieve (CLI, agent-drivable: pe-sieve.exe /pid <PID> /imp 3) or x32dbg + Scylla + ScyllaHide (GUI, handles the anti-debug). Validate the dump by confirming SYS4422/.BIN/DATA1 now appear in plaintext. But prefer Frida dynamic hooking (Phase 3.1) — it avoids the unpack entirely.

19 KiB Raw Blame History Unescape Escape