0ffb9620a3
- rm vm-map/opcodes-himegari.json, vm-map/himegari-opcode-notes.md (folded into opcodes.toml; in history) - PROJECT-STRUCTURE.md: opcodes.toml single-source-of-truth + generated artifacts + convention - repoint dead references in vm-mapping-plan/sys4-format-notes/himegari-port-reference/phase-a-slice-plan Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
159 lines
8.2 KiB
Markdown
159 lines
8.2 KiB
Markdown
# SYS4 Script Format — Reverse-Engineering Notes (hex-first)
|
||
|
||
Derived purely from byte-pattern analysis of the 481 DATA1 `.BIN` scripts, before
|
||
any Ghidra work on `AGE.EXE`. Confidence levels flagged per finding. Probe scripts
|
||
live in `tools/probe_*.py`.
|
||
|
||
> **UPDATE 2026-07-05 — opcode set solved via Kelebek1's table.** Everything below
|
||
> under "Instruction stream — PARTIAL" is now resolved: code = instructions of
|
||
> `<opcode:u32> + argc*(<argtype:u32><value:u32>)`, length `1+2*argc`; inline strings
|
||
> live after code inside `[0,F8)`, so stop decoding at the first type-2/`0x64` arg
|
||
> offset. 476/476 scripts decode clean (0 unknown opcodes). Header fields F0–F5 are
|
||
> **local-variable counts** (F0=local_integer_1, F1=local_floats, F2=local_strings_1,
|
||
> F3=local_integer_2, F4=unknown, F5=local_strings_2). See `vm-mapping-plan.md` and
|
||
> `vm-map/opcodes.toml`. The tag values below (0x71/0x03/0x8F etc.) are the
|
||
> *opcodes at table targets*: 0x8F=`call`, 0x03=`call-script`, 0x71=`u0041A7B0`.
|
||
|
||
## Header — CONFIRMED
|
||
|
||
Fixed 60-byte (0x3C) header: 8-byte magic + thirteen little-endian u32 fields.
|
||
Verified across all 481 files.
|
||
|
||
```
|
||
off field meaning evidence
|
||
0x00 magic "SYS4422 " (0x53 59 53 34 34 32 32 20) 481/481 identical
|
||
0x08 F0 scenario/script id or flags 98 distinct; 0x5 dominant (252×)
|
||
0x0C F1 = 1 always 481/481 == 1
|
||
0x10 F2 = 1 (457×) or 2 (20×); 4 outliers format/feature flag
|
||
0x14 F3 0x06 (268×),0x04,0x01,0x05... 15 distinct — minor version?
|
||
0x18 F4 = 1 always 481/481 == 1
|
||
0x1C F5 = 2 (319×) or 1 (144×); rarely 3/5 flag
|
||
0x20 F6 = 0x1C always 481/481 == 0x1C (header-size marker)
|
||
0x24 F7 table-1 entry count see below
|
||
0x28 F8 table-1 offset == code-section length ordering F8<=F10<=F12<=EOF
|
||
0x2C F9 table-2 entry count
|
||
0x30 F10 table-2 offset
|
||
0x34 F11 table-3 entry count
|
||
0x38 F12 table-3 offset
|
||
0x3C body dword stream (code + 3 tables + strings)
|
||
```
|
||
|
||
**All offsets/counts are in DWORDS (×4 bytes), relative to body start (0x3C).**
|
||
The `0x1C` in F6 is the only *byte* count — it's the offset from F6's own position
|
||
(0x20) back-referenced, i.e. a self-describing "28 bytes of descriptor follow" marker
|
||
consistent with the SYS4/SYS5 family.
|
||
|
||
### Section layout (CONFIRMED — 0 ordering violations, 481/481)
|
||
|
||
```
|
||
body[0 .. F8) CODE bytecode instruction stream
|
||
body[F8 .. F10) TABLE-1 (F7 entries, 1 dword each) -> targets of type 0x71
|
||
body[F10 .. F12) TABLE-2 (F9 entries, 1 dword each) -> targets of type 0x03
|
||
body[F12 .. EOF) TABLE-3 (F11 entries, 1 dword each) -> targets of type 0x8F
|
||
```
|
||
|
||
Every table entry is exactly **1 dword** — a pointer (dword index into body).
|
||
Solved algebraically across the whole corpus: `(F10-F8)/F7 == (F12-F10)/F9 ==
|
||
(EOF-F12)/F11 == 1` with zero non-integer results.
|
||
|
||
### The three tables are typed pointer indexes (CONFIRMED)
|
||
|
||
Each table points at body locations, and the dword *at* every target is a constant
|
||
tag identifying the pointed-to construct:
|
||
|
||
| Table | count/off | Target dword tag | Hits | Meaning (inferred) |
|
||
|---|---|---|---|---|
|
||
| T1 | F7 / F8 | **0x71** | 26,445/26,445 | labels / call targets (operand at +2 is small: mostly 1) |
|
||
| T2 | F9 / F10 | **0x03** | 3,018/3,018 | data/variable entries (operand at +2 large, e.g. addresses) |
|
||
| T3 | F11 / F12| **0x8F** | 72,941/72,941 | instruction/line entries (largest table; operand at +2 huge) |
|
||
|
||
100% type purity — not a single target had a different tag. T3 is the big one
|
||
(~73k entries corpus-wide), consistent with it being a per-instruction or
|
||
per-source-line index (a debug/line table). T1 ≈ labels, T2 ≈ a smaller symbol set.
|
||
|
||
## Instruction stream — PARTIAL
|
||
|
||
The code section is a flat dword stream. Recurring "type/opcode" dwords observed:
|
||
`0x03, 0x55, 0x6E, 0x6F, 0x71, 0x72, 0x8F`. These read as **operand-type tags**
|
||
in a tagged-operand VM rather than raw opcodes, e.g. the repeating shape:
|
||
|
||
```
|
||
... <TAG> <value> ... tag 0x02 => string pointer (see below)
|
||
... 0x71 0x00 0x01 0x55 ... label marker + following instruction
|
||
```
|
||
|
||
- **First body dword is 0x259 (601) in 301/481 files** — likely a standard
|
||
"script entry" / prologue opcode. Second-most-common openers are small ints.
|
||
- `0x55` appears pervasively as an instruction lead — probably the most common
|
||
opcode (statement / expression separator).
|
||
|
||
*Full opcode semantics need the VM dispatch loop in `AGE.EXE` — that's the Ghidra
|
||
task. These tags give a head start on labeling the disassembly.*
|
||
|
||
## Strings — CONFIRMED
|
||
|
||
- Stored inline in the body as **byte-complement (XOR 0xFF) cp932 / Shift-JIS**,
|
||
packed 4 bytes per dword, NUL-terminated (a `\0` byte, i.e. `0xFF` after XOR,
|
||
ends the string), then padded to the next dword.
|
||
- **Referenced by a tagged operand: the dword `0x02` immediately followed by the
|
||
dword-offset of the string.** Confirmed directly by xref:
|
||
- `MENU.BIN`: `...2 30b...` @0x2A9 → offset 0x30B = `"ＭＳ 明朝"`; `...2 30e...` → `"loadmesskip menu"`
|
||
- `SC0030.BIN`: `...2 ef80...` → `"▼G0030 ２章マップ０２１クリア"`; `...2 ef89...` → `"「よし、素晴らしい成果だな」"`
|
||
- Decoder (validated — pulls clean Japanese dialogue):
|
||
```python
|
||
raw = bytes(b ^ 0xFF for b in body[off*4:]) # until a 0x00 appears
|
||
text = raw.split(b"\0")[0].decode("cp932")
|
||
```
|
||
- Scene scripts hold the full dialogue; e.g. `SC0030.BIN` decodes to readable
|
||
story text, choice-branch labels ("本来の分岐", "チェック用"), font names
|
||
("ＭＳ 明朝"), and engine directives ("loadmesskip advset", "ＡＤＶパート").
|
||
- Non-scene scripts (`MENU`, `ADDEXP`) contain only a handful of control strings —
|
||
consistent with the inventory's subsystem/data-table categorization.
|
||
|
||
## Patch-override caveat (re-confirmed)
|
||
|
||
52 loose `.BIN` in the game root shadow their DATA1 copies at runtime and differ
|
||
slightly in size. The disassembler should target the **root** copies where present.
|
||
The header format is identical (same magic/layout) so tooling is copy-agnostic.
|
||
|
||
## What's solid vs. what needs Ghidra
|
||
|
||
**Solid (byte-verified, build a loader now):**
|
||
- 60-byte header, all 13 fields, dword units, section boundaries
|
||
- 3 typed pointer tables (0x71 / 0x03 / 0x8F), 1 dword each, 100% pure
|
||
- String encoding (XOR-0xFF cp932) + reference mechanism (tag 0x02 + offset)
|
||
|
||
**Needs the VM (Ghidra on `AGE.EXE`):**
|
||
- Opcode dispatch — confirm tagged-operand model, enumerate opcodes
|
||
- Meaning of F0/F2/F3/F5 flag fields
|
||
- Exact operand grammar per instruction (how many dwords each opcode consumes)
|
||
- Semantics of T1/T2/T3 beyond "label/data/line" guesses
|
||
|
||
## Loader — DONE
|
||
|
||
`tools/sys4load.py` parses the header, splits the 4 sections, resolves the 3
|
||
tables, decodes inline strings, and emits an assembly-ish listing with strings
|
||
inlined at their `2 <off>` refs (opcodes not yet named — code is chunked by the
|
||
T3 line-index). Importable API (`load()` → `Sys4Script`) plus CLI:
|
||
|
||
```
|
||
sys4load.py <file.BIN> full listing
|
||
sys4load.py <file.BIN> --summary header + section sizes + table/string counts
|
||
sys4load.py <file.BIN> --strings decoded string pool
|
||
sys4load.py <file.BIN> --json machine-readable structure
|
||
sys4load.py <dir> --validate re-check invariants across a folder
|
||
```
|
||
|
||
`--validate` over all 481 DATA1 scripts: **481 parsed clean, 0 failures, 0 impure
|
||
table tags** — the format spec above is fully machine-verified. This listing is the
|
||
artifact to diff against Ghidra output once the VM dispatch loop is mapped.
|
||
|
||
### Observations surfaced by the listing (leads for the VM work)
|
||
- String-display sites look like `<opcode> 0x02 <str-off>` — e.g. opcodes `0x1A7`
|
||
and `0x1A5` immediately precede string refs in `MENU.BIN`. Candidate text/message ops.
|
||
- T3 entries are 3-dword records `[0x8F, 0x00, value]`; T1 labels are `[0x71, 0x00,
|
||
value]`. T3 acts as a per-statement line index (editor metadata) — note scene
|
||
scripts even carry editor annotation strings like `"LABEL"`, `"ループ開始"` (loop start).
|
||
- `0x55` is the most frequent code lead (likely statement/expr separator); `0x09`
|
||
recurs as an operand-type prefix (register/var reference?).
|