chore: initialize age-reimpl repo
Reverse-engineering + open reimplementation workspace for Eushully's AGE/SYS4 engine (first target: Himegari). The repo root is age-reimpl/; the original game install and the extracted ALF data are siblings outside the repo and are never tracked. build/ (derived corpora) is gitignored and regenerated by the tools. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
gamer147
158
docs/sys4-format-notes.md
Normal file
158
docs/sys4-format-notes.md
Normal file
@@ -0,0 +1,158 @@
|
||||
# SYS4 Script Format — Reverse-Engineering Notes (hex-first)
|
||||
|
||||
Derived purely from byte-pattern analysis of the 481 DATA1 `.BIN` scripts, before
|
||||
any Ghidra work on `AGE.EXE`. Confidence levels flagged per finding. Probe scripts
|
||||
live in `tools/probe_*.py`.
|
||||
|
||||
> **UPDATE 2026-07-05 — opcode set solved via Kelebek1's table.** Everything below
|
||||
> under "Instruction stream — PARTIAL" is now resolved: code = instructions of
|
||||
> `<opcode:u32> + argc*(<argtype:u32><value:u32>)`, length `1+2*argc`; inline strings
|
||||
> live after code inside `[0,F8)`, so stop decoding at the first type-2/`0x64` arg
|
||||
> offset. 476/476 scripts decode clean (0 unknown opcodes). Header fields F0–F5 are
|
||||
> **local-variable counts** (F0=local_integer_1, F1=local_floats, F2=local_strings_1,
|
||||
> F3=local_integer_2, F4=unknown, F5=local_strings_2). See `vm-mapping-plan.md` and
|
||||
> `vm-map/opcodes-himegari.json`. The tag values below (0x71/0x03/0x8F etc.) are the
|
||||
> *opcodes at table targets*: 0x8F=`call`, 0x03=`call-script`, 0x71=`u0041A7B0`.
|
||||
|
||||
## Header — CONFIRMED
|
||||
|
||||
Fixed 60-byte (0x3C) header: 8-byte magic + thirteen little-endian u32 fields.
|
||||
Verified across all 481 files.
|
||||
|
||||
```
|
||||
off field meaning evidence
|
||||
0x00 magic "SYS4422 " (0x53 59 53 34 34 32 32 20) 481/481 identical
|
||||
0x08 F0 scenario/script id or flags 98 distinct; 0x5 dominant (252×)
|
||||
0x0C F1 = 1 always 481/481 == 1
|
||||
0x10 F2 = 1 (457×) or 2 (20×); 4 outliers format/feature flag
|
||||
0x14 F3 0x06 (268×),0x04,0x01,0x05... 15 distinct — minor version?
|
||||
0x18 F4 = 1 always 481/481 == 1
|
||||
0x1C F5 = 2 (319×) or 1 (144×); rarely 3/5 flag
|
||||
0x20 F6 = 0x1C always 481/481 == 0x1C (header-size marker)
|
||||
0x24 F7 table-1 entry count see below
|
||||
0x28 F8 table-1 offset == code-section length ordering F8<=F10<=F12<=EOF
|
||||
0x2C F9 table-2 entry count
|
||||
0x30 F10 table-2 offset
|
||||
0x34 F11 table-3 entry count
|
||||
0x38 F12 table-3 offset
|
||||
0x3C body dword stream (code + 3 tables + strings)
|
||||
```
|
||||
|
||||
**All offsets/counts are in DWORDS (×4 bytes), relative to body start (0x3C).**
|
||||
The `0x1C` in F6 is the only *byte* count — it's the offset from F6's own position
|
||||
(0x20) back-referenced, i.e. a self-describing "28 bytes of descriptor follow" marker
|
||||
consistent with the SYS4/SYS5 family.
|
||||
|
||||
### Section layout (CONFIRMED — 0 ordering violations, 481/481)
|
||||
|
||||
```
|
||||
body[0 .. F8) CODE bytecode instruction stream
|
||||
body[F8 .. F10) TABLE-1 (F7 entries, 1 dword each) -> targets of type 0x71
|
||||
body[F10 .. F12) TABLE-2 (F9 entries, 1 dword each) -> targets of type 0x03
|
||||
body[F12 .. EOF) TABLE-3 (F11 entries, 1 dword each) -> targets of type 0x8F
|
||||
```
|
||||
|
||||
Every table entry is exactly **1 dword** — a pointer (dword index into body).
|
||||
Solved algebraically across the whole corpus: `(F10-F8)/F7 == (F12-F10)/F9 ==
|
||||
(EOF-F12)/F11 == 1` with zero non-integer results.
|
||||
|
||||
### The three tables are typed pointer indexes (CONFIRMED)
|
||||
|
||||
Each table points at body locations, and the dword *at* every target is a constant
|
||||
tag identifying the pointed-to construct:
|
||||
|
||||
| Table | count/off | Target dword tag | Hits | Meaning (inferred) |
|
||||
|---|---|---|---|---|
|
||||
| T1 | F7 / F8 | **0x71** | 26,445/26,445 | labels / call targets (operand at +2 is small: mostly 1) |
|
||||
| T2 | F9 / F10 | **0x03** | 3,018/3,018 | data/variable entries (operand at +2 large, e.g. addresses) |
|
||||
| T3 | F11 / F12| **0x8F** | 72,941/72,941 | instruction/line entries (largest table; operand at +2 huge) |
|
||||
|
||||
100% type purity — not a single target had a different tag. T3 is the big one
|
||||
(~73k entries corpus-wide), consistent with it being a per-instruction or
|
||||
per-source-line index (a debug/line table). T1 ≈ labels, T2 ≈ a smaller symbol set.
|
||||
|
||||
## Instruction stream — PARTIAL
|
||||
|
||||
The code section is a flat dword stream. Recurring "type/opcode" dwords observed:
|
||||
`0x03, 0x55, 0x6E, 0x6F, 0x71, 0x72, 0x8F`. These read as **operand-type tags**
|
||||
in a tagged-operand VM rather than raw opcodes, e.g. the repeating shape:
|
||||
|
||||
```
|
||||
... <TAG> <value> ... tag 0x02 => string pointer (see below)
|
||||
... 0x71 0x00 0x01 0x55 ... label marker + following instruction
|
||||
```
|
||||
|
||||
- **First body dword is 0x259 (601) in 301/481 files** — likely a standard
|
||||
"script entry" / prologue opcode. Second-most-common openers are small ints.
|
||||
- `0x55` appears pervasively as an instruction lead — probably the most common
|
||||
opcode (statement / expression separator).
|
||||
|
||||
*Full opcode semantics need the VM dispatch loop in `AGE.EXE` — that's the Ghidra
|
||||
task. These tags give a head start on labeling the disassembly.*
|
||||
|
||||
## Strings — CONFIRMED
|
||||
|
||||
- Stored inline in the body as **byte-complement (XOR 0xFF) cp932 / Shift-JIS**,
|
||||
packed 4 bytes per dword, NUL-terminated (a `\0` byte, i.e. `0xFF` after XOR,
|
||||
ends the string), then padded to the next dword.
|
||||
- **Referenced by a tagged operand: the dword `0x02` immediately followed by the
|
||||
dword-offset of the string.** Confirmed directly by xref:
|
||||
- `MENU.BIN`: `...2 30b...` @0x2A9 → offset 0x30B = `"ＭＳ 明朝"`; `...2 30e...` → `"loadmesskip menu"`
|
||||
- `SC0030.BIN`: `...2 ef80...` → `"▼G0030 ２章マップ０２１クリア"`; `...2 ef89...` → `"「よし、素晴らしい成果だな」"`
|
||||
- Decoder (validated — pulls clean Japanese dialogue):
|
||||
```python
|
||||
raw = bytes(b ^ 0xFF for b in body[off*4:]) # until a 0x00 appears
|
||||
text = raw.split(b"\0")[0].decode("cp932")
|
||||
```
|
||||
- Scene scripts hold the full dialogue; e.g. `SC0030.BIN` decodes to readable
|
||||
story text, choice-branch labels ("本来の分岐", "チェック用"), font names
|
||||
("ＭＳ 明朝"), and engine directives ("loadmesskip advset", "ＡＤＶパート").
|
||||
- Non-scene scripts (`MENU`, `ADDEXP`) contain only a handful of control strings —
|
||||
consistent with the inventory's subsystem/data-table categorization.
|
||||
|
||||
## Patch-override caveat (re-confirmed)
|
||||
|
||||
52 loose `.BIN` in the game root shadow their DATA1 copies at runtime and differ
|
||||
slightly in size. The disassembler should target the **root** copies where present.
|
||||
The header format is identical (same magic/layout) so tooling is copy-agnostic.
|
||||
|
||||
## What's solid vs. what needs Ghidra
|
||||
|
||||
**Solid (byte-verified, build a loader now):**
|
||||
- 60-byte header, all 13 fields, dword units, section boundaries
|
||||
- 3 typed pointer tables (0x71 / 0x03 / 0x8F), 1 dword each, 100% pure
|
||||
- String encoding (XOR-0xFF cp932) + reference mechanism (tag 0x02 + offset)
|
||||
|
||||
**Needs the VM (Ghidra on `AGE.EXE`):**
|
||||
- Opcode dispatch — confirm tagged-operand model, enumerate opcodes
|
||||
- Meaning of F0/F2/F3/F5 flag fields
|
||||
- Exact operand grammar per instruction (how many dwords each opcode consumes)
|
||||
- Semantics of T1/T2/T3 beyond "label/data/line" guesses
|
||||
|
||||
## Loader — DONE
|
||||
|
||||
`tools/sys4load.py` parses the header, splits the 4 sections, resolves the 3
|
||||
tables, decodes inline strings, and emits an assembly-ish listing with strings
|
||||
inlined at their `2 <off>` refs (opcodes not yet named — code is chunked by the
|
||||
T3 line-index). Importable API (`load()` → `Sys4Script`) plus CLI:
|
||||
|
||||
```
|
||||
sys4load.py <file.BIN> full listing
|
||||
sys4load.py <file.BIN> --summary header + section sizes + table/string counts
|
||||
sys4load.py <file.BIN> --strings decoded string pool
|
||||
sys4load.py <file.BIN> --json machine-readable structure
|
||||
sys4load.py <dir> --validate re-check invariants across a folder
|
||||
```
|
||||
|
||||
`--validate` over all 481 DATA1 scripts: **481 parsed clean, 0 failures, 0 impure
|
||||
table tags** — the format spec above is fully machine-verified. This listing is the
|
||||
artifact to diff against Ghidra output once the VM dispatch loop is mapped.
|
||||
|
||||
### Observations surfaced by the listing (leads for the VM work)
|
||||
- String-display sites look like `<opcode> 0x02 <str-off>` — e.g. opcodes `0x1A7`
|
||||
and `0x1A5` immediately precede string refs in `MENU.BIN`. Candidate text/message ops.
|
||||
- T3 entries are 3-dword records `[0x8F, 0x00, value]`; T1 labels are `[0x71, 0x00,
|
||||
value]`. T3 acts as a per-statement line index (editor metadata) — note scene
|
||||
scripts even carry editor annotation strings like `"LABEL"`, `"ループ開始"` (loop start).
|
||||
- `0x55` is the most frequent code lead (likely statement/expr separator); `0x09`
|
||||
recurs as an operand-type prefix (register/var reference?).
|
||||
Reference in New Issue
Block a user