OpenMaidEngine/docs/sys4-format-notes.md

SYS4 Script Format — Reverse-Engineering Notes (hex-first)

Derived purely from byte-pattern analysis of the 481 DATA1 .BIN scripts, before any Ghidra work on AGE.EXE. Confidence levels flagged per finding. Probe scripts live in tools/probe_*.py.

UPDATE 2026-07-05 — opcode set solved via Kelebek1's table. Everything below under "Instruction stream — PARTIAL" is now resolved: code = instructions of <opcode:u32> + argc*(<argtype:u32><value:u32>), length 1+2*argc; inline strings live after code inside [0,F8), so stop decoding at the first type-2/0x64 arg offset. 476/476 scripts decode clean (0 unknown opcodes). Header fields F0–F5 are local-variable counts (F0=local_integer_1, F1=local_floats, F2=local_strings_1, F3=local_integer_2, F4=unknown, F5=local_strings_2). See vm-mapping-plan.md and vm-map/opcodes.toml. The tag values below (0x71/0x03/0x8F etc.) are the opcodes at table targets: 0x8F=call, 0x03=call-script, 0x71=u0041A7B0.

Header — CONFIRMED

Fixed 60-byte (0x3C) header: 8-byte magic + thirteen little-endian u32 fields. Verified across all 481 files.

off   field  meaning                                    evidence
0x00  magic  "SYS4422 " (0x53 59 53 34 34 32 32 20)     481/481 identical
0x08  F0     scenario/script id or flags                98 distinct; 0x5 dominant (252×)
0x0C  F1     = 1 always                                 481/481 == 1
0x10  F2     = 1 (457×) or 2 (20×); 4 outliers          format/feature flag
0x14  F3     0x06 (268×),0x04,0x01,0x05...              15 distinct — minor version?
0x18  F4     = 1 always                                 481/481 == 1
0x1C  F5     = 2 (319×) or 1 (144×); rarely 3/5         flag
0x20  F6     = 0x1C always                              481/481 == 0x1C  (header-size marker)
0x24  F7     table-1 entry count                        see below
0x28  F8     table-1 offset  == code-section length     ordering F8<=F10<=F12<=EOF
0x2C  F9     table-2 entry count
0x30  F10    table-2 offset
0x34  F11    table-3 entry count
0x38  F12    table-3 offset
0x3C  body   dword stream (code + 3 tables + strings)

All offsets/counts are in DWORDS (×4 bytes), relative to body start (0x3C). The 0x1C in F6 is the only byte count — it's the offset from F6's own position (0x20) back-referenced, i.e. a self-describing "28 bytes of descriptor follow" marker consistent with the SYS4/SYS5 family.

Section layout (CONFIRMED — 0 ordering violations, 481/481)

body[0        .. F8)    CODE          bytecode instruction stream
body[F8       .. F10)   TABLE-1  (F7 entries, 1 dword each)  -> targets of type 0x71
body[F10      .. F12)   TABLE-2  (F9 entries, 1 dword each)  -> targets of type 0x03
body[F12      .. EOF)   TABLE-3  (F11 entries, 1 dword each) -> targets of type 0x8F

Every table entry is exactly 1 dword — a pointer (dword index into body). Solved algebraically across the whole corpus: (F10-F8)/F7 == (F12-F10)/F9 == (EOF-F12)/F11 == 1 with zero non-integer results.

The three tables are typed pointer indexes (CONFIRMED)

Each table points at body locations, and the dword at every target is a constant tag identifying the pointed-to construct:

Table	count/off	Target dword tag	Hits	Meaning (inferred)
T1	F7 / F8	0x71	26,445/26,445	labels / call targets (operand at +2 is small: mostly 1)
T2	F9 / F10	0x03	3,018/3,018	data/variable entries (operand at +2 large, e.g. addresses)
T3	F11 / F12	0x8F	72,941/72,941	instruction/line entries (largest table; operand at +2 huge)

100% type purity — not a single target had a different tag. T3 is the big one (~73k entries corpus-wide), consistent with it being a per-instruction or per-source-line index (a debug/line table). T1 ≈ labels, T2 ≈ a smaller symbol set.

Instruction stream — PARTIAL

The code section is a flat dword stream. Recurring "type/opcode" dwords observed: 0x03, 0x55, 0x6E, 0x6F, 0x71, 0x72, 0x8F. These read as operand-type tags in a tagged-operand VM rather than raw opcodes, e.g. the repeating shape:

... <TAG> <value> ...        tag 0x02 => string pointer (see below)
... 0x71 0x00 0x01 0x55 ...  label marker + following instruction

• First body dword is 0x259 (601) in 301/481 files — likely a standard "script entry" / prologue opcode. Second-most-common openers are small ints.
• 0x55 appears pervasively as an instruction lead — probably the most common opcode (statement / expression separator).

Full opcode semantics need the VM dispatch loop in AGE.EXE — that's the Ghidra task. These tags give a head start on labeling the disassembly.

Strings — CONFIRMED

• Stored inline in the body as byte-complement (XOR 0xFF) cp932 / Shift-JIS, packed 4 bytes per dword, NUL-terminated (a \0 byte, i.e. 0xFF after XOR, ends the string), then padded to the next dword.
• Referenced by a tagged operand: the dword 0x02 immediately followed by the dword-offset of the string. Confirmed directly by xref:
  • MENU.BIN: ...2 30b... @0x2A9 → offset 0x30B = "ＭＳ 明朝"; ...2 30e... → "loadmesskip menu"
  • SC0030.BIN: ...2 ef80... → "▼G0030　２章マップ０２１クリア"; ...2 ef89... → "「よし、素晴らしい成果だな」"
• Decoder (validated — pulls clean Japanese dialogue):

  raw = bytes(b ^ 0xFF for b in body[off*4:])   # until a 0x00 appears
  text = raw.split(b"\0")[0].decode("cp932")
  

• Scene scripts hold the full dialogue; e.g. SC0030.BIN decodes to readable story text, choice-branch labels ("本来の分岐", "チェック用"), font names ("ＭＳ 明朝"), and engine directives ("loadmesskip advset", "ＡＤＶパート").
• Non-scene scripts (MENU, ADDEXP) contain only a handful of control strings — consistent with the inventory's subsystem/data-table categorization.

Patch-override caveat (re-confirmed)

52 loose .BIN in the game root shadow their DATA1 copies at runtime and differ slightly in size. The disassembler should target the root copies where present. The header format is identical (same magic/layout) so tooling is copy-agnostic.

What's solid vs. what needs Ghidra

Solid (byte-verified, build a loader now):

• 60-byte header, all 13 fields, dword units, section boundaries
• 3 typed pointer tables (0x71 / 0x03 / 0x8F), 1 dword each, 100% pure
• String encoding (XOR-0xFF cp932) + reference mechanism (tag 0x02 + offset)

Needs the VM (Ghidra on AGE.EXE):

• Opcode dispatch — confirm tagged-operand model, enumerate opcodes
• Meaning of F0/F2/F3/F5 flag fields
• Exact operand grammar per instruction (how many dwords each opcode consumes)
• Semantics of T1/T2/T3 beyond "label/data/line" guesses

Loader — DONE

tools/sys4load.py parses the header, splits the 4 sections, resolves the 3 tables, decodes inline strings, and emits an assembly-ish listing with strings inlined at their 2 <off> refs (opcodes not yet named — code is chunked by the T3 line-index). Importable API (load() → Sys4Script) plus CLI:

sys4load.py <file.BIN>              full listing
sys4load.py <file.BIN> --summary    header + section sizes + table/string counts
sys4load.py <file.BIN> --strings    decoded string pool
sys4load.py <file.BIN> --json       machine-readable structure
sys4load.py <dir> --validate        re-check invariants across a folder

--validate over all 481 DATA1 scripts: 481 parsed clean, 0 failures, 0 impure table tags — the format spec above is fully machine-verified. This listing is the artifact to diff against Ghidra output once the VM dispatch loop is mapped.

Observations surfaced by the listing (leads for the VM work)

• String-display sites look like <opcode> 0x02 <str-off> — e.g. opcodes 0x1A7 and 0x1A5 immediately precede string refs in MENU.BIN. Candidate text/message ops.
• T3 entries are 3-dword records [0x8F, 0x00, value]; T1 labels are [0x71, 0x00, value]. T3 acts as a per-statement line index (editor metadata) — note scene scripts even carry editor annotation strings like "LABEL", "ループ開始" (loop start).
• 0x55 is the most frequent code lead (likely statement/expr separator); 0x09 recurs as an operand-type prefix (register/var reference?).