diff --git a/docs/global-reference.md b/docs/global-reference.md index 3fcfb78..9347ba3 100644 --- a/docs/global-reference.md +++ b/docs/global-reference.md @@ -1,7 +1,7 @@ # Global Variable Reference (generated) -5026 globals (155 curated, 4871 auto shape-inferred). Source of truth: `vm-map/globals.toml`. +5030 globals (167 curated, 4863 auto shape-inferred). Source of truth: `vm-map/globals.toml`. ## choice-output @@ -26,6 +26,7 @@ | address | name | conf | source | usage | |---|---|---|---|---| +| `0x6d3` | story_event_flags | high | investigation | Shared one-based story/event flag bank. Progression, recruitment, item, and stage logic read or write individual cells; STINIT object prerequisite ids are decremented by one before SETOBJ tests this array. | | `0x2e49` | character_voice_suppressed | high | investigation | Base of the per-character voice enable/suppress settings. INITCONFIG zero-fills all 13 cells and registers each with the shared profile service; LOADCONFIG restores them. CONFIG indexes the table to preview a character voice and write 0/1. ROOM reads cell 0 before assigning its selected greeter's greeting/farewell voice ids, so the port's former scalar interpretation of zero-int-range (writing 13 into the base cell) suppressed those voices on every natural boot. This names the script-visible setting array without choosing a persistence backend for op 0x1a2/0x1a3. | | `0x65ce` | skill_acquired_flags | high | investigation | Persistent acquired-skill flags. ADDSKILL sets the selected skill after resolving the unit's equipped-skill slots; FORT checks the flag before granting a skill; CHMENU combines it with skill_change_catalog_eligible to build the available skill-change catalog. | | `0x673c` | party_slot_flags | high | investigation | Per-party-slot state flags for slots 0..99. UNITECH creates the initial unit by setting slot 2 to 0x13; CALCARR counts slots whose flags intersect 0x6, and CHMENU includes slots with bit 1 set. Exact meanings of the remaining bits are not yet classified. | @@ -102,8 +103,18 @@ | `0xaaa1e` | skill_battle_animation_id | high | investigation | Populated for 101 combat skills. BTL and CALCDMG place this value in the battle-animation selector before calling BTANINIT; most skills reuse their own id, while related skills deliberately share an animation and passive reactions use ids 801..808. | | `0xaab4a` | skill_handler_script_id | high | investigation | SKINIT field for all 131 skills. CHMENU and INFOIT look it up and pass it directly to call-script; packed id 0x31ca resolves to SKMES.BIN, the shared per-skill text/behavior dispatcher. | | `0xe7302` | stage_bgm_id | high | investigation | STINIT's per-stage scalar loaded for all 74 records. FIELD passes the value directly to play-bgm when starting the stage. | +| `0xe7303` | stage_target_clear_turns | high | investigation | STINIT's target/par turn count. STAGECLEAR divides elapsed turns by this value to derive a performance multiplier; when replaying an already-cleared ordinary stage, FIELD also uses it as the forced-retreat turn limit. | +| `0xe7304` | stage_clear_performance_bonus | med | investigation | Base stage-clear reward increment. STAGECLEAR multiplies it by the turn-performance percentage derived from stage_target_clear_turns, divides by 100, and adds the result to the capped persistent reward counter at 0x6719. The counter's player-facing resource name remains unresolved. | +| `0xe730b` | stage_replay_rules_disabled | high | investigation | FIELD's already-cleared-stage override gate. For a cleared stage with value 0, FIELD replaces the mission with an all-party retreat objective and a stage_target_clear_turns forced-retreat limit; value 1 suppresses that replay conversion and related revisit handling. | | `0xe730c` | stage_turn_limit | high | investigation | STINIT's per-stage turn limit. DRAWCHP presents the value in the stage information, while FIELD compares the current turn against it when checking stage completion. | | `0xe730d` | stage_turn_limit_outcome | high | investigation | STINIT mode paired with stage_turn_limit. Stage 1 stores 0 and describes 50-turn expiry as defeat; stage 2 stores 1 and explicitly describes 15-turn expiry as a forced-retreat clear. | +| `0xe7311` | stage_map_texture_asset_overrides | high | investigation | Twenty current-stage map texture slots. FIELD loads positive resource ids into tiled surface slots 0x52+index, skips zero, and substitutes the shared per-index fallback for -1; DRAWMAP selects and draws those surfaces through the terrain-to-texture-slot map. | +| `0xe7325` | stage_object_tile_x | high | investigation | X coordinate for each current-stage object slot. DRAWOBJ converts it to centered map-space pixels; FIELD and CALCOCC use it with stage_object_tile_y for object interaction and occupancy. | +| `0xe7357` | stage_object_tile_y | high | investigation | Y coordinate for each current-stage object slot. DRAWOBJ converts it to centered map-space pixels; FIELD and CALCOCC use it with stage_object_tile_x for object interaction and occupancy. | +| `0xe7389` | stage_object_type_id | high | investigation | Object-definition id for each current-stage object slot. SETOBJ decides whether the slot exists, while DRAWOBJ and FIELD use the id to select shared object graphics, dimensions, animation, collision, and behavior metadata. | +| `0xe7483` | stage_object_difficulty_mask | high | investigation | Per-object difficulty inclusion mask. SETOBJ uses check-bit with difficulty_index and rejects the object slot when the selected difficulty bit is absent. | +| `0xe74b5` | stage_object_required_story_flags | high | investigation | Seven positive prerequisites per stage object. SETOBJ subtracts one from every populated id and suppresses the object unless the corresponding story_event_flags cell equals 1. Columns: 0=required_flag_1, 1=required_flag_2, 2=required_flag_3, 3=required_flag_4, 4=required_flag_5, 5=required_flag_6, 6=required_flag_7. | +| `0xe7613` | stage_object_forbidden_story_flags | high | investigation | Five negative prerequisites per stage object. SETOBJ subtracts one from every populated id and suppresses the object when the corresponding story_event_flags cell equals 1. Columns: 0=forbidden_flag_1, 1=forbidden_flag_2, 2=forbidden_flag_3, 3=forbidden_flag_4, 4=forbidden_flag_5. | | `0x5` | — | low | auto-shape | array | | `0xd2` | — | low | auto-shape | array | | `0xd7` | — | low | auto-shape | array | @@ -116,7 +127,6 @@ | `0x4f1` | — | low | auto-shape | array | | `0x5c3` | — | low | auto-shape | array | | `0x69e` | — | low | auto-shape | array | -| `0x6d3` | — | low | auto-shape | array | | `0x6fe` | — | low | auto-shape | array | | `0x712` | — | low | auto-shape | array | | `0x7db` | — | low | auto-shape | array | @@ -250,15 +260,10 @@ | `0xe7172` | — | low | auto-shape | array | | `0xe7305` | — | low | auto-shape | array | | `0xe730e` | — | low | auto-shape | array | -| `0xe7311` | — | low | auto-shape | array | -| `0xe7325` | — | low | auto-shape | array | -| `0xe7357` | — | low | auto-shape | array | -| `0xe7389` | — | low | auto-shape | array | | `0xe73bb` | — | low | auto-shape | array | | `0xe73ed` | — | low | auto-shape | array | | `0xe741f` | — | low | auto-shape | array | | `0xe7451` | — | low | auto-shape | array | -| `0xe7483` | — | low | auto-shape | array | | `0xe770d` | — | low | auto-shape | array | | `0xe773f` | — | low | auto-shape | array | | `0xe775d` | — | low | auto-shape | array | @@ -356,6 +361,7 @@ | address | name | conf | source | usage | |---|---|---|---|---| +| `0x32f1` | difficulty_index | high | investigation | GAMESTART's three-way difficulty selection. SETEN uses the zero-based index to select difficulty stat adjustments; FIELD indexes three-column stage-enemy arrays with it, and SETOBJ tests it against each object's difficulty mask. | | `0x6718` | selected_party_slot | high | investigation | Current/selected slot in the 100-entry party-unit arrays. UNITECH chooses a free slot here before populating it; CHMENU replaces it with the selected sorted roster slot, then uses it to index party_slot_flags, party_slot_character_id, and companion per-slot tables. A natural New Game enters SC0000 with slot 2 selected. | | `0x8c877` | current_item_id | high | investigation | Shared item-id argument/selection slot. Item menus and gameplay scripts write a chosen item id, use it to index ITINIT arrays, and dispatch through item_handler_script_id; ITMES compares it against all 287 item ids to select the matching player-facing title and description. | | `0xa6e59` | current_skill_id | high | investigation | Shared skill-id argument/selection slot. Skill menus and combat scripts write the chosen skill id and use it to index SKINIT arrays; SKMES compares it against all 131 skill ids to select the matching player-facing title and description. | @@ -5013,8 +5019,6 @@ | `0xe6afe` | — | med | auto-shape | record-table[stride 10] | | `0xe6fe2` | — | med | auto-shape | record-table[stride 3] | | `0xe71d6` | — | med | auto-shape | record-table[stride 3] | -| `0xe74b5` | — | med | auto-shape | record-table[stride 7] | -| `0xe7613` | — | med | auto-shape | record-table[stride 5] | | `0xe7889` | — | med | auto-shape | record-table[stride 3] | | `0xe78e3` | — | med | auto-shape | record-table[stride 3] | | `0xe793d` | — | med | auto-shape | record-table[stride 7] | diff --git a/docs/name-resolution.md b/docs/name-resolution.md index fa1aa53..925e946 100644 --- a/docs/name-resolution.md +++ b/docs/name-resolution.md @@ -255,12 +255,30 @@ cell destinations, and 37 footer-array destinations across the 74 records. The r the first nonempty victory-condition string, making consumer/value correlations readable without inventing a stage-name field. -The strongest consumer-backed meanings are curated in `globals.toml`: `0x27b9..0x27bc` are the two -victory and two defeat-condition lines rendered by AIM/FIELD; `0xe7302` is passed by FIELD to `play-bgm`; -`0xe730c` is the turn limit displayed by DRAWCHP and checked by FIELD; and `0xe730d` selects defeat versus -forced-retreat clear when that limit expires. The remaining stage scalars and buffer families stay raw until -their FIELD/SETEN/SETOBJ/DRAWMAP consumers support stable names. In particular, repeated asset-like values -inside `0xe7311` are not promoted merely because they resolve through SYS4INI. +The strongest header meanings are curated in `globals.toml`: `0x27b9..0x27bc` are the two victory and two +defeat-condition lines rendered by AIM/FIELD; `0xe7302` is passed by FIELD to `play-bgm`; `0xe730c` is the +turn limit displayed by DRAWCHP and checked by FIELD; and `0xe730d` selects defeat versus forced-retreat +clear when that limit expires. STAGECLEAR establishes `0xe7303` as the target/par turn count and scales +`0xe7304`'s persistent reward increment by performance against that target. FIELD establishes `0xe730b` as +the gate that disables its already-cleared-stage retreat/replay conversion. + +The first map/object pass resolves seven more buffer families. FIELD loads `0xe7311[1..19]` into tiled +surface slots and DRAWMAP selects those surfaces through terrain metadata, proving it is the current stage's +map-texture override list: positive values are SYS4INI resource ids, zero disables a slot, and -1 selects +the shared fallback. DRAWOBJ converts `0xe7325` and `0xe7357` to map-space coordinates, while +SETOBJ/DRAWOBJ/FIELD use `0xe7389` to index shared object definitions. They are object tile X, tile Y, and +type id. SETOBJ tests the `{3,4,7}` masks in `0xe7483` against GAMESTART's three-way `difficulty_index`, +then applies seven required and five forbidden one-based ids from `0xe74b5`/`0xe7613` against the shared +`story_event_flags` bank. + +Generated INIT records now retain their raw `fields`/`record_fields`/buffer keys and additionally expose a +flat `semantic_fields` projection joined through the top-level `field_semantics` map. For STINIT, the four +confirmed parallel buffers plus both prerequisite tables are also assembled into 2,312 +`object_placements` across 66 stages. Each placement contains its slot, type, tile coordinates, difficulty +mask, and populated positive/negative story prerequisites. The four still-unresolved object-parameter +buffers remain attached under `unknown_fields`, so this convenience view loses no evidence or invents names. +The next structural batch is the separate 30-slot enemy-spawn schema and its difficulty-dependent footer +arrays. ### The curated registry — `vm-map/globals.toml` (2026-07-07) @@ -307,11 +325,11 @@ are *not* story flags — the miner over-tags them; they are recategorized `unkn The v1 map labels *shapes and tables*; the next increments add *meaning*, cheapest first: 1. **Continue INIT semantics by evidence density.** ITINIT/SKINIT, the confirmed EBINIT row layouts, and - STINIT's mixed stage records now have machine-readable investigation surfaces. Next trace STINIT's - highest-coverage object/enemy/map buffers through FIELD, SETOBJ, SETEN, and DRAWMAP; isolate EBINIT's - remaining voice/action slots only when their routed consumers make them distinguishable. Investigate - unread enums and signed classes only when consumer or native evidence appears. Preserve explicit joins - and do not infer meaning from column position alone. + STINIT's mixed stage records now have machine-readable investigation surfaces. Next assemble STINIT's + 30-slot enemy-spawn schema through FIELD/SETEN/ADDEN, including its three-difficulty footer arrays; trace + the four remaining specialized object parameters only when their type-specific branches distinguish + them. Isolate EBINIT's remaining voice/action slots under the same rule. Preserve explicit joins and do + not infer meaning from column position alone. 2. **Extend message-table joins beyond the completed ITMES/SKMES pair** (`VIMES`, other id dispatchers, …) and fold in other `set-string`/`copy-to-global` writers not covered by the `*INIT` set. 3. **Label 2D record tables by their readers** — cross-reference which scripts read each diff --git a/docs/phase-b-framework.md b/docs/phase-b-framework.md index edb5037..2fe94d9 100644 --- a/docs/phase-b-framework.md +++ b/docs/phase-b-framework.md @@ -749,7 +749,11 @@ The remaining EBINIT unknowns are now the unread `0x7843e` enum and the signed m enemy AI appears to live outside the static EBINIT schema. STINIT's separate mixed parser is now complete: all 74 sparse stage ids retain their victory/defeat strings, six scalars, fixed-buffer cells, and 1,396 footer-array copies. AIM/FIELD/DRAWCHP consumers establish the four condition slots, stage BGM, turn limit, -and turn-limit outcome; the object/enemy/map buffer families remain an evidence-driven follow-up. +and turn-limit outcome. The next pass establishes target clear turns, performance reward, replay behavior, +the 20-slot map-texture override list, object type/tile/difficulty fields, and seven required plus five +forbidden story prerequisites. Generated records now carry a joined `semantic_fields` view and assemble +2,312 object placements across 66 stages while retaining four unresolved type-specific parameters by raw +address. The separate 30-slot enemy-spawn family remains the next evidence-driven batch. Once the natural spine and first gameplay loop are trustworthy, broaden in independent tracks: @@ -759,8 +763,8 @@ Once the natural spine and first gameplay loop are trustworthy, broaden in indep - Unit statistics, equipment, inventory, skills, magic, heroine forms, and progression. - Combat resolution presentation, enemy turns/AI services, and win/loss transitions. - A full chapter of ADV and the scene types encountered between gameplay segments. -- Remaining STINIT object/enemy/map buffer semantics and other data schemas when their runtime consumers make - them necessary. +- Remaining STINIT enemy-spawn and specialized object-parameter semantics, plus other data schemas when their + runtime consumers make them necessary. The high-level Phase B direction remains canonical in `docs/remake-architecture-and-roadmap.md`; this file only provides the execution framework. diff --git a/docs/script-inventory.md b/docs/script-inventory.md index 35b8d06..af46602 100644 --- a/docs/script-inventory.md +++ b/docs/script-inventory.md @@ -70,7 +70,9 @@ contains only genuine parallel arrays. STINIT uses the separate mixed shape: 74 sparse `scjump_progress_a` branches load one current-stage work buffer rather than parallel per-id arrays. Its generated records preserve four mission-condition strings, six scalars, cells in 29 preallocated buffers, and all 1,396 length-prefixed footer-array copies. Confirmed -stage field meanings and the evidence workflow live in `docs/name-resolution.md`. +stage field meanings and the evidence workflow live in `docs/name-resolution.md`. The first semantic join +now assembles the confirmed type/X/Y/difficulty and story-prerequisite buffers into 2,312 object placements +across 66 stages; unknown type-specific parameters remain raw, and the enemy-spawn buffers remain separate. ### Message/string tables (`*MES`) `ITMES` (64 KB — item text), `VIMES` (43 KB), `EIMES` (37 KB), `SKMES` (31 KB — skill diff --git a/docs/tools-reference.md b/docs/tools-reference.md index aeaddaa..30f0cd7 100644 --- a/docs/tools-reference.md +++ b/docs/tools-reference.md @@ -55,7 +55,7 @@ All opcode knowledge (ABI, semantics, provenance, `depends_on`) is hand-edited * |---|---|---|---| | `extract_phase2.py` | Batch: disassembly + text corpora for every script. | `extract_phase2.py` | corpus → `build/disasm/*.asm`, `build/text/{dialogue.jsonl,strings.jsonl,*.strings.txt}`, `build/manifest.json` | | `extract_message_table.py` | Discover a repeated global-id dispatch chain such as ITMES/SKMES, reconstruct player-facing title/description lines (including furigana surface text and readings), and emit an ID-keyed message table with bytecode provenance. | `extract_message_table.py [OUTNAME]` | `.BIN` → `build/data/.json` | -| `extract_init.py` | Parse a `*INIT` data table (auto-detects name / numeric / footer / mixed shape). Name tables infer their reserved record span, preserve sparse one-based runtime ids, distinguish lookup bases from first written cells, statically evaluate direct and negative-value writes, and separate parallel `fields` from linked row-major `record_fields`. Mixed tables recover selector-dispatched records, condition strings, scalars, preallocated buffer cells, consumer-confirmed strides, and length-prefixed footer arrays; STINIT is the first such table. ITINIT and SKINIT join ITMES/SKMES messages; top-level `field_semantics` maps raw keys to canonical global/column names. Refreshes the generated data index. | `extract_init.py [OUTNAME] [--mode …]` | `
.BIN` plus matching `.BIN` when supported + `build/globals.json` → `build/data/.json`, `build/data/README.md` | +| `extract_init.py` | Parse a `*INIT` data table (auto-detects name / numeric / footer / mixed shape). Name tables infer their reserved record span, preserve sparse one-based runtime ids, distinguish lookup bases from first written cells, statically evaluate direct and negative-value writes, and separate parallel `fields` from linked row-major `record_fields`. Mixed tables recover selector-dispatched records, condition strings, scalars, preallocated buffer cells, consumer-confirmed strides, and length-prefixed footer arrays; STINIT additionally assembles confirmed buffers into `object_placements`. ITINIT and SKINIT join ITMES/SKMES messages. Top-level `field_semantics` maps raw keys to canonical global/column names, while each record's `semantic_fields` is the generated name-keyed join; raw keys remain provenance. OUTNAME accepts a stem or one `.json` suffix and rejects paths. Refreshes the generated data index. | `extract_init.py
[OUTNAME] [--mode …]` | `
.BIN` plus matching `.BIN` when supported + `build/globals.json` → `build/data/.json`, `build/data/README.md` | | `init_table_profile.py` | Build the static investigation surface for an extracted name/numeric/mixed table: message coverage, per-scalar/string/array-cell/footer-array population and value distributions, representative records, and direct opcode/script consumers. `--message-query REGEX` shows every matching name/message beside all populated fields for semantic correlation. Findings are evidence only; confirmed meanings go in `vm-map/globals.toml`. | `init_table_profile.py
[--build] [--limit N] [--message-query REGEX]` | `build/data/
.json` + corpus → stdout; with `--build`, `build/data/
-field-profile.{json,md}` | | `test_extract_init.py`, `test_init_table_profile.py` | Regression checks for sparse one-based and mixed selector-dispatched INIT extraction, MES reconstruction/joins, footer-array accounting, and field/message profiling. | run each directly | — | | `global_map.py` | Build the partial global-variable name map from static evidence. | `global_map.py` | corpus + `build/data/` → `build/global-var-map.{json,md}` | diff --git a/docs/vm-mapping-plan.md b/docs/vm-mapping-plan.md index e34a5fc..3f582c2 100644 --- a/docs/vm-mapping-plan.md +++ b/docs/vm-mapping-plan.md @@ -54,7 +54,7 @@ - [x] **2.0 — Project structure.** Established `docs/`, `build/{disasm,text,data,scripts-json}/`, `godot/`; game install stays read-only in place. Also relaxed the loader magic check to the `SYS4` family (`SYS4424` patch scripts now parse — was silently skipping 5 scripts). - [x] **2.1 — Text corpora.** `tools/extract_phase2.py` → 481/481 scripts: full disassembly (`build/disasm/*.asm`), per-script strings, `build/text/dialogue.jsonl` (**30,057 show-text lines** — the translation corpus), `build/text/strings.jsonl` (38,449 strings tagged by source opcode), `build/manifest.json`. - [x] **2.2 — `*INIT` data tables → JSON.** `tools/extract_init.py` auto-detects table shape (`name`/`numeric`/`footer`/`mixed`) → **SKINIT (131 skills), ITINIT (287 items), EBINIT (277 units)** [name: sparse one-based name/description/fields], **CGINIT (379 CG entries)** [numeric: index-keyed columns], **MPINIT (1472 map records)** [footer: length-prefixed arrays], and **STINIT (74 stages)** [mixed: selector-dispatched strings/scalars/buffer cells/footer arrays]. Validated; see `build/data/README.md`. Raw addresses remain bytecode provenance; confirmed semantics come from `vm-map/globals.toml`. -- [x] **2.3 — Extract `STINIT`'s 74 sparse stage records.** The mixed mode identifies the dominant `scjump_progress_a` dispatch, recovers 29 preallocated buffer layouts (including six consumer-confirmed row strides), and keeps four condition strings, six scalars, fixed-buffer writes, and all 1,396 footer-array copies separated by stage id. The generated profile supplies population/value and direct-consumer evidence; victory/defeat lines, stage BGM, turn limit, and expiry outcome are curated. +- [x] **2.3 — Extract `STINIT`'s 74 sparse stage records.** The mixed mode identifies the dominant `scjump_progress_a` dispatch, recovers 29 preallocated buffer layouts (including six consumer-confirmed row strides), and keeps four condition strings, six scalars, fixed-buffer writes, and all 1,396 footer-array copies separated by stage id. Profiles supply population/value and direct-consumer evidence. Header/map semantics now cover conditions, BGM, turn/replay/clear-reward settings, and map texture overrides; four parallel object fields plus 7 required/5 forbidden story prerequisites assemble into 2,312 object placements. Raw address views remain alongside generated `semantic_fields`. - [x] **2.4 — Partial global-var map BUILT + wired into the disassembler.** `tools/global_map.py` → `build/global-var-map.{json,md}` (16,354/49,435 globals labelled: string tables, `*INIT` field arrays, 122 record tables w/ strides, current-entity index pointers). `sys4load` renders the labels inline (`=rec[s30]`, `=current-entity-index?`). See `docs/name-resolution.md`. - [ ] **2.5 — Grow the global-var map (future, incremental).** Static first: fold in `*MES` writers; label 2D record tables by their reader scripts. Then Frida to name *which stat* each field is. Full detail: `docs/name-resolution.md` → "Future step — growing the map". Also deferred: `call-script` id→name resolution (engine-level — SCJUMP.BIN decode or Frida; see `docs/name-resolution.md` #1). diff --git a/tools/extract_init.py b/tools/extract_init.py index 700af77..cccb20b 100644 --- a/tools/extract_init.py +++ b/tools/extract_init.py @@ -54,6 +54,16 @@ def resolve(name: str) -> Path: raise SystemExit(f"not found: {name}.BIN") +def normalize_outname(value: str) -> str: + """Accept a generated-file stem, not a path; tolerate one `.json` suffix.""" + if not value or Path(value).name != value or "/" in value or "\\" in value: + raise ValueError("OUTNAME must be a file stem, not a path") + outname = value.removesuffix(".json") + if not outname or outname in {".", ".."}: + raise ValueError("OUTNAME must be a nonempty file stem") + return outname + + def _val(arg): """Render an operand as an int (immediate) or a {type,value} ref.""" t, v = arg @@ -470,7 +480,9 @@ def _global_registry() -> dict: return {} -def field_semantics(records: list[dict]) -> dict[str, str]: +def field_semantics( + records: list[dict], array_layouts: dict[str, dict] | None = None +) -> dict[str, str]: """Map raw extracted field keys to canonical semantic names when available.""" keys = { key @@ -494,7 +506,17 @@ def field_semantics(records: list[dict]) -> dict[str, str]: if not name: continue if len(parts) == 2: - name = f"{name}.index_{parts[1]}" + index = int(parts[1]) + layout = (array_layouts or {}).get(f"0x{int(parts[0], 16):x}", {}) + if stride := layout.get("stride"): + row, column = divmod(index, stride) + column_name = entry.get("columns", {}).get( + str(column), f"column_{column}" + ) + name = f"{name}.row_{row}.{column_name}" + else: + index_name = entry.get("columns", {}).get(str(index), f"index_{index}") + name = f"{name}.{index_name}" elif len(parts) == 3: column = parts[2] column_name = entry.get("columns", {}).get(column, f"column_{column}") @@ -503,6 +525,74 @@ def field_semantics(records: list[dict]) -> dict[str, str]: return semantics +def attach_semantic_fields(records: list[dict], semantics: dict[str, str]) -> None: + """Add a generated name-keyed view while retaining raw address provenance.""" + containers = ( + "string_fields", "fields", "array_fields", "footer_arrays", "record_fields" + ) + for record in records: + semantic_fields = {} + for container in containers: + for key, value in record.get(container, {}).items(): + if semantic_name := semantics.get(key): + if semantic_name in semantic_fields: + raise ValueError( + f"record {record['id']}: duplicate semantic field {semantic_name}" + ) + semantic_fields[semantic_name] = value + if semantic_fields: + record["semantic_fields"] = semantic_fields + else: + record.pop("semantic_fields", None) + + +def attach_stage_object_placements(records: list[dict]) -> None: + """Assemble STINIT's parallel object buffers into modder-facing slot records.""" + known_fields = { + "type_id": "0xe7389", + "tile_x": "0xe7325", + "tile_y": "0xe7357", + "difficulty_mask": "0xe7483", + } + unknown_bases = ("0xe73bb", "0xe73ed", "0xe741f", "0xe7451") + for record in records: + fields = record.get("array_fields", {}) + objects = [] + for slot in range(1, 50): + type_key = f"{known_fields['type_id']}/{slot}" + if type_key not in fields: + continue + obj = {"slot": slot} + for semantic_name, base in known_fields.items(): + if (key := f"{base}/{slot}") in fields: + obj[semantic_name] = fields[key] + required = [ + fields[key] + for column in range(7) + if (key := f"0xe74b5/{slot * 7 + column}") in fields + and fields[key] > 0 + ] + forbidden = [ + fields[key] + for column in range(5) + if (key := f"0xe7613/{slot * 5 + column}") in fields + and fields[key] > 0 + ] + if required: + obj["required_story_flags"] = required + if forbidden: + obj["forbidden_story_flags"] = forbidden + unknown = { + base: fields[f"{base}/{slot}"] + for base in unknown_bases + if f"{base}/{slot}" in fields + } + if unknown: + obj["unknown_fields"] = unknown + objects.append(obj) + record["object_placements"] = objects + + def write_data_index(data_dir: Path) -> None: """Regenerate the disposable build/data index from current table JSONs.""" tables = [] @@ -549,10 +639,13 @@ def write_data_index(data_dir: Path) -> None: "title, description, furigana, and bytecode dispatch offset separately from the", "short description stored by the INIT script.", "Top-level `field_semantics` maps raw array/row-column keys to canonical machine-readable", - "names from `vm-map/globals.toml`; raw keys remain intact as bytecode provenance.", + "names from `vm-map/globals.toml`; each record's generated `semantic_fields` is the joined", + "name-keyed convenience view. Raw keys remain intact as bytecode provenance.", "", "Mixed-mode tables preserve the sparse selector id, branch offset, condition strings,", "scalar fields, cells within preallocated buffers, and length-prefixed footer arrays.", + "STINIT additionally joins the confirmed parallel object buffers into per-slot", + "`object_placements`; unresolved type-specific parameters remain in `unknown_fields`.", "Use `tools/init_table_profile.py
--build` to generate value/population and", "direct-consumer evidence.", "", @@ -579,7 +672,10 @@ def main() -> int: if not argv: raise SystemExit(__doc__) name = argv[0].upper().removesuffix(".BIN") - outname = argv[1] if len(argv) > 1 else name + try: + outname = normalize_outname(argv[1]) if len(argv) > 1 else name + except ValueError as error: + raise SystemExit(str(error)) from error scr = sys4load.load(resolve(name)) mode = mode_arg or detect_mode(scr) @@ -597,7 +693,10 @@ def main() -> int: ) cols = sorted({c for r in recs for c in r.get("fields", {})}, key=lambda h: int(h, 16)) - semantics = field_semantics(recs) + semantics = field_semantics(recs, meta.get("array_layouts")) + attach_semantic_fields(recs, semantics) + if mode == "mixed" and name == "STINIT": + attach_stage_object_placements(recs) out = {"table": name, "source": scr.path.name, "magic": scr.magic, "mode": mode, "record_count": len(recs), **meta, "field_columns": cols if mode != "footer" else None, diff --git a/tools/test_extract_init.py b/tools/test_extract_init.py index d4fc337..a61ab7e 100644 --- a/tools/test_extract_init.py +++ b/tools/test_extract_init.py @@ -66,6 +66,18 @@ def test_static_negative_write() -> None: "INIT subtraction writes preserve negative values") +def test_output_name_validation() -> None: + check(extract_init.normalize_outname("ITINIT.json") == "ITINIT", + "INIT output names tolerate one JSON suffix") + try: + extract_init.normalize_outname("build/data/ITINIT.json") + except ValueError: + rejected = True + else: + rejected = False + check(rejected, "INIT output names reject nested paths") + + def test_real_mixed_table() -> None: script = sys4load.load(extract_init.resolve("STINIT")) check(extract_init.detect_mode(script) == "mixed", @@ -93,6 +105,21 @@ def test_real_mixed_table() -> None: "STINIT length-prefixed footer arrays retain their destination") check(sum(len(record.get("footer_arrays", {})) for record in records) == 1396, "STINIT accounts for every footer-array copy") + extract_init.attach_stage_object_placements(records) + first_object = records[0]["object_placements"][0] + check(first_object == { + "slot": 1, + "type_id": 1, + "tile_x": 13, + "tile_y": 1, + "difficulty_mask": 7, + "unknown_fields": {"0xe73bb": 1}, + }, "STINIT joins confirmed object buffers into one placement record") + stage2_slot3 = next( + obj for obj in records[1]["object_placements"] if obj["slot"] == 3 + ) + check(stage2_slot3["required_story_flags"] == [902], + "STINIT object placements join positive story prerequisites") def test_real_message_tables() -> None: @@ -167,11 +194,29 @@ def test_field_semantics() -> None: "parallel INIT fields expose canonical semantic names") check(semantics["0x9f541/14/8"] == "item_stat_modifiers.critical_chance", "row-table columns expose canonical semantic names") + extract_init.attach_semantic_fields(items, semantics) + check(items[0]["semantic_fields"]["item_sort_key"] == 10, + "records expose a joined semantic field view") + + stages, meta = extract_init.extract_mixed( + sys4load.load(extract_init.resolve("STINIT")) + ) + stage_semantics = extract_init.field_semantics( + stages, meta["array_layouts"] + ) + check(stage_semantics["0xe7325/1"] == "stage_object_tile_x.index_1", + "mixed buffer cells expose canonical semantic names") + check( + stage_semantics["0xe74b5/21"] + == "stage_object_required_story_flags.row_3.required_flag_1", + "mixed row buffers expose row and column semantics", + ) if __name__ == "__main__": test_real_name_tables() test_static_negative_write() + test_output_name_validation() test_real_mixed_table() test_real_message_tables() test_message_join() diff --git a/vm-map/globals.toml b/vm-map/globals.toml index ac3d35e..7f9269f 100644 --- a/vm-map/globals.toml +++ b/vm-map/globals.toml @@ -1671,3 +1671,128 @@ value_domain = "{0,1}; 0 defeat on expiry, 1 forced-retreat clear on expiry" usage = "STINIT mode paired with stage_turn_limit. Stage 1 stores 0 and describes 50-turn expiry as defeat; stage 2 stores 1 and explicitly describes 15-turn expiry as a forced-retreat clear." source = "investigation" confidence = "high" + +[[global]] +address = "0x32f1" +name = "difficulty_index" +category = "index-pointer" +type = "int" +value_domain = "{0,1,2}" +usage = "GAMESTART's three-way difficulty selection. SETEN uses the zero-based index to select difficulty stat adjustments; FIELD indexes three-column stage-enemy arrays with it, and SETOBJ tests it against each object's difficulty mask." +source = "investigation" +confidence = "high" + +[[global]] +address = "0x6d3" +name = "story_event_flags" +category = "data-table" +type = "int[]" +value_domain = "mostly {0,1}" +usage = "Shared one-based story/event flag bank. Progression, recruitment, item, and stage logic read or write individual cells; STINIT object prerequisite ids are decremented by one before SETOBJ tests this array." +source = "investigation" +confidence = "high" + +[[global]] +address = "0xe7303" +name = "stage_target_clear_turns" +category = "data-table" +type = "int" +value_domain = "0..50; 0 disables turn-performance scaling" +usage = "STINIT's target/par turn count. STAGECLEAR divides elapsed turns by this value to derive a performance multiplier; when replaying an already-cleared ordinary stage, FIELD also uses it as the forced-retreat turn limit." +source = "investigation" +confidence = "high" + +[[global]] +address = "0xe7304" +name = "stage_clear_performance_bonus" +category = "data-table" +type = "int" +value_domain = "0..20" +usage = "Base stage-clear reward increment. STAGECLEAR multiplies it by the turn-performance percentage derived from stage_target_clear_turns, divides by 100, and adds the result to the capped persistent reward counter at 0x6719. The counter's player-facing resource name remains unresolved." +source = "investigation" +confidence = "med" + +[[global]] +address = "0xe730b" +name = "stage_replay_rules_disabled" +category = "data-table" +type = "int" +value_domain = "{0,1}; 1 preserves the original mission rules after clear" +usage = "FIELD's already-cleared-stage override gate. For a cleared stage with value 0, FIELD replaces the mission with an all-party retreat objective and a stage_target_clear_turns forced-retreat limit; value 1 suppresses that replay conversion and related revisit handling." +source = "investigation" +confidence = "high" + +[[global]] +address = "0xe7311" +name = "stage_map_texture_asset_overrides" +category = "data-table" +type = "int[20]" +value_domain = "SYS4INI resource id when positive, 0 disables the slot, -1 selects the shared fallback asset" +usage = "Twenty current-stage map texture slots. FIELD loads positive resource ids into tiled surface slots 0x52+index, skips zero, and substitutes the shared per-index fallback for -1; DRAWMAP selects and draws those surfaces through the terrain-to-texture-slot map." +source = "investigation" +confidence = "high" + +[[global]] +address = "0xe7325" +name = "stage_object_tile_x" +category = "data-table" +type = "int[50]" +value_domain = "map tile X coordinate, 1..25 in the shipped stages" +usage = "X coordinate for each current-stage object slot. DRAWOBJ converts it to centered map-space pixels; FIELD and CALCOCC use it with stage_object_tile_y for object interaction and occupancy." +source = "investigation" +confidence = "high" + +[[global]] +address = "0xe7357" +name = "stage_object_tile_y" +category = "data-table" +type = "int[50]" +value_domain = "map tile Y coordinate, 1..800 in the shipped stages" +usage = "Y coordinate for each current-stage object slot. DRAWOBJ converts it to centered map-space pixels; FIELD and CALCOCC use it with stage_object_tile_x for object interaction and occupancy." +source = "investigation" +confidence = "high" + +[[global]] +address = "0xe7389" +name = "stage_object_type_id" +category = "data-table" +type = "int[50]" +value_domain = "object definition id 1..46" +usage = "Object-definition id for each current-stage object slot. SETOBJ decides whether the slot exists, while DRAWOBJ and FIELD use the id to select shared object graphics, dimensions, animation, collision, and behavior metadata." +source = "investigation" +confidence = "high" + +[[global]] +address = "0xe7483" +name = "stage_object_difficulty_mask" +category = "data-table" +type = "int[50]" +value_domain = "{3,4,7}; bit 0/1/2 enables the object on the corresponding difficulty" +usage = "Per-object difficulty inclusion mask. SETOBJ uses check-bit with difficulty_index and rejects the object slot when the selected difficulty bit is absent." +source = "investigation" +confidence = "high" +depends_on = ["0x32f1"] + +[[global]] +address = "0xe74b5" +name = "stage_object_required_story_flags" +category = "data-table" +type = "int[50][7]" +columns = { "0" = "required_flag_1", "1" = "required_flag_2", "2" = "required_flag_3", "3" = "required_flag_4", "4" = "required_flag_5", "5" = "required_flag_6", "6" = "required_flag_7" } +value_domain = "one-based story_event_flags ids; zero means no requirement" +usage = "Seven positive prerequisites per stage object. SETOBJ subtracts one from every populated id and suppresses the object unless the corresponding story_event_flags cell equals 1." +source = "investigation" +confidence = "high" +depends_on = ["0x6d3"] + +[[global]] +address = "0xe7613" +name = "stage_object_forbidden_story_flags" +category = "data-table" +type = "int[50][5]" +columns = { "0" = "forbidden_flag_1", "1" = "forbidden_flag_2", "2" = "forbidden_flag_3", "3" = "forbidden_flag_4", "4" = "forbidden_flag_5" } +value_domain = "one-based story_event_flags ids; zero means no exclusion" +usage = "Five negative prerequisites per stage object. SETOBJ subtracts one from every populated id and suppresses the object when the corresponding story_event_flags cell equals 1." +source = "investigation" +confidence = "high" +depends_on = ["0x6d3"]