Model linked INIT record tables
This commit is contained in:
10976
docs/global-reference.md
10976
docs/global-reference.md
File diff suppressed because it is too large
Load Diff
@@ -87,8 +87,8 @@ free handholds — several of which we've already built:
|
|||||||
1. **The `*INIT` scripts are the writers, and we already extracted them.** `EBINIT`/`ITINIT`/
|
1. **The `*INIT` scripts are the writers, and we already extracted them.** `EBINIT`/`ITINIT`/
|
||||||
`SKINIT`/`CGINIT`/`MPINIT` populate global arrays with names and data (`build/data/*.json`).
|
`SKINIT`/`CGINIT`/`MPINIT` populate global arrays with names and data (`build/data/*.json`).
|
||||||
The base address `EBINIT` writes 277 unit names into *is* the unit-name table. Each JSON's
|
The base address `EBINIT` writes 277 unit names into *is* the unit-name table. Each JSON's
|
||||||
`name_array_base`, `desc_array_bases`, and `field_columns` are literally global addresses we
|
`name_array_base`, `desc_array_bases`, `field_columns`, and `record_field_columns` are literally
|
||||||
can label by which table wrote them.
|
global addresses and access shapes we can label by which table wrote them.
|
||||||
2. **Strings anchor the string side for free.** `set-string` writes skill names to
|
2. **Strings anchor the string side for free.** `set-string` writes skill names to
|
||||||
`global-string 0x23a3…` → that array is the skill-name table. `*MES` tables likewise.
|
`global-string 0x23a3…` → that array is the skill-name table. `*MES` tables likewise.
|
||||||
3. **Access shape reveals structure without names.** A global read as `base[unit*stride + col]`
|
3. **Access shape reveals structure without names.** A global read as `base[unit*stride + col]`
|
||||||
@@ -106,14 +106,14 @@ label the ~dozen hottest globals first (biggest readability payoff), grow the re
|
|||||||
(all evidence) + `build/global-var-map.md` (labelled subset). It ingests `build/data/*.json`
|
(all evidence) + `build/global-var-map.md` (labelled subset). It ingests `build/data/*.json`
|
||||||
(name/desc/field bases), scans the 481-script corpus for each global's **access shape**
|
(name/desc/field bases), scans the 481-script corpus for each global's **access shape**
|
||||||
(2D-table base + stride, 1D-array base, row-index, scalar), and ranks "current entity" index
|
(2D-table base + stride, 1D-array base, row-index, scalar), and ranks "current entity" index
|
||||||
pointers by purity. **Current result: 15,918 of 49,386 distinct globals labelled** —
|
pointers by purity. **Current result: 4,960 of 41,611 distinct globals labelled** —
|
||||||
|
|
||||||
| kind | count | example |
|
| kind | count | example |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| string tables (names/descs/messages) | 3,206 | `0x23a2` = skill-name lookup base |
|
| string tables (names/descs/messages) | 3,206 | `0x23a2` = skill-name lookup base |
|
||||||
| per-entity data-field arrays (from *INIT) | 12,311 | dense = shared fields, `?` = sparse per-entity |
|
| per-entity data-field arrays (from *INIT) | 1,353 | dense = shared fields, `?` = sparse per-entity |
|
||||||
| row-major record tables (from access shape) | 122 | `0x52383` = record-table[stride 30] |
|
| row-major record tables (from access shape) | 122 | `0x52383` = record-table[stride 30] |
|
||||||
| 1D arrays | 307 | |
|
| 1D arrays | 253 | |
|
||||||
| index / "current entity" pointers | 26 | `0x152616` (purity 0.51), `0xeff75` (0.95) |
|
| index / "current entity" pointers | 26 | `0x152616` (purity 0.51), `0xeff75` (0.95) |
|
||||||
|
|
||||||
**Validated against `RECOVER`:** the map independently reproduces its hand-traced layout —
|
**Validated against `RECOVER`:** the map independently reproduces its hand-traced layout —
|
||||||
@@ -148,15 +148,30 @@ Semantic recovery is an evidence ladder, cheapest and strongest first:
|
|||||||
promoting guesses. Use dynamic observation only for fields that remain ambiguous after static consumers.
|
promoting guesses. Use dynamic observation only for fields that remain ambiguous after static consumers.
|
||||||
|
|
||||||
`tools/init_table_profile.py ITINIT --build` materializes steps 1–2 in
|
`tools/init_table_profile.py ITINIT --build` materializes steps 1–2 in
|
||||||
`build/data/ITINIT-field-profile.{json,md}`. The first pass names eleven stable ITINIT arrays: catalog sort
|
`build/data/ITINIT-field-profile.{json,md}`. The initial pass names thirteen parallel arrays: catalog sort
|
||||||
key, random-item tier, item category, icon id, shared ITMES handler id, attack and defense elements, weapon
|
key, random-item tier, item category, icon id, shared ITMES handler id, attack and defense elements, weapon
|
||||||
class, granted skill id, and minimum/maximum range. The strongest joins are independently human-readable:
|
class, granted skill id, minimum/maximum range, essence recovery, and an equipment sex mask. The strongest
|
||||||
attack/defense values index AFINIT's Japanese attribute strings, granted-skill values resolve to SKINIT, all
|
joins are independently human-readable: attack/defense values index AFINIT's Japanese attribute strings,
|
||||||
handler values resolve to ITMES.BIN, and every min/max-range record says `range 2` in its item description.
|
granted-skill values resolve to SKINIT, all handler values resolve to ITMES.BIN, and every min/max-range
|
||||||
|
record says `range 2` in its item description.
|
||||||
|
|
||||||
The remaining 764 ITINIT write bases are not automatically 764 item fields. Most occur once because ITINIT
|
The apparent 764 additional ITINIT field bases were a structural artifact, not 764 sparse arrays. For each
|
||||||
also initializes linked effect/stat arrays while defining an item. Treat the profile's `population` and
|
write, subtracting `item_id * stride` and comparing the destination with corpus-observed `lookup-array-2d`
|
||||||
consumer evidence as a classification aid before assigning table ownership.
|
consumers assigns all 764 writes unambiguously to six row-major tables and 43 populated columns:
|
||||||
|
|
||||||
|
| base | stride | populated writes | semantic role |
|
||||||
|
|---|---:|---:|---|
|
||||||
|
| `0x8e7b9` | 5 | 20 | character-id equipment whitelist |
|
||||||
|
| `0x906f9` | 30 | 36 | attack-inflicted condition levels |
|
||||||
|
| `0x97c29` | 30 | 11 | equipped/passive condition levels |
|
||||||
|
| `0x9f541` | 14 | 301 | additive equipment stat modifiers |
|
||||||
|
| `0xa2bf1` | 10 | 379 | per-stat tuning curve ids |
|
||||||
|
| `0xa5301` | 3 | 17 | HP/SP/FS recovery amounts |
|
||||||
|
|
||||||
|
`extract_init.py` now records these as `record_fields["base/stride/column"]` rather than inventing a
|
||||||
|
one-off `fields` base for every row. Applying the same rule also exposes 17 linked SKINIT columns and 83
|
||||||
|
linked EBINIT columns. This correction reduces the auto map's false INIT-field labels from 12,311 to 1,353;
|
||||||
|
the raw write data was valid, but its former ownership model was not.
|
||||||
|
|
||||||
### The curated registry — `vm-map/globals.toml` (2026-07-07)
|
### The curated registry — `vm-map/globals.toml` (2026-07-07)
|
||||||
|
|
||||||
@@ -202,8 +217,8 @@ are *not* story flags — the miner over-tags them; they are recategorized `unkn
|
|||||||
|
|
||||||
The v1 map labels *shapes and tables*; the next increments add *meaning*, cheapest first:
|
The v1 map labels *shapes and tables*; the next increments add *meaning*, cheapest first:
|
||||||
|
|
||||||
1. **Continue INIT semantics by evidence density.** Finish ITINIT's repeatedly populated fields and linked
|
1. **Continue INIT semantics by evidence density.** Resolve ITINIT stat column 8 and the remaining condition
|
||||||
effect arrays, then run the same profiler on SKINIT and EBINIT. Add explicit foreign-key joins (item →
|
enum columns, then run the same profiler on SKINIT and EBINIT. Add explicit foreign-key joins (item →
|
||||||
skill, unit → attack/skill) once the target table ids are confirmed. Do not infer meaning from column
|
skill, unit → attack/skill) once the target table ids are confirmed. Do not infer meaning from column
|
||||||
position alone.
|
position alone.
|
||||||
2. **Fold in the `*MES` message-table writers** (`ITMES`, `SKMES`, `VIMES`, …) and any other
|
2. **Fold in the `*MES` message-table writers** (`ITMES`, `SKMES`, `VIMES`, …) and any other
|
||||||
|
|||||||
@@ -714,10 +714,12 @@ gate on the one-player-attack acceptance path.
|
|||||||
extractor itself was audited. It now preserves sparse one-based ids and corrects ITINIT from 189 malformed
|
extractor itself was audited. It now preserves sparse one-based ids and corrects ITINIT from 189 malformed
|
||||||
records to 287 items (plus SKINIT 129→131 skills); regression checks cover the real tables. A reusable field
|
records to 287 items (plus SKINIT 129→131 skills); regression checks cover the real tables. A reusable field
|
||||||
profiler reports distributions, examples, and direct script/opcode consumers. The ITINIT pilot has curated
|
profiler reports distributions, examples, and direct script/opcode consumers. The ITINIT pilot has curated
|
||||||
eleven high-confidence arrays in `vm-map/globals.toml`: sort key, random tier, category, icon, behavior
|
thirteen parallel arrays plus six linked row-major tables in `vm-map/globals.toml`. The latter account for
|
||||||
handler, attack/defense element, weapon class, granted skill, and min/max range. Next semantic work should
|
all 764 writes previously misidentified as separate sparse item fields: character restrictions, attack and
|
||||||
classify the remaining repeatedly populated ITINIT bases, then apply the same workflow to SKINIT and EBINIT;
|
equipped condition levels, stat modifiers, tuning curves, and HP/SP/FS recovery. The extractor now emits
|
||||||
STINIT's bespoke parser remains a separate extraction task.
|
these as `record_fields[base/stride/column]`, which also exposes 17 linked SKINIT and 83 linked EBINIT
|
||||||
|
columns. Next semantic work should profile those two tables and resolve ITINIT's remaining unnamed stat and
|
||||||
|
condition columns; STINIT's bespoke parser remains a separate extraction task.
|
||||||
|
|
||||||
Once the natural spine and first gameplay loop are trustworthy, broaden in independent tracks:
|
Once the natural spine and first gameplay loop are trustworthy, broaden in independent tracks:
|
||||||
|
|
||||||
|
|||||||
@@ -60,6 +60,11 @@ items 1–101 into the first record and treated consecutive item names as descri
|
|||||||
and counts are indexed in `build/data/README.md`; field semantics are curated in `vm-map/globals.toml` and
|
and counts are indexed in `build/data/README.md`; field semantics are curated in `vm-map/globals.toml` and
|
||||||
described by the workflow in `docs/name-resolution.md`.
|
described by the workflow in `docs/name-resolution.md`.
|
||||||
|
|
||||||
|
The same audit found that linked row-major writes must not be normalized as independent parallel arrays.
|
||||||
|
Corpus `lookup-array-2d` bases and strides assign every such ITINIT write unambiguously to six tables (43
|
||||||
|
populated columns), while SKINIT and EBINIT expose 17 and 83 linked columns respectively. Generated records
|
||||||
|
now keep these under `record_fields[base/stride/column]`; `fields` contains only genuine parallel arrays.
|
||||||
|
|
||||||
### Message/string tables (`*MES`)
|
### Message/string tables (`*MES`)
|
||||||
`ITMES` (64 KB — item text), `VIMES` (43 KB), `EIMES` (37 KB), `SKMES` (31 KB — skill
|
`ITMES` (64 KB — item text), `VIMES` (43 KB), `EIMES` (37 KB), `SKMES` (31 KB — skill
|
||||||
text), `CIMES` (15 KB), `MAMES`, `INFOMES`, `MES` — where most translatable text
|
text), `CIMES` (15 KB), `MAMES`, `INFOMES`, `MES` — where most translatable text
|
||||||
|
|||||||
@@ -54,8 +54,8 @@ All opcode knowledge (ABI, semantics, provenance, `depends_on`) is hand-edited *
|
|||||||
| Tool | Purpose | Run | Reads → Writes |
|
| Tool | Purpose | Run | Reads → Writes |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `extract_phase2.py` | Batch: disassembly + text corpora for every script. | `extract_phase2.py` | corpus → `build/disasm/*.asm`, `build/text/{dialogue.jsonl,strings.jsonl,*.strings.txt}`, `build/manifest.json` |
|
| `extract_phase2.py` | Batch: disassembly + text corpora for every script. | `extract_phase2.py` | corpus → `build/disasm/*.asm`, `build/text/{dialogue.jsonl,strings.jsonl,*.strings.txt}`, `build/manifest.json` |
|
||||||
| `extract_init.py` | Parse a `*INIT` data table (auto-detects name / numeric / footer shape). Name tables infer their reserved record span, preserve sparse one-based runtime ids, and distinguish lookup bases from first written cells. Also refreshes the generated data index. | `extract_init.py <TABLE> [OUTNAME] [--mode …]` | `<TABLE>.BIN` → `build/data/<OUTNAME>.json`, `build/data/README.md` |
|
| `extract_init.py` | Parse a `*INIT` data table (auto-detects name / numeric / footer shape). Name tables infer their reserved record span, preserve sparse one-based runtime ids, distinguish lookup bases from first written cells, and use corpus-observed 2D consumers to separate parallel `fields` from linked `record_fields` keyed `base/stride/column`. Also refreshes the generated data index. | `extract_init.py <TABLE> [OUTNAME] [--mode …]` | `<TABLE>.BIN` → `build/data/<OUTNAME>.json`, `build/data/README.md` |
|
||||||
| `init_table_profile.py` | Build the static investigation surface for an extracted name/numeric table: per-base population/value distributions, representative records, and direct opcode/script consumers. Findings are evidence only; confirmed meanings go in `vm-map/globals.toml`. | `init_table_profile.py <TABLE> [--build] [--limit N]` | `build/data/<TABLE>.json` + corpus → stdout; with `--build`, `build/data/<TABLE>-field-profile.{json,md}` |
|
| `init_table_profile.py` | Build the static investigation surface for an extracted name/numeric table: per-array and per-record-column population/value distributions, representative records, and direct opcode/script consumers. Findings are evidence only; confirmed meanings go in `vm-map/globals.toml`. | `init_table_profile.py <TABLE> [--build] [--limit N]` | `build/data/<TABLE>.json` + corpus → stdout; with `--build`, `build/data/<TABLE>-field-profile.{json,md}` |
|
||||||
| `test_extract_init.py`, `test_init_table_profile.py` | Regression checks for sparse one-based INIT extraction and field profiling. | run each directly | — |
|
| `test_extract_init.py`, `test_init_table_profile.py` | Regression checks for sparse one-based INIT extraction and field profiling. | run each directly | — |
|
||||||
| `global_map.py` | Build the partial global-variable name map from static evidence. | `global_map.py` | corpus + `build/data/` → `build/global-var-map.{json,md}` |
|
| `global_map.py` | Build the partial global-variable name map from static evidence. | `global_map.py` | corpus + `build/data/` → `build/global-var-map.{json,md}` |
|
||||||
|
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ Usage: py -3.11 -X utf8 tools/extract_init.py <TABLE> [OUTNAME] [--mode name|num
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
import json
|
import json
|
||||||
import sys
|
import sys
|
||||||
|
from functools import cache
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
HERE = Path(__file__).resolve().parent
|
HERE = Path(__file__).resolve().parent
|
||||||
@@ -89,6 +90,44 @@ def _infer_record_span(string_addrs):
|
|||||||
return max(counts, key=lambda delta: (counts[delta], delta))
|
return max(counts, key=lambda delta: (counts[delta], delta))
|
||||||
|
|
||||||
|
|
||||||
|
@cache
|
||||||
|
def _known_record_tables():
|
||||||
|
"""Return corpus-observed (base, stride) pairs used by lookup-array-2d.
|
||||||
|
|
||||||
|
INIT scripts often populate linked row-major tables while defining an
|
||||||
|
entity. Treating every such write as `destination - entity_id` invents a
|
||||||
|
different one-off parallel column for every row. Consumer bytecode gives
|
||||||
|
us the unambiguous table base and stride instead.
|
||||||
|
"""
|
||||||
|
tables = set()
|
||||||
|
for path in paths.scripts().values():
|
||||||
|
try:
|
||||||
|
script = sys4load.load(path)
|
||||||
|
except sys4load.Sys4Error:
|
||||||
|
continue
|
||||||
|
for ins in script.instructions:
|
||||||
|
if (sys4load.display_label(ins.opcode) == "lookup-array-2d"
|
||||||
|
and len(ins.args) >= 5
|
||||||
|
and ins.args[1][0] in (T_GLOBAL_INT, 6)
|
||||||
|
and ins.args[3][0] == T_IMM
|
||||||
|
and ins.args[3][1] > 0):
|
||||||
|
tables.add((ins.args[1][1], ins.args[3][1]))
|
||||||
|
return tuple(sorted(tables))
|
||||||
|
|
||||||
|
|
||||||
|
def _record_table_cell(destination, record_id):
|
||||||
|
matches = []
|
||||||
|
for base, stride in _known_record_tables():
|
||||||
|
column = destination - (base + record_id * stride)
|
||||||
|
if 0 <= column < stride:
|
||||||
|
matches.append((base, stride, column))
|
||||||
|
if len(matches) > 1:
|
||||||
|
raise ValueError(
|
||||||
|
f"ambiguous record-table destination 0x{destination:x} for id {record_id}: {matches}"
|
||||||
|
)
|
||||||
|
return matches[0] if matches else None
|
||||||
|
|
||||||
|
|
||||||
def extract_name(scr):
|
def extract_name(scr):
|
||||||
string_addrs = [
|
string_addrs = [
|
||||||
ins.args[0][1]
|
ins.args[0][1]
|
||||||
@@ -112,17 +151,31 @@ def extract_name(scr):
|
|||||||
# boundary: ITINIT begins with 101 consecutive name-only records,
|
# boundary: ITINIT begins with 101 consecutive name-only records,
|
||||||
# which the old heuristic collapsed into item zero.
|
# which the old heuristic collapsed into item zero.
|
||||||
if name_write_base <= addr < name_write_base + record_span:
|
if name_write_base <= addr < name_write_base + record_span:
|
||||||
cur = {"id": addr - name_base, "name": txt, "fields": {}}
|
cur = {"id": addr - name_base, "name": txt, "fields": {}, "record_fields": {}}
|
||||||
records.append(cur); desc_slot = 0
|
records.append(cur); desc_slot = 0
|
||||||
elif cur is not None:
|
elif cur is not None:
|
||||||
key = "desc" if desc_slot == 0 else f"desc{desc_slot}"
|
key = "desc" if desc_slot == 0 else f"desc{desc_slot}"
|
||||||
cur[key] = txt; desc_bases.setdefault(key, addr - cur["id"]); desc_slot += 1
|
cur[key] = txt; desc_bases.setdefault(key, addr - cur["id"]); desc_slot += 1
|
||||||
elif ins.opcode == MOV and cur is not None and ins.args and ins.args[0][0] == T_GLOBAL_INT:
|
elif ins.opcode == MOV and cur is not None and ins.args and ins.args[0][0] == T_GLOBAL_INT:
|
||||||
cur["fields"][f"0x{ins.args[0][1] - cur['id']:x}"] = _val(ins.args[1])
|
destination = ins.args[0][1]
|
||||||
|
cell = _record_table_cell(destination, cur["id"])
|
||||||
|
if cell is None:
|
||||||
|
cur["fields"][f"0x{destination - cur['id']:x}"] = _val(ins.args[1])
|
||||||
|
else:
|
||||||
|
base, stride, column = cell
|
||||||
|
cur["record_fields"][f"0x{base:x}/{stride}/{column}"] = _val(ins.args[1])
|
||||||
|
for record in records:
|
||||||
|
if not record["record_fields"]:
|
||||||
|
del record["record_fields"]
|
||||||
|
record_columns = sorted(
|
||||||
|
{key for record in records for key in record.get("record_fields", {})},
|
||||||
|
key=lambda key: tuple(int(part, 0) for part in key.split("/")),
|
||||||
|
)
|
||||||
return records, {"name_array_base": f"0x{name_base:x}",
|
return records, {"name_array_base": f"0x{name_base:x}",
|
||||||
"name_write_base": f"0x{name_write_base:x}",
|
"name_write_base": f"0x{name_write_base:x}",
|
||||||
"first_record_id": first_record_id,
|
"first_record_id": first_record_id,
|
||||||
"record_span": record_span,
|
"record_span": record_span,
|
||||||
|
"record_field_columns": record_columns,
|
||||||
"desc_array_bases": {k: f"0x{v:x}" for k, v in sorted(desc_bases.items())}}
|
"desc_array_bases": {k: f"0x{v:x}" for k, v in sorted(desc_bases.items())}}
|
||||||
|
|
||||||
|
|
||||||
@@ -201,17 +254,23 @@ def write_data_index(data_dir: Path) -> None:
|
|||||||
"bases remain available in every record; confirmed field meanings live in",
|
"bases remain available in every record; confirmed field meanings live in",
|
||||||
"`vm-map/globals.toml` and the generated `docs/global-reference.md`.",
|
"`vm-map/globals.toml` and the generated `docs/global-reference.md`.",
|
||||||
"",
|
"",
|
||||||
"| file | mode | records | field bases |",
|
"| file | mode | records | array fields | record columns |",
|
||||||
"|---|---|---:|---:|",
|
"|---|---|---:|---:|---:|",
|
||||||
]
|
]
|
||||||
for filename, data in tables:
|
for filename, data in tables:
|
||||||
columns = len(data.get("field_columns") or [])
|
columns = len(data.get("field_columns") or [])
|
||||||
lines.append(f"| `{filename}` | {data['mode']} | {data['record_count']} | {columns} |")
|
record_columns = len(data.get("record_field_columns") or [])
|
||||||
|
lines.append(
|
||||||
|
f"| `{filename}` | {data['mode']} | {data['record_count']} | "
|
||||||
|
f"{columns} | {record_columns} |"
|
||||||
|
)
|
||||||
lines += [
|
lines += [
|
||||||
"",
|
"",
|
||||||
"Name-mode tables expose one-based runtime `id` values, the lookup `name_array_base`,",
|
"Name-mode tables expose one-based runtime `id` values, the lookup `name_array_base`,",
|
||||||
"the first populated `name_write_base`, and the reserved `record_span`. Fields are keyed",
|
"the first populated `name_write_base`, and the reserved `record_span`. Fields are keyed",
|
||||||
"by the runtime lookup base used by `lookup-array`, not merely the first written cell.",
|
"by the runtime lookup base used by `lookup-array`, not merely the first written cell.",
|
||||||
|
"Linked row-major fields are stored separately in `record_fields`, keyed as",
|
||||||
|
"`base/stride/column` from corpus-observed `lookup-array-2d` consumers.",
|
||||||
"",
|
"",
|
||||||
"Use `tools/init_table_profile.py <TABLE> --build` to generate value/population and",
|
"Use `tools/init_table_profile.py <TABLE> --build` to generate value/population and",
|
||||||
"direct-consumer evidence. `STINIT` still requires a bespoke mixed numeric/string parser.",
|
"direct-consumer evidence. `STINIT` still requires a bespoke mixed numeric/string parser.",
|
||||||
|
|||||||
@@ -49,32 +49,58 @@ def value_key(value) -> str:
|
|||||||
|
|
||||||
def profile_columns(data: dict) -> list[dict]:
|
def profile_columns(data: dict) -> list[dict]:
|
||||||
records = data["records"]
|
records = data["records"]
|
||||||
values: dict[int, list] = collections.defaultdict(list)
|
values: dict[str, list] = collections.defaultdict(list)
|
||||||
examples: dict[int, list[dict]] = collections.defaultdict(list)
|
examples: dict[str, list[dict]] = collections.defaultdict(list)
|
||||||
|
identities: dict[str, dict] = {}
|
||||||
for record in records:
|
for record in records:
|
||||||
for address, value in record.get("fields", {}).items():
|
for address, value in record.get("fields", {}).items():
|
||||||
base = int(address, 16)
|
base = int(address, 16)
|
||||||
values[base].append(value)
|
key = f"0x{base:x}"
|
||||||
if len(examples[base]) < 5:
|
identities[key] = {
|
||||||
examples[base].append({
|
"key": key, "kind": "parallel-array", "base": key,
|
||||||
|
"stride": None, "column": None,
|
||||||
|
}
|
||||||
|
values[key].append(value)
|
||||||
|
if len(examples[key]) < 5:
|
||||||
|
examples[key].append({
|
||||||
|
"id": record["id"],
|
||||||
|
"name": record.get("name", ""),
|
||||||
|
"value": value,
|
||||||
|
})
|
||||||
|
for key, value in record.get("record_fields", {}).items():
|
||||||
|
base_text, stride_text, column_text = key.split("/")
|
||||||
|
base = int(base_text, 16)
|
||||||
|
stride = int(stride_text)
|
||||||
|
column = int(column_text)
|
||||||
|
normalized_key = f"0x{base:x}/{stride}/{column}"
|
||||||
|
identities[normalized_key] = {
|
||||||
|
"key": normalized_key,
|
||||||
|
"kind": "record-column",
|
||||||
|
"base": f"0x{base:x}",
|
||||||
|
"stride": stride,
|
||||||
|
"column": column,
|
||||||
|
}
|
||||||
|
values[normalized_key].append(value)
|
||||||
|
if len(examples[normalized_key]) < 5:
|
||||||
|
examples[normalized_key].append({
|
||||||
"id": record["id"],
|
"id": record["id"],
|
||||||
"name": record.get("name", ""),
|
"name": record.get("name", ""),
|
||||||
"value": value,
|
"value": value,
|
||||||
})
|
})
|
||||||
|
|
||||||
rows = []
|
rows = []
|
||||||
for base, vals in values.items():
|
for key, vals in values.items():
|
||||||
common = collections.Counter(value_key(value) for value in vals).most_common(6)
|
common = collections.Counter(value_key(value) for value in vals).most_common(6)
|
||||||
numeric = vals and all(isinstance(value, int) for value in vals)
|
numeric = vals and all(isinstance(value, int) for value in vals)
|
||||||
rows.append({
|
rows.append({
|
||||||
"base": f"0x{base:x}",
|
**identities[key],
|
||||||
"population": len(vals),
|
"population": len(vals),
|
||||||
"coverage": len(vals) / len(records) if records else 0.0,
|
"coverage": len(vals) / len(records) if records else 0.0,
|
||||||
"distinct_values": len({value_key(value) for value in vals}),
|
"distinct_values": len({value_key(value) for value in vals}),
|
||||||
"min": min(vals) if numeric else None,
|
"min": min(vals) if numeric else None,
|
||||||
"max": max(vals) if numeric else None,
|
"max": max(vals) if numeric else None,
|
||||||
"common": [{"value": value, "count": count} for value, count in common],
|
"common": [{"value": value, "count": count} for value, count in common],
|
||||||
"examples": examples[base],
|
"examples": examples[key],
|
||||||
"references": 0,
|
"references": 0,
|
||||||
"reader_scripts": [],
|
"reader_scripts": [],
|
||||||
"reference_ops": [],
|
"reference_ops": [],
|
||||||
@@ -83,7 +109,9 @@ def profile_columns(data: dict) -> list[dict]:
|
|||||||
|
|
||||||
|
|
||||||
def add_direct_references(rows: list[dict], source_name: str) -> None:
|
def add_direct_references(rows: list[dict], source_name: str) -> None:
|
||||||
by_base = {int(row["base"], 16): row for row in rows}
|
by_base: dict[int, list[dict]] = collections.defaultdict(list)
|
||||||
|
for row in rows:
|
||||||
|
by_base[int(row["base"], 16)].append(row)
|
||||||
scripts: dict[int, collections.Counter] = {
|
scripts: dict[int, collections.Counter] = {
|
||||||
base: collections.Counter() for base in by_base
|
base: collections.Counter() for base in by_base
|
||||||
}
|
}
|
||||||
@@ -104,7 +132,8 @@ def add_direct_references(rows: list[dict], source_name: str) -> None:
|
|||||||
scripts[value][name] += 1
|
scripts[value][name] += 1
|
||||||
ops[value][f"{sys4load.display_label(ins.opcode)}:arg{arg_index + 1}"] += 1
|
ops[value][f"{sys4load.display_label(ins.opcode)}:arg{arg_index + 1}"] += 1
|
||||||
|
|
||||||
for base, row in by_base.items():
|
for base, base_rows in by_base.items():
|
||||||
|
for row in base_rows:
|
||||||
row["references"] = sum(scripts[base].values())
|
row["references"] = sum(scripts[base].values())
|
||||||
row["reader_scripts"] = [
|
row["reader_scripts"] = [
|
||||||
{"script": script, "count": count}
|
{"script": script, "count": count}
|
||||||
@@ -117,7 +146,7 @@ def add_direct_references(rows: list[dict], source_name: str) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def render_markdown(data: dict, rows: list[dict], limit: int) -> str:
|
def render_markdown(data: dict, rows: list[dict], limit: int) -> str:
|
||||||
ranked = sorted(rows, key=lambda row: (-row["population"], -row["references"], row["base"]))
|
ranked = sorted(rows, key=lambda row: (-row["population"], -row["references"], row["key"]))
|
||||||
shown = ranked[:limit]
|
shown = ranked[:limit]
|
||||||
lines = [
|
lines = [
|
||||||
f"# {data['table']} field profile",
|
f"# {data['table']} field profile",
|
||||||
@@ -126,10 +155,10 @@ def render_markdown(data: dict, rows: list[dict], limit: int) -> str:
|
|||||||
"> This is evidence for investigation; confirmed names live in `vm-map/globals.toml`.",
|
"> This is evidence for investigation; confirmed names live in `vm-map/globals.toml`.",
|
||||||
"",
|
"",
|
||||||
f"- records: {data['record_count']}",
|
f"- records: {data['record_count']}",
|
||||||
f"- populated global-array bases: {len(rows)}",
|
f"- populated fields: {len(rows)}",
|
||||||
f"- rows shown: {len(shown)} (ranked by record coverage, then consumer references)",
|
f"- rows shown: {len(shown)} (ranked by record coverage, then consumer references)",
|
||||||
"",
|
"",
|
||||||
"| base | populated | distinct | range | direct refs | readers | common values | examples |",
|
"| field | populated | distinct | range | direct refs | readers | common values | examples |",
|
||||||
"|---|---:|---:|---|---:|---|---|---|",
|
"|---|---:|---:|---|---:|---|---|---|",
|
||||||
]
|
]
|
||||||
for row in shown:
|
for row in shown:
|
||||||
@@ -142,7 +171,7 @@ def render_markdown(data: dict, rows: list[dict], limit: int) -> str:
|
|||||||
for entry in row["examples"][:3]
|
for entry in row["examples"][:3]
|
||||||
).replace("|", "\\|")
|
).replace("|", "\\|")
|
||||||
lines.append(
|
lines.append(
|
||||||
f"| `{row['base']}` | {row['population']}/{data['record_count']} "
|
f"| `{row['key']}` | {row['population']}/{data['record_count']} "
|
||||||
f"({row['coverage']:.0%}) | {row['distinct_values']} | {value_range} | "
|
f"({row['coverage']:.0%}) | {row['distinct_values']} | {value_range} | "
|
||||||
f"{row['references']} | {readers} | {common} | {examples} |"
|
f"{row['references']} | {readers} | {common} | {examples} |"
|
||||||
)
|
)
|
||||||
@@ -165,8 +194,12 @@ def main() -> int:
|
|||||||
"table": data["table"],
|
"table": data["table"],
|
||||||
"source": data["source"],
|
"source": data["source"],
|
||||||
"record_count": data["record_count"],
|
"record_count": data["record_count"],
|
||||||
"field_base_count": len(rows),
|
"field_column_count": len(rows),
|
||||||
"columns": sorted(rows, key=lambda row: int(row["base"], 16)),
|
"parallel_array_count": sum(row["kind"] == "parallel-array" for row in rows),
|
||||||
|
"record_column_count": sum(row["kind"] == "record-column" for row in rows),
|
||||||
|
"columns": sorted(rows, key=lambda row: (
|
||||||
|
int(row["base"], 16), row["stride"] or 0, row["column"] or 0
|
||||||
|
)),
|
||||||
}
|
}
|
||||||
markdown = render_markdown(data, rows, args.limit)
|
markdown = render_markdown(data, rows, args.limit)
|
||||||
print(markdown)
|
print(markdown)
|
||||||
|
|||||||
@@ -45,6 +45,12 @@ def test_real_name_tables() -> None:
|
|||||||
check(by_id[101]["desc"] == "HP30回復", "ITINIT item 101 keeps its description")
|
check(by_id[101]["desc"] == "HP30回復", "ITINIT item 101 keeps its description")
|
||||||
check(by_id[1]["fields"]["0x8c879"] == 10,
|
check(by_id[1]["fields"]["0x8c879"] == 10,
|
||||||
"ITINIT columns use the runtime lookup base")
|
"ITINIT columns use the runtime lookup base")
|
||||||
|
check(len({key for record in items for key in record["fields"]}) == 13,
|
||||||
|
"ITINIT has thirteen parallel-array fields")
|
||||||
|
check(len({key for record in items for key in record.get("record_fields", {})}) == 43,
|
||||||
|
"ITINIT linked row-major tables expose 43 populated columns")
|
||||||
|
check(by_id[101]["record_fields"]["0xa5301/3/0"] == 30,
|
||||||
|
"ITINIT item 101 stores HP recovery in row-major column zero")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -12,12 +12,14 @@ import init_table_profile as profile
|
|||||||
def main() -> int:
|
def main() -> int:
|
||||||
fixture = {
|
fixture = {
|
||||||
"records": [
|
"records": [
|
||||||
{"id": 1, "name": "one", "fields": {"0x10": 2, "0x20": 0}},
|
{"id": 1, "name": "one", "fields": {"0x10": 2, "0x20": 0},
|
||||||
|
"record_fields": {"0x30/3/0": 9}},
|
||||||
{"id": 3, "name": "three", "fields": {"0x10": 2}},
|
{"id": 3, "name": "three", "fields": {"0x10": 2}},
|
||||||
{"id": 7, "name": "seven", "fields": {"0x10": 5}},
|
{"id": 7, "name": "seven", "fields": {"0x10": 5},
|
||||||
|
"record_fields": {"0x30/3/2": 4}},
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
rows = {row["base"]: row for row in profile.profile_columns(fixture)}
|
rows = {row["key"]: row for row in profile.profile_columns(fixture)}
|
||||||
assert rows["0x10"]["population"] == 3
|
assert rows["0x10"]["population"] == 3
|
||||||
assert rows["0x10"]["coverage"] == 1.0
|
assert rows["0x10"]["coverage"] == 1.0
|
||||||
assert rows["0x10"]["distinct_values"] == 2
|
assert rows["0x10"]["distinct_values"] == 2
|
||||||
@@ -25,6 +27,10 @@ def main() -> int:
|
|||||||
assert rows["0x10"]["common"][0] == {"value": "2", "count": 2}
|
assert rows["0x10"]["common"][0] == {"value": "2", "count": 2}
|
||||||
assert rows["0x20"]["population"] == 1
|
assert rows["0x20"]["population"] == 1
|
||||||
assert rows["0x20"]["examples"][0]["name"] == "one"
|
assert rows["0x20"]["examples"][0]["name"] == "one"
|
||||||
|
assert rows["0x30/3/0"]["kind"] == "record-column"
|
||||||
|
assert rows["0x30/3/0"]["base"] == "0x30"
|
||||||
|
assert rows["0x30/3/0"]["stride"] == 3
|
||||||
|
assert rows["0x30/3/2"]["column"] == 2
|
||||||
print("all init_table_profile checks passed")
|
print("all init_table_profile checks passed")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|||||||
@@ -116,6 +116,86 @@ usage = "Populated for the same 12 ranged items as item_min_range. CALCSCOPE rea
|
|||||||
source = "investigation"
|
source = "investigation"
|
||||||
confidence = "high"
|
confidence = "high"
|
||||||
|
|
||||||
|
[[global]]
|
||||||
|
address = "0x8e7b9"
|
||||||
|
name = "item_character_whitelist"
|
||||||
|
category = "data-table"
|
||||||
|
type = "int[1000][5]"
|
||||||
|
value_domain = "EBINIT character ids; up to five per item"
|
||||||
|
usage = "Sparse ITINIT row-major table with stride 5. CHMENU rejects an item when column 0 is populated and the selected party slot's character id is absent from the row. Unique accessories 471/472 allow one character each, while crossover accessories 480..485 allow character ids 2, 3, and 4. Unused trailing columns remain zero."
|
||||||
|
source = "investigation"
|
||||||
|
confidence = "high"
|
||||||
|
|
||||||
|
[[global]]
|
||||||
|
address = "0x8ff29"
|
||||||
|
name = "item_sex_restriction_mask"
|
||||||
|
category = "data-table"
|
||||||
|
type = "int[1000]"
|
||||||
|
value_domain = "bit mask; shipped ITINIT populates value 4 on two items"
|
||||||
|
usage = "Sparse ITINIT equipment restriction. CHMENU tests this mask against EBINIT field 0x71eae, whose values partition male, female, and sexless units; both populated items carry bit 2 and are therefore female-only."
|
||||||
|
source = "investigation"
|
||||||
|
confidence = "med"
|
||||||
|
|
||||||
|
[[global]]
|
||||||
|
address = "0x906f9"
|
||||||
|
name = "item_attack_status_levels"
|
||||||
|
category = "data-table"
|
||||||
|
type = "int[1000][30]"
|
||||||
|
value_domain = "condition strength 1..5; zero means absent"
|
||||||
|
usage = "Sparse ITINIT row-major table consumed by USEITEM and CALCILL when applying an item's attack effects. Confirmed condition columns from item descriptions are 2 HP drain, 3 SP drain, 4 FS drain, 6 charm, 7 confusion, 8 paralysis, 9 poison, 10 water-flow, and 11 fear."
|
||||||
|
source = "investigation"
|
||||||
|
confidence = "high"
|
||||||
|
|
||||||
|
[[global]]
|
||||||
|
address = "0x97c29"
|
||||||
|
name = "item_equipped_status_levels"
|
||||||
|
category = "data-table"
|
||||||
|
type = "int[1000][30]"
|
||||||
|
value_domain = "condition strength 1..5; zero means absent"
|
||||||
|
usage = "Sparse ITINIT row-major table added to a unit's 30-column condition state by CALCREVISE. Item descriptions identify populated columns 9 poison, 11 fear, 13 regeneration, and 14 exaltation; these are passive equipped effects, distinct from item_attack_status_levels."
|
||||||
|
source = "investigation"
|
||||||
|
confidence = "high"
|
||||||
|
|
||||||
|
[[global]]
|
||||||
|
address = "0x9f541"
|
||||||
|
name = "item_stat_modifiers"
|
||||||
|
category = "data-table"
|
||||||
|
type = "int[1000][14]"
|
||||||
|
value_domain = "signed additive stat values; shipped populated values are positive"
|
||||||
|
usage = "ITINIT row-major equipment modifiers added directly to the unit's 14-column stat record by CALCREVISE. Descriptions and consumers establish columns 0 accuracy, 1 evasion, 2 physical attack, 3 physical defense, 4 magic attack, 5 magic defense, 6 speed, 7 luck, 9 capture power, 10 movement, 11 max HP, 12 max SP, and 13 max FS; column 8 remains unnamed."
|
||||||
|
source = "investigation"
|
||||||
|
confidence = "high"
|
||||||
|
|
||||||
|
[[global]]
|
||||||
|
address = "0xa2bf1"
|
||||||
|
name = "item_tuning_curve_ids"
|
||||||
|
category = "data-table"
|
||||||
|
type = "int[1000][10]"
|
||||||
|
value_domain = "curve ids 1..18; zero means the field cannot be tuned"
|
||||||
|
usage = "ITINIT row-major table selecting an equipment-growth curve for each of the ten tunable fields. TUNE, IMPROVE, DRAWTIP, and CALCREVISE combine each nonzero curve id with the item's corresponding tuning level and index the shared curve-value table at 0xab6fa. Columns align with item_stat_modifiers columns 0..9."
|
||||||
|
source = "investigation"
|
||||||
|
confidence = "high"
|
||||||
|
|
||||||
|
[[global]]
|
||||||
|
address = "0xa5301"
|
||||||
|
name = "item_resource_recovery_amounts"
|
||||||
|
category = "data-table"
|
||||||
|
type = "int[1000][3]"
|
||||||
|
value_domain = "recovery amount; zero means absent"
|
||||||
|
usage = "Sparse ITINIT row-major consumable table. USEITEM applies columns 0, 1, and 2 to the matching three-column unit resource record; descriptions prove these are HP, SP, and FS respectively (for example item 101 stores HP 30, and item 107 stores 999/99/99 for full recovery)."
|
||||||
|
source = "investigation"
|
||||||
|
confidence = "high"
|
||||||
|
|
||||||
|
[[global]]
|
||||||
|
address = "0xa62a1"
|
||||||
|
name = "item_essence_recovery_amount"
|
||||||
|
category = "data-table"
|
||||||
|
type = "int[1000]"
|
||||||
|
value_domain = "50 in the shipped table"
|
||||||
|
usage = "Sparse ITINIT consumable field. Item 106, Blood Price Healing Hand, describes essence recovery 50 and stores 50 here; USEITEM follows the dedicated essence-recovery path and scales the value before updating the selected unit."
|
||||||
|
source = "investigation"
|
||||||
|
confidence = "high"
|
||||||
|
|
||||||
[[global]]
|
[[global]]
|
||||||
address = "0x0"
|
address = "0x0"
|
||||||
name = "system_flow_request"
|
name = "system_flow_request"
|
||||||
|
|||||||
Reference in New Issue
Block a user