Model linked INIT record tables

This commit is contained in:
gamer147
2026-07-22 16:57:59 -04:00
parent d6a69ba9b4
commit 5f743275bf
10 changed files with 268 additions and 11020 deletions

File diff suppressed because it is too large Load Diff

View File

@@ -87,8 +87,8 @@ free handholds — several of which we've already built:
1. **The `*INIT` scripts are the writers, and we already extracted them.** `EBINIT`/`ITINIT`/ 1. **The `*INIT` scripts are the writers, and we already extracted them.** `EBINIT`/`ITINIT`/
`SKINIT`/`CGINIT`/`MPINIT` populate global arrays with names and data (`build/data/*.json`). `SKINIT`/`CGINIT`/`MPINIT` populate global arrays with names and data (`build/data/*.json`).
The base address `EBINIT` writes 277 unit names into *is* the unit-name table. Each JSON's The base address `EBINIT` writes 277 unit names into *is* the unit-name table. Each JSON's
`name_array_base`, `desc_array_bases`, and `field_columns` are literally global addresses we `name_array_base`, `desc_array_bases`, `field_columns`, and `record_field_columns` are literally
can label by which table wrote them. global addresses and access shapes we can label by which table wrote them.
2. **Strings anchor the string side for free.** `set-string` writes skill names to 2. **Strings anchor the string side for free.** `set-string` writes skill names to
`global-string 0x23a3…` → that array is the skill-name table. `*MES` tables likewise. `global-string 0x23a3…` → that array is the skill-name table. `*MES` tables likewise.
3. **Access shape reveals structure without names.** A global read as `base[unit*stride + col]` 3. **Access shape reveals structure without names.** A global read as `base[unit*stride + col]`
@@ -106,14 +106,14 @@ label the ~dozen hottest globals first (biggest readability payoff), grow the re
(all evidence) + `build/global-var-map.md` (labelled subset). It ingests `build/data/*.json` (all evidence) + `build/global-var-map.md` (labelled subset). It ingests `build/data/*.json`
(name/desc/field bases), scans the 481-script corpus for each global's **access shape** (name/desc/field bases), scans the 481-script corpus for each global's **access shape**
(2D-table base + stride, 1D-array base, row-index, scalar), and ranks "current entity" index (2D-table base + stride, 1D-array base, row-index, scalar), and ranks "current entity" index
pointers by purity. **Current result: 15,918 of 49,386 distinct globals labelled** pointers by purity. **Current result: 4,960 of 41,611 distinct globals labelled**
| kind | count | example | | kind | count | example |
|---|---|---| |---|---|---|
| string tables (names/descs/messages) | 3,206 | `0x23a2` = skill-name lookup base | | string tables (names/descs/messages) | 3,206 | `0x23a2` = skill-name lookup base |
| per-entity data-field arrays (from *INIT) | 12,311 | dense = shared fields, `?` = sparse per-entity | | per-entity data-field arrays (from *INIT) | 1,353 | dense = shared fields, `?` = sparse per-entity |
| row-major record tables (from access shape) | 122 | `0x52383` = record-table[stride 30] | | row-major record tables (from access shape) | 122 | `0x52383` = record-table[stride 30] |
| 1D arrays | 307 | | | 1D arrays | 253 | |
| index / "current entity" pointers | 26 | `0x152616` (purity 0.51), `0xeff75` (0.95) | | index / "current entity" pointers | 26 | `0x152616` (purity 0.51), `0xeff75` (0.95) |
**Validated against `RECOVER`:** the map independently reproduces its hand-traced layout — **Validated against `RECOVER`:** the map independently reproduces its hand-traced layout —
@@ -148,15 +148,30 @@ Semantic recovery is an evidence ladder, cheapest and strongest first:
promoting guesses. Use dynamic observation only for fields that remain ambiguous after static consumers. promoting guesses. Use dynamic observation only for fields that remain ambiguous after static consumers.
`tools/init_table_profile.py ITINIT --build` materializes steps 12 in `tools/init_table_profile.py ITINIT --build` materializes steps 12 in
`build/data/ITINIT-field-profile.{json,md}`. The first pass names eleven stable ITINIT arrays: catalog sort `build/data/ITINIT-field-profile.{json,md}`. The initial pass names thirteen parallel arrays: catalog sort
key, random-item tier, item category, icon id, shared ITMES handler id, attack and defense elements, weapon key, random-item tier, item category, icon id, shared ITMES handler id, attack and defense elements, weapon
class, granted skill id, and minimum/maximum range. The strongest joins are independently human-readable: class, granted skill id, minimum/maximum range, essence recovery, and an equipment sex mask. The strongest
attack/defense values index AFINIT's Japanese attribute strings, granted-skill values resolve to SKINIT, all joins are independently human-readable: attack/defense values index AFINIT's Japanese attribute strings,
handler values resolve to ITMES.BIN, and every min/max-range record says `range 2` in its item description. granted-skill values resolve to SKINIT, all handler values resolve to ITMES.BIN, and every min/max-range
record says `range 2` in its item description.
The remaining 764 ITINIT write bases are not automatically 764 item fields. Most occur once because ITINIT The apparent 764 additional ITINIT field bases were a structural artifact, not 764 sparse arrays. For each
also initializes linked effect/stat arrays while defining an item. Treat the profile's `population` and write, subtracting `item_id * stride` and comparing the destination with corpus-observed `lookup-array-2d`
consumer evidence as a classification aid before assigning table ownership. consumers assigns all 764 writes unambiguously to six row-major tables and 43 populated columns:
| base | stride | populated writes | semantic role |
|---|---:|---:|---|
| `0x8e7b9` | 5 | 20 | character-id equipment whitelist |
| `0x906f9` | 30 | 36 | attack-inflicted condition levels |
| `0x97c29` | 30 | 11 | equipped/passive condition levels |
| `0x9f541` | 14 | 301 | additive equipment stat modifiers |
| `0xa2bf1` | 10 | 379 | per-stat tuning curve ids |
| `0xa5301` | 3 | 17 | HP/SP/FS recovery amounts |
`extract_init.py` now records these as `record_fields["base/stride/column"]` rather than inventing a
one-off `fields` base for every row. Applying the same rule also exposes 17 linked SKINIT columns and 83
linked EBINIT columns. This correction reduces the auto map's false INIT-field labels from 12,311 to 1,353;
the raw write data was valid, but its former ownership model was not.
### The curated registry — `vm-map/globals.toml` (2026-07-07) ### The curated registry — `vm-map/globals.toml` (2026-07-07)
@@ -202,8 +217,8 @@ are *not* story flags — the miner over-tags them; they are recategorized `unkn
The v1 map labels *shapes and tables*; the next increments add *meaning*, cheapest first: The v1 map labels *shapes and tables*; the next increments add *meaning*, cheapest first:
1. **Continue INIT semantics by evidence density.** Finish ITINIT's repeatedly populated fields and linked 1. **Continue INIT semantics by evidence density.** Resolve ITINIT stat column 8 and the remaining condition
effect arrays, then run the same profiler on SKINIT and EBINIT. Add explicit foreign-key joins (item → enum columns, then run the same profiler on SKINIT and EBINIT. Add explicit foreign-key joins (item →
skill, unit → attack/skill) once the target table ids are confirmed. Do not infer meaning from column skill, unit → attack/skill) once the target table ids are confirmed. Do not infer meaning from column
position alone. position alone.
2. **Fold in the `*MES` message-table writers** (`ITMES`, `SKMES`, `VIMES`, …) and any other 2. **Fold in the `*MES` message-table writers** (`ITMES`, `SKMES`, `VIMES`, …) and any other

View File

@@ -714,10 +714,12 @@ gate on the one-player-attack acceptance path.
extractor itself was audited. It now preserves sparse one-based ids and corrects ITINIT from 189 malformed extractor itself was audited. It now preserves sparse one-based ids and corrects ITINIT from 189 malformed
records to 287 items (plus SKINIT 129→131 skills); regression checks cover the real tables. A reusable field records to 287 items (plus SKINIT 129→131 skills); regression checks cover the real tables. A reusable field
profiler reports distributions, examples, and direct script/opcode consumers. The ITINIT pilot has curated profiler reports distributions, examples, and direct script/opcode consumers. The ITINIT pilot has curated
eleven high-confidence arrays in `vm-map/globals.toml`: sort key, random tier, category, icon, behavior thirteen parallel arrays plus six linked row-major tables in `vm-map/globals.toml`. The latter account for
handler, attack/defense element, weapon class, granted skill, and min/max range. Next semantic work should all 764 writes previously misidentified as separate sparse item fields: character restrictions, attack and
classify the remaining repeatedly populated ITINIT bases, then apply the same workflow to SKINIT and EBINIT; equipped condition levels, stat modifiers, tuning curves, and HP/SP/FS recovery. The extractor now emits
STINIT's bespoke parser remains a separate extraction task. these as `record_fields[base/stride/column]`, which also exposes 17 linked SKINIT and 83 linked EBINIT
columns. Next semantic work should profile those two tables and resolve ITINIT's remaining unnamed stat and
condition columns; STINIT's bespoke parser remains a separate extraction task.
Once the natural spine and first gameplay loop are trustworthy, broaden in independent tracks: Once the natural spine and first gameplay loop are trustworthy, broaden in independent tracks:

View File

@@ -60,6 +60,11 @@ items 1101 into the first record and treated consecutive item names as descri
and counts are indexed in `build/data/README.md`; field semantics are curated in `vm-map/globals.toml` and and counts are indexed in `build/data/README.md`; field semantics are curated in `vm-map/globals.toml` and
described by the workflow in `docs/name-resolution.md`. described by the workflow in `docs/name-resolution.md`.
The same audit found that linked row-major writes must not be normalized as independent parallel arrays.
Corpus `lookup-array-2d` bases and strides assign every such ITINIT write unambiguously to six tables (43
populated columns), while SKINIT and EBINIT expose 17 and 83 linked columns respectively. Generated records
now keep these under `record_fields[base/stride/column]`; `fields` contains only genuine parallel arrays.
### Message/string tables (`*MES`) ### Message/string tables (`*MES`)
`ITMES` (64 KB — item text), `VIMES` (43 KB), `EIMES` (37 KB), `SKMES` (31 KB — skill `ITMES` (64 KB — item text), `VIMES` (43 KB), `EIMES` (37 KB), `SKMES` (31 KB — skill
text), `CIMES` (15 KB), `MAMES`, `INFOMES`, `MES` — where most translatable text text), `CIMES` (15 KB), `MAMES`, `INFOMES`, `MES` — where most translatable text

View File

@@ -54,8 +54,8 @@ All opcode knowledge (ABI, semantics, provenance, `depends_on`) is hand-edited *
| Tool | Purpose | Run | Reads → Writes | | Tool | Purpose | Run | Reads → Writes |
|---|---|---|---| |---|---|---|---|
| `extract_phase2.py` | Batch: disassembly + text corpora for every script. | `extract_phase2.py` | corpus → `build/disasm/*.asm`, `build/text/{dialogue.jsonl,strings.jsonl,*.strings.txt}`, `build/manifest.json` | | `extract_phase2.py` | Batch: disassembly + text corpora for every script. | `extract_phase2.py` | corpus → `build/disasm/*.asm`, `build/text/{dialogue.jsonl,strings.jsonl,*.strings.txt}`, `build/manifest.json` |
| `extract_init.py` | Parse a `*INIT` data table (auto-detects name / numeric / footer shape). Name tables infer their reserved record span, preserve sparse one-based runtime ids, and distinguish lookup bases from first written cells. Also refreshes the generated data index. | `extract_init.py <TABLE> [OUTNAME] [--mode …]` | `<TABLE>.BIN``build/data/<OUTNAME>.json`, `build/data/README.md` | | `extract_init.py` | Parse a `*INIT` data table (auto-detects name / numeric / footer shape). Name tables infer their reserved record span, preserve sparse one-based runtime ids, distinguish lookup bases from first written cells, and use corpus-observed 2D consumers to separate parallel `fields` from linked `record_fields` keyed `base/stride/column`. Also refreshes the generated data index. | `extract_init.py <TABLE> [OUTNAME] [--mode …]` | `<TABLE>.BIN``build/data/<OUTNAME>.json`, `build/data/README.md` |
| `init_table_profile.py` | Build the static investigation surface for an extracted name/numeric table: per-base population/value distributions, representative records, and direct opcode/script consumers. Findings are evidence only; confirmed meanings go in `vm-map/globals.toml`. | `init_table_profile.py <TABLE> [--build] [--limit N]` | `build/data/<TABLE>.json` + corpus → stdout; with `--build`, `build/data/<TABLE>-field-profile.{json,md}` | | `init_table_profile.py` | Build the static investigation surface for an extracted name/numeric table: per-array and per-record-column population/value distributions, representative records, and direct opcode/script consumers. Findings are evidence only; confirmed meanings go in `vm-map/globals.toml`. | `init_table_profile.py <TABLE> [--build] [--limit N]` | `build/data/<TABLE>.json` + corpus → stdout; with `--build`, `build/data/<TABLE>-field-profile.{json,md}` |
| `test_extract_init.py`, `test_init_table_profile.py` | Regression checks for sparse one-based INIT extraction and field profiling. | run each directly | — | | `test_extract_init.py`, `test_init_table_profile.py` | Regression checks for sparse one-based INIT extraction and field profiling. | run each directly | — |
| `global_map.py` | Build the partial global-variable name map from static evidence. | `global_map.py` | corpus + `build/data/``build/global-var-map.{json,md}` | | `global_map.py` | Build the partial global-variable name map from static evidence. | `global_map.py` | corpus + `build/data/``build/global-var-map.{json,md}` |

View File

@@ -20,6 +20,7 @@ Usage: py -3.11 -X utf8 tools/extract_init.py <TABLE> [OUTNAME] [--mode name|num
from __future__ import annotations from __future__ import annotations
import json import json
import sys import sys
from functools import cache
from pathlib import Path from pathlib import Path
HERE = Path(__file__).resolve().parent HERE = Path(__file__).resolve().parent
@@ -89,6 +90,44 @@ def _infer_record_span(string_addrs):
return max(counts, key=lambda delta: (counts[delta], delta)) return max(counts, key=lambda delta: (counts[delta], delta))
@cache
def _known_record_tables():
"""Return corpus-observed (base, stride) pairs used by lookup-array-2d.
INIT scripts often populate linked row-major tables while defining an
entity. Treating every such write as `destination - entity_id` invents a
different one-off parallel column for every row. Consumer bytecode gives
us the unambiguous table base and stride instead.
"""
tables = set()
for path in paths.scripts().values():
try:
script = sys4load.load(path)
except sys4load.Sys4Error:
continue
for ins in script.instructions:
if (sys4load.display_label(ins.opcode) == "lookup-array-2d"
and len(ins.args) >= 5
and ins.args[1][0] in (T_GLOBAL_INT, 6)
and ins.args[3][0] == T_IMM
and ins.args[3][1] > 0):
tables.add((ins.args[1][1], ins.args[3][1]))
return tuple(sorted(tables))
def _record_table_cell(destination, record_id):
matches = []
for base, stride in _known_record_tables():
column = destination - (base + record_id * stride)
if 0 <= column < stride:
matches.append((base, stride, column))
if len(matches) > 1:
raise ValueError(
f"ambiguous record-table destination 0x{destination:x} for id {record_id}: {matches}"
)
return matches[0] if matches else None
def extract_name(scr): def extract_name(scr):
string_addrs = [ string_addrs = [
ins.args[0][1] ins.args[0][1]
@@ -112,17 +151,31 @@ def extract_name(scr):
# boundary: ITINIT begins with 101 consecutive name-only records, # boundary: ITINIT begins with 101 consecutive name-only records,
# which the old heuristic collapsed into item zero. # which the old heuristic collapsed into item zero.
if name_write_base <= addr < name_write_base + record_span: if name_write_base <= addr < name_write_base + record_span:
cur = {"id": addr - name_base, "name": txt, "fields": {}} cur = {"id": addr - name_base, "name": txt, "fields": {}, "record_fields": {}}
records.append(cur); desc_slot = 0 records.append(cur); desc_slot = 0
elif cur is not None: elif cur is not None:
key = "desc" if desc_slot == 0 else f"desc{desc_slot}" key = "desc" if desc_slot == 0 else f"desc{desc_slot}"
cur[key] = txt; desc_bases.setdefault(key, addr - cur["id"]); desc_slot += 1 cur[key] = txt; desc_bases.setdefault(key, addr - cur["id"]); desc_slot += 1
elif ins.opcode == MOV and cur is not None and ins.args and ins.args[0][0] == T_GLOBAL_INT: elif ins.opcode == MOV and cur is not None and ins.args and ins.args[0][0] == T_GLOBAL_INT:
cur["fields"][f"0x{ins.args[0][1] - cur['id']:x}"] = _val(ins.args[1]) destination = ins.args[0][1]
cell = _record_table_cell(destination, cur["id"])
if cell is None:
cur["fields"][f"0x{destination - cur['id']:x}"] = _val(ins.args[1])
else:
base, stride, column = cell
cur["record_fields"][f"0x{base:x}/{stride}/{column}"] = _val(ins.args[1])
for record in records:
if not record["record_fields"]:
del record["record_fields"]
record_columns = sorted(
{key for record in records for key in record.get("record_fields", {})},
key=lambda key: tuple(int(part, 0) for part in key.split("/")),
)
return records, {"name_array_base": f"0x{name_base:x}", return records, {"name_array_base": f"0x{name_base:x}",
"name_write_base": f"0x{name_write_base:x}", "name_write_base": f"0x{name_write_base:x}",
"first_record_id": first_record_id, "first_record_id": first_record_id,
"record_span": record_span, "record_span": record_span,
"record_field_columns": record_columns,
"desc_array_bases": {k: f"0x{v:x}" for k, v in sorted(desc_bases.items())}} "desc_array_bases": {k: f"0x{v:x}" for k, v in sorted(desc_bases.items())}}
@@ -201,17 +254,23 @@ def write_data_index(data_dir: Path) -> None:
"bases remain available in every record; confirmed field meanings live in", "bases remain available in every record; confirmed field meanings live in",
"`vm-map/globals.toml` and the generated `docs/global-reference.md`.", "`vm-map/globals.toml` and the generated `docs/global-reference.md`.",
"", "",
"| file | mode | records | field bases |", "| file | mode | records | array fields | record columns |",
"|---|---|---:|---:|", "|---|---|---:|---:|---:|",
] ]
for filename, data in tables: for filename, data in tables:
columns = len(data.get("field_columns") or []) columns = len(data.get("field_columns") or [])
lines.append(f"| `{filename}` | {data['mode']} | {data['record_count']} | {columns} |") record_columns = len(data.get("record_field_columns") or [])
lines.append(
f"| `{filename}` | {data['mode']} | {data['record_count']} | "
f"{columns} | {record_columns} |"
)
lines += [ lines += [
"", "",
"Name-mode tables expose one-based runtime `id` values, the lookup `name_array_base`,", "Name-mode tables expose one-based runtime `id` values, the lookup `name_array_base`,",
"the first populated `name_write_base`, and the reserved `record_span`. Fields are keyed", "the first populated `name_write_base`, and the reserved `record_span`. Fields are keyed",
"by the runtime lookup base used by `lookup-array`, not merely the first written cell.", "by the runtime lookup base used by `lookup-array`, not merely the first written cell.",
"Linked row-major fields are stored separately in `record_fields`, keyed as",
"`base/stride/column` from corpus-observed `lookup-array-2d` consumers.",
"", "",
"Use `tools/init_table_profile.py <TABLE> --build` to generate value/population and", "Use `tools/init_table_profile.py <TABLE> --build` to generate value/population and",
"direct-consumer evidence. `STINIT` still requires a bespoke mixed numeric/string parser.", "direct-consumer evidence. `STINIT` still requires a bespoke mixed numeric/string parser.",

View File

@@ -49,32 +49,58 @@ def value_key(value) -> str:
def profile_columns(data: dict) -> list[dict]: def profile_columns(data: dict) -> list[dict]:
records = data["records"] records = data["records"]
values: dict[int, list] = collections.defaultdict(list) values: dict[str, list] = collections.defaultdict(list)
examples: dict[int, list[dict]] = collections.defaultdict(list) examples: dict[str, list[dict]] = collections.defaultdict(list)
identities: dict[str, dict] = {}
for record in records: for record in records:
for address, value in record.get("fields", {}).items(): for address, value in record.get("fields", {}).items():
base = int(address, 16) base = int(address, 16)
values[base].append(value) key = f"0x{base:x}"
if len(examples[base]) < 5: identities[key] = {
examples[base].append({ "key": key, "kind": "parallel-array", "base": key,
"stride": None, "column": None,
}
values[key].append(value)
if len(examples[key]) < 5:
examples[key].append({
"id": record["id"],
"name": record.get("name", ""),
"value": value,
})
for key, value in record.get("record_fields", {}).items():
base_text, stride_text, column_text = key.split("/")
base = int(base_text, 16)
stride = int(stride_text)
column = int(column_text)
normalized_key = f"0x{base:x}/{stride}/{column}"
identities[normalized_key] = {
"key": normalized_key,
"kind": "record-column",
"base": f"0x{base:x}",
"stride": stride,
"column": column,
}
values[normalized_key].append(value)
if len(examples[normalized_key]) < 5:
examples[normalized_key].append({
"id": record["id"], "id": record["id"],
"name": record.get("name", ""), "name": record.get("name", ""),
"value": value, "value": value,
}) })
rows = [] rows = []
for base, vals in values.items(): for key, vals in values.items():
common = collections.Counter(value_key(value) for value in vals).most_common(6) common = collections.Counter(value_key(value) for value in vals).most_common(6)
numeric = vals and all(isinstance(value, int) for value in vals) numeric = vals and all(isinstance(value, int) for value in vals)
rows.append({ rows.append({
"base": f"0x{base:x}", **identities[key],
"population": len(vals), "population": len(vals),
"coverage": len(vals) / len(records) if records else 0.0, "coverage": len(vals) / len(records) if records else 0.0,
"distinct_values": len({value_key(value) for value in vals}), "distinct_values": len({value_key(value) for value in vals}),
"min": min(vals) if numeric else None, "min": min(vals) if numeric else None,
"max": max(vals) if numeric else None, "max": max(vals) if numeric else None,
"common": [{"value": value, "count": count} for value, count in common], "common": [{"value": value, "count": count} for value, count in common],
"examples": examples[base], "examples": examples[key],
"references": 0, "references": 0,
"reader_scripts": [], "reader_scripts": [],
"reference_ops": [], "reference_ops": [],
@@ -83,7 +109,9 @@ def profile_columns(data: dict) -> list[dict]:
def add_direct_references(rows: list[dict], source_name: str) -> None: def add_direct_references(rows: list[dict], source_name: str) -> None:
by_base = {int(row["base"], 16): row for row in rows} by_base: dict[int, list[dict]] = collections.defaultdict(list)
for row in rows:
by_base[int(row["base"], 16)].append(row)
scripts: dict[int, collections.Counter] = { scripts: dict[int, collections.Counter] = {
base: collections.Counter() for base in by_base base: collections.Counter() for base in by_base
} }
@@ -104,20 +132,21 @@ def add_direct_references(rows: list[dict], source_name: str) -> None:
scripts[value][name] += 1 scripts[value][name] += 1
ops[value][f"{sys4load.display_label(ins.opcode)}:arg{arg_index + 1}"] += 1 ops[value][f"{sys4load.display_label(ins.opcode)}:arg{arg_index + 1}"] += 1
for base, row in by_base.items(): for base, base_rows in by_base.items():
row["references"] = sum(scripts[base].values()) for row in base_rows:
row["reader_scripts"] = [ row["references"] = sum(scripts[base].values())
{"script": script, "count": count} row["reader_scripts"] = [
for script, count in scripts[base].most_common() {"script": script, "count": count}
] for script, count in scripts[base].most_common()
row["reference_ops"] = [ ]
{"operation": operation, "count": count} row["reference_ops"] = [
for operation, count in ops[base].most_common() {"operation": operation, "count": count}
] for operation, count in ops[base].most_common()
]
def render_markdown(data: dict, rows: list[dict], limit: int) -> str: def render_markdown(data: dict, rows: list[dict], limit: int) -> str:
ranked = sorted(rows, key=lambda row: (-row["population"], -row["references"], row["base"])) ranked = sorted(rows, key=lambda row: (-row["population"], -row["references"], row["key"]))
shown = ranked[:limit] shown = ranked[:limit]
lines = [ lines = [
f"# {data['table']} field profile", f"# {data['table']} field profile",
@@ -126,10 +155,10 @@ def render_markdown(data: dict, rows: list[dict], limit: int) -> str:
"> This is evidence for investigation; confirmed names live in `vm-map/globals.toml`.", "> This is evidence for investigation; confirmed names live in `vm-map/globals.toml`.",
"", "",
f"- records: {data['record_count']}", f"- records: {data['record_count']}",
f"- populated global-array bases: {len(rows)}", f"- populated fields: {len(rows)}",
f"- rows shown: {len(shown)} (ranked by record coverage, then consumer references)", f"- rows shown: {len(shown)} (ranked by record coverage, then consumer references)",
"", "",
"| base | populated | distinct | range | direct refs | readers | common values | examples |", "| field | populated | distinct | range | direct refs | readers | common values | examples |",
"|---|---:|---:|---|---:|---|---|---|", "|---|---:|---:|---|---:|---|---|---|",
] ]
for row in shown: for row in shown:
@@ -142,7 +171,7 @@ def render_markdown(data: dict, rows: list[dict], limit: int) -> str:
for entry in row["examples"][:3] for entry in row["examples"][:3]
).replace("|", "\\|") ).replace("|", "\\|")
lines.append( lines.append(
f"| `{row['base']}` | {row['population']}/{data['record_count']} " f"| `{row['key']}` | {row['population']}/{data['record_count']} "
f"({row['coverage']:.0%}) | {row['distinct_values']} | {value_range} | " f"({row['coverage']:.0%}) | {row['distinct_values']} | {value_range} | "
f"{row['references']} | {readers} | {common} | {examples} |" f"{row['references']} | {readers} | {common} | {examples} |"
) )
@@ -165,8 +194,12 @@ def main() -> int:
"table": data["table"], "table": data["table"],
"source": data["source"], "source": data["source"],
"record_count": data["record_count"], "record_count": data["record_count"],
"field_base_count": len(rows), "field_column_count": len(rows),
"columns": sorted(rows, key=lambda row: int(row["base"], 16)), "parallel_array_count": sum(row["kind"] == "parallel-array" for row in rows),
"record_column_count": sum(row["kind"] == "record-column" for row in rows),
"columns": sorted(rows, key=lambda row: (
int(row["base"], 16), row["stride"] or 0, row["column"] or 0
)),
} }
markdown = render_markdown(data, rows, args.limit) markdown = render_markdown(data, rows, args.limit)
print(markdown) print(markdown)

View File

@@ -45,6 +45,12 @@ def test_real_name_tables() -> None:
check(by_id[101]["desc"] == "HP30回復", "ITINIT item 101 keeps its description") check(by_id[101]["desc"] == "HP30回復", "ITINIT item 101 keeps its description")
check(by_id[1]["fields"]["0x8c879"] == 10, check(by_id[1]["fields"]["0x8c879"] == 10,
"ITINIT columns use the runtime lookup base") "ITINIT columns use the runtime lookup base")
check(len({key for record in items for key in record["fields"]}) == 13,
"ITINIT has thirteen parallel-array fields")
check(len({key for record in items for key in record.get("record_fields", {})}) == 43,
"ITINIT linked row-major tables expose 43 populated columns")
check(by_id[101]["record_fields"]["0xa5301/3/0"] == 30,
"ITINIT item 101 stores HP recovery in row-major column zero")
if __name__ == "__main__": if __name__ == "__main__":

View File

@@ -12,12 +12,14 @@ import init_table_profile as profile
def main() -> int: def main() -> int:
fixture = { fixture = {
"records": [ "records": [
{"id": 1, "name": "one", "fields": {"0x10": 2, "0x20": 0}}, {"id": 1, "name": "one", "fields": {"0x10": 2, "0x20": 0},
"record_fields": {"0x30/3/0": 9}},
{"id": 3, "name": "three", "fields": {"0x10": 2}}, {"id": 3, "name": "three", "fields": {"0x10": 2}},
{"id": 7, "name": "seven", "fields": {"0x10": 5}}, {"id": 7, "name": "seven", "fields": {"0x10": 5},
"record_fields": {"0x30/3/2": 4}},
] ]
} }
rows = {row["base"]: row for row in profile.profile_columns(fixture)} rows = {row["key"]: row for row in profile.profile_columns(fixture)}
assert rows["0x10"]["population"] == 3 assert rows["0x10"]["population"] == 3
assert rows["0x10"]["coverage"] == 1.0 assert rows["0x10"]["coverage"] == 1.0
assert rows["0x10"]["distinct_values"] == 2 assert rows["0x10"]["distinct_values"] == 2
@@ -25,6 +27,10 @@ def main() -> int:
assert rows["0x10"]["common"][0] == {"value": "2", "count": 2} assert rows["0x10"]["common"][0] == {"value": "2", "count": 2}
assert rows["0x20"]["population"] == 1 assert rows["0x20"]["population"] == 1
assert rows["0x20"]["examples"][0]["name"] == "one" assert rows["0x20"]["examples"][0]["name"] == "one"
assert rows["0x30/3/0"]["kind"] == "record-column"
assert rows["0x30/3/0"]["base"] == "0x30"
assert rows["0x30/3/0"]["stride"] == 3
assert rows["0x30/3/2"]["column"] == 2
print("all init_table_profile checks passed") print("all init_table_profile checks passed")
return 0 return 0

View File

@@ -116,6 +116,86 @@ usage = "Populated for the same 12 ranged items as item_min_range. CALCSCOPE rea
source = "investigation" source = "investigation"
confidence = "high" confidence = "high"
[[global]]
address = "0x8e7b9"
name = "item_character_whitelist"
category = "data-table"
type = "int[1000][5]"
value_domain = "EBINIT character ids; up to five per item"
usage = "Sparse ITINIT row-major table with stride 5. CHMENU rejects an item when column 0 is populated and the selected party slot's character id is absent from the row. Unique accessories 471/472 allow one character each, while crossover accessories 480..485 allow character ids 2, 3, and 4. Unused trailing columns remain zero."
source = "investigation"
confidence = "high"
[[global]]
address = "0x8ff29"
name = "item_sex_restriction_mask"
category = "data-table"
type = "int[1000]"
value_domain = "bit mask; shipped ITINIT populates value 4 on two items"
usage = "Sparse ITINIT equipment restriction. CHMENU tests this mask against EBINIT field 0x71eae, whose values partition male, female, and sexless units; both populated items carry bit 2 and are therefore female-only."
source = "investigation"
confidence = "med"
[[global]]
address = "0x906f9"
name = "item_attack_status_levels"
category = "data-table"
type = "int[1000][30]"
value_domain = "condition strength 1..5; zero means absent"
usage = "Sparse ITINIT row-major table consumed by USEITEM and CALCILL when applying an item's attack effects. Confirmed condition columns from item descriptions are 2 HP drain, 3 SP drain, 4 FS drain, 6 charm, 7 confusion, 8 paralysis, 9 poison, 10 water-flow, and 11 fear."
source = "investigation"
confidence = "high"
[[global]]
address = "0x97c29"
name = "item_equipped_status_levels"
category = "data-table"
type = "int[1000][30]"
value_domain = "condition strength 1..5; zero means absent"
usage = "Sparse ITINIT row-major table added to a unit's 30-column condition state by CALCREVISE. Item descriptions identify populated columns 9 poison, 11 fear, 13 regeneration, and 14 exaltation; these are passive equipped effects, distinct from item_attack_status_levels."
source = "investigation"
confidence = "high"
[[global]]
address = "0x9f541"
name = "item_stat_modifiers"
category = "data-table"
type = "int[1000][14]"
value_domain = "signed additive stat values; shipped populated values are positive"
usage = "ITINIT row-major equipment modifiers added directly to the unit's 14-column stat record by CALCREVISE. Descriptions and consumers establish columns 0 accuracy, 1 evasion, 2 physical attack, 3 physical defense, 4 magic attack, 5 magic defense, 6 speed, 7 luck, 9 capture power, 10 movement, 11 max HP, 12 max SP, and 13 max FS; column 8 remains unnamed."
source = "investigation"
confidence = "high"
[[global]]
address = "0xa2bf1"
name = "item_tuning_curve_ids"
category = "data-table"
type = "int[1000][10]"
value_domain = "curve ids 1..18; zero means the field cannot be tuned"
usage = "ITINIT row-major table selecting an equipment-growth curve for each of the ten tunable fields. TUNE, IMPROVE, DRAWTIP, and CALCREVISE combine each nonzero curve id with the item's corresponding tuning level and index the shared curve-value table at 0xab6fa. Columns align with item_stat_modifiers columns 0..9."
source = "investigation"
confidence = "high"
[[global]]
address = "0xa5301"
name = "item_resource_recovery_amounts"
category = "data-table"
type = "int[1000][3]"
value_domain = "recovery amount; zero means absent"
usage = "Sparse ITINIT row-major consumable table. USEITEM applies columns 0, 1, and 2 to the matching three-column unit resource record; descriptions prove these are HP, SP, and FS respectively (for example item 101 stores HP 30, and item 107 stores 999/99/99 for full recovery)."
source = "investigation"
confidence = "high"
[[global]]
address = "0xa62a1"
name = "item_essence_recovery_amount"
category = "data-table"
type = "int[1000]"
value_domain = "50 in the shipped table"
usage = "Sparse ITINIT consumable field. Item 106, Blood Price Healing Hand, describes essence recovery 50 and stores 50 here; USEITEM follows the dedicated essence-recovery path and scales the value before updating the selected unit."
source = "investigation"
confidence = "high"
[[global]] [[global]]
address = "0x0" address = "0x0"
name = "system_flow_request" name = "system_flow_request"