1048 lines
47 KiB
Markdown
1048 lines
47 KiB
Markdown
# Globals Registry + Story-Flag Miner Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** Build a hand-curated, single-source semantic registry for VM global variables (`vm-map/globals.toml`), generated into machine + human views and wired into `sys4load` labeling, plus a static miner that discovers story-state flags and proposes skeleton entries.
|
||
|
||
**Architecture:** Mirror the existing opcode-reference toolchain. `vm-map/globals.toml` is the only hand-edited source (like `opcodes.toml`). `tools/globals_build.py` merges curated entries *over* the auto-generated shape map (`build/global-var-map.json`) into `build/globals.json` + generated `docs/global-reference.md`. `tools/story_flags.py` statically scans the 481-script corpus for globals that feed branch conditions, emits ranked candidates, and can bootstrap skeletons into `globals.toml`. `sys4load` switches its operand-label source from the raw auto map to the merged `build/globals.json`.
|
||
|
||
**Tech Stack:** Python 3.11 (`py -3.11 -X utf8`), stdlib `tomllib`/`json`/`argparse`/`re`. Reuses `tools/paths.py`, `tools/sys4load.py`, `tools/age_opcodes.py`. Standalone test scripts (no pytest), matching `tools/test_opcodes.py`.
|
||
|
||
## Global Constraints
|
||
|
||
- Run every tool as `py -3.11 -X utf8 tools/<name>.py …` (the `-X utf8` is mandatory on Windows for cp932/Shift-JIS text). Source text is cp932; everything we generate is UTF-8.
|
||
- Tools never hard-code paths — always `import paths` (`tools/paths.py`) and derive from it.
|
||
- Never hand-edit a generated file. Generated files carry a "DO NOT EDIT — generated from …" header. `build/globals.json` and `docs/global-reference.md` are generated from `vm-map/globals.toml`; `build/global-var-map.json` is generated by `tools/global_map.py` (do not edit either).
|
||
- `tools/age_opcodes.py` is pristine upstream Kelebek data — never edit it.
|
||
- Provenance vocabulary is fixed: `category` ∈ {`story-flag`, `index-pointer`, `data-table`, `string-table`, `ui-toggle`, `choice-output`, `counter`, `unknown`}; `source` ∈ {`investigation`, `harness`, `inference`, `auto-shape`}; `confidence` ∈ {`low`, `med`, `high`}. An `auto-shape` source may never claim `high` confidence.
|
||
- Git repo root is `age-reimpl/`. Commit from there. End commit messages with the `Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>` trailer.
|
||
- Tests are standalone scripts using a `check(cond, msg)` helper that records failures and exits nonzero if any fail (pattern: `tools/test_opcodes.py`). Run: `py -3.11 -X utf8 tools/test_globals.py`.
|
||
|
||
**Reference facts (verified against the corpus, 2026-07-07):**
|
||
- Comparison opcodes (result→arg0, operands at arg indices 1 & 2): `0x5a eq`, `0x5b ne`, `0x5c lt`, `0x5d lte`, `0x5e gr`, `0x5f gre`. Logical (operands at 1 & 2): `0x56 and`, `0x57 or`. `0xa0 jcc` reads its condition at arg index 0 (args 1 & 2 are jump targets). `0x55 mov` writes arg0.
|
||
- Operand shape: `Instruction.args` is a list of `(atype, value)` tuples. Global atypes = {3,4,5,6,8} (int/float/string/ptr/string-ptr). Immediate atype = 0.
|
||
- `sys4load.load(path)` → object with `.instructions`; each `Instruction` has `.opcode`, `.label`, `.args`, `.offset`.
|
||
- `build/global-var-map.json` top-level keys: `globals` (dict `"0xADDR"` → `{type, uses, roles, stride_candidates, label, confidence, kind, table}`) and `current_entity_index_candidates` (list of `{addr, used_as_row_index, also_scalar, purity}`).
|
||
- Corpus scan of the two known flags confirms the heuristic: `0x3234` (chapter) → scene-reach 0, total branch-reads exist, compared consts `{1..8}`, writer `FIELD.BIN`; `0xa57` (Lily form A) → scene-reach 78, compared const `{1}`, no `mov` writer. **The miner must track total-reach (not only scene-reach) and must not require a static writer.**
|
||
|
||
---
|
||
|
||
### Task 1: `globals.toml` source + loader + lint
|
||
|
||
Create the hand-edited registry with first-pass curated content, plus the load/lint core of `globals_build.py`.
|
||
|
||
**Files:**
|
||
- Create: `vm-map/globals.toml`
|
||
- Create: `tools/globals_build.py`
|
||
- Create: `tools/test_globals.py`
|
||
|
||
**Interfaces:**
|
||
- Produces (consumed by Tasks 2, 4, 5):
|
||
- `globals_build.load_toml(path: Path) -> tuple[dict[int, dict], dict]` — returns `(entries_by_addr, meta)`; each entry is the raw TOML table dict with an added `_addr: int` key.
|
||
- `globals_build.lint(entries: dict[int, dict], all_addrs: set[int]) -> tuple[list[str], list[str]]` — `(errors, warnings)`. `all_addrs` is the set of addresses valid for `depends_on` resolution (curated ∪ auto).
|
||
- Module constants `CATEGORIES: set[str]`, `SOURCES: set[str]`, `CONFIDENCE: dict[str,int]`.
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
Create `tools/test_globals.py`:
|
||
|
||
```python
|
||
#!/usr/bin/env python3
|
||
"""Standalone tests for the globals registry tooling. Run: py -3.11 -X utf8 tools/test_globals.py"""
|
||
import os, sys
|
||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||
import paths
|
||
import globals_build as G
|
||
|
||
FAILS = []
|
||
def check(cond, msg):
|
||
print((" ok " if cond else " FAIL ") + msg)
|
||
if not cond: FAILS.append(msg)
|
||
|
||
def test_load_and_lint():
|
||
entries, meta = G.load_toml(paths.VM_MAP / "globals.toml")
|
||
check(0xa57 in entries, "0xa57 present in globals.toml")
|
||
check(entries[0xa57]["category"] == "story-flag", "0xa57 is category story-flag")
|
||
check(entries[0x3234]["category"] == "story-flag", "0x3234 is category story-flag")
|
||
check({0xa57, 0xa58, 0xa59} <= set(entries), "Lily form flags A/B/C all present")
|
||
# lint clean against a permissive address universe (curated addrs are self-consistent)
|
||
errors, warnings = G.lint(entries, set(entries))
|
||
check(errors == [], f"globals.toml lints clean (errors={errors})")
|
||
|
||
def test_lint_catches_bad_vocab():
|
||
bad = {0x1: {"_addr": 0x1, "name": "x", "category": "bogus",
|
||
"source": "auto-shape", "confidence": "high"}}
|
||
errors, _ = G.lint(bad, {0x1})
|
||
check(any("category" in e for e in errors), "lint flags bad category")
|
||
check(any("confidence" in e for e in errors), "lint flags auto-shape claiming high confidence")
|
||
|
||
if __name__ == "__main__":
|
||
test_load_and_lint()
|
||
test_lint_catches_bad_vocab()
|
||
print(f"\n{len(FAILS)} failures")
|
||
sys.exit(1 if FAILS else 0)
|
||
```
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `py -3.11 -X utf8 tools/test_globals.py`
|
||
Expected: FAIL — `ModuleNotFoundError: No module named 'globals_build'` (or, once the module exists but `globals.toml` doesn't, a load error).
|
||
|
||
- [ ] **Step 3: Create `globals.toml` with schema + first-pass curated content**
|
||
|
||
Create `vm-map/globals.toml`:
|
||
|
||
```toml
|
||
# vm-map/globals.toml -- CANONICAL living global-variable registry (hand-edited).
|
||
# Generated artifacts (build/globals.json, docs/global-reference.md) come from this file via
|
||
# tools/globals_build.py --build. Do not edit those. Skeletons are appended by
|
||
# tools/story_flags.py --bootstrap; enrich each entry as we investigate.
|
||
# See docs/superpowers/specs/2026-07-07-globals-registry-and-story-flags-design.md.
|
||
[meta]
|
||
note = "Curated global addresses override the auto shape-inference map (build/global-var-map.json)."
|
||
|
||
[[global]]
|
||
address = "0x3234"
|
||
name = "chapter_mode"
|
||
category = "story-flag"
|
||
type = "int"
|
||
value_domain = "1..9"
|
||
usage = "Progression chapter/mode selector. SCJUMP's top-level switch keys on it; branch-read by progression scripts (FIELD etc.), not directly by SC/SP scenes."
|
||
source = "investigation"
|
||
confidence = "high"
|
||
depends_on = ["0x62ccf"]
|
||
|
||
[[global]]
|
||
address = "0xa57"
|
||
name = "lily_form_a"
|
||
category = "story-flag"
|
||
type = "int"
|
||
value_domain = "{0,1}"
|
||
usage = "Lily current-form flag A. Exactly one of form A/B/C is 1; gates form-specific voiced dialogue (seeding 0xa57=1 -> SC0000 186->229 lines). Set externally (menu/save), no static writer."
|
||
source = "investigation"
|
||
confidence = "high"
|
||
depends_on = ["0xa58", "0xa59"]
|
||
|
||
[[global]]
|
||
address = "0xa58"
|
||
name = "lily_form_b"
|
||
category = "story-flag"
|
||
type = "int"
|
||
value_domain = "{0,1}"
|
||
usage = "Lily current-form flag B. See lily_form_a."
|
||
source = "investigation"
|
||
confidence = "high"
|
||
depends_on = ["0xa57", "0xa59"]
|
||
|
||
[[global]]
|
||
address = "0xa59"
|
||
name = "lily_form_c"
|
||
category = "story-flag"
|
||
type = "int"
|
||
value_domain = "{0,1}"
|
||
usage = "Lily current-form flag C. See lily_form_a."
|
||
source = "investigation"
|
||
confidence = "high"
|
||
depends_on = ["0xa57", "0xa58"]
|
||
|
||
[[global]]
|
||
address = "0x6c9"
|
||
name = "ui_toggle_0"
|
||
category = "ui-toggle"
|
||
type = "int"
|
||
value_domain = "{0,1}"
|
||
usage = "ADV-chrome hotspot button toggle (op 0x90 site, near-universal across scenes)."
|
||
source = "investigation"
|
||
confidence = "med"
|
||
depends_on = []
|
||
|
||
[[global]]
|
||
address = "0x6ca"
|
||
name = "ui_toggle_1"
|
||
category = "ui-toggle"
|
||
type = "int"
|
||
value_domain = "{0,1}"
|
||
usage = "ADV-chrome hotspot button toggle. See ui_toggle_0."
|
||
source = "investigation"
|
||
confidence = "med"
|
||
depends_on = []
|
||
|
||
[[global]]
|
||
address = "0x6cb"
|
||
name = "ui_toggle_2"
|
||
category = "ui-toggle"
|
||
type = "int"
|
||
value_domain = "{0,1}"
|
||
usage = "ADV-chrome hotspot button toggle. See ui_toggle_0."
|
||
source = "investigation"
|
||
confidence = "med"
|
||
depends_on = []
|
||
|
||
[[global]]
|
||
address = "0x6cc"
|
||
name = "ui_toggle_3"
|
||
category = "ui-toggle"
|
||
type = "int"
|
||
value_domain = "{0,1}"
|
||
usage = "ADV-chrome hotspot button toggle. See ui_toggle_0."
|
||
source = "investigation"
|
||
confidence = "med"
|
||
depends_on = []
|
||
|
||
[[global]]
|
||
address = "0x6cd"
|
||
name = "ui_toggle_4"
|
||
category = "ui-toggle"
|
||
type = "int"
|
||
value_domain = "{0,1}"
|
||
usage = "ADV-chrome hotspot button toggle. See ui_toggle_0."
|
||
source = "investigation"
|
||
confidence = "med"
|
||
depends_on = []
|
||
|
||
[[global]]
|
||
address = "0x62ccf"
|
||
name = "scjump_decision_out"
|
||
category = "choice-output"
|
||
type = "int"
|
||
value_domain = "?"
|
||
usage = "One of SCJUMP's output/decision globals (progression state machine writes it). Related to chapter_mode."
|
||
source = "inference"
|
||
confidence = "low"
|
||
depends_on = []
|
||
|
||
[[global]]
|
||
address = "0xeff75"
|
||
name = "current_entity_index_hi"
|
||
category = "index-pointer"
|
||
type = "int"
|
||
value_domain = "row index"
|
||
usage = "High-purity current-entity row index (purity 0.95 in the auto shape map); dominant 2D-table row selector."
|
||
source = "inference"
|
||
confidence = "med"
|
||
depends_on = []
|
||
|
||
[[global]]
|
||
address = "0x152616"
|
||
name = "current_entity_index"
|
||
category = "index-pointer"
|
||
type = "int"
|
||
value_domain = "row index"
|
||
usage = "Primary current-entity row index (RECOVER-confirmed; purity 0.51, 363 row-index uses)."
|
||
source = "investigation"
|
||
confidence = "med"
|
||
depends_on = []
|
||
```
|
||
|
||
- [ ] **Step 4: Implement `load_toml` + `lint` in `globals_build.py`**
|
||
|
||
Create `tools/globals_build.py`:
|
||
|
||
```python
|
||
#!/usr/bin/env python3
|
||
"""Generator + linter for the living global-variable registry (vm-map/globals.toml).
|
||
--build merge curated over build/global-var-map.json -> build/globals.json + docs/global-reference.md
|
||
--lint run the linter, print errors/warnings, exit nonzero on errors
|
||
See docs/superpowers/specs/2026-07-07-globals-registry-and-story-flags-design.md."""
|
||
from __future__ import annotations
|
||
import os, sys, json, argparse, collections, tomllib
|
||
from pathlib import Path
|
||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||
import paths
|
||
|
||
TOML_DEFAULT = paths.VM_MAP / "globals.toml"
|
||
AUTO_MAP = paths.BUILD / "global-var-map.json"
|
||
|
||
CATEGORIES = {"story-flag", "index-pointer", "data-table", "string-table",
|
||
"ui-toggle", "choice-output", "counter", "unknown"}
|
||
SOURCES = {"investigation", "harness", "inference", "auto-shape"}
|
||
CONFIDENCE = {"low": 1, "med": 2, "high": 3}
|
||
|
||
|
||
def _parse_addr(v) -> int:
|
||
return int(v, 16) if isinstance(v, str) else int(v)
|
||
|
||
|
||
def load_toml(path) -> tuple[dict[int, dict], dict]:
|
||
"""Return ({addr:int -> entry-dict (with _addr set)}, meta-dict)."""
|
||
data = tomllib.loads(Path(path).read_text(encoding="utf-8"))
|
||
out: dict[int, dict] = {}
|
||
for e in data.get("global", []):
|
||
addr = _parse_addr(e["address"])
|
||
e = dict(e)
|
||
e["_addr"] = addr
|
||
out[addr] = e
|
||
return out, data.get("meta", {})
|
||
|
||
|
||
def lint(entries: dict[int, dict], all_addrs: set[int]) -> tuple[list[str], list[str]]:
|
||
"""Errors: bad vocabulary, auto-shape@high, dangling depends_on, dup handled by dict.
|
||
Warnings: (reserved)."""
|
||
errors: list[str] = []
|
||
warnings: list[str] = []
|
||
for addr, e in sorted(entries.items()):
|
||
tag = f"0x{addr:x}"
|
||
cat, src, conf = e.get("category"), e.get("source"), e.get("confidence")
|
||
if cat not in CATEGORIES:
|
||
errors.append(f"{tag}: bad category {cat!r}")
|
||
if src not in SOURCES:
|
||
errors.append(f"{tag}: bad source {src!r}")
|
||
if conf not in CONFIDENCE:
|
||
errors.append(f"{tag}: bad confidence {conf!r}")
|
||
if src == "auto-shape" and conf == "high":
|
||
errors.append(f"{tag}: auto-shape source may not claim high confidence")
|
||
for dep in e.get("depends_on", []):
|
||
if _parse_addr(dep) not in all_addrs:
|
||
errors.append(f"{tag}: depends_on missing address {dep}")
|
||
return errors, warnings
|
||
|
||
|
||
def main(argv=None):
|
||
ap = argparse.ArgumentParser()
|
||
ap.add_argument("--build", action="store_true")
|
||
ap.add_argument("--lint", action="store_true")
|
||
ap.add_argument("--toml", default=str(TOML_DEFAULT))
|
||
args = ap.parse_args(argv)
|
||
tp = Path(args.toml)
|
||
if args.lint:
|
||
entries, _ = load_toml(tp)
|
||
errors, warnings = lint(entries, set(entries))
|
||
for m in warnings: print("warn:", m)
|
||
for m in errors: print("error:", m)
|
||
print(f"lint: {len(errors)} errors, {len(warnings)} warnings")
|
||
return 1 if errors else 0
|
||
if args.build:
|
||
return build(tp) # implemented in Task 2
|
||
ap.error("no action (expected --build/--lint)")
|
||
|
||
|
||
if __name__ == "__main__":
|
||
sys.exit(main())
|
||
```
|
||
|
||
Note: `main` references `build()` for `--build`, added in Task 2. `--lint` and the loader are fully functional now; do not call `--build` until Task 2.
|
||
|
||
- [ ] **Step 5: Run tests to verify they pass**
|
||
|
||
Run: `py -3.11 -X utf8 tools/test_globals.py`
|
||
Expected: PASS — all checks `ok`, `0 failures`, exit 0.
|
||
|
||
Also run the linter directly:
|
||
Run: `py -3.11 -X utf8 tools/globals_build.py --lint`
|
||
Expected: `lint: 0 errors, 0 warnings`, exit 0.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add vm-map/globals.toml tools/globals_build.py tools/test_globals.py
|
||
git commit -m "feat: globals.toml registry source + loader/lint (Task 1)
|
||
|
||
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: `globals_build.py --build` — merge + generate
|
||
|
||
Merge curated entries over the auto shape map and emit the machine JSON + human MD.
|
||
|
||
**Files:**
|
||
- Modify: `tools/globals_build.py` (add `build()`, `merge()`, `emit_json()`, `emit_reference_md()`, `_auto_category()`)
|
||
- Modify: `tools/test_globals.py` (add merge-precedence test)
|
||
|
||
**Interfaces:**
|
||
- Consumes: `load_toml`, `lint`, `AUTO_MAP`, `CATEGORIES` from Task 1.
|
||
- Produces (consumed by Task 3 + Task 6):
|
||
- `globals_build.merge(curated: dict[int,dict], auto: dict) -> dict[int, dict]` — merged registry keyed by addr; each value has keys `address, name, category, type, value_domain, usage, source, confidence, depends_on, provenance` where `provenance ∈ {"curated","auto"}`.
|
||
- `build/globals.json` — `{"meta":…, "generated_from":"vm-map/globals.toml", "globals": {"0xADDR": entry}}`.
|
||
- `docs/global-reference.md` — generated human view.
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
Add to `tools/test_globals.py` (call it from `__main__` after the existing tests):
|
||
|
||
```python
|
||
def test_merge_precedence():
|
||
curated, _ = G.load_toml(paths.VM_MAP / "globals.toml")
|
||
auto = G.load_auto(paths.BUILD / "global-var-map.json")
|
||
merged = G.merge(curated, auto)
|
||
check(merged[0xa57]["name"] == "lily_form_a", "curated 0xa57 name wins over auto label")
|
||
check(merged[0xa57]["category"] == "story-flag", "curated 0xa57 category overrides auto string-table")
|
||
check(merged[0xa57]["provenance"] == "curated", "0xa57 marked curated")
|
||
# an address only in the auto map falls through as provenance=auto
|
||
auto_only = next((a for a in auto.get("globals", {})
|
||
if int(a, 16) not in curated and auto["globals"][a].get("label")), None)
|
||
check(auto_only is not None and merged[int(auto_only, 16)]["provenance"] == "auto",
|
||
"auto-only address retained with provenance=auto")
|
||
```
|
||
|
||
Add `test_merge_precedence()` to the `__main__` block.
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `py -3.11 -X utf8 tools/test_globals.py`
|
||
Expected: FAIL — `AttributeError: module 'globals_build' has no attribute 'load_auto'` (and `merge`).
|
||
|
||
- [ ] **Step 3: Implement `load_auto`, `merge`, emitters, and `build` in `globals_build.py`**
|
||
|
||
Add to `tools/globals_build.py` (above `main`):
|
||
|
||
```python
|
||
GEN_JSON_META = {"generated_from": "vm-map/globals.toml",
|
||
"note": "DO NOT EDIT -- generated by tools/globals_build.py --build"}
|
||
|
||
def load_auto(path) -> dict:
|
||
try:
|
||
return json.loads(Path(path).read_text(encoding="utf-8"))
|
||
except Exception:
|
||
return {}
|
||
|
||
def _auto_category(e: dict) -> str:
|
||
"""Best-effort category for an auto shape-map entry (unreliable; curation overrides)."""
|
||
kind = e.get("kind") or ""
|
||
if kind == "record-table":
|
||
return "data-table"
|
||
if kind == "array-1d":
|
||
return "data-table"
|
||
lbl = e.get("label") or ""
|
||
if lbl.startswith("string-table"):
|
||
return "string-table"
|
||
return "unknown"
|
||
|
||
def merge(curated: dict[int, dict], auto: dict) -> dict[int, dict]:
|
||
out: dict[int, dict] = {}
|
||
for addr_s, e in auto.get("globals", {}).items():
|
||
if not e.get("label"):
|
||
continue
|
||
addr = int(addr_s, 16)
|
||
out[addr] = {"address": f"0x{addr:x}", "name": None, "category": _auto_category(e),
|
||
"type": e.get("type"), "value_domain": None, "usage": e.get("label"),
|
||
"source": "auto-shape", "confidence": e.get("confidence") or "low",
|
||
"depends_on": [], "provenance": "auto"}
|
||
for addr, e in curated.items():
|
||
out[addr] = {"address": f"0x{addr:x}", "name": e.get("name"),
|
||
"category": e.get("category", "unknown"), "type": e.get("type"),
|
||
"value_domain": e.get("value_domain"), "usage": e.get("usage", ""),
|
||
"source": e.get("source", "inference"), "confidence": e.get("confidence", "low"),
|
||
"depends_on": [f"0x{_parse_addr(d):x}" for d in e.get("depends_on", [])],
|
||
"provenance": "curated"}
|
||
return out
|
||
|
||
def emit_json(merged: dict[int, dict], meta: dict) -> str:
|
||
out = {"meta": {**GEN_JSON_META, **meta},
|
||
"globals": {e["address"]: e for _, e in sorted(merged.items())}}
|
||
return json.dumps(out, ensure_ascii=False, indent=2) + "\n"
|
||
|
||
def emit_reference_md(merged: dict[int, dict]) -> str:
|
||
L = ["<!-- DO NOT EDIT -- generated from vm-map/globals.toml by tools/globals_build.py --build -->",
|
||
"# Global Variable Reference (generated)", "",
|
||
f"{len(merged)} globals ({sum(1 for e in merged.values() if e['provenance']=='curated')} curated, "
|
||
f"{sum(1 for e in merged.values() if e['provenance']=='auto')} auto shape-inferred). "
|
||
"Source of truth: `vm-map/globals.toml`.", ""]
|
||
by_cat = collections.defaultdict(list)
|
||
for addr, e in merged.items():
|
||
by_cat[e["category"]].append(addr)
|
||
for cat in sorted(by_cat):
|
||
L += [f"## {cat}", "", "| address | name | conf | source | usage |", "|---|---|---|---|---|"]
|
||
# curated first, then by address
|
||
for addr in sorted(by_cat[cat], key=lambda a: (merged[a]["provenance"] != "curated", a)):
|
||
e = merged[addr]
|
||
name = e["name"] or "—"
|
||
usage = (e["usage"] or "").replace("|", "\\|").replace("\n", " ")
|
||
L.append(f"| `{e['address']}` | {name} | {e['confidence']} | {e['source']} | {usage} |")
|
||
L.append("")
|
||
return "\n".join(L) + "\n"
|
||
|
||
def build(tp: Path) -> int:
|
||
curated, meta = load_toml(tp)
|
||
auto = load_auto(AUTO_MAP)
|
||
all_addrs = set(curated) | {int(a, 16) for a in auto.get("globals", {})}
|
||
errors, warnings = lint(curated, all_addrs)
|
||
for m in warnings: print("warn:", m)
|
||
if errors:
|
||
for m in errors: print("error:", m)
|
||
return 1
|
||
merged = merge(curated, auto)
|
||
paths.BUILD.mkdir(parents=True, exist_ok=True)
|
||
(paths.BUILD / "globals.json").write_text(emit_json(merged, meta), encoding="utf-8")
|
||
(paths.REPO / "docs" / "global-reference.md").write_text(emit_reference_md(merged), encoding="utf-8")
|
||
print(f"build: {len(merged)} globals -> build/globals.json, docs/global-reference.md")
|
||
return 0
|
||
```
|
||
|
||
`main` already calls `build(tp)` for `--build` (the line was written in Task 1). No further change to `main` is needed.
|
||
|
||
- [ ] **Step 4: Run tests + build**
|
||
|
||
Run: `py -3.11 -X utf8 tools/globals_build.py --build`
|
||
Expected: `build: N globals -> build/globals.json, docs/global-reference.md` (N in the low thousands — every auto entry with a label plus curated), exit 0.
|
||
|
||
Run: `py -3.11 -X utf8 tools/test_globals.py`
|
||
Expected: PASS — all checks `ok`, `0 failures`.
|
||
|
||
- [ ] **Step 5: Sanity-check the generated JSON**
|
||
|
||
Run: `py -3.11 -X utf8 -c "import json; d=json.load(open('build/globals.json',encoding='utf-8')); print(d['globals']['0xa57'])"`
|
||
Expected: shows `name: lily_form_a`, `category: story-flag`, `provenance: curated`.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add tools/globals_build.py tools/test_globals.py build/globals.json docs/global-reference.md
|
||
git commit -m "feat: globals_build --build merge + generated JSON/MD (Task 2)
|
||
|
||
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: Wire `sys4load` to the merged `build/globals.json`
|
||
|
||
Switch the disassembler's global-operand labels from the raw auto map to the merged registry, so curated names/categories show.
|
||
|
||
**Files:**
|
||
- Modify: `tools/sys4load.py:48-76` (the `_load_global_labels` block)
|
||
- Modify: `tools/test_globals.py` (add a sys4load-label test)
|
||
|
||
**Interfaces:**
|
||
- Consumes: `build/globals.json` from Task 2 (`globals` dict, entries with `name`, `category`, `usage`).
|
||
- Produces: `sys4load.GLOBAL_LABELS: dict[int, str]` now sourced from the merged registry (curated names win; auto tail kept for high/med only). No signature change — downstream renderers unaffected.
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
Add to `tools/test_globals.py`:
|
||
|
||
```python
|
||
def test_sys4load_labels_from_registry():
|
||
import importlib, sys4load
|
||
importlib.reload(sys4load) # re-run _load_global_labels against current build/globals.json
|
||
lbl = sys4load.GLOBAL_LABELS.get(0x3234, "")
|
||
check("chapter_mode" in lbl, f"sys4load labels 0x3234 with curated name (got {lbl!r})")
|
||
lbl2 = sys4load.GLOBAL_LABELS.get(0xa57, "")
|
||
check("lily_form_a" in lbl2, f"sys4load labels 0xa57 with curated name (got {lbl2!r})")
|
||
```
|
||
|
||
Add `test_sys4load_labels_from_registry()` to `__main__`.
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `py -3.11 -X utf8 tools/test_globals.py`
|
||
Expected: FAIL — `sys4load labels 0x3234 with curated name` fails (current `GLOBAL_LABELS` comes from `global-var-map.json`, which has no `chapter_mode`).
|
||
|
||
- [ ] **Step 3: Rewrite `_load_global_labels` in `sys4load.py`**
|
||
|
||
Replace the block at `tools/sys4load.py:48-76` (from the `# Global-variable labels` comment through `GLOBAL_LABELS = _load_global_labels()`) with:
|
||
|
||
```python
|
||
# Global-variable labels (optional): annotate global operands from the merged registry
|
||
# build/globals.json (curated vm-map/globals.toml over the auto shape map), produced by
|
||
# tools/globals_build.py --build. Curated entries show their name+category; the auto tail
|
||
# keeps only high/med confidence to stay readable. Degrades to {} if the file is absent.
|
||
GLOBAL_ATYPES = {3, 4, 5, 6, 8} # global-int/float/string/ptr/string-ptr
|
||
|
||
|
||
def _short_global_label(lbl: str) -> str:
|
||
if lbl.startswith("record-table[stride "):
|
||
return "rec[s" + lbl[len("record-table[stride "):-1] + "]"
|
||
if lbl.startswith("string-table (written by "):
|
||
return "str<" + lbl[len("string-table (written by "):-1] + ">"
|
||
return lbl
|
||
|
||
|
||
def _load_global_labels() -> dict:
|
||
try:
|
||
p = Path(__file__).resolve().parent.parent / "build" / "globals.json"
|
||
data = json.loads(p.read_text(encoding="utf-8"))
|
||
except Exception:
|
||
return {}
|
||
out = {}
|
||
for addr_s, e in data.get("globals", {}).items():
|
||
addr = int(addr_s, 16)
|
||
if e.get("provenance") == "curated" and e.get("name"):
|
||
cat = e.get("category")
|
||
out[addr] = f"{e['name']}({cat})" if cat and cat != "unknown" else e["name"]
|
||
elif e.get("usage") and e.get("confidence") in ("high", "med"):
|
||
out[addr] = _short_global_label(e["usage"])
|
||
return out
|
||
|
||
|
||
GLOBAL_LABELS = _load_global_labels()
|
||
```
|
||
|
||
- [ ] **Step 4: Run test to verify it passes**
|
||
|
||
Run: `py -3.11 -X utf8 tools/test_globals.py`
|
||
Expected: PASS — the two label checks `ok`.
|
||
|
||
- [ ] **Step 5: Verify the loader still decodes the whole corpus**
|
||
|
||
Run: `py -3.11 -X utf8 tools/sys4load.py --validate`
|
||
Expected: `481/481` container-clean AND opcode-decode clean (unchanged from before — labels are cosmetic).
|
||
|
||
Spot-check a curated label renders in a listing:
|
||
Run: `py -3.11 -X utf8 -c "import sys; sys.path.insert(0,'tools'); import sys4load,paths; print(sys4load.GLOBAL_LABELS[0x3234], '|', sys4load.GLOBAL_LABELS[0xa57])"`
|
||
Expected: prints `chapter_mode(story-flag) | lily_form_a(story-flag)`.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add tools/sys4load.py tools/test_globals.py
|
||
git commit -m "feat: sys4load labels globals from merged build/globals.json (Task 3)
|
||
|
||
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: `story_flags.py` — static miner → candidates JSON
|
||
|
||
Scan the corpus for globals that feed branch conditions; emit ranked candidates with evidence.
|
||
|
||
**Files:**
|
||
- Create: `tools/story_flags.py`
|
||
- Modify: `tools/test_globals.py` (add miner regression-anchor test)
|
||
|
||
**Interfaces:**
|
||
- Consumes: `sys4load.load`, `paths.scripts`, `build/global-var-map.json` (for shape exclusion), `globals_build.load_toml` (Task 5).
|
||
- Produces (consumed by Task 5 + Task 6):
|
||
- `story_flags.mine() -> dict[int, dict]` — `addr -> evidence` with keys: `address(str)`, `reach_total(int)`, `reach_scenes(int)`, `consts(sorted list[int])`, `writers(sorted list[str])`, `writers_progression(sorted list[str])`, `atypes(sorted list[int])`, `near_universal(bool)`, `category(str)`, `confidence(str)`.
|
||
- `build/story-flags-candidates.json` — `{"meta":…, "candidates":[evidence…]}` sorted by (`reach_scenes` desc, `reach_total` desc).
|
||
- `story_flags.assign_category(ev) -> tuple[str,str]` — returns `(category, confidence)`.
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
Add to `tools/test_globals.py`:
|
||
|
||
```python
|
||
def test_miner_finds_known_flags():
|
||
import story_flags
|
||
cands = story_flags.mine()
|
||
check(0x3234 in cands, "miner surfaces chapter flag 0x3234")
|
||
check(set(range(1, 9)) <= set(cands[0x3234]["consts"]), "0x3234 compared against 1..8 enum")
|
||
check(cands[0x3234]["category"] == "story-flag", "0x3234 classified story-flag")
|
||
check(0xa57 in cands, "miner surfaces Lily form flag 0xa57")
|
||
check(cands[0xa57]["reach_scenes"] >= 70, "0xa57 high scene reach")
|
||
check(cands[0xa57]["category"] == "story-flag", "0xa57 classified story-flag")
|
||
```
|
||
|
||
Add `test_miner_finds_known_flags()` to `__main__`.
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `py -3.11 -X utf8 tools/test_globals.py`
|
||
Expected: FAIL — `ModuleNotFoundError: No module named 'story_flags'`.
|
||
|
||
- [ ] **Step 3: Implement `tools/story_flags.py`**
|
||
|
||
Create `tools/story_flags.py`:
|
||
|
||
```python
|
||
#!/usr/bin/env python3
|
||
"""Static story-state flag miner. Scans the 481-script corpus for global variables that feed
|
||
branch conditions (comparisons / jcc), gathers evidence, classifies candidates, and emits a
|
||
ranked review surface. 100% static -- no runtime, no sweep. See
|
||
docs/superpowers/specs/2026-07-07-globals-registry-and-story-flags-design.md.
|
||
(no flag) -> build/story-flags-candidates.json
|
||
--bootstrap append skeleton [[global]] entries to vm-map/globals.toml (Task 5)"""
|
||
from __future__ import annotations
|
||
import os, sys, re, json, argparse, collections
|
||
from pathlib import Path
|
||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||
import paths
|
||
import sys4load
|
||
|
||
CMP_OPS = {0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f} # eq ne lt lte gr gre (operands at arg idx 1,2)
|
||
LOGIC_OPS = {0x56, 0x57} # and or (operands at arg idx 1,2)
|
||
JCC_OP = 0xa0 # condition at arg idx 0
|
||
ASSIGN_OP = 0x55 # mov -> writes arg0
|
||
GLOBAL_ATYPES = {3, 4, 5, 6, 8}
|
||
SCENE_RE = re.compile(r"^S[CP]\d{4}\.BIN$")
|
||
KNOWN_UI_TOGGLES = {0x6c9, 0x6ca, 0x6cb, 0x6cd, 0x6cc}
|
||
NEAR_UNIVERSAL = 250 # scene-reach at/above this = ADV-chrome-wide, not a story flag
|
||
|
||
|
||
def _excluded_addrs() -> set[int]:
|
||
"""Addresses the auto shape map classifies as genuine tables/index pointers -- excluded from
|
||
story-flag candidacy. Deliberately does NOT exclude the auto 'string-table' *label* guesses
|
||
(those are unreliable: 0xa57, a real story flag, is mislabelled string-table)."""
|
||
try:
|
||
data = json.loads((paths.BUILD / "global-var-map.json").read_text(encoding="utf-8"))
|
||
except Exception:
|
||
return set()
|
||
excl = {int(c["addr"], 16) for c in data.get("current_entity_index_candidates", [])}
|
||
for addr_s, e in data.get("globals", {}).items():
|
||
if e.get("kind") == "record-table" and e.get("stride_candidates"):
|
||
excl.add(int(addr_s, 16))
|
||
return excl
|
||
|
||
|
||
def assign_category(ev: dict) -> tuple[str, str]:
|
||
"""(category, confidence) from evidence. Never 'high' -- auto-shape."""
|
||
addr = int(ev["address"], 16)
|
||
consts = set(ev["consts"])
|
||
domain_bool = consts <= {0, 1}
|
||
enum_like = len(consts) >= 3 and (max(consts) if consts else 0) <= 32
|
||
has_prog_writer = bool(ev["writers_progression"])
|
||
scene_only_writers = bool(ev["writers"]) and not has_prog_writer
|
||
if addr in KNOWN_UI_TOGGLES or (ev["near_universal"] and domain_bool):
|
||
cat = "ui-toggle"
|
||
elif enum_like and has_prog_writer:
|
||
cat = "story-flag" # chapter-like (0x3234)
|
||
elif has_prog_writer and ev["reach_total"] >= 1:
|
||
cat = "story-flag"
|
||
elif not ev["writers"] and ev["reach_scenes"] >= 3 and domain_bool:
|
||
cat = "story-flag" # externally/natively set form-like (0xa57)
|
||
elif scene_only_writers:
|
||
cat = "choice-output"
|
||
elif ev["reach_scenes"] >= 2 or ev["reach_total"] >= 2:
|
||
cat = "story-flag"
|
||
else:
|
||
cat = "unknown"
|
||
conf = "med" if (has_prog_writer and ev["reach_total"] >= 3) or ev["reach_scenes"] >= 10 else "low"
|
||
return cat, conf
|
||
|
||
|
||
def mine() -> dict[int, dict]:
|
||
reach_total = collections.defaultdict(set)
|
||
reach_scenes = collections.defaultdict(set)
|
||
consts = collections.defaultdict(set)
|
||
writers = collections.defaultdict(set)
|
||
atypes = collections.defaultdict(set)
|
||
for name, path in paths.scripts().items():
|
||
try:
|
||
scr = sys4load.load(path)
|
||
except Exception:
|
||
continue
|
||
is_scene = bool(SCENE_RE.match(name))
|
||
for ins in scr.instructions:
|
||
op, a = ins.opcode, ins.args
|
||
if op in CMP_OPS or op in LOGIC_OPS:
|
||
idxs = [1, 2]
|
||
elif op == JCC_OP:
|
||
idxs = [0]
|
||
else:
|
||
idxs = []
|
||
for i in idxs:
|
||
if i < len(a) and a[i][0] in GLOBAL_ATYPES:
|
||
addr = a[i][1]
|
||
reach_total[addr].add(name)
|
||
atypes[addr].add(a[i][0])
|
||
if is_scene:
|
||
reach_scenes[addr].add(name)
|
||
for j in idxs: # compared-against immediates
|
||
if j != i and j < len(a) and a[j][0] == 0:
|
||
consts[addr].add(a[j][1])
|
||
if op == ASSIGN_OP and len(a) >= 1 and a[0][0] in GLOBAL_ATYPES:
|
||
writers[a[0][1]].add(name)
|
||
excl = _excluded_addrs()
|
||
out: dict[int, dict] = {}
|
||
for addr in reach_total:
|
||
if addr in excl:
|
||
continue
|
||
w = writers.get(addr, set())
|
||
w_prog = {n for n in w if not SCENE_RE.match(n)}
|
||
ev = {"address": f"0x{addr:x}",
|
||
"reach_total": len(reach_total[addr]),
|
||
"reach_scenes": len(reach_scenes.get(addr, set())),
|
||
"consts": sorted(consts.get(addr, set())),
|
||
"writers": sorted(w),
|
||
"writers_progression": sorted(w_prog),
|
||
"atypes": sorted(atypes[addr]),
|
||
"near_universal": len(reach_scenes.get(addr, set())) >= NEAR_UNIVERSAL}
|
||
ev["category"], ev["confidence"] = assign_category(ev)
|
||
out[addr] = ev
|
||
return out
|
||
|
||
|
||
def write_candidates(cands: dict[int, dict]) -> Path:
|
||
ranked = sorted(cands.values(), key=lambda e: (-e["reach_scenes"], -e["reach_total"]))
|
||
paths.BUILD.mkdir(parents=True, exist_ok=True)
|
||
p = paths.BUILD / "story-flags-candidates.json"
|
||
p.write_text(json.dumps({"meta": {"note": "static branch-condition mining; review surface, "
|
||
"not ground truth", "count": len(ranked)},
|
||
"candidates": ranked}, ensure_ascii=False, indent=2) + "\n",
|
||
encoding="utf-8")
|
||
return p
|
||
|
||
|
||
def main(argv=None):
|
||
ap = argparse.ArgumentParser()
|
||
ap.add_argument("--bootstrap", action="store_true") # implemented in Task 5
|
||
args = ap.parse_args(argv)
|
||
cands = mine()
|
||
if args.bootstrap:
|
||
return bootstrap(cands) # Task 5
|
||
p = write_candidates(cands)
|
||
story = sum(1 for e in cands.values() if e["category"] == "story-flag")
|
||
print(f"mined {len(cands)} branch-read globals ({story} story-flag candidates) -> {p}")
|
||
return 0
|
||
|
||
|
||
if __name__ == "__main__":
|
||
sys.exit(main())
|
||
```
|
||
|
||
Note: `main` references `bootstrap()` for `--bootstrap`, added in Task 5. Running with no flag is fully functional now.
|
||
|
||
- [ ] **Step 4: Run test + miner**
|
||
|
||
Run: `py -3.11 -X utf8 tools/test_globals.py`
|
||
Expected: PASS — the six miner checks `ok`.
|
||
|
||
Run: `py -3.11 -X utf8 tools/story_flags.py`
|
||
Expected: `mined N branch-read globals (M story-flag candidates) -> …/build/story-flags-candidates.json`.
|
||
|
||
- [ ] **Step 5: Eyeball the top candidates**
|
||
|
||
Run: `py -3.11 -X utf8 -c "import json; d=json.load(open('build/story-flags-candidates.json',encoding='utf-8')); [print(c['address'], c['category'], 'scenes=%d'%c['reach_scenes'], 'consts=%s'%c['consts'][:6]) for c in d['candidates'][:15]]"`
|
||
Expected: `0xa57`/`0xa58`/`0xa59` near the top with high `reach_scenes`; entries look plausible (booleans and small enums).
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add tools/story_flags.py tools/test_globals.py build/story-flags-candidates.json
|
||
git commit -m "feat: static story-flag miner -> candidates JSON (Task 4)
|
||
|
||
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: `story_flags.py --bootstrap` — seed skeletons into `globals.toml`
|
||
|
||
Append skeleton entries for newly discovered addresses, preserving all hand edits.
|
||
|
||
**Files:**
|
||
- Modify: `tools/story_flags.py` (add `bootstrap()`, `skeleton_toml()`)
|
||
- Modify: `tools/test_globals.py` (add bootstrap test using a temp toml)
|
||
|
||
**Interfaces:**
|
||
- Consumes: `mine()` (Task 4), `globals_build.load_toml` (Task 1).
|
||
- Produces: `story_flags.bootstrap(cands, toml_path=…) -> int` — appends only new-address skeletons (`source="auto-shape"`, confidence from evidence, `usage="TODO: …"`); idempotent; never rewrites existing entries.
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
Add to `tools/test_globals.py`:
|
||
|
||
```python
|
||
def test_bootstrap_is_additive_and_idempotent():
|
||
import tempfile, pathlib, story_flags
|
||
# start from a copy of the real toml so curated entries are present
|
||
src = (paths.VM_MAP / "globals.toml").read_text(encoding="utf-8")
|
||
with tempfile.TemporaryDirectory() as d:
|
||
tp = pathlib.Path(d) / "globals.toml"
|
||
tp.write_text(src, encoding="utf-8")
|
||
before, _ = G.load_toml(tp)
|
||
cands = story_flags.mine()
|
||
story_flags.bootstrap(cands, toml_path=tp)
|
||
after, _ = G.load_toml(tp)
|
||
check(len(after) > len(before), "bootstrap adds new skeleton entries")
|
||
check(after[0xa57]["name"] == "lily_form_a", "bootstrap preserves curated 0xa57")
|
||
# idempotent: second run adds nothing
|
||
n1 = len(after)
|
||
story_flags.bootstrap(story_flags.mine(), toml_path=tp)
|
||
after2, _ = G.load_toml(tp)
|
||
check(len(after2) == n1, "second bootstrap is a no-op (idempotent)")
|
||
# every skeleton is auto-shape and not high-confidence
|
||
added = set(after) - set(before)
|
||
check(all(after[a]["source"] == "auto-shape" for a in added), "skeletons are source=auto-shape")
|
||
check(all(after[a]["confidence"] != "high" for a in added), "skeletons never high confidence")
|
||
```
|
||
|
||
Add `test_bootstrap_is_additive_and_idempotent()` to `__main__`. (Also add `import globals_build as G` is already at top; `story_flags` imported inside the test.)
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `py -3.11 -X utf8 tools/test_globals.py`
|
||
Expected: FAIL — `AttributeError: module 'story_flags' has no attribute 'bootstrap'`.
|
||
|
||
- [ ] **Step 3: Implement `bootstrap` + `skeleton_toml` in `story_flags.py`**
|
||
|
||
Add to `tools/story_flags.py` (above `main`):
|
||
|
||
```python
|
||
import globals_build as _gb
|
||
|
||
def skeleton_toml(ev: dict) -> str:
|
||
consts = ", ".join(str(c) for c in ev["consts"][:8])
|
||
domain = "{0,1}" if set(ev["consts"]) <= {0, 1} and ev["consts"] else (f"one of {{{consts}}}" if consts else "?")
|
||
usage = (f"TODO: confirm. Branch-read in {ev['reach_scenes']} scenes / {ev['reach_total']} scripts; "
|
||
f"compared against [{consts}]; "
|
||
f"writers={ev['writers'][:4] or 'none (external/native?)'}.")
|
||
lines = ["[[global]]",
|
||
f'address = "{ev["address"]}"',
|
||
f'name = ""',
|
||
f'category = "{ev["category"]}"',
|
||
'type = "int"',
|
||
f'value_domain = "{domain}"',
|
||
f'usage = "{usage}"',
|
||
'source = "auto-shape"',
|
||
f'confidence = "{ev["confidence"]}"',
|
||
"depends_on = []"]
|
||
return "\n".join(lines) + "\n"
|
||
|
||
def bootstrap(cands: dict[int, dict], toml_path=None) -> int:
|
||
toml_path = Path(toml_path) if toml_path else (paths.VM_MAP / "globals.toml")
|
||
present = set(_gb.load_toml(toml_path)[0]) if toml_path.exists() else set()
|
||
# only story-flag / ui-toggle / choice-output candidates are worth seeding for curation
|
||
seedable = {a: e for a, e in cands.items()
|
||
if e["category"] in ("story-flag", "ui-toggle", "choice-output") and a not in present}
|
||
if not seedable:
|
||
print(f"bootstrap: nothing new to add ({len(present)} already present).")
|
||
return 0
|
||
blocks = [skeleton_toml(seedable[a]) for a in sorted(seedable)]
|
||
with toml_path.open("a", encoding="utf-8") as f:
|
||
f.write("\n" + "\n".join(blocks))
|
||
print(f"bootstrap: appended {len(blocks)} skeletons -> {toml_path}")
|
||
return 0
|
||
```
|
||
|
||
The `name = ""` skeletons parse fine but should be named before they add value; leaving `name` empty means `sys4load` keeps the auto label until curated (curated-with-empty-name is treated as auto by the label rule in Task 3, which requires `name`). That's intended: a skeleton is a to-do, not a finished label.
|
||
|
||
- [ ] **Step 4: Run test to verify it passes**
|
||
|
||
Run: `py -3.11 -X utf8 tools/test_globals.py`
|
||
Expected: PASS — all bootstrap checks `ok`, `0 failures`.
|
||
|
||
- [ ] **Step 5: Verify against the real file is non-destructive (dry check)**
|
||
|
||
Run: `py -3.11 -X utf8 -c "import sys; sys.path.insert(0,'tools'); import story_flags,globals_build as g; before=set(g.load_toml('vm-map/globals.toml')[0]); print('curated now:', len(before))"`
|
||
Expected: prints the curated count (13). Do **not** run `--bootstrap` on the real file yet — that happens deliberately in Task 6 under review.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add tools/story_flags.py tools/test_globals.py
|
||
git commit -m "feat: story_flags --bootstrap seeds skeletons into globals.toml (Task 5)
|
||
|
||
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6: Curate strong candidates + docs + status memory
|
||
|
||
Run the miner against the real registry, promote the confident findings to named entries, rebuild, and update all documentation homes.
|
||
|
||
**Files:**
|
||
- Modify: `vm-map/globals.toml` (bootstrap + hand-name the strong candidates)
|
||
- Regenerate: `build/globals.json`, `docs/global-reference.md`, `build/story-flags-candidates.json`
|
||
- Modify: `docs/name-resolution.md` (§2 registry description + new "Story-state flags" subsection)
|
||
- Modify: `CLAUDE.md` (canonical-documents map row + single-source table row)
|
||
- Modify: `docs/tools-reference.md` (add `globals_build.py`, `story_flags.py`, `test_globals.py`)
|
||
- Modify: `~/.claude/…/memory/himegari-port-status.md` + `MEMORY.md` (status entry)
|
||
|
||
**Interfaces:**
|
||
- Consumes: all prior tasks.
|
||
- Produces: a populated registry + current docs. No new code interfaces.
|
||
|
||
- [ ] **Step 1: Bootstrap the real registry**
|
||
|
||
Run: `py -3.11 -X utf8 tools/story_flags.py --bootstrap`
|
||
Expected: `bootstrap: appended K skeletons -> …/vm-map/globals.toml`.
|
||
|
||
Run: `py -3.11 -X utf8 tools/globals_build.py --lint`
|
||
Expected: `lint: 0 errors, …` (skeletons are auto-shape + non-high, so they lint clean).
|
||
|
||
- [ ] **Step 2: Hand-name the confident candidates**
|
||
|
||
Open `build/story-flags-candidates.json`. For each candidate with `confidence == "med"` (strong reach and/or a progression writer), edit its skeleton in `vm-map/globals.toml`: set a `name` (snake_case), tighten `value_domain`, replace the `TODO:` `usage` with a real description, and bump `source`/`confidence` if you are confident (e.g. `inference`/`med`). Leave weak `low`-confidence skeletons with empty `name` (they stay to-dos). Do not invent meaning you cannot support — an honest `unknown`-ish skeleton is fine.
|
||
|
||
Guidance: cross-reference the reader scripts in a listing to infer purpose, e.g.
|
||
`py -3.11 -X utf8 tools/sys4load.py <SCRIPT.BIN>` and search for the address. Use `Age.Cli sweep 0xADDR=1` (the separate dynamic reach tool) to confirm a flag actually moves dialogue before naming it a story-flag.
|
||
|
||
- [ ] **Step 3: Rebuild generated artifacts + full test**
|
||
|
||
Run: `py -3.11 -X utf8 tools/globals_build.py --build`
|
||
Expected: `build: N globals -> build/globals.json, docs/global-reference.md`.
|
||
|
||
Run: `py -3.11 -X utf8 tools/story_flags.py`
|
||
Expected: refreshed candidates JSON.
|
||
|
||
Run: `py -3.11 -X utf8 tools/test_globals.py`
|
||
Expected: PASS, `0 failures`.
|
||
|
||
Run: `py -3.11 -X utf8 tools/sys4load.py --validate`
|
||
Expected: `481/481` clean.
|
||
|
||
- [ ] **Step 4: Update `docs/name-resolution.md`**
|
||
|
||
In §2 ("The global-variable map"), after the existing "Partial map — BUILT (v1)" content, add a subsection describing the curated registry. Insert this text:
|
||
|
||
```markdown
|
||
### The curated registry — `vm-map/globals.toml` (2026-07-07)
|
||
|
||
The v1 auto map (`build/global-var-map.json`) infers *shapes* but cannot recover branch-flag
|
||
*meaning* — and is sometimes wrong (it labels `0xa57`, the Lily form-A story flag, as a
|
||
"string-table"). The curated registry fixes this, modelled exactly on `vm-map/opcodes.toml`:
|
||
|
||
- **`vm-map/globals.toml`** — the only hand-edited source. One `[[global]]` per known address:
|
||
`name`, `category` (`story-flag`/`index-pointer`/`data-table`/`string-table`/`ui-toggle`/
|
||
`choice-output`/`counter`/`unknown`), `type`, `value_domain`, `usage`, and provenance
|
||
(`source`/`confidence`/`depends_on`).
|
||
- **`tools/globals_build.py --build`** merges curated entries *over* the auto map →
|
||
`build/globals.json` (machine) + `docs/global-reference.md` (generated human view). `--lint`
|
||
checks vocabulary, the auto-shape≠high rule, and dangling `depends_on`. `sys4load` reads
|
||
`build/globals.json` for operand labels (curated names win; auto tail kept at high/med).
|
||
|
||
#### Story-state flags (the first populated category)
|
||
|
||
Story flags are scalar globals that ADV/progression logic *branches on* (chapter, character
|
||
forms, choices) — a category the auto shape map never enumerated. **`tools/story_flags.py`** is a
|
||
100% static miner: it flags a global as a candidate when it feeds a comparison (`eq`/`ne`/`lt`/
|
||
`lte`/`gr`/`gre`), a logical (`and`/`or`), or a `jcc` condition, and is not a genuine table/index
|
||
in the shape map. Per candidate it records compared-against constants (→ value domain), the writer
|
||
set (progression-written but scene-read = strong story flag), total- and scene-reach, and
|
||
near-universal (ADV-chrome) status → an auto category + confidence. Output:
|
||
`build/story-flags-candidates.json` (review surface); `--bootstrap` appends skeletons to
|
||
`globals.toml` for human naming. Dynamic confirmation of a flag's reach stays separate —
|
||
`Age.Cli sweep 0xADDR=VAL`. Known anchors: `0x3234` chapter (enum 1..9, progression-read,
|
||
scene-reach 0), `0xa57/8/9` Lily forms (boolean, scene-read, externally set).
|
||
```
|
||
|
||
- [ ] **Step 5: Update `CLAUDE.md` (two tables)**
|
||
|
||
In the **canonical-documents map** table, add a row after the opcode row:
|
||
|
||
```markdown
|
||
| Global variable semantics / addresses (names, story-flags) | `age-reimpl/vm-map/globals.toml` *(source; generated → `build/globals.json`, `docs/global-reference.md`)* |
|
||
```
|
||
|
||
In the **single source of truth** table, add a row:
|
||
|
||
```markdown
|
||
| `vm-map/globals.toml` | `py -3.11 -X utf8 tools/globals_build.py --build` | `build/globals.json`, `docs/global-reference.md` |
|
||
```
|
||
|
||
- [ ] **Step 6: Update `docs/tools-reference.md`**
|
||
|
||
Add rows (match the existing table's column shape — tool / purpose / reads / writes):
|
||
|
||
```markdown
|
||
| `globals_build.py` | Merge curated globals.toml over the auto shape map; generate registry. | `vm-map/globals.toml`, `build/global-var-map.json` | `build/globals.json`, `docs/global-reference.md` |
|
||
| `story_flags.py` | Static story-flag miner (branch-condition mining) + `--bootstrap`. | corpus, `build/global-var-map.json` | `build/story-flags-candidates.json`, appends `vm-map/globals.toml` |
|
||
| `test_globals.py` | Unit tests for the globals registry + miner. | — | — |
|
||
```
|
||
|
||
- [ ] **Step 7: Commit code + docs**
|
||
|
||
```bash
|
||
git add vm-map/globals.toml build/globals.json docs/global-reference.md build/story-flags-candidates.json docs/name-resolution.md CLAUDE.md docs/tools-reference.md
|
||
git commit -m "feat: curate story-flags into registry + docs (Task 6)
|
||
|
||
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
|
||
```
|
||
|
||
- [ ] **Step 8: Update the status memory**
|
||
|
||
Edit `~/.claude/projects/S--Game-Hacking-Eushully-Himegari/memory/himegari-port-status.md`: add a bullet under Phase B recording that the globals registry (`vm-map/globals.toml`) + static story-flag miner (`tools/story_flags.py`) landed — the curated home for global semantics (opcodes.toml analogue), story-flags its first category, `sys4load` now labels from the merged `build/globals.json`; N story-flag candidates catalogued, K named. Update the `MEMORY.md` one-line index entry to mention story-flag mapping is underway/done. Convert "2026-07-07" to the absolute date. (Memory files are outside the git repo — no commit needed.)
|
||
|
||
---
|
||
|
||
## Notes for the implementer
|
||
|
||
- **Ordering matters:** Task 2's `build()` is referenced by Task 1's `main` but only *defined* in Task 2 — do not run `--build` until Task 2 is complete. Likewise `story_flags.bootstrap` is referenced in Task 4's `main` but defined in Task 5 — do not run `--bootstrap` until Task 5.
|
||
- **`py -3.11 -X utf8` always** — omitting `-X utf8` will corrupt cp932 script text on Windows.
|
||
- The candidates JSON is a *review surface*, not ground truth. Curation (naming) is a human judgment step (Task 6 Step 2); it is fine to leave most low-confidence skeletons unnamed.
|
||
- If `tomllib` is missing (Python < 3.11), the tools fail loudly — this project standardizes on 3.11.
|