# Globals Registry + Story-Flag Miner Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Build a hand-curated, single-source semantic registry for VM global variables (`vm-map/globals.toml`), generated into machine + human views and wired into `sys4load` labeling, plus a static miner that discovers story-state flags and proposes skeleton entries. **Architecture:** Mirror the existing opcode-reference toolchain. `vm-map/globals.toml` is the only hand-edited source (like `opcodes.toml`). `tools/globals_build.py` merges curated entries *over* the auto-generated shape map (`build/global-var-map.json`) into `build/globals.json` + generated `docs/global-reference.md`. `tools/story_flags.py` statically scans the 481-script corpus for globals that feed branch conditions, emits ranked candidates, and can bootstrap skeletons into `globals.toml`. `sys4load` switches its operand-label source from the raw auto map to the merged `build/globals.json`. **Tech Stack:** Python 3.11 (`py -3.11 -X utf8`), stdlib `tomllib`/`json`/`argparse`/`re`. Reuses `tools/paths.py`, `tools/sys4load.py`, `tools/age_opcodes.py`. Standalone test scripts (no pytest), matching `tools/test_opcodes.py`. ## Global Constraints - Run every tool as `py -3.11 -X utf8 tools/.py …` (the `-X utf8` is mandatory on Windows for cp932/Shift-JIS text). Source text is cp932; everything we generate is UTF-8. - Tools never hard-code paths — always `import paths` (`tools/paths.py`) and derive from it. - Never hand-edit a generated file. Generated files carry a "DO NOT EDIT — generated from …" header. `build/globals.json` and `docs/global-reference.md` are generated from `vm-map/globals.toml`; `build/global-var-map.json` is generated by `tools/global_map.py` (do not edit either). - `tools/age_opcodes.py` is pristine upstream Kelebek data — never edit it. - Provenance vocabulary is fixed: `category` ∈ {`story-flag`, `index-pointer`, `data-table`, `string-table`, `ui-toggle`, `choice-output`, `counter`, `unknown`}; `source` ∈ {`investigation`, `harness`, `inference`, `auto-shape`}; `confidence` ∈ {`low`, `med`, `high`}. An `auto-shape` source may never claim `high` confidence. - Git repo root is `age-reimpl/`. Commit from there. End commit messages with the `Co-Authored-By: Claude Opus 4.8 (1M context) ` trailer. - Tests are standalone scripts using a `check(cond, msg)` helper that records failures and exits nonzero if any fail (pattern: `tools/test_opcodes.py`). Run: `py -3.11 -X utf8 tools/test_globals.py`. **Reference facts (verified against the corpus, 2026-07-07):** - Comparison opcodes (result→arg0, operands at arg indices 1 & 2): `0x5a eq`, `0x5b ne`, `0x5c lt`, `0x5d lte`, `0x5e gr`, `0x5f gre`. Logical (operands at 1 & 2): `0x56 and`, `0x57 or`. `0xa0 jcc` reads its condition at arg index 0 (args 1 & 2 are jump targets). `0x55 mov` writes arg0. - Operand shape: `Instruction.args` is a list of `(atype, value)` tuples. Global atypes = {3,4,5,6,8} (int/float/string/ptr/string-ptr). Immediate atype = 0. - `sys4load.load(path)` → object with `.instructions`; each `Instruction` has `.opcode`, `.label`, `.args`, `.offset`. - `build/global-var-map.json` top-level keys: `globals` (dict `"0xADDR"` → `{type, uses, roles, stride_candidates, label, confidence, kind, table}`) and `current_entity_index_candidates` (list of `{addr, used_as_row_index, also_scalar, purity}`). - Corpus scan of the two known flags confirms the heuristic: `0x3234` (chapter) → scene-reach 0, total branch-reads exist, compared consts `{1..8}`, writer `FIELD.BIN`; `0xa57` (Lily form A) → scene-reach 78, compared const `{1}`, no `mov` writer. **The miner must track total-reach (not only scene-reach) and must not require a static writer.** --- ### Task 1: `globals.toml` source + loader + lint Create the hand-edited registry with first-pass curated content, plus the load/lint core of `globals_build.py`. **Files:** - Create: `vm-map/globals.toml` - Create: `tools/globals_build.py` - Create: `tools/test_globals.py` **Interfaces:** - Produces (consumed by Tasks 2, 4, 5): - `globals_build.load_toml(path: Path) -> tuple[dict[int, dict], dict]` — returns `(entries_by_addr, meta)`; each entry is the raw TOML table dict with an added `_addr: int` key. - `globals_build.lint(entries: dict[int, dict], all_addrs: set[int]) -> tuple[list[str], list[str]]` — `(errors, warnings)`. `all_addrs` is the set of addresses valid for `depends_on` resolution (curated ∪ auto). - Module constants `CATEGORIES: set[str]`, `SOURCES: set[str]`, `CONFIDENCE: dict[str,int]`. - [ ] **Step 1: Write the failing test** Create `tools/test_globals.py`: ```python #!/usr/bin/env python3 """Standalone tests for the globals registry tooling. Run: py -3.11 -X utf8 tools/test_globals.py""" import os, sys sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) import paths import globals_build as G FAILS = [] def check(cond, msg): print((" ok " if cond else " FAIL ") + msg) if not cond: FAILS.append(msg) def test_load_and_lint(): entries, meta = G.load_toml(paths.VM_MAP / "globals.toml") check(0xa57 in entries, "0xa57 present in globals.toml") check(entries[0xa57]["category"] == "story-flag", "0xa57 is category story-flag") check(entries[0x3234]["category"] == "story-flag", "0x3234 is category story-flag") check({0xa57, 0xa58, 0xa59} <= set(entries), "Lily form flags A/B/C all present") # lint clean against a permissive address universe (curated addrs are self-consistent) errors, warnings = G.lint(entries, set(entries)) check(errors == [], f"globals.toml lints clean (errors={errors})") def test_lint_catches_bad_vocab(): bad = {0x1: {"_addr": 0x1, "name": "x", "category": "bogus", "source": "auto-shape", "confidence": "high"}} errors, _ = G.lint(bad, {0x1}) check(any("category" in e for e in errors), "lint flags bad category") check(any("confidence" in e for e in errors), "lint flags auto-shape claiming high confidence") if __name__ == "__main__": test_load_and_lint() test_lint_catches_bad_vocab() print(f"\n{len(FAILS)} failures") sys.exit(1 if FAILS else 0) ``` - [ ] **Step 2: Run test to verify it fails** Run: `py -3.11 -X utf8 tools/test_globals.py` Expected: FAIL — `ModuleNotFoundError: No module named 'globals_build'` (or, once the module exists but `globals.toml` doesn't, a load error). - [ ] **Step 3: Create `globals.toml` with schema + first-pass curated content** Create `vm-map/globals.toml`: ```toml # vm-map/globals.toml -- CANONICAL living global-variable registry (hand-edited). # Generated artifacts (build/globals.json, docs/global-reference.md) come from this file via # tools/globals_build.py --build. Do not edit those. Skeletons are appended by # tools/story_flags.py --bootstrap; enrich each entry as we investigate. # See docs/superpowers/specs/2026-07-07-globals-registry-and-story-flags-design.md. [meta] note = "Curated global addresses override the auto shape-inference map (build/global-var-map.json)." [[global]] address = "0x3234" name = "chapter_mode" category = "story-flag" type = "int" value_domain = "1..9" usage = "Progression chapter/mode selector. SCJUMP's top-level switch keys on it; branch-read by progression scripts (FIELD etc.), not directly by SC/SP scenes." source = "investigation" confidence = "high" depends_on = ["0x62ccf"] [[global]] address = "0xa57" name = "lily_form_a" category = "story-flag" type = "int" value_domain = "{0,1}" usage = "Lily current-form flag A. Exactly one of form A/B/C is 1; gates form-specific voiced dialogue (seeding 0xa57=1 -> SC0000 186->229 lines). Set externally (menu/save), no static writer." source = "investigation" confidence = "high" depends_on = ["0xa58", "0xa59"] [[global]] address = "0xa58" name = "lily_form_b" category = "story-flag" type = "int" value_domain = "{0,1}" usage = "Lily current-form flag B. See lily_form_a." source = "investigation" confidence = "high" depends_on = ["0xa57", "0xa59"] [[global]] address = "0xa59" name = "lily_form_c" category = "story-flag" type = "int" value_domain = "{0,1}" usage = "Lily current-form flag C. See lily_form_a." source = "investigation" confidence = "high" depends_on = ["0xa57", "0xa58"] [[global]] address = "0x6c9" name = "ui_toggle_0" category = "ui-toggle" type = "int" value_domain = "{0,1}" usage = "ADV-chrome hotspot button toggle (op 0x90 site, near-universal across scenes)." source = "investigation" confidence = "med" depends_on = [] [[global]] address = "0x6ca" name = "ui_toggle_1" category = "ui-toggle" type = "int" value_domain = "{0,1}" usage = "ADV-chrome hotspot button toggle. See ui_toggle_0." source = "investigation" confidence = "med" depends_on = [] [[global]] address = "0x6cb" name = "ui_toggle_2" category = "ui-toggle" type = "int" value_domain = "{0,1}" usage = "ADV-chrome hotspot button toggle. See ui_toggle_0." source = "investigation" confidence = "med" depends_on = [] [[global]] address = "0x6cc" name = "ui_toggle_3" category = "ui-toggle" type = "int" value_domain = "{0,1}" usage = "ADV-chrome hotspot button toggle. See ui_toggle_0." source = "investigation" confidence = "med" depends_on = [] [[global]] address = "0x6cd" name = "ui_toggle_4" category = "ui-toggle" type = "int" value_domain = "{0,1}" usage = "ADV-chrome hotspot button toggle. See ui_toggle_0." source = "investigation" confidence = "med" depends_on = [] [[global]] address = "0x62ccf" name = "scjump_decision_out" category = "choice-output" type = "int" value_domain = "?" usage = "One of SCJUMP's output/decision globals (progression state machine writes it). Related to chapter_mode." source = "inference" confidence = "low" depends_on = [] [[global]] address = "0xeff75" name = "current_entity_index_hi" category = "index-pointer" type = "int" value_domain = "row index" usage = "High-purity current-entity row index (purity 0.95 in the auto shape map); dominant 2D-table row selector." source = "inference" confidence = "med" depends_on = [] [[global]] address = "0x152616" name = "current_entity_index" category = "index-pointer" type = "int" value_domain = "row index" usage = "Primary current-entity row index (RECOVER-confirmed; purity 0.51, 363 row-index uses)." source = "investigation" confidence = "med" depends_on = [] ``` - [ ] **Step 4: Implement `load_toml` + `lint` in `globals_build.py`** Create `tools/globals_build.py`: ```python #!/usr/bin/env python3 """Generator + linter for the living global-variable registry (vm-map/globals.toml). --build merge curated over build/global-var-map.json -> build/globals.json + docs/global-reference.md --lint run the linter, print errors/warnings, exit nonzero on errors See docs/superpowers/specs/2026-07-07-globals-registry-and-story-flags-design.md.""" from __future__ import annotations import os, sys, json, argparse, collections, tomllib from pathlib import Path sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) import paths TOML_DEFAULT = paths.VM_MAP / "globals.toml" AUTO_MAP = paths.BUILD / "global-var-map.json" CATEGORIES = {"story-flag", "index-pointer", "data-table", "string-table", "ui-toggle", "choice-output", "counter", "unknown"} SOURCES = {"investigation", "harness", "inference", "auto-shape"} CONFIDENCE = {"low": 1, "med": 2, "high": 3} def _parse_addr(v) -> int: return int(v, 16) if isinstance(v, str) else int(v) def load_toml(path) -> tuple[dict[int, dict], dict]: """Return ({addr:int -> entry-dict (with _addr set)}, meta-dict).""" data = tomllib.loads(Path(path).read_text(encoding="utf-8")) out: dict[int, dict] = {} for e in data.get("global", []): addr = _parse_addr(e["address"]) e = dict(e) e["_addr"] = addr out[addr] = e return out, data.get("meta", {}) def lint(entries: dict[int, dict], all_addrs: set[int]) -> tuple[list[str], list[str]]: """Errors: bad vocabulary, auto-shape@high, dangling depends_on, dup handled by dict. Warnings: (reserved).""" errors: list[str] = [] warnings: list[str] = [] for addr, e in sorted(entries.items()): tag = f"0x{addr:x}" cat, src, conf = e.get("category"), e.get("source"), e.get("confidence") if cat not in CATEGORIES: errors.append(f"{tag}: bad category {cat!r}") if src not in SOURCES: errors.append(f"{tag}: bad source {src!r}") if conf not in CONFIDENCE: errors.append(f"{tag}: bad confidence {conf!r}") if src == "auto-shape" and conf == "high": errors.append(f"{tag}: auto-shape source may not claim high confidence") for dep in e.get("depends_on", []): if _parse_addr(dep) not in all_addrs: errors.append(f"{tag}: depends_on missing address {dep}") return errors, warnings def main(argv=None): ap = argparse.ArgumentParser() ap.add_argument("--build", action="store_true") ap.add_argument("--lint", action="store_true") ap.add_argument("--toml", default=str(TOML_DEFAULT)) args = ap.parse_args(argv) tp = Path(args.toml) if args.lint: entries, _ = load_toml(tp) errors, warnings = lint(entries, set(entries)) for m in warnings: print("warn:", m) for m in errors: print("error:", m) print(f"lint: {len(errors)} errors, {len(warnings)} warnings") return 1 if errors else 0 if args.build: return build(tp) # implemented in Task 2 ap.error("no action (expected --build/--lint)") if __name__ == "__main__": sys.exit(main()) ``` Note: `main` references `build()` for `--build`, added in Task 2. `--lint` and the loader are fully functional now; do not call `--build` until Task 2. - [ ] **Step 5: Run tests to verify they pass** Run: `py -3.11 -X utf8 tools/test_globals.py` Expected: PASS — all checks `ok`, `0 failures`, exit 0. Also run the linter directly: Run: `py -3.11 -X utf8 tools/globals_build.py --lint` Expected: `lint: 0 errors, 0 warnings`, exit 0. - [ ] **Step 6: Commit** ```bash git add vm-map/globals.toml tools/globals_build.py tools/test_globals.py git commit -m "feat: globals.toml registry source + loader/lint (Task 1) Co-Authored-By: Claude Opus 4.8 (1M context) " ``` --- ### Task 2: `globals_build.py --build` — merge + generate Merge curated entries over the auto shape map and emit the machine JSON + human MD. **Files:** - Modify: `tools/globals_build.py` (add `build()`, `merge()`, `emit_json()`, `emit_reference_md()`, `_auto_category()`) - Modify: `tools/test_globals.py` (add merge-precedence test) **Interfaces:** - Consumes: `load_toml`, `lint`, `AUTO_MAP`, `CATEGORIES` from Task 1. - Produces (consumed by Task 3 + Task 6): - `globals_build.merge(curated: dict[int,dict], auto: dict) -> dict[int, dict]` — merged registry keyed by addr; each value has keys `address, name, category, type, value_domain, usage, source, confidence, depends_on, provenance` where `provenance ∈ {"curated","auto"}`. - `build/globals.json` — `{"meta":…, "generated_from":"vm-map/globals.toml", "globals": {"0xADDR": entry}}`. - `docs/global-reference.md` — generated human view. - [ ] **Step 1: Write the failing test** Add to `tools/test_globals.py` (call it from `__main__` after the existing tests): ```python def test_merge_precedence(): curated, _ = G.load_toml(paths.VM_MAP / "globals.toml") auto = G.load_auto(paths.BUILD / "global-var-map.json") merged = G.merge(curated, auto) check(merged[0xa57]["name"] == "lily_form_a", "curated 0xa57 name wins over auto label") check(merged[0xa57]["category"] == "story-flag", "curated 0xa57 category overrides auto string-table") check(merged[0xa57]["provenance"] == "curated", "0xa57 marked curated") # an address only in the auto map falls through as provenance=auto auto_only = next((a for a in auto.get("globals", {}) if int(a, 16) not in curated and auto["globals"][a].get("label")), None) check(auto_only is not None and merged[int(auto_only, 16)]["provenance"] == "auto", "auto-only address retained with provenance=auto") ``` Add `test_merge_precedence()` to the `__main__` block. - [ ] **Step 2: Run test to verify it fails** Run: `py -3.11 -X utf8 tools/test_globals.py` Expected: FAIL — `AttributeError: module 'globals_build' has no attribute 'load_auto'` (and `merge`). - [ ] **Step 3: Implement `load_auto`, `merge`, emitters, and `build` in `globals_build.py`** Add to `tools/globals_build.py` (above `main`): ```python GEN_JSON_META = {"generated_from": "vm-map/globals.toml", "note": "DO NOT EDIT -- generated by tools/globals_build.py --build"} def load_auto(path) -> dict: try: return json.loads(Path(path).read_text(encoding="utf-8")) except Exception: return {} def _auto_category(e: dict) -> str: """Best-effort category for an auto shape-map entry (unreliable; curation overrides).""" kind = e.get("kind") or "" if kind == "record-table": return "data-table" if kind == "array-1d": return "data-table" lbl = e.get("label") or "" if lbl.startswith("string-table"): return "string-table" return "unknown" def merge(curated: dict[int, dict], auto: dict) -> dict[int, dict]: out: dict[int, dict] = {} for addr_s, e in auto.get("globals", {}).items(): if not e.get("label"): continue addr = int(addr_s, 16) out[addr] = {"address": f"0x{addr:x}", "name": None, "category": _auto_category(e), "type": e.get("type"), "value_domain": None, "usage": e.get("label"), "source": "auto-shape", "confidence": e.get("confidence") or "low", "depends_on": [], "provenance": "auto"} for addr, e in curated.items(): out[addr] = {"address": f"0x{addr:x}", "name": e.get("name"), "category": e.get("category", "unknown"), "type": e.get("type"), "value_domain": e.get("value_domain"), "usage": e.get("usage", ""), "source": e.get("source", "inference"), "confidence": e.get("confidence", "low"), "depends_on": [f"0x{_parse_addr(d):x}" for d in e.get("depends_on", [])], "provenance": "curated"} return out def emit_json(merged: dict[int, dict], meta: dict) -> str: out = {"meta": {**GEN_JSON_META, **meta}, "globals": {e["address"]: e for _, e in sorted(merged.items())}} return json.dumps(out, ensure_ascii=False, indent=2) + "\n" def emit_reference_md(merged: dict[int, dict]) -> str: L = ["", "# Global Variable Reference (generated)", "", f"{len(merged)} globals ({sum(1 for e in merged.values() if e['provenance']=='curated')} curated, " f"{sum(1 for e in merged.values() if e['provenance']=='auto')} auto shape-inferred). " "Source of truth: `vm-map/globals.toml`.", ""] by_cat = collections.defaultdict(list) for addr, e in merged.items(): by_cat[e["category"]].append(addr) for cat in sorted(by_cat): L += [f"## {cat}", "", "| address | name | conf | source | usage |", "|---|---|---|---|---|"] # curated first, then by address for addr in sorted(by_cat[cat], key=lambda a: (merged[a]["provenance"] != "curated", a)): e = merged[addr] name = e["name"] or "—" usage = (e["usage"] or "").replace("|", "\\|").replace("\n", " ") L.append(f"| `{e['address']}` | {name} | {e['confidence']} | {e['source']} | {usage} |") L.append("") return "\n".join(L) + "\n" def build(tp: Path) -> int: curated, meta = load_toml(tp) auto = load_auto(AUTO_MAP) all_addrs = set(curated) | {int(a, 16) for a in auto.get("globals", {})} errors, warnings = lint(curated, all_addrs) for m in warnings: print("warn:", m) if errors: for m in errors: print("error:", m) return 1 merged = merge(curated, auto) paths.BUILD.mkdir(parents=True, exist_ok=True) (paths.BUILD / "globals.json").write_text(emit_json(merged, meta), encoding="utf-8") (paths.REPO / "docs" / "global-reference.md").write_text(emit_reference_md(merged), encoding="utf-8") print(f"build: {len(merged)} globals -> build/globals.json, docs/global-reference.md") return 0 ``` `main` already calls `build(tp)` for `--build` (the line was written in Task 1). No further change to `main` is needed. - [ ] **Step 4: Run tests + build** Run: `py -3.11 -X utf8 tools/globals_build.py --build` Expected: `build: N globals -> build/globals.json, docs/global-reference.md` (N in the low thousands — every auto entry with a label plus curated), exit 0. Run: `py -3.11 -X utf8 tools/test_globals.py` Expected: PASS — all checks `ok`, `0 failures`. - [ ] **Step 5: Sanity-check the generated JSON** Run: `py -3.11 -X utf8 -c "import json; d=json.load(open('build/globals.json',encoding='utf-8')); print(d['globals']['0xa57'])"` Expected: shows `name: lily_form_a`, `category: story-flag`, `provenance: curated`. - [ ] **Step 6: Commit** ```bash git add tools/globals_build.py tools/test_globals.py build/globals.json docs/global-reference.md git commit -m "feat: globals_build --build merge + generated JSON/MD (Task 2) Co-Authored-By: Claude Opus 4.8 (1M context) " ``` --- ### Task 3: Wire `sys4load` to the merged `build/globals.json` Switch the disassembler's global-operand labels from the raw auto map to the merged registry, so curated names/categories show. **Files:** - Modify: `tools/sys4load.py:48-76` (the `_load_global_labels` block) - Modify: `tools/test_globals.py` (add a sys4load-label test) **Interfaces:** - Consumes: `build/globals.json` from Task 2 (`globals` dict, entries with `name`, `category`, `usage`). - Produces: `sys4load.GLOBAL_LABELS: dict[int, str]` now sourced from the merged registry (curated names win; auto tail kept for high/med only). No signature change — downstream renderers unaffected. - [ ] **Step 1: Write the failing test** Add to `tools/test_globals.py`: ```python def test_sys4load_labels_from_registry(): import importlib, sys4load importlib.reload(sys4load) # re-run _load_global_labels against current build/globals.json lbl = sys4load.GLOBAL_LABELS.get(0x3234, "") check("chapter_mode" in lbl, f"sys4load labels 0x3234 with curated name (got {lbl!r})") lbl2 = sys4load.GLOBAL_LABELS.get(0xa57, "") check("lily_form_a" in lbl2, f"sys4load labels 0xa57 with curated name (got {lbl2!r})") ``` Add `test_sys4load_labels_from_registry()` to `__main__`. - [ ] **Step 2: Run test to verify it fails** Run: `py -3.11 -X utf8 tools/test_globals.py` Expected: FAIL — `sys4load labels 0x3234 with curated name` fails (current `GLOBAL_LABELS` comes from `global-var-map.json`, which has no `chapter_mode`). - [ ] **Step 3: Rewrite `_load_global_labels` in `sys4load.py`** Replace the block at `tools/sys4load.py:48-76` (from the `# Global-variable labels` comment through `GLOBAL_LABELS = _load_global_labels()`) with: ```python # Global-variable labels (optional): annotate global operands from the merged registry # build/globals.json (curated vm-map/globals.toml over the auto shape map), produced by # tools/globals_build.py --build. Curated entries show their name+category; the auto tail # keeps only high/med confidence to stay readable. Degrades to {} if the file is absent. GLOBAL_ATYPES = {3, 4, 5, 6, 8} # global-int/float/string/ptr/string-ptr def _short_global_label(lbl: str) -> str: if lbl.startswith("record-table[stride "): return "rec[s" + lbl[len("record-table[stride "):-1] + "]" if lbl.startswith("string-table (written by "): return "str<" + lbl[len("string-table (written by "):-1] + ">" return lbl def _load_global_labels() -> dict: try: p = Path(__file__).resolve().parent.parent / "build" / "globals.json" data = json.loads(p.read_text(encoding="utf-8")) except Exception: return {} out = {} for addr_s, e in data.get("globals", {}).items(): addr = int(addr_s, 16) if e.get("provenance") == "curated" and e.get("name"): cat = e.get("category") out[addr] = f"{e['name']}({cat})" if cat and cat != "unknown" else e["name"] elif e.get("usage") and e.get("confidence") in ("high", "med"): out[addr] = _short_global_label(e["usage"]) return out GLOBAL_LABELS = _load_global_labels() ``` - [ ] **Step 4: Run test to verify it passes** Run: `py -3.11 -X utf8 tools/test_globals.py` Expected: PASS — the two label checks `ok`. - [ ] **Step 5: Verify the loader still decodes the whole corpus** Run: `py -3.11 -X utf8 tools/sys4load.py --validate` Expected: `481/481` container-clean AND opcode-decode clean (unchanged from before — labels are cosmetic). Spot-check a curated label renders in a listing: Run: `py -3.11 -X utf8 -c "import sys; sys.path.insert(0,'tools'); import sys4load,paths; print(sys4load.GLOBAL_LABELS[0x3234], '|', sys4load.GLOBAL_LABELS[0xa57])"` Expected: prints `chapter_mode(story-flag) | lily_form_a(story-flag)`. - [ ] **Step 6: Commit** ```bash git add tools/sys4load.py tools/test_globals.py git commit -m "feat: sys4load labels globals from merged build/globals.json (Task 3) Co-Authored-By: Claude Opus 4.8 (1M context) " ``` --- ### Task 4: `story_flags.py` — static miner → candidates JSON Scan the corpus for globals that feed branch conditions; emit ranked candidates with evidence. **Files:** - Create: `tools/story_flags.py` - Modify: `tools/test_globals.py` (add miner regression-anchor test) **Interfaces:** - Consumes: `sys4load.load`, `paths.scripts`, `build/global-var-map.json` (for shape exclusion), `globals_build.load_toml` (Task 5). - Produces (consumed by Task 5 + Task 6): - `story_flags.mine() -> dict[int, dict]` — `addr -> evidence` with keys: `address(str)`, `reach_total(int)`, `reach_scenes(int)`, `consts(sorted list[int])`, `writers(sorted list[str])`, `writers_progression(sorted list[str])`, `atypes(sorted list[int])`, `near_universal(bool)`, `category(str)`, `confidence(str)`. - `build/story-flags-candidates.json` — `{"meta":…, "candidates":[evidence…]}` sorted by (`reach_scenes` desc, `reach_total` desc). - `story_flags.assign_category(ev) -> tuple[str,str]` — returns `(category, confidence)`. - [ ] **Step 1: Write the failing test** Add to `tools/test_globals.py`: ```python def test_miner_finds_known_flags(): import story_flags cands = story_flags.mine() check(0x3234 in cands, "miner surfaces chapter flag 0x3234") check(set(range(1, 9)) <= set(cands[0x3234]["consts"]), "0x3234 compared against 1..8 enum") check(cands[0x3234]["category"] == "story-flag", "0x3234 classified story-flag") check(0xa57 in cands, "miner surfaces Lily form flag 0xa57") check(cands[0xa57]["reach_scenes"] >= 70, "0xa57 high scene reach") check(cands[0xa57]["category"] == "story-flag", "0xa57 classified story-flag") ``` Add `test_miner_finds_known_flags()` to `__main__`. - [ ] **Step 2: Run test to verify it fails** Run: `py -3.11 -X utf8 tools/test_globals.py` Expected: FAIL — `ModuleNotFoundError: No module named 'story_flags'`. - [ ] **Step 3: Implement `tools/story_flags.py`** Create `tools/story_flags.py`: ```python #!/usr/bin/env python3 """Static story-state flag miner. Scans the 481-script corpus for global variables that feed branch conditions (comparisons / jcc), gathers evidence, classifies candidates, and emits a ranked review surface. 100% static -- no runtime, no sweep. See docs/superpowers/specs/2026-07-07-globals-registry-and-story-flags-design.md. (no flag) -> build/story-flags-candidates.json --bootstrap append skeleton [[global]] entries to vm-map/globals.toml (Task 5)""" from __future__ import annotations import os, sys, re, json, argparse, collections from pathlib import Path sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) import paths import sys4load CMP_OPS = {0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f} # eq ne lt lte gr gre (operands at arg idx 1,2) LOGIC_OPS = {0x56, 0x57} # and or (operands at arg idx 1,2) JCC_OP = 0xa0 # condition at arg idx 0 ASSIGN_OP = 0x55 # mov -> writes arg0 GLOBAL_ATYPES = {3, 4, 5, 6, 8} SCENE_RE = re.compile(r"^S[CP]\d{4}\.BIN$") KNOWN_UI_TOGGLES = {0x6c9, 0x6ca, 0x6cb, 0x6cd, 0x6cc} NEAR_UNIVERSAL = 250 # scene-reach at/above this = ADV-chrome-wide, not a story flag def _excluded_addrs() -> set[int]: """Addresses the auto shape map classifies as genuine tables/index pointers -- excluded from story-flag candidacy. Deliberately does NOT exclude the auto 'string-table' *label* guesses (those are unreliable: 0xa57, a real story flag, is mislabelled string-table).""" try: data = json.loads((paths.BUILD / "global-var-map.json").read_text(encoding="utf-8")) except Exception: return set() excl = {int(c["addr"], 16) for c in data.get("current_entity_index_candidates", [])} for addr_s, e in data.get("globals", {}).items(): if e.get("kind") == "record-table" and e.get("stride_candidates"): excl.add(int(addr_s, 16)) return excl def assign_category(ev: dict) -> tuple[str, str]: """(category, confidence) from evidence. Never 'high' -- auto-shape.""" addr = int(ev["address"], 16) consts = set(ev["consts"]) domain_bool = consts <= {0, 1} enum_like = len(consts) >= 3 and (max(consts) if consts else 0) <= 32 has_prog_writer = bool(ev["writers_progression"]) scene_only_writers = bool(ev["writers"]) and not has_prog_writer if addr in KNOWN_UI_TOGGLES or (ev["near_universal"] and domain_bool): cat = "ui-toggle" elif enum_like and has_prog_writer: cat = "story-flag" # chapter-like (0x3234) elif has_prog_writer and ev["reach_total"] >= 1: cat = "story-flag" elif not ev["writers"] and ev["reach_scenes"] >= 3 and domain_bool: cat = "story-flag" # externally/natively set form-like (0xa57) elif scene_only_writers: cat = "choice-output" elif ev["reach_scenes"] >= 2 or ev["reach_total"] >= 2: cat = "story-flag" else: cat = "unknown" conf = "med" if (has_prog_writer and ev["reach_total"] >= 3) or ev["reach_scenes"] >= 10 else "low" return cat, conf def mine() -> dict[int, dict]: reach_total = collections.defaultdict(set) reach_scenes = collections.defaultdict(set) consts = collections.defaultdict(set) writers = collections.defaultdict(set) atypes = collections.defaultdict(set) for name, path in paths.scripts().items(): try: scr = sys4load.load(path) except Exception: continue is_scene = bool(SCENE_RE.match(name)) for ins in scr.instructions: op, a = ins.opcode, ins.args if op in CMP_OPS or op in LOGIC_OPS: idxs = [1, 2] elif op == JCC_OP: idxs = [0] else: idxs = [] for i in idxs: if i < len(a) and a[i][0] in GLOBAL_ATYPES: addr = a[i][1] reach_total[addr].add(name) atypes[addr].add(a[i][0]) if is_scene: reach_scenes[addr].add(name) for j in idxs: # compared-against immediates if j != i and j < len(a) and a[j][0] == 0: consts[addr].add(a[j][1]) if op == ASSIGN_OP and len(a) >= 1 and a[0][0] in GLOBAL_ATYPES: writers[a[0][1]].add(name) excl = _excluded_addrs() out: dict[int, dict] = {} for addr in reach_total: if addr in excl: continue w = writers.get(addr, set()) w_prog = {n for n in w if not SCENE_RE.match(n)} ev = {"address": f"0x{addr:x}", "reach_total": len(reach_total[addr]), "reach_scenes": len(reach_scenes.get(addr, set())), "consts": sorted(consts.get(addr, set())), "writers": sorted(w), "writers_progression": sorted(w_prog), "atypes": sorted(atypes[addr]), "near_universal": len(reach_scenes.get(addr, set())) >= NEAR_UNIVERSAL} ev["category"], ev["confidence"] = assign_category(ev) out[addr] = ev return out def write_candidates(cands: dict[int, dict]) -> Path: ranked = sorted(cands.values(), key=lambda e: (-e["reach_scenes"], -e["reach_total"])) paths.BUILD.mkdir(parents=True, exist_ok=True) p = paths.BUILD / "story-flags-candidates.json" p.write_text(json.dumps({"meta": {"note": "static branch-condition mining; review surface, " "not ground truth", "count": len(ranked)}, "candidates": ranked}, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") return p def main(argv=None): ap = argparse.ArgumentParser() ap.add_argument("--bootstrap", action="store_true") # implemented in Task 5 args = ap.parse_args(argv) cands = mine() if args.bootstrap: return bootstrap(cands) # Task 5 p = write_candidates(cands) story = sum(1 for e in cands.values() if e["category"] == "story-flag") print(f"mined {len(cands)} branch-read globals ({story} story-flag candidates) -> {p}") return 0 if __name__ == "__main__": sys.exit(main()) ``` Note: `main` references `bootstrap()` for `--bootstrap`, added in Task 5. Running with no flag is fully functional now. - [ ] **Step 4: Run test + miner** Run: `py -3.11 -X utf8 tools/test_globals.py` Expected: PASS — the six miner checks `ok`. Run: `py -3.11 -X utf8 tools/story_flags.py` Expected: `mined N branch-read globals (M story-flag candidates) -> …/build/story-flags-candidates.json`. - [ ] **Step 5: Eyeball the top candidates** Run: `py -3.11 -X utf8 -c "import json; d=json.load(open('build/story-flags-candidates.json',encoding='utf-8')); [print(c['address'], c['category'], 'scenes=%d'%c['reach_scenes'], 'consts=%s'%c['consts'][:6]) for c in d['candidates'][:15]]"` Expected: `0xa57`/`0xa58`/`0xa59` near the top with high `reach_scenes`; entries look plausible (booleans and small enums). - [ ] **Step 6: Commit** ```bash git add tools/story_flags.py tools/test_globals.py build/story-flags-candidates.json git commit -m "feat: static story-flag miner -> candidates JSON (Task 4) Co-Authored-By: Claude Opus 4.8 (1M context) " ``` --- ### Task 5: `story_flags.py --bootstrap` — seed skeletons into `globals.toml` Append skeleton entries for newly discovered addresses, preserving all hand edits. **Files:** - Modify: `tools/story_flags.py` (add `bootstrap()`, `skeleton_toml()`) - Modify: `tools/test_globals.py` (add bootstrap test using a temp toml) **Interfaces:** - Consumes: `mine()` (Task 4), `globals_build.load_toml` (Task 1). - Produces: `story_flags.bootstrap(cands, toml_path=…) -> int` — appends only new-address skeletons (`source="auto-shape"`, confidence from evidence, `usage="TODO: …"`); idempotent; never rewrites existing entries. - [ ] **Step 1: Write the failing test** Add to `tools/test_globals.py`: ```python def test_bootstrap_is_additive_and_idempotent(): import tempfile, pathlib, story_flags # start from a copy of the real toml so curated entries are present src = (paths.VM_MAP / "globals.toml").read_text(encoding="utf-8") with tempfile.TemporaryDirectory() as d: tp = pathlib.Path(d) / "globals.toml" tp.write_text(src, encoding="utf-8") before, _ = G.load_toml(tp) cands = story_flags.mine() story_flags.bootstrap(cands, toml_path=tp) after, _ = G.load_toml(tp) check(len(after) > len(before), "bootstrap adds new skeleton entries") check(after[0xa57]["name"] == "lily_form_a", "bootstrap preserves curated 0xa57") # idempotent: second run adds nothing n1 = len(after) story_flags.bootstrap(story_flags.mine(), toml_path=tp) after2, _ = G.load_toml(tp) check(len(after2) == n1, "second bootstrap is a no-op (idempotent)") # every skeleton is auto-shape and not high-confidence added = set(after) - set(before) check(all(after[a]["source"] == "auto-shape" for a in added), "skeletons are source=auto-shape") check(all(after[a]["confidence"] != "high" for a in added), "skeletons never high confidence") ``` Add `test_bootstrap_is_additive_and_idempotent()` to `__main__`. (Also add `import globals_build as G` is already at top; `story_flags` imported inside the test.) - [ ] **Step 2: Run test to verify it fails** Run: `py -3.11 -X utf8 tools/test_globals.py` Expected: FAIL — `AttributeError: module 'story_flags' has no attribute 'bootstrap'`. - [ ] **Step 3: Implement `bootstrap` + `skeleton_toml` in `story_flags.py`** Add to `tools/story_flags.py` (above `main`): ```python import globals_build as _gb def skeleton_toml(ev: dict) -> str: consts = ", ".join(str(c) for c in ev["consts"][:8]) domain = "{0,1}" if set(ev["consts"]) <= {0, 1} and ev["consts"] else (f"one of {{{consts}}}" if consts else "?") usage = (f"TODO: confirm. Branch-read in {ev['reach_scenes']} scenes / {ev['reach_total']} scripts; " f"compared against [{consts}]; " f"writers={ev['writers'][:4] or 'none (external/native?)'}.") lines = ["[[global]]", f'address = "{ev["address"]}"', f'name = ""', f'category = "{ev["category"]}"', 'type = "int"', f'value_domain = "{domain}"', f'usage = "{usage}"', 'source = "auto-shape"', f'confidence = "{ev["confidence"]}"', "depends_on = []"] return "\n".join(lines) + "\n" def bootstrap(cands: dict[int, dict], toml_path=None) -> int: toml_path = Path(toml_path) if toml_path else (paths.VM_MAP / "globals.toml") present = set(_gb.load_toml(toml_path)[0]) if toml_path.exists() else set() # only story-flag / ui-toggle / choice-output candidates are worth seeding for curation seedable = {a: e for a, e in cands.items() if e["category"] in ("story-flag", "ui-toggle", "choice-output") and a not in present} if not seedable: print(f"bootstrap: nothing new to add ({len(present)} already present).") return 0 blocks = [skeleton_toml(seedable[a]) for a in sorted(seedable)] with toml_path.open("a", encoding="utf-8") as f: f.write("\n" + "\n".join(blocks)) print(f"bootstrap: appended {len(blocks)} skeletons -> {toml_path}") return 0 ``` The `name = ""` skeletons parse fine but should be named before they add value; leaving `name` empty means `sys4load` keeps the auto label until curated (curated-with-empty-name is treated as auto by the label rule in Task 3, which requires `name`). That's intended: a skeleton is a to-do, not a finished label. - [ ] **Step 4: Run test to verify it passes** Run: `py -3.11 -X utf8 tools/test_globals.py` Expected: PASS — all bootstrap checks `ok`, `0 failures`. - [ ] **Step 5: Verify against the real file is non-destructive (dry check)** Run: `py -3.11 -X utf8 -c "import sys; sys.path.insert(0,'tools'); import story_flags,globals_build as g; before=set(g.load_toml('vm-map/globals.toml')[0]); print('curated now:', len(before))"` Expected: prints the curated count (13). Do **not** run `--bootstrap` on the real file yet — that happens deliberately in Task 6 under review. - [ ] **Step 6: Commit** ```bash git add tools/story_flags.py tools/test_globals.py git commit -m "feat: story_flags --bootstrap seeds skeletons into globals.toml (Task 5) Co-Authored-By: Claude Opus 4.8 (1M context) " ``` --- ### Task 6: Curate strong candidates + docs + status memory Run the miner against the real registry, promote the confident findings to named entries, rebuild, and update all documentation homes. **Files:** - Modify: `vm-map/globals.toml` (bootstrap + hand-name the strong candidates) - Regenerate: `build/globals.json`, `docs/global-reference.md`, `build/story-flags-candidates.json` - Modify: `docs/name-resolution.md` (§2 registry description + new "Story-state flags" subsection) - Modify: `CLAUDE.md` (canonical-documents map row + single-source table row) - Modify: `docs/tools-reference.md` (add `globals_build.py`, `story_flags.py`, `test_globals.py`) - Modify: `~/.claude/…/memory/himegari-port-status.md` + `MEMORY.md` (status entry) **Interfaces:** - Consumes: all prior tasks. - Produces: a populated registry + current docs. No new code interfaces. - [ ] **Step 1: Bootstrap the real registry** Run: `py -3.11 -X utf8 tools/story_flags.py --bootstrap` Expected: `bootstrap: appended K skeletons -> …/vm-map/globals.toml`. Run: `py -3.11 -X utf8 tools/globals_build.py --lint` Expected: `lint: 0 errors, …` (skeletons are auto-shape + non-high, so they lint clean). - [ ] **Step 2: Hand-name the confident candidates** Open `build/story-flags-candidates.json`. For each candidate with `confidence == "med"` (strong reach and/or a progression writer), edit its skeleton in `vm-map/globals.toml`: set a `name` (snake_case), tighten `value_domain`, replace the `TODO:` `usage` with a real description, and bump `source`/`confidence` if you are confident (e.g. `inference`/`med`). Leave weak `low`-confidence skeletons with empty `name` (they stay to-dos). Do not invent meaning you cannot support — an honest `unknown`-ish skeleton is fine. Guidance: cross-reference the reader scripts in a listing to infer purpose, e.g. `py -3.11 -X utf8 tools/sys4load.py ` and search for the address. Use `Age.Cli sweep 0xADDR=1` (the separate dynamic reach tool) to confirm a flag actually moves dialogue before naming it a story-flag. - [ ] **Step 3: Rebuild generated artifacts + full test** Run: `py -3.11 -X utf8 tools/globals_build.py --build` Expected: `build: N globals -> build/globals.json, docs/global-reference.md`. Run: `py -3.11 -X utf8 tools/story_flags.py` Expected: refreshed candidates JSON. Run: `py -3.11 -X utf8 tools/test_globals.py` Expected: PASS, `0 failures`. Run: `py -3.11 -X utf8 tools/sys4load.py --validate` Expected: `481/481` clean. - [ ] **Step 4: Update `docs/name-resolution.md`** In §2 ("The global-variable map"), after the existing "Partial map — BUILT (v1)" content, add a subsection describing the curated registry. Insert this text: ```markdown ### The curated registry — `vm-map/globals.toml` (2026-07-07) The v1 auto map (`build/global-var-map.json`) infers *shapes* but cannot recover branch-flag *meaning* — and is sometimes wrong (it labels `0xa57`, the Lily form-A story flag, as a "string-table"). The curated registry fixes this, modelled exactly on `vm-map/opcodes.toml`: - **`vm-map/globals.toml`** — the only hand-edited source. One `[[global]]` per known address: `name`, `category` (`story-flag`/`index-pointer`/`data-table`/`string-table`/`ui-toggle`/ `choice-output`/`counter`/`unknown`), `type`, `value_domain`, `usage`, and provenance (`source`/`confidence`/`depends_on`). - **`tools/globals_build.py --build`** merges curated entries *over* the auto map → `build/globals.json` (machine) + `docs/global-reference.md` (generated human view). `--lint` checks vocabulary, the auto-shape≠high rule, and dangling `depends_on`. `sys4load` reads `build/globals.json` for operand labels (curated names win; auto tail kept at high/med). #### Story-state flags (the first populated category) Story flags are scalar globals that ADV/progression logic *branches on* (chapter, character forms, choices) — a category the auto shape map never enumerated. **`tools/story_flags.py`** is a 100% static miner: it flags a global as a candidate when it feeds a comparison (`eq`/`ne`/`lt`/ `lte`/`gr`/`gre`), a logical (`and`/`or`), or a `jcc` condition, and is not a genuine table/index in the shape map. Per candidate it records compared-against constants (→ value domain), the writer set (progression-written but scene-read = strong story flag), total- and scene-reach, and near-universal (ADV-chrome) status → an auto category + confidence. Output: `build/story-flags-candidates.json` (review surface); `--bootstrap` appends skeletons to `globals.toml` for human naming. Dynamic confirmation of a flag's reach stays separate — `Age.Cli sweep 0xADDR=VAL`. Known anchors: `0x3234` chapter (enum 1..9, progression-read, scene-reach 0), `0xa57/8/9` Lily forms (boolean, scene-read, externally set). ``` - [ ] **Step 5: Update `CLAUDE.md` (two tables)** In the **canonical-documents map** table, add a row after the opcode row: ```markdown | Global variable semantics / addresses (names, story-flags) | `age-reimpl/vm-map/globals.toml` *(source; generated → `build/globals.json`, `docs/global-reference.md`)* | ``` In the **single source of truth** table, add a row: ```markdown | `vm-map/globals.toml` | `py -3.11 -X utf8 tools/globals_build.py --build` | `build/globals.json`, `docs/global-reference.md` | ``` - [ ] **Step 6: Update `docs/tools-reference.md`** Add rows (match the existing table's column shape — tool / purpose / reads / writes): ```markdown | `globals_build.py` | Merge curated globals.toml over the auto shape map; generate registry. | `vm-map/globals.toml`, `build/global-var-map.json` | `build/globals.json`, `docs/global-reference.md` | | `story_flags.py` | Static story-flag miner (branch-condition mining) + `--bootstrap`. | corpus, `build/global-var-map.json` | `build/story-flags-candidates.json`, appends `vm-map/globals.toml` | | `test_globals.py` | Unit tests for the globals registry + miner. | — | — | ``` - [ ] **Step 7: Commit code + docs** ```bash git add vm-map/globals.toml build/globals.json docs/global-reference.md build/story-flags-candidates.json docs/name-resolution.md CLAUDE.md docs/tools-reference.md git commit -m "feat: curate story-flags into registry + docs (Task 6) Co-Authored-By: Claude Opus 4.8 (1M context) " ``` - [ ] **Step 8: Update the status memory** Edit `~/.claude/projects/S--Game-Hacking-Eushully-Himegari/memory/himegari-port-status.md`: add a bullet under Phase B recording that the globals registry (`vm-map/globals.toml`) + static story-flag miner (`tools/story_flags.py`) landed — the curated home for global semantics (opcodes.toml analogue), story-flags its first category, `sys4load` now labels from the merged `build/globals.json`; N story-flag candidates catalogued, K named. Update the `MEMORY.md` one-line index entry to mention story-flag mapping is underway/done. Convert "2026-07-07" to the absolute date. (Memory files are outside the git repo — no commit needed.) --- ## Notes for the implementer - **Ordering matters:** Task 2's `build()` is referenced by Task 1's `main` but only *defined* in Task 2 — do not run `--build` until Task 2 is complete. Likewise `story_flags.bootstrap` is referenced in Task 4's `main` but defined in Task 5 — do not run `--bootstrap` until Task 5. - **`py -3.11 -X utf8` always** — omitting `-X utf8` will corrupt cp932 script text on Windows. - The candidates JSON is a *review surface*, not ground truth. Curation (naming) is a human judgment step (Task 6 Step 2); it is fine to leave most low-confidence skeletons unnamed. - If `tomllib` is missing (Python < 3.11), the tools fail loudly — this project standardizes on 3.11.