diff --git a/docs/superpowers/specs/2026-07-07-globals-registry-and-story-flags-design.md b/docs/superpowers/specs/2026-07-07-globals-registry-and-story-flags-design.md new file mode 100644 index 0000000..8fcb71d --- /dev/null +++ b/docs/superpowers/specs/2026-07-07-globals-registry-and-story-flags-design.md @@ -0,0 +1,218 @@ +# Globals registry (`globals.toml`) + story-flag miner — design + +**Date:** 2026-07-07 +**Status:** approved (brainstorming) → ready for implementation plan +**Slice:** Phase B story-state layer, static-catalog sub-slice (#1 of the story-state work). + +--- + +## 1. Problem & goal + +The VM addresses one flat **global memory bank** by raw offset (`global-int 0x3234`). A subset of +those globals are **story-state flags** — scalar values that ADV scene logic *branches on* (chapter, +Lily forms, choices, UI toggles). Seeding one changes the executed dialogue (`0xa57=1` → SC0000 +186→229 lines), so they are the lever behind cross-scene state divergence and the bg/sprite drift. + +**Today there is no structured catalog and no curated semantic home for globals.** What exists: + +- `build/global-var-map.json/md` — **auto-generated shape inference** (`tools/global_map.py`), from + access patterns + which `*INIT` script wrote each address. Regenerated wholesale; not hand-editable; + no curated semantics; no provenance model. **Actively wrong on story flags**: it labels `0xa57` + (the Lily form-A flag) as `string-table (written by EBINIT)` because it saw `EBINIT` touch the + address. Shape inference cannot recover branch-flag meaning — only curation can. +- Scattered prose mentions of the few known flags in `docs/name-resolution.md`, `phase-a-slice-plan.md`, + the Phase B spec, and the status memory. No table, no single source. + +**Goal.** Build the missing sibling to `vm-map/opcodes.toml`: a **hand-curated, single-source semantic +registry for globals**, generated into machine + human views and wired into `sys4load` labeling. +Story-flags are its first fully-populated category, discovered by a separate **static** miner that +proposes skeleton entries for human naming. + +**Non-goals (this slice).** +- No dynamic validation in the catalog build. The existing `Age.Cli sweep 0xADDR=VAL` reach-explorer + stays a **separate, on-demand** tool for confirming a flag moves dialogue. (Seed-sweeping every + candidate × 297 scenes × 2 runs is far too expensive to bake into a build.) +- No SCJUMP decode and no `call-script` id resolution (both remain deferred, tracked in + `name-resolution.md §1`). SCJUMP is only referenced as a *writer source* when hand-curating. +- No Frida / runtime observation (that layer is shelved; see `docs/global-memory-re.md`). +- Not attempting to hand-name all 16k auto-labelled globals — curation is incremental, hottest first. + +--- + +## 2. Architecture — three pieces + +| Piece | Role | Models after | +|---|---|---| +| `vm-map/globals.toml` | hand-edited **source of truth** for curated globals | `vm-map/opcodes.toml` | +| `tools/globals_build.py` | merge curated-over-auto → generate JSON/MD + lint | `tools/opcodes_build.py` | +| `tools/story_flags.py` | **static** miner: discover story-flag candidates + evidence + `--bootstrap` | `tools/opcode_context.py` + opcodes `--bootstrap` | + +Data flow: + +``` +tools/global_map.py ──► build/global-var-map.json (auto shape inference; existing, unchanged) + │ +vm-map/globals.toml ─────┐ │ + (curated source) ├─ merge (curated wins) ─► tools/globals_build.py --build +tools/story_flags.py ────┘ │ + (--bootstrap seeds skeletons ├─► build/globals.json (machine, merged, provenance-tagged) + back into globals.toml) └─► docs/global-reference.md (human, GENERATED) + │ + tools/sys4load.py reads build/globals.json for operand labels +``` + +--- + +## 3. `vm-map/globals.toml` schema + +Array-of-tables, one `[[global]]` per **known/curated** address. Mirrors opcodes.toml's provenance +fields (`source`/`confidence`/`depends_on`). + +```toml +# generated-consumers: build/globals.json, docs/global-reference.md (never hand-edit those) + +[[global]] +address = "0x3234" # canonical key; hex string, lowercase 0x +name = "chapter_mode" # snake_case identifier used in labels +category = "story-flag" # see vocabulary below +type = "int" # int | string | float +value_domain = "1..9" # free text; e.g. "{0,1}", "1..9", "resId" +usage = "Progression chapter/mode selector; SCJUMP's top-level switch keys on it." +source = "investigation" # investigation | inference | harness | auto-shape +confidence = "high" # high | med | low +depends_on = ["0x62ccf"] # optional: related addresses (choice outputs, sibling forms) +``` + +**`category` vocabulary:** `story-flag`, `index-pointer`, `data-table`, `string-table`, `ui-toggle`, +`choice-output`, `counter`, `unknown`. (Extensible; validated by lint against this list.) + +**`source` semantics:** `investigation` = confirmed by us (e.g. seed→divergence); `harness` = +confirmed by VM/test; `inference` = static reasoning, unconfirmed; `auto-shape` = machine-proposed +skeleton from the miner / shape map, awaiting human confirmation. + +**Confidence ceiling (lint):** an `auto-shape` source may not claim `high` confidence. + +--- + +## 4. `tools/story_flags.py` — the static miner + +Purely static; walks the 481-script disassembly via `sys4load` (reuses the loaded `Script` + +opcode table; no re-decode). No sweep, no runtime. + +### Candidate detection +A global address is a **story-flag candidate** when all hold: +1. **feeds a branch** — it is an operand of a comparison/logical opcode (`eq`/`ne`/`lt`/`gt`/`and`/`or`, + per the opcode table's classification) or a direct operand of `jcc` (0xa0); +2. **scalar** — read/written as a plain global operand (argtype 3), *not* as a 2D `base[i*stride+col]` + access (those are data-tables); +3. **not already a structural global** — excluded if `global_map.py`'s output classifies it as an + index-pointer, 2D-table base, or string-table base (dedupe against the auto map). + +### Evidence gathered per candidate +- **compared-against constants** — the immediate operands (argtype 0) it is compared with → infers + `value_domain` (`{0,1}` boolean flag vs `1..n` enum like `0x3234`); +- **writer set** — which scripts `mov` (0x55) / assign the address as an lvalue. Classification signal: + *written by progression/menu scripts (SCJUMP, SAVE, menu) but only read in SC/SP scenes* = externally + set → strong `story-flag`; *written and read within the same scene* = local `choice-output`/scratch; +- **reach** — count of distinct SC/SP scenes with a branch-read → breadth of influence; +- **chrome-vs-scene-unique** — whether all read sites are inside the shared ADV-chrome subroutine + (the byte-identical block carrying `0x6c9–0x6cd` toggles and `0x90` hotspots) or scene-specific. + +From this it auto-assigns a **sub-category** (`story-flag` / `ui-toggle` / `choice-output`) and a +**confidence** (never `high`; `auto-shape` source). + +### Outputs +- `build/story-flags-candidates.json` — always: ranked candidates with full evidence (read sites, + writers, compared constants, reach, category, confidence). This is the review surface. +- `--bootstrap` — appends skeleton `[[global]]` entries **only for addresses not already in + `globals.toml`**, `source = "auto-shape"`, low confidence, `usage` left as a `TODO:` stub for the + human. Idempotent; never overwrites a curated entry. (Mirrors `opcodes_build.py --bootstrap`.) + +--- + +## 5. `tools/globals_build.py` — merge & generate + +- `--build` — merge, then write generated files: + - **merge rule:** per address, a curated `globals.toml` entry wins; otherwise fall back to the + `build/global-var-map.json` auto entry. Output tags each with provenance (`curated` vs `auto`). + - `build/globals.json` — merged machine view (address → {name, category, type, value_domain, usage, + source, confidence, provenance}). Consumed by `sys4load` and, later, the C# VM. + - `docs/global-reference.md` — generated human view (header stamped "generated — do not edit"), + grouped by category, curated entries first. Parallels `docs/opcode-reference.md`. +- `--lint` — dangling `depends_on` (address not present in either source), unknown `category`, + confidence-ceiling violations (`auto-shape` claiming `high`), duplicate addresses. Mirrors + opcodes lint. + +Imports `tools/paths.py` for all paths (no hard-coding). + +--- + +## 6. `sys4load` wiring + +`sys4load` currently annotates global operands from `build/global-var-map.json`. Switch its label +source to the merged **`build/globals.json`** (curated names/categories win; auto shape labels remain +as fallback for the long tail). Degrades gracefully if the file is absent (same as today). Result: +curated globals render by name — e.g. `global-int 0x3234 =chapter_mode(story-flag)`, +`global-int 0xa57 =lily_form_a(story-flag)` — instead of the wrong auto shape label. + +Regenerate the `.asm` corpus (`tools/extract_phase2.py`) to pick up new labels, same as the existing +global-map flow. + +--- + +## 7. Docs & canonical-map + +- **Extend `docs/name-resolution.md §2`** (already the "global-variable map" home): describe the + curated `globals.toml` registry + `globals_build.py` build step + the merge-over-auto model, and add + a **"Story-state flags"** subsection (what they are, the miner, how to curate). The existing auto-map + narrative stays as the "auto shape inference" layer feeding the merge. +- **`docs/global-reference.md`** — new, but **generated** (owned by `globals_build.py`), so it is not a + hand-authored doc; it is the human view of the registry. +- **CLAUDE.md canonical-documents map** — add a row: + `Global variable semantics / addresses → vm-map/globals.toml` (source), paralleling the opcode row. +- **CLAUDE.md single-source-of-truth table** — add `vm-map/globals.toml` → `globals_build.py --build` + → `build/globals.json`, `docs/global-reference.md` (generated; do not edit). +- **`docs/tools-reference.md`** — add `story_flags.py` and `globals_build.py` (purpose/usage/I-O). + +--- + +## 8. First-pass registry content + +Hand-author the flags we already know into `globals.toml` (so the registry is useful on day one and +the miner has regression anchors): + +- `0x3234` `chapter_mode` (story-flag, `1..9`, SCJUMP switch) — high, investigation. +- `0xa57` / `0xa58` / `0xa59` `lily_form_a` / `_b` / `_c` (story-flag, `{0,1}`; exactly one set = + current form; gates form-specific voiced dialogue) — high, investigation; `depends_on` each other. +- `0x6c9`–`0x6cd` `ui_toggle_*` (ui-toggle, `{0,1}`; the 5 ADV-chrome buttons) — med, investigation. +- `0x0`, `0x62ccf` choice outputs (choice-output) — med/inference, from SCJUMP recon. +- High-purity index pointers `0xeff75`, `0x152616` (index-pointer) — med, from the auto map's purity + ranking (carries them over so the merged view is complete for the known-structural ones too). + +Then run `story_flags.py --bootstrap` to propose the rest for review. + +--- + +## 9. Testing — `tools/test_globals.py` + +- **schema/lint:** `globals.toml` parses; `--lint` passes (no dangling `depends_on`, valid categories, + no confidence-ceiling violations). +- **merge precedence:** curated `0xa57` in the merged `build/globals.json` reads `lily_form_a + (story-flag)`, overriding the auto map's `string-table` label. +- **miner regression anchors:** running `story_flags.py` over the corpus surfaces the *known* flags + `0x3234` and `0xa57` as story-flag candidates with nonzero reach (guards the detection heuristic). +- **generated == source:** `build/globals.json` reflects `globals.toml` (round-trip on a curated + entry's fields). +- **loader intact:** `sys4load --validate` still 481/481 clean with the new label source. + +--- + +## 10. Implementation order (for the plan) + +1. `globals.toml` skeleton + schema + hand-authored first-pass content (§8). +2. `globals_build.py` (`--build` merge + generate, `--lint`) + `build/globals.json` + + `docs/global-reference.md`. +3. Wire `sys4load` to `build/globals.json`; regenerate a sample `.asm`; verify 481/481. +4. `story_flags.py` (static miner → candidates JSON; `--bootstrap`). +5. `test_globals.py`; run miner, review candidates, curate the strong ones into `globals.toml`. +6. Docs: `name-resolution.md §2`, CLAUDE.md two tables, `tools-reference.md`. Update status memory.