OpenMaidEngine/docs/superpowers/specs/2026-07-06-opcode-reference-design.md

Design: Living Opcode Reference (single-source-of-truth + provenance)

Status: approved (design) · Date: 2026-07-06 · Author: session work Related: docs/phase-a-slice-plan.md, vm-map/himegari-opcode-notes.md, docs/PROJECT-STRUCTURE.md

Problem

Opcode knowledge for the AGE/SYS4 VM is currently spread across five artifacts:

File	Role today	Consumed by
tools/age_opcodes.py	Kelebek table verbatim: {op: (label, argc)} + arg-types	sys4load, vm0
tools/age_opcodes_himegari.py	our INFERRED dict (name/category/noop/confidence/method/note)	sys4load
vm-map/opcodes-himegari.json	data snapshot (arg-types, header fields, per-op array)	— (reference)
vm-map/himegari-opcode-notes.md	prose evidence, buckets A–F	— (humans)
build/opcode-coverage.md	coverage tiers	— (humans)

The same fact lives in several places, so they drift (this session already found stale "6/8 scenes" wording, and age_opcodes_himegari.py's method field duplicates prose in the notes doc). None of them record why we believe a claim or what other claims it rests on — so when a reversal proves one opcode wrong, there is no way to find the downstream inferences that depended on it.

Goals

1. One canonical, hand-edited file as the single source of truth; every other opcode artifact is generated from it and can never disagree with it or with the VM.
2. Provenance per claim — record the source (Kelebek / our harness / our investigation / Frida / Unicorn) and confidence, with the near-certain ABI facts kept separate from the fallible semantics.
3. Dependency tracking — a semantic claim can declare the opcodes whose interpretation it rests on, so a correction's blast radius is visible and mechanically checkable.
4. Zero disruption to working tooling — sys4load/vm0 keep importing the same Python interface, now generated.
5. Complete map — an entry for every one of the 248 opcodes Himegari uses, so coverage is measurable and any opcode can be named as a dependency.

Non-goals

• Not touching tools/age_opcodes.py (the Kelebek table): it stays pristine as the ABI baseline and the bootstrap input.
• No per-call-site type modelling (arg types are per-instruction in the bytecode, not fixed per opcode).
• No live-capture work (Frida/Unicorn) here — this is the reference structure; those remain sources a future entry can cite.
• Not covering the ~300 Kelebek opcodes Himegari never uses.

Architecture

One canonical file; everything else is generated:

vm-map/opcodes.toml            ← CANONICAL, hand-edited. [meta] + 248 [[opcode]] tables.
        │
        ▼  tools/opcodes_build.py         (generator + linter; stdlib tomllib read, no new dep)
        ├─ tools/age_opcodes_himegari.py  GENERATED — exposes INFERRED (sys4load imports unchanged)
        ├─ build/opcodes.json             GENERATED — language-neutral machine view (future C# VM)
        ├─ docs/opcode-reference.md        GENERATED — human reference incl. reverse dependents index
        └─ build/opcode-coverage.md        GENERATED — counts by source / confidence / category

• tools/age_opcodes.py (Kelebek) stays: ABI baseline + bootstrap input, never hand-edited.
• Retired (content migrates into opcodes.toml, then re-emitted): hand-maintained age_opcodes_himegari.py (now generated), vm-map/opcodes-himegari.json (→ build/opcodes.json), vm-map/himegari-opcode-notes.md (per-op evidence → each entry's details; rendered into docs/opcode-reference.md), and hand-maintained build/opcode-coverage.md.
• Editing loop: edit opcodes.toml → run opcodes_build.py → tooling, machine view, human doc, and coverage all update together.

Canonical file format (vm-map/opcodes.toml)

[meta] (migrated from opcodes-himegari.json)

• instruction_model — the 1 + 2*argc decode description (string).
• [meta.arg_types] — type-code → name (0x0→immediate, 0x2→string, 0x9→local-int, …).
• [meta.header_fields] — F0–F12 meanings.
• [meta.sources], [meta.confidence], [meta.categories] — controlled vocabularies (documented below), so the linter can reject unknown values.

[[opcode]] — ABI block (high trust)

field	meaning
op	opcode number (TOML 0x..)
label	Kelebek engine address (e.g. "u0041BEB0")
argc	argument count — validated by our 481/481 clean decode (len = 1 + 2*argc)
code_target_args	1-based arg indices that are jump targets (from Kelebek's notes; optional)
abi_source	e.g. "kelebek+decode-validated"
abi_note	optional caveat

[opcode.semantics] — our fallible layer (where cascades live)

field	meaning
name	short mnemonic for the disassembler (the one field sys4load requires)
category	marker|structural|control|adv|draw|audio|input|compute|unknown
summary	one-line description
noop_headless	bool — safe for the headless VM v1 to skip/fall-through (no state/visible effect)
source	kelebek|harness|investigation|frida|unicorn|inference
confidence	high|med|low
depends_on	list of op ids whose interpretation this claim rests on (default [])
evidence	concise grounding (one line)
details	optional multi-line long-form evidence (migrated from the notes doc)
confirm_by	optional — what would raise confidence or falsify this

[[opcode.semantics.args]] — per-arg roles (array of tables, optional)

field	meaning
i	1-based arg index
role	semantic role ("x", "y", "target:click", "count", …)
observed_types	list of type names actually seen at this position (auto-filled from corpus)
note	optional

Source vocabulary (grounding roots vs. fallible)

• kelebek — from the Kelebek table (ABI trusted; semantics from a later AGE title → treat as a lead, not truth). A root.
• harness — confirmed by our dialogue-oracle diff (build/text/dialogue.jsonl). Empirically grounded root.
• investigation — our own static/corpus analysis (may depends_on other ops).
• frida / unicorn — live runtime capture / micro-execution. Roots (direct observation).
• inference — reasoned from neighbours/context; weakest, usually has depends_on.

Generator + linter (tools/opcodes_build.py)

Subcommands:

• --bootstrap — create opcodes.toml: the used-opcode set is computed by scanning the corpus with sys4load (authoritative, reproducible); seed each with op/label/argc from age_opcodes.py, abi_source="kelebek+decode-validated" (argc is validated for all 248 by our 481/481 decode), semantics.source="kelebek", confidence = med if Kelebek names it else low (bare u004xxx/dev_ukn label → category="unknown"); auto-fill observed_types per arg position from the corpus scan. Idempotent: re-running preserves hand-edited entries (only fills missing).
• --build (default) — read opcodes.toml, run lint, then emit the four generated artifacts. The generated age_opcodes_himegari.py exposes INFERRED: dict[int, dict] with at least name (plus category/noop/confidence/source for future consumers) — a drop-in for the current interface.
• --lint — run the three checks; nonzero exit on any error.

Generated-file guard: each emitted file carries a DO NOT EDIT — generated from vm-map/opcodes.toml header.

Lint checks (the cascade mechanism)

1. Dangling-ref (error): every id in a depends_on must be an existing op. No orphan dependencies.
2. Reverse index / dependents (report): build A → [ops whose semantics depend on A]; render it in docs/opcode-reference.md as a "depended on by: 0x…" line per opcode, so a revision's blast radius is visible. Also emitted to the lint output.
3. Confidence-ceiling (warning): an entry's confidence may not exceed the minimum confidence among its depends_on targets (can't be high while resting on a low). Surfaces shaky foundations.
4. Vocabulary (error): category/source/confidence must be in the [meta] controlled lists.

Chains terminate at a kelebek/harness/frida/unicorn root or at direct evidence.

Bootstrap / migration procedure

1. Implement opcodes_build.py; run --bootstrap → opcodes.toml with 248 seeded skeletons + observed arg-type histograms.
2. Hand-migrate our real inferences into their entries: the ~26 in age_opcodes_himegari.py plus the evidence in himegari-opcode-notes.md (buckets A–F, incl. the 0x90/0x97 deep-dive), each with proper source, depends_on, evidence, details.
3. Run --build; confirm generated age_opcodes_himegari.py reproduces the current names, and sys4load … --validate + vm0.py --test + vm0.py --sweep are unchanged (differential check).
4. Delete the retired hand-maintained files; update docs/PROJECT-STRUCTURE.md and the memory index.

Testing / verification

• Regression (must be byte-for-byte where it matters): after migration, sys4load disassembly of a few scripts (MENU, SC0830) shows the same mnemonics; vm0.py --test PASS and --sweep still 282/294. This proves the generated shim is a true drop-in.
• Generator round-trip: --build is deterministic; re-running produces no diff.
• Lint unit checks: craft a tiny fixture TOML exercising dangling-ref (error), a confidence-ceiling violation (warning), and an unknown category (error).
• Coverage sanity: generated opcode-coverage.md totals reconcile with 248 used opcodes.

Risks / open questions

• TOML verbosity for 248 entries — acceptable; entries are mostly skeletons and diffs read cleanly.
• observed_types staleness — it's derived; re-run --bootstrap --refresh-observed (fills only that field) if the corpus set changes. Not load-bearing (evidence, not truth).
• Deciding confidence for Kelebek-named ops — Kelebek names are ABI-labels from another title; default seed = med for named, and we downgrade/confirm as the VM exercises them (harness).
• git — workspace isn't a git repo, so the design/spec aren't committed; offer git init later.

Out of scope (future)

• Emitting a C#-native binding from build/opcodes.json (A1 will consume the JSON directly).
• A GUI/queryable browser over the reference.
• Automated "review-needed" flags on git-diff of opcodes.toml (the dependents index already gives the manual signal).