Define future modding and OMP architecture

docs/remake-architecture-and-roadmap.md

@@ -63,7 +63,7 @@ Three layers, cleanly separated:
 - **Runtime language.** VM core in **C#** (Godot's C# support) — a 1.5M-instruction fetch/execute
   loop is too hot for GDScript. Presentation, UI, mod tooling, and export targets use Godot. This
   is why **Godot now fits**: under the earlier "faithful port" framing it was overkill (you'd use
-  ~10% of it); under *remake/enhance/mod* its editor, UI toolkit, asset pipeline, GDScript modding,
+  ~10% of it); under *remake/enhance/mod* its editor, UI toolkit, asset pipeline, mod workbench,
   and multi-platform export all earn their keep. Current OS dependencies and the gates for future exports
   are tracked in `docs/platform-portability.md`; they do not expand the active Phase A scope.
 - **Toolchain vs runtime.** The runtime owns the canonical parser+VM (C#). The Python tools remain
@@ -81,42 +81,354 @@ Three layers, cleanly separated:
 
 ---
 
-## 3. How modding works with a bytecode VM
+## 3. Future modding architecture
 
-Modding is tiered from trivial to deep. The first two tiers cover the large majority of "proper
-modding" and need **no decompilation**.
+> **Status and scope:** this section is design guidance for Phases C–E, not an active implementation
+> task. The immediate priority remains a faithful, validated runtime. Preserve the seams this design
+> will need, but do not build the mod platform before the base game works.
 
-**Tier 1 — assets & data (easy, no tools needed beyond a text/image editor).**
-- *Asset overrides:* drop replacement textures/CGs/voices/BGM into a mod folder; the content loader
-  resolves mod → loose-override → archive (generalizing the engine's native override behavior).
-- *Data edits:* game data (skills/items/units/maps/stages) is **externalized to editable files**
-  (JSON) that the runtime loads, bootstrapped from our `*INIT` extraction. Rebalancing, new items,
-  new skills = editing JSON. No bytecode involved.
+The faithful VM is the **compatibility floor, not the feature ceiling**. Original scripts must keep
+their historical behavior, but mods targeting our runtime need not remain constrained to what the
+proprietary `AGE.EXE` could load. Extended mode may add virtual resources and scripts, namespaced mod
+state, host hooks and new services without changing the interpretation of unmodified AGE bytecode.
 
-**Tier 2 — logic (medium; asm-level or host-language).**
-- *Script patches:* disassemble → edit the `.age-asm` → reassemble to `.BIN` (Kelebek's project has
-  a reassembler to adapt). Mods ship patched/replacement scripts; the VM runs them unmodified.
-- *Host hooks:* a mod API lets mods register callbacks in **GDScript/C#** — fire before/after a
-  script, intercept an opcode, replace a script by id, react to events, add UI. Original scripts run
-  as-is; mods augment. (This is the BepInEx/script-extender model and avoids a bytecode compiler for
-  most behavioral mods.)
+### 3.1 Two execution modes
 
-**Tier 3 — a friendly modding language (stretch, later).**
-- A high-level decompiled DSL + a compiler back to bytecode, so mods are written in readable source.
-  This is a real compiler project and its quality is bounded by how complete the global-var map and
-  call-script resolution are. Realistic as a *later* milestone, not near-term.
+- **Compatibility mode** runs original content with the selected game/version profile's historical
+  semantics. A mod limited to replacement AGE scripts and original-format assets may also remain usable
+  with the proprietary engine, although that is not a product requirement.
+- **Extended mode** layers engine-owned services on top: mod manifests and load order, virtual asset and
+  script identities, runtime callbacks, semantic events, extra saved state, modern UI and optional new
+  script facilities. These mods target Open Maid Engine and are not expected to run under `AGE.EXE`.
 
-**Readability, concretely:** annotated disassembly is achievable today (opcode names + global-var
-aliases + eventually call-script names). Pseudo-decompilation for *reading* is feasible (demonstrated
-on RECOVER). Clean round-trippable *source* is Tier 3. So near-term "how readable" = well-annotated
-assembly + external data/assets + host hooks; the read-like-C dream is a stretch goal.
+The modes share one VM. Extended mode is an additive dispatcher and content layer, not a forked
+interpreter or a second implementation of the game rules.
 
-**Two enablers become load-bearing under this goal** (they were "polish" for a port):
-- **Global-var map** — modders must know what game state a global is to touch it safely.
-- **Call-script resolution** — needed both to *run* scripts and to *add/replace* scenes. **✅ SOLVED
-  (2026-07-07):** `call-script <id>` = a raw index into the SYS4INI file table (native-RE via Ghidra;
-  `docs/engine-re.md`, `name-resolution.md §1`), and the C# VM now **executes** it (loads the target
-  `.BIN` as a nested subroutine frame). Resolution + execution both done; see the status memory.
+### 3.2 Modding surfaces and realistic boundaries
+
+Use the least invasive surface that can express a change:
+
+| Surface | Intended use | Relative cost |
+|---|---|---|
+| Declarative assets/text/data | replacements, translations, balance values, load order | low |
+| Semantic events | common gameplay and presentation changes through named typed contexts | low–medium |
+| Named bytecode patch points | game-specific inline behavior for which no general event exists | medium |
+| AGE assembly patches | direct changes to original control flow and script behavior | medium–high |
+| Raw VM/opcode hooks | expert escape hatch and discovery tool | high/brittle |
+| New engine services | genuinely new UI, state or mechanics beyond the AGE ABI | engine-extension work |
+
+**High-confidence uses once the full runtime exists:** layered asset/text/audio/font replacement;
+presentation and accessibility changes; balance edits; bug fixes; cheats and challenge modes;
+randomizers; replacement dialogue and event branches; debugging overlays; and new ADV scenes built
+from existing operations. These are where the reimplementation improves most over the native loose-file
+and translation workflows: mods can be small, ordered, validated and composed without repacking the
+original archives.
+
+**Feasible but subsystem-sized:** adding rather than replacing skills/items/units; new maps or routes;
+new battle effects; larger rosters or inventories; cross-game content ports; and narrative total
+conversions that retain the original game systems. The VM can grow its storage, but original scripts
+also encode table strides, loop bounds, switches, UI capacity and save assumptions. Expanding a table
+therefore requires auditing all of its consumers; increasing a C# dictionary's capacity alone does not
+teach the bytecode that a new entry exists.
+
+**Effectively an engine/game fork:** real-time combat, multiplayer, a 3D conversion, or replacement of
+the SRPG rules with an unrelated genre. Open source makes these possible in the literal sense, but they
+are not ordinary VM mods and should not distort the base mod API.
+
+For data edits, prefer named **post-initialization patches** over replacing whole `*INIT` scripts where
+possible. The original initialization remains the compatibility oracle; the profile applies declarative
+changes to named globals/records afterward. Replacement is easy, while expansion retains the capacity
+caveats above. Longer-term external JSON data remains useful, but it should have an explicit mapping to
+the VM state rather than becoming an unsynchronized second source of truth.
+
+### 3.3 A layered authoring stack
+
+No single language should serve every layer:
+
+| Layer | Preferred representation |
+|---|---|
+| Mod identity, dependencies, load order, assets, data patches and hook selectors | declarative manifest/patch files |
+| Runtime behavior, event callbacks and UI glue | embedded scripting language; **Lua is the leading candidate**, pending a later integration spike |
+| Direct modification of original scripts | annotated AGE assembly + assembler |
+| Substantial new narrative/content authoring | optional future **Open Maid Script** scenario DSL |
+
+Lua and Open Maid Script solve different problems. Lua would run **alongside** the AGE VM and talk to a
+controlled engine API; it would not need to compile into AGE bytecode. Open Maid Script, if justified
+later, should be a narrow content language for scenes, dialogue, choices and common presentation rather
+than an attempt to replace Lua as a general-purpose language. Its eventual backend could be AGE
+bytecode, an engine-owned extended representation, or generated Lua coroutines; defer that choice until
+the ADV and hook APIs expose stable abstractions.
+
+AGE assembly remains important. It is the most exact route for changing original logic and the only
+route likely to preserve native-engine compatibility, but it should not be the normal requirement for
+behavioral mods.
+
+### 3.4 Hook selectors: identify inline code once
+
+Script-level hooks are useful (`before/after ADDEXP.BIN`) but insufficient because much AGE logic is
+inlined. A global `before opcode 0x55` callback followed by several global checks is technically
+powerful and ergonomically unacceptable. The profile/toolchain must pay the identification cost once
+so every mod author does not rediscover the same context.
+
+Every executable site exposed to mods needs a stable **original-site identity** containing at least:
+
+```text
+game/profile + script catalog identity + base-script fingerprint + original dword offset
+```
+
+Use the instruction's original body/dword offset, not its decoded-list index. Filename alone is not
+enough because loose patches, append content and game revisions may collide. The base fingerprint lets
+the loader reject an unsupported revision explicitly instead of silently attaching a hook to the wrong
+instruction.
+
+Profiles then map those sites or regions to durable names:
+
+```text
+himegari.experience.before_award
+himegari.experience.after_calculation
+himegari.battle.before_damage
+himegari.unit.on_level_up
+himegari.scene.before_transition
+```
+
+A named patch point records the ugly game-specific facts: exact scripts/sites, supported fingerprints,
+captured operands or globals, expected inputs/results and any revision variants. A mod consumes a typed
+event or named point and should not need to know that its value currently lives at `G[0x152616]`.
+
+When one implementation is repeated inline, resolve it **statically at mod load**, not on every VM
+step. A selector may match a short normalized instruction sequence with typed-operand constraints,
+wildcards/captures, nearby strings/comments/calls/labels and an expected match count. Resolution produces
+an O(1) runtime table:
+
+```text
+(script identity, original offset) -> patch-point id(s)
+```
+
+Start with exact/syntactic patterns. Only add local-number normalization or control-flow-aware matching
+when real mods demonstrate the need; a general semantic decompiler is not a prerequisite. Zero or
+ambiguous matches are hard validation errors. Never guess a nearby site.
+
+Many useful changes concern a calculation **region**, not one opcode. A region patch point can expose
+entry/exit sites, captured inputs and a result lvalue, enabling `before`, `after`, or `override result`
+behavior. Result override is safer and more composable than skipping an arbitrary instruction range.
+Whole-region replacement remains an advanced operation because omitted side effects or temporary-state
+updates can violate script assumptions.
+
+### 3.5 Preserve identities and detect conflicts
+
+Resolve all profile selectors against the pristine decoded base script before applying mod transforms:
+
+```text
+load and fingerprint base script
+    -> decode and assign immutable original-site ids
+    -> resolve profile and mod selectors
+    -> apply ordered declarative transforms
+    -> attach callbacks to original-site ids
+    -> execute
+```
+
+Inserted instructions receive mod-owned identities; later insertions do not move another mod's target.
+A full replacement `.BIN` is intentionally coarse-grained: two mods replacing the same script generally
+cannot be merged and should produce a clear conflict. Named hooks, data patches and constrained
+instruction transforms are the composable path. Load order must be deterministic, visible and recorded
+in saves/diagnostics.
+
+The compatibility contract is deliberately limited:
+
+> Open Maid Engine detects structural conflicts it can prove, reports interactions it can observe and
+> makes composition order deterministic. Absence of a detected conflict does **not** imply that mods are
+> behaviorally compatible.
+
+Classify interactions rather than returning one misleading compatible/incompatible bit:
+
+- **Hard conflict:** overlapping replacement/deletion, incompatible whole-script replacements, an
+  unsatisfied dependency or a selector that is invalid for the installed base.
+- **Ordered interaction:** multiple inserts at one anchor, callbacks on the same hook point, or multiple
+  data operations whose declared order determines the result.
+- **Override:** more than one package supplies one logical asset/script and the visible load order chooses
+  a winner.
+- **Potential semantic interaction:** different sites touch the same named state, connected control-flow
+  regions or related events. Report when known, but do not pretend it proves incompatibility.
+- **Unknown/opaque behavior:** Lua or other extension code whose effects cannot be established statically,
+  especially code using the raw `maid.vm` surface.
+
+Two mods can edit disjoint instructions yet still invalidate each other's assumptions, redirect flow
+away from a hook, or mutate the same state through unrelated sites. Conversely, two callbacks at the same
+event may compose intentionally. Mechanical overlap is valuable evidence, not a compatibility proof.
+
+Improve runtime diagnosis by attributing normal-API behavior to the responsible mod: registered hook
+points and callback order, named globals/state read or changed, event values before/after each callback,
+asset/script resolution, cancellation/control-flow overrides and exceptions. An optional development
+trace should make chains such as `base 50 -> difficulty mod 35 -> double-exp mod 70` explicit. Raw VM
+access remains less observable and should be marked advanced/opaque rather than falsely analyzed.
+
+Raw opcode hooks remain available as the expert escape hatch, but their filters should be declarative
+and compiled into site sets—for example opcode + script + offset range + named global read/write + call
+context. Lua/C# callbacks should run only for resolved matches, never receive every arithmetic opcode and
+discard 99.99% themselves.
+
+### 3.6 Open Maid Pack (`.omp`)
+
+Use **Open Maid Pack (OMP)** as the canonical distribution and mount unit. An OMP is a specification and
+manifest layered on a standard ZIP container, not a new compression algorithm. Existing ZIP tools,
+checksums and libraries remain usable; already-compressed/streamed media can be stored without additional
+compression. The same layout must work unpacked as a development mod for source control and hot reload.
+
+Conceptual layout:
+
+```text
+example-mod.omp
+├── manifest.toml
+├── assets/
+├── data/
+├── text/
+├── lua/
+├── selectors/
+├── patches/
+├── open-maid-script/
+├── config/
+├── README.md
+└── LICENSE
+```
+
+The manifest owns mod id/name/version, OMP format version, engine-API range, supported game/profile and
+base fingerprints, dependencies/conflicts/load-order hints, requested capabilities, Lua entry points,
+selectors/transforms, virtual assets/scripts, configuration schema and saved-state schema/version.
+Package paths and ids are namespaced; no OMP writes into the original install or scatters loose files
+beside `AGE.EXE`.
+
+The content resolver mounts packages as ordered layers above the native loose/append/base sources and
+can show the full provider/override chain for every logical resource. Multiple asset providers are an
+explicit ordered override, not automatically a hard conflict. Runtime script hooks normally attach to
+the dispatcher without inserting AGE instructions; structural bytecode patches follow the pristine-base
+resolution and one-time link pipeline in §3.5.
+
+Patch operations should express intent (`insert before/after site`, `replace operand`, `replace/delete
+region`, `redirect branch`, `override result`) rather than shipping adjusted byte offsets. Inserts at one
+anchor can be deterministically ordered; edits to the same operand or overlapping replacement/deletion
+are conflicts. A patch that intentionally targets code introduced by another mod must declare that
+dependency and target an exported mod-owned site identity/version.
+
+Loading is transactional: validate archive paths/sizes, resolve dependencies and a deterministic order,
+verify base fingerprints, resolve all selectors, build the combined transform plan, report interactions,
+link final scripts/catalogs and only then start the game. Cache the linked result by base/profile plus the
+ordered package hashes and configuration; never cache by mutating game files.
+
+Record the resolved OMP ids, versions, hashes, order and relevant configuration with each save. A changed
+stack produces a clear warning, not a claim that the save is safe or unsafe. Packages are untrusted input:
+reject path traversal/case collisions/decompression abuse, do not load native binaries by default and
+grant filesystem/network/process/.NET capabilities only through explicit user approval. Signatures and a
+repository can come later; deterministic builds, hashes and strict validation come first.
+
+### 3.7 GUI-assisted discovery and authoring
+
+The future GUI should let an author work from either direction:
+
+1. Pick a curated profile event/patch point and generate a manifest entry plus handler stub.
+2. Browse annotated disassembly or the best supported pseudo-decompilation, select an instruction or
+   region, and have the tool generate a fingerprinted selector.
+3. Start from live gameplay: break on named-global reads/writes, inspect the current script/offset and
+   call stacks, record a trace region, search the corpus for similar regions, then export a selector
+   draft with expected match counts.
+4. Assemble assets/data/scripts, validate them against a chosen base and mod stack, preview the merged
+   script/resource plan, then build a deterministic `.omp` (or run the unpacked form directly).
+
+The ADV page locator already demonstrates the core vocabulary: translate a visible game location into a
+stable bytecode coordinate. A mod workbench generalizes that workflow. The GUI must show when a selector
+is exact, pattern-based, revision-specific, ambiguous or invalid; generated configuration remains plain
+text and reviewable rather than becoming an opaque editor project. Its conflict view should distinguish
+hard overlap, ordered callbacks/patches, overrides, potential semantic interactions and opaque runtime
+code, and show a merged-script view where practical.
+
+### 3.8 Runtime scripting API
+
+Lua is attractive for runtime mods because it is familiar, embeddable, dynamically loadable and already
+has mature editor support. It avoids requiring users to compile C# assemblies and keeps drop-in mods
+behind an explicit API boundary. The integration choice (managed interpreter versus native runtime) is a
+later portability/performance spike, not a decision to make during VM bring-up.
+
+Keep the selector outside the handler language so it can be validated before executing mod code:
+
+```toml
+[[hooks]]
+point = "himegari.experience.after_calculation"
+handler = "scripts/double_exp.lua:on_experience"
+priority = 100
+```
+
+Conceptual handler:
+
+```lua
+function on_experience(event)
+    event.amount = event.amount * 2
+end
+```
+
+The normal API should be named and capability-oriented:
+
+- `maid.events` — typed semantic events.
+- `maid.hooks` — curated and custom bytecode patch points.
+- `maid.game` — profile-defined named game-state access.
+- `maid.assets`, `maid.audio`, `maid.ui` — controlled host services.
+- `maid.state` — namespaced serializable mod state.
+- `maid.log` — attribution-aware diagnostics.
+- `maid.vm` — explicitly advanced raw scripts/globals/operands/call-stack access.
+
+Do not expose the VM dictionaries, arbitrary .NET reflection or raw Godot nodes as the normal API.
+Semantic events may be implemented using opaque globals underneath, but each mod should receive named
+values and controlled mutation points. The profile pays that reverse-engineering cost once.
+
+Runtime constraints are part of the contract:
+
+- VM-triggered callbacks execute on the VM thread; Godot/UI work crosses the host boundary to the main
+  thread. A callback never keeps a raw Godot object.
+- Arbitrary opcode/region hooks cannot yield. Only explicitly suspendable operations such as dialogue,
+  choices and waits may use coroutine-style scripting.
+- Each mod owns a serializable namespace. Initially persist only simple scalars, arrays and string-keyed
+  tables—not functions, engine handles or arbitrary object graphs.
+- Callback order is deterministic. Errors identify the mod, handler and bytecode context; time/instruction
+  budgets prevent accidental infinite callbacks.
+- Treat scripting as a capability boundary, not a perfect security sandbox. File/network/process/.NET
+  access is absent unless the user explicitly grants it.
+- Version the API and generated editor type stubs independently of game profiles.
+
+### 3.9 Load-bearing prerequisites already identified
+
+- **Global-var map** — named state access, selector captures and semantic events all depend on knowing
+  what hot globals mean.
+- **Call-script resolution — ✅ SOLVED (2026-07-07).** `call-script <id>` is a raw index into the
+  SYS4INI file table (`docs/engine-re.md`, `name-resolution.md §1`), and the C# VM executes it as a
+  nested frame. This supplies stable script/catalog identity and makes virtual script dispatch feasible.
+- **Trace/diagnostic seam — LANDED.** Frame enter/exit and per-instruction diagnostics already carry the
+  beginnings of script/site context. The future mod dispatcher should evolve beside this seam, not turn
+  diagnostics themselves into a mutable public API.
+- **Validated subsystem behavior.** Semantic hooks are trustworthy only where the underlying VM/profile
+  behavior has an independent correctness oracle. Modding does not relax the validation discipline in §5.
+
+### 3.10 Deferred implementation sequence
+
+When Phase C/D actually begins, proceed incrementally:
+
+1. Define OMP's language-independent manifest/layout, dependencies/load order and selector schema; support
+   both packed `.omp` and unpacked development directories.
+2. Preserve immutable script/catalog identity and original dword offsets throughout execution.
+3. Generalize diagnostics into a read-only, prefiltered execution observer.
+4. Implement profile-owned named patch points and strict fingerprint validation.
+5. Add `before`/`after` contexts, then safe result/write override; test the contracts directly in C#.
+6. Add load-time pattern resolution, transactional combination and classified interaction reporting.
+7. Put user scripting behind an `IModRuntime`-style boundary and spike Lua as the first runtime.
+8. Add typed semantic events where actual mod use cases justify them.
+9. Add namespaced saved state, virtual assets/scripts and controlled UI/audio services.
+10. Integrate AGE assembly/reassembly and instruction-level transforms.
+11. Consider Open Maid Script only after the stable ADV/event vocabulary is known.
+
+This ordering avoids baking Lua or a speculative DSL into the VM core. It also leaves room for another
+runtime later while making the selectors, event contracts and mod packages language-independent.
+
+**Readability, concretely:** annotated disassembly is available now (opcode names, raw offsets,
+call-script names and a growing global map). Pseudo-decompilation for reading is feasible and has been
+demonstrated on RECOVER. Clean round-trippable high-level source remains a compiler project and a stretch
+goal; none of the Tier-2 hook design depends on solving it.
 
 ---
 
@@ -150,14 +462,16 @@ the entry condition.
   real playthrough — reversible struct work).
 
 ### Phase C — Externalize & modding foundation
-- Move game data from bytecode-embedded tables to **editable external files** the runtime loads.
-- Generalize the **override/mod-loading** (mod folders, load order) from the engine's native
-  loose-file mechanism.
-- Asset pipeline: AGF↔PNG, audio, packaging. → Tier-1 modding works.
+- Add **editable named data overlays** mapped explicitly onto the VM's `*INIT`-produced state; external
+  files must not become an unsynchronized second source of truth.
+- Generalize the **override/mod-loading** from the engine's native loose-file mechanism into ordered OMP
+  mounts, deterministic dependency resolution and unpacked development packages (§3.6).
+- Asset pipeline: AGF↔PNG, audio and deterministic `.omp` packaging. → Tier-1 modding works.
 
 ### Phase D — Logic modding
 - Integrate the **assembler** (Tier-2 bytecode-patch mods) and ship the **host hook API**
-  (GDScript/C#). → Tier-2 modding works.
+  behind a language-independent runtime boundary; spike Lua as the first user-facing runtime per §3.
+  → Tier-2 modding works.
 - Optionally invest in decompiler quality toward Tier 3.
 
 ### Phase E — Enhance, polish, productize