From f049ce786e527f13624a717fd886d0361bb8ec62 Mon Sep 17 00:00:00 2001 From: gamer147 Date: Sat, 18 Jul 2026 17:17:47 -0400 Subject: [PATCH] Define future modding and OMP architecture --- docs/remake-architecture-and-roadmap.md | 386 +++++++++++++++++++++--- 1 file changed, 350 insertions(+), 36 deletions(-) diff --git a/docs/remake-architecture-and-roadmap.md b/docs/remake-architecture-and-roadmap.md index 7944fae..0470bb9 100644 --- a/docs/remake-architecture-and-roadmap.md +++ b/docs/remake-architecture-and-roadmap.md @@ -63,7 +63,7 @@ Three layers, cleanly separated: - **Runtime language.** VM core in **C#** (Godot's C# support) — a 1.5M-instruction fetch/execute loop is too hot for GDScript. Presentation, UI, mod tooling, and export targets use Godot. This is why **Godot now fits**: under the earlier "faithful port" framing it was overkill (you'd use - ~10% of it); under *remake/enhance/mod* its editor, UI toolkit, asset pipeline, GDScript modding, + ~10% of it); under *remake/enhance/mod* its editor, UI toolkit, asset pipeline, mod workbench, and multi-platform export all earn their keep. Current OS dependencies and the gates for future exports are tracked in `docs/platform-portability.md`; they do not expand the active Phase A scope. - **Toolchain vs runtime.** The runtime owns the canonical parser+VM (C#). The Python tools remain @@ -81,42 +81,354 @@ Three layers, cleanly separated: --- -## 3. How modding works with a bytecode VM +## 3. Future modding architecture -Modding is tiered from trivial to deep. The first two tiers cover the large majority of "proper -modding" and need **no decompilation**. +> **Status and scope:** this section is design guidance for Phases C–E, not an active implementation +> task. The immediate priority remains a faithful, validated runtime. Preserve the seams this design +> will need, but do not build the mod platform before the base game works. -**Tier 1 — assets & data (easy, no tools needed beyond a text/image editor).** -- *Asset overrides:* drop replacement textures/CGs/voices/BGM into a mod folder; the content loader - resolves mod → loose-override → archive (generalizing the engine's native override behavior). -- *Data edits:* game data (skills/items/units/maps/stages) is **externalized to editable files** - (JSON) that the runtime loads, bootstrapped from our `*INIT` extraction. Rebalancing, new items, - new skills = editing JSON. No bytecode involved. +The faithful VM is the **compatibility floor, not the feature ceiling**. Original scripts must keep +their historical behavior, but mods targeting our runtime need not remain constrained to what the +proprietary `AGE.EXE` could load. Extended mode may add virtual resources and scripts, namespaced mod +state, host hooks and new services without changing the interpretation of unmodified AGE bytecode. -**Tier 2 — logic (medium; asm-level or host-language).** -- *Script patches:* disassemble → edit the `.age-asm` → reassemble to `.BIN` (Kelebek's project has - a reassembler to adapt). Mods ship patched/replacement scripts; the VM runs them unmodified. -- *Host hooks:* a mod API lets mods register callbacks in **GDScript/C#** — fire before/after a - script, intercept an opcode, replace a script by id, react to events, add UI. Original scripts run - as-is; mods augment. (This is the BepInEx/script-extender model and avoids a bytecode compiler for - most behavioral mods.) +### 3.1 Two execution modes -**Tier 3 — a friendly modding language (stretch, later).** -- A high-level decompiled DSL + a compiler back to bytecode, so mods are written in readable source. - This is a real compiler project and its quality is bounded by how complete the global-var map and - call-script resolution are. Realistic as a *later* milestone, not near-term. +- **Compatibility mode** runs original content with the selected game/version profile's historical + semantics. A mod limited to replacement AGE scripts and original-format assets may also remain usable + with the proprietary engine, although that is not a product requirement. +- **Extended mode** layers engine-owned services on top: mod manifests and load order, virtual asset and + script identities, runtime callbacks, semantic events, extra saved state, modern UI and optional new + script facilities. These mods target Open Maid Engine and are not expected to run under `AGE.EXE`. -**Readability, concretely:** annotated disassembly is achievable today (opcode names + global-var -aliases + eventually call-script names). Pseudo-decompilation for *reading* is feasible (demonstrated -on RECOVER). Clean round-trippable *source* is Tier 3. So near-term "how readable" = well-annotated -assembly + external data/assets + host hooks; the read-like-C dream is a stretch goal. +The modes share one VM. Extended mode is an additive dispatcher and content layer, not a forked +interpreter or a second implementation of the game rules. -**Two enablers become load-bearing under this goal** (they were "polish" for a port): -- **Global-var map** — modders must know what game state a global is to touch it safely. -- **Call-script resolution** — needed both to *run* scripts and to *add/replace* scenes. **✅ SOLVED - (2026-07-07):** `call-script ` = a raw index into the SYS4INI file table (native-RE via Ghidra; - `docs/engine-re.md`, `name-resolution.md §1`), and the C# VM now **executes** it (loads the target - `.BIN` as a nested subroutine frame). Resolution + execution both done; see the status memory. +### 3.2 Modding surfaces and realistic boundaries + +Use the least invasive surface that can express a change: + +| Surface | Intended use | Relative cost | +|---|---|---| +| Declarative assets/text/data | replacements, translations, balance values, load order | low | +| Semantic events | common gameplay and presentation changes through named typed contexts | low–medium | +| Named bytecode patch points | game-specific inline behavior for which no general event exists | medium | +| AGE assembly patches | direct changes to original control flow and script behavior | medium–high | +| Raw VM/opcode hooks | expert escape hatch and discovery tool | high/brittle | +| New engine services | genuinely new UI, state or mechanics beyond the AGE ABI | engine-extension work | + +**High-confidence uses once the full runtime exists:** layered asset/text/audio/font replacement; +presentation and accessibility changes; balance edits; bug fixes; cheats and challenge modes; +randomizers; replacement dialogue and event branches; debugging overlays; and new ADV scenes built +from existing operations. These are where the reimplementation improves most over the native loose-file +and translation workflows: mods can be small, ordered, validated and composed without repacking the +original archives. + +**Feasible but subsystem-sized:** adding rather than replacing skills/items/units; new maps or routes; +new battle effects; larger rosters or inventories; cross-game content ports; and narrative total +conversions that retain the original game systems. The VM can grow its storage, but original scripts +also encode table strides, loop bounds, switches, UI capacity and save assumptions. Expanding a table +therefore requires auditing all of its consumers; increasing a C# dictionary's capacity alone does not +teach the bytecode that a new entry exists. + +**Effectively an engine/game fork:** real-time combat, multiplayer, a 3D conversion, or replacement of +the SRPG rules with an unrelated genre. Open source makes these possible in the literal sense, but they +are not ordinary VM mods and should not distort the base mod API. + +For data edits, prefer named **post-initialization patches** over replacing whole `*INIT` scripts where +possible. The original initialization remains the compatibility oracle; the profile applies declarative +changes to named globals/records afterward. Replacement is easy, while expansion retains the capacity +caveats above. Longer-term external JSON data remains useful, but it should have an explicit mapping to +the VM state rather than becoming an unsynchronized second source of truth. + +### 3.3 A layered authoring stack + +No single language should serve every layer: + +| Layer | Preferred representation | +|---|---| +| Mod identity, dependencies, load order, assets, data patches and hook selectors | declarative manifest/patch files | +| Runtime behavior, event callbacks and UI glue | embedded scripting language; **Lua is the leading candidate**, pending a later integration spike | +| Direct modification of original scripts | annotated AGE assembly + assembler | +| Substantial new narrative/content authoring | optional future **Open Maid Script** scenario DSL | + +Lua and Open Maid Script solve different problems. Lua would run **alongside** the AGE VM and talk to a +controlled engine API; it would not need to compile into AGE bytecode. Open Maid Script, if justified +later, should be a narrow content language for scenes, dialogue, choices and common presentation rather +than an attempt to replace Lua as a general-purpose language. Its eventual backend could be AGE +bytecode, an engine-owned extended representation, or generated Lua coroutines; defer that choice until +the ADV and hook APIs expose stable abstractions. + +AGE assembly remains important. It is the most exact route for changing original logic and the only +route likely to preserve native-engine compatibility, but it should not be the normal requirement for +behavioral mods. + +### 3.4 Hook selectors: identify inline code once + +Script-level hooks are useful (`before/after ADDEXP.BIN`) but insufficient because much AGE logic is +inlined. A global `before opcode 0x55` callback followed by several global checks is technically +powerful and ergonomically unacceptable. The profile/toolchain must pay the identification cost once +so every mod author does not rediscover the same context. + +Every executable site exposed to mods needs a stable **original-site identity** containing at least: + +```text +game/profile + script catalog identity + base-script fingerprint + original dword offset +``` + +Use the instruction's original body/dword offset, not its decoded-list index. Filename alone is not +enough because loose patches, append content and game revisions may collide. The base fingerprint lets +the loader reject an unsupported revision explicitly instead of silently attaching a hook to the wrong +instruction. + +Profiles then map those sites or regions to durable names: + +```text +himegari.experience.before_award +himegari.experience.after_calculation +himegari.battle.before_damage +himegari.unit.on_level_up +himegari.scene.before_transition +``` + +A named patch point records the ugly game-specific facts: exact scripts/sites, supported fingerprints, +captured operands or globals, expected inputs/results and any revision variants. A mod consumes a typed +event or named point and should not need to know that its value currently lives at `G[0x152616]`. + +When one implementation is repeated inline, resolve it **statically at mod load**, not on every VM +step. A selector may match a short normalized instruction sequence with typed-operand constraints, +wildcards/captures, nearby strings/comments/calls/labels and an expected match count. Resolution produces +an O(1) runtime table: + +```text +(script identity, original offset) -> patch-point id(s) +``` + +Start with exact/syntactic patterns. Only add local-number normalization or control-flow-aware matching +when real mods demonstrate the need; a general semantic decompiler is not a prerequisite. Zero or +ambiguous matches are hard validation errors. Never guess a nearby site. + +Many useful changes concern a calculation **region**, not one opcode. A region patch point can expose +entry/exit sites, captured inputs and a result lvalue, enabling `before`, `after`, or `override result` +behavior. Result override is safer and more composable than skipping an arbitrary instruction range. +Whole-region replacement remains an advanced operation because omitted side effects or temporary-state +updates can violate script assumptions. + +### 3.5 Preserve identities and detect conflicts + +Resolve all profile selectors against the pristine decoded base script before applying mod transforms: + +```text +load and fingerprint base script + -> decode and assign immutable original-site ids + -> resolve profile and mod selectors + -> apply ordered declarative transforms + -> attach callbacks to original-site ids + -> execute +``` + +Inserted instructions receive mod-owned identities; later insertions do not move another mod's target. +A full replacement `.BIN` is intentionally coarse-grained: two mods replacing the same script generally +cannot be merged and should produce a clear conflict. Named hooks, data patches and constrained +instruction transforms are the composable path. Load order must be deterministic, visible and recorded +in saves/diagnostics. + +The compatibility contract is deliberately limited: + +> Open Maid Engine detects structural conflicts it can prove, reports interactions it can observe and +> makes composition order deterministic. Absence of a detected conflict does **not** imply that mods are +> behaviorally compatible. + +Classify interactions rather than returning one misleading compatible/incompatible bit: + +- **Hard conflict:** overlapping replacement/deletion, incompatible whole-script replacements, an + unsatisfied dependency or a selector that is invalid for the installed base. +- **Ordered interaction:** multiple inserts at one anchor, callbacks on the same hook point, or multiple + data operations whose declared order determines the result. +- **Override:** more than one package supplies one logical asset/script and the visible load order chooses + a winner. +- **Potential semantic interaction:** different sites touch the same named state, connected control-flow + regions or related events. Report when known, but do not pretend it proves incompatibility. +- **Unknown/opaque behavior:** Lua or other extension code whose effects cannot be established statically, + especially code using the raw `maid.vm` surface. + +Two mods can edit disjoint instructions yet still invalidate each other's assumptions, redirect flow +away from a hook, or mutate the same state through unrelated sites. Conversely, two callbacks at the same +event may compose intentionally. Mechanical overlap is valuable evidence, not a compatibility proof. + +Improve runtime diagnosis by attributing normal-API behavior to the responsible mod: registered hook +points and callback order, named globals/state read or changed, event values before/after each callback, +asset/script resolution, cancellation/control-flow overrides and exceptions. An optional development +trace should make chains such as `base 50 -> difficulty mod 35 -> double-exp mod 70` explicit. Raw VM +access remains less observable and should be marked advanced/opaque rather than falsely analyzed. + +Raw opcode hooks remain available as the expert escape hatch, but their filters should be declarative +and compiled into site sets—for example opcode + script + offset range + named global read/write + call +context. Lua/C# callbacks should run only for resolved matches, never receive every arithmetic opcode and +discard 99.99% themselves. + +### 3.6 Open Maid Pack (`.omp`) + +Use **Open Maid Pack (OMP)** as the canonical distribution and mount unit. An OMP is a specification and +manifest layered on a standard ZIP container, not a new compression algorithm. Existing ZIP tools, +checksums and libraries remain usable; already-compressed/streamed media can be stored without additional +compression. The same layout must work unpacked as a development mod for source control and hot reload. + +Conceptual layout: + +```text +example-mod.omp +├── manifest.toml +├── assets/ +├── data/ +├── text/ +├── lua/ +├── selectors/ +├── patches/ +├── open-maid-script/ +├── config/ +├── README.md +└── LICENSE +``` + +The manifest owns mod id/name/version, OMP format version, engine-API range, supported game/profile and +base fingerprints, dependencies/conflicts/load-order hints, requested capabilities, Lua entry points, +selectors/transforms, virtual assets/scripts, configuration schema and saved-state schema/version. +Package paths and ids are namespaced; no OMP writes into the original install or scatters loose files +beside `AGE.EXE`. + +The content resolver mounts packages as ordered layers above the native loose/append/base sources and +can show the full provider/override chain for every logical resource. Multiple asset providers are an +explicit ordered override, not automatically a hard conflict. Runtime script hooks normally attach to +the dispatcher without inserting AGE instructions; structural bytecode patches follow the pristine-base +resolution and one-time link pipeline in §3.5. + +Patch operations should express intent (`insert before/after site`, `replace operand`, `replace/delete +region`, `redirect branch`, `override result`) rather than shipping adjusted byte offsets. Inserts at one +anchor can be deterministically ordered; edits to the same operand or overlapping replacement/deletion +are conflicts. A patch that intentionally targets code introduced by another mod must declare that +dependency and target an exported mod-owned site identity/version. + +Loading is transactional: validate archive paths/sizes, resolve dependencies and a deterministic order, +verify base fingerprints, resolve all selectors, build the combined transform plan, report interactions, +link final scripts/catalogs and only then start the game. Cache the linked result by base/profile plus the +ordered package hashes and configuration; never cache by mutating game files. + +Record the resolved OMP ids, versions, hashes, order and relevant configuration with each save. A changed +stack produces a clear warning, not a claim that the save is safe or unsafe. Packages are untrusted input: +reject path traversal/case collisions/decompression abuse, do not load native binaries by default and +grant filesystem/network/process/.NET capabilities only through explicit user approval. Signatures and a +repository can come later; deterministic builds, hashes and strict validation come first. + +### 3.7 GUI-assisted discovery and authoring + +The future GUI should let an author work from either direction: + +1. Pick a curated profile event/patch point and generate a manifest entry plus handler stub. +2. Browse annotated disassembly or the best supported pseudo-decompilation, select an instruction or + region, and have the tool generate a fingerprinted selector. +3. Start from live gameplay: break on named-global reads/writes, inspect the current script/offset and + call stacks, record a trace region, search the corpus for similar regions, then export a selector + draft with expected match counts. +4. Assemble assets/data/scripts, validate them against a chosen base and mod stack, preview the merged + script/resource plan, then build a deterministic `.omp` (or run the unpacked form directly). + +The ADV page locator already demonstrates the core vocabulary: translate a visible game location into a +stable bytecode coordinate. A mod workbench generalizes that workflow. The GUI must show when a selector +is exact, pattern-based, revision-specific, ambiguous or invalid; generated configuration remains plain +text and reviewable rather than becoming an opaque editor project. Its conflict view should distinguish +hard overlap, ordered callbacks/patches, overrides, potential semantic interactions and opaque runtime +code, and show a merged-script view where practical. + +### 3.8 Runtime scripting API + +Lua is attractive for runtime mods because it is familiar, embeddable, dynamically loadable and already +has mature editor support. It avoids requiring users to compile C# assemblies and keeps drop-in mods +behind an explicit API boundary. The integration choice (managed interpreter versus native runtime) is a +later portability/performance spike, not a decision to make during VM bring-up. + +Keep the selector outside the handler language so it can be validated before executing mod code: + +```toml +[[hooks]] +point = "himegari.experience.after_calculation" +handler = "scripts/double_exp.lua:on_experience" +priority = 100 +``` + +Conceptual handler: + +```lua +function on_experience(event) + event.amount = event.amount * 2 +end +``` + +The normal API should be named and capability-oriented: + +- `maid.events` — typed semantic events. +- `maid.hooks` — curated and custom bytecode patch points. +- `maid.game` — profile-defined named game-state access. +- `maid.assets`, `maid.audio`, `maid.ui` — controlled host services. +- `maid.state` — namespaced serializable mod state. +- `maid.log` — attribution-aware diagnostics. +- `maid.vm` — explicitly advanced raw scripts/globals/operands/call-stack access. + +Do not expose the VM dictionaries, arbitrary .NET reflection or raw Godot nodes as the normal API. +Semantic events may be implemented using opaque globals underneath, but each mod should receive named +values and controlled mutation points. The profile pays that reverse-engineering cost once. + +Runtime constraints are part of the contract: + +- VM-triggered callbacks execute on the VM thread; Godot/UI work crosses the host boundary to the main + thread. A callback never keeps a raw Godot object. +- Arbitrary opcode/region hooks cannot yield. Only explicitly suspendable operations such as dialogue, + choices and waits may use coroutine-style scripting. +- Each mod owns a serializable namespace. Initially persist only simple scalars, arrays and string-keyed + tables—not functions, engine handles or arbitrary object graphs. +- Callback order is deterministic. Errors identify the mod, handler and bytecode context; time/instruction + budgets prevent accidental infinite callbacks. +- Treat scripting as a capability boundary, not a perfect security sandbox. File/network/process/.NET + access is absent unless the user explicitly grants it. +- Version the API and generated editor type stubs independently of game profiles. + +### 3.9 Load-bearing prerequisites already identified + +- **Global-var map** — named state access, selector captures and semantic events all depend on knowing + what hot globals mean. +- **Call-script resolution — ✅ SOLVED (2026-07-07).** `call-script ` is a raw index into the + SYS4INI file table (`docs/engine-re.md`, `name-resolution.md §1`), and the C# VM executes it as a + nested frame. This supplies stable script/catalog identity and makes virtual script dispatch feasible. +- **Trace/diagnostic seam — LANDED.** Frame enter/exit and per-instruction diagnostics already carry the + beginnings of script/site context. The future mod dispatcher should evolve beside this seam, not turn + diagnostics themselves into a mutable public API. +- **Validated subsystem behavior.** Semantic hooks are trustworthy only where the underlying VM/profile + behavior has an independent correctness oracle. Modding does not relax the validation discipline in §5. + +### 3.10 Deferred implementation sequence + +When Phase C/D actually begins, proceed incrementally: + +1. Define OMP's language-independent manifest/layout, dependencies/load order and selector schema; support + both packed `.omp` and unpacked development directories. +2. Preserve immutable script/catalog identity and original dword offsets throughout execution. +3. Generalize diagnostics into a read-only, prefiltered execution observer. +4. Implement profile-owned named patch points and strict fingerprint validation. +5. Add `before`/`after` contexts, then safe result/write override; test the contracts directly in C#. +6. Add load-time pattern resolution, transactional combination and classified interaction reporting. +7. Put user scripting behind an `IModRuntime`-style boundary and spike Lua as the first runtime. +8. Add typed semantic events where actual mod use cases justify them. +9. Add namespaced saved state, virtual assets/scripts and controlled UI/audio services. +10. Integrate AGE assembly/reassembly and instruction-level transforms. +11. Consider Open Maid Script only after the stable ADV/event vocabulary is known. + +This ordering avoids baking Lua or a speculative DSL into the VM core. It also leaves room for another +runtime later while making the selectors, event contracts and mod packages language-independent. + +**Readability, concretely:** annotated disassembly is available now (opcode names, raw offsets, +call-script names and a growing global map). Pseudo-decompilation for reading is feasible and has been +demonstrated on RECOVER. Clean round-trippable high-level source remains a compiler project and a stretch +goal; none of the Tier-2 hook design depends on solving it. --- @@ -150,14 +462,16 @@ the entry condition. real playthrough — reversible struct work). ### Phase C — Externalize & modding foundation -- Move game data from bytecode-embedded tables to **editable external files** the runtime loads. -- Generalize the **override/mod-loading** (mod folders, load order) from the engine's native - loose-file mechanism. -- Asset pipeline: AGF↔PNG, audio, packaging. → Tier-1 modding works. +- Add **editable named data overlays** mapped explicitly onto the VM's `*INIT`-produced state; external + files must not become an unsynchronized second source of truth. +- Generalize the **override/mod-loading** from the engine's native loose-file mechanism into ordered OMP + mounts, deterministic dependency resolution and unpacked development packages (§3.6). +- Asset pipeline: AGF↔PNG, audio and deterministic `.omp` packaging. → Tier-1 modding works. ### Phase D — Logic modding - Integrate the **assembler** (Tier-2 bytecode-patch mods) and ship the **host hook API** - (GDScript/C#). → Tier-2 modding works. + behind a language-independent runtime boundary; spike Lua as the first user-facing runtime per §3. + → Tier-2 modding works. - Optionally invest in decompiler quality toward Tier 3. ### Phase E — Enhance, polish, productize