Files

gamer147 1fbaa363f9 Split Godot host audio operations

2026-08-02 21:02:27 -04:00

69 KiB

Raw Blame History

AGE Remake — Architecture & Roadmap (option 3: remake / enhance / mod)

Decided goal (2026-07-06): not a translation patch (a translated build is already playable), not a bare cross-platform port — but an open reimplementation of the AGE engine that runs the original games and makes modding a first-class feature. Himegari (SYS4) is the first target; the design must extend to other AGE games and engine versions (SYS3/SYS5).

The right mental model is ScummVM / OpenMW for Eushully's AGE engine: we ship an engine; the user supplies the original game data they own; mods layer on top. Those projects prove this shape is feasible — and that it's a large, long-lived effort. Our decoding work (done) is the foundation; the runtime + backends + mod system is the bulk of the remaining work.

1. Guiding principles

Bytecode-faithful logic, pragmatic presentation. Run the original .BIN scripts on a reimplemented VM — you get every gameplay rule (damage, dungeon, battle, recovery) correct by construction, without re-deriving them. Reimplement the effectful ops (draw/text/audio/input) against a clean modern backend that looks right, not byte-identical to D3D9.
One engine, many profiles. Do NOT fork per game or per version. A shared VM core, with version front-ends (parser/codec/opcode table) and per-game profiles (maps, data schemas, asset rules) selected by a manifest.
Modding is architecture, not an afterthought. The data model, content loading, and script dispatch are designed so mods can override assets, edit data, patch scripts, and inject host-language hooks. The engine already hints at this: 49 loose root script .BIN files natively shadow their archived copies — a built-in override mechanism VFS-A generalizes.
The original owns the content; we own the engine. Users provide their AGE install; the runtime imports/loads it. This keeps us on the right side of distribution and mirrors ScummVM.
De-risk with vertical slices. Prove "run one scene end-to-end" before breadth. Nothing is real until a script executes and renders.

2. Target architecture — the split

Three layers, cleanly separated:

┌──────────────────────────────────────────────────────────────────────┐
│ RUNTIME  (Godot + C#)  — ships to players & modders                    │
│  ├─ AGE VM core (C#)      fetch/execute, typed var banks, control flow  │
│  ├─ Version front-ends    SYS3 / SYS4 / SYS5: header, string codec,     │
│  │                        opcode table, archive format                  │
│  ├─ Backend adapters      render(Godot 2D) · text/msg · audio · input   │
│  │                        — the effectful opcodes call into these       │
│  ├─ Content loader        ALF archives + loose overrides + mod folders  │
│  ├─ Data layer            game data from moddable files (bootstrapped    │
│  │                        from *INIT extraction)                         │
│  └─ Mod system            override resolution · hook API · patch loader  │
├──────────────────────────────────────────────────────────────────────┤
│ PROFILE / MANIFEST  (per game)  — data, not code                        │
│   engine_version, archives, string_codec, opcode_table_ref,             │
│   global_var_map, callscript_map, data_schemas, boot_entry,             │
│   asset_conversion rules                                                │
├──────────────────────────────────────────────────────────────────────┤
│ TOOLCHAIN  (Python — what we've already built)  — offline, for modders  │
│   disassembler · assembler · extractors · global-map builder ·          │
│   data exporters · mod packaging                                        │
└──────────────────────────────────────────────────────────────────────┘

Runtime language. VM core in C# (Godot's C# support) — a 1.5M-instruction fetch/execute loop is too hot for GDScript. Presentation, UI, mod tooling, and export targets use Godot. This is why Godot now fits: under the earlier "faithful port" framing it was overkill (you'd use ~10% of it); under remake/enhance/mod its editor, UI toolkit, asset pipeline, mod workbench, and multi-platform export all earn their keep. Current OS dependencies and the gates for future exports are tracked in docs/platform-portability.md; they do not expand the active Phase A scope.
Toolchain vs runtime. The runtime owns the canonical parser+VM (C#). The Python tools remain the offline analysis/authoring chain; they were the reference implementation and stay useful for modders. They share the format spec (documented), not code — acceptable for a small, stable container format.
Native content foundation status (2026-07-11). VFS-A runtime-parses the base SYS4 catalog and applies loose-first bounded ALF reads; VFS-B mounts selector-keyed S4AC append catalogs and resolves the native high-byte/low-24-bit id split; VFS-C decodes AGF directly to platform-neutral RGBA8 and feeds Godot without pre-extracted or pre-converted texture files. BGM, voice, and SFX now open catalog entries through the same store and decode OGG/WAV from bytes in Godot, so runtime scripts, textures, and audio no longer depend on extracted/.
Profile = manifest. Adding a game = a new profile + its maps. Adding an engine version = a new front-end plugin + profiles that reference it. See §5.

3. Future modding architecture

Status and scope: this section is design guidance for Phases C–E, not an active implementation task. The immediate priority remains a faithful, validated runtime. Preserve the seams this design will need, but do not build the mod platform before the base game works.

The faithful VM is the compatibility floor, not the feature ceiling. Original scripts must keep their historical behavior, but mods targeting our runtime need not remain constrained to what the proprietary AGE.EXE could load. Extended mode may add virtual resources and scripts, namespaced mod state, host hooks and new services without changing the interpretation of unmodified AGE bytecode.

3.1 Two execution modes

Compatibility mode runs original content with the selected game/version profile's historical semantics. A mod limited to replacement AGE scripts and original-format assets may also remain usable with the proprietary engine, although that is not a product requirement.
Extended mode layers engine-owned services on top: mod manifests and load order, virtual asset and script identities, runtime callbacks, semantic events, extra saved state, modern UI and optional new script facilities. These mods target Open Maid Engine and are not expected to run under AGE.EXE.

The modes share one VM. Extended mode is an additive dispatcher and content layer, not a forked interpreter or a second implementation of the game rules.

3.2 Modding surfaces and realistic boundaries

Use the least invasive surface that can express a change:

Surface	Intended use	Relative cost
Declarative assets/text/data	replacements, translations, balance values, load order	low
Semantic events	common gameplay and presentation changes through named typed contexts	low–medium
Named bytecode patch points	game-specific inline behavior for which no general event exists	medium
AGE assembly patches	direct changes to original control flow and script behavior	medium–high
Raw VM/opcode hooks	expert escape hatch and discovery tool	high/brittle
New engine services	genuinely new UI, state or mechanics beyond the AGE ABI	engine-extension work

High-confidence uses once the full runtime exists: layered asset/text/audio/font replacement; presentation and accessibility changes; balance edits; bug fixes; cheats and challenge modes; randomizers; replacement dialogue and event branches; debugging overlays; and new ADV scenes built from existing operations. These are where the reimplementation improves most over the native loose-file and translation workflows: mods can be small, ordered, validated and composed without repacking the original archives.

Feasible but subsystem-sized: adding rather than replacing skills/items/units; new maps or routes; new battle effects; larger rosters or inventories; cross-game content ports; and narrative total conversions that retain the original game systems. The VM can grow its storage, but original scripts also encode table strides, loop bounds, switches, UI capacity and save assumptions. Expanding a table therefore requires auditing all of its consumers; increasing a C# dictionary's capacity alone does not teach the bytecode that a new entry exists.

Effectively an engine/game fork: real-time combat, multiplayer, a 3D conversion, or replacement of the SRPG rules with an unrelated genre. Open source makes these possible in the literal sense, but they are not ordinary VM mods and should not distort the base mod API.

For data edits, prefer named post-initialization patches over replacing whole *INIT scripts where possible. The original initialization remains the compatibility oracle; the profile applies declarative changes to named globals/records afterward. Replacement is easy, while expansion retains the capacity caveats above. Longer-term external JSON data remains useful, but it should have an explicit mapping to the VM state rather than becoming an unsynchronized second source of truth.

3.3 A layered authoring stack

No single language should serve every layer:

Layer	Preferred representation
Mod identity, dependencies, load order, assets, data patches and hook selectors	declarative manifest/patch files
Runtime behavior, event callbacks and UI glue	embedded scripting language; Lua is the leading candidate, pending a later integration spike
Direct modification of original scripts	annotated AGE assembly + assembler
Substantial new narrative/content authoring	optional future Open Maid Script scenario DSL

Lua and Open Maid Script solve different problems. Lua would run alongside the AGE VM and talk to a controlled engine API; it would not need to compile into AGE bytecode. Open Maid Script, if justified later, should be a narrow content language for scenes, dialogue, choices and common presentation rather than an attempt to replace Lua as a general-purpose language. Its eventual backend could be AGE bytecode, an engine-owned extended representation, or generated Lua coroutines; defer that choice until the ADV and hook APIs expose stable abstractions.

AGE assembly remains important. It is the most exact route for changing original logic and the only route likely to preserve native-engine compatibility, but it should not be the normal requirement for behavioral mods.

3.4 Hook selectors: identify inline code once

Script-level hooks are useful (before/after ADDEXP.BIN) but insufficient because much AGE logic is inlined. A global before opcode 0x55 callback followed by several global checks is technically powerful and ergonomically unacceptable. The profile/toolchain must pay the identification cost once so every mod author does not rediscover the same context.

Every executable site exposed to mods needs a stable original-site identity containing at least:

game/profile + script catalog identity + base-script fingerprint + original dword offset

Use the instruction's original body/dword offset, not its decoded-list index. Filename alone is not enough because loose patches, append content and game revisions may collide. The base fingerprint lets the loader reject an unsupported revision explicitly instead of silently attaching a hook to the wrong instruction.

Profiles then map those sites or regions to durable names:

himegari.experience.before_award
himegari.experience.after_calculation
himegari.battle.before_damage
himegari.unit.on_level_up
himegari.scene.before_transition

A named patch point records the ugly game-specific facts: exact scripts/sites, supported fingerprints, captured operands or globals, expected inputs/results and any revision variants. A mod consumes a typed event or named point and should not need to know that its value currently lives at G[0x152616].

When one implementation is repeated inline, resolve it statically at mod load, not on every VM step. A selector may match a short normalized instruction sequence with typed-operand constraints, wildcards/captures, nearby strings/comments/calls/labels and an expected match count. Resolution produces an O(1) runtime table:

(script identity, original offset) -> patch-point id(s)

Start with exact/syntactic patterns. Only add local-number normalization or control-flow-aware matching when real mods demonstrate the need; a general semantic decompiler is not a prerequisite. Zero or ambiguous matches are hard validation errors. Never guess a nearby site.

Many useful changes concern a calculation region, not one opcode. A region patch point can expose entry/exit sites, captured inputs and a result lvalue, enabling before, after, or override result behavior. Result override is safer and more composable than skipping an arbitrary instruction range. Whole-region replacement remains an advanced operation because omitted side effects or temporary-state updates can violate script assumptions.

3.5 Preserve identities and detect conflicts

Resolve all profile selectors against the pristine decoded base script before applying mod transforms:

load and fingerprint base script
    -> decode and assign immutable original-site ids
    -> resolve profile and mod selectors
    -> apply ordered declarative transforms
    -> attach callbacks to original-site ids
    -> execute

Inserted instructions receive mod-owned identities; later insertions do not move another mod's target. A full replacement .BIN is intentionally coarse-grained: two mods replacing the same script generally cannot be merged and should produce a clear conflict. Named hooks, data patches and constrained instruction transforms are the composable path. Load order must be deterministic, visible and recorded in saves/diagnostics.

The compatibility contract is deliberately limited:

Open Maid Engine detects structural conflicts it can prove, reports interactions it can observe and makes composition order deterministic. Absence of a detected conflict does not imply that mods are behaviorally compatible.

Classify interactions rather than returning one misleading compatible/incompatible bit:

Hard conflict: overlapping replacement/deletion, incompatible whole-script replacements, an unsatisfied dependency or a selector that is invalid for the installed base.
Ordered interaction: multiple inserts at one anchor, callbacks on the same hook point, or multiple data operations whose declared order determines the result.
Override: more than one package supplies one logical asset/script and the visible load order chooses a winner.
Potential semantic interaction: different sites touch the same named state, connected control-flow regions or related events. Report when known, but do not pretend it proves incompatibility.
Unknown/opaque behavior: Lua or other extension code whose effects cannot be established statically, especially code using the raw maid.vm surface.

Two mods can edit disjoint instructions yet still invalidate each other's assumptions, redirect flow away from a hook, or mutate the same state through unrelated sites. Conversely, two callbacks at the same event may compose intentionally. Mechanical overlap is valuable evidence, not a compatibility proof.

Improve runtime diagnosis by attributing normal-API behavior to the responsible mod: registered hook points and callback order, named globals/state read or changed, event values before/after each callback, asset/script resolution, cancellation/control-flow overrides and exceptions. An optional development trace should make chains such as base 50 -> level-gap mod 25 -> double-exp mod 50 explicit. Raw VM access remains less observable and should be marked advanced/opaque rather than falsely analyzed.

Raw opcode hooks remain available as the expert escape hatch, but their filters should be declarative and compiled into site sets—for example opcode + script + offset range + named global read/write + call context. Lua/C# callbacks should run only for resolved matches, never receive every arithmetic opcode and discard 99.99% themselves.

3.6 Open Maid Pack (`.omp`)

Use Open Maid Pack (OMP) as the canonical distribution and mount unit. An OMP is a specification and manifest layered on a standard ZIP container, not a new compression algorithm. Existing ZIP tools, checksums and libraries remain usable; already-compressed/streamed media can be stored without additional compression. The same layout must work unpacked as a development mod for source control and hot reload.

Conceptual layout:

example-mod.omp
├── manifest.toml
├── assets/
├── data/
├── text/
├── lua/
├── selectors/
├── patches/
├── open-maid-script/
├── config/
├── README.md
└── LICENSE

The manifest owns mod id/name/version, OMP format version, engine-API range, supported game/profile and base fingerprints, dependencies/conflicts/load-order hints, requested capabilities, Lua entry points, selectors/transforms, virtual assets/scripts, configuration schema and saved-state schema/version. Package paths and ids are namespaced; no OMP writes into the original install or scatters loose files beside AGE.EXE.

The content resolver mounts packages as ordered layers above the native loose/append/base sources and can show the full provider/override chain for every logical resource. Multiple asset providers are an explicit ordered override, not automatically a hard conflict. Runtime script hooks normally attach to the dispatcher without inserting AGE instructions; structural bytecode patches follow the pristine-base resolution and one-time link pipeline in §3.5.

Patch operations should express intent (insert before/after site, replace operand, replace/delete region, redirect branch, override result) rather than shipping adjusted byte offsets. Inserts at one anchor can be deterministically ordered; edits to the same operand or overlapping replacement/deletion are conflicts. A patch that intentionally targets code introduced by another mod must declare that dependency and target an exported mod-owned site identity/version.

Loading is transactional: validate archive paths/sizes, resolve dependencies and a deterministic order, verify base fingerprints, resolve all selectors, build the combined transform plan, report interactions, link final scripts/catalogs and only then start the game. Cache the linked result by base/profile plus the ordered package hashes and configuration; never cache by mutating game files.

Record the resolved OMP ids, versions, hashes, order and relevant configuration with each save. A changed stack produces a clear warning, not a claim that the save is safe or unsafe. Packages are untrusted input: reject path traversal/case collisions/decompression abuse, do not load native binaries by default and grant filesystem/network/process/.NET capabilities only through explicit user approval. Signatures and a repository can come later; deterministic builds, hashes and strict validation come first.

3.7 GUI-assisted discovery and authoring

The future GUI should let an author work from either direction:

Pick a curated profile event/patch point and generate a manifest entry plus handler stub.
Browse annotated disassembly or the best supported pseudo-decompilation, select an instruction or region, and have the tool generate a fingerprinted selector.
Start from live gameplay: break on named-global reads/writes, inspect the current script/offset and call stacks, record a trace region, search the corpus for similar regions, then export a selector draft with expected match counts.
Assemble assets/data/scripts, validate them against a chosen base and mod stack, preview the merged script/resource plan, then build a deterministic .omp (or run the unpacked form directly).

The ADV page locator already demonstrates the core vocabulary: translate a visible game location into a stable bytecode coordinate. A mod workbench generalizes that workflow. The GUI must show when a selector is exact, pattern-based, revision-specific, ambiguous or invalid; generated configuration remains plain text and reviewable rather than becoming an opaque editor project. Its conflict view should distinguish hard overlap, ordered callbacks/patches, overrides, potential semantic interactions and opaque runtime code, and show a merged-script view where practical.

3.8 Runtime scripting API

Lua is attractive for runtime mods because it is familiar, embeddable, dynamically loadable and already has mature editor support. It avoids requiring users to compile C# assemblies and keeps drop-in mods behind an explicit API boundary. The integration choice (managed interpreter versus native runtime) is a later portability/performance spike, not a decision to make during VM bring-up.

Keep the selector outside the handler language so it can be validated before executing mod code:

[[hooks]]
point = "himegari.experience.after_calculation"
handler = "scripts/double_exp.lua:on_experience"
priority = 100

Conceptual handler:

function on_experience(event)
    event.amount = event.amount * 2
end

The normal API should be named and capability-oriented:

maid.events — typed semantic events.
maid.hooks — curated and custom bytecode patch points.
maid.game — profile-defined named game-state access.
maid.assets, maid.audio, maid.ui — controlled host services.
maid.state — namespaced serializable mod state.
maid.log — attribution-aware diagnostics.
maid.vm — explicitly advanced raw scripts/globals/operands/call-stack access.

Do not expose the VM dictionaries, arbitrary .NET reflection or raw Godot nodes as the normal API. Semantic events may be implemented using opaque globals underneath, but each mod should receive named values and controlled mutation points. The profile pays that reverse-engineering cost once.

Runtime constraints are part of the contract:

VM-triggered callbacks execute on the VM thread; Godot/UI work crosses the host boundary to the main thread. A callback never keeps a raw Godot object.
Arbitrary opcode/region hooks cannot yield. Only explicitly suspendable operations such as dialogue, choices and waits may use coroutine-style scripting.
Each mod owns a serializable namespace. Initially persist only simple scalars, arrays and string-keyed tables—not functions, engine handles or arbitrary object graphs.
Callback order is deterministic. Errors identify the mod, handler and bytecode context; time/instruction budgets prevent accidental infinite callbacks.
Treat scripting as a capability boundary, not a perfect security sandbox. File/network/process/.NET access is absent unless the user explicitly grants it.
Version the API and generated editor type stubs independently of game profiles.

3.9 Load-bearing prerequisites already identified

Global-var map — named state access, selector captures and semantic events all depend on knowing what hot globals mean.
Call-script resolution — ✅ SOLVED (2026-07-07). call-script <id> is a raw index into the SYS4INI file table (docs/engine-re.md, name-resolution.md §1), and the C# VM executes it as a nested frame. This supplies stable script/catalog identity and makes virtual script dispatch feasible.
Trace/diagnostic seam — LANDED. Frame enter/exit and per-instruction diagnostics already carry the beginnings of script/site context. The future mod dispatcher should evolve beside this seam, not turn diagnostics themselves into a mutable public API.
Validated subsystem behavior. Semantic hooks are trustworthy only where the underlying VM/profile behavior has an independent correctness oracle. Modding does not relax the validation discipline in §5.

3.10 Deferred implementation sequence

When Phase C/D actually begins, proceed incrementally:

Define OMP's language-independent manifest/layout, dependencies/load order and selector schema; support both packed .omp and unpacked development directories.
Preserve immutable script/catalog identity and original dword offsets throughout execution.
Generalize diagnostics into a read-only, prefiltered execution observer.
Implement profile-owned named patch points and strict fingerprint validation.
Add before/after contexts, then safe result/write override; test the contracts directly in C#.
Add load-time pattern resolution, transactional combination and classified interaction reporting.
Put user scripting behind an IModRuntime-style boundary and spike Lua as the first runtime.
Add typed semantic events where actual mod use cases justify them.
Add namespaced saved state, virtual assets/scripts and controlled UI/audio services.
Integrate AGE assembly/reassembly and instruction-level transforms.
Consider Open Maid Script only after the stable ADV/event vocabulary is known.

This ordering avoids baking Lua or a speculative DSL into the VM core. It also leaves room for another runtime later while making the selectors, event contracts and mod packages language-independent.

Readability, concretely: annotated disassembly is available now (opcode names, raw offsets, call-script names and a growing global map). Pseudo-decompilation for reading is feasible and has been demonstrated on RECOVER. Clean round-trippable high-level source remains a compiler project and a stretch goal; none of the Tier-2 hook design depends on solving it.

4. Roadmap — high-level progression

Each phase ends with something demonstrable. The documented side-tasks map into these phases (noted).

Phase A — Prove the VM (vertical slice) — completed

Goal: run one ADV scene end-to-end in the new runtime — background image + dialogue + a choice + a voice line — and match its show-text sequence to build/text/dialogue.jsonl. Forces, and thereby de-risks, every core unknown at once:

Port the container parser + VM core to C#.
Implement the ADV effectful ops against Godot: show-text, end-text-line, wait-for-input, set-font, play-voice, play-bgm, draw-texture/create-texture/draw-string, choices.
~~Resolve just enough call-script to enter/leave a scene~~ ✅ DONE — full call-script resolution + execution landed (nested subroutine frames sharing globals). Scene→scene chaining (decision→scene) still needs the SCJUMP decision→scene-id native hop.
AGF → texture for the one scene's art (side-task; AGF2BMP2AGF.exe already on disk).
Treat the classified no-op markers as skips; validate the tentative-no-op ops via the dialogue diff.

Phase B — Broaden coverage (playable ADV, then systems)

Execution order and decision gates are tracked in docs/phase-b-framework.md; Phase A completion remains the entry condition.

Implement the remaining effectful ops; Frida sessions for the opaque ones (the shortlist in build/opcode-coverage.md); Unicorn for 0x215-style computational ops.
Grow the global-var map (side-task 2.5: *MES writers → record-table readers → Frida field naming) — now a core enabler, not polish.
Get a full chapter of ADV playable; then the dungeon/battle/menu systems (they run as bytecode — we implement the ops they use, not the rules).
Side-tasks absorbed here: STINIT parser, remaining data schemas, save-file format (needed for a real playthrough — reversible struct work).

Persistence compatibility floor (decided 2026-07-24): the 1.0 path uses AGE's native binary domains and lifecycle—shared SAVE.DAT/SAVE.BAK, RT.DAT/RT.BAK, numbered SAVE##.DAT, and paired BMP .STH thumbnails. Keep the codec behind a profile/save-service boundary. Human-readable JSON inspection/export, migrations, and namespaced mod state are additive extended-mode work, not a replacement for compatibility-mode import/export. The recovered native contract lives in docs/engine-re.md. The common container codec/store and typed shared SAVE.DAT payload landed on 2026-07-24. Profile-owned selected integer/string cells now survive across scene VMs and are wired to their four native opcodes. The base/append catalog sections are now decoded as encrypted resource-seen markers: opcode 0x19d queries them, successful VFS opens add them, and compatibility saves write native-decodable tables. Native RT.DAT import/export and the packed-script/T1 ReadTextDB queue/commit/query lifecycle are also implemented, including message:ReadTextSkip ops 0x1ca/0x1cb and state query 0x1cc. Numbered active-frame state is now implemented in native layout 3: metadata query, paired .DAT/.STH lifecycle, exact native BMP thumbnail I/O, six global banks, retained surface/gfx state, history, and nested frame restoration through the 0xae rendezvous. The real SAVE.BIN script is covered end to end for listing and loading; the read-only installed-save gate now continues through CALLBACK_LOAD, reconstructs SYSTEM4.BIN → FORT.BIN, and reaches FORT's CHMENU gameplay poll. Full restoration replaces only the serialized mutable bank prefixes (preserving initialized unit/stage/string definitions), restores the retained BGM/SFX state, preserves initialized flag-zero system surfaces while overlaying explicit saved reload records, reproduces the native configuration-gated all-surface release when explicitly enabled, and writes native per-slot reload/created metadata rather than treating every texture as reloadable. Opcode 0x259 supplies its real script-entry reload-policy clear. Selected offscreen render targets now receive actual retained-object pixels, closing the black-thumbnail half of the first port-authored-save failure. The following immediate-resave oracle exposed and fixed two separate numbered-state defects: 0xae now re-establishes the saved 0x1ad gameplay-frame boundary before a later save UI opens, and retained objects use AGE's sparse stride plus native-initialized matrix/default bytes while preserving unnamed fields. The next fresh rewrite also proved that reconstructed managed ancestor frames must retain their original T1/T2/T3 coordinates while parked at synthetic 0xae; preserving those coordinates fixes the later SYSTEM4 unwind that otherwise re-entered the Eushully intro. Slot 005 subsequently passed the base-load and stage-launch round trip. Dungeon-authored slot 006 exposed one further native-ordering requirement: restored scripts must begin at their ordinary entry, run frame-local prologues, and reach 0xae themselves. Matching that order restores FIELD's 80% zoom table entry and produces the dungeon map from the unchanged slot; interactive slot-006 confirmation passed. The shared-profile lifecycle is also closed: accepted frontend shutdown stops and joins the VM worker before writing SAVE.DAT plus RT.DAT, honors native NoSaveDat=0 shutdown policy without suppressing numbered-save flushes, preserves loaded accumulated playtime, and handles repeated teardown and I/O failure safely. Repeated live saving and loading through the early dungeon playthrough is now user-accepted: numbered saves restore into active gameplay and remain usable across repeated cycles without a newly observed persistence failure. Treat native-layout gameplay save/load as confirmed for the current Phase-B slice; future concrete edge cases can reopen it without retaining a generic acceptance blocker. JSON inspection/export, namespaced mod data, and migrations remain additive extended-mode work rather than 1.0 compatibility requirements.

Phase C — Externalize & modding foundation

Add editable named data overlays mapped explicitly onto the VM's *INIT-produced state; external files must not become an unsynchronized second source of truth.
Generalize the override/mod-loading from the engine's native loose-file mechanism into ordered OMP mounts, deterministic dependency resolution and unpacked development packages (§3.6).
Asset pipeline: AGF↔PNG, audio and deterministic .omp packaging. → Tier-1 modding works.

Phase D — Logic modding

Integrate the assembler (Tier-2 bytecode-patch mods) and ship the host hook API behind a language-independent runtime boundary; spike Lua as the first user-facing runtime per §3. → Tier-2 modding works.
Optionally invest in decompiler quality toward Tier 3.

Phase E — Enhance, polish, productize

Enhancements the VM unlocks: higher/wide resolution, faster text, QoL, save-anywhere, new-content mods. Modding docs + tools. Save/UX polish.

Planned maintenance slice — codebase consolidation

Status (2026-08-02): step 1 complete; step 2 underway. The runtime and tooling now have enough independent regression coverage to support behavior-preserving cleanup: 590 engine tests, eleven directly runnable Python tool suites, the Godot build/self-test, and the installed-corpus gates. The project layout itself is sound, but a few files have become navigation and ownership bottlenecks: VirtualMachine.cs, GfxState.cs, Main.cs, GodotAdvHost.cs, and extract_init.py. Repository entry points are also partly machine-local, and the test project recompiles frontend/tool sources by link instead of consuming one production assembly.

This is a maintenance effort, not a runtime redesign. Preserve bytecode behavior, native-format compatibility, current command paths, generated-file ownership, and the one-engine/many-profiles direction. Each step should land in bounded commits with the full validation level appropriate to the touched boundary; do not mix mechanical moves with semantic changes.

Create a reproducible project front door — completed 2026-08-02. Replace the locally excluded, machine-path-specific run-godot.ps1/.cmd workflow with a tracked launcher whose explicit parameters and documented environment fallbacks select Godot and the game root. Implement the already-listed one-command validation driver from docs/tools-reference.md, with named levels that distinguish hermetic/core, workspace-corpus, Godot runtime, and complete validation rather than silently skipping unavailable prerequisites. Add a navigation-only repository README that links canonical references instead of copying their facts. Keep personal defaults in ignored local state, never in the tracked launcher.

Gate: from a clean checkout, a developer can discover prerequisites, generate required runtime metadata, build/test the engine, and launch or self-test Godot through tracked commands. Every validation level exits nonzero on a required failure, summarizes pass/fail/skip with reasons, and detects leaked child processes. No tracked script contains this workspace's absolute paths.
Perform behavior-neutral physical splits. Keep public types, namespaces, commands, and dispatch behavior stable while dividing the large runtime files by existing domains. Move Main self-test code first, then Godot audio/movie/compositor/input sections; split GodotAdvHost into ADV text, presentation/input, surfaces, movies, and audio; split GfxState contracts, surfaces, retained objects, animation, and presentation; and thin VirtualMachine.Step through domain handler methods without replacing the proven dispatcher. Keep tools/extract_init.py as the documented CLI while moving its stage, battle, card, routine, gallery, and general table implementations plus tests into importable modules.

Progress (2026-08-02): five bounded splits completed the planned Main decomposition: its synthetic scene builder/threaded self-test harness into godot/Main.SelfTest.cs, then its BGM, voice, sound-effect, mixer, and bus-control surface into godot/Main.Audio.cs, then decoder staging, movie frame/audio publication, completion, and teardown into godot/Main.Movie.cs, then retained GPU/software composition, texture caching, transitions, raster helpers, and decision logging into godot/Main.Compositor.cs, and finally input routing, locator/debug hotkeys, debug-scene dispatch, cursor/dialog handling, and full-width text entry into godot/Main.Input.cs. The partial class retains the same node type, fields, signatures, execution order, and call sites; runtime validation, including all 590 engine tests and the Godot threaded self-test, remains green after each move.

The first bounded GodotAdvHost split moved live/retained ADV text, surface glyph rasterization and cache state, history presentation, message-window alpha, and retained wait-indicator publication into godot/GodotAdvHost.AdvText.cs. The host remains a sealed IHost implementation, and presentation/input, reset, and surface-lifecycle consumers retain direct partial-class access to the moved state. Runtime validation remains green.

The second bounded GodotAdvHost split moved script/presentation barriers, diagnostic and text-entry waits, input/message-skip services, cursor/foreground waits, frame and backbuffer publication, legacy transitions, scene reset/stop, frame pulse, and timed waits into godot/GodotAdvHost.PresentationInput.cs. Shared surface, movie, audio, and retained-text state remains directly accessible through the sealed partial class; runtime validation remains green.

The third bounded GodotAdvHost split moved decoded-image caching, mutable surface pixel/resource/dimension state, allocation and binding, fill/copy/resolve operations, render-target publication, release/range teardown, and texture decode caching into godot/GodotAdvHost.Surfaces.cs. Movie publication and teardown retain direct access to the surface store through the sealed partial class; runtime validation remains green.

The fourth bounded GodotAdvHost split moved movie surface bindings, ordinary and modal playback, movie-mask capture/publication and teardown, movie diagnostics, and frame/completion publication into godot/GodotAdvHost.Movies.cs. Presentation waits and mutable surface storage retain direct access through the sealed partial class; runtime validation remains green.

The fifth bounded GodotAdvHost split moved BGM, voice and SFX resolution/dispatch, delayed voice state, volume/routing control, and blocking BGM fades into godot/GodotAdvHost.Audio.cs. Message-skip release, scene reset, and frame-pulse consumers retain direct access through the sealed partial class. The planned host decomposition is complete; runtime validation remains green.

Gate: no externally visible behavior or command changes; generated artifacts are byte-identical where deterministic, and the corresponding engine, Python, Godot, and corpus validations remain green after each domain move.
Clarify runtime contracts. Split the broad IHost surface into ADV, graphics, audio, movie, input, diagnostics, and lifecycle contracts while retaining one aggregate host accepted by the VM. Move neutral transport records such as RGBA images and movie/surface requests out of format-specific or host-specific namespaces so Model, Hosting, and Sys4 no longer form avoidable dependency cycles. Preserve explicit headless defaults and diagnostics; interface cleanup must not turn intentionally unsupported presentation into false success.
Make the build graph express source ownership. Stop linking production .cs files from godot/ and tools/movie-corpus-gate/ into Age.Engine.Tests. Extract the platform-neutral frontend/movie/diagnostic code into a small production project referenced by Godot, tests, and the corpus gate. Retain both existing solution files because Godot export requires the frontend-local solution.
Close repository-policy and storage gaps. Add an explicit project license, third-party notices and checksums/provenance for committed binaries, consistent editor/build policy, a pinned .NET SDK, and a CI entry point built on the same validation driver. Remove the empty workspace-root .git directory and obsolete ignored output nests after separately verifying their exact contents; compact the real repository only after active work is committed. Configure an off-machine remote or equivalent backup before relying on local history as the sole recovery path.

Not cleanup targets: generated build/ output, the two intentional solution files, historical docs/superpowers/ plans/specifications, and fidelity-specific complexity that is directly covered by the native ABI. Reorganization is successful when ownership and reproduction become clearer, not when the raw file count is minimized.

Planned polish slice — AGE-exact retained glyph-mask text renderer

Scheduling (2026-07-30): resume this as the next bounded presentation-polish subsystem, while still allowing urgent playthrough blockers to preempt it. The original deferral gate has been met: the natural gameplay spine and first-dungeon loop are playable, numbered save/load is provisionally accepted, the shipped corpus has no remaining decoded effectful-opcode gaps, and representative system/menu presentation is live. More importantly, the detached Label representation now causes behavioral defects rather than only cosmetic native-parity differences: labels sit above the complete retained canvas, surface-drawn text does not inherit its object's alpha/tint, and modal/publication paths need explicit visibility and replacement exceptions. The current backend remains the migration fallback, not a target for further screenshot-driven font tuning.

Behavioral source: implement the native contract decoded in docs/engine-re.md; screenshots are integration/regression evidence, not tuning inputs. AGE selects an exact LOGFONTA into a CreateICA("DISPLAY") information DC, obtains per-glyph metrics and 0..16 coverage through GetGlyphOutlineA(GGO_GRAY4_BITMAP), and applies its own integer coverage, outline-sampling, and surface compositor. Godot currently substitutes FreeType FontFile/FontVariation masks and the Label outline primitive, which cannot be made equivalent by choosing another embolden constant.

Target architecture:

Add a platform-neutral glyph request/result contract. The request carries the authored face, height, derived width, weight, character/code point, and raster policy. The result carries coverage bytes, dimensions/stride, glyph origin/bearings, and cell advance.
Move AGE's coverage conversion, mode-1 displacement, mode-3 rounded ellipse samples, clipping, and integer surface compositing into shared managed code. This layer must not know whether the mask came from GDI or FreeType.
Add a Windows GDI rasterizer using the decoded CreateICA/CreateFontIndirectA/ GetGlyphOutlineA calls. Keep it as the shipped highest-fidelity Windows backend and as the independent reference oracle for the shared contract; it is not disposable calibration scaffolding.
Add an explicitly defined portable rasterizer behind the same contract. A FreeType/Godot implementation may produce different hinted pixels, especially when the proprietary requested face is unavailable; profile/configuration must select requested-face substitutions and the default fallback rather than claiming GDI pixel equivalence.
Route live ADV, retained text, History, immediate surface strings, and later ruby/furigana presentation through one cached glyph service. Retain the existing Label path as a temporary portable fallback until that integration is complete, then remove synthetic embolden/glyph-spacing constants from the exact path.

Execution sequence:

Close the retained-record contract and pin baselines. Before changing presentation, record the current semantic coordinates and native-backed checks for SC0000 voiced/unvoiced pages, STUDY/MAMES, HISTORY, BUNKI, and save/load restoration. Confirm the remaining implementation-sensitive native details: overflow/kinsoku behavior, object-range exhaustion, layout reset/republication, and restoration of live layout surfaces. The 2026-07-30 baseline closes those questions: the 20-byte glyph record is {publication-chain flag, left, top, right, bottom}; publication binds layout_first_handle + reveal_index from surface layout_slot + 0x14, using those four edges as the native source rectangle and layout_origin + (left,top) as the destination. Overflow compares glyph right/bottom strictly against the configured bounds. Horizontal wrapping leaves CP932 、, 。, and 」 attached to the preceding line. Timed reveal stops at the configured handle capacity; all nine Himegari layouts reserve 500 handles, while the largest individual static corpus string has 68 characters. Reset clears the complete layout surface and handle interval. Op 0x20a erases that interval and republishes through the current reveal index, clamping to record count but not capacity. Numbered saves persist the History backlog, not live layout surfaces or glyph records; initialized layout bindings survive history-tail decode and the ordinary saved-frame prologue/republication path reconstructs current-page presentation. AdvRetainedTextContract and focused engine tests pin the platform-neutral record, overflow, punctuation, capacity, SYSTEM4 binding, and restore boundaries before presentation changes.
Land the backend-neutral mask and compositor core. Completed 2026-07-30 without changing live presentation. Age.Engine.Text now owns immutable GlyphRasterRequest/GlyphMask contracts and the IGlyphMaskRasterizer seam outside the VM. Requests preserve Unicode scalar plus optional original CP932 code and explicitly select native-CP932 or portable-Unicode policy; results normalize every backend to 0..16 coverage, stride, GDI-style origin, cell extent, and cell advance. AgeGlyphMaskCompositor implements AGE's coverage conversion, modes 0–3, clipping, transparent-destination rule, maximum-alpha and integer RGB blend. RetainedGlyphLayoutEngine returns native-edge records, wrapping/kinsoku decisions, overflow state, and final cursor while rasterizing into a backend-neutral RGBA surface. A generic bounded LRU supports disposable backend font resources and CachedGlyphMaskRasterizer bounds masks. Fifteen synthetic-mask tests, independent of OS fonts, pin coverage rounding, bearings/clipping, mode-1 displacement, mode-2 quarter coverage, mode-3 sampling/overlap, wrapping, punctuation, vertical overflow, cursor results, and cache eviction.
Add the independent Windows GDI reference backend. Completed 2026-07-30 as the separate Age.Engine.Text.Windows library; the VM and platform-neutral Age.Engine still call no OS APIs. WindowsGdiGlyphMaskRasterizer owns a CreateICA("DISPLAY") information context, bounded/safely disposed CreateFontIndirectA handles, CP932 GetTextExtentPoint32A, identity MAT2, and GetGlyphOutlineA(GGO_GRAY4_BITMAP). It accepts only explicit native-CP932 requests and reports backend id, policy, exactness, and call-chain detail through GlyphRasterizerBackendInfo. Availability requires Windows system ACP 932; another OS/ACP reports a diagnostic reason instead of silently claiming parity. Three representative regular/bold Mincho and bold Gothic requests byte-match an independent Unicode GDI oracle for coverage, stride, GLYPHMETRICS, extent, and advance. A later live capture resolved the apparent lighter-weight discrepancy: the AGE Patch.exe/jprun.dll reference launch realizes AGE's authored 24px bold Mincho request as MS Gothic. Thirteen intercepted masks match an independent Gothic request byte-for-byte and Mincho 0/13. That result is specific to the wrapper and does not supersede the true native Mincho target; a realized-face override is optional launch-profile compatibility, not an acceptance requirement for the exact backend. This reference backend remains opt-in and is not yet connected to live presentation; the existing Label path therefore remains the current fallback.
Move immediate surface strings first. Completed 2026-07-30 for the exact Windows path. Ops 0x204/0x205 now convert the complete Unicode string to explicit CP932 glyph identities before changing any pixels, derive AGE's rebuilt negative height/half-width and weight, and composite the exact masks into a cloned numbered RgbaImage snapshot. Successful draws publish no SurfaceTextDraw, so both retained renderers consume the same pixels in ordinary handle order and naturally apply object alpha/tint, affine transforms, source clipping, capture, fill/copy, and backbuffer preservation. Synthetic integration tests pin later-handle occlusion, source clipping, scale/translation, static alpha/tint, animated fade, capture, copy, and clear. The Godot self-test additionally exercises the GDI backend, 2,048-entry mask-cache bound, dynamic GPU upload, and surface mutation path. Backend absence, an unrepresentable glyph, or an unresolved surface falls back atomically to the prior metadata/Label projection with an explicit diagnostic; that projection remains only for this temporary unavailable-backend case until the portable backend lands.
Materialize live ADV glyphs as ordinary retained objects. Completed 2026-07-30 for the exact Windows path. The VM passes GfxState plus the transient layout binding into presentation and accepts the measured final cursor in return; the persisted History ABI is unchanged. RetainedAdvTextLayoutPresentation retains each native edge record with the layout origin active for that run, builds the complete line into the layout surface once, and binds first_handle + glyph_index in reveal order. Consecutive literal/dynamic runs therefore begin at GDI's returned cursor instead of relying on Label concatenation. Timed reveal stops at capacity, while click/Skip publishes the remaining available handles through the existing consumed-click branch. Partial 0x1f7 erases ordinary handles; 0x20a recreates all records through the current reveal count. HIDEWIN suspension erases that range and restoration republishes it. Layout reset erases the interval, clears its RGBA surface and transient records, and saved-frame-style reconstruction deterministically rebuilds the same bindings. Exact runs are excluded from the live Label snapshot; unavailable-backend runs retain it unchanged. Focused tests cover handle order/capacity, per-run origins, partial erase/republication, suspension, measured cursor handoff, reset, and reconstruction. The synthesized Godot VM self-test exercises timed exact reveal and reports live-adv-text=retained-glyphs; real GPU captures of SC0000 pages 1 and 7 confirm ordinary and voiced dialogue placement/wrapping alongside the immediate speaker-name surface. A 2026-07-30 clipping follow-up keeps those native measured-cell records intact but gives transient placements a separate vertically ink-safe source crop, because backend glyph black boxes and mode-3 outlines can extend below the measured cell; horizontal crops remain cell-bounded to preserve reveal isolation.
Migrate History and the ADV wait indicator through the same retained path. — Completed 2026-07-30. Op 0x1d1 now rasterizes each HISTORY row into its target layout surface and publishes its native glyph interval through ordinary GfxState bindings. Row reset, wheel redraw, the broad native exit erase, recording re-enable, and surface release all remove the transient presentation without changing the persisted backlog. The configured wait atlas now binds at op 0x212's handle, advances source cells only on the configured frame boundary, participates in op-0x20a republication and HIDEWIN suspension, and disappears when the wait service stops. The standalone Godot wait TextureRect and its raw-callback visibility exception are gone. At this checkpoint exact History no longer entered the layout-keyed Label pool; step 7 subsequently supplied the portable backend and removed every remaining gameplay Label pool. Focused lifecycle tests, all 579 engine tests, the exact History Godot self-test, and a real SC0000 Vulkan capture cover the retained path.
Complete and select the portable backend — completed 2026-07-30. Godot's TextServer atlas is the portable mask/metrics source, avoiding another native dependency. Profile data in godot/config/himegari-text-rendering.json owns ordered Mincho/Gothic substitutions and bounded cache sizes. auto selects exact GDI when its ACP-932 gate passes and otherwise selects the explicitly non-pixel-exact Unicode backend; development runs can force either with --text-backend. Both modes pass the same immediate/live/History retained-glyph self-test. Gameplay Label pools and surface-text overlay projection are removed, so text always participates in the ordinary surface compositor. Publication uses the transient ink-safe crop established in step 5 in both backends.

Acceptance gates:

Fixed synthetic-mask tests byte-match AGE's decoded 16-/32-bit compositor, including overlapping mode-3 neighbors, clipping, alpha, and RGB integer rounding.
On Windows, representative CP932 glyph masks, GLYPHMETRICS, and advances match a direct invocation of the decoded GDI request; tests compare returned data, not screenshot histograms.
If wrapper compatibility is requested, a launch profile may override the realized face ahead of the raster request. Himegari's AGE Patch.exe/jprun.dll path realizes 24px authored Mincho as MS Gothic, but the true-native acceptance target remains the authored Mincho request.
Surface-string tests prove text is occluded by later handles and inherits the bound object's alpha, tint, affine transform, source clipping, offscreen capture, and transition behavior in both render backends.
Retained-layout tests prove per-glyph handle order, capacity bounds, partial/full erase, reset/republication, final cursor, reveal/Skip click consumption, and save/load reconstruction without Label visibility shims.
SC0000, STUDY/MAMES, HISTORY, HIDEWIN, BUNKI, and save/load checks retain placement, wrapping, line advance, reveal timing, colors, effects, publication behavior, and native-backed lifetime after the backend swap.
The portable backend starts without Windows fonts or GDI, reports/uses its selected substitution policy, and passes layout/legibility tests without being labeled pixel-identical to native AGE.
Glyph/font caches and retained node/surface pools remain bounded. A complete line is rasterized once before reveal, so revealing another glyph does not rerasterize or upload the whole text surface and introduces no visible frame stalls.

5. The VM as our analysis instrument — and the correctness bootstrap

Beyond being the runtime, the VM is the best analysis tool we can build — but only once it is validated-correct, and that ordering is load-bearing.

Upside: owning the VM turns static RE into dynamic observation. Instrumenting the original packed AGE.EXE means fighting an anti-debug binary with Frida; instrumenting our interpreter is one line in a handler. So a class of documented side-tasks become built-in debugger views instead of separate investigations:

Live named-global watch — with the global-var map, watch state change in real time; take damage → see which global moved → that is the "name which stat" step (name-resolution.md → Future), now a debugger feature, not a Frida session.
Call-graph / dispatch trace — watch call-script resolve live, helping crack the entry-point registry dynamically.
Breakpoints, single-step, var-bank inspection, opcode/script/global coverage; and because VM state is just variable banks + a program counter, cheap snapshot / rewind (time-travel debugging nearly falls out of the design).
State-divergence differ (backlog — the payoff consumer of the trace facility). Builds on the landed diagnostics seam (Age.Engine/Diagnostics/ITraceSink; spec/plan docs/superpowers/{specs, plans}/2026-07-07-engine-diagnostics*): align our per-scene trace against a reference stream (a Frida capture of the real engine's pc/branch sequence, a known-good seeded run, or the same scene ±a candidate seed) and report the first instruction where control forks plus the global that fed the branch — turning "the opening loops/drifts somewhere in state" into "fork at jcc g[0x…]; seed this flag." The fine-grained successor to Age.Cli sweep 0xADDR=VAL (which only reports which scenes change ±seed, not where/why); it directly attacks the state-divergence root cause behind the 13 STEP-LIMIT scenes, the gfx geometry drift, and the form-gated silences. Requires: --trace-steps granularity (have it), robust sequence alignment (LCS-style, not naive zip — the real work), and treating native-nondeterministic ops (rand-like 0x60 in SCJUMP) as expected-to-differ. Enables the "differential checks against known-correct behavior" named in the bootstrap order below.

The hard caveat: the instrument is only as trustworthy as the VM is correct. A VM that executes wrong produces wrong observations — and circularly so: you would "learn" false facts about game state from a broken interpreter and bake them into the global map, the dispatch model, and everything downstream. The analysis power is unlocked by correctness, not a substitute for achieving it. You cannot debug the unknown with an instrument you have not first validated. Until the VM is running mostly correctly, using it for analysis is meaningless.

Therefore the bootstrap order matters:

Build the VM.
Validate it against ground truth we already hold, by independent means — chiefly the dialogue oracle (build/text/dialogue.jsonl: the VM's show-text sequence per scene must match), plus differential checks against known-correct behavior. This external oracle certifies the instrument.
Only then use the validated core's observability to understand the adjacent unknown — which globals mean what, dispatch, effectful-op behavior. Correctness propagates outward from validated anchors.

Two refinements that bound the trust:

Trust is per-subsystem, and only as strong as the oracle covering it. The dialogue diff strongly validates the ADV/text layer. But computational/battle logic has weaker oracles — a wrong damage number can look plausible and pass unnoticed. So "mostly working" must mean demonstrated correct per subsystem; where oracles are weak (battle math), the VM-as-instrument is correspondingly less trustworthy, and Frida/Unicorn cross-checks retain their value there (exactly why Unicorn stayed on the list for computational ops). A green ADV oracle does not imply the battle math is right.
Coverage-limited, plus a chicken-and-egg. Runtime tools only observe what a playthrough exercises (rare branches stay dark), so they complement rather than replace static analysis. And bootstrapping the VM needs just enough call-script dispatch to run before observation can help refine dispatch — hence Phase A hardcodes a minimal dispatch first.

Net effect on sequencing: the runtime debugger makes most of the Frida/Unicorn side-tasks cheaper or obsolete — but only after the VM earns trust on a validated core. So the priority is to reach validated correctness on the ADV layer first (via the dialogue oracle); that trusted anchor is what makes the instrument usable for everything else. The debugger is a force multiplier on a correct VM and dead weight on an incorrect one.

6. Extending to other AGE games and versions

Other AGE games, same version (e.g. Kamidori, also SYS4)

Reused for free: container parser, VM core, backend adapters, the engine-version opcode ABI, disassembler/assembler, and the whole extraction methodology. The ABI is shared across the family, but semantic and implementation coverage still grows as another game's corpus exercises services that the first game did not use. Per-game (inherent content work): the global-var map (globals are game-specific), the data-table layouts (each game's *INIT differs), assets, and any game-specific effectful behavior. The call-script registry is no longer a per-game long pole — it's a raw index into that game's SYS4INI file table, parsed directly by the runtime catalog (and exported by parse_sys4ini.py as build/callscript-names.json for tooling); the resolver is generic. So the remaining long pole is really just the global-var map. Process: point the toolchain at the new game's archives, re-run extraction, rebuild its global map, author a profile. This is the core payoff of the VM approach: the engine cost amortizes across all AGE games; only content-mapping recurs — far less than re-coding each game's logic bespoke.

The first runtime install-selection boundary is now in place. Godot receives one normalized game root, defaulting to the executable directory with a current-working-directory fallback and accepting an explicit --game-root override; the selected root is injected into the generic SYS4 catalog/store instead of inferred from the Himegari repository layout. A future multi-profile launcher can therefore own install discovery and pass the chosen profile's absolute root through the same stable argument. Generated engine/profile metadata still needs an export-owned bundle before this constitutes complete drop-in packaging; invocation details live in tools-reference.md.

Opcode ABI registry must be independent of per-game coverage

A cursory Kamidori boot probe on 2026-07-20 validated much of this boundary: its own SYS4INI.BIN and archives loaded, SYSTEM4.BIN completed the initializer chain, call-script resolution entered TITLE.BIN, and title graphics/resource commands resolved. The first hard stop was not a different container or catalog. Kamidori's title loop reached opcode 0x1be (u0041D9D0, argc 2) at bytecode offset 0xd7. That opcode existed in Kelebek's full AGE/SYS4 table, but was absent from the generated build/opcodes.json because the runtime artifact contained only opcodes observed in Himegari. Sys4Loader therefore stopped decoding at the unknown opcode; TITLE.BIN returned as if it had ended, and Godot displayed its ordinary — end — marker instead of reporting an incompatibility.

ABI-registry floor completed 2026-07-28. vm-map/opcodes.toml and generated build/opcodes.json now contain the complete 548-entry Kelebek AGE catalog: 248 instructions observed in Himegari and 300 catalog-only compatibility entries (including the already mapped but unused persistence opcode 0x19f). Each entry carries observed_in_himegari; per-game observation no longer limits decoding. A recognized but unimplemented opcode reaches the VM's traced stub-and-advance fallback, so a probe continues beyond it instead of losing the rest of the script. Catalog-only entries deliberately remain noop_headless=false: skipping them is a temporary compatibility-probe behavior, not evidence that their native effects are semantically safe to omit.

The multi-game design keeps three separate concepts:

A version ABI registry containing every known opcode number, operand count/shape, and version gate needed to decode that SYS generation, regardless of whether the active game uses it.
Per-game observed coverage, used for prioritization, provenance, and regression reporting but never to decide which otherwise-known instructions the runtime parser is allowed to decode.
Semantic/runtime implementation coverage, which may remain incomplete and should report a precise unsupported-service error containing game/profile, script, opcode, and bytecode offset.

An opcode that is absent even from the selected version ABI should still become a structured decode error, not a synthetic final instruction or natural script return; that diagnostic hardening remains open. Supporting a new same-version game now means selecting the complete shared ABI, measuring its corpus against existing semantics, and implementing only the newly exercised services. It does not require cloning the VM or manufacturing a new parser table from that game's corpus.

The probe also exposed a separate profile/presentation concern: Kamidori creates a 1024x576 render target while the current Himegari frontend assumes an 800x600 presentation. Logical canvas geometry, scaling, and other game-specific defaults therefore belong in the selected game profile or script-driven surface state rather than in a forked frontend. The source is now known: after its asset directory and VM-bank metadata, SYS4INI carries a typed per-game startup-settings record. Himegari uses it for the canvas, text face/raster mode, ADV input/skip policy, save ABI/path, audio initialization, and native Windows compatibility metadata. The runtime now parses that record once and preserves unknown/profile- specific data for diagnostics; semantic settings are applied through explicit cross-platform seams while legacy renderer/registration switches remain classified rather than blindly emulated. The first bounded application slice is complete: SCREENX/SCREENY select the validated logical canvas, all presentation allocations and primary bounds consume it, and it is the default windowed size. Other settings remain on their existing paths while gameplay is the priority. The second slice is also complete: independent physical --window-width/--window-height overrides change only the windowed client while the logical canvas, VM coordinates, and surface geometry remain fixed. The executable task plan and gates live in phase-a-slice-plan.md. This experiment was diagnostic only; no Kamidori support or 0x1be semantics were implemented.

Other engine versions (SYS3 / SYS5) — one app, not many

Versions differ in: header (SYS4 0x3C vs SYS5 0x44), string codec (SYS4 cp932^0xFF vs SYS5 UTF-16^0xFFFF), opcode set (overlapping, version-specific; Kelebek's table already spans the family and notes version-gated ops), operand types (SYS5 adds 0x8003+), and archive magic (S3IC/S4IC/ S5IN). Architecture answer: version-parameterize, don't fork. One runtime with:

a version-detection step (magic → SYS3/4/5),
pluggable front-ends (header parser, string decoder, opcode table, archive reader per version),
the shared VM core (the execution model — opcode+typed operands, var banks, control flow — is the same engine evolving),
per-game profiles that name the version + game-specific maps.

So: not a separate application per version — a manifest/profile selects the front-end + game data. SYS5 is well-covered (it's Kelebek's target); SYS3 is older and less documented and would need more front-end work (Kelebek's parser only does SYS4/SYS5), but the plug-in shape accommodates it. Result: one "AGE Engine" app that, given a profile, runs Himegari, Kamidori, a SYS5 title, etc., each moddable through the same system.

7. Feasibility, risks, open questions

Feasible? Yes — but it is the largest phase of the whole effort, on the scale of a small ScummVM target. The decoding groundwork substantially de-risks it (we understand the format, 97% of opcodes, the data, a partial global map). Biggest risks, with mitigations:

~~call-script dispatch entangled in the packed AGE.EXE~~ ✅ RESOLVED — cracked statically via the Ghidra dispatch table (no Frida): call-script <id> = a raw SYS4INI file index; the C# VM executes it. (SCJUMP was not the registry, as first guessed.)
Effectful-op surface is large and quirk-laden (esp. SRPG battle/dungeon UI) → ADV-first; defer SRPG; lean on the Frida shortlist.
AGF graphics → low risk; AGF2BMP2AGF.exe (asmodean) already present.
Save format → reversible struct work; needed before a full playthrough.
Toolchain (Python) vs runtime (C#) drift → share the documented format spec; runtime is canonical.

Open questions to resolve early: ~~exact call-script mechanism~~ (solved); how scenes chain (the SCJUMP decision→scene-id native hop — needed to sequence and to add content); how much the SRPG layer's rendering diverges from ADV; save layout.

8. Immediate next step

Continue step 2 of the codebase consolidation maintenance slice: behavior-neutral physical splits backed by the tracked launcher and layered validation driver. With the planned Main and GodotAdvHost domains isolated, begin the GfxState decomposition with one existing ownership domain at a time, preserving public types, commands, and generated output. Concrete playthrough blockers may still preempt this bounded maintenance work; the consolidation effort does not replace Phase B gameplay validation or the open cross-platform gates.

69 KiB Raw Blame History Unescape Escape