diff --git a/docs/asset-resolution-re.md b/docs/asset-resolution-re.md index 8019bf5..aa1f5c2 100644 --- a/docs/asset-resolution-re.md +++ b/docs/asset-resolution-re.md @@ -54,7 +54,8 @@ highest-risk area of the port. This doc is the steering state; it feeds the A2b all 13206 `offset+size` fit inside their real `.ALF`; 837 name-matched files → 0 size mismatches. `files[]` preserves directory order (feeds step 2's order-correlation). Re-run: `py -3.11 -X utf8 tools/parse_sys4ini.py --check`. (Ref: asmodean's `exs4alf` / GARbro Eushully `ArcALF.cs`.) -2. **Resolve `resId → asset file`.** **✅ SOLVED (2026-07-06) — fully static & general; NO runtime capture.** +2. **Resolve `resId → asset file`.** **✅ SOLVED for scene-manifest references (2026-07-06); + system/global raw ids are a separate path identified 2026-07-10.** **The rule:** SYS4INI's file list is organized into **SECTIONS, one per scene** — each is a `SCxxxx.BIN` script entry followed by that scene's **asset MANIFEST**: every asset it references, @@ -74,6 +75,14 @@ highest-risk area of the port. This doc is the steering state; it feeds the A2b AGE game with the same container — **the "scope" was just which SYS4INI section the scene lives in.** (The old `play-bgm 5→BGM006` validation point was a mis-attribution — the real game plays BGM005.) + **System/global-id exception (identified 2026-07-10; not yet implemented).** Some SYSTEM4 loads use + the SYS4INI record's universal `raw_index` directly, including the two `@` placeholder records, rather + than a scene-local manifest index. `SYSTEM4.BIN` writes `G[0x69b]=0x337e`, then + `set-texture(G[0x69b], slot=0x11)`. SYS4INI `raw_index 0x337e` is `DATA1/SO001.AGF`, the shared + 800×300 RGBA system-chrome sheet. The filtered `files[]` list omits placeholders, so treating `0x337e` + as a `files[]` position currently mis-resolves it to `SETROUTE.BIN`. This path needs a distinct + `raw_index → entry` lookup; the scene-manifest rule above remains correct for SC texture/voice ids. + *How we got here (condensed):* first confirmed `resId == file_number` via Frida load-order correlation for SC0000's opening, but `file_number` is not globally unique so a per-scene "scope" was needed. A long hunt for the selector (thought it was native scene state; even tried reading `G[0x62424]` live — the @@ -94,6 +103,17 @@ highest-risk area of the port. This doc is the steering state; it feeds the A2b (`AE*`) drew opaque (no alpha); the slot model approximated the game's immediate-mode blit-onto-slot-0 canvas. See `docs/phase-a-slice-plan.md` (A2b section) for the implementation history and current retained-object model. + + **Current system-chrome shortcut/gap (confirmed 2026-07-10).** `Main --boot` executes only + `INITCONFIG/INIT2/INIT` through `CaptureHost` and copies their globals into the scene VM; it does not + replay SYSTEM4's graphics side effects through `GodotAdvHost`. `convert_agf.py --scene SC0000` also + converts only SC0000's manifest, so `build/textures/SO001.BMP` is absent. `CALLBACK_WINDOW.BIN` expects + slot `0x11` to already contain SO001, draws the 800×227 textbox from `(0,0)`, and crops the lower-right + buttons from the same sheet. In the port slot 17 is unpopulated, so retained handle `0xd2f0` resolves as + a colored surfaceless object and the compositor draws the observed opaque black fill. A temporary decode + verified SO001 is 800×300, 32-bpp, with substantial per-pixel alpha; the current rasterizer already + consumes source alpha. The missing prerequisites are system-asset resolution/conversion and retained + slot initialization, not new textbox drawing or button interaction. 4. **Audio.** **✅ WIRED (2026-07-06) — no Frida needed.** Same rule as textures: `play-bgm(id)`/`play-voice(id)` → `files[section_base(scene)+id]` → OGG. `IHost.PlayBgm/PlayVoice` + VM dispatch (`play-bgm` 0xbf / `play-voice` 0xc4, both argc 1); `ResourceMap.AudioPath` → loose @@ -134,3 +154,72 @@ layer renders end-to-end from the bytecode. Remaining (next chunk): the **graphi subsystem** — native geometry ops (`0x208` + sprite position/animation) so sprites/`BG*` position, plus alpha/blend for fades + chromakey. See `docs/phase-a-slice-plan.md` (A2b). Audio (step 4): **`play-voice` uses the manifest** (`files[base+id]`); **`play-bgm` uses direct names** (`BGM{id:03d}.OGG`) — NOT unified. + +## Candidate runtime asset-VFS track (scoped 2026-07-10; not started) + +The pre-extracted tree and `build/textures/*.BMP` pipeline were a Phase-A bootstrap, not the desired final +runtime. The native-compatible target is a read-only virtual filesystem that preserves AGE's translation/mod +behavior: + +> resolve the resource record → try a loose file with that record's name in the game/mod root → otherwise +> read exactly `offset..offset+size` from the record's ALF → decode the contained format in process. + +Native evidence already proves this ordering for scripts: `resource_open_by_raw_id@0x44f390` indexes the +80-byte SYS4 record and calls `CreateFileA(record.name)` before opening `record.archive`, seeking to +`record.offset`, and reading `record.size`. The same service is the correct common seam for scripts, +graphics, voice/SFX, and later movie bytes. Resolution and opening must remain separate: scene-local ids and +universal `raw_index` ids select a record differently, but both records flow through the same loose-first +store. + +### Proposed layers + +1. **Catalog + read-only ALF store.** Parse SYS4INI at runtime while preserving all 13208 raw records + (including the two `@` placeholders), archive names, scene sections, and the existing three lookup modes: + universal raw id, scene-local manifest id, and direct name where the opcode family genuinely uses one. + An ALF is a payload container at this layer: open the named archive and return a bounded stream/byte range + at the indexed offset/size. Before that fallback, probe the configured loose override roots by the record's + exact basename. Keep `build/asset-index.json` as a diagnostic artifact, not a runtime dependency. +2. **AAI append mount.** Parse the installed `APPEND01.AAI` (`S4AC422`) and its paired `APPEND01.ALF` with + the same catalog abstractions. First prove whether Himegari joins append records by a separate pack/tag, + by name replacement, or by another table selected by the native high-byte-id path; do not invent mount + precedence. Validate every parsed append entry against `BinExtractALF.exe` output before exposing it to + the runtime. +3. **AGF decoder.** Decode an opened AGF stream directly to width/height + RGBA8. The MIT-licensed GARbro + `ArcFormats/Eushully/ImageAGF.cs` provides a compact reference: `ACGF` (or zero) signature, type 1/2, + LZSS-or-raw header section, 4/8/truecolor source pixels, LZSS-or-raw pixel section, bottom-up row/stride + conversion, and optional `ACIF` LZSS alpha plane. Port only the algorithm and attribution into + platform-neutral .NET code; do not carry GARbro's WPF/GameRes dependencies. Kelebek's extractor and the + on-disk `BinExtractALF.exe` are validation references; the Kelebek repository exposes no clear license, + so its code should not be copied without clarification. +4. **Runtime consumers.** Make script loading accept streams/bytes from the store, make texture surfaces own + decoded RGBA pixels rather than BMP paths, and load OGG/WAV from store bytes. Migrate one consumer at a + time; retain extraction/conversion tools as diagnostics until parity is established. + +### Acceptance gates + +- Catalog: 13208 raw slots / 13206 real base entries; every ALF range is in bounds; scene-local mappings + remain identical to the current resolver and `raw_index 0x337e` resolves to `SO001.AGF`. +- Store: representative base reads are byte-identical to `extracted/`; a temporary loose file with the same + record name wins, and removing it deterministically reveals the archive bytes. Root path traversal is + rejected and archive reads are bounded/thread-safe. +- AAI: entry names/counts/ranges and representative bytes match a disposable `BinExtractALF APPEND01.AAI` + extraction; the native mount/selection rule is documented before integration. +- AGF: a sample matrix covers compressed/uncompressed sections, 4/8/24-bit source pixels, type 1/2, and + alpha/no-alpha. Decoded dimensions and RGBA hashes/pixels match GARbro or `AGF2BMP2AGF`; `SO001.AGF` + specifically decodes as 800×300 with its alpha plane intact. +- End to end: SC0000 can run without consulting `extracted/` or `build/textures`; slot 17 receives SO001, + the translucent textbox/button chrome appears, root `.BIN` overrides still win, and the standard VM/Godot + validation matrix remains green. + +### Deliberate non-goals + +- Writing/repacking ALF or AAI; loose overrides already provide the native mod/translation workflow. +- AGF encoding, movie/video decoding for MPEG-like `OP/MVB*.AGF`, or SFX channel semantics. +- A generalized multi-mod dependency manager. Start with native game-root loose overrides; configurable + ordered mod roots can be layered onto the same store later. +- Removing the extraction/conversion tools immediately. They remain independent parity oracles until the + runtime readers have broad corpus coverage. + +Primary implementation reference: [GARbro's Eushully AGF reader](https://github.com/morkt/GARbro/blob/master/ArcFormats/Eushully/ImageAGF.cs) +and [ALF reader](https://github.com/morkt/GARbro/blob/master/ArcFormats/Eushully/ArcALF.cs), MIT licensed. +Secondary validation reference: [Kelebek's extractor](https://github.com/Kelebek1/Eushully-Decompiler/blob/master/extract_alf.py). diff --git a/docs/phase-a-slice-plan.md b/docs/phase-a-slice-plan.md index 16db1bc..f7d2eb7 100644 --- a/docs/phase-a-slice-plan.md +++ b/docs/phase-a-slice-plan.md @@ -1049,3 +1049,43 @@ clean apart from the pre-existing nullable warning and threaded `SELFTEST OK`; a opcode/ctx lint, 481-script decode, RECOVER, and `git diff --check` clean. SC0000 coverage rises from **85/129 to 87/129 handled (67.4%)**, with 42 GAP ops / 188 GAP instructions. Movie `0x236` and SFX remain separate slices. No implementation commit was made. + +### Candidate foundation track — native asset VFS + ALF/AAI/AGF readers (SCOPED 2026-07-10; NOT STARTED) + +This is an optional high-leverage detour before movie `0x236` or SFX. It replaces the Phase-A +pre-extracted/pre-converted asset bootstrap with the native loose-override/archive-fallback model and removes +the runtime dependency on `extracted/` plus `build/textures/`. Canonical format/architecture detail and source +references live in `docs/asset-resolution-re.md` §“Candidate runtime asset-VFS track”; this section defines +slice boundaries only. + +Land it as three bounded slices, not one archive/codec rewrite: + +VFS-A is the common prerequisite. VFS-B and VFS-C may be swapped afterward: choose A→C→B for the fastest +SC0000 textbox/chrome payoff, or A→B→C when complete base+append archive mounting is the priority. + +1. **VFS-A — base SYS4 catalog + ALF byte reads.** Runtime-parse SYS4INI into raw-id, scene-local, and name + lookup views; preserve placeholders; expose bounded asset streams. Apply native precedence + `loose root/ → indexed ALF range`. Convert `Sys4Loader`/`Sys4ScriptProvider` to consume bytes + from this seam first, proving the existing 52 patched root BINs still shadow archive copies. Gate: sampled + and corpus-size/range validation against `extracted/`, plus synthetic loose-override tests. +2. **VFS-B — APPEND01 AAI mount.** Parse `S4AC422` and read `APPEND01.ALF`. Use Ghidra/native observation + only as needed to settle pack selection/mount precedence; compare the entire parsed directory and sampled + payloads with `BinExtractALF.exe`. Gate: no guessed name override or high-byte-id behavior reaches runtime. +3. **VFS-C — in-process AGF → RGBA8.** Port the MIT GARbro decode algorithm into platform-neutral engine + code with a focused LZSS primitive, palette/truecolor expansion, stride/orientation handling, and optional + ACIF alpha. Replace BMP-path texture loading with decoded pixel surfaces. Gate: codec sample matrix plus + exact SO001 dimensions/alpha, then windowed SC0000 textbox/button pixels with `extracted/` and + `build/textures/` unavailable to the runtime. + +After those gates, move OGG/WAV consumers onto the same byte store as a small follow-up. That prepares SFX +without conflating its channel/timing semantics with archive access. Movie `0x236` may likewise consume raw +AGF/MPEG bytes later, but video decoding remains explicitly outside this track. + +Expected authored seams (names provisional): `Sys4AssetCatalog`, `IAssetStore`/`Sys4AssetStore`, bounded +archive stream/reader, `LzssDecoder`, and `AgfDecoder`; focused tests belong in `Age.Engine.Tests`. Runtime +caches should key decoded assets by catalog identity plus loose-file timestamp, while tests use synthetic +fixtures and installed-game integration checks rather than committing proprietary assets. + +**Decision point:** this track is worthwhile before broadening beyond SC0000 because it establishes the +modding contract and benefits scripts, UI chrome, SFX, and movies. It is not required to continue opcode +coverage immediately, so choosing movie/SFX next remains valid. diff --git a/docs/tools-reference.md b/docs/tools-reference.md index a3befc7..40c212b 100644 --- a/docs/tools-reference.md +++ b/docs/tools-reference.md @@ -191,6 +191,24 @@ texture ops (no GPU context) — run windowed for real scenes. User args (after | `engine_ctx_build.py` | Build the `EngineCtx` struct artifacts from `vm-map/engine-ctx.toml` (canonical `ctx`-field registry). `--lint` = overlap/OOB/dup/type checks. The struct is then applied to the `/v2` image via `run_script_inline` (creates `EngineCtx`, retypes all dispatch-handler `this` → `EngineCtx *`) so handlers decompile `ctx->field` not `param_1+0x…`. Grows one `[[field]]` at a time. | `engine_ctx_build.py --build` · `--lint` | ⚙ `vm-map/engine-ctx.toml` → ⚙ `build/engine-ctx.json`, ⚙ `docs/engine-ctx-reference.md` | | `test_engine_ctx.py` | Unit tests for the ctx builder (load/lint/emit; plain runner). | `test_engine_ctx.py` | — | +## Tooling improvement backlog + +Prioritized from the 2026-07-10 Phase-A retrospective. These are proposed capabilities, not tools that +already exist. Prefer additions that produce reusable, offset-keyed evidence over one-off capture scripts. + +| Priority | Improvement | Payoff / acceptance boundary | +|---|---|---| +| P0 | **Native/port checkpoint harness** | Launch either runtime windowed, identify the process/window, record script offset + timestamp, capture a screenshot and retained-state/trace excerpt, and classify normal exit vs harness stop vs crash. A checkpoint should produce one comparable artifact bundle for the same bytecode offset in both runtimes. | +| P0 | **Sanctioned Windows GUI automation** (small Win32 helper, AutoHotkey, or pywinauto) | Repeatably focus the game, capture window geometry, and inject only explicitly requested clicks/keys. Manual input remains the independent oracle; automation removes capture/setup repetition. | +| P0 | **Scriptable 32-bit debugger** (x32dbg or WinDbg) | Breakpoints and memory snapshots for hot render/audio workers that are unsafe to hook densely with Frida. Keep Frida probes on known low-frequency handlers; never restore hot interpreter/glyph/render hooks merely for convenience. | +| P1 | **Timestamped video capture** (ffmpeg desktop capture or command-controlled OBS) | Frame-by-frame native/port evidence for movie `0x236`, fades, and short animation boundaries without PNG-per-frame overhead. | +| P1 | **WASAPI loopback/audio capture** | Objective SFX/BGM/voice start time, channel reuse, volume, stop, and waveform comparison. This is the main evidence upgrade for the pending SFX slice. | +| P1 | **One-command validation driver** | Run engine tests with shared compilation disabled, sweep, Godot build/selftest, Python suites, generated-reference lints, decode/RECOVER checks, `git diff --check`, and report leaked child processes in one summarized result. | +| P1 | **Golden SC0000 checkpoint corpus** | Preserve the first 10-15 native pages as offset-keyed screenshots, click/wait events, retained-state summaries, and trace excerpts. Port regressions should be comparable without replaying the entire investigation. | +| P2 | **Opcode dossier generator** | Combine corpus callsites/operands, native handler/worker addresses, runtime samples, Ghidra names, opcode provenance, and port coverage into a per-op investigation packet. | +| P2 | **More typed Ghidra state** | Materialize retained-object, surface-slot, text-layout, and audio-channel structures so related handlers decompile against shared named fields. | +| P2 | **Narrow write access for cross-session memory** | Let mandatory milestone closeout update `MEMORY.md` and `himegari-port-status.md` directly; the current out-of-workspace copy/approval path adds avoidable handoff friction. | + ## Historical / one-off | Tool | Purpose |