Decode AGF textures through the asset store

This commit is contained in:
gamer147
2026-07-11 09:55:58 -04:00
parent b6c3b1660d
commit a1f6d0ab51
15 changed files with 468 additions and 158 deletions

View File

@@ -85,7 +85,7 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
│ │ └── strings.jsonl every string, tagged by source opcode
│ ├── data/ parsed data tables (*INIT → JSON)
│ ├── scripts-json/ machine-readable full dumps (on demand via --json)
│ ├── textures/ AGFBMP stills (convert_agf.py) — feeds the Godot render
│ ├── textures/ AGF-to-BMP stills (convert_agf.py) - diagnostic pixel oracle only
│ ├── engine-dump/ UNPACKED engine dump (frida/dump_engine.py): range_<base>.bin + manifest.json
│ ├── asset-index.json, asset-sections.json asset resolver data (parse_sys4ini / resolve_asset)
│ ├── global-var-map.{json,md} partial global-variable name map (auto shape inference; feeds globals.toml merge)
@@ -97,7 +97,7 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
├── engine/ DELIVERABLE — the .NET VM core (AgeEngine.sln: Age.Engine / Age.Cli / tests)
│ └── Age.Engine/Sys4/ runtime catalog parser, loose-first bounded ALF asset store,
│ script provider, and temporary resource facade
│ script provider, AGF-to-RGBA8/LZSS decoder, and resource facade
├── tools/frida/ runtime-capture + engine-dump scripts (see tools/frida/README.md)
└── godot/ DELIVERABLE — the Godot/C# ADV front-end (references Age.Engine)
```

View File

@@ -166,7 +166,7 @@ The current Phase-A backend deliberately continues through the extracted-file bo
accepts both OGG and WAV and Godot loads the WAV bytes into its fixed SC0000 channel pool. This does not change
the scoped VFS plan below: ALF/AAI mounting and in-process asset reads remain a separate foundation track.
## Runtime asset-VFS track (VFS-A complete 2026-07-11)
## Runtime asset-VFS track (VFS-A and VFS-C complete 2026-07-11)
The pre-extracted tree and `build/textures/*.BMP` pipeline were a Phase-A bootstrap, not the desired final
runtime. The native-compatible target is a read-only virtual filesystem that preserves AGE's translation/mod
@@ -199,16 +199,20 @@ store.
by name replacement, or by another table selected by the native high-byte-id path; do not invent mount
precedence. Validate every parsed append entry against `BinExtractALF.exe` output before exposing it to
the runtime.
3. **AGF decoder.** Decode an opened AGF stream directly to width/height + RGBA8. The MIT-licensed GARbro
3. **AGF decoder (VFS-C DONE).** `Age.Engine/Sys4/AgfDecoder.cs` decodes an opened AGF payload directly
to a tightly packed, top-down width/height + RGBA8 surface. The MIT-licensed GARbro
`ArcFormats/Eushully/ImageAGF.cs` provides a compact reference: `ACGF` (or zero) signature, type 1/2,
LZSS-or-raw header section, 4/8/truecolor source pixels, LZSS-or-raw pixel section, bottom-up row/stride
conversion, and optional `ACIF` LZSS alpha plane. Port only the algorithm and attribution into
platform-neutral .NET code; do not carry GARbro's WPF/GameRes dependencies. Kelebek's extractor and the
on-disk `BinExtractALF.exe` are validation references; the Kelebek repository exposes no clear license,
so its code should not be copied without clarification.
4. **Runtime consumers.** Script loading is complete. Next make texture surfaces own
decoded RGBA pixels rather than BMP paths, and load OGG/WAV from store bytes. Migrate one consumer at a
time; retain extraction/conversion tools as diagnostics until parity is established.
so its code should not be copied without clarification. The focused `LzssDecoder` is shared with
`Sys4AssetCatalog`; raw and compressed information/pixel/ACIF sections use the same bounded primitive.
4. **Runtime consumers.** Script and texture loading are complete. `ResourceMap.ResolveTexture` preserves
scene-local resolution and falls back to universal raw ids for SYSTEM4 assets; `GodotAdvHost` caches
decoded RGBA surfaces by catalog identity and supplies synchronous dimensions to opcode `0x208`.
Godot no longer reads `build/textures/*.BMP`. OGG/WAV byte migration remains a separate follow-up;
retain extraction/conversion tools as diagnostics until parity is established.
### Acceptance gates
@@ -233,8 +237,17 @@ are byte-identical to `extracted/`, and synthetic removal of a loose override re
Traversal, past-range seek/read, and concurrent reads are covered. Installed override enumeration corrected
an older inventory error: this tree contains 51 loose root BINs, comprising **49 archive-backed v1.03 script
overrides** (all byte-proven to win and differ from DATA1) plus root-only `SYS4INI.BIN` and `SYS4AB.BIN`.
There are not 52 archive copies available to shadow. APPEND01/AAI, AGF decode, audio consumers, and movie
`0x236` remain unimplemented by design.
There are not 52 archive copies available to shadow.
VFS-C passes its bounded gates in `AgfDecoderTests`: synthesized fixtures cover raw/compressed sections,
4/8-bit palettes, 24/32-bit truecolor expansion, padded bottom-up rows, type 1/2, and ACIF/no-ACIF alpha.
Five installed assets spanning raw/compressed metadata and pixel/alpha combinations match the existing
`AGF2BMP2AGF` BMP oracle pixel-for-pixel. SO001 resolves through universal raw id `0x337e`, decodes to
800×300 with intermediate alpha values, and is inherited in surface slot 17 before SC0000. A windowed
page-1 capture with `build/textures/` moved aside showed the translucent textbox edge and bottom-right
controls. Texture runtime no longer consults `extracted/` or `build/textures/`; current audio consumers
still use extracted OGG/WAV paths by design. APPEND01/AAI, audio migration, and movie `0x236` remain
unimplemented by this slice.
### Deliberate non-goals

View File

@@ -1050,7 +1050,7 @@ opcode/ctx lint, 481-script decode, RECOVER, and `git diff --check` clean. SC000
**85/129 to 87/129 handled (67.4%)**, with 42 GAP ops / 188 GAP instructions. Movie `0x236` and SFX remain
separate slices. No implementation commit was made.
### Candidate foundation track — native asset VFS + ALF/AAI/AGF readers (SCOPED 2026-07-10; NOT STARTED)
### Foundation track — native asset VFS + ALF/AAI/AGF readers (VFS-A/C DONE; VFS-B PENDING)
This is an optional high-leverage detour before movie `0x236` or SFX. It replaces the Phase-A
pre-extracted/pre-converted asset bootstrap with the native loose-override/archive-fallback model and removes
@@ -1105,6 +1105,26 @@ archive-backed script overrides and two root-only BIN engine files, not the hist
shadowing scripts; all 49 were byte-proven to win and differ from their archive payload. VFS-B APPEND01/AAI,
VFS-C AGF, audio migration, and movie `0x236` remain separate.
### VFS-C — in-process AGF to RGBA8 + SC0000 system chrome DONE (2026-07-11)
`AgfDecoder` now consumes `IAssetStore` bytes and emits a platform-neutral, tightly packed top-down RGBA8
surface. Its shared 4 KiB-ring `LzssDecoder` covers raw/compressed information, pixel, and optional ACIF
alpha sections; expansion covers 4/8-bit palettes, 24/32-bit truecolor, DIB row padding, and bottom-up
orientation. Synthetic fixtures exercise the whole format matrix. Installed `AE000A`, `AE001A`,
`BG030A`, `EV052CA`, and `SO001` match the existing converter's pixels exactly; SO001 is 800×300
with non-binary per-pixel alpha.
`ResourceMap` now resolves texture ids through the scene manifest with a universal raw-id fallback,
then decodes through the loose-first VFS. Godot and the CLI no longer use `BmpHeader`, `TexturePath`, or
pre-converted BMPs. SYSTEM4's inherited SO001 surface is seeded into slot `0x11` before SC0000, allowing
the existing callback-window crops to render the textbox and control icons. Windowed page-1 validation
passed with `build/textures/` temporarily unavailable; source alpha produced the translucent upper edge
and the bottom-right control row was visible.
Validation: 124/124 engine tests, CLI/Godot builds, Godot threaded selftest, pixel parity for five installed
assets, and the windowed no-BMP capture. VFS-B APPEND01/AAI, OGG/WAV migration, movie `0x236`, and unrelated
opcode work remain separate.
### Phase A — native SC0000 SFX family (`0xb4`/`0xb5`/`0xb6`/`0xc2`/`0xd9`) DONE (2026-07-11)
Native RE and the matching trace resolve the bounded family. `0xb4(resource,channel)` synchronously loads

View File

@@ -69,6 +69,10 @@ Three layers, cleanly separated:
the offline analysis/authoring chain; they were the reference implementation and stay useful for
modders. They share the *format spec* (documented), not code — acceptable for a small, stable
container format.
- **Native content foundation status (2026-07-11).** VFS-A runtime-parses the base SYS4 catalog and
applies loose-first bounded ALF reads; VFS-C decodes AGF directly to platform-neutral RGBA8 and feeds
Godot without pre-extracted or pre-converted texture files. APPEND01/AAI mounting and audio-byte consumers
remain separate follow-ups.
- **Profile = manifest.** Adding a game = a new profile + its maps. Adding an engine version = a new
front-end plugin + profiles that reference it. See §5.

View File

@@ -160,7 +160,7 @@ texture ops (no GPU context) — run windowed for real scenes. User args (after
| `parse_sys4ini.py` | Parse `SYS4INI.BIN` (S4IC422, LZSS-compressed) into the diagnostic JSON asset-index mirror — name ↔ archive ↔ offset ↔ size for all DATA*.ALF. Each real entry carries universal `raw_index`; the runtime parses SYS4INI itself, while these generated files remain tooling/test oracles. Also emits the `call-script <id> → name` annotation map. | `parse_sys4ini.py [--check]` (`--check` validates vs `extracted/` + `.ALF` sizes) | `姫狩り…/SYS4INI.BIN``build/asset-index.json` + `build/callscript-names.json` |
| `resolve_asset.py` | ★ **The static asset resolver.** SYS4INI is sectioned (one per scene: `SCxxxx.BIN` + its cross-archive manifest; `file_number` = index within section). Resolves `resId → files[section_base(scene) + resId]` for graphics AND audio, no capture. | `resolve_asset.py --build` · `resolve_asset.py <SCENE> [resId]` | `build/asset-index.json``build/asset-sections.json`; resolves any (scene, resId) |
| `resolve_frida_reads.py` | Rescue noisy Frida archive-read offsets → asset names via the index (per-archive range search; drops 0x20000 paging reads); recovers the per-scene asset load order. | `resolve_frida_reads.py [reads.log] [-o out.json]` | `build/frida-reads.log` + `build/asset-index.json``build/frida-asset-loads.json` |
| `convert_agf.py` | Convert AGF stills to BMP via `AGF2BMP2AGF.exe` (searches all `extracted/DATA*`). `--scene` batch-converts a scene's whole SYS4INI manifest — feeds the Godot render. | `convert_agf.py EV052CA.AGF …` · `convert_agf.py --scene SC0000` | `extracted/DATA*/*.AGF``build/textures/*.BMP` |
| `convert_agf.py` | Convert AGF stills to BMP via `AGF2BMP2AGF.exe` (searches all `extracted/DATA*`). `--scene` batch-converts a scene's whole SYS4INI manifest. Since VFS-C, output is a diagnostic pixel-parity oracle; the runtime decodes AGF bytes directly. | `convert_agf.py EV052CA.AGF …` · `convert_agf.py --scene SC0000` | `extracted/DATA*/*.AGF``build/textures/*.BMP` |
## Runtime capture (Frida)