Decode AGF textures through the asset store

This commit is contained in:
gamer147
2026-07-11 09:55:58 -04:00
parent 8e0f769a6e
commit ec07c76721
15 changed files with 468 additions and 158 deletions

View File

@@ -85,7 +85,7 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
│ │ └── strings.jsonl every string, tagged by source opcode
│ ├── data/ parsed data tables (*INIT → JSON)
│ ├── scripts-json/ machine-readable full dumps (on demand via --json)
│ ├── textures/ AGFBMP stills (convert_agf.py) — feeds the Godot render
│ ├── textures/ AGF-to-BMP stills (convert_agf.py) - diagnostic pixel oracle only
│ ├── engine-dump/ UNPACKED engine dump (frida/dump_engine.py): range_<base>.bin + manifest.json
│ ├── asset-index.json, asset-sections.json asset resolver data (parse_sys4ini / resolve_asset)
│ ├── global-var-map.{json,md} partial global-variable name map (auto shape inference; feeds globals.toml merge)
@@ -97,7 +97,7 @@ S:\Game Hacking\Eushully\Himegari\ ← workspace root (three siblings)
├── engine/ DELIVERABLE — the .NET VM core (AgeEngine.sln: Age.Engine / Age.Cli / tests)
│ └── Age.Engine/Sys4/ runtime catalog parser, loose-first bounded ALF asset store,
│ script provider, and temporary resource facade
│ script provider, AGF-to-RGBA8/LZSS decoder, and resource facade
├── tools/frida/ runtime-capture + engine-dump scripts (see tools/frida/README.md)
└── godot/ DELIVERABLE — the Godot/C# ADV front-end (references Age.Engine)
```

View File

@@ -166,7 +166,7 @@ The current Phase-A backend deliberately continues through the extracted-file bo
accepts both OGG and WAV and Godot loads the WAV bytes into its fixed SC0000 channel pool. This does not change
the scoped VFS plan below: ALF/AAI mounting and in-process asset reads remain a separate foundation track.
## Runtime asset-VFS track (VFS-A complete 2026-07-11)
## Runtime asset-VFS track (VFS-A and VFS-C complete 2026-07-11)
The pre-extracted tree and `build/textures/*.BMP` pipeline were a Phase-A bootstrap, not the desired final
runtime. The native-compatible target is a read-only virtual filesystem that preserves AGE's translation/mod
@@ -199,16 +199,20 @@ store.
by name replacement, or by another table selected by the native high-byte-id path; do not invent mount
precedence. Validate every parsed append entry against `BinExtractALF.exe` output before exposing it to
the runtime.
3. **AGF decoder.** Decode an opened AGF stream directly to width/height + RGBA8. The MIT-licensed GARbro
3. **AGF decoder (VFS-C DONE).** `Age.Engine/Sys4/AgfDecoder.cs` decodes an opened AGF payload directly
to a tightly packed, top-down width/height + RGBA8 surface. The MIT-licensed GARbro
`ArcFormats/Eushully/ImageAGF.cs` provides a compact reference: `ACGF` (or zero) signature, type 1/2,
LZSS-or-raw header section, 4/8/truecolor source pixels, LZSS-or-raw pixel section, bottom-up row/stride
conversion, and optional `ACIF` LZSS alpha plane. Port only the algorithm and attribution into
platform-neutral .NET code; do not carry GARbro's WPF/GameRes dependencies. Kelebek's extractor and the
on-disk `BinExtractALF.exe` are validation references; the Kelebek repository exposes no clear license,
so its code should not be copied without clarification.
4. **Runtime consumers.** Script loading is complete. Next make texture surfaces own
decoded RGBA pixels rather than BMP paths, and load OGG/WAV from store bytes. Migrate one consumer at a
time; retain extraction/conversion tools as diagnostics until parity is established.
so its code should not be copied without clarification. The focused `LzssDecoder` is shared with
`Sys4AssetCatalog`; raw and compressed information/pixel/ACIF sections use the same bounded primitive.
4. **Runtime consumers.** Script and texture loading are complete. `ResourceMap.ResolveTexture` preserves
scene-local resolution and falls back to universal raw ids for SYSTEM4 assets; `GodotAdvHost` caches
decoded RGBA surfaces by catalog identity and supplies synchronous dimensions to opcode `0x208`.
Godot no longer reads `build/textures/*.BMP`. OGG/WAV byte migration remains a separate follow-up;
retain extraction/conversion tools as diagnostics until parity is established.
### Acceptance gates
@@ -233,8 +237,17 @@ are byte-identical to `extracted/`, and synthetic removal of a loose override re
Traversal, past-range seek/read, and concurrent reads are covered. Installed override enumeration corrected
an older inventory error: this tree contains 51 loose root BINs, comprising **49 archive-backed v1.03 script
overrides** (all byte-proven to win and differ from DATA1) plus root-only `SYS4INI.BIN` and `SYS4AB.BIN`.
There are not 52 archive copies available to shadow. APPEND01/AAI, AGF decode, audio consumers, and movie
`0x236` remain unimplemented by design.
There are not 52 archive copies available to shadow.
VFS-C passes its bounded gates in `AgfDecoderTests`: synthesized fixtures cover raw/compressed sections,
4/8-bit palettes, 24/32-bit truecolor expansion, padded bottom-up rows, type 1/2, and ACIF/no-ACIF alpha.
Five installed assets spanning raw/compressed metadata and pixel/alpha combinations match the existing
`AGF2BMP2AGF` BMP oracle pixel-for-pixel. SO001 resolves through universal raw id `0x337e`, decodes to
800×300 with intermediate alpha values, and is inherited in surface slot 17 before SC0000. A windowed
page-1 capture with `build/textures/` moved aside showed the translucent textbox edge and bottom-right
controls. Texture runtime no longer consults `extracted/` or `build/textures/`; current audio consumers
still use extracted OGG/WAV paths by design. APPEND01/AAI, audio migration, and movie `0x236` remain
unimplemented by this slice.
### Deliberate non-goals

View File

@@ -1050,7 +1050,7 @@ opcode/ctx lint, 481-script decode, RECOVER, and `git diff --check` clean. SC000
**85/129 to 87/129 handled (67.4%)**, with 42 GAP ops / 188 GAP instructions. Movie `0x236` and SFX remain
separate slices. No implementation commit was made.
### Candidate foundation track — native asset VFS + ALF/AAI/AGF readers (SCOPED 2026-07-10; NOT STARTED)
### Foundation track — native asset VFS + ALF/AAI/AGF readers (VFS-A/C DONE; VFS-B PENDING)
This is an optional high-leverage detour before movie `0x236` or SFX. It replaces the Phase-A
pre-extracted/pre-converted asset bootstrap with the native loose-override/archive-fallback model and removes
@@ -1105,6 +1105,26 @@ archive-backed script overrides and two root-only BIN engine files, not the hist
shadowing scripts; all 49 were byte-proven to win and differ from their archive payload. VFS-B APPEND01/AAI,
VFS-C AGF, audio migration, and movie `0x236` remain separate.
### VFS-C — in-process AGF to RGBA8 + SC0000 system chrome DONE (2026-07-11)
`AgfDecoder` now consumes `IAssetStore` bytes and emits a platform-neutral, tightly packed top-down RGBA8
surface. Its shared 4 KiB-ring `LzssDecoder` covers raw/compressed information, pixel, and optional ACIF
alpha sections; expansion covers 4/8-bit palettes, 24/32-bit truecolor, DIB row padding, and bottom-up
orientation. Synthetic fixtures exercise the whole format matrix. Installed `AE000A`, `AE001A`,
`BG030A`, `EV052CA`, and `SO001` match the existing converter's pixels exactly; SO001 is 800×300
with non-binary per-pixel alpha.
`ResourceMap` now resolves texture ids through the scene manifest with a universal raw-id fallback,
then decodes through the loose-first VFS. Godot and the CLI no longer use `BmpHeader`, `TexturePath`, or
pre-converted BMPs. SYSTEM4's inherited SO001 surface is seeded into slot `0x11` before SC0000, allowing
the existing callback-window crops to render the textbox and control icons. Windowed page-1 validation
passed with `build/textures/` temporarily unavailable; source alpha produced the translucent upper edge
and the bottom-right control row was visible.
Validation: 124/124 engine tests, CLI/Godot builds, Godot threaded selftest, pixel parity for five installed
assets, and the windowed no-BMP capture. VFS-B APPEND01/AAI, OGG/WAV migration, movie `0x236`, and unrelated
opcode work remain separate.
### Phase A — native SC0000 SFX family (`0xb4`/`0xb5`/`0xb6`/`0xc2`/`0xd9`) DONE (2026-07-11)
Native RE and the matching trace resolve the bounded family. `0xb4(resource,channel)` synchronously loads

View File

@@ -69,6 +69,10 @@ Three layers, cleanly separated:
the offline analysis/authoring chain; they were the reference implementation and stay useful for
modders. They share the *format spec* (documented), not code — acceptable for a small, stable
container format.
- **Native content foundation status (2026-07-11).** VFS-A runtime-parses the base SYS4 catalog and
applies loose-first bounded ALF reads; VFS-C decodes AGF directly to platform-neutral RGBA8 and feeds
Godot without pre-extracted or pre-converted texture files. APPEND01/AAI mounting and audio-byte consumers
remain separate follow-ups.
- **Profile = manifest.** Adding a game = a new profile + its maps. Adding an engine version = a new
front-end plugin + profiles that reference it. See §5.

View File

@@ -160,7 +160,7 @@ texture ops (no GPU context) — run windowed for real scenes. User args (after
| `parse_sys4ini.py` | Parse `SYS4INI.BIN` (S4IC422, LZSS-compressed) into the diagnostic JSON asset-index mirror — name ↔ archive ↔ offset ↔ size for all DATA*.ALF. Each real entry carries universal `raw_index`; the runtime parses SYS4INI itself, while these generated files remain tooling/test oracles. Also emits the `call-script <id> → name` annotation map. | `parse_sys4ini.py [--check]` (`--check` validates vs `extracted/` + `.ALF` sizes) | `姫狩り…/SYS4INI.BIN``build/asset-index.json` + `build/callscript-names.json` |
| `resolve_asset.py` | ★ **The static asset resolver.** SYS4INI is sectioned (one per scene: `SCxxxx.BIN` + its cross-archive manifest; `file_number` = index within section). Resolves `resId → files[section_base(scene) + resId]` for graphics AND audio, no capture. | `resolve_asset.py --build` · `resolve_asset.py <SCENE> [resId]` | `build/asset-index.json``build/asset-sections.json`; resolves any (scene, resId) |
| `resolve_frida_reads.py` | Rescue noisy Frida archive-read offsets → asset names via the index (per-archive range search; drops 0x20000 paging reads); recovers the per-scene asset load order. | `resolve_frida_reads.py [reads.log] [-o out.json]` | `build/frida-reads.log` + `build/asset-index.json``build/frida-asset-loads.json` |
| `convert_agf.py` | Convert AGF stills to BMP via `AGF2BMP2AGF.exe` (searches all `extracted/DATA*`). `--scene` batch-converts a scene's whole SYS4INI manifest — feeds the Godot render. | `convert_agf.py EV052CA.AGF …` · `convert_agf.py --scene SC0000` | `extracted/DATA*/*.AGF``build/textures/*.BMP` |
| `convert_agf.py` | Convert AGF stills to BMP via `AGF2BMP2AGF.exe` (searches all `extracted/DATA*`). `--scene` batch-converts a scene's whole SYS4INI manifest. Since VFS-C, output is a diagnostic pixel-parity oracle; the runtime decodes AGF bytes directly. | `convert_agf.py EV052CA.AGF …` · `convert_agf.py --scene SC0000` | `extracted/DATA*/*.AGF``build/textures/*.BMP` |
## Runtime capture (Frida)

View File

@@ -373,7 +373,7 @@ sealed class GfxTraceHost : IHost
{
private readonly ResourceMap _res;
private readonly string _scene;
private readonly Dictionary<int, string?> _slotBmp = new(); // slot -> resolved BMP path (or null)
private readonly Dictionary<int, string?> _slotAsset = new(); // slot -> resolved AGF name (or null)
// slot -> dims. Slot 0 is the primary/screen surface (800x600), normally created at engine boot which
// the single-scene harness skips; seed it so the first CG's anchor math stays correct (not 0x0).
private readonly Dictionary<int, (int W, int H)> _slotDims = new() { { 0, (800, 600) } };
@@ -389,19 +389,19 @@ sealed class GfxTraceHost : IHost
public void SetTexture(long resId, int slot)
{
var e = _res.Resolve(_scene, resId);
var bmp = e != null ? ResourceMap.TexturePath(e) : null;
_slotBmp[slot] = bmp;
_slotDims[slot] = BmpHeader.ReadDims(bmp);
var e = _res.ResolveTexture(_scene, resId);
RgbaImage? image = e != null ? _res.DecodeTexture(e) : null;
_slotAsset[slot] = e?.Name;
_slotDims[slot] = image != null ? (image.Width, image.Height) : (0, 0);
Events.Add($"set-texture slot={slot} res=0x{resId:x} -> {(e?.Name ?? "<unresolved>")}"
+ (bmp == null ? " [NO BMP]" : ""));
+ (image == null ? " [NO AGF]" : ""));
}
public void DrawTexture(int slot, int sx, int sy, int w, int h, int dx, int dy)
{
_slotBmp.TryGetValue(slot, out var bmp);
_slotAsset.TryGetValue(slot, out var asset);
Events.Add($"draw-texture slot={slot} src=({sx},{sy} {w}x{h}) dst=({dx},{dy}) "
+ $"file={(bmp != null ? System.IO.Path.GetFileName(bmp) : "<none>")}");
+ $"file={(asset ?? "<none>")}");
}
public void CreateTexture(int slot, int width, int height)

View File

@@ -0,0 +1,164 @@
using System.Buffers.Binary;
using Age.Engine.Sys4;
namespace Age.Engine.Tests;
public class AgfDecoderTests
{
[Fact]
public void DecodesRawFourBitPaletteWithBottomUpStride()
{
byte[] palette = Palette((0x10, 0x20, 0x30), (0x40, 0x50, 0x60), (0x70, 0x80, 0x90));
byte[] bottomUp = { 0x12, 0x10, 0, 0, 0x01, 0x20, 0, 0 };
var image = AgfDecoder.Decode(BuildAgf(1, 3, 2, 4, palette, bottomUp));
Assert.Equal((3, 2), (image.Width, image.Height));
Assert.Equal(new byte[] {
0x10,0x20,0x30,255, 0x40,0x50,0x60,255, 0x70,0x80,0x90,255,
0x40,0x50,0x60,255, 0x70,0x80,0x90,255, 0x40,0x50,0x60,255,
}, image.Pixels);
}
[Fact]
public void DecodesCompressedEightBitPaletteAndAcifAlpha()
{
byte[] palette = Palette(256, (1, 2, 3), (4, 5, 6), (7, 8, 9));
byte[] bottomUp = { 2, 1, 0, 0, 0, 1, 2, 0 };
byte[] alpha = { 10, 20, 30, 40, 50, 60 };
var image = AgfDecoder.Decode(BuildAgf(2, 3, 2, 8, palette, bottomUp, alpha,
compressInfo: true, compressPixels: true, compressAlpha: true));
Assert.Equal(new byte[] {
1,2,3,10, 4,5,6,20, 7,8,9,30,
7,8,9,40, 4,5,6,50, 1,2,3,60,
}, image.Pixels);
}
[Fact]
public void DecodesTruecolorAndDefaultsMissingAlphaToOpaque()
{
byte[] bottomUp = { 30,20,10,0, 60,50,40,0 };
var image = AgfDecoder.Decode(BuildAgf(2, 2, 1, 32, null, bottomUp));
Assert.Equal(new byte[] { 10,20,30,255, 40,50,60,255 }, image.Pixels);
}
[Fact]
public void DecodesThroughAssetStoreByteSeam()
{
byte[] agf = BuildAgf(1, 1, 1, 24, null, new byte[] { 3, 2, 1, 0 });
var entry = new AssetEntry("TEST.AGF", "DATA.ALF", 0, agf.Length);
var image = AgfDecoder.Decode(new MemoryStore(agf), entry);
Assert.Equal(new byte[] { 1, 2, 3, 255 }, image.Pixels);
}
[Theory]
[InlineData("AE000A.AGF")]
[InlineData("AE001A.AGF")]
[InlineData("BG030A.AGF")]
[InlineData("EV052CA.AGF")]
[InlineData("SO001.AGF")]
public void InstalledAssetMatchesExistingConverterPixels(string name)
{
string bmpPath = Path.Combine(Paths.Textures, Path.ChangeExtension(name, ".BMP"));
if (!File.Exists(bmpPath)) return;
var catalog = Sys4AssetCatalog.Load(Paths.Sys4Ini);
var store = new Sys4AssetStore(catalog, Paths.GameDir, Paths.GameDir);
var image = AgfDecoder.Decode(store, catalog.ResolveName(name)!);
var oracle = ReadBmp32(bmpPath);
Assert.Equal((oracle.Width, oracle.Height), (image.Width, image.Height));
Assert.Equal(oracle.Pixels, image.Pixels);
}
[Fact]
public void InstalledSo001HasExpectedAlphaBearingDimensions()
{
var catalog = Sys4AssetCatalog.Load(Paths.Sys4Ini);
var store = new Sys4AssetStore(catalog, Paths.GameDir, Paths.GameDir);
var resources = new ResourceMap(catalog, store);
Assert.Equal("SO001.AGF", resources.ResolveTexture("SC0000", 0x337e)?.Name);
var image = AgfDecoder.Decode(store, catalog.ResolveRaw(0x337e)!);
Assert.Equal((800, 300), (image.Width, image.Height));
Assert.Contains(image.Pixels.Where((_, i) => (i & 3) == 3), a => a is > 0 and < 255);
}
private sealed class MemoryStore(byte[] bytes) : IAssetStore
{
public Stream Open(AssetEntry entry) => new MemoryStream(bytes, writable: false);
public byte[] ReadAll(AssetEntry entry) => bytes;
}
private static byte[] Palette(params (byte R, byte G, byte B)[] colors) => Palette(16, colors);
private static byte[] Palette(int count, params (byte R, byte G, byte B)[] colors)
{
var result = new byte[count * 4];
for (int i = 0; i < colors.Length; i++)
{ result[i * 4] = colors[i].B; result[i * 4 + 1] = colors[i].G; result[i * 4 + 2] = colors[i].R; }
return result;
}
private static byte[] BuildAgf(int type, int width, int height, int bpp, byte[]? palette, byte[] pixels,
byte[]? alpha = null, bool compressInfo = false,
bool compressPixels = false, bool compressAlpha = false)
{
var info = new byte[0x38 + (palette?.Length ?? 0)];
Put32(info, 0x14, width); Put32(info, 0x18, height); Put16(info, 0x1c, 1); Put16(info, 0x1e, bpp);
palette?.CopyTo(info, 0x38);
byte[] packedInfo = compressInfo ? LiteralLzss(info) : info;
byte[] packedPixels = compressPixels ? LiteralLzss(pixels) : pixels;
byte[]? packedAlpha = alpha == null ? null : compressAlpha ? LiteralLzss(alpha) : alpha;
int length = 0x18 + packedInfo.Length + 12 + packedPixels.Length +
(alpha == null ? 0 : 0x24 + packedAlpha!.Length);
var file = new byte[length];
"ACGF"u8.CopyTo(file); Put32(file, 4, type);
Put32(file, 0x0c, info.Length); Put32(file, 0x14, packedInfo.Length);
packedInfo.CopyTo(file, 0x18);
int p = 0x18 + packedInfo.Length;
Put32(file, p + 4, pixels.Length); Put32(file, p + 8, packedPixels.Length);
packedPixels.CopyTo(file, p + 12); p += 12 + packedPixels.Length;
if (alpha != null)
{
"ACIF"u8.CopyTo(file.AsSpan(p)); Put32(file, p + 0x1c, alpha.Length);
Put32(file, p + 0x20, packedAlpha!.Length); packedAlpha.CopyTo(file, p + 0x24);
}
return file;
}
private static byte[] LiteralLzss(byte[] source)
{
var output = new List<byte>();
for (int p = 0; p < source.Length;)
{
int count = Math.Min(8, source.Length - p);
output.Add((byte)((1 << count) - 1));
for (int i = 0; i < count; i++) output.Add(source[p++]);
}
return output.ToArray();
}
private static RgbaImage ReadBmp32(string path)
{
byte[] b = File.ReadAllBytes(path);
int offset = BinaryPrimitives.ReadInt32LittleEndian(b.AsSpan(10));
int width = BinaryPrimitives.ReadInt32LittleEndian(b.AsSpan(18));
int signedHeight = BinaryPrimitives.ReadInt32LittleEndian(b.AsSpan(22));
int bpp = BinaryPrimitives.ReadInt16LittleEndian(b.AsSpan(28));
Assert.True(bpp is 24 or 32);
int height = Math.Abs(signedHeight);
int stride = ((width * bpp / 8) + 3) & ~3;
var rgba = new byte[width * height * 4];
for (int y = 0; y < height; y++)
{
int sy = signedHeight > 0 ? height - 1 - y : y;
for (int x = 0; x < width; x++)
{
int s = offset + sy * stride + x * (bpp / 8), d = (y * width + x) * 4;
rgba[d] = b[s + 2]; rgba[d + 1] = b[s + 1]; rgba[d + 2] = b[s];
rgba[d + 3] = bpp == 32 ? b[s + 3] : (byte)255;
}
}
return new RgbaImage(width, height, rgba);
}
private static void Put32(byte[] b, int p, int value) => BinaryPrimitives.WriteInt32LittleEndian(b.AsSpan(p), value);
private static void Put16(byte[] b, int p, int value) => BinaryPrimitives.WriteInt16LittleEndian(b.AsSpan(p), (short)value);
}

View File

@@ -1,33 +0,0 @@
using System.IO;
using Age.Engine.Sys4;
using Xunit;
public class BmpHeaderTests
{
[Fact]
public void ReadDimsReadsWidthAndHeightFromBmpHeader()
{
// Minimal 54-byte BMP header (BITMAPFILEHEADER 14 + BITMAPINFOHEADER 40); width=4, height=3.
var b = new byte[54];
b[0] = (byte)'B'; b[1] = (byte)'M';
System.BitConverter.GetBytes(40).CopyTo(b, 14); // header size
System.BitConverter.GetBytes(4).CopyTo(b, 18); // width
System.BitConverter.GetBytes(3).CopyTo(b, 22); // height
var tmp = Path.Combine(Path.GetTempPath(), "agehdr_test.bmp");
File.WriteAllBytes(tmp, b);
try
{
var (w, h) = BmpHeader.ReadDims(tmp);
Assert.Equal(4, w);
Assert.Equal(3, h);
}
finally { File.Delete(tmp); }
}
[Fact]
public void ReadDimsReturnsZeroForMissingFile()
{
var (w, h) = BmpHeader.ReadDims(Path.Combine(Path.GetTempPath(), "does_not_exist_agehdr.bmp"));
Assert.Equal((0, 0), (w, h));
}
}

View File

@@ -0,0 +1,132 @@
using System.Buffers.Binary;
namespace Age.Engine.Sys4;
/// <summary>A decoded, tightly packed, top-down RGBA8 image.</summary>
public sealed record RgbaImage(int Width, int Height, byte[] Pixels);
/// <summary>
/// Platform-neutral Eushully AGF decoder. Format algorithm ported from GARbro's MIT-licensed
/// ArcFormats/Eushully/ImageAGF.cs (Copyright (C) 2015 morkt).
/// </summary>
public static class AgfDecoder
{
private const int OuterHeaderSize = 0x18;
public static RgbaImage Decode(IAssetStore store, AssetEntry entry)
=> Decode(store.ReadAll(entry), entry.Name);
public static RgbaImage Decode(ReadOnlySpan<byte> file, string name = "AGF")
{
if (file.Length < OuterHeaderSize ||
!(file[..4].SequenceEqual("ACGF"u8) || BinaryPrimitives.ReadUInt32LittleEndian(file) == 0))
throw new InvalidDataException($"{name}: expected an ACGF image");
int type = I32(file, 4, name);
if (type is not (1 or 2)) throw new InvalidDataException($"{name}: unsupported AGF type {type}");
int infoSize = PositiveSize(file, 0x0c, name, "information expanded size");
int infoPacked = PositiveSize(file, 0x14, name, "information packed size");
byte[] info = OpenSection(Slice(file, OuterHeaderSize, infoPacked, name, "information section"),
infoSize, infoPacked, name, "information");
if (info.Length < 0x20) throw new InvalidDataException($"{name}: information header is truncated");
int width = PositiveSize(info, 0x14, name, "width");
int height = PositiveSize(info, 0x18, name, "height");
int sourceBpp = I16(info, 0x1e, name);
if (sourceBpp is not (4 or 8 or 24 or 32))
throw new InvalidDataException($"{name}: unsupported source depth {sourceBpp}");
long pixelCount = (long)width * height;
if (pixelCount > int.MaxValue / 4) throw new InvalidDataException($"{name}: dimensions are too large");
ReadOnlySpan<byte> palette = default;
if (sourceBpp <= 8)
palette = Slice(info, 0x38, checked((1 << sourceBpp) * 4), name, "palette");
int dataOffset = checked(OuterHeaderSize + infoPacked);
ReadOnlySpan<byte> dataHeader = Slice(file, dataOffset, 12, name, "pixel section header");
int dataSize = PositiveSize(dataHeader, 4, name, "pixel expanded size");
int dataPacked = PositiveSize(dataHeader, 8, name, "pixel packed size");
int dataPos = checked(dataOffset + 12);
byte[] pixels = OpenSection(Slice(file, dataPos, dataPacked, name, "pixel section"),
dataSize, dataPacked, name, "pixels");
byte[]? alpha = null;
if (type == 2)
{
int alphaOffset = checked(dataPos + dataPacked);
if (alphaOffset <= file.Length - 0x24 && file.Slice(alphaOffset, 4).SequenceEqual("ACIF"u8))
{
ReadOnlySpan<byte> alphaHeader = file.Slice(alphaOffset, 0x24);
int alphaSize = PositiveSize(alphaHeader, 0x1c, name, "alpha expanded size");
int alphaPacked = PositiveSize(alphaHeader, 0x20, name, "alpha packed size");
if (alphaSize != pixelCount) throw new InvalidDataException($"{name}: alpha dimensions do not match image");
alpha = OpenSection(Slice(file, alphaOffset + 0x24, alphaPacked, name, "alpha section"),
alphaSize, alphaPacked, name, "alpha");
}
}
int sourceRowBytes = checked((checked(width * sourceBpp) + 7) / 8);
int sourceStride = checked((sourceRowBytes + 3) & ~3);
if ((long)sourceStride * height > pixels.Length)
throw new InvalidDataException($"{name}: pixel section is shorter than its bitmap stride");
var rgba = new byte[checked((int)pixelCount * 4)];
for (int y = 0; y < height; y++)
{
int src = checked((height - 1 - y) * sourceStride);
int dst = checked(y * width * 4);
int alphaAt = y * width;
for (int x = 0; x < width; x++, dst += 4)
{
if (sourceBpp == 4)
{
int index = (pixels[src + (x >> 1)] >> ((x & 1) == 0 ? 4 : 0)) & 0x0f;
CopyPalette(palette, index, rgba, dst);
}
else if (sourceBpp == 8)
CopyPalette(palette, pixels[src + x], rgba, dst);
else
{
int at = src + x * (sourceBpp / 8);
rgba[dst] = pixels[at + 2];
rgba[dst + 1] = pixels[at + 1];
rgba[dst + 2] = pixels[at];
}
rgba[dst + 3] = alpha?[alphaAt + x] ?? (byte)255;
}
}
return new RgbaImage(width, height, rgba);
}
private static void CopyPalette(ReadOnlySpan<byte> palette, int index, byte[] rgba, int dst)
{
int p = index * 4;
rgba[dst] = palette[p + 2]; rgba[dst + 1] = palette[p + 1]; rgba[dst + 2] = palette[p];
}
private static byte[] OpenSection(ReadOnlySpan<byte> source, int expanded, int packed,
string name, string section)
=> expanded == packed ? source.ToArray() : LzssDecoder.Decode(source, expanded, $"{name}: {section}");
private static int I32(ReadOnlySpan<byte> data, int offset, string name)
{
if (offset < 0 || offset > data.Length - 4) throw new InvalidDataException($"{name}: header is truncated");
return BinaryPrimitives.ReadInt32LittleEndian(data.Slice(offset, 4));
}
private static int I16(ReadOnlySpan<byte> data, int offset, string name)
{
if (offset < 0 || offset > data.Length - 2) throw new InvalidDataException($"{name}: header is truncated");
return BinaryPrimitives.ReadInt16LittleEndian(data.Slice(offset, 2));
}
private static int PositiveSize(ReadOnlySpan<byte> data, int offset, string name, string field)
{
int value = I32(data, offset, name);
if (value <= 0) throw new InvalidDataException($"{name}: invalid {field} {value}");
return value;
}
private static ReadOnlySpan<byte> Slice(ReadOnlySpan<byte> data, int offset, int count, string name, string field)
{
if (offset < 0 || count < 0 || offset > data.Length - count)
throw new InvalidDataException($"{name}: {field} is truncated");
return data.Slice(offset, count);
}
}

View File

@@ -1,25 +0,0 @@
namespace Age.Engine.Sys4;
/// <summary>
/// Reads pixel dimensions from a BMP file header (BITMAPINFOHEADER: width at byte 18, height at byte 22,
/// both little-endian int32; height may be negative for top-down bitmaps). Used to give the VM the
/// texture size that opcode 0x208 (get-texture-size) needs, without decoding pixels. Our textures are
/// pre-converted BMPs (tools/convert_agf.py).
/// </summary>
public static class BmpHeader
{
public static (int Width, int Height) ReadDims(string? path)
{
if (string.IsNullOrEmpty(path) || !File.Exists(path)) return (0, 0);
try
{
var b = new byte[26];
using var fs = File.OpenRead(path);
if (fs.Read(b, 0, 26) < 26 || b[0] != (byte)'B' || b[1] != (byte)'M') return (0, 0);
int w = System.BitConverter.ToInt32(b, 18);
int h = System.BitConverter.ToInt32(b, 22);
return (System.Math.Abs(w), System.Math.Abs(h));
}
catch { return (0, 0); }
}
}

View File

@@ -0,0 +1,44 @@
namespace Age.Engine.Sys4;
/// <summary>Eushully's 4 KiB-ring LZSS stream used by SYS4 catalogs and AGF sections.</summary>
public static class LzssDecoder
{
public static byte[] Decode(ReadOnlySpan<byte> source, int expectedSize, string name = "LZSS")
{
if (expectedSize < 0) throw new InvalidDataException($"{name}: negative expanded size");
var frame = new byte[0x1000];
var result = new byte[expectedSize];
int framePos = 0xfee, input = 0, output = 0;
while (output < expectedSize)
{
if (input >= source.Length) throw new InvalidDataException($"{name}: stream ended early");
int control = source[input++];
for (int bit = 1; bit <= 0x80 && output < expectedSize; bit <<= 1)
{
if ((control & bit) != 0)
{
if (input >= source.Length) throw new InvalidDataException($"{name}: truncated literal");
byte value = source[input++];
result[output++] = value;
frame[framePos] = value;
framePos = (framePos + 1) & 0xfff;
}
else
{
if (input > source.Length - 2) throw new InvalidDataException($"{name}: truncated back-reference");
int lo = source[input++], hi = source[input++];
int readPos = ((hi & 0xf0) << 4) | lo;
int length = 3 + (hi & 0x0f);
for (int j = 0; j < length && output < expectedSize; j++)
{
byte value = frame[readPos++ & 0xfff];
result[output++] = value;
frame[framePos] = value;
framePos = (framePos + 1) & 0xfff;
}
}
}
}
return result;
}
}

View File

@@ -9,8 +9,13 @@ namespace Age.Engine.Sys4;
public sealed class ResourceMap
{
private readonly Sys4AssetCatalog _catalog;
private readonly IAssetStore _store;
public ResourceMap(Sys4AssetCatalog catalog) => _catalog = catalog;
public ResourceMap(Sys4AssetCatalog catalog, IAssetStore? store = null)
{
_catalog = catalog;
_store = store ?? new Sys4AssetStore(catalog, Paths.GameDir, Paths.GameDir);
}
public static ResourceMap Load() => new(Sys4AssetCatalog.Load(Paths.Sys4Ini));
@@ -20,14 +25,20 @@ public sealed class ResourceMap
return _catalog.ResolveScene(scene, resId);
}
/// <summary>Pre-converted BMP path for an AGF asset (see tools/convert_agf.py).</summary>
public static string? TexturePath(AssetEntry a)
/// <summary>Resolve graphics normally through the scene manifest, with the universal raw-id
/// fallback used by SYSTEM4-owned assets such as SO001.</summary>
public AssetEntry? ResolveTexture(string scene, long resId)
{
if (!a.Name.EndsWith(".AGF", StringComparison.OrdinalIgnoreCase)) return null;
var bmp = Path.Combine(Paths.Textures, Path.GetFileNameWithoutExtension(a.Name) + ".BMP");
return File.Exists(bmp) ? bmp : null;
var entry = _catalog.ResolveScene(scene, resId) ?? _catalog.ResolveRaw(resId);
return entry is { IsPlaceholder: false } &&
entry.Name.EndsWith(".AGF", StringComparison.OrdinalIgnoreCase) ? entry : null;
}
/// <summary>Decode an AGF directly from loose-first VFS bytes.</summary>
public RgbaImage DecodeTexture(AssetEntry entry) => AgfDecoder.Decode(_store, entry);
public AssetEntry? ResolveName(string name) => _catalog.ResolveName(name);
/// <summary>
/// Resolve a BGM id to its OGG path. BGM is addressed by DIRECT LITERAL NAME (BGM{id:D3}.OGG), NOT the
/// per-scene section manifest that voices/textures use. Confirmed by ear (play-bgm 5->BGM005, 8->BGM008)

View File

@@ -55,8 +55,8 @@ public sealed class Sys4AssetCatalog
if (expandedSize == 0 || expandedSize > int.MaxValue)
throw new InvalidDataException($"{name}: invalid expanded size {expandedSize}");
var blob = DecompressLzss(data.AsSpan(PackedSizeOffset + 4, checked((int)packedSize)),
checked((int)expandedSize), name);
var blob = LzssDecoder.Decode(data.AsSpan(PackedSizeOffset + 4, checked((int)packedSize)),
checked((int)expandedSize), name);
int p = 0;
uint ReadU32()
{
@@ -145,41 +145,4 @@ public sealed class Sys4AssetCatalog
return Encoding.GetEncoding(932).GetString(bytes);
}
private static byte[] DecompressLzss(ReadOnlySpan<byte> source, int expectedSize, string name)
{
var frame = new byte[0x1000];
int framePos = 0xfee, input = 0, output = 0;
var result = new byte[expectedSize];
while (output < expectedSize)
{
if (input >= source.Length) throw new InvalidDataException($"{name}: LZSS stream ended early");
int control = source[input++];
for (int bit = 1; bit <= 0x80 && output < expectedSize; bit <<= 1)
{
if ((control & bit) != 0)
{
if (input >= source.Length) throw new InvalidDataException($"{name}: truncated LZSS literal");
byte value = source[input++];
result[output++] = value;
frame[framePos] = value;
framePos = (framePos + 1) & 0xfff;
}
else
{
if (input > source.Length - 2) throw new InvalidDataException($"{name}: truncated LZSS back-reference");
int lo = source[input++], hi = source[input++];
int readPos = ((hi & 0xf0) << 4) | lo;
int length = 3 + (hi & 0x0f);
for (int j = 0; j < length && output < expectedSize; j++)
{
byte value = frame[readPos++ & 0xfff];
result[output++] = value;
frame[framePos] = value;
framePos = (framePos + 1) & 0xfff;
}
}
}
}
return result;
}
}

View File

@@ -9,7 +9,8 @@ public sealed class GodotAdvHost : IHost
private readonly Main _main;
private readonly ResourceMap _res;
private readonly string _scene; // e.g. "SC0000" — for section_base
private readonly Dictionary<int, string?> _slotBmp = new(); // slot -> pre-converted BMP path
private readonly object _imageLock = new();
private readonly Dictionary<int, RgbaImage?> _images = new(); // raw catalog id -> decoded pixels
private readonly string?[] _sfxPaths = new string?[10]; // SC0000 native channel subset
// slot -> dims. Slot 0 is the primary/screen surface (800x600), normally created at engine boot which
// the single-scene harness skips; seed it so the first CG's anchor math stays correct (not 0x0).
@@ -230,22 +231,20 @@ public sealed class GodotAdvHost : IHost
public void CreateTexture(int slot, int width, int height)
{
lock (_textLock) _surfaceText.Remove(slot);
_slotBmp[slot] = null; _slotDims[slot] = (width, height);
_slotDims[slot] = (width, height);
if (TraceOps) Godot.GD.Print($"[op] create-texture slot={slot} {width}x{height}");
}
public void SetTexture(long resourceId, int slot)
{
lock (_textLock) _surfaceText.Remove(slot);
var asset = _res.Resolve(_scene, resourceId);
var bmp = asset != null ? ResourceMap.TexturePath(asset) : null;
_slotBmp[slot] = bmp;
_slotDims[slot] = BmpHeader.ReadDims(bmp); // synchronous: dims from the header, no Godot Image
if (TraceOps) Godot.GD.Print($"[op] set-texture slot={slot} resId=0x{resourceId:x} -> {(bmp != null ? System.IO.Path.GetFileName(bmp) : "<none>")}");
var asset = _res.ResolveTexture(_scene, resourceId);
var image = asset != null ? Decode(asset) : null;
_slotDims[slot] = image != null ? (image.Width, image.Height) : (0, 0);
if (TraceOps) Godot.GD.Print($"[op] set-texture slot={slot} resId=0x{resourceId:x} -> {(asset?.Name ?? "<none>")}");
}
// Dims are read from the BMP header on the VM thread so the bytecode's geometry math (which calls this
// synchronously right after set-texture) sees the real size. Pixels are blitted later on the main thread.
// AGF is decoded synchronously on the VM thread so geometry queried immediately afterward sees real dims.
public (int Width, int Height) GetTextureSize(int slot)
=> _slotDims.TryGetValue(slot, out var d) ? (d.W, d.H) : (0, 0);
@@ -253,12 +252,28 @@ public sealed class GodotAdvHost : IHost
// the visible objects each frame in ascending-handle order. No immediate blit here.
public void DrawTexture(int slot, int srcX, int srcY, int width, int height, int dstX, int dstY) { }
/// <summary>Resolve a gfx surface's resId to its pre-converted BMP path (Main's per-frame compositor
/// resolves each visible object's surface through this).</summary>
public string? ResolveResIdTexture(long resId)
/// <summary>Resolve a gfx surface through scene-local or universal raw-id addressing and decode it
/// from the loose-first asset store.</summary>
public (RgbaImage Image, string Name, int AssetId)? ResolveResIdTexture(long resId)
{
var asset = _res.Resolve(_scene, resId);
return asset != null ? ResourceMap.TexturePath(asset) : null;
var asset = _res.ResolveTexture(_scene, resId);
var image = asset != null ? Decode(asset) : null;
return asset != null && image != null ? (image, asset.Name, asset.RawIndex) : null;
}
private RgbaImage? Decode(AssetEntry asset)
{
lock (_imageLock)
{
if (_images.TryGetValue(asset.RawIndex, out var cached)) return cached;
try { return _images[asset.RawIndex] = _res.DecodeTexture(asset); }
catch (System.Exception e)
{
Godot.GD.Print($"AGF decode failed {asset.Name}: {e.Message}");
_images[asset.RawIndex] = null;
return null;
}
}
}
// ---- audio ops (OGG plays natively in Godot) ----

View File

@@ -154,7 +154,8 @@ public partial class Main : Godot.Control
if (_selftest) (script, provider) = BuildSelfTestScene(table);
else { scripts = Sys4ScriptProvider.Load(table); script = scripts.RequireByName(scene + ".BIN"); provider = scripts; }
if (_timelineLogPath != null) _timeline = new GodotTimelineLog(_timelineLogPath);
_host = new GodotAdvHost(this, scripts != null ? new ResourceMap(scripts.Catalog) : ResourceMap.Load(), scene, _clock, _timeline) { SleepScale = sleepScale, TraceOps = _gfxLogPath != null };
var resources = scripts != null ? new ResourceMap(scripts.Catalog) : ResourceMap.Load();
_host = new GodotAdvHost(this, resources, scene, _clock, _timeline) { SleepScale = sleepScale, TraceOps = _gfxLogPath != null };
_trace = new GodotTraceSink(_timeline);
// --trace-histogram: aggregate op/call-site execution counts of the REAL Godot run (headless flow
// diverges — wait-for-input is a no-op there — so this is the only way to profile the live path).
@@ -164,6 +165,13 @@ public partial class Main : Godot.Control
if (histFile != null) { _hist = new Age.Engine.Diagnostics.HistogramTraceSink();
sink = new Age.Engine.Diagnostics.CompositeTraceSink(_trace, _hist); }
_vm = new VirtualMachine(script, table, _host, new VmOptions(MaxSteps: 20_000_000), provider, sink);
// SYSTEM4 loads the shared SO001 chrome sheet into surface slot 17 before any scene runs.
// Seed that inherited retained-surface state without replaying the entrypoint's unrelated UI flow.
if (!_selftest && resources.ResolveName("SO001.AGF") is { } systemChrome)
{
_host.SetTexture(systemChrome.RawIndex, 0x11);
_vm.Gfx.SetSurface(0x11, systemChrome.RawIndex, 0);
}
// --boot: run SYSTEM4's state prefix (INITCONFIG/INIT2/INIT) so the scene sees boot state — chiefly
// INIT2's gfx handle array 0x62455.. (skips the UI scripts LOGO/OP/TITLE). State carries via globals.
if (boot && !_selftest)
@@ -279,10 +287,10 @@ public partial class Main : Godot.Control
// ---- retained per-frame compositor (main thread, from _Process) ----
// Clear the screen and composite the VM's current VISIBLE gfx objects in ascending-handle order (= the
// engine's z-order), each blitting its live surface's rect at its position. Surfaces are cached by BMP
// path (this runs every frame). Native scale/translation matrix channels are sampled independently by
// engine's z-order), each blitting its live surface's rect at its position. Decoded AGF surfaces are cached
// by catalog identity (this runs every frame). Native scale/translation matrix channels are sampled independently by
// GfxState and applied here; object opacity comes only from the actual blend/color path.
private readonly System.Collections.Generic.Dictionary<(string Path, long Key), Image?> _imgCache = new();
private readonly System.Collections.Generic.Dictionary<(int AssetId, long Key), Image?> _imgCache = new();
private void Recomposite()
{
@@ -328,14 +336,14 @@ public partial class Main : Godot.Control
}
else
{
var bmp = _host.ResolveResIdTexture(v.SurfaceResId);
if (bmp == null) outcome = $"SKIP(resId=0x{v.SurfaceResId:x} UNRESOLVED)";
var texture = _host.ResolveResIdTexture(v.SurfaceResId);
if (texture == null) outcome = $"SKIP(resId=0x{v.SurfaceResId:x} UNRESOLVED)";
else
{
BlitLayer(bmp, v.ColorKey, v.Tint, strength, v.SrcX, v.SrcY, v.W, v.H,
BlitLayer(texture.Value.Image, texture.Value.AssetId, v.ColorKey, v.Tint, strength, v.SrcX, v.SrcY, v.W, v.H,
localToDest, opacity, v.MultiplyTint);
var raw = _vm.Gfx.TryGet(v.Handle);
outcome = $"slot={raw?.SourceSlot} DRAWN resId=0x{v.SurfaceResId:x} {System.IO.Path.GetFileName(bmp)} " +
outcome = $"slot={raw?.SourceSlot} DRAWN resId=0x{v.SurfaceResId:x} {texture.Value.Name} " +
$"src=({v.SrcX},{v.SrcY} {v.W}x{v.H}) base=({v.DstX},{v.DstY}) " +
$"anchor=({t.AnchorX:0.0},{t.AnchorY:0.0}) dst=({dstX},{dstY}) " +
$"scale=({t.ScaleX:0.00},{t.ScaleY:0.00}) trans=({t.TranslateX:0.0},{t.TranslateY:0.0}) " +
@@ -395,9 +403,9 @@ public partial class Main : Godot.Control
}
else
{
var bmp = _host.ResolveResIdTexture(source.SurfaceResId);
if (bmp == null) continue;
BlitLayer(bmp, source.ColorKey, source.Tint, source.TintStrength / 255f,
var texture = _host.ResolveResIdTexture(source.SurfaceResId);
if (texture == null) continue;
BlitLayer(texture.Value.Image, texture.Value.AssetId, source.ColorKey, source.Tint, source.TintStrength / 255f,
source.SrcX, source.SrcY, source.W, source.H, affine, opacity, source.MultiplyTint);
}
drawn++;
@@ -439,20 +447,14 @@ public partial class Main : Godot.Control
// matching the surface colorkey are made transparent (native bakes the key at load — engine-re.md §Blend).
// Mode 0 uses tintStrength to LERP texel RGB toward tint. Mode 1 sets multiplyTint and uses packed RGB as
// multiplicative modulation while alpha is object opacity.
private void BlitLayer(string bmpPath, long colorKey, long tint, float tintStrength, int srcX, int srcY, int w, int h,
private void BlitLayer(RgbaImage decoded, int assetId, long colorKey, long tint, float tintStrength, int srcX, int srcY, int w, int h,
Age.Engine.Model.Affine2D localToDest, float alpha = 1f, bool multiplyTint = false)
{
var cacheKey = (bmpPath, colorKey);
var cacheKey = (assetId, colorKey);
if (!_imgCache.TryGetValue(cacheKey, out var src))
{
src = new Image();
if (src.LoadBmpFromBuffer(System.IO.File.ReadAllBytes(bmpPath)) != Error.Ok)
{ GD.Print($"BMP load failed {bmpPath}"); src = null; }
else
{
if (src.GetFormat() != Image.Format.Rgba8) src.Convert(Image.Format.Rgba8);
if (Age.Engine.Model.BlendMath.HasColorKey(colorKey)) BakeColorKey(src, colorKey);
}
src = Image.CreateFromData(decoded.Width, decoded.Height, false, Image.Format.Rgba8, decoded.Pixels);
if (Age.Engine.Model.BlendMath.HasColorKey(colorKey)) BakeColorKey(src, colorKey);
_imgCache[cacheKey] = src;
}
if (src == null) return;