conco/OpenMaidEngine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Mount SYS4 append asset catalogs

Browse Source
This commit is contained in:
gamer147
2026-07-11 10:42:33 -04:00
parent ec07c76721
commit d6da43c818
9 changed files with 235 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
docs/asset-resolution-re.md
View File

@@ -166,7 +166,7 @@ The current Phase-A backend deliberately continues through the extracted-file bo
accepts both OGG and WAV and Godot loads the WAV bytes into its fixed SC0000 channel pool. This does not change
the scoped VFS plan below: ALF/AAI mounting and in-process asset reads remain a separate foundation track.
## Runtime asset-VFS track (VFS-A and VFS-C complete 2026-07-11)
## Runtime asset-VFS track (VFS-A/B/C complete 2026-07-11)
The pre-extracted tree and `build/textures/*.BMP` pipeline were a Phase-A bootstrap, not the desired final
runtime. The native-compatible target is a read-only virtual filesystem that preserves AGE's translation/mod
@@ -194,11 +194,16 @@ store.
`call-script` targets through this seam; `ResourceMap` uses the same live catalog. `build/asset-index.json`,
`build/asset-sections.json`, `build/callscript-names.json`, and `extracted/` are validation/temporary
graphics-audio artifacts, not script-runtime dependencies.
2. **AAI append mount.** Parse the installed `APPEND01.AAI` (`S4AC422`) and its paired `APPEND01.ALF` with
the same catalog abstractions. First prove whether Himegari joins append records by a separate pack/tag,
by name replacement, or by another table selected by the native high-byte-id path; do not invent mount
precedence. Validate every parsed append entry against `BinExtractALF.exe` output before exposing it to
the runtime.
2. **AAI append mount (VFS-B DONE).** `Sys4AssetCatalog` also parses the installed `APPEND01.AAI`
(`S4AC422`) and mounts its paired `APPEND01.ALF`. This is a separate selector-keyed catalog, not a
filename overlay: AGE scans `*.AAI`, reads the selector at AAI header offset `0x108`, and stores the
successfully loaded catalog in `mounted_aai[selector]`. A resource id with a nonzero high byte selects
that slot; its low 24 bits index only the selected append table. Installed `APPEND01` is selector 1.
Within that catalog the ordinary exact loose-record-name then ALF-range precedence still applies, so the
literal `$1$...` names are preserved. A later successfully enumerated AAI with the same selector replaces
the earlier pointer in the native loop; no base/append name replacement occurs. Native extracts the
selector with arithmetic `SAR 24`; the port rejects sign-bit selectors rather than guessing behavior for
ids that would index before AGE's mount table.
3. **AGF decoder (VFS-C DONE).** `Age.Engine/Sys4/AgfDecoder.cs` decodes an opened AGF payload directly
to a tightly packed, top-down width/height + RGBA8 surface. The MIT-licensed GARbro
`ArcFormats/Eushully/ImageAGF.cs` provides a compact reference: `ACGF` (or zero) signature, type 1/2,
@@ -246,8 +251,15 @@ Five installed assets spanning raw/compressed metadata and pixel/alpha combinati
800×300 with intermediate alpha values, and is inherited in surface slot 17 before SC0000. A windowed
page-1 capture with `build/textures/` moved aside showed the translucent textbox edge and bottom-right
controls. Texture runtime no longer consults `extracted/` or `build/textures/`; current audio consumers
still use extracted OGG/WAV paths by design. APPEND01/AAI, audio migration, and movie `0x236` remain
unimplemented by this slice.
still use extracted OGG/WAV paths by design. Audio migration and movie `0x236` remain unimplemented.
VFS-B passes its bounded gates in `Sys4AssetStoreTests`: the installed AAI expands from the LZSS stream at
`0x118` (expanded size at `0x110`, packed size at `0x114`) to one `APPEND01.ALF` archive and 81 80-byte
records. All records carry selector 1 and literal `$1$` names. The full directory has stable SHA-256
`23F0C104A45C099CEFB7D333362716EDE6F20B9EC53E4C3705A8E3A87063708E` over its ordered record fields.
The integration gate runs `BinExtractALF.exe` into a disposable directory, compares all 81 names, validates
every range and size, and byte-compares all 81 payloads. `Sys4ScriptProvider` resolves a real append script
through `0x01xxxxxx`; direct base-name lookup deliberately does not see append records.
### Deliberate non-goals

19
docs/engine-re.md
View File

@@ -1045,6 +1045,25 @@ includes the scene's own early writes (can exclude by codebase for a pure pre-sc
---
### S4AC append catalogs and packed resource ids (2026-07-11)
`asset_mount_append_catalogs@0x44f120` scans `*.AAI`, constructs `AAIFileDB` objects, and calls
`aai_catalog_load@0x401110`. Installed `S4AC422` uses selector `1` at header offset `0x108`, expanded size
at `0x110`, packed size at `0x114`, and an LZSS directory stream at `0x118`. The expanded directory is the
same archive-count / 256-byte archive-name / file-count / 80-byte-record layout as SYS4INI. Its 81 names are
literal `$1$...` basenames in `APPEND01.ALF`.
After a successful load, the scan writes the catalog pointer to `FileDB+0x3028 + selector*4`; a later
successful discovery of the same selector overwrites that slot. `asset_open_indexed_entry@0x44f390` splits
nonzero-high-byte ids into `mounted_aai[id >> 24]` and record index `id & 0xffffff`, while high-byte-zero ids
stay in the base SYS4 table. `aai_open_indexed_entry@0x401630` then applies the same exact loose basename
before indexed ALF fallback as the base path. Thus append selection is explicit pack selection, not filename
replacement. The selector extraction is an arithmetic `SAR 24`, so ids whose high byte has its sign bit set
index before the mount table rather than slots `0x80..0xff`; the port rejects those selectors instead of
inventing unsigned behavior. `/v2` names/comments this mount/load/open chain and is saved.
---
## Native walls backlog (targets for this loop)
- ~~**call-script dispatch**~~ — **SOLVED** (above): `call-script <id>` = raw SYS4INI file index.

24
docs/phase-a-slice-plan.md
View File

@@ -1050,7 +1050,7 @@ opcode/ctx lint, 481-script decode, RECOVER, and `git diff --check` clean. SC000
**85/129 to 87/129 handled (67.4%)**, with 42 GAP ops / 188 GAP instructions. Movie `0x236` and SFX remain
separate slices. No implementation commit was made.
### Foundation track — native asset VFS + ALF/AAI/AGF readers (VFS-A/C DONE; VFS-B PENDING)
### Foundation track — native asset VFS + ALF/AAI/AGF readers (VFS-A/B/C DONE)
This is an optional high-leverage detour before movie `0x236` or SFX. It replaces the Phase-A
pre-extracted/pre-converted asset bootstrap with the native loose-override/archive-fallback model and removes
@@ -1125,6 +1125,28 @@ Validation: 124/124 engine tests, CLI/Godot builds, Godot threaded selftest, pix
assets, and the windowed no-BMP capture. VFS-B APPEND01/AAI, OGG/WAV migration, movie `0x236`, and unrelated
opcode work remain separate.
### VFS-B — APPEND01 AAI mount DONE (2026-07-11)
`Sys4AssetCatalog.Load` now discovers installed `*.AAI` files beside `SYS4INI.BIN`, parses the `S4AC`
directory at its native offsets, and mounts each catalog by the selector stored at header offset `0x108`.
For Himegari, `APPEND01.AAI` mounts selector 1, names one `APPEND01.ALF`, and contains 81 records whose
literal names all begin `$1$`. `AssetEntry` retains the pack id, `ResolvePacked` applies the evidenced
`high byte -> mounted catalog; low 24 bits -> record` rule, and `Sys4ScriptProvider` routes append call ids
through the existing `IAssetStore`. Base direct-name lookup remains base-only: append records are not guessed
name replacements. The selected catalog still probes the record's exact loose basename before its ALF range.
Native `/v2` now names/comments the AAI constructor/load/table/open/get chain and
`asset_mount_append_catalogs@0x44f120`; `asset_open_indexed_entry@0x44f390` documents the packed-id split.
The native scan stores each successful catalog directly into its selector slot, so a later enumerated AAI
with the same selector replaces the earlier pointer. Its `SAR 24` selector extraction makes sign-bit high
bytes negative table indexes, so the port rejects them rather than inventing `0x80..0xff` behavior. Only
selector 1 is installed and exercised here.
Validation: 127/127 engine tests; stable complete-directory digest; all 81 ranges fit; a disposable
`BinExtractALF.exe APPEND01.AAI` extraction matches all 81 names, sizes, and payload bytes; real
`$1$SC1260.BIN` loads through its packed id; CLI/Godot builds and threaded `SELFTEST OK`. Audio migration,
movie `0x236`, AGF work, and unrelated opcodes were untouched.
### Phase A — native SC0000 SFX family (`0xb4`/`0xb5`/`0xb6`/`0xc2`/`0xd9`) DONE (2026-07-11)
Native RE and the matching trace resolve the bounded family. `0xb4(resource,channel)` synchronously loads

6
docs/remake-architecture-and-roadmap.md
View File

@@ -70,9 +70,9 @@ Three layers, cleanly separated:
modders. They share the *format spec* (documented), not code — acceptable for a small, stable
container format.
- **Native content foundation status (2026-07-11).** VFS-A runtime-parses the base SYS4 catalog and
applies loose-first bounded ALF reads; VFS-C decodes AGF directly to platform-neutral RGBA8 and feeds
Godot without pre-extracted or pre-converted texture files. APPEND01/AAI mounting and audio-byte consumers
remain separate follow-ups.
applies loose-first bounded ALF reads; VFS-B mounts selector-keyed S4AC append catalogs and resolves the
native high-byte/low-24-bit id split; VFS-C decodes AGF directly to platform-neutral RGBA8 and feeds Godot
without pre-extracted or pre-converted texture files. Audio-byte consumers remain a separate follow-up.
- **Profile = manifest.** Adding a game = a new profile + its maps. Adding an engine version = a new
front-end plugin + profiles that reference it. See §5.

82
engine/Age.Engine.Tests/Sys4AssetStoreTests.cs
View File

@@ -1,9 +1,91 @@
using System.Text.Json;
using System.Security.Cryptography;
using System.Text;
using System.Diagnostics;
using Age.Engine.Sys4;
using Xunit;
public class Sys4AssetStoreTests
{
[Fact]
public void InstalledAppendCatalogHasNativePackSelectionAndStableDirectory()
{
var catalog = Sys4AssetCatalog.Load(Paths.Sys4Ini);
var append = Assert.Single(catalog.AppendPacks).Value;
Assert.Equal(1, append.PackId);
Assert.Equal("S4AC422 ", append.Magic);
Assert.Equal(catalog.Title, append.Title);
Assert.Equal(new[] { "APPEND01.ALF" }, append.Archives);
Assert.Equal(81, append.RawSlots.Count);
Assert.Equal(81, append.Files.Count);
Assert.All(append.Files, entry =>
{
Assert.Equal(1, entry.PackId);
Assert.StartsWith("$1$", entry.Name);
Assert.Equal("APPEND01.ALF", entry.Archive);
});
Assert.Same(append.RawSlots[0], catalog.ResolvePacked(0x01000000));
Assert.Same(append.RawSlots[^1], catalog.ResolvePacked(0x01000050));
Assert.Null(catalog.ResolvePacked(0x02000000));
Assert.Null(catalog.ResolvePacked(0x80000000));
Assert.Null(catalog.ResolveName(append.RawSlots[0].Name));
string directory = string.Join("\n", append.RawSlots.Select(e =>
$"{e.RawIndex}|{e.Name}|{e.ArchiveId}|{e.Archive}|{e.FileNumber}|{e.Offset}|{e.Size}"));
string digest = Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(directory)));
Assert.Equal("23F0C104A45C099CEFB7D333362716EDE6F20B9EC53E4C3705A8E3A87063708E", digest);
}
[Fact]
public void CompleteAppendDirectoryAndPayloadsMatchBinExtractAlf()
{
string temp = Path.Combine(Path.GetTempPath(), "age-vfsb-oracle-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(temp);
try
{
var start = new ProcessStartInfo(Paths.BinExtractAlf)
{
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true,
CreateNoWindow = true,
};
start.ArgumentList.Add(Paths.Append01Aai);
start.ArgumentList.Add(temp);
using var process = Process.Start(start)!;
string stdout = process.StandardOutput.ReadToEnd();
string stderr = process.StandardError.ReadToEnd();
process.WaitForExit();
Assert.True(process.ExitCode == 0, stdout + stderr);
var catalog = Sys4AssetCatalog.Load(Paths.Sys4Ini);
var append = catalog.AppendPacks[1];
var store = new Sys4AssetStore(catalog, Paths.GameDir, Paths.GameDir);
string output = Path.Combine(temp, "APPEND01");
var oracleNames = Directory.EnumerateFiles(output).Select(Path.GetFileName)
.Order(StringComparer.OrdinalIgnoreCase).ToArray();
var actualNames = append.Files.Select(e => e.Name)
.Order(StringComparer.OrdinalIgnoreCase).ToArray();
Assert.Equal(actualNames, oracleNames);
long archiveLength = new FileInfo(Path.Combine(Paths.GameDir, "APPEND01.ALF")).Length;
foreach (var entry in append.Files)
{
Assert.InRange(entry.Offset, 0, archiveLength);
Assert.InRange(entry.Size, 0, archiveLength - entry.Offset);
byte[] oracle = File.ReadAllBytes(Path.Combine(output, entry.Name));
Assert.Equal(entry.Size, oracle.LongLength);
Assert.Equal(oracle, store.ReadAll(entry));
}
}
finally
{
Directory.Delete(temp, recursive: true);
}
}
[Fact]
public void RuntimeCatalogMatchesDiagnosticCatalogAndSceneViews()
{

15
engine/Age.Engine.Tests/Sys4ScriptProviderTests.cs
View File

@@ -18,6 +18,21 @@ public class Sys4ScriptProviderTests
Assert.Null(provider.GetById(long.MaxValue)); // unknown id
}
[Fact]
public void HighByteSelectsAppendPackWithoutReplacingBaseNames()
{
var table = OpcodeTableJson.Load(Paths.OpcodesJson);
var provider = Sys4ScriptProvider.Load(table);
var append = provider.Catalog.AppendPacks[1];
var entry = append.Files.Single(e => e.Name == "$1$SC1260.BIN");
long packedId = 0x01000000L | (uint)entry.RawIndex;
var script = provider.GetById(packedId);
Assert.NotNull(script);
Assert.True(script!.Instructions.Count > 0);
Assert.Null(provider.GetByName("$1$SC1260.BIN"));
}
[Fact]
public void RootScriptLoadingUsesTheSameAssetStoreAndLoosePrecedence()
{

2
engine/Age.Engine/Sys4/Paths.cs
View File

@@ -13,6 +13,8 @@ public static class Paths
public static string CallscriptNamesJson => Path.Combine(Build, "callscript-names.json");
public static string Textures => Path.Combine(Build, "textures");
public static string Sys4Ini => Path.Combine(GameDir, "SYS4INI.BIN");
public static string Append01Aai => Path.Combine(GameDir, "APPEND01.AAI");
public static string BinExtractAlf => Path.Combine(Repo, "bin", "BinExtractALF.exe");
private static string FindRepo()
{

84
engine/Age.Engine/Sys4/Sys4AssetCatalog.cs
View File

@@ -13,27 +13,33 @@ public sealed record AssetEntry(
int RawIndex = -1,
int ArchiveId = -1,
int FileNumber = -1,
bool IsPlaceholder = false);
bool IsPlaceholder = false,
int PackId = 0);
/// <summary>Runtime parser and lookup views for a base S4IC SYS4INI catalog.</summary>
/// <summary>Runtime parser and lookup views for a base S4IC SYS4INI catalog and its S4AC append mounts.</summary>
public sealed class Sys4AssetCatalog
{
private const int PackedSizeOffset = 0x134;
private const int ExpandedSizeOffset = 0x12c;
private const int ArchiveNameSize = 256;
private const int RecordSize = 80;
private readonly Dictionary<string, AssetEntry> _byName;
private readonly Dictionary<string, (int Start, int End)> _sceneRanges;
private readonly Dictionary<int, Sys4AssetCatalog> _appendPacks = new();
public string Magic { get; }
public string Title { get; }
public int PackId { get; }
public IReadOnlyList<string> Archives { get; }
public IReadOnlyList<AssetEntry> RawSlots { get; }
public IReadOnlyList<AssetEntry> Files { get; }
public IReadOnlyDictionary<int, Sys4AssetCatalog> AppendPacks => _appendPacks;
private Sys4AssetCatalog(string magic, List<string> archives, List<AssetEntry> rawSlots)
private Sys4AssetCatalog(string magic, string title, int packId,
List<string> archives, List<AssetEntry> rawSlots)
{
Magic = magic;
Title = title;
PackId = packId;
Archives = archives;
RawSlots = rawSlots;
Files = rawSlots.Where(r => !r.IsPlaceholder).ToArray();
@@ -41,21 +47,38 @@ public sealed class Sys4AssetCatalog
_sceneRanges = BuildSceneRanges(Files);
}
public static Sys4AssetCatalog Load(string path) => Parse(File.ReadAllBytes(path), Path.GetFileName(path));
public static Sys4AssetCatalog Load(string path)
{
var catalog = Parse(File.ReadAllBytes(path), Path.GetFileName(path));
if (!catalog.Magic.StartsWith("S4IC", StringComparison.Ordinal)) return catalog;
string? root = Path.GetDirectoryName(Path.GetFullPath(path));
if (root == null) return catalog;
foreach (string appendPath in Directory.EnumerateFiles(root, "*.AAI"))
catalog.MountAppend(Parse(File.ReadAllBytes(appendPath), Path.GetFileName(appendPath)));
return catalog;
}
public static Sys4AssetCatalog Parse(byte[] data, string name = "SYS4INI.BIN")
{
if (data.Length < PackedSizeOffset + 4 || !data.AsSpan(0, 4).SequenceEqual("S4IC"u8))
throw new InvalidDataException($"{name}: expected an S4IC catalog");
bool isBase = data.Length >= 4 && data.AsSpan(0, 4).SequenceEqual("S4IC"u8);
bool isAppend = data.Length >= 4 && data.AsSpan(0, 4).SequenceEqual("S4AC"u8);
if (!isBase && !isAppend)
throw new InvalidDataException($"{name}: expected an S4IC or S4AC catalog");
uint expandedSize = BinaryPrimitives.ReadUInt32LittleEndian(data.AsSpan(ExpandedSizeOffset, 4));
uint packedSize = BinaryPrimitives.ReadUInt32LittleEndian(data.AsSpan(PackedSizeOffset, 4));
if (packedSize > data.Length - (PackedSizeOffset + 4))
int expandedSizeOffset = isAppend ? 0x110 : 0x12c;
int packedSizeOffset = isAppend ? 0x114 : 0x134;
if (data.Length < packedSizeOffset + 4)
throw new InvalidDataException($"{name}: catalog header is truncated");
uint expandedSize = BinaryPrimitives.ReadUInt32LittleEndian(data.AsSpan(expandedSizeOffset, 4));
uint packedSize = BinaryPrimitives.ReadUInt32LittleEndian(data.AsSpan(packedSizeOffset, 4));
if (packedSize > data.Length - (packedSizeOffset + 4))
throw new InvalidDataException($"{name}: packed directory is truncated");
if (expandedSize == 0 || expandedSize > int.MaxValue)
throw new InvalidDataException($"{name}: invalid expanded size {expandedSize}");
var blob = LzssDecoder.Decode(data.AsSpan(PackedSizeOffset + 4, checked((int)packedSize)),
var blob = LzssDecoder.Decode(data.AsSpan(packedSizeOffset + 4, checked((int)packedSize)),
checked((int)expandedSize), name);
int p = 0;
uint ReadU32()
@@ -76,6 +99,13 @@ public sealed class Sys4AssetCatalog
uint fileCount = ReadU32();
if (fileCount is 0 or >= 0x400000 || fileCount > (blob.Length - p) / RecordSize)
throw new InvalidDataException($"{name}: invalid file count {fileCount}");
int packId = isAppend
? checked((int)BinaryPrimitives.ReadUInt32LittleEndian(data.AsSpan(0x108, 4))) : 0;
// AGE uses SAR 24 before indexing its mount table, so selectors with the sign bit set do
// not address slots 0x80..0xff. Reject them instead of inventing unsigned behavior.
if (isAppend && packId is not (> 0 and < 0x80))
throw new InvalidDataException($"{name}: invalid append pack selector {packId}");
var slots = new List<AssetEntry>(checked((int)fileCount));
for (int i = 0; i < fileCount; i++, p += RecordSize)
{
@@ -87,17 +117,43 @@ public sealed class Sys4AssetCatalog
long size = BinaryPrimitives.ReadUInt32LittleEndian(row.Slice(76, 4));
string archive = archiveId >= 0 && archiveId < archives.Count ? archives[archiveId] : "";
bool placeholder = fileName is "" or "@";
slots.Add(new AssetEntry(fileName, archive, offset, size, i, archiveId, fileNumber, placeholder));
slots.Add(new AssetEntry(fileName, archive, offset, size, i, archiveId, fileNumber,
placeholder, packId));
}
string magic = ReadCString(data.AsSpan(0, Math.Min(8, data.Length)));
return new Sys4AssetCatalog(magic, archives, slots);
string title = ReadCString(data.AsSpan(8, Math.Min(256, data.Length - 8)));
return new Sys4AssetCatalog(magic, title, packId, archives, slots);
}
/// <summary>Mount an append catalog by its native header selector. A later mount of the same
/// selector replaces the earlier one, matching AGE's FindNextFile/store loop.</summary>
public void MountAppend(Sys4AssetCatalog append)
{
if (PackId != 0) throw new InvalidOperationException("append catalogs cannot own append mounts");
if (!append.Magic.StartsWith("S4AC", StringComparison.Ordinal) || append.PackId == 0)
throw new InvalidDataException("mounted catalog is not an S4AC append pack");
if (!string.Equals(Title, append.Title, StringComparison.Ordinal))
throw new InvalidDataException($"append title mismatch: {append.Title}");
_appendPacks[append.PackId] = append;
}
/// <summary>Universal raw-id lookup. Placeholder slots are returned, not collapsed.</summary>
public AssetEntry? ResolveRaw(long rawId)
=> rawId >= 0 && rawId < RawSlots.Count ? RawSlots[(int)rawId] : null;
/// <summary>Native packed-id lookup: zero high byte selects SYS4INI; otherwise the high byte
/// selects an S4AC mount and the low 24 bits index that pack.</summary>
public AssetEntry? ResolvePacked(long id)
{
if (id < 0 || id > uint.MaxValue) return null;
int pack = (int)((id >> 24) & 0xff);
if (pack >= 0x80) return null;
long index = id & 0xffffff;
return pack == 0 ? ResolveRaw(index)
: _appendPacks.TryGetValue(pack, out var append) ? append.ResolveRaw(index) : null;
}
/// <summary>Case-insensitive exact-name lookup over real records.</summary>
public AssetEntry? ResolveName(string name)
=> _byName.TryGetValue(Path.GetFileName(name), out var entry) && Path.GetFileName(name) == name

2
engine/Age.Engine/Sys4/Sys4ScriptProvider.cs
View File

@@ -27,7 +27,7 @@ public sealed class Sys4ScriptProvider : IScriptProvider
public Script? GetById(long id)
{
if (_cache.TryGetValue(id, out var cached)) return cached;
var entry = Catalog.ResolveRaw(id);
var entry = Catalog.ResolvePacked(id);
Script? script = entry is { IsPlaceholder: false }
&& entry.Name.EndsWith(".BIN", StringComparison.OrdinalIgnoreCase)
? Parse(entry) : null;