conco/OpenMaidEngine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Sanitize CP932 WAV metadata for Godot

Browse Source
This commit is contained in:
gamer147
2026-07-22 10:00:13 -04:00
parent 4f5137a98f
commit fb84bc3fa9
6 changed files with 204 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
docs/asset-resolution-re.md
View File

@@ -169,6 +169,21 @@ The Phase-A backend now resolves the OGG/WAV catalog entry and opens it through
the returned bytes into its existing BGM, voice, and fixed SC0000 SFX channel players. The earlier
`ResourceMap.AudioPath` extracted-file bootstrap is retired.
Godot's WAV loader adds one host-specific compatibility boundary. SC0000 `0xc29` starts `E0808.WAV` with
the EV052DA glow. Its PCM is valid, but its trailing RIFF `LIST/INFO` fields contain Japanese CP932 text
(`IPRD`, `IGNR`, and `ICMT`). Godot assumes INFO text is UTF-8 and formerly emitted one Unicode warning per
invalid CP932 byte each time the sound was loaded; the duplicated SC0000 burst came from loading raw id
`0x28` on two channels. A complete extracted-corpus scan found 238 RIFF/WAVE files, 61 INFO chunks, no other
LIST type, and no invalid RIFF containers, explaining the same warnings around combat SFX.
The correction is deliberately confined to the Godot frontend. `RiffWaveSanitizer` removes only `LIST` chunks
whose form type is `INFO` from the transient byte array passed to `AudioStreamWav.LoadFromBuffer`, updates the
RIFF length, and preserves every functional chunk (`fmt `, `data`, `smpl`, `cue `, and unknown chunks)
byte-for-byte, including padding. The shared engine, original loose/archive payloads, and
`ResourceMap.ReadAudio` output remain untouched. The real E0808 Godot input shrinks from 688,570 to 688,336
bytes while retaining an identical PCM data chunk and loads headlessly without a Unicode warning. Synthetic
chunk/padding tests and the installed E0808 regression cover the adapter.
## Runtime asset-VFS track (VFS-A/B/C complete 2026-07-11)
The pre-extracted tree and `build/textures/*.BMP` pipeline were a Phase-A bootstrap, not the desired final

10
docs/phase-b-framework.md
View File

@@ -685,6 +685,16 @@ that invalid fallback; it does not retain the frame or decoder. The focused clea
tests, zero-warning Godot build, and threaded selftest pass. **NEXT:** perform a longer combat/return-to-FIELD
acceptance run, then delete DirectShow if the remaining live gate stays clean.
An independent long-standing console-spam issue was also localized at the same SC0000 third-CG boundary.
`play-sound-effect 0x28` loads `E0808.WAV`, whose RIFF `LIST/INFO` metadata uses CP932; Godot assumes those
unused fields are UTF-8 and logged each invalid byte twice because the script loaded the effect on two channels.
This was neither the AE glow renderer nor corrupt PCM. The complete extracted corpus contains 61 affected INFO
chunks among 238 valid WAVs, which also explains intermittent combat spam. A Godot-only adapter now strips only
INFO metadata from the transient buffer immediately before `AudioStreamWav.LoadFromBuffer`; the shared engine,
VFS bytes, and functional RIFF chunks are unchanged. Synthetic preservation tests and the real E0808 regression
pass, as do all 331 engine tests, the zero-warning Godot build, and the headless loader selftest with no Unicode
warnings.
**Mutable-surface fill/blend regression corrected.** The first visual recheck exposed BUNKI's menu interior
as transparent. SYSTEM4 creates 800x600 surface 3 and fills it opaque white through `0x20b`; the metadata-only
host fill left the new pixel buffer transparent. Implementing the fill alone made the panel solid gray and

1
engine/Age.Engine.Tests/Age.Engine.Tests.csproj
View File

@@ -30,6 +30,7 @@
<Compile Include="..\..\godot\GodotTimelineLog.cs" Link="GodotTimelineLog.cs" />
<Compile Include="..\..\godot\GodotTraceSink.cs" Link="GodotTraceSink.cs" />
<Compile Include="..\..\godot\MovieSurfaceRegistry.cs" Link="MovieSurfaceRegistry.cs" />
<Compile Include="..\..\godot\RiffWaveSanitizer.cs" Link="RiffWaveSanitizer.cs" />
<Compile Include="..\..\tools\movie-corpus-gate\MovieCorpusGate.cs" Link="MovieCorpusGate.cs" />
<Compile Include="..\..\godot\DirectShowMovieDecoder.cs" Link="DirectShowMovieDecoder.cs" />
</ItemGroup>

94
engine/Age.Engine.Tests/RiffWaveSanitizerTests.cs Normal file
View File

@@ -0,0 +1,94 @@
using System.Buffers.Binary;
using System.Text;
using Age.Engine.Sys4;
public class RiffWaveSanitizerTests
{
[Fact]
public void RemovesOnlyInfoListAndPreservesFunctionalChunksAndPadding()
{
byte[] fmt = { 1, 0, 1, 0 };
byte[] info = { (byte)'I', (byte)'N', (byte)'F', (byte)'O',
(byte)'I', (byte)'P', (byte)'R', (byte)'D',
3, 0, 0, 0, 0x81, 0x45, 0 };
byte[] sampleLoop = { 9, 8, 7, 6 };
byte[] data = { 1, 2, 3 };
byte[] source = Wave(("fmt ", fmt), ("LIST", info), ("smpl", sampleLoop), ("data", data));
byte[] sanitized = RiffWaveSanitizer.RemoveInfoMetadata(source);
Assert.NotSame(source, sanitized);
Assert.Equal(sanitized.Length - 8, BinaryPrimitives.ReadInt32LittleEndian(sanitized.AsSpan(4, 4)));
Assert.Equal(new[] { "fmt ", "smpl", "data" }, ChunkIds(sanitized));
Assert.Equal(fmt, Chunk(sanitized, "fmt "));
Assert.Equal(sampleLoop, Chunk(sanitized, "smpl"));
Assert.Equal(data, Chunk(sanitized, "data"));
}
[Fact]
public void LeavesWaveWithoutInfoMetadataUnchanged()
{
byte[] source = Wave(("fmt ", new byte[] { 1, 0 }), ("data", new byte[] { 1, 2 }));
Assert.Same(source, RiffWaveSanitizer.RemoveInfoMetadata(source));
}
[Fact]
public void InstalledSc0000GlowSoundDropsCp932InfoBlockWithoutChangingPcmData()
{
var resources = ResourceMap.Load();
AudioPayload audio = resources.ReadAudio(resources.ResolveSoundEffect(0x28)!);
byte[] sanitized = RiffWaveSanitizer.RemoveInfoMetadata(audio.Bytes);
Assert.Equal("E0808.WAV", audio.Name);
Assert.Equal(688_570, audio.Bytes.Length);
Assert.Equal(688_336, sanitized.Length);
Assert.DoesNotContain("LIST", ChunkIds(sanitized));
Assert.Equal(Chunk(audio.Bytes, "data"), Chunk(sanitized, "data"));
}
private static byte[] Wave(params (string Id, byte[] Payload)[] chunks)
{
int length = 12 + chunks.Sum(chunk => 8 + chunk.Payload.Length + (chunk.Payload.Length & 1));
var result = new byte[length];
Encoding.ASCII.GetBytes("RIFF").CopyTo(result, 0);
BinaryPrimitives.WriteInt32LittleEndian(result.AsSpan(4, 4), length - 8);
Encoding.ASCII.GetBytes("WAVE").CopyTo(result, 8);
int offset = 12;
foreach (var chunk in chunks)
{
Encoding.ASCII.GetBytes(chunk.Id).CopyTo(result, offset);
BinaryPrimitives.WriteInt32LittleEndian(result.AsSpan(offset + 4, 4), chunk.Payload.Length);
chunk.Payload.CopyTo(result, offset + 8);
offset += 8 + chunk.Payload.Length + (chunk.Payload.Length & 1);
}
return result;
}
private static string[] ChunkIds(byte[] wave)
{
var ids = new List<string>();
int offset = 12;
while (offset < wave.Length)
{
ids.Add(Encoding.ASCII.GetString(wave, offset, 4));
int length = BinaryPrimitives.ReadInt32LittleEndian(wave.AsSpan(offset + 4, 4));
offset += 8 + length + (length & 1);
}
return ids.ToArray();
}
private static byte[] Chunk(byte[] wave, string wanted)
{
int offset = 12;
while (offset < wave.Length)
{
string id = Encoding.ASCII.GetString(wave, offset, 4);
int length = BinaryPrimitives.ReadInt32LittleEndian(wave.AsSpan(offset + 4, 4));
if (id == wanted) return wave.AsSpan(offset + 8, length).ToArray();
offset += 8 + length + (length & 1);
}
throw new InvalidDataException($"missing RIFF chunk {wanted}");
}
}

16
godot/Main.cs
View File

@@ -1208,7 +1208,10 @@ public partial class Main : Godot.Control
_sfxGenerations[channel]++;
_sfx[channel].Stop();
_sfx[channel].Stream = null;
var stream = AudioStreamWav.LoadFromBuffer(wavBytes);
// This is intentionally a Godot-only compatibility boundary. The VFS and engine retain the
// original WAV bytes; only Godot's UTF-8-assuming INFO parser sees the sanitized copy.
byte[] godotWav = RiffWaveSanitizer.RemoveInfoMetadata(wavBytes);
var stream = AudioStreamWav.LoadFromBuffer(godotWav);
if (stream == null) { GD.Print($"WAV load failed {assetName}"); return; }
stream.LoopMode = AudioStreamWav.LoopModeEnum.Disabled;
_sfx[channel].VolumeDb = 0;
@@ -1389,13 +1392,18 @@ public partial class Main : Godot.Control
new InputEventKey { PhysicalKeycode = Key.Up }, out int upVk) && upVk == 0x26
&& Win32VirtualKeyTranslator.TryTranslate(
new InputEventKey { PhysicalKeycode = Key.Ctrl }, out int ctrlVk) && ctrlVk == 0x11;
ok &= launcherOk && sleepMinimumOk && inputTranslationOk;
var selftestResources = ResourceMap.Load();
AudioPayload glowSfx = selftestResources.ReadAudio(selftestResources.ResolveSoundEffect(0x28)!);
byte[] glowGodotWav = RiffWaveSanitizer.RemoveInfoMetadata(glowSfx.Bytes);
bool cp932WavMetadataOk = glowGodotWav.Length == 688_336
&& AudioStreamWav.LoadFromBuffer(glowGodotWav) != null;
ok &= launcherOk && sleepMinimumOk && inputTranslationOk && cp932WavMetadataOk;
if (ok) GD.Print($"SELFTEST OK: threaded host matches headless ({actual.Count} lines, full handling); " +
$"debug launcher catalog/UI smoke ({debugEntries.Count} packed scripts); " +
$"sleep-min=1ms; native-key-translation=ok");
$"sleep-min=1ms; native-key-translation=ok; cp932-wav-info=ok");
else GD.Print($"SELFTEST FAIL: threaded={actual.Count} vs headless={expected.Count}; " +
$"debug-launcher={launcherOk}; sleep-min={sleepMinimumOk}; " +
$"native-key-translation={inputTranslationOk}");
$"native-key-translation={inputTranslationOk}; cp932-wav-info={cp932WavMetadataOk}");
GetTree().Quit(ok ? 0 : 1);
}

72
godot/RiffWaveSanitizer.cs Normal file
View File

@@ -0,0 +1,72 @@
using System;
using System.Buffers.Binary;
/// <summary>Godot-specific WAV input adapter. Godot assumes RIFF INFO strings are UTF-8, while AGE's
/// Japanese assets commonly store them as CP932. Playback does not consume these tags, so remove only
/// INFO metadata while preserving every functional RIFF chunk byte-for-byte.</summary>
internal static class RiffWaveSanitizer
{
public static byte[] RemoveInfoMetadata(byte[] wavBytes)
{
ReadOnlySpan<byte> input = wavBytes;
if (input.Length < 12 || !HasId(input, 0, "RIFF") || !HasId(input, 8, "WAVE"))
return wavBytes;
ulong declaredEnd64 = 8UL + BinaryPrimitives.ReadUInt32LittleEndian(input.Slice(4, 4));
if (declaredEnd64 < 12 || declaredEnd64 > (ulong)input.Length || declaredEnd64 > int.MaxValue)
return wavBytes;
int declaredEnd = (int)declaredEnd64;
int cursor = 12;
int removedBytes = 0;
while (cursor < declaredEnd)
{
if (!TryGetChunk(input, cursor, declaredEnd, out int chunkBytes, out bool isInfoList))
return wavBytes;
if (isInfoList) removedBytes = checked(removedBytes + chunkBytes);
cursor += chunkBytes;
}
if (removedBytes == 0) return wavBytes;
var output = new byte[checked(wavBytes.Length - removedBytes)];
input[..12].CopyTo(output);
cursor = 12;
int destination = 12;
while (cursor < declaredEnd)
{
_ = TryGetChunk(input, cursor, declaredEnd, out int chunkBytes, out bool isInfoList);
if (!isInfoList)
{
input.Slice(cursor, chunkBytes).CopyTo(output.AsSpan(destination));
destination += chunkBytes;
}
cursor += chunkBytes;
}
int newDeclaredEnd = declaredEnd - removedBytes;
input[declaredEnd..].CopyTo(output.AsSpan(newDeclaredEnd));
BinaryPrimitives.WriteUInt32LittleEndian(output.AsSpan(4, 4),
checked((uint)(newDeclaredEnd - 8)));
return output;
}
private static bool TryGetChunk(ReadOnlySpan<byte> input, int offset, int declaredEnd,
out int chunkBytes, out bool isInfoList)
{
chunkBytes = 0;
isInfoList = false;
if (offset > declaredEnd - 8) return false;
uint payloadBytes = BinaryPrimitives.ReadUInt32LittleEndian(input.Slice(offset + 4, 4));
ulong total64 = 8UL + payloadBytes + (payloadBytes & 1U);
if (total64 > int.MaxValue || total64 > (ulong)(declaredEnd - offset)) return false;
chunkBytes = (int)total64;
isInfoList = payloadBytes >= 4 && HasId(input, offset, "LIST")
&& HasId(input, offset + 8, "INFO");
return true;
}
private static bool HasId(ReadOnlySpan<byte> bytes, int offset, string id)
=> offset >= 0 && offset <= bytes.Length - 4
&& bytes[offset] == id[0] && bytes[offset + 1] == id[1]
&& bytes[offset + 2] == id[2] && bytes[offset + 3] == id[3];
}