Sanitize CP932 WAV metadata for Godot
This commit is contained in:
@@ -169,6 +169,21 @@ The Phase-A backend now resolves the OGG/WAV catalog entry and opens it through
|
|||||||
the returned bytes into its existing BGM, voice, and fixed SC0000 SFX channel players. The earlier
|
the returned bytes into its existing BGM, voice, and fixed SC0000 SFX channel players. The earlier
|
||||||
`ResourceMap.AudioPath` extracted-file bootstrap is retired.
|
`ResourceMap.AudioPath` extracted-file bootstrap is retired.
|
||||||
|
|
||||||
|
Godot's WAV loader adds one host-specific compatibility boundary. SC0000 `0xc29` starts `E0808.WAV` with
|
||||||
|
the EV052DA glow. Its PCM is valid, but its trailing RIFF `LIST/INFO` fields contain Japanese CP932 text
|
||||||
|
(`IPRD`, `IGNR`, and `ICMT`). Godot assumes INFO text is UTF-8 and formerly emitted one Unicode warning per
|
||||||
|
invalid CP932 byte each time the sound was loaded; the duplicated SC0000 burst came from loading raw id
|
||||||
|
`0x28` on two channels. A complete extracted-corpus scan found 238 RIFF/WAVE files, 61 INFO chunks, no other
|
||||||
|
LIST type, and no invalid RIFF containers, explaining the same warnings around combat SFX.
|
||||||
|
|
||||||
|
The correction is deliberately confined to the Godot frontend. `RiffWaveSanitizer` removes only `LIST` chunks
|
||||||
|
whose form type is `INFO` from the transient byte array passed to `AudioStreamWav.LoadFromBuffer`, updates the
|
||||||
|
RIFF length, and preserves every functional chunk (`fmt `, `data`, `smpl`, `cue `, and unknown chunks)
|
||||||
|
byte-for-byte, including padding. The shared engine, original loose/archive payloads, and
|
||||||
|
`ResourceMap.ReadAudio` output remain untouched. The real E0808 Godot input shrinks from 688,570 to 688,336
|
||||||
|
bytes while retaining an identical PCM data chunk and loads headlessly without a Unicode warning. Synthetic
|
||||||
|
chunk/padding tests and the installed E0808 regression cover the adapter.
|
||||||
|
|
||||||
## Runtime asset-VFS track (VFS-A/B/C complete 2026-07-11)
|
## Runtime asset-VFS track (VFS-A/B/C complete 2026-07-11)
|
||||||
|
|
||||||
The pre-extracted tree and `build/textures/*.BMP` pipeline were a Phase-A bootstrap, not the desired final
|
The pre-extracted tree and `build/textures/*.BMP` pipeline were a Phase-A bootstrap, not the desired final
|
||||||
|
|||||||
@@ -685,6 +685,16 @@ that invalid fallback; it does not retain the frame or decoder. The focused clea
|
|||||||
tests, zero-warning Godot build, and threaded selftest pass. **NEXT:** perform a longer combat/return-to-FIELD
|
tests, zero-warning Godot build, and threaded selftest pass. **NEXT:** perform a longer combat/return-to-FIELD
|
||||||
acceptance run, then delete DirectShow if the remaining live gate stays clean.
|
acceptance run, then delete DirectShow if the remaining live gate stays clean.
|
||||||
|
|
||||||
|
An independent long-standing console-spam issue was also localized at the same SC0000 third-CG boundary.
|
||||||
|
`play-sound-effect 0x28` loads `E0808.WAV`, whose RIFF `LIST/INFO` metadata uses CP932; Godot assumes those
|
||||||
|
unused fields are UTF-8 and logged each invalid byte twice because the script loaded the effect on two channels.
|
||||||
|
This was neither the AE glow renderer nor corrupt PCM. The complete extracted corpus contains 61 affected INFO
|
||||||
|
chunks among 238 valid WAVs, which also explains intermittent combat spam. A Godot-only adapter now strips only
|
||||||
|
INFO metadata from the transient buffer immediately before `AudioStreamWav.LoadFromBuffer`; the shared engine,
|
||||||
|
VFS bytes, and functional RIFF chunks are unchanged. Synthetic preservation tests and the real E0808 regression
|
||||||
|
pass, as do all 331 engine tests, the zero-warning Godot build, and the headless loader selftest with no Unicode
|
||||||
|
warnings.
|
||||||
|
|
||||||
**Mutable-surface fill/blend regression corrected.** The first visual recheck exposed BUNKI's menu interior
|
**Mutable-surface fill/blend regression corrected.** The first visual recheck exposed BUNKI's menu interior
|
||||||
as transparent. SYSTEM4 creates 800x600 surface 3 and fills it opaque white through `0x20b`; the metadata-only
|
as transparent. SYSTEM4 creates 800x600 surface 3 and fills it opaque white through `0x20b`; the metadata-only
|
||||||
host fill left the new pixel buffer transparent. Implementing the fill alone made the panel solid gray and
|
host fill left the new pixel buffer transparent. Implementing the fill alone made the panel solid gray and
|
||||||
|
|||||||
@@ -30,6 +30,7 @@
|
|||||||
<Compile Include="..\..\godot\GodotTimelineLog.cs" Link="GodotTimelineLog.cs" />
|
<Compile Include="..\..\godot\GodotTimelineLog.cs" Link="GodotTimelineLog.cs" />
|
||||||
<Compile Include="..\..\godot\GodotTraceSink.cs" Link="GodotTraceSink.cs" />
|
<Compile Include="..\..\godot\GodotTraceSink.cs" Link="GodotTraceSink.cs" />
|
||||||
<Compile Include="..\..\godot\MovieSurfaceRegistry.cs" Link="MovieSurfaceRegistry.cs" />
|
<Compile Include="..\..\godot\MovieSurfaceRegistry.cs" Link="MovieSurfaceRegistry.cs" />
|
||||||
|
<Compile Include="..\..\godot\RiffWaveSanitizer.cs" Link="RiffWaveSanitizer.cs" />
|
||||||
<Compile Include="..\..\tools\movie-corpus-gate\MovieCorpusGate.cs" Link="MovieCorpusGate.cs" />
|
<Compile Include="..\..\tools\movie-corpus-gate\MovieCorpusGate.cs" Link="MovieCorpusGate.cs" />
|
||||||
<Compile Include="..\..\godot\DirectShowMovieDecoder.cs" Link="DirectShowMovieDecoder.cs" />
|
<Compile Include="..\..\godot\DirectShowMovieDecoder.cs" Link="DirectShowMovieDecoder.cs" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
|
|||||||
94
engine/Age.Engine.Tests/RiffWaveSanitizerTests.cs
Normal file
94
engine/Age.Engine.Tests/RiffWaveSanitizerTests.cs
Normal file
@@ -0,0 +1,94 @@
|
|||||||
|
using System.Buffers.Binary;
|
||||||
|
using System.Text;
|
||||||
|
using Age.Engine.Sys4;
|
||||||
|
|
||||||
|
public class RiffWaveSanitizerTests
|
||||||
|
{
|
||||||
|
[Fact]
|
||||||
|
public void RemovesOnlyInfoListAndPreservesFunctionalChunksAndPadding()
|
||||||
|
{
|
||||||
|
byte[] fmt = { 1, 0, 1, 0 };
|
||||||
|
byte[] info = { (byte)'I', (byte)'N', (byte)'F', (byte)'O',
|
||||||
|
(byte)'I', (byte)'P', (byte)'R', (byte)'D',
|
||||||
|
3, 0, 0, 0, 0x81, 0x45, 0 };
|
||||||
|
byte[] sampleLoop = { 9, 8, 7, 6 };
|
||||||
|
byte[] data = { 1, 2, 3 };
|
||||||
|
byte[] source = Wave(("fmt ", fmt), ("LIST", info), ("smpl", sampleLoop), ("data", data));
|
||||||
|
|
||||||
|
byte[] sanitized = RiffWaveSanitizer.RemoveInfoMetadata(source);
|
||||||
|
|
||||||
|
Assert.NotSame(source, sanitized);
|
||||||
|
Assert.Equal(sanitized.Length - 8, BinaryPrimitives.ReadInt32LittleEndian(sanitized.AsSpan(4, 4)));
|
||||||
|
Assert.Equal(new[] { "fmt ", "smpl", "data" }, ChunkIds(sanitized));
|
||||||
|
Assert.Equal(fmt, Chunk(sanitized, "fmt "));
|
||||||
|
Assert.Equal(sampleLoop, Chunk(sanitized, "smpl"));
|
||||||
|
Assert.Equal(data, Chunk(sanitized, "data"));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void LeavesWaveWithoutInfoMetadataUnchanged()
|
||||||
|
{
|
||||||
|
byte[] source = Wave(("fmt ", new byte[] { 1, 0 }), ("data", new byte[] { 1, 2 }));
|
||||||
|
|
||||||
|
Assert.Same(source, RiffWaveSanitizer.RemoveInfoMetadata(source));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void InstalledSc0000GlowSoundDropsCp932InfoBlockWithoutChangingPcmData()
|
||||||
|
{
|
||||||
|
var resources = ResourceMap.Load();
|
||||||
|
AudioPayload audio = resources.ReadAudio(resources.ResolveSoundEffect(0x28)!);
|
||||||
|
|
||||||
|
byte[] sanitized = RiffWaveSanitizer.RemoveInfoMetadata(audio.Bytes);
|
||||||
|
|
||||||
|
Assert.Equal("E0808.WAV", audio.Name);
|
||||||
|
Assert.Equal(688_570, audio.Bytes.Length);
|
||||||
|
Assert.Equal(688_336, sanitized.Length);
|
||||||
|
Assert.DoesNotContain("LIST", ChunkIds(sanitized));
|
||||||
|
Assert.Equal(Chunk(audio.Bytes, "data"), Chunk(sanitized, "data"));
|
||||||
|
}
|
||||||
|
|
||||||
|
private static byte[] Wave(params (string Id, byte[] Payload)[] chunks)
|
||||||
|
{
|
||||||
|
int length = 12 + chunks.Sum(chunk => 8 + chunk.Payload.Length + (chunk.Payload.Length & 1));
|
||||||
|
var result = new byte[length];
|
||||||
|
Encoding.ASCII.GetBytes("RIFF").CopyTo(result, 0);
|
||||||
|
BinaryPrimitives.WriteInt32LittleEndian(result.AsSpan(4, 4), length - 8);
|
||||||
|
Encoding.ASCII.GetBytes("WAVE").CopyTo(result, 8);
|
||||||
|
int offset = 12;
|
||||||
|
foreach (var chunk in chunks)
|
||||||
|
{
|
||||||
|
Encoding.ASCII.GetBytes(chunk.Id).CopyTo(result, offset);
|
||||||
|
BinaryPrimitives.WriteInt32LittleEndian(result.AsSpan(offset + 4, 4), chunk.Payload.Length);
|
||||||
|
chunk.Payload.CopyTo(result, offset + 8);
|
||||||
|
offset += 8 + chunk.Payload.Length + (chunk.Payload.Length & 1);
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string[] ChunkIds(byte[] wave)
|
||||||
|
{
|
||||||
|
var ids = new List<string>();
|
||||||
|
int offset = 12;
|
||||||
|
while (offset < wave.Length)
|
||||||
|
{
|
||||||
|
ids.Add(Encoding.ASCII.GetString(wave, offset, 4));
|
||||||
|
int length = BinaryPrimitives.ReadInt32LittleEndian(wave.AsSpan(offset + 4, 4));
|
||||||
|
offset += 8 + length + (length & 1);
|
||||||
|
}
|
||||||
|
return ids.ToArray();
|
||||||
|
}
|
||||||
|
|
||||||
|
private static byte[] Chunk(byte[] wave, string wanted)
|
||||||
|
{
|
||||||
|
int offset = 12;
|
||||||
|
while (offset < wave.Length)
|
||||||
|
{
|
||||||
|
string id = Encoding.ASCII.GetString(wave, offset, 4);
|
||||||
|
int length = BinaryPrimitives.ReadInt32LittleEndian(wave.AsSpan(offset + 4, 4));
|
||||||
|
if (id == wanted) return wave.AsSpan(offset + 8, length).ToArray();
|
||||||
|
offset += 8 + length + (length & 1);
|
||||||
|
}
|
||||||
|
throw new InvalidDataException($"missing RIFF chunk {wanted}");
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1208,7 +1208,10 @@ public partial class Main : Godot.Control
|
|||||||
_sfxGenerations[channel]++;
|
_sfxGenerations[channel]++;
|
||||||
_sfx[channel].Stop();
|
_sfx[channel].Stop();
|
||||||
_sfx[channel].Stream = null;
|
_sfx[channel].Stream = null;
|
||||||
var stream = AudioStreamWav.LoadFromBuffer(wavBytes);
|
// This is intentionally a Godot-only compatibility boundary. The VFS and engine retain the
|
||||||
|
// original WAV bytes; only Godot's UTF-8-assuming INFO parser sees the sanitized copy.
|
||||||
|
byte[] godotWav = RiffWaveSanitizer.RemoveInfoMetadata(wavBytes);
|
||||||
|
var stream = AudioStreamWav.LoadFromBuffer(godotWav);
|
||||||
if (stream == null) { GD.Print($"WAV load failed {assetName}"); return; }
|
if (stream == null) { GD.Print($"WAV load failed {assetName}"); return; }
|
||||||
stream.LoopMode = AudioStreamWav.LoopModeEnum.Disabled;
|
stream.LoopMode = AudioStreamWav.LoopModeEnum.Disabled;
|
||||||
_sfx[channel].VolumeDb = 0;
|
_sfx[channel].VolumeDb = 0;
|
||||||
@@ -1389,13 +1392,18 @@ public partial class Main : Godot.Control
|
|||||||
new InputEventKey { PhysicalKeycode = Key.Up }, out int upVk) && upVk == 0x26
|
new InputEventKey { PhysicalKeycode = Key.Up }, out int upVk) && upVk == 0x26
|
||||||
&& Win32VirtualKeyTranslator.TryTranslate(
|
&& Win32VirtualKeyTranslator.TryTranslate(
|
||||||
new InputEventKey { PhysicalKeycode = Key.Ctrl }, out int ctrlVk) && ctrlVk == 0x11;
|
new InputEventKey { PhysicalKeycode = Key.Ctrl }, out int ctrlVk) && ctrlVk == 0x11;
|
||||||
ok &= launcherOk && sleepMinimumOk && inputTranslationOk;
|
var selftestResources = ResourceMap.Load();
|
||||||
|
AudioPayload glowSfx = selftestResources.ReadAudio(selftestResources.ResolveSoundEffect(0x28)!);
|
||||||
|
byte[] glowGodotWav = RiffWaveSanitizer.RemoveInfoMetadata(glowSfx.Bytes);
|
||||||
|
bool cp932WavMetadataOk = glowGodotWav.Length == 688_336
|
||||||
|
&& AudioStreamWav.LoadFromBuffer(glowGodotWav) != null;
|
||||||
|
ok &= launcherOk && sleepMinimumOk && inputTranslationOk && cp932WavMetadataOk;
|
||||||
if (ok) GD.Print($"SELFTEST OK: threaded host matches headless ({actual.Count} lines, full handling); " +
|
if (ok) GD.Print($"SELFTEST OK: threaded host matches headless ({actual.Count} lines, full handling); " +
|
||||||
$"debug launcher catalog/UI smoke ({debugEntries.Count} packed scripts); " +
|
$"debug launcher catalog/UI smoke ({debugEntries.Count} packed scripts); " +
|
||||||
$"sleep-min=1ms; native-key-translation=ok");
|
$"sleep-min=1ms; native-key-translation=ok; cp932-wav-info=ok");
|
||||||
else GD.Print($"SELFTEST FAIL: threaded={actual.Count} vs headless={expected.Count}; " +
|
else GD.Print($"SELFTEST FAIL: threaded={actual.Count} vs headless={expected.Count}; " +
|
||||||
$"debug-launcher={launcherOk}; sleep-min={sleepMinimumOk}; " +
|
$"debug-launcher={launcherOk}; sleep-min={sleepMinimumOk}; " +
|
||||||
$"native-key-translation={inputTranslationOk}");
|
$"native-key-translation={inputTranslationOk}; cp932-wav-info={cp932WavMetadataOk}");
|
||||||
GetTree().Quit(ok ? 0 : 1);
|
GetTree().Quit(ok ? 0 : 1);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
72
godot/RiffWaveSanitizer.cs
Normal file
72
godot/RiffWaveSanitizer.cs
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
using System;
|
||||||
|
using System.Buffers.Binary;
|
||||||
|
|
||||||
|
/// <summary>Godot-specific WAV input adapter. Godot assumes RIFF INFO strings are UTF-8, while AGE's
|
||||||
|
/// Japanese assets commonly store them as CP932. Playback does not consume these tags, so remove only
|
||||||
|
/// INFO metadata while preserving every functional RIFF chunk byte-for-byte.</summary>
|
||||||
|
internal static class RiffWaveSanitizer
|
||||||
|
{
|
||||||
|
public static byte[] RemoveInfoMetadata(byte[] wavBytes)
|
||||||
|
{
|
||||||
|
ReadOnlySpan<byte> input = wavBytes;
|
||||||
|
if (input.Length < 12 || !HasId(input, 0, "RIFF") || !HasId(input, 8, "WAVE"))
|
||||||
|
return wavBytes;
|
||||||
|
|
||||||
|
ulong declaredEnd64 = 8UL + BinaryPrimitives.ReadUInt32LittleEndian(input.Slice(4, 4));
|
||||||
|
if (declaredEnd64 < 12 || declaredEnd64 > (ulong)input.Length || declaredEnd64 > int.MaxValue)
|
||||||
|
return wavBytes;
|
||||||
|
int declaredEnd = (int)declaredEnd64;
|
||||||
|
|
||||||
|
int cursor = 12;
|
||||||
|
int removedBytes = 0;
|
||||||
|
while (cursor < declaredEnd)
|
||||||
|
{
|
||||||
|
if (!TryGetChunk(input, cursor, declaredEnd, out int chunkBytes, out bool isInfoList))
|
||||||
|
return wavBytes;
|
||||||
|
if (isInfoList) removedBytes = checked(removedBytes + chunkBytes);
|
||||||
|
cursor += chunkBytes;
|
||||||
|
}
|
||||||
|
if (removedBytes == 0) return wavBytes;
|
||||||
|
|
||||||
|
var output = new byte[checked(wavBytes.Length - removedBytes)];
|
||||||
|
input[..12].CopyTo(output);
|
||||||
|
cursor = 12;
|
||||||
|
int destination = 12;
|
||||||
|
while (cursor < declaredEnd)
|
||||||
|
{
|
||||||
|
_ = TryGetChunk(input, cursor, declaredEnd, out int chunkBytes, out bool isInfoList);
|
||||||
|
if (!isInfoList)
|
||||||
|
{
|
||||||
|
input.Slice(cursor, chunkBytes).CopyTo(output.AsSpan(destination));
|
||||||
|
destination += chunkBytes;
|
||||||
|
}
|
||||||
|
cursor += chunkBytes;
|
||||||
|
}
|
||||||
|
|
||||||
|
int newDeclaredEnd = declaredEnd - removedBytes;
|
||||||
|
input[declaredEnd..].CopyTo(output.AsSpan(newDeclaredEnd));
|
||||||
|
BinaryPrimitives.WriteUInt32LittleEndian(output.AsSpan(4, 4),
|
||||||
|
checked((uint)(newDeclaredEnd - 8)));
|
||||||
|
return output;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool TryGetChunk(ReadOnlySpan<byte> input, int offset, int declaredEnd,
|
||||||
|
out int chunkBytes, out bool isInfoList)
|
||||||
|
{
|
||||||
|
chunkBytes = 0;
|
||||||
|
isInfoList = false;
|
||||||
|
if (offset > declaredEnd - 8) return false;
|
||||||
|
uint payloadBytes = BinaryPrimitives.ReadUInt32LittleEndian(input.Slice(offset + 4, 4));
|
||||||
|
ulong total64 = 8UL + payloadBytes + (payloadBytes & 1U);
|
||||||
|
if (total64 > int.MaxValue || total64 > (ulong)(declaredEnd - offset)) return false;
|
||||||
|
chunkBytes = (int)total64;
|
||||||
|
isInfoList = payloadBytes >= 4 && HasId(input, offset, "LIST")
|
||||||
|
&& HasId(input, offset + 8, "INFO");
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool HasId(ReadOnlySpan<byte> bytes, int offset, string id)
|
||||||
|
=> offset >= 0 && offset <= bytes.Length - 4
|
||||||
|
&& bytes[offset] == id[0] && bytes[offset + 1] == id[1]
|
||||||
|
&& bytes[offset + 2] == id[2] && bytes[offset + 3] == id[3];
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user