Add synchronized MPEG movie audio

This commit is contained in:
gamer147
2026-07-25 11:49:50 -04:00
parent 1828701872
commit 843b282cc4
24 changed files with 1336 additions and 558 deletions

View File

@@ -351,8 +351,8 @@ until DirectShow EOF, then scene cleanup stops the movie. The initial invisible
the static `(assetId,colorKey)` image cache froze the first movie sample, while an extra current-sample
background copy was covered by the correctly positioned retained movie object. Dynamic movie surfaces now
bypass that cache and publish only at their retained z-position. A windowed run reached first frame 101 and
stop frame 190, and manual observation confirmed visible changing video. Movie audio remains intentionally
unrendered. The real-scene trace remains a separate extracted-present test
stop frame 190, and manual observation confirmed visible changing video. That milestone left movie audio
unrendered; the synchronized FFmpeg/Godot audio path landed later as described below. The real-scene trace remains a separate extracted-present test
because the current `Paths.Scripts()` test bootstrap still locates its root `*.BIN` fixtures there; migrating
that test/bootstrap path is unrelated to movie asset loading and was not folded into this slice.
@@ -410,7 +410,9 @@ reported combat stall survives this guard, capture the VM/service coordinate aft
a separate BTL timed-presentation bug.
The FFmpeg replacement and destination-surface work are recorded in `docs/platform-portability.md` and
`docs/phase-b-framework.md`. Movie identity also remains typed across the entire retained-surface lifecycle.
`docs/phase-b-framework.md`. After video/audio corpus validation and clean audible LOGO/OP/CHAPTER acceptance,
the DirectShow/temp-file adapter was deleted; packed MPEG decode now has one runtime path. Movie identity also
remains typed across the entire retained-surface lifecycle.
The packed catalog is immutable while mounted, so once an id enters the movie path its `.AGF`-named MPEG
payload cannot fall through to the ACGF still decoder before the first frame or after the last live binding
is detached. The latter guard closes a real cross-thread cleanup window: BTL clears `GfxState` and releases
@@ -430,10 +432,12 @@ depend on the movie service itself to park until EOF/input cancellation before t
`ResourceMap.ResolveMovie` supplies that typed universal lookup, while `ReadMovie` remains the MPEG
signature gate. `IHost.PlayModalMovieToSurface` is distinct from the non-modal call for lifecycle only: Godot
reuses the asynchronous DirectShow frame decoder and retained compositor but parks the VM thread until EOF
reuses the asynchronous FFmpeg decoder and retained compositor but parks the VM thread until EOF
or mouse/Accept/Cancel input. The wrapper's following release then tears down the completed/cancelled movie.
The decoder still intentionally leaves audio unrendered; OP/ED audio parity needs an explicit synchronized
movie-audio/backend contract rather than an unmanaged default-device side path.
The FFmpeg backend now renders embedded audio through an engine-owned synchronized path rather than an
unmanaged default-device side path. Timestamped stereo float PCM feeds a per-playback Godot
`AudioStreamGenerator`; the sound-hardware position drives video presentation, and modal cancellation or
surface cleanup releases both sides together.
VFS-B passes its bounded gates in `Sys4AssetStoreTests`: the installed AAI expands from the LZSS stream at
`0x118` (expanded size at `0x110`, packed size at `0x114`) to one `APPEND01.ALF` archive and 81 80-byte
@@ -446,8 +450,8 @@ through `0x01xxxxxx`; direct base-name lookup deliberately does not see append r
### Deliberate non-goals
- Writing/repacking ALF or AAI; loose overrides already provide the native mod/translation workflow.
- AGF encoding or movie audio. A decoder interface/software MPEG fallback is now required for the installed
non-16-aligned MVB effects; synchronized audio remains a separate slice.
- AGF/MPEG encoding. Runtime MPEG decode and synchronized movie audio now live behind the portable FFmpeg
boundary documented in `docs/platform-portability.md`; authoring or repacking those streams remains out of scope.
- A generalized multi-mod dependency manager. Start with native game-root loose overrides; configurable
ordered mod roots can be layered onto the same store later.
- Removing the extraction/conversion tools immediately. They remain independent parity oracles until the

View File

@@ -362,8 +362,9 @@ The port implements `0x20f` through a typed packed-movie resolver and a distinct
asynchronous decoder publishes frames through the retained surface while only the VM thread is parked;
EOF, mouse click, Accept, or Cancel resumes the wrapper so its scripted cleanup releases surface 42.
`0x236` retains its non-modal contract; native passes its packed resource operand through the same universal
catalog opener as `0x20f`. Both paths still intentionally leave the MPEG audio
pin unrendered; synchronized movie audio remains a deliberate backend/audio-clock slice.
catalog opener as `0x20f`. Both paths now decode the MPEG audio pin through FFmpeg into timestamped stereo
float PCM. Godot owns one `AudioStreamGenerator` per playback instance, applies the native flag-selected
mute/music/SE/voice or default movie route, and uses the sound-hardware position as the presentation clock.
An existing native operand trace identifies every observed heap codebase by a 100% match against its static
instruction-offset set. The captured New Game route is:
@@ -1279,8 +1280,8 @@ queries `IGraphBuilder`, `IMediaControl`, `IMediaPosition`, `IMediaEvent`, and `
`CMovieTextureRenderer` accepts RGB samples and copies the bottom-up frame into the retained texture.
Operand 3 is retained movie mode plus sound-route policy. Bits `0x10000/0x20000/0x40000/0x80000` force
sound route 0/1/2/3; without an override, native setting `set:DependMovieSound` supplies the route. This
slice deliberately does not implement the audio branch. Operand 4 is stored at movie object `+0x42c` as
mute/music/SE/voice routing; without an override, native setting `set:DependMovieSound` supplies the normal
movie route. The port applies those categories through Godot audio buses. Operand 4 is stored at movie object `+0x42c` as
the sync/device mask; it is not a duration or loop count. Replacing or releasing the owning surface stops
the graph and detaches the renderer.
@@ -1300,7 +1301,8 @@ because every sample has the same `(assetId,colorKey)`. An unconditional current
not help because normal retained composition immediately covered it with that cached first sample. Movie
surfaces now bypass the static cache and publish only through the retained object. A windowed run records
first frame 101 and stop frame 190; the user manually confirmed visible playback. The separate lower white
textbox-area object remains outside this finding, as does the deliberately unrendered movie audio stream.
textbox-area object remains outside this finding. Embedded audio was added later through the portable
FFmpeg/audio-clock path described in `docs/platform-portability.md`.
The `/v2` image names/comments the handler; movie ctor/interface/open/play/volume/release workers; sound
route helpers; renderer media-type/sample workers; and stop/detach/destructor lifecycle. The image was
@@ -1351,6 +1353,12 @@ reported stop time plus two seconds, maximum five minutes; 30 seconds without ti
hold the shared `0x21c` wait indefinitely. A normal Game Start through SC0000 was manually validated with
ordinary positive timing.
**Portable replacement closeout (2026-07-25):** the synchronous metadata, asynchronous presentation,
failure-as-completed, and watchdog contracts above remain, but `FfmpegMovieDecoder` is now the sole backend.
After the 213/213 video/audio corpus gate and clean audible LOGO/OP/CHAPTER acceptance, the port deleted
`DirectShowMovieDecoder`, its COM/temp-file adapter, compatibility test, and managed Windows annotations.
Native AGE's DirectShow behavior remains relevant evidence for opcode semantics; it is no longer port code.
### Grey-background root cause — slot collision + tint-strength (2026-07-08, gfx-log)
Diagnosed with the new `--gfx-log` compositor/op trace (docs/tools-reference.md). The grey background has

View File

@@ -726,7 +726,7 @@ This is a target-pixel operation, not retained-object teardown. It invokes IDire
- **grounding:** source=investigation, confidence=high
- **evidence:** Ghidra /v2: op_0x20f_play_modal_movie_to_surface@0x422e50 shares the movie-object allocation, DirectShow graph open, audio-route, and volume setup used by 0x236, then calls movie_start_modal_playback@0x463280, ORs EngineCtx+0xa0ce4 with 0x2000, and marks movie presentation dirty. The main loop and window procedure special-case run-state 0x2000. Corpus has exactly three sites: LOGO (0x335f,42,4), OP (0x3364,42,4), and ED (0x3324,42,dynamic flags). Those ids are universal raw SYS4INI indexes for MPEG-pack LOGO.AGF, OP.AGF, and ED.AGF; each script releases its surface only after 0x20f resumes.
Implemented through IHost.PlayModalMovieToSurface. Its operand uses the same native universal packed-id catalog contract as 0x236; the separate host call exists for modal wait/cancel lifecycle, not a different resolver. ResourceMap.ResolveMovie selects through ResolvePacked and retains MPEG signature validation in ReadMovie. Godot reuses the asynchronous decoder/retained-surface compositor, parks only the VM thread until EOF, and treats mouse click or Accept/Cancel input as completion before wrapper cleanup releases the decoder. MPEG audio remains a separate backend/audio-clock contract.
Implemented through IHost.PlayModalMovieToSurface. Its operand uses the same native universal packed-id catalog contract as 0x236; the separate host call exists for modal wait/cancel lifecycle, not a different resolver. ResourceMap.ResolveMovie selects through ResolvePacked and retains MPEG signature validation in ReadMovie. Godot reuses the asynchronous FFmpeg decoder/retained-surface compositor, parks only the VM thread until EOF, and treats mouse click or Accept/Cancel input as completion before wrapper cleanup releases both video and per-playback audio output. Audio-bearing movies deliver timestamped stereo float PCM through AudioStreamGenerator and use the sound-hardware position as the video/master clock; video-only movies retain monotonic-clock pacing.
### 0x212 `set-gfx-field64` (set-gfx-field64, argc 2)
- **summary:** 0x212 (obj_idx)(val) — handler gfx_op_0x212_set_field64 @0x4230c0: obj=[ctx+0x14d54 + obj_idx*4]; if obj: *(obj+0x64)=val. The generic instruction length is 5 dwords. See docs/engine-re.md gfx op-contract table.
@@ -856,7 +856,7 @@ Implemented through IHost.PlayModalMovieToSurface. Its operand uses the same nat
- **grounding:** source=investigation, confidence=high
- **evidence:** Ghidra /v2 op_0x236_play_movie_to_surface@0x423ee0 fetches operand 1 and passes it unchanged to movie_to_texture_open_asset_graph@0x463e20, which passes it unchanged to asset_open_indexed_entry@0x44f390. The opener directly indexes the flat base table or selected AAI table and has no scene input. SC0000 native operand capture and exact 0x13c8->0x13d1 trace prove nonblocking behavior. BTL's live 0x2b21 site supplies 0x2af1/0x2af5/0x2bca/0x2bd8/0x2bde, the exact base entries MVB001/MVB004/MVB914/MVB958/MVB955.
The handler requires an existing destination texture, allocates/reuses a 0x478-byte movie-to-texture object for the surface, opens operand 1 through the native universal packed-id reader, builds FilterGraph/IGraphBuilder/IMediaControl/IMediaPosition/IMediaEvent/IBasicAudio, and presents bottom-up RGB samples through the movie texture renderer. Operand 3 selects movie/sound routing policy: bits 0x10000/0x20000/0x40000/0x80000 force sound route 0/1/2/3, otherwise set:DependMovieSound is used; SC0000's low value 2 is retained as native movie mode state. Operand 4 is the movie sync/device mask. Static layer preparation after 0x236 does not terminate the retained movie; 0x21c services it through EOF and subsequent surface cleanup stops/detaches it. The port type-checks the selected ResolvePacked record as MPEG without adding a scene base or fallback.
The handler requires an existing destination texture, allocates/reuses a 0x478-byte movie-to-texture object for the surface, opens operand 1 through the native universal packed-id reader, builds FilterGraph/IGraphBuilder/IMediaControl/IMediaPosition/IMediaEvent/IBasicAudio, and presents bottom-up RGB samples through the movie texture renderer. Operand 3 selects movie/sound routing policy: bits 0x10000/0x20000/0x40000/0x80000 force mute/music/SE/voice routes, otherwise set:DependMovieSound selects the normal movie route; SC0000's low value 2 is retained as native movie mode state. Godot maps those categories to audio buses and sends FFmpeg-decoded timestamped stereo float PCM through a per-playback AudioStreamGenerator. Audio-bearing movies use the sound-hardware position as the master presentation clock; video-only movies retain monotonic pacing. Operand 4 is the movie sync/device mask. Static layer preparation after 0x236 does not terminate the retained movie; 0x21c services it through EOF and subsequent surface cleanup stops/detaches both decoder and audio output. The port type-checks the selected ResolvePacked record as MPEG without adding a scene base or fallback.
### 0x238 `set-anim-clock` (set-anim-clock, argc 1)
- **summary:** (duration) — configure the separate global finite-animation service window: native ctx+0x51b78=0 (start), +0x51b7c=duration. The generic instruction length is 3 dwords. NON-BLOCKING: gfx_animation_service_poll seeds the start from the shared frame timestamp and keeps redraw active through the duration. This is not the time source or cadence for cyclic object channels, which sample ctx+0x51b64/+0x51b68 with object-local starts and periods. SC0000 opening @0x123bd/@0x13858. Handler 0x4240e0; Kelebek VA 0x422390 is drift.

View File

@@ -127,8 +127,8 @@ explained the direct jump to TITLE. LOGO and OP then use the modal movie op
non-modal movie-to-surface path and uses the same packed lookup. The VM now models the initial-root flag and clears
it at the op-`0x9` whole-stack root-reload boundary. Godot resolves a typed MPEG asset, reuses the asynchronous decoder
and retained compositor, and parks the VM until EOF or mouse/Accept/Cancel input. Focused natural-boot tests
prove `LOGO -> OP -> INIT -> TITLE` ordering and exact movie operands. MPEG audio remains explicitly deferred
until the decoder abstraction has an engine-owned synchronized audio/volume contract.
prove `LOGO -> OP -> INIT -> TITLE` ordering and exact movie operands. MPEG audio was explicitly deferred
at this milestone; the engine-owned synchronized audio/volume contract landed on 2026-07-25 as recorded below.
## Stage B1 — Persistent session and scene coordinator
@@ -636,8 +636,8 @@ dependency pin, corpus codec inventory, timing contract, and packaging rules liv
as a separate battle-timeline defect.
6. **Complete distributable packaging.** Bundle pinned dynamically linked libraries and exact license/source
provenance per target, validate loader isolation from machine-installed codecs, then add Linux/macOS build and
smoke gates. Timestamped PCM delivery and Godot `AudioStreamGenerator` integration remain a later movie-audio
slice; the video replacement must merely avoid closing that path.
smoke gates. Timestamped PCM delivery and Godot `AudioStreamGenerator` integration were assigned to a later
movie-audio slice; that slice landed on 2026-07-25 as recorded below.
Steps 1 and 2 landed on 2026-07-21 without changing live playback: `Main` owns an
`IMovieDecoderFactory`, `MovieRuntime` is backend-neutral, and the default factory still constructs DirectShow.
@@ -726,6 +726,42 @@ tests, the zero-warning Godot build, and threaded selftest pass. Manual recheck
BUNKI panel and minimap paper are correct. The movement/overlay recheck described above is now the next manual
gate on the one-player-attack acceptance path.
### MPEG movie audio implemented; audible acceptance in progress (2026-07-25)
The FFmpeg shim ABI is now version 2. Each in-memory MPEG payload feeds independent seekable video and audio
demuxers, preserving concurrent decode without temporary files, and exposes timestamped interleaved stereo
float PCM resampled at the source rate. Video and audio timestamps share the same normalized media origin.
The managed decoder owns bounded video and PCM queues plus separate workers; audio-bearing movies use the
Godot audio-device clock as the presentation master, while video-only movies retain monotonic stopwatch pacing.
Completion waits for the final video interval and for decoded PCM to be submitted.
Godot creates a per-playback `AudioStreamGenerator`, compensates for output latency, handles timestamp gaps and
overlaps, and tears it down with the corresponding playback instance. Native movie flags route sound to the
Movie bus by default, with exact overrides for mute (`0x10000`), Music (`0x20000`), SFX (`0x40000`), and Voice
(`0x80000`). F6 diagnostics now expose audio format, decode/submission state, route, clock, and underruns.
The full installed-corpus gate passes all 213 movies: 29 audio-bearing assets decode 17,537 blocks /
18,185,856 stereo PCM frames at 44.1 kHz with nondecreasing timestamps and non-silent signal, while all 184
video-only assets remain audio-free. At the initial landing, focused real-stream tests covered MP2 and MP1
content, all 404 engine tests passed, the Godot build was warning-free, threaded selftest passed, and a natural
headless boot completed LOGO with its audio path active before opening OP. This moved acceptance to audible,
synchronized LOGO/OP and representative CHAPTER/MVS playback in a normal windowed run; DirectShow remained
unselected but in-tree until that live gate.
The first normal windowed LOGO/OP run confirmed that audio reaches the intended output, but it sounded
crackly/warbled. Corpus endpoints showed continuous decoded PCM; the fault was presentation alignment.
Audio PTS crosses ABI v2 in whole milliseconds, losing up to 44 samples of precision at 44.1 kHz, while the
initial sink inserted or dropped that tiny discrepancy at every MPEG block boundary. Established timelines
now tolerate 2 ms of timestamp quantization and remain sample-contiguous, while initial offsets and material
later gaps/overlaps still insert silence or trim PCM. A regression simulates all 4,093 OP blocks without a
splice and separately pins real discontinuity handling. The repeat LOGO/OP run and SC0000 CHAPTER playback are
audibly clean. Together with signal-bearing decode of all 29 audio streams, this closes the movie-audio acceptance
gate: missing, distorted, or unsynchronized audio is now a runtime bug. The unused DirectShow implementation,
COM/temp-file adapter, compatibility test, and managed Windows platform annotations are deleted. **NEXT:** add
Linux/macOS FFmpeg builds and packaging smoke gates when work returns to cross-platform distribution.
Post-cleanup validation passes all 405 engine tests, the warning-free Godot build, threaded selftest, opcode
lint/tooling, and the 213/213 installed video/audio corpus gate.
## Later Phase B breadth
**INIT data-semantics side track started (2026-07-22).** Before naming more gameplay state, the static

View File

@@ -18,19 +18,19 @@ The VM and content pipeline are already mostly platform-neutral:
- Native-compatible persistence uses managed streams behind `INativeDatStore`; Godot redirects AGE's
engine-owned root to `user://SAVE` while retaining its fixed portable DAT/STH filenames.
The selected movie path now uses the project-owned FFmpeg C ABI rather than a Windows multimedia API, but only a
Windows-x64 native bundle is built and staged today. The retired-live DirectShow implementation remains in-tree
until the corpus and manual gates pass. There are also softer Windows assumptions that should be tested or
replaced before claiming portable exports.
The sole movie path now uses the project-owned FFmpeg C ABI rather than a Windows multimedia API, but only a
Windows-x64 native bundle is built and staged today. The accepted DirectShow fallback was deleted after the
corpus and audible LOGO/OP/CHAPTER gates passed. There are also softer Windows assumptions that should be tested
or replaced before claiming portable exports.
## Dependency inventory
| Area | Current dependency | Runtime impact | Portability status / future action |
|---|---|---|---|
| Retained graphics presentation | Backend-neutral `GfxState`; accepted default Godot `Sprite2D` GPU stage plus the retained software pixel oracle, using runtime `ImageTexture`, canvas transforms/materials, and no native graphics API | GPU backend caches static/color-key variants, updates dynamic surfaces, handles retained range transitions, and falls back whole-frame for the legacy host screen-transition path | Godot owns D3D/Vulkan/Metal/OpenGL selection; validate shader/blend/filter behavior per target rather than adding a platform renderer |
| AGE movie decode (`0x236` scene movies; `0x20f` modal LOGO/OP/ED) | `FfmpegMovieDecoder` is the selected live factory over the project-owned `native/age_movie_ffmpeg` ABI; `DirectShowMovieDecoder` remains unselected pending deletion | Windows-x64 passes the complete 213-payload installed-corpus gate; other native targets and the remaining windowed CHAPTER/combat checks are pending | Run the windowed live gates, then delete DirectShow and add target-specific native builds |
| Movie integration | Each surface owns a unique playback-instance id; `MovieRuntime` owns `IMovieDecoder` from an injected factory, and the FFmpeg worker paces PTS against a monotonic clock with cancellation/failure completion | Concurrent/restarted uses of one asset have independent frame/completion/teardown state; `Main` remains annotated Windows because only the win-x64 bundle is available | Recheck the corrected combat lifecycle, then add Linux/macOS builds and remove the Windows annotation after DirectShow is deleted |
| Movie audio | FFmpeg detects the audio stream but the current ABI returns video frames only | MPEG movie audio remains intentionally silent | Extend the ABI with timestamped PCM and select an audio/presentation clock; separate feature slice |
| AGE movie decode (`0x236` scene movies; `0x20f` modal LOGO/OP/ED) | `FfmpegMovieDecoder` is the sole factory over the project-owned `native/age_movie_ffmpeg` ABI | Windows-x64 passes the complete 213-payload installed video/audio corpus gate plus audible LOGO/OP/CHAPTER playback | Add target-specific native builds and export packaging |
| Movie integration | Each surface owns a unique playback-instance id; `MovieRuntime` owns `IMovieDecoder` from an injected factory; video-only streams use monotonic pacing while audio-bearing streams use the Godot output clock | Concurrent/restarted uses of one asset have independent frame/audio/completion/teardown state; managed code is no longer Windows-annotated, while only the win-x64 native bundle exists today | Add Linux/macOS native builds and smoke gates |
| Movie audio | ABI v2 returns timestamped stereo float PCM; bounded managed buffering feeds a per-playback Godot `AudioStreamGenerator` and routes native movie flags to engine buses | All 29 installed audio-bearing streams decode with signal; synchronized LOGO/OP/CHAPTER playback is audibly accepted | Treat absent, distorted, or unsynchronized audio from an audio-bearing movie as a runtime bug |
| ADV font discovery | `godot/Main.cs` probes `C:/Windows/Fonts` for Japanese fonts | Harmless fallback today, but appearance depends on host fonts | Bundle/configure a redistributable font or add platform-specific discovery |
| Filesystem semantics | Several filename and containment comparisons use `OrdinalIgnoreCase`; installed assets are conventionally uppercase | Needs validation on case-sensitive filesystems; may hide casing or containment mistakes | Add Linux/macOS tests with mixed-case synthetic roots and use filesystem-appropriate containment rules |
| Save/profile storage | Managed `DirectoryNativeDatStore` under Godot `user://SAVE`; native S3SD/S4SD/S3RT files and 24-bit BMP thumbnails | No Win32 path API at runtime; port saves remain isolated from the original installation | Validate replace/flush, case, permissions, and interrupted-write behavior on each export target |
@@ -53,41 +53,43 @@ VM op 0x236 (non-modal) / op 0x20f (modal)
-> IHost.PlayMovieToSurface / PlayModalMovieToSurface
-> VFS-owned MoviePayload bytes
-> IMovieDecoderFactory
-> FfmpegMovieDecoder (current live selection)
-> FfmpegMovieDecoder
-> FfmpegMovieSession -> age_movie C ABI
-> DirectShowMovieDecoder (unselected; retained only through acceptance)
-> newest RGBA frame
-> retained movie surface
-> Godot compositor
-> newest due RGBA frame -> retained movie surface -> Godot compositor
-> timestamped stereo float PCM -> per-playback AudioStreamGenerator
```
Everything before and after the selected decoder is portable. The replacement decision is an in-process
FFmpeg backend behind a project-owned C ABI, not raw FFmpeg structs in Godot/C# and not a subprocess. FFmpeg
`n8.1.2-29-g703dcc25b9` is pinned by immutable release URL and SHA-256 in
`native/age_movie_ffmpeg/dependency-win64.json`; changing that pin requires rerunning the full installed-movie
gate. The shim dynamically links an LGPL build made without GPL or nonfree components and
uses only `libavformat`, `libavcodec`, `libavutil`, and `libswscale` for the video slice. `libswresample` and the
MPEG audio decoders may be packaged now, but PCM delivery remains a separate slice. Release artifacts must carry
gate. The shim dynamically links an LGPL build made without GPL or nonfree components and uses
`libavformat`, `libavcodec`, `libavutil`, `libswscale`, and `libswresample`. Release artifacts must carry
the matching FFmpeg source/configuration and notices required by FFmpeg's
[license checklist](https://ffmpeg.org/legal.html).
The boundary has two layers:
1. A small native `age_movie` ABI owns all FFmpeg objects and version-sensitive calls. It accepts a borrowed
byte span only for the duration of `open`, copies it into native-owned memory, creates a seekable custom
`AVIOContext`, probes the MPEG program stream, and returns immutable video metadata. Sequential decode returns
one top-down tightly packed RGBA8 frame plus its normalized presentation timestamp. Conversion occurs in an
byte span only for the duration of `open`, copies it once into native-owned memory, and creates independent
seekable custom `AVIOContext` views for video and audio so either stream can be pulled without duplicating the
payload or sharing a demux cursor. ABI v2 returns immutable video/audio metadata. Sequential video decode
returns one top-down tightly packed RGBA8 frame; sequential audio decode returns one interleaved stereo float32
PCM block. Both carry timestamps normalized to a common media origin. Video conversion occurs in an
FFmpeg-owned aligned frame; only exact visible row bytes are copied into the caller's tightly packed buffer.
`libswresample` converts the source channel/sample layout to the Godot generator contract.
Return statuses distinguish frame, EOF, invalid arguments, undersized output, and decoder failure; the open
call returns a bounded UTF-8 diagnostic and an opened handle retains its last decode error. Close accepts a
null handle, and managed `SafeHandle` ownership guarantees one close for each successfully opened handle. No
FFmpeg pointer crosses the ABI.
2. Managed `IMovieDecoder` owns the native handle and the paced worker. Synchronous construction provides
`StopTimeMs` before `0x236` returns. The worker uses a monotonic playback origin, decodes ahead by at most one
frame, publishes frames when their timestamps become due, and retains newest-frame-wins behavior if Godot is
late. EOF becomes completion only after the final frame's presentation interval/stop time, so surface cleanup
cannot erase the last frame immediately. Disposal interrupts waits and joins without depending on Godot's main
thread.
2. Managed `IMovieDecoder` owns the native handle and cancellable video/audio workers. Synchronous construction
provides `StopTimeMs` and audio format before `0x236` returns. Video remains at most one due frame ahead; PCM
uses a bounded roughly half-second queue, so the 263-second ending never becomes a whole-file allocation.
Video-only movies use the monotonic playback origin. Audio-bearing movies feed a per-instance
`AudioStreamGenerator`; the monotonic-clamped Godot sound-hardware position
(`playback + time-since-mix - cached output latency`) advances due video frames. Completion requires video
stop time, audio decode EOF, and PCM submission. Disposal interrupts waits, joins both workers, and stops the
generator without depending on native default-device playback.
`StopTimeMs` performs a bounded packet scan over the seekable in-memory payload and takes the longest usable
FFmpeg format duration, video-stream duration, timestamp span, or constant-frame-rate packet-count duration. This
@@ -99,12 +101,11 @@ safety rule still converts any backend initialization failure into an
explicitly completed zero-duration movie identity, and the presentation watchdog remains a last-resort guard for
an initialized backend that never reaches EOF.
The factory is injection for tests and future decoder replacement, not runtime codec roulette. FFmpeg is now the
selected Windows-x64 live backend; once its corpus and live gates pass, DirectShow is deleted rather than shipped
as a fallback. Modal
completion/cancel remains owned above the decoder by the existing `0x20f` host path. A future audio implementation
will consume timestamped PCM and may become the presentation clock; it must not change the VM-facing stop-time,
surface, or cancellation contracts.
The factory is injection for tests and future decoder replacement, not runtime codec roulette. FFmpeg is the
sole backend; DirectShow was deleted rather than shipped as a fallback after the live gate passed. Modal
completion/cancel remains owned above the decoder by the existing `0x20f` host path and tears down its generator
with the video instance. Native movie flag overrides select mute/music/SE/voice audio buses; the ordinary path
uses the movie bus.
This replacement is now also required for Windows gameplay parity. Archive-backed probes on 2026-07-21
show that the current DirectShow graph accepts tested MPEG widths divisible by 16 (208, 288, 304, 400, and
@@ -120,6 +121,18 @@ but only decodes Layer II audio and explicitly ignores program-stream PTS in its
FFmpeg covers the complete installed codec set and leaves the mod/profile boundary open without selecting a
different decoder per effect.
ABI v2 and MPEG audio playback landed on 2026-07-25. The expanded installed-corpus gate passes all 213 assets:
184 remain strictly video-only and all 29 audio-bearing streams decode to finite 44.1 kHz stereo PCM with
monotonic timestamps and clean EOF/teardown. The gate decoded 17,537 audio blocks / 18,185,856 stereo PCM
frames in the same eight-second unpaced run. A headless natural boot completed audio-clocked `LOGO.AGF` at
7,288 ms, released it without generator errors, and opened `OP.AGF`. A first windowed run confirmed audible
LOGO/OP output but exposed crackle/warble. The decoded PCM was continuous; the sink had treated each whole-
millisecond PTS rounding error (up to 44 samples at 44.1 kHz) as a real boundary and inserted or removed samples
from every MPEG block. The sink now ignores established-timeline discrepancies within 2 ms while still honoring
the first timestamp and material later gaps/overlaps. A 4,093-block OP regression pins splice-free continuity.
The clean audible recheck passed for LOGO/OP and CHAPTER. DirectShow and the managed Windows platform annotations
were then deleted; missing, distorted, or unsynchronized audio on any of the 29 audio-bearing assets is now a bug.
Native deliverables are RID-specific and bundled with the Godot export; the runtime must not discover an
arbitrary system FFmpeg. The first implementation gate is Windows x64 because that is the current runnable
target, but the C ABI and loader paths must reserve Windows x64, Linux x64, macOS x64, and macOS arm64 from the
@@ -137,7 +150,8 @@ nondecreasing timestamps. Malformed input and repeated teardown are covered. Thi
packaging. `FfmpegMovieDecoder` now adds cancellable timestamp pacing and is the live selection. The Godot build
copies the shim, five required shared libraries, and FFmpeg license beside `Himegari.dll`; a natural SYSTEM4
smoke completed 7288 ms `LOGO.AGF`, opened 106919 ms `OP.AGF`, and published frames from both without loading
DirectShow; the user subsequently confirmed both opening movies work in normal windowed playback. Full
DirectShow; the user subsequently confirmed both opening movies work in normal windowed playback. DirectShow
was deleted after the later synchronized-audio gate. Full
export/source-offer packaging and non-Windows builds remain outstanding.
The Windows-x64 installed-corpus gate passed on 2026-07-22 with the pinned dependency: all 213 signature-selected
@@ -145,7 +159,8 @@ MPEG payloads opened and reached EOF under a 30-second per-item bound. It decode
display sizes (120x120 through 800x600), independently matched sequence-header dimensions, found positive
duration and frame-rate metadata, verified tightly packed RGBA size and nondecreasing timestamps, observed
changing imagery in every asset, and closed every session. The set includes all 184 video-only and 29 audio-bearing
streams; audio presence is detected but PCM remains intentionally undelivered. The 263-second `ED.AGF` was the
streams; at that video-only milestone audio presence was detected but PCM remained intentionally undelivered.
The later ABI-v2 gate above supersedes that limitation. The 263-second `ED.AGF` was the
slowest decode at 4.4 seconds in the deliberately unpaced gate, so its prior 30-second failure was test-harness
whole-frame hashing overhead rather than a decoder incompatibility. The disposable machine report is
`build/movie-corpus-ffmpeg.json`; invocation and report semantics are canonical in `docs/tools-reference.md`.

View File

@@ -219,7 +219,7 @@ immutable LGPL shared FFmpeg archive and SHA-256.
|---|---|---|---|
| `native/age_movie_ffmpeg/bootstrap-win64.ps1` | Download, hash-check, extract, and version-check the pinned FFmpeg SDK. Returns the resolved SDK root. | `.\\native\\age_movie_ffmpeg\\bootstrap-win64.ps1 [-Destination <dir>]` | `dependency-win64.json`, network/archive cache → disposable `build/downloads/`, `build/ffmpeg-sdk/` |
| `native/age_movie_ffmpeg/build-win64.ps1` | Discover the MSVC x64 toolchain, build `age_movie_ffmpeg.dll`, and stage its exact shared-library/license dependencies. | `.\\native\\age_movie_ffmpeg\\build-win64.ps1 -SdkRoot <bootstrap-output> [-OutputDirectory <dir>]` | C ABI source + FFmpeg SDK → disposable `build/native/win-x64/` by default |
| `tools/movie-corpus-gate` | Discover every MPEG program stream stored under an `.AGF` catalog entry, decode every video frame through the unpaced FFmpeg session, validate independent sequence dimensions, metadata, RGBA size, timestamps, EOF, timeout, and teardown, then emit a per-asset JSON report. Exit 0 means the expected corpus count and every asset passed; exit 1 is a gate failure; exit 2 means the native shim is absent. | `dotnet run --project tools/movie-corpus-gate -- --output build/movie-corpus-ffmpeg.json --expected-count 213 --max-item-ms 30000` · optional `--native-dir <dir>` | `SYS4INI.BIN` + loose/ALF VFS assets + staged FFmpeg shim → stdout progress + disposable `build/movie-corpus-ffmpeg.json` |
| `tools/movie-corpus-gate` | Discover every MPEG program stream stored under an `.AGF` catalog entry; decode every video frame and, when present, every audio block through the unpaced FFmpeg session; validate independent sequence dimensions, metadata, RGBA size, stereo finite PCM, monotonic video/audio timestamps, EOF, timeout, and teardown; then emit a per-asset JSON report. Audio fields include sample rate/channels, block and PCM-frame counts, first/last PTS, signal presence, and decode time. Exit 0 means the expected corpus count and every asset passed; exit 1 is a gate failure; exit 2 means the native shim is absent. | `dotnet run --project tools/movie-corpus-gate -- --output build/movie-corpus-ffmpeg.json --expected-count 213 --max-item-ms 30000` · optional `--native-dir <dir>` | `SYS4INI.BIN` + loose/ALF VFS assets + staged FFmpeg shim → stdout progress + disposable `build/movie-corpus-ffmpeg.json` |
The managed isolated probes load from `AGE_FFMPEG_NATIVE_DIR` when set, then application-local and
`runtimes/win-x64/native` locations. Build the shim before running the `FfmpegShim*` tests; no original-game
@@ -227,8 +227,8 @@ movie is copied into the repository or native output. When `build/native/win-x64
`dotnet build godot/Himegari.csproj` also stages the shim, its five DLL dependencies, and `FFmpeg-LICENSE.txt`
beside `Himegari.dll` for development playback.
The corpus gate intentionally bypasses presentation waits: it validates decode compatibility and lifecycle,
not wall-clock playback pacing. `--expected-count` makes additions, omissions, or profile changes explicit;
The corpus gate intentionally bypasses presentation waits: it validates video/audio decode compatibility and
lifecycle, not wall-clock playback pacing or audible output. `--expected-count` makes additions, omissions, or profile changes explicit;
changing the pinned FFmpeg dependency requires rerunning this gate.
## Engine (C#) — VM core, CLI, Godot frontend