From 4b5b451df03110f79a20dc4f2c78a44a7e718615 Mon Sep 17 00:00:00 2001 From: gamer147 Date: Sat, 25 Jul 2026 11:49:50 -0400 Subject: [PATCH] Add synchronized MPEG movie audio --- docs/asset-resolution-re.md | 20 +- docs/engine-re.md | 18 +- docs/opcode-reference.md | 4 +- docs/phase-b-framework.md | 44 +- docs/platform-portability.md | 79 +-- docs/tools-reference.md | 6 +- .../Age.Engine.Tests/Age.Engine.Tests.csproj | 2 +- .../Age.Engine.Tests/MovieCorpusGateTests.cs | 33 ++ engine/Age.Engine.Tests/MovieOpcodeTests.cs | 161 ++++-- engine/Age.Engine/Hosting/IHost.cs | 2 +- godot/DirectShowMovieDecoder.cs | 304 ----------- godot/FfmpegMovieDecoder.cs | 196 ++++++- godot/FfmpegMovieNative.cs | 98 +++- godot/GodotAdvHost.cs | 7 +- godot/IMovieDecoder.cs | 19 +- godot/Main.cs | 85 ++- godot/MovieAudioOutput.cs | 182 +++++++ godot/MovieAudioTimeline.cs | 50 ++ godot/MovieRuntime.cs | 6 +- native/age_movie_ffmpeg/age_movie.c | 489 ++++++++++++++---- native/age_movie_ffmpeg/age_movie.h | 14 + native/age_movie_ffmpeg/build-win64.ps1 | 2 +- tools/movie-corpus-gate/MovieCorpusGate.cs | 69 ++- vm-map/opcodes.toml | 4 +- 24 files changed, 1336 insertions(+), 558 deletions(-) delete mode 100644 godot/DirectShowMovieDecoder.cs create mode 100644 godot/MovieAudioOutput.cs create mode 100644 godot/MovieAudioTimeline.cs diff --git a/docs/asset-resolution-re.md b/docs/asset-resolution-re.md index ba1a3fd..aeed339 100644 --- a/docs/asset-resolution-re.md +++ b/docs/asset-resolution-re.md @@ -351,8 +351,8 @@ until DirectShow EOF, then scene cleanup stops the movie. The initial invisible the static `(assetId,colorKey)` image cache froze the first movie sample, while an extra current-sample background copy was covered by the correctly positioned retained movie object. Dynamic movie surfaces now bypass that cache and publish only at their retained z-position. A windowed run reached first frame 101 and -stop frame 190, and manual observation confirmed visible changing video. Movie audio remains intentionally -unrendered. The real-scene trace remains a separate extracted-present test +stop frame 190, and manual observation confirmed visible changing video. That milestone left movie audio +unrendered; the synchronized FFmpeg/Godot audio path landed later as described below. The real-scene trace remains a separate extracted-present test because the current `Paths.Scripts()` test bootstrap still locates its root `*.BIN` fixtures there; migrating that test/bootstrap path is unrelated to movie asset loading and was not folded into this slice. @@ -410,7 +410,9 @@ reported combat stall survives this guard, capture the VM/service coordinate aft a separate BTL timed-presentation bug. The FFmpeg replacement and destination-surface work are recorded in `docs/platform-portability.md` and -`docs/phase-b-framework.md`. Movie identity also remains typed across the entire retained-surface lifecycle. +`docs/phase-b-framework.md`. After video/audio corpus validation and clean audible LOGO/OP/CHAPTER acceptance, +the DirectShow/temp-file adapter was deleted; packed MPEG decode now has one runtime path. Movie identity also +remains typed across the entire retained-surface lifecycle. The packed catalog is immutable while mounted, so once an id enters the movie path its `.AGF`-named MPEG payload cannot fall through to the ACGF still decoder before the first frame or after the last live binding is detached. The latter guard closes a real cross-thread cleanup window: BTL clears `GfxState` and releases @@ -430,10 +432,12 @@ depend on the movie service itself to park until EOF/input cancellation before t `ResourceMap.ResolveMovie` supplies that typed universal lookup, while `ReadMovie` remains the MPEG signature gate. `IHost.PlayModalMovieToSurface` is distinct from the non-modal call for lifecycle only: Godot -reuses the asynchronous DirectShow frame decoder and retained compositor but parks the VM thread until EOF +reuses the asynchronous FFmpeg decoder and retained compositor but parks the VM thread until EOF or mouse/Accept/Cancel input. The wrapper's following release then tears down the completed/cancelled movie. -The decoder still intentionally leaves audio unrendered; OP/ED audio parity needs an explicit synchronized -movie-audio/backend contract rather than an unmanaged default-device side path. +The FFmpeg backend now renders embedded audio through an engine-owned synchronized path rather than an +unmanaged default-device side path. Timestamped stereo float PCM feeds a per-playback Godot +`AudioStreamGenerator`; the sound-hardware position drives video presentation, and modal cancellation or +surface cleanup releases both sides together. VFS-B passes its bounded gates in `Sys4AssetStoreTests`: the installed AAI expands from the LZSS stream at `0x118` (expanded size at `0x110`, packed size at `0x114`) to one `APPEND01.ALF` archive and 81 80-byte @@ -446,8 +450,8 @@ through `0x01xxxxxx`; direct base-name lookup deliberately does not see append r ### Deliberate non-goals - Writing/repacking ALF or AAI; loose overrides already provide the native mod/translation workflow. -- AGF encoding or movie audio. A decoder interface/software MPEG fallback is now required for the installed - non-16-aligned MVB effects; synchronized audio remains a separate slice. +- AGF/MPEG encoding. Runtime MPEG decode and synchronized movie audio now live behind the portable FFmpeg + boundary documented in `docs/platform-portability.md`; authoring or repacking those streams remains out of scope. - A generalized multi-mod dependency manager. Start with native game-root loose overrides; configurable ordered mod roots can be layered onto the same store later. - Removing the extraction/conversion tools immediately. They remain independent parity oracles until the diff --git a/docs/engine-re.md b/docs/engine-re.md index 33a1936..ac2ef45 100644 --- a/docs/engine-re.md +++ b/docs/engine-re.md @@ -362,8 +362,9 @@ The port implements `0x20f` through a typed packed-movie resolver and a distinct asynchronous decoder publishes frames through the retained surface while only the VM thread is parked; EOF, mouse click, Accept, or Cancel resumes the wrapper so its scripted cleanup releases surface 42. `0x236` retains its non-modal contract; native passes its packed resource operand through the same universal -catalog opener as `0x20f`. Both paths still intentionally leave the MPEG audio -pin unrendered; synchronized movie audio remains a deliberate backend/audio-clock slice. +catalog opener as `0x20f`. Both paths now decode the MPEG audio pin through FFmpeg into timestamped stereo +float PCM. Godot owns one `AudioStreamGenerator` per playback instance, applies the native flag-selected +mute/music/SE/voice or default movie route, and uses the sound-hardware position as the presentation clock. An existing native operand trace identifies every observed heap codebase by a 100% match against its static instruction-offset set. The captured New Game route is: @@ -1279,8 +1280,8 @@ queries `IGraphBuilder`, `IMediaControl`, `IMediaPosition`, `IMediaEvent`, and ` `CMovieTextureRenderer` accepts RGB samples and copies the bottom-up frame into the retained texture. Operand 3 is retained movie mode plus sound-route policy. Bits `0x10000/0x20000/0x40000/0x80000` force -sound route 0/1/2/3; without an override, native setting `set:DependMovieSound` supplies the route. This -slice deliberately does not implement the audio branch. Operand 4 is stored at movie object `+0x42c` as +mute/music/SE/voice routing; without an override, native setting `set:DependMovieSound` supplies the normal +movie route. The port applies those categories through Godot audio buses. Operand 4 is stored at movie object `+0x42c` as the sync/device mask; it is not a duration or loop count. Replacing or releasing the owning surface stops the graph and detaches the renderer. @@ -1300,7 +1301,8 @@ because every sample has the same `(assetId,colorKey)`. An unconditional current not help because normal retained composition immediately covered it with that cached first sample. Movie surfaces now bypass the static cache and publish only through the retained object. A windowed run records first frame 101 and stop frame 190; the user manually confirmed visible playback. The separate lower white -textbox-area object remains outside this finding, as does the deliberately unrendered movie audio stream. +textbox-area object remains outside this finding. Embedded audio was added later through the portable +FFmpeg/audio-clock path described in `docs/platform-portability.md`. The `/v2` image names/comments the handler; movie ctor/interface/open/play/volume/release workers; sound route helpers; renderer media-type/sample workers; and stop/detach/destructor lifecycle. The image was @@ -1351,6 +1353,12 @@ reported stop time plus two seconds, maximum five minutes; 30 seconds without ti hold the shared `0x21c` wait indefinitely. A normal Game Start through SC0000 was manually validated with ordinary positive timing. +**Portable replacement closeout (2026-07-25):** the synchronous metadata, asynchronous presentation, +failure-as-completed, and watchdog contracts above remain, but `FfmpegMovieDecoder` is now the sole backend. +After the 213/213 video/audio corpus gate and clean audible LOGO/OP/CHAPTER acceptance, the port deleted +`DirectShowMovieDecoder`, its COM/temp-file adapter, compatibility test, and managed Windows annotations. +Native AGE's DirectShow behavior remains relevant evidence for opcode semantics; it is no longer port code. + ### Grey-background root cause — slot collision + tint-strength (2026-07-08, gfx-log) Diagnosed with the new `--gfx-log` compositor/op trace (docs/tools-reference.md). The grey background has diff --git a/docs/opcode-reference.md b/docs/opcode-reference.md index a3d193c..54462d8 100644 --- a/docs/opcode-reference.md +++ b/docs/opcode-reference.md @@ -726,7 +726,7 @@ This is a target-pixel operation, not retained-object teardown. It invokes IDire - **grounding:** source=investigation, confidence=high - **evidence:** Ghidra /v2: op_0x20f_play_modal_movie_to_surface@0x422e50 shares the movie-object allocation, DirectShow graph open, audio-route, and volume setup used by 0x236, then calls movie_start_modal_playback@0x463280, ORs EngineCtx+0xa0ce4 with 0x2000, and marks movie presentation dirty. The main loop and window procedure special-case run-state 0x2000. Corpus has exactly three sites: LOGO (0x335f,42,4), OP (0x3364,42,4), and ED (0x3324,42,dynamic flags). Those ids are universal raw SYS4INI indexes for MPEG-pack LOGO.AGF, OP.AGF, and ED.AGF; each script releases its surface only after 0x20f resumes. -Implemented through IHost.PlayModalMovieToSurface. Its operand uses the same native universal packed-id catalog contract as 0x236; the separate host call exists for modal wait/cancel lifecycle, not a different resolver. ResourceMap.ResolveMovie selects through ResolvePacked and retains MPEG signature validation in ReadMovie. Godot reuses the asynchronous decoder/retained-surface compositor, parks only the VM thread until EOF, and treats mouse click or Accept/Cancel input as completion before wrapper cleanup releases the decoder. MPEG audio remains a separate backend/audio-clock contract. +Implemented through IHost.PlayModalMovieToSurface. Its operand uses the same native universal packed-id catalog contract as 0x236; the separate host call exists for modal wait/cancel lifecycle, not a different resolver. ResourceMap.ResolveMovie selects through ResolvePacked and retains MPEG signature validation in ReadMovie. Godot reuses the asynchronous FFmpeg decoder/retained-surface compositor, parks only the VM thread until EOF, and treats mouse click or Accept/Cancel input as completion before wrapper cleanup releases both video and per-playback audio output. Audio-bearing movies deliver timestamped stereo float PCM through AudioStreamGenerator and use the sound-hardware position as the video/master clock; video-only movies retain monotonic-clock pacing. ### 0x212 `set-gfx-field64` (set-gfx-field64, argc 2) - **summary:** 0x212 (obj_idx)(val) — handler gfx_op_0x212_set_field64 @0x4230c0: obj=[ctx+0x14d54 + obj_idx*4]; if obj: *(obj+0x64)=val. The generic instruction length is 5 dwords. See docs/engine-re.md gfx op-contract table. @@ -856,7 +856,7 @@ Implemented through IHost.PlayModalMovieToSurface. Its operand uses the same nat - **grounding:** source=investigation, confidence=high - **evidence:** Ghidra /v2 op_0x236_play_movie_to_surface@0x423ee0 fetches operand 1 and passes it unchanged to movie_to_texture_open_asset_graph@0x463e20, which passes it unchanged to asset_open_indexed_entry@0x44f390. The opener directly indexes the flat base table or selected AAI table and has no scene input. SC0000 native operand capture and exact 0x13c8->0x13d1 trace prove nonblocking behavior. BTL's live 0x2b21 site supplies 0x2af1/0x2af5/0x2bca/0x2bd8/0x2bde, the exact base entries MVB001/MVB004/MVB914/MVB958/MVB955. -The handler requires an existing destination texture, allocates/reuses a 0x478-byte movie-to-texture object for the surface, opens operand 1 through the native universal packed-id reader, builds FilterGraph/IGraphBuilder/IMediaControl/IMediaPosition/IMediaEvent/IBasicAudio, and presents bottom-up RGB samples through the movie texture renderer. Operand 3 selects movie/sound routing policy: bits 0x10000/0x20000/0x40000/0x80000 force sound route 0/1/2/3, otherwise set:DependMovieSound is used; SC0000's low value 2 is retained as native movie mode state. Operand 4 is the movie sync/device mask. Static layer preparation after 0x236 does not terminate the retained movie; 0x21c services it through EOF and subsequent surface cleanup stops/detaches it. The port type-checks the selected ResolvePacked record as MPEG without adding a scene base or fallback. +The handler requires an existing destination texture, allocates/reuses a 0x478-byte movie-to-texture object for the surface, opens operand 1 through the native universal packed-id reader, builds FilterGraph/IGraphBuilder/IMediaControl/IMediaPosition/IMediaEvent/IBasicAudio, and presents bottom-up RGB samples through the movie texture renderer. Operand 3 selects movie/sound routing policy: bits 0x10000/0x20000/0x40000/0x80000 force mute/music/SE/voice routes, otherwise set:DependMovieSound selects the normal movie route; SC0000's low value 2 is retained as native movie mode state. Godot maps those categories to audio buses and sends FFmpeg-decoded timestamped stereo float PCM through a per-playback AudioStreamGenerator. Audio-bearing movies use the sound-hardware position as the master presentation clock; video-only movies retain monotonic pacing. Operand 4 is the movie sync/device mask. Static layer preparation after 0x236 does not terminate the retained movie; 0x21c services it through EOF and subsequent surface cleanup stops/detaches both decoder and audio output. The port type-checks the selected ResolvePacked record as MPEG without adding a scene base or fallback. ### 0x238 `set-anim-clock` (set-anim-clock, argc 1) - **summary:** (duration) — configure the separate global finite-animation service window: native ctx+0x51b78=0 (start), +0x51b7c=duration. The generic instruction length is 3 dwords. NON-BLOCKING: gfx_animation_service_poll seeds the start from the shared frame timestamp and keeps redraw active through the duration. This is not the time source or cadence for cyclic object channels, which sample ctx+0x51b64/+0x51b68 with object-local starts and periods. SC0000 opening @0x123bd/@0x13858. Handler 0x4240e0; Kelebek VA 0x422390 is drift. diff --git a/docs/phase-b-framework.md b/docs/phase-b-framework.md index 3346049..cd9ee66 100644 --- a/docs/phase-b-framework.md +++ b/docs/phase-b-framework.md @@ -127,8 +127,8 @@ explained the direct jump to TITLE. LOGO and OP then use the modal movie op non-modal movie-to-surface path and uses the same packed lookup. The VM now models the initial-root flag and clears it at the op-`0x9` whole-stack root-reload boundary. Godot resolves a typed MPEG asset, reuses the asynchronous decoder and retained compositor, and parks the VM until EOF or mouse/Accept/Cancel input. Focused natural-boot tests -prove `LOGO -> OP -> INIT -> TITLE` ordering and exact movie operands. MPEG audio remains explicitly deferred -until the decoder abstraction has an engine-owned synchronized audio/volume contract. +prove `LOGO -> OP -> INIT -> TITLE` ordering and exact movie operands. MPEG audio was explicitly deferred +at this milestone; the engine-owned synchronized audio/volume contract landed on 2026-07-25 as recorded below. ## Stage B1 — Persistent session and scene coordinator @@ -636,8 +636,8 @@ dependency pin, corpus codec inventory, timing contract, and packaging rules liv as a separate battle-timeline defect. 6. **Complete distributable packaging.** Bundle pinned dynamically linked libraries and exact license/source provenance per target, validate loader isolation from machine-installed codecs, then add Linux/macOS build and - smoke gates. Timestamped PCM delivery and Godot `AudioStreamGenerator` integration remain a later movie-audio - slice; the video replacement must merely avoid closing that path. + smoke gates. Timestamped PCM delivery and Godot `AudioStreamGenerator` integration were assigned to a later + movie-audio slice; that slice landed on 2026-07-25 as recorded below. Steps 1 and 2 landed on 2026-07-21 without changing live playback: `Main` owns an `IMovieDecoderFactory`, `MovieRuntime` is backend-neutral, and the default factory still constructs DirectShow. @@ -726,6 +726,42 @@ tests, the zero-warning Godot build, and threaded selftest pass. Manual recheck BUNKI panel and minimap paper are correct. The movement/overlay recheck described above is now the next manual gate on the one-player-attack acceptance path. +### MPEG movie audio implemented; audible acceptance in progress (2026-07-25) + +The FFmpeg shim ABI is now version 2. Each in-memory MPEG payload feeds independent seekable video and audio +demuxers, preserving concurrent decode without temporary files, and exposes timestamped interleaved stereo +float PCM resampled at the source rate. Video and audio timestamps share the same normalized media origin. +The managed decoder owns bounded video and PCM queues plus separate workers; audio-bearing movies use the +Godot audio-device clock as the presentation master, while video-only movies retain monotonic stopwatch pacing. +Completion waits for the final video interval and for decoded PCM to be submitted. + +Godot creates a per-playback `AudioStreamGenerator`, compensates for output latency, handles timestamp gaps and +overlaps, and tears it down with the corresponding playback instance. Native movie flags route sound to the +Movie bus by default, with exact overrides for mute (`0x10000`), Music (`0x20000`), SFX (`0x40000`), and Voice +(`0x80000`). F6 diagnostics now expose audio format, decode/submission state, route, clock, and underruns. + +The full installed-corpus gate passes all 213 movies: 29 audio-bearing assets decode 17,537 blocks / +18,185,856 stereo PCM frames at 44.1 kHz with nondecreasing timestamps and non-silent signal, while all 184 +video-only assets remain audio-free. At the initial landing, focused real-stream tests covered MP2 and MP1 +content, all 404 engine tests passed, the Godot build was warning-free, threaded selftest passed, and a natural +headless boot completed LOGO with its audio path active before opening OP. This moved acceptance to audible, +synchronized LOGO/OP and representative CHAPTER/MVS playback in a normal windowed run; DirectShow remained +unselected but in-tree until that live gate. + +The first normal windowed LOGO/OP run confirmed that audio reaches the intended output, but it sounded +crackly/warbled. Corpus endpoints showed continuous decoded PCM; the fault was presentation alignment. +Audio PTS crosses ABI v2 in whole milliseconds, losing up to 44 samples of precision at 44.1 kHz, while the +initial sink inserted or dropped that tiny discrepancy at every MPEG block boundary. Established timelines +now tolerate 2 ms of timestamp quantization and remain sample-contiguous, while initial offsets and material +later gaps/overlaps still insert silence or trim PCM. A regression simulates all 4,093 OP blocks without a +splice and separately pins real discontinuity handling. The repeat LOGO/OP run and SC0000 CHAPTER playback are +audibly clean. Together with signal-bearing decode of all 29 audio streams, this closes the movie-audio acceptance +gate: missing, distorted, or unsynchronized audio is now a runtime bug. The unused DirectShow implementation, +COM/temp-file adapter, compatibility test, and managed Windows platform annotations are deleted. **NEXT:** add +Linux/macOS FFmpeg builds and packaging smoke gates when work returns to cross-platform distribution. +Post-cleanup validation passes all 405 engine tests, the warning-free Godot build, threaded selftest, opcode +lint/tooling, and the 213/213 installed video/audio corpus gate. + ## Later Phase B breadth **INIT data-semantics side track started (2026-07-22).** Before naming more gameplay state, the static diff --git a/docs/platform-portability.md b/docs/platform-portability.md index 36ee712..aeedd64 100644 --- a/docs/platform-portability.md +++ b/docs/platform-portability.md @@ -18,19 +18,19 @@ The VM and content pipeline are already mostly platform-neutral: - Native-compatible persistence uses managed streams behind `INativeDatStore`; Godot redirects AGE's engine-owned root to `user://SAVE` while retaining its fixed portable DAT/STH filenames. -The selected movie path now uses the project-owned FFmpeg C ABI rather than a Windows multimedia API, but only a -Windows-x64 native bundle is built and staged today. The retired-live DirectShow implementation remains in-tree -until the corpus and manual gates pass. There are also softer Windows assumptions that should be tested or -replaced before claiming portable exports. +The sole movie path now uses the project-owned FFmpeg C ABI rather than a Windows multimedia API, but only a +Windows-x64 native bundle is built and staged today. The accepted DirectShow fallback was deleted after the +corpus and audible LOGO/OP/CHAPTER gates passed. There are also softer Windows assumptions that should be tested +or replaced before claiming portable exports. ## Dependency inventory | Area | Current dependency | Runtime impact | Portability status / future action | |---|---|---|---| | Retained graphics presentation | Backend-neutral `GfxState`; accepted default Godot `Sprite2D` GPU stage plus the retained software pixel oracle, using runtime `ImageTexture`, canvas transforms/materials, and no native graphics API | GPU backend caches static/color-key variants, updates dynamic surfaces, handles retained range transitions, and falls back whole-frame for the legacy host screen-transition path | Godot owns D3D/Vulkan/Metal/OpenGL selection; validate shader/blend/filter behavior per target rather than adding a platform renderer | -| AGE movie decode (`0x236` scene movies; `0x20f` modal LOGO/OP/ED) | `FfmpegMovieDecoder` is the selected live factory over the project-owned `native/age_movie_ffmpeg` ABI; `DirectShowMovieDecoder` remains unselected pending deletion | Windows-x64 passes the complete 213-payload installed-corpus gate; other native targets and the remaining windowed CHAPTER/combat checks are pending | Run the windowed live gates, then delete DirectShow and add target-specific native builds | -| Movie integration | Each surface owns a unique playback-instance id; `MovieRuntime` owns `IMovieDecoder` from an injected factory, and the FFmpeg worker paces PTS against a monotonic clock with cancellation/failure completion | Concurrent/restarted uses of one asset have independent frame/completion/teardown state; `Main` remains annotated Windows because only the win-x64 bundle is available | Recheck the corrected combat lifecycle, then add Linux/macOS builds and remove the Windows annotation after DirectShow is deleted | -| Movie audio | FFmpeg detects the audio stream but the current ABI returns video frames only | MPEG movie audio remains intentionally silent | Extend the ABI with timestamped PCM and select an audio/presentation clock; separate feature slice | +| AGE movie decode (`0x236` scene movies; `0x20f` modal LOGO/OP/ED) | `FfmpegMovieDecoder` is the sole factory over the project-owned `native/age_movie_ffmpeg` ABI | Windows-x64 passes the complete 213-payload installed video/audio corpus gate plus audible LOGO/OP/CHAPTER playback | Add target-specific native builds and export packaging | +| Movie integration | Each surface owns a unique playback-instance id; `MovieRuntime` owns `IMovieDecoder` from an injected factory; video-only streams use monotonic pacing while audio-bearing streams use the Godot output clock | Concurrent/restarted uses of one asset have independent frame/audio/completion/teardown state; managed code is no longer Windows-annotated, while only the win-x64 native bundle exists today | Add Linux/macOS native builds and smoke gates | +| Movie audio | ABI v2 returns timestamped stereo float PCM; bounded managed buffering feeds a per-playback Godot `AudioStreamGenerator` and routes native movie flags to engine buses | All 29 installed audio-bearing streams decode with signal; synchronized LOGO/OP/CHAPTER playback is audibly accepted | Treat absent, distorted, or unsynchronized audio from an audio-bearing movie as a runtime bug | | ADV font discovery | `godot/Main.cs` probes `C:/Windows/Fonts` for Japanese fonts | Harmless fallback today, but appearance depends on host fonts | Bundle/configure a redistributable font or add platform-specific discovery | | Filesystem semantics | Several filename and containment comparisons use `OrdinalIgnoreCase`; installed assets are conventionally uppercase | Needs validation on case-sensitive filesystems; may hide casing or containment mistakes | Add Linux/macOS tests with mixed-case synthetic roots and use filesystem-appropriate containment rules | | Save/profile storage | Managed `DirectoryNativeDatStore` under Godot `user://SAVE`; native S3SD/S4SD/S3RT files and 24-bit BMP thumbnails | No Win32 path API at runtime; port saves remain isolated from the original installation | Validate replace/flush, case, permissions, and interrupted-write behavior on each export target | @@ -53,41 +53,43 @@ VM op 0x236 (non-modal) / op 0x20f (modal) -> IHost.PlayMovieToSurface / PlayModalMovieToSurface -> VFS-owned MoviePayload bytes -> IMovieDecoderFactory - -> FfmpegMovieDecoder (current live selection) + -> FfmpegMovieDecoder -> FfmpegMovieSession -> age_movie C ABI - -> DirectShowMovieDecoder (unselected; retained only through acceptance) - -> newest RGBA frame - -> retained movie surface - -> Godot compositor + -> newest due RGBA frame -> retained movie surface -> Godot compositor + -> timestamped stereo float PCM -> per-playback AudioStreamGenerator ``` Everything before and after the selected decoder is portable. The replacement decision is an in-process FFmpeg backend behind a project-owned C ABI, not raw FFmpeg structs in Godot/C# and not a subprocess. FFmpeg `n8.1.2-29-g703dcc25b9` is pinned by immutable release URL and SHA-256 in `native/age_movie_ffmpeg/dependency-win64.json`; changing that pin requires rerunning the full installed-movie -gate. The shim dynamically links an LGPL build made without GPL or nonfree components and -uses only `libavformat`, `libavcodec`, `libavutil`, and `libswscale` for the video slice. `libswresample` and the -MPEG audio decoders may be packaged now, but PCM delivery remains a separate slice. Release artifacts must carry +gate. The shim dynamically links an LGPL build made without GPL or nonfree components and uses +`libavformat`, `libavcodec`, `libavutil`, `libswscale`, and `libswresample`. Release artifacts must carry the matching FFmpeg source/configuration and notices required by FFmpeg's [license checklist](https://ffmpeg.org/legal.html). The boundary has two layers: 1. A small native `age_movie` ABI owns all FFmpeg objects and version-sensitive calls. It accepts a borrowed - byte span only for the duration of `open`, copies it into native-owned memory, creates a seekable custom - `AVIOContext`, probes the MPEG program stream, and returns immutable video metadata. Sequential decode returns - one top-down tightly packed RGBA8 frame plus its normalized presentation timestamp. Conversion occurs in an + byte span only for the duration of `open`, copies it once into native-owned memory, and creates independent + seekable custom `AVIOContext` views for video and audio so either stream can be pulled without duplicating the + payload or sharing a demux cursor. ABI v2 returns immutable video/audio metadata. Sequential video decode + returns one top-down tightly packed RGBA8 frame; sequential audio decode returns one interleaved stereo float32 + PCM block. Both carry timestamps normalized to a common media origin. Video conversion occurs in an FFmpeg-owned aligned frame; only exact visible row bytes are copied into the caller's tightly packed buffer. + `libswresample` converts the source channel/sample layout to the Godot generator contract. Return statuses distinguish frame, EOF, invalid arguments, undersized output, and decoder failure; the open call returns a bounded UTF-8 diagnostic and an opened handle retains its last decode error. Close accepts a null handle, and managed `SafeHandle` ownership guarantees one close for each successfully opened handle. No FFmpeg pointer crosses the ABI. -2. Managed `IMovieDecoder` owns the native handle and the paced worker. Synchronous construction provides - `StopTimeMs` before `0x236` returns. The worker uses a monotonic playback origin, decodes ahead by at most one - frame, publishes frames when their timestamps become due, and retains newest-frame-wins behavior if Godot is - late. EOF becomes completion only after the final frame's presentation interval/stop time, so surface cleanup - cannot erase the last frame immediately. Disposal interrupts waits and joins without depending on Godot's main - thread. +2. Managed `IMovieDecoder` owns the native handle and cancellable video/audio workers. Synchronous construction + provides `StopTimeMs` and audio format before `0x236` returns. Video remains at most one due frame ahead; PCM + uses a bounded roughly half-second queue, so the 263-second ending never becomes a whole-file allocation. + Video-only movies use the monotonic playback origin. Audio-bearing movies feed a per-instance + `AudioStreamGenerator`; the monotonic-clamped Godot sound-hardware position + (`playback + time-since-mix - cached output latency`) advances due video frames. Completion requires video + stop time, audio decode EOF, and PCM submission. Disposal interrupts waits, joins both workers, and stops the + generator without depending on native default-device playback. `StopTimeMs` performs a bounded packet scan over the seekable in-memory payload and takes the longest usable FFmpeg format duration, video-stream duration, timestamp span, or constant-frame-rate packet-count duration. This @@ -99,12 +101,11 @@ safety rule still converts any backend initialization failure into an explicitly completed zero-duration movie identity, and the presentation watchdog remains a last-resort guard for an initialized backend that never reaches EOF. -The factory is injection for tests and future decoder replacement, not runtime codec roulette. FFmpeg is now the -selected Windows-x64 live backend; once its corpus and live gates pass, DirectShow is deleted rather than shipped -as a fallback. Modal -completion/cancel remains owned above the decoder by the existing `0x20f` host path. A future audio implementation -will consume timestamped PCM and may become the presentation clock; it must not change the VM-facing stop-time, -surface, or cancellation contracts. +The factory is injection for tests and future decoder replacement, not runtime codec roulette. FFmpeg is the +sole backend; DirectShow was deleted rather than shipped as a fallback after the live gate passed. Modal +completion/cancel remains owned above the decoder by the existing `0x20f` host path and tears down its generator +with the video instance. Native movie flag overrides select mute/music/SE/voice audio buses; the ordinary path +uses the movie bus. This replacement is now also required for Windows gameplay parity. Archive-backed probes on 2026-07-21 show that the current DirectShow graph accepts tested MPEG widths divisible by 16 (208, 288, 304, 400, and @@ -120,6 +121,18 @@ but only decodes Layer II audio and explicitly ignores program-stream PTS in its FFmpeg covers the complete installed codec set and leaves the mod/profile boundary open without selecting a different decoder per effect. +ABI v2 and MPEG audio playback landed on 2026-07-25. The expanded installed-corpus gate passes all 213 assets: +184 remain strictly video-only and all 29 audio-bearing streams decode to finite 44.1 kHz stereo PCM with +monotonic timestamps and clean EOF/teardown. The gate decoded 17,537 audio blocks / 18,185,856 stereo PCM +frames in the same eight-second unpaced run. A headless natural boot completed audio-clocked `LOGO.AGF` at +7,288 ms, released it without generator errors, and opened `OP.AGF`. A first windowed run confirmed audible +LOGO/OP output but exposed crackle/warble. The decoded PCM was continuous; the sink had treated each whole- +millisecond PTS rounding error (up to 44 samples at 44.1 kHz) as a real boundary and inserted or removed samples +from every MPEG block. The sink now ignores established-timeline discrepancies within 2 ms while still honoring +the first timestamp and material later gaps/overlaps. A 4,093-block OP regression pins splice-free continuity. +The clean audible recheck passed for LOGO/OP and CHAPTER. DirectShow and the managed Windows platform annotations +were then deleted; missing, distorted, or unsynchronized audio on any of the 29 audio-bearing assets is now a bug. + Native deliverables are RID-specific and bundled with the Godot export; the runtime must not discover an arbitrary system FFmpeg. The first implementation gate is Windows x64 because that is the current runnable target, but the C ABI and loader paths must reserve Windows x64, Linux x64, macOS x64, and macOS arm64 from the @@ -137,7 +150,8 @@ nondecreasing timestamps. Malformed input and repeated teardown are covered. Thi packaging. `FfmpegMovieDecoder` now adds cancellable timestamp pacing and is the live selection. The Godot build copies the shim, five required shared libraries, and FFmpeg license beside `Himegari.dll`; a natural SYSTEM4 smoke completed 7288 ms `LOGO.AGF`, opened 106919 ms `OP.AGF`, and published frames from both without loading -DirectShow; the user subsequently confirmed both opening movies work in normal windowed playback. Full +DirectShow; the user subsequently confirmed both opening movies work in normal windowed playback. DirectShow +was deleted after the later synchronized-audio gate. Full export/source-offer packaging and non-Windows builds remain outstanding. The Windows-x64 installed-corpus gate passed on 2026-07-22 with the pinned dependency: all 213 signature-selected @@ -145,7 +159,8 @@ MPEG payloads opened and reached EOF under a 30-second per-item bound. It decode display sizes (120x120 through 800x600), independently matched sequence-header dimensions, found positive duration and frame-rate metadata, verified tightly packed RGBA size and nondecreasing timestamps, observed changing imagery in every asset, and closed every session. The set includes all 184 video-only and 29 audio-bearing -streams; audio presence is detected but PCM remains intentionally undelivered. The 263-second `ED.AGF` was the +streams; at that video-only milestone audio presence was detected but PCM remained intentionally undelivered. +The later ABI-v2 gate above supersedes that limitation. The 263-second `ED.AGF` was the slowest decode at 4.4 seconds in the deliberately unpaced gate, so its prior 30-second failure was test-harness whole-frame hashing overhead rather than a decoder incompatibility. The disposable machine report is `build/movie-corpus-ffmpeg.json`; invocation and report semantics are canonical in `docs/tools-reference.md`. diff --git a/docs/tools-reference.md b/docs/tools-reference.md index b153a24..aae82c7 100644 --- a/docs/tools-reference.md +++ b/docs/tools-reference.md @@ -219,7 +219,7 @@ immutable LGPL shared FFmpeg archive and SHA-256. |---|---|---|---| | `native/age_movie_ffmpeg/bootstrap-win64.ps1` | Download, hash-check, extract, and version-check the pinned FFmpeg SDK. Returns the resolved SDK root. | `.\\native\\age_movie_ffmpeg\\bootstrap-win64.ps1 [-Destination ]` | `dependency-win64.json`, network/archive cache → disposable `build/downloads/`, `build/ffmpeg-sdk/` | | `native/age_movie_ffmpeg/build-win64.ps1` | Discover the MSVC x64 toolchain, build `age_movie_ffmpeg.dll`, and stage its exact shared-library/license dependencies. | `.\\native\\age_movie_ffmpeg\\build-win64.ps1 -SdkRoot [-OutputDirectory ]` | C ABI source + FFmpeg SDK → disposable `build/native/win-x64/` by default | -| `tools/movie-corpus-gate` | Discover every MPEG program stream stored under an `.AGF` catalog entry, decode every video frame through the unpaced FFmpeg session, validate independent sequence dimensions, metadata, RGBA size, timestamps, EOF, timeout, and teardown, then emit a per-asset JSON report. Exit 0 means the expected corpus count and every asset passed; exit 1 is a gate failure; exit 2 means the native shim is absent. | `dotnet run --project tools/movie-corpus-gate -- --output build/movie-corpus-ffmpeg.json --expected-count 213 --max-item-ms 30000` · optional `--native-dir ` | `SYS4INI.BIN` + loose/ALF VFS assets + staged FFmpeg shim → stdout progress + disposable `build/movie-corpus-ffmpeg.json` | +| `tools/movie-corpus-gate` | Discover every MPEG program stream stored under an `.AGF` catalog entry; decode every video frame and, when present, every audio block through the unpaced FFmpeg session; validate independent sequence dimensions, metadata, RGBA size, stereo finite PCM, monotonic video/audio timestamps, EOF, timeout, and teardown; then emit a per-asset JSON report. Audio fields include sample rate/channels, block and PCM-frame counts, first/last PTS, signal presence, and decode time. Exit 0 means the expected corpus count and every asset passed; exit 1 is a gate failure; exit 2 means the native shim is absent. | `dotnet run --project tools/movie-corpus-gate -- --output build/movie-corpus-ffmpeg.json --expected-count 213 --max-item-ms 30000` · optional `--native-dir ` | `SYS4INI.BIN` + loose/ALF VFS assets + staged FFmpeg shim → stdout progress + disposable `build/movie-corpus-ffmpeg.json` | The managed isolated probes load from `AGE_FFMPEG_NATIVE_DIR` when set, then application-local and `runtimes/win-x64/native` locations. Build the shim before running the `FfmpegShim*` tests; no original-game @@ -227,8 +227,8 @@ movie is copied into the repository or native output. When `build/native/win-x64 `dotnet build godot/Himegari.csproj` also stages the shim, its five DLL dependencies, and `FFmpeg-LICENSE.txt` beside `Himegari.dll` for development playback. -The corpus gate intentionally bypasses presentation waits: it validates decode compatibility and lifecycle, -not wall-clock playback pacing. `--expected-count` makes additions, omissions, or profile changes explicit; +The corpus gate intentionally bypasses presentation waits: it validates video/audio decode compatibility and +lifecycle, not wall-clock playback pacing or audible output. `--expected-count` makes additions, omissions, or profile changes explicit; changing the pinned FFmpeg dependency requires rerunning this gate. ## Engine (C#) — VM core, CLI, Godot frontend diff --git a/engine/Age.Engine.Tests/Age.Engine.Tests.csproj b/engine/Age.Engine.Tests/Age.Engine.Tests.csproj index ec8c49e..533b180 100644 --- a/engine/Age.Engine.Tests/Age.Engine.Tests.csproj +++ b/engine/Age.Engine.Tests/Age.Engine.Tests.csproj @@ -24,6 +24,7 @@ + @@ -33,7 +34,6 @@ - diff --git a/engine/Age.Engine.Tests/MovieCorpusGateTests.cs b/engine/Age.Engine.Tests/MovieCorpusGateTests.cs index f9b3059..4dc8745 100644 --- a/engine/Age.Engine.Tests/MovieCorpusGateTests.cs +++ b/engine/Age.Engine.Tests/MovieCorpusGateTests.cs @@ -6,10 +6,17 @@ public class MovieCorpusGateTests params FfmpegVideoFrame[] frames) : IFfmpegFrameSource { private readonly Queue _frames = new(frames); + private readonly Queue _audio = new(); public FfmpegMovieInfo Info { get; } = info; public bool Disposed { get; private set; } public bool TryDecodeNextVideoFrame(out FfmpegVideoFrame frame) => _frames.TryDequeue(out frame!); + public void EnqueueAudio(params FfmpegAudioChunk[] chunks) + { + foreach (FfmpegAudioChunk chunk in chunks) _audio.Enqueue(chunk); + } + public bool TryDecodeNextAudioChunk(out FfmpegAudioChunk chunk) + => _audio.TryDequeue(out chunk!); public void Dispose() => Disposed = true; } @@ -86,6 +93,32 @@ public class MovieCorpusGateTests Assert.True(source.Disposed); } + [Fact] + public void GateValidatesAndReportsTimestampedStereoPcm() + { + var source = new FakeFrameSource( + new FfmpegMovieInfo(16, 16, 100, 20, 1, true, 1000, 2, 50), + Frame(16, 16, 1, 0), Frame(16, 16, 2, 50)); + source.EnqueueAudio( + new FfmpegAudioChunk(new float[100], 50, 0), + new FfmpegAudioChunk(Enumerable.Repeat(0.5f, 100).ToArray(), 50, 50)); + var input = new MovieCorpusInput(1, "AUDIO.AGF", () => + new MoviePayload("AUDIO.AGF", SyntheticPayload(16, 16))); + + MovieCorpusReport report = new MovieCorpusGate(_ => source).Run([input], 1, 1000); + + var item = Assert.Single(report.Movies); + Assert.True(item.Passed); + Assert.True(item.HasAudio); + Assert.Equal(1000, item.AudioSampleRate); + Assert.Equal(2, item.AudioChannels); + Assert.Equal(2, item.AudioBlockCount); + Assert.Equal(100, item.AudioFrameCount); + Assert.Equal(0, item.FirstAudioPresentationTimeMs); + Assert.Equal(50, item.LastAudioPresentationTimeMs); + Assert.True(item.AudioHasSignal); + } + private static byte[] SyntheticPayload(int width, int height) => [ 0, 0, 1, 0xba, diff --git a/engine/Age.Engine.Tests/MovieOpcodeTests.cs b/engine/Age.Engine.Tests/MovieOpcodeTests.cs index d562c16..84fda99 100644 --- a/engine/Age.Engine.Tests/MovieOpcodeTests.cs +++ b/engine/Age.Engine.Tests/MovieOpcodeTests.cs @@ -42,6 +42,7 @@ public class MovieOpcodeTests params FfmpegVideoFrame[] frames) : IFfmpegFrameSource { private readonly Queue _frames = new(frames); + private readonly Queue _audio = new(); private int _decodeCalls; public FfmpegMovieInfo Info { get; } = info; public int FailOnDecodeCall { get; init; } = -1; @@ -53,6 +54,12 @@ public class MovieOpcodeTests throw new InvalidDataException("synthetic decode failure"); return _frames.TryDequeue(out frame!); } + public void EnqueueAudio(params FfmpegAudioChunk[] chunks) + { + foreach (FfmpegAudioChunk chunk in chunks) _audio.Enqueue(chunk); + } + public bool TryDecodeNextAudioChunk(out FfmpegAudioChunk chunk) + => _audio.TryDequeue(out chunk!); public void Dispose() => Disposed = true; } @@ -201,6 +208,91 @@ public class MovieOpcodeTests Assert.Equal("synthetic decode failure", decoder.Failure); } + [Fact] + public void FfmpegAudioDecoderQueuesTimestampedPcmAndCompletesAfterSubmissionAndClock() + { + var source = new FakeFfmpegFrameSource( + new FfmpegMovieInfo(1, 1, 100, 20, 1, true, 1000, 2, 50), + SyntheticMovieFrame(1, 0), SyntheticMovieFrame(2, 50)); + source.EnqueueAudio( + new FfmpegAudioChunk(new float[100], 50, 0), + new FfmpegAudioChunk(Enumerable.Repeat(0.25f, 100).ToArray(), 50, 50)); + using var clock = new ManualMoviePacingClock(); + using var decoder = new FfmpegMovieDecoder(source, clock); + + var chunks = new List(); + Assert.True(SpinWait.SpinUntil(() => + { + while (decoder.TryTakeAudioChunk(out var chunk)) chunks.Add(chunk); + return decoder.AudioDecodingCompleted && chunks.Count == 2; + }, 1000)); + Assert.Equal(new long[] { 0, 50 }, chunks.Select(chunk => chunk.PresentationTimeMs)); + Assert.Equal(100, chunks.Sum(chunk => chunk.FrameCount)); + + decoder.MarkAudioSubmitted(); + clock.AdvanceTo(99); + Assert.False(decoder.IsCompleted); + clock.AdvanceTo(100); + Assert.True(SpinWait.SpinUntil(() => decoder.IsCompleted, 1000)); + Assert.Null(decoder.Failure); + } + + [Fact] + public void MovieAudioTimelineDoesNotSpliceContinuousMpegBlocksAtRoundedMillisecondTimestamps() + { + const int sampleRate = 44100; + const int blockFrames = 1152; + const int blockCount = 4093; // OP.AGF's installed MPEG audio block count + long submittedFrames = 0; + bool anchored = false; + + for (int block = 0; block < blockCount; block++) + { + long exactStartFrame = block * (long)blockFrames; + long roundedDownTimestampMs = exactStartFrame * 1000 / sampleRate; + long targetFrame = MovieAudioTimeline.PresentationFrame( + roundedDownTimestampMs, sampleRate); + MovieAudioAdjustment adjustment = MovieAudioTimeline.Align( + submittedFrames, targetFrame, blockFrames, sampleRate, anchored); + + Assert.Equal(default, adjustment); + submittedFrames += blockFrames; + anchored = true; + } + + Assert.Equal(blockCount * (long)blockFrames, submittedFrames); + } + + [Fact] + public void MovieAudioTimelineStillPreservesMaterialGapsAndOverlaps() + { + const int sampleRate = 44100; + + var initialGap = MovieAudioTimeline.Align( + submittedFrames: 0, + targetFrame: MovieAudioTimeline.PresentationFrame(100, sampleRate), + chunkFrames: 1152, + sampleRate: sampleRate, + timelineAnchored: false); + Assert.Equal(new MovieAudioAdjustment(4410, 0), initialGap); + + var laterGap = MovieAudioTimeline.Align( + submittedFrames: 2304, + targetFrame: MovieAudioTimeline.PresentationFrame(100, sampleRate), + chunkFrames: 1152, + sampleRate: sampleRate, + timelineAnchored: true); + Assert.Equal(new MovieAudioAdjustment(2106, 0), laterGap); + + var overlap = MovieAudioTimeline.Align( + submittedFrames: 2304, + targetFrame: MovieAudioTimeline.PresentationFrame(40, sampleRate), + chunkFrames: 1152, + sampleRate: sampleRate, + timelineAnchored: true); + Assert.Equal(new MovieAudioAdjustment(0, 540), overlap); + } + [Fact] public void InitialRootFlagStartsSetAndClearsWhenExitScriptRuns() { @@ -413,39 +505,6 @@ public class MovieOpcodeTests Assert.Equal(new byte[] { 0, 0, 1, 0xba }, movie.Bytes[..4]); } - [Fact] - public void Sc0000MoviePayloadDecodesAn800By600FrameOnWindows() - { - if (!OperatingSystem.IsWindows()) return; - var catalog = Sys4AssetCatalog.Load(Paths.Sys4Ini); - var resources = new ResourceMap(catalog, new Sys4AssetStore(catalog, Paths.GameDir)); - var entry = resources.ResolveMovie(0x33)!; - using var decoder = new DirectShowMovieDecoder(resources.ReadMovie(entry)); - - Assert.True(decoder.StopTimeMs > 0, "DirectShow should expose a positive IMediaPosition stop time"); - - var deadline = DateTime.UtcNow.AddSeconds(10); - RgbaImage? frame = null; - while (DateTime.UtcNow < deadline && !decoder.TryTakeFrame(out frame)) - Thread.Sleep(20); - - Assert.NotNull(frame); - Assert.Equal(800, frame!.Width); - Assert.Equal(600, frame.Height); - Assert.Equal(800 * 600 * 4, frame.Pixels.Length); - - byte[] firstPixels = frame.Pixels; - var changeDeadline = DateTime.UtcNow.AddSeconds(2); - bool changed = false; - while (DateTime.UtcNow < changeDeadline && !changed) - { - Thread.Sleep(20); - if (decoder.TryTakeFrame(out var later)) - changed = !firstPixels.AsSpan().SequenceEqual(later.Pixels); - } - Assert.True(changed, "DirectShow should deliver changing MPEG frames, not one retained still"); - } - [Theory] [InlineData(0x2be3, "MVB961.AGF", 280, 352, 500)] [InlineData(0x2b94, "MVB238.AGF", 280, 352, 866)] @@ -485,6 +544,42 @@ public class MovieOpcodeTests Assert.True(changed, $"{expectedName} should deliver changing decoded frames"); } + [Theory] + [InlineData(0x33, "CHAPTER.AGF")] + [InlineData(0x2bf1, "MVS001.AGF")] + [InlineData(0x335f, "LOGO.AGF")] + public void FfmpegShimDecodesRepresentativeMpegAudio(int resourceId, string expectedName) + { + if (!OperatingSystem.IsWindows()) return; + ConfigureFfmpegNativeProbe(); + var catalog = Sys4AssetCatalog.Load(Paths.Sys4Ini); + var resources = new ResourceMap(catalog, new Sys4AssetStore(catalog, Paths.GameDir)); + var payload = resources.ReadMovie(resources.ResolveMovie(resourceId)!); + + using var movie = new FfmpegMovieSession(payload); + + Assert.Equal(expectedName, payload.Name); + Assert.True(movie.Info.HasAudio); + Assert.Equal(44100, movie.Info.AudioSampleRate); + Assert.Equal(2, movie.Info.AudioChannels); + Assert.True(movie.TryDecodeNextAudioChunk(out var first)); + Assert.True(first.FrameCount > 0); + Assert.Equal(first.FrameCount * 2, first.InterleavedStereo.Length); + Assert.True(first.PresentationTimeMs >= 0); + Assert.All(first.InterleavedStereo, sample => Assert.True(float.IsFinite(sample))); + bool heardSignal = first.InterleavedStereo.Any(sample => Math.Abs(sample) > 0.00001f); + long priorTimestamp = first.PresentationTimeMs; + for (int block = 0; block < 500 && !heardSignal + && movie.TryDecodeNextAudioChunk(out var later); block++) + { + Assert.True(later.PresentationTimeMs >= priorTimestamp); + Assert.All(later.InterleavedStereo, sample => Assert.True(float.IsFinite(sample))); + priorTimestamp = later.PresentationTimeMs; + heardSignal = later.InterleavedStereo.Any(sample => Math.Abs(sample) > 0.00001f); + } + Assert.True(heardSignal, $"{expectedName} should contain non-silent MPEG audio"); + } + [Fact] public void FfmpegShimRejectsTruncatedMovieWithBoundedDiagnostic() { diff --git a/engine/Age.Engine/Hosting/IHost.cs b/engine/Age.Engine/Hosting/IHost.cs index 077e71c..433ad1d 100644 --- a/engine/Age.Engine/Hosting/IHost.cs +++ b/engine/Age.Engine/Hosting/IHost.cs @@ -123,7 +123,7 @@ public interface IHost void ScheduleSoundEffectStart(int channel, int startMode, long delayMs) { } void ReleaseSoundEffect(int channel) { } void FadeBgm(int targetPercent, long durationMs) { } - // Native op 0x236 binds a DirectShow movie decoder to an existing retained texture surface. + // Native op 0x236 binds a movie decoder to an existing retained texture surface. // Playback is non-modal: the VM advances to the following instruction while the host publishes frames. /// The initialized movie graph's stop position in truncated integer milliseconds, or null /// when the host could not obtain usable timing metadata. Native op 0x23f queries this state diff --git a/godot/DirectShowMovieDecoder.cs b/godot/DirectShowMovieDecoder.cs deleted file mode 100644 index 4142aca..0000000 --- a/godot/DirectShowMovieDecoder.cs +++ /dev/null @@ -1,304 +0,0 @@ -using System; -using System.IO; -using System.Runtime.InteropServices; -using System.Runtime.Versioning; -using System.Threading; -using Age.Engine.Sys4; - -[ComVisible(true), Guid("0579154A-2B53-4994-B0D0-E773148EFF85"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)] -internal interface ISampleGrabberCB -{ - [PreserveSig] int SampleCB(double time, IntPtr sample); - [PreserveSig] int BufferCB(double time, IntPtr buffer, int length); -} - -/// -/// Windows DirectShow bridge for AGE's MPEG program-stream movie payloads. The payload is supplied by -/// IAssetStore; a private temporary file only adapts those owned bytes to DirectShow's stock MPEG source. -/// Decoded RGB32 samples are copied into process memory and consumed by Godot's retained compositor. -/// -[SupportedOSPlatform("windows")] -internal sealed class DirectShowMovieDecoder : IMovieDecoder, ISampleGrabberCB -{ - private readonly byte[] _payload; - private readonly Thread _thread; - private readonly ManualResetEventSlim _ready = new(false); - private readonly object _frameLock = new(); - private byte[]? _latestRgba; - private int _width, _height; - private bool _bottomUp; - private volatile bool _stopping; - private volatile bool _completed; - private string? _error; - private string? _tempPath; - private object? _graphObject; - private IMediaControl? _control; - - public bool IsCompleted => _completed; - public string? Failure => _error; - /// The graph's IMediaPosition stop time converted exactly as native op 0x23f does: - /// seconds * 1000, truncated toward zero. Null means DirectShow supplied no usable value. - public long? StopTimeMs { get; private set; } - - public DirectShowMovieDecoder(MoviePayload movie) - { - _payload = movie.Bytes; - _thread = new Thread(DecodeThread) { IsBackground = true, Name = $"AGE movie {movie.Name}" }; - _thread.SetApartmentState(ApartmentState.MTA); - _thread.Start(); - // Native 0x236 builds the graph synchronously. Preserve that initialization boundary while keeping - // actual playback asynchronous; timeout becomes a deterministic load error rather than a VM hang. - if (!_ready.Wait(TimeSpan.FromSeconds(10))) throw new InvalidOperationException("movie graph initialization timed out"); - if (_error != null) throw new InvalidOperationException(_error); - } - - public bool TryTakeFrame(out RgbaImage frame) - { - lock (_frameLock) - { - if (_latestRgba == null) { frame = default!; return false; } - frame = new RgbaImage(_width, _height, _latestRgba); - _latestRgba = null; - return true; - } - } - - private void DecodeThread() - { - int co = CoInitializeEx(IntPtr.Zero, 0); // COINIT_MULTITHREADED; DirectShow graph and callbacks share this apartment. - try - { - _tempPath = Path.Combine(Path.GetTempPath(), $"age-movie-{Guid.NewGuid():N}.mpg"); - File.WriteAllBytes(_tempPath, _payload); - - _graphObject = Activator.CreateInstance(Type.GetTypeFromCLSID(CLSID_FilterGraph, throwOnError: true)!)!; - var graph = (IGraphBuilder)_graphObject; - var grabberObject = Activator.CreateInstance(Type.GetTypeFromCLSID(CLSID_SampleGrabber, true)!)!; - var nullObject = Activator.CreateInstance(Type.GetTypeFromCLSID(CLSID_NullRenderer, true)!)!; - var grabberFilter = (IBaseFilter)grabberObject; - var nullFilter = (IBaseFilter)nullObject; - var grabber = (ISampleGrabber)grabberObject; - - Check(graph.AddFilter(grabberFilter, "AGE Sample Grabber"), "add sample grabber"); - Check(graph.AddFilter(nullFilter, "AGE Null Renderer"), "add null renderer"); - var requested = new AMMediaType { MajorType = MEDIATYPE_Video, SubType = MEDIASUBTYPE_RGB32, - FormatType = FORMAT_VideoInfo }; - Check(grabber.SetMediaType(ref requested), "request RGB32 movie output"); - Check(graph.AddSourceFilter(_tempPath, "AGE VFS MPEG Source", out var source), "open MPEG source"); - - var sourceOut = FirstPin(source, PinDirection.Output); - var grabberIn = FirstPin(grabberFilter, PinDirection.Input); - var grabberOut = FirstPin(grabberFilter, PinDirection.Output); - var nullIn = FirstPin(nullFilter, PinDirection.Input); - // Intelligent connection inserts only the MPEG splitter/video decoder needed to reach RGB32. - // The splitter's audio pin remains unrendered, keeping movie audio outside this slice. - Check(graph.Connect(sourceOut, grabberIn), "connect MPEG video decoder"); - Check(graph.ConnectDirect(grabberOut, nullIn, IntPtr.Zero), "connect null renderer"); - - var connected = new AMMediaType(); - Check(grabber.GetConnectedMediaType(ref connected), "query movie format"); - try - { - if (connected.FormatPtr == IntPtr.Zero) throw new InvalidDataException("movie decoder returned no VIDEOINFOHEADER"); - var vi = Marshal.PtrToStructure(connected.FormatPtr); - _width = vi.BitmapInfo.Width; - _bottomUp = vi.BitmapInfo.Height > 0; - _height = Math.Abs(vi.BitmapInfo.Height); - if (_width <= 0 || _height <= 0) throw new InvalidDataException($"invalid movie dimensions {_width}x{_height}"); - } - finally { FreeMediaType(ref connected); } - - Check(grabber.SetOneShot(false), "configure continuous samples"); - Check(grabber.SetBufferSamples(false), "disable redundant frame buffering"); - Check(grabber.SetCallback(this, 1), "install decoded frame callback"); - _control = (IMediaControl)_graphObject; - Check(_control.Run(), "start movie graph"); - int stateHr = _control.GetState(5000, out int graphState); - if (stateHr < 0) Check(stateHr, "wait for running movie graph"); - if (graphState != 2) throw new InvalidOperationException($"movie graph entered unexpected state {graphState}"); - var mediaPosition = (IMediaPosition)_graphObject; - int stopHr = mediaPosition.get_StopTime(out double stopTimeSeconds); - double stopTimeMilliseconds = stopTimeSeconds * 1000.0; - if (stopHr >= 0 && double.IsFinite(stopTimeMilliseconds) - && stopTimeMilliseconds >= int.MinValue - && stopTimeMilliseconds <= int.MaxValue) - StopTimeMs = (long)System.Math.Truncate(stopTimeMilliseconds); - _ready.Set(); - var mediaEvent = (IMediaEvent)_graphObject; - while (!_stopping) - { - int eventHr = mediaEvent.WaitForCompletion(0, out _); - if (eventHr >= 0) { _completed = true; break; } - if (eventHr != E_ABORT) Check(eventHr, "poll movie completion"); - Thread.Sleep(10); - } - } - catch (Exception e) - { - _error = $"DirectShow MPEG decode failed: {e}"; - _ready.Set(); - } - finally - { - try { _control?.Stop(); } catch { } - ReleaseCom(_control); _control = null; - ReleaseCom(_graphObject); _graphObject = null; - if (_tempPath != null) try { File.Delete(_tempPath); } catch { } - if (co >= 0) CoUninitialize(); - } - } - - public int SampleCB(double sampleTime, IntPtr sample) => 0; - - public int BufferCB(double sampleTime, IntPtr buffer, int length) - { AcceptBgra(buffer, length); return 0; } - - private void AcceptBgra(IntPtr buffer, int length) - { - int rowBytes = checked(_width * 4); - if (buffer == IntPtr.Zero || length < rowBytes * _height) return; - byte[] bgra = new byte[rowBytes * _height]; - Marshal.Copy(buffer, bgra, 0, bgra.Length); - byte[] rgba = new byte[bgra.Length]; - for (int y = 0; y < _height; y++) - { - int src = (_bottomUp ? _height - 1 - y : y) * rowBytes; - int dst = y * rowBytes; - for (int x = 0; x < _width; x++, src += 4, dst += 4) - { - rgba[dst] = bgra[src + 2]; rgba[dst + 1] = bgra[src + 1]; - rgba[dst + 2] = bgra[src]; rgba[dst + 3] = 255; - } - } - lock (_frameLock) _latestRgba = rgba; // newest decoded frame wins if Godot renders more slowly - } - - public void Dispose() - { - _stopping = true; - if (_thread.IsAlive) _thread.Join(TimeSpan.FromSeconds(3)); - _ready.Dispose(); - } - - private static IPin FirstPin(IBaseFilter filter, PinDirection direction) - { - Check(filter.EnumPins(out var pins), "enumerate source pins"); - var one = new IPin[1]; - while (pins.Next(1, one, IntPtr.Zero) == 0) - { - Check(one[0].QueryDirection(out var found), "query source pin direction"); - if (found == direction) return one[0]; - } - throw new InvalidOperationException($"DirectShow source has no {direction} pin"); - } - - private static void Check(int hr, string operation) - { if (hr < 0) throw new COMException($"{operation} failed (HRESULT 0x{hr:x8})", hr); } - private static void ReleaseCom(object? value) - { if (value != null && Marshal.IsComObject(value)) try { Marshal.ReleaseComObject(value); } catch { } } - private static void FreeMediaType(ref AMMediaType mt) - { - if (mt.FormatPtr != IntPtr.Zero) { Marshal.FreeCoTaskMem(mt.FormatPtr); mt.FormatPtr = IntPtr.Zero; } - if (mt.UnknownPtr != IntPtr.Zero) { Marshal.Release(mt.UnknownPtr); mt.UnknownPtr = IntPtr.Zero; } - } - - private static readonly Guid CLSID_FilterGraph = new("E436EBB3-524F-11CE-9F53-0020AF0BA770"); - private static readonly Guid CLSID_SampleGrabber = new("C1F400A0-3F08-11D3-9F0B-006008039E37"); - private static readonly Guid CLSID_NullRenderer = new("C1F400A4-3F08-11D3-9F0B-006008039E37"); - private static readonly Guid MEDIATYPE_Video = new("73646976-0000-0010-8000-00AA00389B71"); - private static readonly Guid MEDIASUBTYPE_RGB32 = new("E436EB7E-524F-11CE-9F53-0020AF0BA770"); - private static readonly Guid FORMAT_VideoInfo = new("05589F80-C356-11CE-BF01-00AA0055595A"); - private const int E_ABORT = unchecked((int)0x80004004); - - [DllImport("ole32.dll")] private static extern int CoInitializeEx(IntPtr reserved, uint coInit); - [DllImport("ole32.dll")] private static extern void CoUninitialize(); - - private enum PinDirection { Input, Output } - [StructLayout(LayoutKind.Sequential)] private struct AMMediaType - { - public Guid MajorType, SubType; [MarshalAs(UnmanagedType.Bool)] public bool FixedSizeSamples; - [MarshalAs(UnmanagedType.Bool)] public bool TemporalCompression; public int SampleSize; - public Guid FormatType; public IntPtr UnknownPtr; public int FormatSize; public IntPtr FormatPtr; - } - [StructLayout(LayoutKind.Sequential)] private struct DsRect { public int Left, Top, Right, Bottom; } - [StructLayout(LayoutKind.Sequential)] private struct BitmapInfoHeader - { public int Size, Width, Height; public short Planes, BitCount; public int Compression, ImageSize, XPelsPerMeter, YPelsPerMeter, ColorsUsed, ColorsImportant; } - [StructLayout(LayoutKind.Sequential)] private struct VideoInfoHeader - { public DsRect Source, Target; public int BitRate, BitErrorRate; public long AvgTimePerFrame; public BitmapInfoHeader BitmapInfo; } - - [ComImport, Guid("6B652FFF-11FE-4FCE-92AD-0266B5D7C78F"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)] - private interface ISampleGrabber - { - [PreserveSig] int SetOneShot([MarshalAs(UnmanagedType.Bool)] bool value); - [PreserveSig] int SetMediaType(ref AMMediaType type); - [PreserveSig] int GetConnectedMediaType(ref AMMediaType type); - [PreserveSig] int SetBufferSamples([MarshalAs(UnmanagedType.Bool)] bool value); - [PreserveSig] int GetCurrentBuffer(ref int size, IntPtr buffer); - [PreserveSig] int GetCurrentSample(out IntPtr sample); - [PreserveSig] int SetCallback(ISampleGrabberCB callback, int method); - } - [ComImport, Guid("56A868A9-0AD4-11CE-B03A-0020AF0BA770"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)] - private interface IGraphBuilder - { - [PreserveSig] int AddFilter(IBaseFilter filter, [MarshalAs(UnmanagedType.LPWStr)] string name); - [PreserveSig] int RemoveFilter(IBaseFilter filter); [PreserveSig] int EnumFilters(out IntPtr filters); - [PreserveSig] int FindFilterByName([MarshalAs(UnmanagedType.LPWStr)] string name, out IBaseFilter filter); - [PreserveSig] int ConnectDirect(IPin output, IPin input, IntPtr mediaType); - [PreserveSig] int Reconnect(IPin pin); [PreserveSig] int Disconnect(IPin pin); [PreserveSig] int SetDefaultSyncSource(); - [PreserveSig] int Connect(IPin output, IPin input); [PreserveSig] int Render(IPin output); - [PreserveSig] int RenderFile([MarshalAs(UnmanagedType.LPWStr)] string file, [MarshalAs(UnmanagedType.LPWStr)] string? playList); - [PreserveSig] int AddSourceFilter([MarshalAs(UnmanagedType.LPWStr)] string file, [MarshalAs(UnmanagedType.LPWStr)] string name, out IBaseFilter filter); - [PreserveSig] int SetLogFile(IntPtr file); [PreserveSig] int Abort(); [PreserveSig] int ShouldOperationContinue(); - } - [ComImport, Guid("56A86895-0AD4-11CE-B03A-0020AF0BA770"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)] - private interface IBaseFilter - { - [PreserveSig] int GetClassID(out Guid clsid); [PreserveSig] int Stop(); [PreserveSig] int Pause(); - [PreserveSig] int Run(long start); [PreserveSig] int GetState(int timeout, out int state); - [PreserveSig] int SetSyncSource(IntPtr clock); [PreserveSig] int GetSyncSource(out IntPtr clock); - [PreserveSig] int EnumPins(out IEnumPins pins); [PreserveSig] int FindPin([MarshalAs(UnmanagedType.LPWStr)] string id, out IPin pin); - [PreserveSig] int QueryFilterInfo(IntPtr info); [PreserveSig] int JoinFilterGraph(IntPtr graph, [MarshalAs(UnmanagedType.LPWStr)] string name); - [PreserveSig] int QueryVendorInfo([MarshalAs(UnmanagedType.LPWStr)] out string vendor); - } - [ComImport, Guid("56A86892-0AD4-11CE-B03A-0020AF0BA770"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)] - private interface IEnumPins - { [PreserveSig] int Next(int count, [Out, MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 0)] IPin[] pins, IntPtr fetched); [PreserveSig] int Skip(int count); [PreserveSig] int Reset(); [PreserveSig] int Clone(out IEnumPins clone); } - [ComImport, Guid("56A86891-0AD4-11CE-B03A-0020AF0BA770"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)] - private interface IPin - { - [PreserveSig] int Connect(IPin receive, IntPtr mediaType); [PreserveSig] int ReceiveConnection(IPin connector, IntPtr mediaType); - [PreserveSig] int Disconnect(); [PreserveSig] int ConnectedTo(out IPin pin); [PreserveSig] int ConnectionMediaType(IntPtr mediaType); - [PreserveSig] int QueryPinInfo(IntPtr info); [PreserveSig] int QueryDirection(out PinDirection direction); - [PreserveSig] int QueryId([MarshalAs(UnmanagedType.LPWStr)] out string id); [PreserveSig] int QueryAccept(IntPtr mediaType); - [PreserveSig] int EnumMediaTypes(out IntPtr types); [PreserveSig] int QueryInternalConnections(IntPtr pins, ref int count); - [PreserveSig] int EndOfStream(); [PreserveSig] int BeginFlush(); [PreserveSig] int EndFlush(); [PreserveSig] int NewSegment(long start, long stop, double rate); - } - [ComImport, Guid("56A868B1-0AD4-11CE-B03A-0020AF0BA770"), InterfaceType(ComInterfaceType.InterfaceIsDual)] - private interface IMediaControl - { [PreserveSig] int Run(); [PreserveSig] int Pause(); [PreserveSig] int Stop(); [PreserveSig] int GetState(int timeout, out int state); } - [ComImport, Guid("56A868B2-0AD4-11CE-B03A-0020AF0BA770"), InterfaceType(ComInterfaceType.InterfaceIsDual)] - private interface IMediaPosition - { - [PreserveSig] int get_Duration(out double seconds); - [PreserveSig] int put_CurrentPosition(double seconds); - [PreserveSig] int get_CurrentPosition(out double seconds); - [PreserveSig] int get_StopTime(out double seconds); - [PreserveSig] int put_StopTime(double seconds); - [PreserveSig] int get_PrerollTime(out double seconds); - [PreserveSig] int put_PrerollTime(double seconds); - [PreserveSig] int put_Rate(double rate); - [PreserveSig] int get_Rate(out double rate); - [PreserveSig] int CanSeekForward(out int canSeekForward); - [PreserveSig] int CanSeekBackward(out int canSeekBackward); - } - [ComImport, Guid("56A868B6-0AD4-11CE-B03A-0020AF0BA770"), InterfaceType(ComInterfaceType.InterfaceIsDual)] - private interface IMediaEvent - { - [PreserveSig] int GetEventHandle(out IntPtr eventHandle); - [PreserveSig] int GetEvent(out int eventCode, out IntPtr param1, out IntPtr param2, int timeoutMs); - [PreserveSig] int WaitForCompletion(int timeoutMs, out int eventCode); - [PreserveSig] int CancelDefaultHandling(int eventCode); - [PreserveSig] int RestoreDefaultHandling(int eventCode); - [PreserveSig] int FreeEventParams(int eventCode, IntPtr param1, IntPtr param2); - } -} diff --git a/godot/FfmpegMovieDecoder.cs b/godot/FfmpegMovieDecoder.cs index 118b358..c9bbcae 100644 --- a/godot/FfmpegMovieDecoder.cs +++ b/godot/FfmpegMovieDecoder.cs @@ -1,6 +1,7 @@ using System; using System.Diagnostics; using System.IO; +using System.Collections.Generic; using System.Threading; using Age.Engine.Sys4; @@ -9,6 +10,11 @@ internal interface IMoviePacingClock bool WaitUntil(long elapsedMilliseconds, WaitHandle cancellation); } +internal interface IExternallyAdvancedMoviePacingClock : IMoviePacingClock +{ + void AdvanceTo(long elapsedMilliseconds); +} + internal sealed class StopwatchMoviePacingClock : IMoviePacingClock { private readonly long _startedAt = Stopwatch.GetTimestamp(); @@ -26,9 +32,39 @@ internal sealed class StopwatchMoviePacingClock : IMoviePacingClock } } +internal sealed class ExternallyAdvancedMoviePacingClock : IExternallyAdvancedMoviePacingClock, IDisposable +{ + private readonly AutoResetEvent _advanced = new(false); + private long _now; + + public bool WaitUntil(long elapsedMilliseconds, WaitHandle cancellation) + { + while (Interlocked.Read(ref _now) < elapsedMilliseconds) + { + int signalled = WaitHandle.WaitAny([cancellation, _advanced]); + if (signalled == 0) return false; + } + return true; + } + + public void AdvanceTo(long elapsedMilliseconds) + { + long current; + do + { + current = Interlocked.Read(ref _now); + if (elapsedMilliseconds <= current) return; + } + while (Interlocked.CompareExchange(ref _now, elapsedMilliseconds, current) != current); + _advanced.Set(); + } + + public void Dispose() => _advanced.Dispose(); +} + /// -/// Timestamp-paced FFmpeg video delivery. The worker decodes no more than one frame ahead, publishes only -/// when its presentation timestamp is due, and reports completion after the final presentation interval. +/// Timestamp-paced FFmpeg delivery. Video-only streams retain monotonic stopwatch pacing. Audio-bearing +/// streams decode PCM into a bounded queue and advance video against the sound-device clock supplied by Godot. /// internal sealed class FfmpegMovieDecoder : IMovieDecoder { @@ -36,33 +72,70 @@ internal sealed class FfmpegMovieDecoder : IMovieDecoder private readonly IMoviePacingClock _clock; private readonly ManualResetEvent _cancel = new(false); private readonly Thread _thread; + private readonly Thread? _audioThread; private readonly object _frameLock = new(); + private readonly object _audioLock = new(); + private readonly Queue _audioChunks = new(); + private readonly AutoResetEvent _audioSpace = new(false); + private readonly int _maximumQueuedAudioFrames; private RgbaImage? _latestFrame; private volatile bool _completed; + private volatile bool _videoTimelineCompleted; + private volatile bool _audioDecodingCompleted; + private volatile bool _audioSubmitted; private string? _failure; + private int _queuedAudioFrames; + private int _activeWorkers; private int _disposed; public long? StopTimeMs => _source.Info.StopTimeMs; public bool IsCompleted => _completed; public string? Failure => Volatile.Read(ref _failure); + public MovieAudioInfo? AudioInfo { get; } + public bool AudioDecodingCompleted => _audioDecodingCompleted; public FfmpegMovieDecoder(MoviePayload movie) - : this(new FfmpegMovieSession(movie), new StopwatchMoviePacingClock()) { } + : this(new FfmpegMovieSession(movie), null) { } - internal FfmpegMovieDecoder(IFfmpegFrameSource source, IMoviePacingClock clock) + internal FfmpegMovieDecoder(IFfmpegFrameSource source, IMoviePacingClock? clock) { _source = source ?? throw new ArgumentNullException(nameof(source)); - _clock = clock ?? throw new ArgumentNullException(nameof(clock)); + _clock = clock ?? (source.Info.HasAudio + ? new ExternallyAdvancedMoviePacingClock() + : new StopwatchMoviePacingClock()); + if (source.Info.HasAudio) + { + AudioInfo = new MovieAudioInfo(source.Info.AudioSampleRate, source.Info.AudioChannels); + _maximumQueuedAudioFrames = Math.Max(source.Info.AudioSampleRate / 2, + source.Info.AudioFrameSamples * 2); + } _thread = new Thread(DecodeThread) { IsBackground = true, - Name = "AGE FFmpeg movie", + Name = "AGE FFmpeg movie video", }; - try { _thread.Start(); } + _audioThread = source.Info.HasAudio + ? new Thread(AudioDecodeThread) + { + IsBackground = true, + Name = "AGE FFmpeg movie audio", + } + : null; + _activeWorkers = _audioThread == null ? 1 : 2; + try + { + _thread.Start(); + _audioThread?.Start(); + } catch { + _cancel.Set(); + if (_thread.IsAlive) _thread.Join(); + if (_audioThread?.IsAlive == true) _audioThread.Join(); _source.Dispose(); _cancel.Dispose(); + _audioSpace.Dispose(); + if (_clock is IDisposable disposableClock) disposableClock.Dispose(); throw; } } @@ -82,6 +155,34 @@ internal sealed class FfmpegMovieDecoder : IMovieDecoder } } + public bool TryTakeAudioChunk(out MovieAudioChunk chunk) + { + lock (_audioLock) + { + if (_audioChunks.Count == 0) + { + chunk = default!; + return false; + } + chunk = _audioChunks.Dequeue(); + _queuedAudioFrames -= chunk.FrameCount; + } + _audioSpace.Set(); + return true; + } + + public void AdvancePlaybackClock(long elapsedMilliseconds) + { + if (_clock is IExternallyAdvancedMoviePacingClock external) + external.AdvanceTo(Math.Max(0, elapsedMilliseconds)); + } + + public void MarkAudioSubmitted() + { + _audioSubmitted = true; + UpdateCompletion(); + } + private void DecodeThread() { try @@ -96,7 +197,11 @@ internal sealed class FfmpegMovieDecoder : IMovieDecoder throw new InvalidDataException("FFmpeg stream ended before producing a video frame"); long completionTime = Math.Max(_source.Info.StopTimeMs, lastTimestamp + FrameIntervalMilliseconds(_source.Info)); - if (_clock.WaitUntil(completionTime, _cancel)) _completed = true; + if (_clock.WaitUntil(completionTime, _cancel)) + { + _videoTimelineCompleted = true; + UpdateCompletion(); + } return; } if (frame.PresentationTimeMs < 0 || frame.PresentationTimeMs < lastTimestamp) @@ -110,15 +215,79 @@ internal sealed class FfmpegMovieDecoder : IMovieDecoder } catch (Exception error) { - Volatile.Write(ref _failure, error.Message); - _completed = true; // decode failure must never strand an AGE movie wait + Fail(error); } finally { - _source.Dispose(); + WorkerCompleted(); } } + private void AudioDecodeThread() + { + try + { + long priorTimestamp = -1; + while (!_cancel.WaitOne(0)) + { + while (Volatile.Read(ref _queuedAudioFrames) >= _maximumQueuedAudioFrames) + { + int signalled = WaitHandle.WaitAny([_cancel, _audioSpace]); + if (signalled == 0) return; + } + if (!_source.TryDecodeNextAudioChunk(out FfmpegAudioChunk decoded)) + { + _audioDecodingCompleted = true; + UpdateCompletion(); + return; + } + if (decoded.FrameCount <= 0 + || decoded.InterleavedStereo.Length != checked(decoded.FrameCount * 2) + || decoded.PresentationTimeMs < 0 + || decoded.PresentationTimeMs < priorTimestamp) + throw new InvalidDataException( + $"FFmpeg returned invalid audio block {decoded.FrameCount}f " + + $"at {decoded.PresentationTimeMs} ms after {priorTimestamp} ms"); + var chunk = new MovieAudioChunk(decoded.InterleavedStereo, decoded.FrameCount, + decoded.PresentationTimeMs); + lock (_audioLock) + { + _audioChunks.Enqueue(chunk); + _queuedAudioFrames += chunk.FrameCount; + } + priorTimestamp = decoded.PresentationTimeMs; + } + } + catch (Exception error) + { + Fail(error); + } + finally + { + WorkerCompleted(); + } + } + + private void Fail(Exception error) + { + Interlocked.CompareExchange(ref _failure, error.Message, null); + _completed = true; // decode failure must never strand an AGE movie wait + _cancel.Set(); + _audioSpace.Set(); + } + + private void UpdateCompletion() + { + if (_failure != null || !_videoTimelineCompleted) return; + if (AudioInfo != null && (!_audioDecodingCompleted || !_audioSubmitted)) return; + _completed = true; + } + + private void WorkerCompleted() + { + if (Interlocked.Decrement(ref _activeWorkers) == 0) _source.Dispose(); + } + private static long FrameIntervalMilliseconds(FfmpegMovieInfo info) { if (info.FrameRateNumerator <= 0 || info.FrameRateDenominator <= 0) return 0; @@ -130,9 +299,14 @@ internal sealed class FfmpegMovieDecoder : IMovieDecoder { if (Interlocked.Exchange(ref _disposed, 1) != 0) return; _cancel.Set(); + _audioSpace.Set(); if (_thread.IsAlive && Thread.CurrentThread != _thread) _thread.Join(); + if (_audioThread?.IsAlive == true && Thread.CurrentThread != _audioThread) + _audioThread.Join(); _cancel.Dispose(); + _audioSpace.Dispose(); + if (_clock is IDisposable disposableClock) disposableClock.Dispose(); } } diff --git a/godot/FfmpegMovieNative.cs b/godot/FfmpegMovieNative.cs index 78fedc8..fdd92e3 100644 --- a/godot/FfmpegMovieNative.cs +++ b/godot/FfmpegMovieNative.cs @@ -10,26 +10,36 @@ internal readonly record struct FfmpegMovieInfo( long StopTimeMs, int FrameRateNumerator, int FrameRateDenominator, - bool HasAudio); + bool HasAudio, + int AudioSampleRate = 0, + int AudioChannels = 0, + int AudioFrameSamples = 0); internal sealed record FfmpegVideoFrame(RgbaImage Image, long PresentationTimeMs); +internal sealed record FfmpegAudioChunk(float[] InterleavedStereo, int FrameCount, long PresentationTimeMs); internal interface IFfmpegFrameSource : IDisposable { FfmpegMovieInfo Info { get; } bool TryDecodeNextVideoFrame(out FfmpegVideoFrame frame); + bool TryDecodeNextAudioChunk(out FfmpegAudioChunk chunk) + { + chunk = default!; + return false; + } } /// Sequential, unpaced access to the project-owned FFmpeg C ABI for isolated probes and playback. internal sealed class FfmpegMovieSession : IFfmpegFrameSource { private FfmpegMovieHandle _handle; + private readonly object _decodeLock = new(); public FfmpegMovieInfo Info { get; } public FfmpegMovieSession(MoviePayload movie) { ArgumentNullException.ThrowIfNull(movie); - if (FfmpegMovieNative.AbiVersion() != 1) + if (FfmpegMovieNative.AbiVersion() != 2) throw new InvalidOperationException("unsupported age_movie_ffmpeg ABI version"); byte[] error = new byte[1024]; @@ -42,31 +52,97 @@ internal sealed class FfmpegMovieSession : IFfmpegFrameSource $"FFmpeg movie open failed for {movie.Name}: {FfmpegMovieNative.DecodeUtf8(error)}"); } Info = new FfmpegMovieInfo(nativeInfo.Width, nativeInfo.Height, nativeInfo.StopTimeMs, - nativeInfo.FrameRateNumerator, nativeInfo.FrameRateDenominator, nativeInfo.HasAudio != 0); + nativeInfo.FrameRateNumerator, nativeInfo.FrameRateDenominator, nativeInfo.HasAudio != 0, + nativeInfo.AudioSampleRate, nativeInfo.AudioChannels, nativeInfo.AudioFrameSamples); if (Info.Width <= 0 || Info.Height <= 0 || Info.StopTimeMs <= 0) { _handle.Dispose(); throw new InvalidDataException( $"FFmpeg returned invalid metadata for {movie.Name}: {Info.Width}x{Info.Height}, {Info.StopTimeMs} ms"); } + if (Info.HasAudio && (Info.AudioSampleRate <= 0 || Info.AudioChannels != 2 + || Info.AudioFrameSamples <= 0)) + { + _handle.Dispose(); + throw new InvalidDataException( + $"FFmpeg returned invalid audio metadata for {movie.Name}: " + + $"{Info.AudioSampleRate} Hz, {Info.AudioChannels} channels, {Info.AudioFrameSamples} frames"); + } } public bool TryDecodeNextVideoFrame(out FfmpegVideoFrame frame) { ObjectDisposedException.ThrowIf(_handle.IsClosed, this); byte[] pixels = new byte[checked(Info.Width * Info.Height * 4)]; - int result = FfmpegMovieNative.DecodeVideo(_handle, pixels, (nuint)pixels.Length, out long ptsMs); + int result; + long ptsMs; + string? error = null; + lock (_decodeLock) + { + result = FfmpegMovieNative.DecodeVideo( + _handle, pixels, (nuint)pixels.Length, out ptsMs); + if (result != FfmpegMovieNative.EndOfFile && result != FfmpegMovieNative.Frame) + error = FfmpegMovieNative.LastError(_handle); + } if (result == FfmpegMovieNative.EndOfFile) { frame = default!; return false; } if (result != FfmpegMovieNative.Frame) - throw new InvalidDataException($"FFmpeg movie decode failed: {FfmpegMovieNative.LastError(_handle)}"); + throw new InvalidDataException($"FFmpeg movie decode failed: {error}"); frame = new FfmpegVideoFrame(new RgbaImage(Info.Width, Info.Height, pixels), ptsMs); return true; } + public bool TryDecodeNextAudioChunk(out FfmpegAudioChunk chunk) + { + ObjectDisposedException.ThrowIf(_handle.IsClosed, this); + if (!Info.HasAudio) + { + chunk = default!; + return false; + } + + int capacity = Math.Max(Info.AudioFrameSamples, 1); + while (true) + { + float[] samples = new float[checked(capacity * 2)]; + int result; + int frameCount; + long ptsMs; + string? error = null; + lock (_decodeLock) + { + result = FfmpegMovieNative.DecodeAudio( + _handle, samples, (nuint)capacity, out frameCount, out ptsMs); + if (result != FfmpegMovieNative.EndOfFile + && result != FfmpegMovieNative.Frame + && result != FfmpegMovieNative.BufferTooSmall) + error = FfmpegMovieNative.LastError(_handle); + } + if (result == FfmpegMovieNative.EndOfFile) + { + chunk = default!; + return false; + } + if (result == FfmpegMovieNative.BufferTooSmall) + { + if (frameCount <= capacity) + throw new InvalidDataException("FFmpeg audio decode reported an invalid required buffer size"); + capacity = frameCount; + continue; + } + if (result != FfmpegMovieNative.Frame) + throw new InvalidDataException($"FFmpeg movie audio decode failed: {error}"); + if (frameCount <= 0 || frameCount > capacity) + throw new InvalidDataException($"FFmpeg returned invalid audio frame count {frameCount}"); + if (frameCount != capacity) Array.Resize(ref samples, checked(frameCount * 2)); + chunk = new FfmpegAudioChunk(samples, frameCount, ptsMs); + return true; + } + } + public void Dispose() => _handle.Dispose(); } @@ -84,6 +160,7 @@ internal static class FfmpegMovieNative { internal const int EndOfFile = 0; internal const int Frame = 1; + internal const int BufferTooSmall = -3; private const string LibraryName = "age_movie_ffmpeg"; [StructLayout(LayoutKind.Sequential)] @@ -95,6 +172,9 @@ internal static class FfmpegMovieNative public int FrameRateNumerator; public int FrameRateDenominator; public int HasAudio; + public int AudioSampleRate; + public int AudioChannels; + public int AudioFrameSamples; } static FfmpegMovieNative() @@ -160,6 +240,14 @@ internal static class FfmpegMovieNative nuint rgbaSize, out long presentationTimeMs); + [DllImport(LibraryName, EntryPoint = "age_movie_decode_audio", CallingConvention = CallingConvention.Cdecl)] + internal static extern int DecodeAudio( + FfmpegMovieHandle movie, + [Out] float[] stereo, + nuint frameCapacity, + out int frameCount, + out long presentationTimeMs); + [DllImport(LibraryName, EntryPoint = "age_movie_last_error", CallingConvention = CallingConvention.Cdecl)] private static extern IntPtr LastErrorNative(FfmpegMovieHandle movie); diff --git a/godot/GodotAdvHost.cs b/godot/GodotAdvHost.cs index 5119ec0..907a224 100644 --- a/godot/GodotAdvHost.cs +++ b/godot/GodotAdvHost.cs @@ -1,7 +1,6 @@ using System; using System.Collections.Generic; using System.Linq; -using System.Runtime.Versioning; using System.Threading; using Age.Engine.Hosting; using Age.Engine.Model; @@ -18,7 +17,6 @@ public enum HostPresentationReason DiscreteSourceCell = 16, } -[SupportedOSPlatform("windows")] public sealed class GodotAdvHost : IHost { private readonly Main _main; @@ -1061,8 +1059,9 @@ public sealed class GodotAdvHost : IHost ["surface"] = surfaceSlot, ["file"] = movie.Name, ["flags"] = movieFlags, ["sync_mask"] = syncMask, ["modal"] = modal, }); - bool started = _main.TryPlayMovie(movie.Bytes, movie.Name, playbackId, resourceId, asset.PackedId, - out stopTimeMs); + bool started = _main.TryPlayMovie( + movie.Bytes, movie.Name, playbackId, resourceId, asset.PackedId, movieFlags, + out stopTimeMs); if (!started) { stopTimeMs = 0; diff --git a/godot/IMovieDecoder.cs b/godot/IMovieDecoder.cs index f630db0..aa3792b 100644 --- a/godot/IMovieDecoder.cs +++ b/godot/IMovieDecoder.cs @@ -1,7 +1,9 @@ using System; -using System.Runtime.Versioning; using Age.Engine.Sys4; +internal readonly record struct MovieAudioInfo(int SampleRate, int Channels); +internal sealed record MovieAudioChunk(float[] InterleavedStereo, int FrameCount, long PresentationTimeMs); + /// /// Platform-neutral movie playback boundary. Construction is synchronous so metadata needed by the VM is /// available before op 0x236 returns; frame delivery and completion remain asynchronous. @@ -12,15 +14,18 @@ internal interface IMovieDecoder : IDisposable bool IsCompleted { get; } string? Failure { get; } bool TryTakeFrame(out RgbaImage frame); + MovieAudioInfo? AudioInfo => null; + bool AudioDecodingCompleted => true; + bool TryTakeAudioChunk(out MovieAudioChunk chunk) + { + chunk = default!; + return false; + } + void AdvancePlaybackClock(long elapsedMilliseconds) { } + void MarkAudioSubmitted() { } } internal interface IMovieDecoderFactory { IMovieDecoder Open(MoviePayload movie); } - -[SupportedOSPlatform("windows")] -internal sealed class DirectShowMovieDecoderFactory : IMovieDecoderFactory -{ - public IMovieDecoder Open(MoviePayload movie) => new DirectShowMovieDecoder(movie); -} diff --git a/godot/Main.cs b/godot/Main.cs index 6552e92..091583e 100644 --- a/godot/Main.cs +++ b/godot/Main.cs @@ -1,7 +1,6 @@ using System.Collections.Generic; using System.Diagnostics; using System.Linq; -using System.Runtime.Versioning; using System.Text.Json; using System.Threading.Tasks; using Godot; @@ -13,7 +12,6 @@ using Age.Engine.Sys4; using Age.Engine.Vm; using Script = Age.Engine.Model.Script; // disambiguate from Godot.Script -[SupportedOSPlatform("windows")] public partial class Main : Godot.Control { private const int ScreenWidth = 800; @@ -56,10 +54,12 @@ public partial class Main : Godot.Control private IReadOnlyList _debugSceneEntries = System.Array.Empty(); private readonly Age.Engine.Hosting.FrameClock _clock = new(); private readonly System.Collections.Generic.Dictionary _movies = new(); + private readonly System.Collections.Generic.Dictionary _movieAudio = new(); // 0x236 opens its decoder synchronously on the VM thread so 0x23f can query timing immediately. // Presentation ownership transfers here; _Process adopts staged decoders before sampling frames. private readonly System.Collections.Concurrent.ConcurrentDictionary _pendingMovies = new(); private IMovieDecoderFactory _movieDecoderFactory = new FfmpegMovieDecoderFactory(); + private double _audioOutputLatencySeconds; private readonly System.Collections.Generic.HashSet _movieFrameSeen = new(); private readonly System.Collections.Generic.HashSet _movieCompletionNotified = new(); private GodotTraceSink _trace = null!; @@ -161,13 +161,18 @@ public partial class Main : Godot.Control _bgm = new AudioStreamPlayer(); _voice = new AudioStreamPlayer(); + EnsureAudioBuses(); + _bgm.Bus = "Music"; + _voice.Bus = "Voice"; AddChild(_bgm); AddChild(_voice); for (int i = 0; i < _sfx.Length; i++) { _sfx[i] = new AudioStreamPlayer(); + _sfx[i].Bus = "SFX"; AddChild(_sfx[i]); } + _audioOutputLatencySeconds = AudioServer.GetOutputLatency(); var userArgs = OS.GetCmdlineUserArgs(); _selftest = System.Array.IndexOf(userArgs, "--selftest") >= 0; @@ -582,19 +587,30 @@ public partial class Main : Godot.Control GodotTraceSnapshot trace = _trace.Snapshot(); var activeMovies = _movies .OrderBy(pair => pair.Key) - .Select(pair => new + .Select(pair => { - playback_id = pair.Key, - resource_id = pair.Value.ResourceId, - name = pair.Value.Name, - asset_id = pair.Value.AssetId, - stop_time_ms = pair.Value.Decoder.StopTimeMs, - decoder_completed = pair.Value.Decoder.IsCompleted, - decoder_failure = pair.Value.Decoder.Failure, - frame_seen = _movieFrameSeen.Contains(pair.Key), - completion_notified = _movieCompletionNotified.Contains(pair.Key), - watchdog_ms = pair.Value.WatchdogMs, - elapsed_ms = (long)Stopwatch.GetElapsedTime(pair.Value.StartedAtTimestamp).TotalMilliseconds, + _movieAudio.TryGetValue(pair.Key, out var audio); + return new + { + playback_id = pair.Key, + resource_id = pair.Value.ResourceId, + name = pair.Value.Name, + asset_id = pair.Value.AssetId, + stop_time_ms = pair.Value.Decoder.StopTimeMs, + decoder_completed = pair.Value.Decoder.IsCompleted, + decoder_failure = pair.Value.Decoder.Failure, + frame_seen = _movieFrameSeen.Contains(pair.Key), + completion_notified = _movieCompletionNotified.Contains(pair.Key), + watchdog_ms = pair.Value.WatchdogMs, + elapsed_ms = (long)Stopwatch.GetElapsedTime(pair.Value.StartedAtTimestamp).TotalMilliseconds, + audio_sample_rate = pair.Value.Decoder.AudioInfo?.SampleRate, + audio_decode_completed = pair.Value.Decoder.AudioInfo == null + ? (bool?)null : pair.Value.Decoder.AudioDecodingCompleted, + audio_route = audio?.Route.ToString(), + audio_clock_ms = audio?.ClockMs, + audio_submitted_through_ms = audio?.SubmittedThroughMs, + audio_buffer_underruns = audio?.BufferUnderruns, + }; }) .ToArray(); var pendingMovies = _pendingMovies @@ -794,6 +810,8 @@ public partial class Main : Godot.Control GD.Print($"[perf-log] wrote {_perf.FrameCount} frames / {_perf.RecompositeCount} recomposites -> {_perf.Path}"); _perf = null; } + foreach (var audio in _movieAudio.Values) audio.Dispose(); + _movieAudio.Clear(); foreach (var movie in _pendingMovies.Values) movie.Decoder.Dispose(); _pendingMovies.Clear(); foreach (var movie in _movies.Values) movie.Decoder.Dispose(); @@ -1662,14 +1680,15 @@ public partial class Main : Godot.Control } public bool TryPlayMovie(byte[] mpegBytes, string assetName, long playbackId, - long resourceId, int assetId, + long resourceId, int assetId, long movieFlags, out long? stopTimeMs) { stopTimeMs = null; try { var payload = new Age.Engine.Sys4.MoviePayload(assetName, mpegBytes); - var runtime = MovieRuntime.Open(assetName, assetId, resourceId, payload, _movieDecoderFactory); + var runtime = MovieRuntime.Open( + assetName, assetId, resourceId, payload, _movieDecoderFactory, movieFlags); stopTimeMs = runtime.Decoder.StopTimeMs; while (!_pendingMovies.TryAdd(playbackId, runtime)) if (_pendingMovies.TryRemove(playbackId, out var prior)) prior.Decoder.Dispose(); @@ -1689,9 +1708,19 @@ public partial class Main : Godot.Control if (!_pendingMovies.TryRemove(playbackId, out var movie)) continue; if (_movies.Remove(playbackId, out var prior)) prior.Decoder.Dispose(); _movies[playbackId] = movie; + if (movie.Decoder.AudioInfo != null) + { + if (_movieAudio.Remove(playbackId, out var priorAudio)) priorAudio.Dispose(); + _movieAudio[playbackId] = new MovieAudioOutput( + this, movie.Decoder, MovieAudioRouteFromFlags(movie.MovieFlags), + _audioOutputLatencySeconds); + } _movieCompletionNotified.Remove(playbackId); GD.Print($"movie started {movie.Name} playback={playbackId} " + - $"({movie.Decoder.StopTimeMs?.ToString() ?? "unknown"} ms from VFS)"); + $"({movie.Decoder.StopTimeMs?.ToString() ?? "unknown"} ms from VFS" + + (movie.Decoder.AudioInfo is { } audio + ? $", audio={audio.SampleRate}Hz stereo route={MovieAudioRouteFromFlags(movie.MovieFlags)}" + : "") + ")"); } } @@ -1700,6 +1729,7 @@ public partial class Main : Godot.Control if (_host == null) return; foreach (var (playbackId, movie) in _movies) { + if (_movieAudio.TryGetValue(playbackId, out var audio)) audio.Update(); if (movie.Decoder.TryTakeFrame(out var frame)) { _host.PublishMovieFrame(playbackId, movie.Name, movie.AssetId, frame); @@ -1722,6 +1752,7 @@ public partial class Main : Godot.Control public void StopMovie(long playbackId) { + if (_movieAudio.Remove(playbackId, out var audio)) audio.Dispose(); if (_pendingMovies.TryRemove(playbackId, out var pending)) pending.Decoder.Dispose(); if (_movies.Remove(playbackId, out var movie)) { @@ -1732,6 +1763,26 @@ public partial class Main : Godot.Control _movieCompletionNotified.Remove(playbackId); } + private static MovieAudioRoute MovieAudioRouteFromFlags(long flags) + { + ulong value = unchecked((ulong)flags); + if ((value & 0x10000) != 0) return MovieAudioRoute.Muted; + if ((value & 0x20000) != 0) return MovieAudioRoute.Music; + if ((value & 0x40000) != 0) return MovieAudioRoute.SoundEffect; + if ((value & 0x80000) != 0) return MovieAudioRoute.Voice; + return MovieAudioRoute.Movie; + } + + private static void EnsureAudioBuses() + { + foreach (string name in new[] { "Music", "SFX", "Voice", "Movie" }) + { + if (AudioServer.GetBusIndex(name) >= 0) continue; + AudioServer.AddBus(); + AudioServer.SetBusName(AudioServer.BusCount - 1, name); + } + } + public void AppendLine(string text) => _text.Text += text + "\n"; public void PageBreak() { diff --git a/godot/MovieAudioOutput.cs b/godot/MovieAudioOutput.cs new file mode 100644 index 0000000..8a352a8 --- /dev/null +++ b/godot/MovieAudioOutput.cs @@ -0,0 +1,182 @@ +using System; +using Godot; + +internal enum MovieAudioRoute +{ + Muted = 0, + Music = 1, + SoundEffect = 2, + Voice = 3, + Movie = 4, +} + +/// Godot-main-thread PCM sink and sound-hardware clock for one movie playback. +internal sealed class MovieAudioOutput : IDisposable +{ + private const int MaximumPushFrames = 4096; + + private readonly IMovieDecoder _decoder; + private readonly AudioStreamPlayer _player; + private readonly int _sampleRate; + private readonly double _outputLatencySeconds; + private AudioStreamGeneratorPlayback? _playback; + private MovieAudioChunk? _pending; + private int _pendingOffsetFrames; + private long _pendingGapFrames; + private long _submittedFrames; + private long _clockMs; + private bool _timelineAnchored; + private bool _started; + private bool _prerolling; + private bool _submissionCompleted; + private bool _disposed; + + public MovieAudioRoute Route { get; } + public long ClockMs => _clockMs; + public long SubmittedThroughMs => _sampleRate <= 0 ? 0 : _submittedFrames * 1000 / _sampleRate; + public int BufferUnderruns => _playback?.GetSkips() ?? 0; + + public MovieAudioOutput(Node owner, IMovieDecoder decoder, MovieAudioRoute route, + double outputLatencySeconds) + { + ArgumentNullException.ThrowIfNull(owner); + _decoder = decoder ?? throw new ArgumentNullException(nameof(decoder)); + MovieAudioInfo info = decoder.AudioInfo + ?? throw new ArgumentException("movie decoder has no audio stream", nameof(decoder)); + if (info.SampleRate <= 0 || info.Channels != 2) + throw new ArgumentOutOfRangeException(nameof(decoder), "movie PCM must be stereo with a positive sample rate"); + + Route = route; + _sampleRate = info.SampleRate; + _outputLatencySeconds = Math.Max(0, outputLatencySeconds); + var generator = new AudioStreamGenerator + { + MixRate = _sampleRate, + BufferLength = 0.25f, + }; + _player = new AudioStreamPlayer + { + Stream = generator, + Bus = RouteBus(route), + VolumeDb = route == MovieAudioRoute.Muted ? -80.0f : 0.0f, + }; + owner.AddChild(_player); + } + + public void Update() + { + if (_disposed) return; + if (_started) + { + double audibleSeconds = _player.GetPlaybackPosition() + + AudioServer.GetTimeSinceLastMix() + - _outputLatencySeconds; + long candidate = (long)Math.Floor(Math.Max(0, audibleSeconds) * 1000.0); + if (candidate > _clockMs) _clockMs = candidate; + } + FeedAvailablePcm(); + _decoder.AdvancePlaybackClock(_clockMs); + } + + private void FeedAvailablePcm() + { + while (true) + { + if (_pending == null && !_decoder.TryTakeAudioChunk(out _pending)) + { + if (_decoder.AudioDecodingCompleted && !_submissionCompleted) + { + _submissionCompleted = true; + _decoder.MarkAudioSubmitted(); + } + return; + } + if (_pending == null) return; + if (!_started) Start(); + + if (_pendingOffsetFrames == 0 && _pendingGapFrames == 0) + { + long audibleFrame = _clockMs * _sampleRate / 1000; + if (audibleFrame > _submittedFrames) _submittedFrames = audibleFrame; + long targetFrame = MovieAudioTimeline.PresentationFrame( + _pending.PresentationTimeMs, _sampleRate); + MovieAudioAdjustment adjustment = MovieAudioTimeline.Align( + _submittedFrames, targetFrame, _pending.FrameCount, _sampleRate, _timelineAnchored); + _pendingGapFrames = adjustment.GapFrames; + _pendingOffsetFrames = adjustment.SkipFrames; + _timelineAnchored = true; + } + + int available = _playback!.GetFramesAvailable(); + if (available <= 0) return; + if (_pendingGapFrames > 0) + { + int count = (int)Math.Min(Math.Min(_pendingGapFrames, available), MaximumPushFrames); + if (!_playback.PushBuffer(new Vector2[count])) return; + FinishPreroll(); + _pendingGapFrames -= count; + _submittedFrames += count; + continue; + } + + int remaining = _pending.FrameCount - _pendingOffsetFrames; + if (remaining <= 0) + { + _pending = null; + _pendingOffsetFrames = 0; + continue; + } + int frames = Math.Min(Math.Min(remaining, available), MaximumPushFrames); + var output = new Vector2[frames]; + for (int index = 0; index < frames; index++) + { + int source = checked((_pendingOffsetFrames + index) * 2); + output[index] = new Vector2( + _pending.InterleavedStereo[source], + _pending.InterleavedStereo[source + 1]); + } + if (!_playback.PushBuffer(output)) return; + FinishPreroll(); + _pendingOffsetFrames += frames; + _submittedFrames += frames; + if (_pendingOffsetFrames == _pending.FrameCount) + { + _pending = null; + _pendingOffsetFrames = 0; + } + } + } + + private void Start() + { + _player.Play(); + _player.StreamPaused = true; + _playback = _player.GetStreamPlayback() as AudioStreamGeneratorPlayback + ?? throw new InvalidOperationException("Godot did not create movie audio generator playback"); + _started = true; + _prerolling = true; + } + + private void FinishPreroll() + { + if (!_prerolling) return; + _prerolling = false; + _player.StreamPaused = false; + } + + private static StringName RouteBus(MovieAudioRoute route) => route switch + { + MovieAudioRoute.Music => "Music", + MovieAudioRoute.SoundEffect => "SFX", + MovieAudioRoute.Voice => "Voice", + _ => "Movie", + }; + + public void Dispose() + { + if (_disposed) return; + _disposed = true; + _player.Stop(); + _player.QueueFree(); + } +} diff --git a/godot/MovieAudioTimeline.cs b/godot/MovieAudioTimeline.cs new file mode 100644 index 0000000..7f20fd4 --- /dev/null +++ b/godot/MovieAudioTimeline.cs @@ -0,0 +1,50 @@ +using System; + +internal readonly record struct MovieAudioAdjustment(long GapFrames, int SkipFrames); + +/// +/// Converts coarse MPEG timestamps into PCM alignment decisions without turning sub-millisecond +/// timestamp quantization into an audible splice at every decoded block boundary. +/// +internal static class MovieAudioTimeline +{ + private const int TimestampJitterToleranceMilliseconds = 2; + + public static long PresentationFrame(long presentationTimeMs, int sampleRate) + { + if (presentationTimeMs < 0) + throw new ArgumentOutOfRangeException(nameof(presentationTimeMs)); + if (sampleRate <= 0) + throw new ArgumentOutOfRangeException(nameof(sampleRate)); + return checked((long)Math.Round( + presentationTimeMs * (double)sampleRate / 1000.0, + MidpointRounding.AwayFromZero)); + } + + public static MovieAudioAdjustment Align( + long submittedFrames, + long targetFrame, + int chunkFrames, + int sampleRate, + bool timelineAnchored) + { + if (submittedFrames < 0) + throw new ArgumentOutOfRangeException(nameof(submittedFrames)); + if (targetFrame < 0) + throw new ArgumentOutOfRangeException(nameof(targetFrame)); + if (chunkFrames <= 0) + throw new ArgumentOutOfRangeException(nameof(chunkFrames)); + if (sampleRate <= 0) + throw new ArgumentOutOfRangeException(nameof(sampleRate)); + + long delta = targetFrame - submittedFrames; + long tolerance = timelineAnchored + ? Math.Max(1L, (sampleRate * (long)TimestampJitterToleranceMilliseconds + 999L) / 1000L) + : 0L; + if (delta > tolerance) + return new MovieAudioAdjustment(delta, 0); + if (delta < -tolerance) + return new MovieAudioAdjustment(0, checked((int)Math.Min(chunkFrames, -delta))); + return default; + } +} diff --git a/godot/MovieRuntime.cs b/godot/MovieRuntime.cs index da165c1..80c5d66 100644 --- a/godot/MovieRuntime.cs +++ b/godot/MovieRuntime.cs @@ -4,14 +4,14 @@ using Age.Engine.Sys4; /// Presentation-side ownership for one decoder plus its fail-safe completion deadline. internal sealed record MovieRuntime(string Name, int AssetId, long ResourceId, IMovieDecoder Decoder, - long StartedAtTimestamp, long WatchdogMs) + long MovieFlags, long StartedAtTimestamp, long WatchdogMs) { public static MovieRuntime Open(string name, int assetId, long resourceId, MoviePayload payload, - IMovieDecoderFactory factory) + IMovieDecoderFactory factory, long movieFlags = 0) { ArgumentNullException.ThrowIfNull(factory); IMovieDecoder decoder = factory.Open(payload); - return new MovieRuntime(name, assetId, resourceId, decoder, Stopwatch.GetTimestamp(), + return new MovieRuntime(name, assetId, resourceId, decoder, movieFlags, Stopwatch.GetTimestamp(), decoder.StopTimeMs is >= 0 and var stopTime ? Math.Clamp(stopTime + 2000, 5000, 300000) : 30000); diff --git a/native/age_movie_ffmpeg/age_movie.c b/native/age_movie_ffmpeg/age_movie.c index 8fc79fc..c684280 100644 --- a/native/age_movie_ffmpeg/age_movie.c +++ b/native/age_movie_ffmpeg/age_movie.c @@ -12,31 +12,55 @@ #include #include #include +#include +#include #include -#define AGE_MOVIE_ABI_VERSION 1u +#define AGE_MOVIE_ABI_VERSION 2u #define AGE_MOVIE_IO_BUFFER_SIZE 32768 #define AGE_MOVIE_ERROR_SIZE 512 +typedef struct age_movie_io_state { + const uint8_t *payload; + size_t payload_size; + size_t position; +} age_movie_io_state; + struct age_movie { uint8_t *payload; size_t payload_size; - size_t payload_position; - AVIOContext *io; - AVFormatContext *format; + age_movie_io_state video_io_state; + AVIOContext *video_io; + AVFormatContext *video_format; AVCodecContext *video_codec; - AVPacket *packet; - AVFrame *frame; + AVPacket *video_packet; + AVFrame *video_frame; AVFrame *rgba_frame; struct SwsContext *sws; int video_stream_index; AVRational video_time_base; - int64_t timestamp_origin; - int64_t first_frame_timestamp; - int input_eof; - int decoder_draining; - int decoder_eof; - int64_t decoded_frame_count; + int video_input_eof; + int video_decoder_draining; + int video_decoder_eof; + int64_t decoded_video_frame_count; + + age_movie_io_state audio_io_state; + AVIOContext *audio_io; + AVFormatContext *audio_format; + AVCodecContext *audio_codec; + AVPacket *audio_packet; + AVFrame *audio_frame; + SwrContext *swr; + int audio_stream_index; + AVRational audio_time_base; + int audio_input_eof; + int audio_decoder_draining; + int audio_decoder_eof; + int audio_frame_pending; + int64_t decoded_audio_sample_count; + int audio_sample_rate; + + int64_t timeline_origin_ms; int width; int height; char error[AGE_MOVIE_ERROR_SIZE]; @@ -58,28 +82,28 @@ static void set_error(age_movie *movie, const char *operation, int ffmpeg_error) } static int read_packet(void *opaque, uint8_t *buffer, int buffer_size) { - age_movie *movie = (age_movie *)opaque; - if (movie == NULL || buffer == NULL || buffer_size <= 0) { + age_movie_io_state *state = (age_movie_io_state *)opaque; + if (state == NULL || buffer == NULL || buffer_size <= 0) { return AVERROR(EINVAL); } - if (movie->payload_position >= movie->payload_size) { + if (state->position >= state->payload_size) { return AVERROR_EOF; } - size_t remaining = movie->payload_size - movie->payload_position; + size_t remaining = state->payload_size - state->position; size_t requested = (size_t)buffer_size; size_t count = remaining < requested ? remaining : requested; - memcpy(buffer, movie->payload + movie->payload_position, count); - movie->payload_position += count; + memcpy(buffer, state->payload + state->position, count); + state->position += count; return (int)count; } static int64_t seek_payload(void *opaque, int64_t offset, int whence) { - age_movie *movie = (age_movie *)opaque; - if (movie == NULL) { + age_movie_io_state *state = (age_movie_io_state *)opaque; + if (state == NULL) { return AVERROR(EINVAL); } if ((whence & AVSEEK_SIZE) != 0) { - return movie->payload_size <= INT64_MAX ? (int64_t)movie->payload_size : AVERROR(EOVERFLOW); + return state->payload_size <= INT64_MAX ? (int64_t)state->payload_size : AVERROR(EOVERFLOW); } int base_whence = whence & ~AVSEEK_FORCE; @@ -87,12 +111,12 @@ static int64_t seek_payload(void *opaque, int64_t offset, int whence) { switch (base_whence) { case SEEK_SET: base = 0; break; case SEEK_CUR: - if (movie->payload_position > INT64_MAX) return AVERROR(EOVERFLOW); - base = (int64_t)movie->payload_position; + if (state->position > INT64_MAX) return AVERROR(EOVERFLOW); + base = (int64_t)state->position; break; case SEEK_END: - if (movie->payload_size > INT64_MAX) return AVERROR(EOVERFLOW); - base = (int64_t)movie->payload_size; + if (state->payload_size > INT64_MAX) return AVERROR(EOVERFLOW); + base = (int64_t)state->payload_size; break; default: return AVERROR(EINVAL); } @@ -101,10 +125,10 @@ static int64_t seek_payload(void *opaque, int64_t offset, int whence) { return AVERROR(EOVERFLOW); } int64_t target = base + offset; - if (target < 0 || (uint64_t)target > movie->payload_size) { + if (target < 0 || (uint64_t)target > state->payload_size) { return AVERROR(EINVAL); } - movie->payload_position = (size_t)target; + state->position = (size_t)target; return target; } @@ -113,14 +137,65 @@ static int64_t rescale_ms_down(int64_t value, AVRational time_base) { AV_ROUND_DOWN | AV_ROUND_PASS_MINMAX); } +static int open_format_context(age_movie *movie, age_movie_io_state *state, + AVIOContext **out_io, AVFormatContext **out_format) { + state->payload = movie->payload; + state->payload_size = movie->payload_size; + state->position = 0; + uint8_t *io_buffer = (uint8_t *)av_malloc(AGE_MOVIE_IO_BUFFER_SIZE); + if (io_buffer == NULL) return AVERROR(ENOMEM); + AVIOContext *io = avio_alloc_context(io_buffer, AGE_MOVIE_IO_BUFFER_SIZE, 0, + state, read_packet, NULL, seek_payload); + if (io == NULL) { + av_free(io_buffer); + return AVERROR(ENOMEM); + } + AVFormatContext *format = avformat_alloc_context(); + if (format == NULL) { + av_freep(&io->buffer); + avio_context_free(&io); + return AVERROR(ENOMEM); + } + format->pb = io; + format->flags |= AVFMT_FLAG_CUSTOM_IO; + int result = avformat_open_input(&format, NULL, NULL, NULL); + if (result >= 0) result = avformat_find_stream_info(format, NULL); + if (result < 0) { + avformat_close_input(&format); + av_freep(&io->buffer); + avio_context_free(&io); + return result; + } + *out_io = io; + *out_format = format; + return 0; +} + +static int64_t determine_timeline_origin_ms(const AVFormatContext *format) { + int64_t origin_ms = INT64_MAX; + for (unsigned int index = 0; index < format->nb_streams; index++) { + const AVStream *stream = format->streams[index]; + enum AVMediaType type = stream->codecpar->codec_type; + if (type != AVMEDIA_TYPE_VIDEO && type != AVMEDIA_TYPE_AUDIO) continue; + if (stream->start_time == AV_NOPTS_VALUE) continue; + int64_t candidate = rescale_ms_down(stream->start_time, stream->time_base); + if (candidate < origin_ms) origin_ms = candidate; + } + if (origin_ms == INT64_MAX && format->start_time != AV_NOPTS_VALUE) + origin_ms = av_rescale_q_rnd(format->start_time, + (AVRational){1, AV_TIME_BASE}, (AVRational){1, 1000}, + AV_ROUND_DOWN | AV_ROUND_PASS_MINMAX); + return origin_ms == INT64_MAX ? 0 : origin_ms; +} + static int determine_duration(age_movie *movie, AVStream *stream, int64_t *out_duration_ms) { int64_t best_duration_ms = 0; if (stream->duration != AV_NOPTS_VALUE && stream->duration > 0) { int64_t candidate = rescale_ms_down(stream->duration, stream->time_base); if (candidate > best_duration_ms) best_duration_ms = candidate; } - if (movie->format->duration != AV_NOPTS_VALUE && movie->format->duration > 0) { - int64_t candidate = av_rescale_q_rnd(movie->format->duration, + if (movie->video_format->duration != AV_NOPTS_VALUE && movie->video_format->duration > 0) { + int64_t candidate = av_rescale_q_rnd(movie->video_format->duration, (AVRational){1, AV_TIME_BASE}, (AVRational){1, 1000}, AV_ROUND_DOWN | AV_ROUND_PASS_MINMAX); if (candidate > best_duration_ms) best_duration_ms = candidate; @@ -132,7 +207,19 @@ static int determine_duration(age_movie *movie, AVStream *stream, int64_t *out_d AVPacket *probe = av_packet_alloc(); if (probe == NULL) return AVERROR(ENOMEM); int result; - while ((result = av_read_frame(movie->format, probe)) >= 0) { + int64_t media_end_ms = 0; + while ((result = av_read_frame(movie->video_format, probe)) >= 0) { + AVStream *packet_stream = movie->video_format->streams[probe->stream_index]; + enum AVMediaType packet_type = packet_stream->codecpar->codec_type; + if ((packet_type == AVMEDIA_TYPE_VIDEO || packet_type == AVMEDIA_TYPE_AUDIO) + && probe->pts != AV_NOPTS_VALUE) { + int64_t packet_end = probe->pts; + if (probe->duration > 0 && packet_end <= INT64_MAX - probe->duration) + packet_end += probe->duration; + int64_t candidate = rescale_ms_down(packet_end, packet_stream->time_base) + - movie->timeline_origin_ms; + if (candidate > media_end_ms) media_end_ms = candidate; + } if (probe->stream_index == movie->video_stream_index) { video_packet_count++; int64_t timestamp = probe->pts != AV_NOPTS_VALUE ? probe->pts : probe->dts; @@ -149,6 +236,7 @@ static int determine_duration(age_movie *movie, AVStream *stream, int64_t *out_d } av_packet_free(&probe); if (result != AVERROR_EOF) return result; + if (media_end_ms > best_duration_ms) best_duration_ms = media_end_ms; if (first != AV_NOPTS_VALUE && end != AV_NOPTS_VALUE && end > first) { int64_t candidate = rescale_ms_down(end - first, stream->time_base); if (candidate > best_duration_ms) best_duration_ms = candidate; @@ -165,32 +253,58 @@ static int determine_duration(age_movie *movie, AVStream *stream, int64_t *out_d static int rewind_for_decode(age_movie *movie, AVStream *stream) { int64_t target = stream->start_time != AV_NOPTS_VALUE ? stream->start_time : 0; - int result = av_seek_frame(movie->format, movie->video_stream_index, target, AVSEEK_FLAG_BACKWARD); + int result = av_seek_frame(movie->video_format, movie->video_stream_index, target, AVSEEK_FLAG_BACKWARD); if (result < 0) { - movie->payload_position = 0; - if (movie->io != NULL) { - avio_flush(movie->io); - movie->io->eof_reached = 0; - movie->io->error = 0; + movie->video_io_state.position = 0; + if (movie->video_io != NULL) { + avio_flush(movie->video_io); + movie->video_io->eof_reached = 0; + movie->video_io->error = 0; } - result = avformat_seek_file(movie->format, movie->video_stream_index, + result = avformat_seek_file(movie->video_format, movie->video_stream_index, INT64_MIN, target, INT64_MAX, 0); } if (result >= 0) avcodec_flush_buffers(movie->video_codec); return result; } +static int rewind_audio_for_decode(age_movie *movie, AVStream *stream) { + int64_t target = stream->start_time != AV_NOPTS_VALUE ? stream->start_time : 0; + int result = av_seek_frame(movie->audio_format, movie->audio_stream_index, target, AVSEEK_FLAG_BACKWARD); + if (result < 0) { + movie->audio_io_state.position = 0; + if (movie->audio_io != NULL) { + avio_flush(movie->audio_io); + movie->audio_io->eof_reached = 0; + movie->audio_io->error = 0; + } + result = avformat_seek_file(movie->audio_format, movie->audio_stream_index, + INT64_MIN, target, INT64_MAX, 0); + } + if (result >= 0) avcodec_flush_buffers(movie->audio_codec); + return result; +} + static void destroy_movie(age_movie *movie) { if (movie == NULL) return; + swr_free(&movie->swr); sws_freeContext(movie->sws); + av_frame_free(&movie->audio_frame); av_frame_free(&movie->rgba_frame); - av_frame_free(&movie->frame); - av_packet_free(&movie->packet); + av_frame_free(&movie->video_frame); + av_packet_free(&movie->audio_packet); + av_packet_free(&movie->video_packet); + avcodec_free_context(&movie->audio_codec); avcodec_free_context(&movie->video_codec); - avformat_close_input(&movie->format); - if (movie->io != NULL) { - av_freep(&movie->io->buffer); - avio_context_free(&movie->io); + avformat_close_input(&movie->audio_format); + avformat_close_input(&movie->video_format); + if (movie->audio_io != NULL) { + av_freep(&movie->audio_io->buffer); + avio_context_free(&movie->audio_io); + } + if (movie->video_io != NULL) { + av_freep(&movie->video_io->buffer); + avio_context_free(&movie->video_io); } av_free(movie->payload); free(movie); @@ -216,8 +330,7 @@ int32_t AGE_MOVIE_CALL age_movie_open(const uint8_t *bytes, size_t length, return AGE_MOVIE_ERROR; } movie->video_stream_index = -1; - movie->timestamp_origin = AV_NOPTS_VALUE; - movie->first_frame_timestamp = AV_NOPTS_VALUE; + movie->audio_stream_index = -1; if (length > SIZE_MAX - AV_INPUT_BUFFER_PADDING_SIZE) { snprintf(movie->error, sizeof(movie->error), "movie payload is too large"); @@ -231,47 +344,23 @@ int32_t AGE_MOVIE_CALL age_movie_open(const uint8_t *bytes, size_t length, memcpy(movie->payload, bytes, length); movie->payload_size = length; - uint8_t *io_buffer = (uint8_t *)av_malloc(AGE_MOVIE_IO_BUFFER_SIZE); - if (io_buffer == NULL) { - snprintf(movie->error, sizeof(movie->error), "movie IO buffer allocation failed"); - goto failure; - } - movie->io = avio_alloc_context(io_buffer, AGE_MOVIE_IO_BUFFER_SIZE, 0, - movie, read_packet, NULL, seek_payload); - if (movie->io == NULL) { - av_free(io_buffer); - snprintf(movie->error, sizeof(movie->error), "movie AVIO context allocation failed"); - goto failure; - } - - movie->format = avformat_alloc_context(); - if (movie->format == NULL) { - snprintf(movie->error, sizeof(movie->error), "movie format context allocation failed"); - goto failure; - } - movie->format->pb = movie->io; - movie->format->flags |= AVFMT_FLAG_CUSTOM_IO; - int result = avformat_open_input(&movie->format, NULL, NULL, NULL); + int result = open_format_context(movie, &movie->video_io_state, + &movie->video_io, &movie->video_format); if (result < 0) { set_error(movie, "open MPEG program stream", result); goto failure; } - result = avformat_find_stream_info(movie->format, NULL); - if (result < 0) { - set_error(movie, "probe MPEG streams", result); - goto failure; - } + movie->timeline_origin_ms = determine_timeline_origin_ms(movie->video_format); const AVCodec *video_decoder = NULL; - result = av_find_best_stream(movie->format, AVMEDIA_TYPE_VIDEO, -1, -1, &video_decoder, 0); + result = av_find_best_stream(movie->video_format, AVMEDIA_TYPE_VIDEO, -1, -1, &video_decoder, 0); if (result < 0 || video_decoder == NULL) { set_error(movie, "find MPEG video stream", result < 0 ? result : AVERROR_DECODER_NOT_FOUND); goto failure; } movie->video_stream_index = result; - AVStream *stream = movie->format->streams[movie->video_stream_index]; + AVStream *stream = movie->video_format->streams[movie->video_stream_index]; movie->video_time_base = stream->time_base; - movie->timestamp_origin = stream->start_time; movie->video_codec = avcodec_alloc_context3(video_decoder); if (movie->video_codec == NULL) { @@ -316,10 +405,10 @@ int32_t AGE_MOVIE_CALL age_movie_open(const uint8_t *bytes, size_t length, goto failure; } - movie->packet = av_packet_alloc(); - movie->frame = av_frame_alloc(); + movie->video_packet = av_packet_alloc(); + movie->video_frame = av_frame_alloc(); movie->rgba_frame = av_frame_alloc(); - if (movie->packet == NULL || movie->frame == NULL || movie->rgba_frame == NULL) { + if (movie->video_packet == NULL || movie->video_frame == NULL || movie->rgba_frame == NULL) { snprintf(movie->error, sizeof(movie->error), "video packet/frame allocation failed"); goto failure; } @@ -332,17 +421,86 @@ int32_t AGE_MOVIE_CALL age_movie_open(const uint8_t *bytes, size_t length, goto failure; } - AVRational frame_rate = av_guess_frame_rate(movie->format, stream, NULL); + const AVCodec *audio_decoder = NULL; + result = av_find_best_stream(movie->video_format, AVMEDIA_TYPE_AUDIO, -1, -1, &audio_decoder, 0); + if (result >= 0 && audio_decoder != NULL) { + result = open_format_context(movie, &movie->audio_io_state, + &movie->audio_io, &movie->audio_format); + if (result < 0) { + set_error(movie, "open MPEG audio program stream", result); + goto failure; + } + result = av_find_best_stream(movie->audio_format, AVMEDIA_TYPE_AUDIO, + -1, -1, &audio_decoder, 0); + if (result < 0 || audio_decoder == NULL) { + set_error(movie, "find MPEG audio stream", + result < 0 ? result : AVERROR_DECODER_NOT_FOUND); + goto failure; + } + movie->audio_stream_index = result; + AVStream *audio_stream = movie->audio_format->streams[movie->audio_stream_index]; + movie->audio_time_base = audio_stream->time_base; + movie->audio_codec = avcodec_alloc_context3(audio_decoder); + if (movie->audio_codec == NULL) { + snprintf(movie->error, sizeof(movie->error), "audio decoder allocation failed"); + goto failure; + } + result = avcodec_parameters_to_context(movie->audio_codec, audio_stream->codecpar); + if (result < 0) { + set_error(movie, "copy audio parameters", result); + goto failure; + } + result = avcodec_open2(movie->audio_codec, audio_decoder, NULL); + if (result < 0) { + set_error(movie, "open MPEG audio decoder", result); + goto failure; + } + movie->audio_sample_rate = movie->audio_codec->sample_rate; + if (movie->audio_sample_rate <= 0) { + snprintf(movie->error, sizeof(movie->error), "invalid audio sample rate %d", + movie->audio_sample_rate); + goto failure; + } + AVChannelLayout stereo_layout = AV_CHANNEL_LAYOUT_STEREO; + result = swr_alloc_set_opts2(&movie->swr, + &stereo_layout, AV_SAMPLE_FMT_FLT, movie->audio_sample_rate, + &movie->audio_codec->ch_layout, movie->audio_codec->sample_fmt, + movie->audio_sample_rate, 0, NULL); + if (result < 0 || movie->swr == NULL) { + set_error(movie, "configure stereo PCM conversion", + result < 0 ? result : AVERROR(ENOMEM)); + goto failure; + } + result = swr_init(movie->swr); + if (result < 0) { + set_error(movie, "initialize stereo PCM conversion", result); + goto failure; + } + result = rewind_audio_for_decode(movie, audio_stream); + if (result < 0) { + set_error(movie, "rewind MPEG audio stream", result); + goto failure; + } + movie->audio_packet = av_packet_alloc(); + movie->audio_frame = av_frame_alloc(); + if (movie->audio_packet == NULL || movie->audio_frame == NULL) { + snprintf(movie->error, sizeof(movie->error), "audio packet/frame allocation failed"); + goto failure; + } + } + + AVRational frame_rate = av_guess_frame_rate(movie->video_format, stream, NULL); out_info->width = movie->width; out_info->height = movie->height; out_info->stop_time_ms = duration_ms; out_info->frame_rate_num = frame_rate.num; out_info->frame_rate_den = frame_rate.den; - for (unsigned int index = 0; index < movie->format->nb_streams; index++) { - if (movie->format->streams[index]->codecpar->codec_type == AVMEDIA_TYPE_AUDIO) { - out_info->has_audio = 1; - break; - } + if (movie->audio_stream_index >= 0) { + out_info->has_audio = 1; + out_info->audio_sample_rate = movie->audio_sample_rate; + out_info->audio_channels = 2; + out_info->audio_frame_samples = movie->audio_codec->frame_size > 0 + ? movie->audio_codec->frame_size : 4096; } *out_movie = movie; copy_error(error_buffer, error_buffer_size, ""); @@ -359,13 +517,14 @@ int32_t AGE_MOVIE_CALL age_movie_decode_video(age_movie *movie, if (movie == NULL || rgba == NULL || out_pts_ms == NULL) return AGE_MOVIE_INVALID_ARGUMENT; size_t required = (size_t)movie->width * (size_t)movie->height * 4u; if (rgba_size < required) return AGE_MOVIE_BUFFER_TOO_SMALL; - if (movie->decoder_eof) return AGE_MOVIE_EOF; + if (movie->video_decoder_eof) return AGE_MOVIE_EOF; for (;;) { - int result = avcodec_receive_frame(movie->video_codec, movie->frame); + int result = avcodec_receive_frame(movie->video_codec, movie->video_frame); if (result == 0) { movie->sws = sws_getCachedContext(movie->sws, - movie->frame->width, movie->frame->height, (enum AVPixelFormat)movie->frame->format, + movie->video_frame->width, movie->video_frame->height, + (enum AVPixelFormat)movie->video_frame->format, movie->width, movie->height, AV_PIX_FMT_RGBA, SWS_BILINEAR, NULL, NULL, NULL); if (movie->sws == NULL) { @@ -378,8 +537,8 @@ int32_t AGE_MOVIE_CALL age_movie_decode_video(age_movie *movie, return AGE_MOVIE_ERROR; } result = sws_scale(movie->sws, - (const uint8_t * const *)movie->frame->data, movie->frame->linesize, - 0, movie->frame->height, movie->rgba_frame->data, movie->rgba_frame->linesize); + (const uint8_t * const *)movie->video_frame->data, movie->video_frame->linesize, + 0, movie->video_frame->height, movie->rgba_frame->data, movie->rgba_frame->linesize); if (result != movie->height) { snprintf(movie->error, sizeof(movie->error), "RGBA conversion returned %d of %d rows", result, movie->height); return AGE_MOVIE_ERROR; @@ -391,33 +550,33 @@ int32_t AGE_MOVIE_CALL age_movie_decode_video(age_movie *movie, row_size); } - int64_t timestamp = movie->frame->best_effort_timestamp; + int64_t timestamp = movie->video_frame->best_effort_timestamp; if (timestamp == AV_NOPTS_VALUE) { - timestamp = movie->frame->pts; + timestamp = movie->video_frame->pts; } if (timestamp == AV_NOPTS_VALUE) { - AVRational rate = av_guess_frame_rate(movie->format, - movie->format->streams[movie->video_stream_index], movie->frame); + AVRational rate = av_guess_frame_rate(movie->video_format, + movie->video_format->streams[movie->video_stream_index], movie->video_frame); if (rate.num <= 0 || rate.den <= 0) { snprintf(movie->error, sizeof(movie->error), "decoded frame has no timestamp or frame rate"); return AGE_MOVIE_ERROR; } - *out_pts_ms = av_rescale_q_rnd(movie->decoded_frame_count, av_inv_q(rate), + int64_t stream_start = movie->video_format->streams[movie->video_stream_index]->start_time; + int64_t start_ms = stream_start == AV_NOPTS_VALUE ? 0 + : rescale_ms_down(stream_start, movie->video_time_base) - movie->timeline_origin_ms; + *out_pts_ms = start_ms + av_rescale_q_rnd(movie->decoded_video_frame_count, av_inv_q(rate), (AVRational){1, 1000}, AV_ROUND_DOWN | AV_ROUND_PASS_MINMAX); } else { - if (movie->timestamp_origin == AV_NOPTS_VALUE) { - if (movie->first_frame_timestamp == AV_NOPTS_VALUE) movie->first_frame_timestamp = timestamp; - movie->timestamp_origin = movie->first_frame_timestamp; - } - int64_t relative = timestamp >= movie->timestamp_origin ? timestamp - movie->timestamp_origin : 0; - *out_pts_ms = rescale_ms_down(relative, movie->video_time_base); + *out_pts_ms = rescale_ms_down(timestamp, movie->video_time_base) + - movie->timeline_origin_ms; } - movie->decoded_frame_count++; - av_frame_unref(movie->frame); + if (*out_pts_ms < 0) *out_pts_ms = 0; + movie->decoded_video_frame_count++; + av_frame_unref(movie->video_frame); return AGE_MOVIE_FRAME; } if (result == AVERROR_EOF) { - movie->decoder_eof = 1; + movie->video_decoder_eof = 1; return AGE_MOVIE_EOF; } if (result != AVERROR(EAGAIN)) { @@ -425,14 +584,14 @@ int32_t AGE_MOVIE_CALL age_movie_decode_video(age_movie *movie, return AGE_MOVIE_ERROR; } - if (movie->input_eof) { - if (!movie->decoder_draining) { + if (movie->video_input_eof) { + if (!movie->video_decoder_draining) { result = avcodec_send_packet(movie->video_codec, NULL); if (result < 0 && result != AVERROR_EOF) { set_error(movie, "flush MPEG-1 video decoder", result); return AGE_MOVIE_ERROR; } - movie->decoder_draining = 1; + movie->video_decoder_draining = 1; continue; } snprintf(movie->error, sizeof(movie->error), "video decoder requested input after drain"); @@ -440,25 +599,25 @@ int32_t AGE_MOVIE_CALL age_movie_decode_video(age_movie *movie, } for (;;) { - result = av_read_frame(movie->format, movie->packet); + result = av_read_frame(movie->video_format, movie->video_packet); if (result == AVERROR_EOF) { - movie->input_eof = 1; + movie->video_input_eof = 1; break; } if (result < 0) { set_error(movie, "read MPEG video packet", result); return AGE_MOVIE_ERROR; } - if (movie->packet->stream_index != movie->video_stream_index) { - av_packet_unref(movie->packet); + if (movie->video_packet->stream_index != movie->video_stream_index) { + av_packet_unref(movie->video_packet); continue; } break; } - if (movie->input_eof) continue; - result = avcodec_send_packet(movie->video_codec, movie->packet); - av_packet_unref(movie->packet); + if (movie->video_input_eof) continue; + result = avcodec_send_packet(movie->video_codec, movie->video_packet); + av_packet_unref(movie->video_packet); if (result < 0) { set_error(movie, "send MPEG video packet", result); return AGE_MOVIE_ERROR; @@ -466,6 +625,112 @@ int32_t AGE_MOVIE_CALL age_movie_decode_video(age_movie *movie, } } +int32_t AGE_MOVIE_CALL age_movie_decode_audio(age_movie *movie, + float *stereo, size_t frame_capacity, int32_t *out_frame_count, int64_t *out_pts_ms) { + if (out_frame_count != NULL) *out_frame_count = 0; + if (movie == NULL || out_frame_count == NULL || out_pts_ms == NULL) + return AGE_MOVIE_INVALID_ARGUMENT; + if (movie->audio_stream_index < 0) return AGE_MOVIE_EOF; + if (movie->audio_decoder_eof) return AGE_MOVIE_EOF; + + for (;;) { + if (movie->audio_frame_pending) { + int required = swr_get_out_samples(movie->swr, movie->audio_frame->nb_samples); + if (required < 0) { + set_error(movie, "size stereo PCM conversion", required); + return AGE_MOVIE_ERROR; + } + *out_frame_count = required; + if (stereo == NULL || frame_capacity < (size_t)required) + return AGE_MOVIE_BUFFER_TOO_SMALL; + + int64_t timestamp = movie->audio_frame->best_effort_timestamp; + if (timestamp == AV_NOPTS_VALUE) timestamp = movie->audio_frame->pts; + if (timestamp == AV_NOPTS_VALUE) { + AVStream *audio_stream = movie->audio_format->streams[movie->audio_stream_index]; + int64_t stream_start = audio_stream->start_time; + int64_t start_ms = stream_start == AV_NOPTS_VALUE ? 0 + : rescale_ms_down(stream_start, movie->audio_time_base) + - movie->timeline_origin_ms; + *out_pts_ms = start_ms + av_rescale_q_rnd(movie->decoded_audio_sample_count, + (AVRational){1, movie->audio_sample_rate}, (AVRational){1, 1000}, + AV_ROUND_DOWN | AV_ROUND_PASS_MINMAX); + } else { + *out_pts_ms = rescale_ms_down(timestamp, movie->audio_time_base) + - movie->timeline_origin_ms; + } + if (*out_pts_ms < 0) *out_pts_ms = 0; + + uint8_t *output[1] = {(uint8_t *)stereo}; + int converted = swr_convert(movie->swr, output, (int)frame_capacity, + (const uint8_t **)movie->audio_frame->extended_data, + movie->audio_frame->nb_samples); + if (converted < 0) { + set_error(movie, "convert MPEG audio to stereo PCM", converted); + return AGE_MOVIE_ERROR; + } + *out_frame_count = converted; + movie->decoded_audio_sample_count += converted; + movie->audio_frame_pending = 0; + av_frame_unref(movie->audio_frame); + return AGE_MOVIE_FRAME; + } + + int result = avcodec_receive_frame(movie->audio_codec, movie->audio_frame); + if (result == 0) { + movie->audio_frame_pending = 1; + continue; + } + if (result == AVERROR_EOF) { + movie->audio_decoder_eof = 1; + return AGE_MOVIE_EOF; + } + if (result != AVERROR(EAGAIN)) { + set_error(movie, "receive decoded audio frame", result); + return AGE_MOVIE_ERROR; + } + + if (movie->audio_input_eof) { + if (!movie->audio_decoder_draining) { + result = avcodec_send_packet(movie->audio_codec, NULL); + if (result < 0 && result != AVERROR_EOF) { + set_error(movie, "flush MPEG audio decoder", result); + return AGE_MOVIE_ERROR; + } + movie->audio_decoder_draining = 1; + continue; + } + snprintf(movie->error, sizeof(movie->error), "audio decoder requested input after drain"); + return AGE_MOVIE_ERROR; + } + + for (;;) { + result = av_read_frame(movie->audio_format, movie->audio_packet); + if (result == AVERROR_EOF) { + movie->audio_input_eof = 1; + break; + } + if (result < 0) { + set_error(movie, "read MPEG audio packet", result); + return AGE_MOVIE_ERROR; + } + if (movie->audio_packet->stream_index != movie->audio_stream_index) { + av_packet_unref(movie->audio_packet); + continue; + } + break; + } + + if (movie->audio_input_eof) continue; + result = avcodec_send_packet(movie->audio_codec, movie->audio_packet); + av_packet_unref(movie->audio_packet); + if (result < 0) { + set_error(movie, "send MPEG audio packet", result); + return AGE_MOVIE_ERROR; + } + } +} + const char *AGE_MOVIE_CALL age_movie_last_error(const age_movie *movie) { return movie != NULL ? movie->error : "movie handle is null"; } diff --git a/native/age_movie_ffmpeg/age_movie.h b/native/age_movie_ffmpeg/age_movie.h index 4b530c2..25f5399 100644 --- a/native/age_movie_ffmpeg/age_movie.h +++ b/native/age_movie_ffmpeg/age_movie.h @@ -25,6 +25,9 @@ typedef struct age_movie_info { int32_t frame_rate_num; int32_t frame_rate_den; int32_t has_audio; + int32_t audio_sample_rate; + int32_t audio_channels; + int32_t audio_frame_samples; } age_movie_info; enum age_movie_result { @@ -53,6 +56,17 @@ AGE_MOVIE_API int32_t AGE_MOVIE_CALL age_movie_decode_video( size_t rgba_size, int64_t *out_pts_ms); +/* + * Writes one interleaved stereo float32 PCM block and its presentation timestamp. + * out_frame_count is also populated with the required frame count when the buffer is too small. + */ +AGE_MOVIE_API int32_t AGE_MOVIE_CALL age_movie_decode_audio( + age_movie *movie, + float *stereo, + size_t frame_capacity, + int32_t *out_frame_count, + int64_t *out_pts_ms); + AGE_MOVIE_API const char *AGE_MOVIE_CALL age_movie_last_error(const age_movie *movie); AGE_MOVIE_API void AGE_MOVIE_CALL age_movie_close(age_movie *movie); diff --git a/native/age_movie_ffmpeg/build-win64.ps1 b/native/age_movie_ffmpeg/build-win64.ps1 index a31920d..6cb3b98 100644 --- a/native/age_movie_ffmpeg/build-win64.ps1 +++ b/native/age_movie_ffmpeg/build-win64.ps1 @@ -32,7 +32,7 @@ $source = Join-Path $PSScriptRoot 'age_movie.c' $object = Join-Path $OutputDirectory 'age_movie.obj' $dll = Join-Path $OutputDirectory 'age_movie_ffmpeg.dll' $importLibrary = Join-Path $OutputDirectory 'age_movie_ffmpeg.lib' -$compile = 'call "{0}" && cl.exe /nologo /std:c11 /utf-8 /O2 /MD /W4 /external:I"{1}" /external:W0 /LD /Fo"{2}" "{3}" /link /OUT:"{4}" /IMPLIB:"{5}" /LIBPATH:"{6}" avformat.lib avcodec.lib avutil.lib swscale.lib' -f +$compile = 'call "{0}" && cl.exe /nologo /std:c11 /utf-8 /O2 /MD /W4 /external:I"{1}" /external:W0 /LD /Fo"{2}" "{3}" /link /OUT:"{4}" /IMPLIB:"{5}" /LIBPATH:"{6}" avformat.lib avcodec.lib avutil.lib swscale.lib swresample.lib' -f $vcvars, $include, $object, $source, $dll, $importLibrary, $lib & cmd.exe /d /s /c $compile if ($LASTEXITCODE -ne 0) { throw "age_movie_ffmpeg compilation failed with exit code $LASTEXITCODE" } diff --git a/tools/movie-corpus-gate/MovieCorpusGate.cs b/tools/movie-corpus-gate/MovieCorpusGate.cs index 21420b5..cd1dd94 100644 --- a/tools/movie-corpus-gate/MovieCorpusGate.cs +++ b/tools/movie-corpus-gate/MovieCorpusGate.cs @@ -16,6 +16,13 @@ internal sealed record MovieCorpusItemResult( int FrameRateNumerator, int FrameRateDenominator, bool HasAudio, + int AudioSampleRate, + int AudioChannels, + long AudioBlockCount, + long AudioFrameCount, + long FirstAudioPresentationTimeMs, + long LastAudioPresentationTimeMs, + bool AudioHasSignal, long FrameCount, long FirstPresentationTimeMs, long LastPresentationTimeMs, @@ -23,6 +30,7 @@ internal sealed record MovieCorpusItemResult( long ReadMilliseconds, long OpenMilliseconds, long DecodeMilliseconds, + long AudioDecodeMilliseconds, long DisposeMilliseconds, bool Passed, string? Error); @@ -123,7 +131,10 @@ internal sealed class MovieCorpusGate long stopTimeMs = 0, frameCount = 0, firstPts = -1, lastPts = -1; int frameRateNumerator = 0, frameRateDenominator = 0; bool hasAudio = false, framesChanged = false; - long readMs = 0, openMs = 0, decodeMs = 0, disposeMs = 0; + int audioSampleRate = 0, audioChannels = 0; + long audioBlockCount = 0, audioFrameCount = 0, firstAudioPts = -1, lastAudioPts = -1; + bool audioHasSignal = false; + long readMs = 0, openMs = 0, decodeMs = 0, audioDecodeMs = 0, disposeMs = 0; string? error = null; IFfmpegFrameSource? source = null; var itemTime = Stopwatch.StartNew(); @@ -149,6 +160,8 @@ internal sealed class MovieCorpusGate frameRateNumerator = info.FrameRateNumerator; frameRateDenominator = info.FrameRateDenominator; hasAudio = info.HasAudio; + audioSampleRate = info.AudioSampleRate; + audioChannels = info.AudioChannels; if (width != expectedWidth || height != expectedHeight) throw new InvalidDataException( $"decoder dimensions {width}x{height} differ from MPEG sequence {expectedWidth}x{expectedHeight}"); @@ -194,6 +207,53 @@ internal sealed class MovieCorpusGate decodeMs = decodeTime.ElapsedMilliseconds; } if (frameCount == 0) throw new InvalidDataException("decoder reached EOF without a video frame"); + + var audioTime = Stopwatch.StartNew(); + try + { + if (hasAudio) + { + if (audioSampleRate <= 0 || audioChannels != 2 || info.AudioFrameSamples <= 0) + throw new InvalidDataException( + $"invalid audio metadata {audioSampleRate} Hz, {audioChannels} channels, " + + $"{info.AudioFrameSamples} frame capacity"); + while (source.TryDecodeNextAudioChunk(out FfmpegAudioChunk chunk)) + { + if (chunk.FrameCount <= 0 + || chunk.InterleavedStereo.Length != checked(chunk.FrameCount * 2)) + throw new InvalidDataException( + $"audio block {audioBlockCount} has invalid shape " + + $"{chunk.FrameCount}f/{chunk.InterleavedStereo.Length} samples"); + if (chunk.PresentationTimeMs < 0 || chunk.PresentationTimeMs < lastAudioPts) + throw new InvalidDataException( + $"audio block {audioBlockCount} timestamp {chunk.PresentationTimeMs} follows {lastAudioPts}"); + foreach (float sample in chunk.InterleavedStereo) + { + if (!float.IsFinite(sample)) + throw new InvalidDataException( + $"audio block {audioBlockCount} contains non-finite PCM"); + if (Math.Abs(sample) > 0.00001f) audioHasSignal = true; + } + if (audioBlockCount == 0) firstAudioPts = chunk.PresentationTimeMs; + lastAudioPts = chunk.PresentationTimeMs; + audioBlockCount++; + audioFrameCount += chunk.FrameCount; + if (itemTime.ElapsedMilliseconds > maximumItemMilliseconds) + throw new TimeoutException( + $"item exceeded {maximumItemMilliseconds} ms before audio EOF"); + } + if (audioBlockCount == 0) + throw new InvalidDataException("audio stream reached EOF without a PCM block"); + } + else if (audioSampleRate != 0 || audioChannels != 0 + || source.TryDecodeNextAudioChunk(out _)) + throw new InvalidDataException("video-only stream exposed unexpected audio metadata or PCM"); + } + finally + { + audioTime.Stop(); + audioDecodeMs = audioTime.ElapsedMilliseconds; + } } catch (Exception exception) { @@ -221,8 +281,11 @@ internal sealed class MovieCorpusGate return new MovieCorpusItemResult( input.PackedId, $"0x{input.PackedId:x}", input.Name, payloadBytes, expectedWidth, expectedHeight, width, height, stopTimeMs, - frameRateNumerator, frameRateDenominator, hasAudio, frameCount, firstPts, lastPts, - framesChanged, readMs, openMs, decodeMs, disposeMs, error == null, error); + frameRateNumerator, frameRateDenominator, hasAudio, + audioSampleRate, audioChannels, audioBlockCount, audioFrameCount, + firstAudioPts, lastAudioPts, audioHasSignal, + frameCount, firstPts, lastPts, framesChanged, + readMs, openMs, decodeMs, audioDecodeMs, disposeMs, error == null, error); } } diff --git a/vm-map/opcodes.toml b/vm-map/opcodes.toml index 3f8529e..79e0b0d 100644 --- a/vm-map/opcodes.toml +++ b/vm-map/opcodes.toml @@ -5256,7 +5256,7 @@ source = "investigation" confidence = "high" depends_on = [] evidence = "Ghidra /v2: op_0x20f_play_modal_movie_to_surface@0x422e50 shares the movie-object allocation, DirectShow graph open, audio-route, and volume setup used by 0x236, then calls movie_start_modal_playback@0x463280, ORs EngineCtx+0xa0ce4 with 0x2000, and marks movie presentation dirty. The main loop and window procedure special-case run-state 0x2000. Corpus has exactly three sites: LOGO (0x335f,42,4), OP (0x3364,42,4), and ED (0x3324,42,dynamic flags). Those ids are universal raw SYS4INI indexes for MPEG-pack LOGO.AGF, OP.AGF, and ED.AGF; each script releases its surface only after 0x20f resumes." -details = "Implemented through IHost.PlayModalMovieToSurface. Its operand uses the same native universal packed-id catalog contract as 0x236; the separate host call exists for modal wait/cancel lifecycle, not a different resolver. ResourceMap.ResolveMovie selects through ResolvePacked and retains MPEG signature validation in ReadMovie. Godot reuses the asynchronous decoder/retained-surface compositor, parks only the VM thread until EOF, and treats mouse click or Accept/Cancel input as completion before wrapper cleanup releases the decoder. MPEG audio remains a separate backend/audio-clock contract." +details = "Implemented through IHost.PlayModalMovieToSurface. Its operand uses the same native universal packed-id catalog contract as 0x236; the separate host call exists for modal wait/cancel lifecycle, not a different resolver. ResourceMap.ResolveMovie selects through ResolvePacked and retains MPEG signature validation in ReadMovie. Godot reuses the asynchronous FFmpeg decoder/retained-surface compositor, parks only the VM thread until EOF, and treats mouse click or Accept/Cancel input as completion before wrapper cleanup releases both video and per-playback audio output. Audio-bearing movies deliver timestamped stereo float PCM through AudioStreamGenerator and use the sound-hardware position as the video/master clock; video-only movies retain monotonic-clock pacing." [[opcode.semantics.args]] i = 1 @@ -6247,7 +6247,7 @@ source = "investigation" confidence = "high" depends_on = [] evidence = "Ghidra /v2 op_0x236_play_movie_to_surface@0x423ee0 fetches operand 1 and passes it unchanged to movie_to_texture_open_asset_graph@0x463e20, which passes it unchanged to asset_open_indexed_entry@0x44f390. The opener directly indexes the flat base table or selected AAI table and has no scene input. SC0000 native operand capture and exact 0x13c8->0x13d1 trace prove nonblocking behavior. BTL's live 0x2b21 site supplies 0x2af1/0x2af5/0x2bca/0x2bd8/0x2bde, the exact base entries MVB001/MVB004/MVB914/MVB958/MVB955." -details = "The handler requires an existing destination texture, allocates/reuses a 0x478-byte movie-to-texture object for the surface, opens operand 1 through the native universal packed-id reader, builds FilterGraph/IGraphBuilder/IMediaControl/IMediaPosition/IMediaEvent/IBasicAudio, and presents bottom-up RGB samples through the movie texture renderer. Operand 3 selects movie/sound routing policy: bits 0x10000/0x20000/0x40000/0x80000 force sound route 0/1/2/3, otherwise set:DependMovieSound is used; SC0000's low value 2 is retained as native movie mode state. Operand 4 is the movie sync/device mask. Static layer preparation after 0x236 does not terminate the retained movie; 0x21c services it through EOF and subsequent surface cleanup stops/detaches it. The port type-checks the selected ResolvePacked record as MPEG without adding a scene base or fallback." +details = "The handler requires an existing destination texture, allocates/reuses a 0x478-byte movie-to-texture object for the surface, opens operand 1 through the native universal packed-id reader, builds FilterGraph/IGraphBuilder/IMediaControl/IMediaPosition/IMediaEvent/IBasicAudio, and presents bottom-up RGB samples through the movie texture renderer. Operand 3 selects movie/sound routing policy: bits 0x10000/0x20000/0x40000/0x80000 force mute/music/SE/voice routes, otherwise set:DependMovieSound selects the normal movie route; SC0000's low value 2 is retained as native movie mode state. Godot maps those categories to audio buses and sends FFmpeg-decoded timestamped stereo float PCM through a per-playback AudioStreamGenerator. Audio-bearing movies use the sound-hardware position as the master presentation clock; video-only movies retain monotonic pacing. Operand 4 is the movie sync/device mask. Static layer preparation after 0x236 does not terminate the retained movie; 0x21c services it through EOF and subsequent surface cleanup stops/detaches both decoder and audio output. The port type-checks the selected ResolvePacked record as MPEG without adding a scene base or fallback." [[opcode.semantics.args]] i = 1