Add synchronized MPEG movie audio
This commit is contained in:
@@ -351,8 +351,8 @@ until DirectShow EOF, then scene cleanup stops the movie. The initial invisible
|
||||
the static `(assetId,colorKey)` image cache froze the first movie sample, while an extra current-sample
|
||||
background copy was covered by the correctly positioned retained movie object. Dynamic movie surfaces now
|
||||
bypass that cache and publish only at their retained z-position. A windowed run reached first frame 101 and
|
||||
stop frame 190, and manual observation confirmed visible changing video. Movie audio remains intentionally
|
||||
unrendered. The real-scene trace remains a separate extracted-present test
|
||||
stop frame 190, and manual observation confirmed visible changing video. That milestone left movie audio
|
||||
unrendered; the synchronized FFmpeg/Godot audio path landed later as described below. The real-scene trace remains a separate extracted-present test
|
||||
because the current `Paths.Scripts()` test bootstrap still locates its root `*.BIN` fixtures there; migrating
|
||||
that test/bootstrap path is unrelated to movie asset loading and was not folded into this slice.
|
||||
|
||||
@@ -410,7 +410,9 @@ reported combat stall survives this guard, capture the VM/service coordinate aft
|
||||
a separate BTL timed-presentation bug.
|
||||
|
||||
The FFmpeg replacement and destination-surface work are recorded in `docs/platform-portability.md` and
|
||||
`docs/phase-b-framework.md`. Movie identity also remains typed across the entire retained-surface lifecycle.
|
||||
`docs/phase-b-framework.md`. After video/audio corpus validation and clean audible LOGO/OP/CHAPTER acceptance,
|
||||
the DirectShow/temp-file adapter was deleted; packed MPEG decode now has one runtime path. Movie identity also
|
||||
remains typed across the entire retained-surface lifecycle.
|
||||
The packed catalog is immutable while mounted, so once an id enters the movie path its `.AGF`-named MPEG
|
||||
payload cannot fall through to the ACGF still decoder before the first frame or after the last live binding
|
||||
is detached. The latter guard closes a real cross-thread cleanup window: BTL clears `GfxState` and releases
|
||||
@@ -430,10 +432,12 @@ depend on the movie service itself to park until EOF/input cancellation before t
|
||||
|
||||
`ResourceMap.ResolveMovie` supplies that typed universal lookup, while `ReadMovie` remains the MPEG
|
||||
signature gate. `IHost.PlayModalMovieToSurface` is distinct from the non-modal call for lifecycle only: Godot
|
||||
reuses the asynchronous DirectShow frame decoder and retained compositor but parks the VM thread until EOF
|
||||
reuses the asynchronous FFmpeg decoder and retained compositor but parks the VM thread until EOF
|
||||
or mouse/Accept/Cancel input. The wrapper's following release then tears down the completed/cancelled movie.
|
||||
The decoder still intentionally leaves audio unrendered; OP/ED audio parity needs an explicit synchronized
|
||||
movie-audio/backend contract rather than an unmanaged default-device side path.
|
||||
The FFmpeg backend now renders embedded audio through an engine-owned synchronized path rather than an
|
||||
unmanaged default-device side path. Timestamped stereo float PCM feeds a per-playback Godot
|
||||
`AudioStreamGenerator`; the sound-hardware position drives video presentation, and modal cancellation or
|
||||
surface cleanup releases both sides together.
|
||||
|
||||
VFS-B passes its bounded gates in `Sys4AssetStoreTests`: the installed AAI expands from the LZSS stream at
|
||||
`0x118` (expanded size at `0x110`, packed size at `0x114`) to one `APPEND01.ALF` archive and 81 80-byte
|
||||
@@ -446,8 +450,8 @@ through `0x01xxxxxx`; direct base-name lookup deliberately does not see append r
|
||||
### Deliberate non-goals
|
||||
|
||||
- Writing/repacking ALF or AAI; loose overrides already provide the native mod/translation workflow.
|
||||
- AGF encoding or movie audio. A decoder interface/software MPEG fallback is now required for the installed
|
||||
non-16-aligned MVB effects; synchronized audio remains a separate slice.
|
||||
- AGF/MPEG encoding. Runtime MPEG decode and synchronized movie audio now live behind the portable FFmpeg
|
||||
boundary documented in `docs/platform-portability.md`; authoring or repacking those streams remains out of scope.
|
||||
- A generalized multi-mod dependency manager. Start with native game-root loose overrides; configurable
|
||||
ordered mod roots can be layered onto the same store later.
|
||||
- Removing the extraction/conversion tools immediately. They remain independent parity oracles until the
|
||||
|
||||
@@ -362,8 +362,9 @@ The port implements `0x20f` through a typed packed-movie resolver and a distinct
|
||||
asynchronous decoder publishes frames through the retained surface while only the VM thread is parked;
|
||||
EOF, mouse click, Accept, or Cancel resumes the wrapper so its scripted cleanup releases surface 42.
|
||||
`0x236` retains its non-modal contract; native passes its packed resource operand through the same universal
|
||||
catalog opener as `0x20f`. Both paths still intentionally leave the MPEG audio
|
||||
pin unrendered; synchronized movie audio remains a deliberate backend/audio-clock slice.
|
||||
catalog opener as `0x20f`. Both paths now decode the MPEG audio pin through FFmpeg into timestamped stereo
|
||||
float PCM. Godot owns one `AudioStreamGenerator` per playback instance, applies the native flag-selected
|
||||
mute/music/SE/voice or default movie route, and uses the sound-hardware position as the presentation clock.
|
||||
|
||||
An existing native operand trace identifies every observed heap codebase by a 100% match against its static
|
||||
instruction-offset set. The captured New Game route is:
|
||||
@@ -1279,8 +1280,8 @@ queries `IGraphBuilder`, `IMediaControl`, `IMediaPosition`, `IMediaEvent`, and `
|
||||
`CMovieTextureRenderer` accepts RGB samples and copies the bottom-up frame into the retained texture.
|
||||
|
||||
Operand 3 is retained movie mode plus sound-route policy. Bits `0x10000/0x20000/0x40000/0x80000` force
|
||||
sound route 0/1/2/3; without an override, native setting `set:DependMovieSound` supplies the route. This
|
||||
slice deliberately does not implement the audio branch. Operand 4 is stored at movie object `+0x42c` as
|
||||
mute/music/SE/voice routing; without an override, native setting `set:DependMovieSound` supplies the normal
|
||||
movie route. The port applies those categories through Godot audio buses. Operand 4 is stored at movie object `+0x42c` as
|
||||
the sync/device mask; it is not a duration or loop count. Replacing or releasing the owning surface stops
|
||||
the graph and detaches the renderer.
|
||||
|
||||
@@ -1300,7 +1301,8 @@ because every sample has the same `(assetId,colorKey)`. An unconditional current
|
||||
not help because normal retained composition immediately covered it with that cached first sample. Movie
|
||||
surfaces now bypass the static cache and publish only through the retained object. A windowed run records
|
||||
first frame 101 and stop frame 190; the user manually confirmed visible playback. The separate lower white
|
||||
textbox-area object remains outside this finding, as does the deliberately unrendered movie audio stream.
|
||||
textbox-area object remains outside this finding. Embedded audio was added later through the portable
|
||||
FFmpeg/audio-clock path described in `docs/platform-portability.md`.
|
||||
|
||||
The `/v2` image names/comments the handler; movie ctor/interface/open/play/volume/release workers; sound
|
||||
route helpers; renderer media-type/sample workers; and stop/detach/destructor lifecycle. The image was
|
||||
@@ -1351,6 +1353,12 @@ reported stop time plus two seconds, maximum five minutes; 30 seconds without ti
|
||||
hold the shared `0x21c` wait indefinitely. A normal Game Start through SC0000 was manually validated with
|
||||
ordinary positive timing.
|
||||
|
||||
**Portable replacement closeout (2026-07-25):** the synchronous metadata, asynchronous presentation,
|
||||
failure-as-completed, and watchdog contracts above remain, but `FfmpegMovieDecoder` is now the sole backend.
|
||||
After the 213/213 video/audio corpus gate and clean audible LOGO/OP/CHAPTER acceptance, the port deleted
|
||||
`DirectShowMovieDecoder`, its COM/temp-file adapter, compatibility test, and managed Windows annotations.
|
||||
Native AGE's DirectShow behavior remains relevant evidence for opcode semantics; it is no longer port code.
|
||||
|
||||
### Grey-background root cause — slot collision + tint-strength (2026-07-08, gfx-log)
|
||||
|
||||
Diagnosed with the new `--gfx-log` compositor/op trace (docs/tools-reference.md). The grey background has
|
||||
|
||||
@@ -726,7 +726,7 @@ This is a target-pixel operation, not retained-object teardown. It invokes IDire
|
||||
- **grounding:** source=investigation, confidence=high
|
||||
- **evidence:** Ghidra /v2: op_0x20f_play_modal_movie_to_surface@0x422e50 shares the movie-object allocation, DirectShow graph open, audio-route, and volume setup used by 0x236, then calls movie_start_modal_playback@0x463280, ORs EngineCtx+0xa0ce4 with 0x2000, and marks movie presentation dirty. The main loop and window procedure special-case run-state 0x2000. Corpus has exactly three sites: LOGO (0x335f,42,4), OP (0x3364,42,4), and ED (0x3324,42,dynamic flags). Those ids are universal raw SYS4INI indexes for MPEG-pack LOGO.AGF, OP.AGF, and ED.AGF; each script releases its surface only after 0x20f resumes.
|
||||
|
||||
Implemented through IHost.PlayModalMovieToSurface. Its operand uses the same native universal packed-id catalog contract as 0x236; the separate host call exists for modal wait/cancel lifecycle, not a different resolver. ResourceMap.ResolveMovie selects through ResolvePacked and retains MPEG signature validation in ReadMovie. Godot reuses the asynchronous decoder/retained-surface compositor, parks only the VM thread until EOF, and treats mouse click or Accept/Cancel input as completion before wrapper cleanup releases the decoder. MPEG audio remains a separate backend/audio-clock contract.
|
||||
Implemented through IHost.PlayModalMovieToSurface. Its operand uses the same native universal packed-id catalog contract as 0x236; the separate host call exists for modal wait/cancel lifecycle, not a different resolver. ResourceMap.ResolveMovie selects through ResolvePacked and retains MPEG signature validation in ReadMovie. Godot reuses the asynchronous FFmpeg decoder/retained-surface compositor, parks only the VM thread until EOF, and treats mouse click or Accept/Cancel input as completion before wrapper cleanup releases both video and per-playback audio output. Audio-bearing movies deliver timestamped stereo float PCM through AudioStreamGenerator and use the sound-hardware position as the video/master clock; video-only movies retain monotonic-clock pacing.
|
||||
|
||||
### 0x212 `set-gfx-field64` (set-gfx-field64, argc 2)
|
||||
- **summary:** 0x212 (obj_idx)(val) — handler gfx_op_0x212_set_field64 @0x4230c0: obj=[ctx+0x14d54 + obj_idx*4]; if obj: *(obj+0x64)=val. The generic instruction length is 5 dwords. See docs/engine-re.md gfx op-contract table.
|
||||
@@ -856,7 +856,7 @@ Implemented through IHost.PlayModalMovieToSurface. Its operand uses the same nat
|
||||
- **grounding:** source=investigation, confidence=high
|
||||
- **evidence:** Ghidra /v2 op_0x236_play_movie_to_surface@0x423ee0 fetches operand 1 and passes it unchanged to movie_to_texture_open_asset_graph@0x463e20, which passes it unchanged to asset_open_indexed_entry@0x44f390. The opener directly indexes the flat base table or selected AAI table and has no scene input. SC0000 native operand capture and exact 0x13c8->0x13d1 trace prove nonblocking behavior. BTL's live 0x2b21 site supplies 0x2af1/0x2af5/0x2bca/0x2bd8/0x2bde, the exact base entries MVB001/MVB004/MVB914/MVB958/MVB955.
|
||||
|
||||
The handler requires an existing destination texture, allocates/reuses a 0x478-byte movie-to-texture object for the surface, opens operand 1 through the native universal packed-id reader, builds FilterGraph/IGraphBuilder/IMediaControl/IMediaPosition/IMediaEvent/IBasicAudio, and presents bottom-up RGB samples through the movie texture renderer. Operand 3 selects movie/sound routing policy: bits 0x10000/0x20000/0x40000/0x80000 force sound route 0/1/2/3, otherwise set:DependMovieSound is used; SC0000's low value 2 is retained as native movie mode state. Operand 4 is the movie sync/device mask. Static layer preparation after 0x236 does not terminate the retained movie; 0x21c services it through EOF and subsequent surface cleanup stops/detaches it. The port type-checks the selected ResolvePacked record as MPEG without adding a scene base or fallback.
|
||||
The handler requires an existing destination texture, allocates/reuses a 0x478-byte movie-to-texture object for the surface, opens operand 1 through the native universal packed-id reader, builds FilterGraph/IGraphBuilder/IMediaControl/IMediaPosition/IMediaEvent/IBasicAudio, and presents bottom-up RGB samples through the movie texture renderer. Operand 3 selects movie/sound routing policy: bits 0x10000/0x20000/0x40000/0x80000 force mute/music/SE/voice routes, otherwise set:DependMovieSound selects the normal movie route; SC0000's low value 2 is retained as native movie mode state. Godot maps those categories to audio buses and sends FFmpeg-decoded timestamped stereo float PCM through a per-playback AudioStreamGenerator. Audio-bearing movies use the sound-hardware position as the master presentation clock; video-only movies retain monotonic pacing. Operand 4 is the movie sync/device mask. Static layer preparation after 0x236 does not terminate the retained movie; 0x21c services it through EOF and subsequent surface cleanup stops/detaches both decoder and audio output. The port type-checks the selected ResolvePacked record as MPEG without adding a scene base or fallback.
|
||||
|
||||
### 0x238 `set-anim-clock` (set-anim-clock, argc 1)
|
||||
- **summary:** (duration) — configure the separate global finite-animation service window: native ctx+0x51b78=0 (start), +0x51b7c=duration. The generic instruction length is 3 dwords. NON-BLOCKING: gfx_animation_service_poll seeds the start from the shared frame timestamp and keeps redraw active through the duration. This is not the time source or cadence for cyclic object channels, which sample ctx+0x51b64/+0x51b68 with object-local starts and periods. SC0000 opening @0x123bd/@0x13858. Handler 0x4240e0; Kelebek VA 0x422390 is drift.
|
||||
|
||||
@@ -127,8 +127,8 @@ explained the direct jump to TITLE. LOGO and OP then use the modal movie op
|
||||
non-modal movie-to-surface path and uses the same packed lookup. The VM now models the initial-root flag and clears
|
||||
it at the op-`0x9` whole-stack root-reload boundary. Godot resolves a typed MPEG asset, reuses the asynchronous decoder
|
||||
and retained compositor, and parks the VM until EOF or mouse/Accept/Cancel input. Focused natural-boot tests
|
||||
prove `LOGO -> OP -> INIT -> TITLE` ordering and exact movie operands. MPEG audio remains explicitly deferred
|
||||
until the decoder abstraction has an engine-owned synchronized audio/volume contract.
|
||||
prove `LOGO -> OP -> INIT -> TITLE` ordering and exact movie operands. MPEG audio was explicitly deferred
|
||||
at this milestone; the engine-owned synchronized audio/volume contract landed on 2026-07-25 as recorded below.
|
||||
|
||||
## Stage B1 — Persistent session and scene coordinator
|
||||
|
||||
@@ -636,8 +636,8 @@ dependency pin, corpus codec inventory, timing contract, and packaging rules liv
|
||||
as a separate battle-timeline defect.
|
||||
6. **Complete distributable packaging.** Bundle pinned dynamically linked libraries and exact license/source
|
||||
provenance per target, validate loader isolation from machine-installed codecs, then add Linux/macOS build and
|
||||
smoke gates. Timestamped PCM delivery and Godot `AudioStreamGenerator` integration remain a later movie-audio
|
||||
slice; the video replacement must merely avoid closing that path.
|
||||
smoke gates. Timestamped PCM delivery and Godot `AudioStreamGenerator` integration were assigned to a later
|
||||
movie-audio slice; that slice landed on 2026-07-25 as recorded below.
|
||||
|
||||
Steps 1 and 2 landed on 2026-07-21 without changing live playback: `Main` owns an
|
||||
`IMovieDecoderFactory`, `MovieRuntime` is backend-neutral, and the default factory still constructs DirectShow.
|
||||
@@ -726,6 +726,42 @@ tests, the zero-warning Godot build, and threaded selftest pass. Manual recheck
|
||||
BUNKI panel and minimap paper are correct. The movement/overlay recheck described above is now the next manual
|
||||
gate on the one-player-attack acceptance path.
|
||||
|
||||
### MPEG movie audio implemented; audible acceptance in progress (2026-07-25)
|
||||
|
||||
The FFmpeg shim ABI is now version 2. Each in-memory MPEG payload feeds independent seekable video and audio
|
||||
demuxers, preserving concurrent decode without temporary files, and exposes timestamped interleaved stereo
|
||||
float PCM resampled at the source rate. Video and audio timestamps share the same normalized media origin.
|
||||
The managed decoder owns bounded video and PCM queues plus separate workers; audio-bearing movies use the
|
||||
Godot audio-device clock as the presentation master, while video-only movies retain monotonic stopwatch pacing.
|
||||
Completion waits for the final video interval and for decoded PCM to be submitted.
|
||||
|
||||
Godot creates a per-playback `AudioStreamGenerator`, compensates for output latency, handles timestamp gaps and
|
||||
overlaps, and tears it down with the corresponding playback instance. Native movie flags route sound to the
|
||||
Movie bus by default, with exact overrides for mute (`0x10000`), Music (`0x20000`), SFX (`0x40000`), and Voice
|
||||
(`0x80000`). F6 diagnostics now expose audio format, decode/submission state, route, clock, and underruns.
|
||||
|
||||
The full installed-corpus gate passes all 213 movies: 29 audio-bearing assets decode 17,537 blocks /
|
||||
18,185,856 stereo PCM frames at 44.1 kHz with nondecreasing timestamps and non-silent signal, while all 184
|
||||
video-only assets remain audio-free. At the initial landing, focused real-stream tests covered MP2 and MP1
|
||||
content, all 404 engine tests passed, the Godot build was warning-free, threaded selftest passed, and a natural
|
||||
headless boot completed LOGO with its audio path active before opening OP. This moved acceptance to audible,
|
||||
synchronized LOGO/OP and representative CHAPTER/MVS playback in a normal windowed run; DirectShow remained
|
||||
unselected but in-tree until that live gate.
|
||||
|
||||
The first normal windowed LOGO/OP run confirmed that audio reaches the intended output, but it sounded
|
||||
crackly/warbled. Corpus endpoints showed continuous decoded PCM; the fault was presentation alignment.
|
||||
Audio PTS crosses ABI v2 in whole milliseconds, losing up to 44 samples of precision at 44.1 kHz, while the
|
||||
initial sink inserted or dropped that tiny discrepancy at every MPEG block boundary. Established timelines
|
||||
now tolerate 2 ms of timestamp quantization and remain sample-contiguous, while initial offsets and material
|
||||
later gaps/overlaps still insert silence or trim PCM. A regression simulates all 4,093 OP blocks without a
|
||||
splice and separately pins real discontinuity handling. The repeat LOGO/OP run and SC0000 CHAPTER playback are
|
||||
audibly clean. Together with signal-bearing decode of all 29 audio streams, this closes the movie-audio acceptance
|
||||
gate: missing, distorted, or unsynchronized audio is now a runtime bug. The unused DirectShow implementation,
|
||||
COM/temp-file adapter, compatibility test, and managed Windows platform annotations are deleted. **NEXT:** add
|
||||
Linux/macOS FFmpeg builds and packaging smoke gates when work returns to cross-platform distribution.
|
||||
Post-cleanup validation passes all 405 engine tests, the warning-free Godot build, threaded selftest, opcode
|
||||
lint/tooling, and the 213/213 installed video/audio corpus gate.
|
||||
|
||||
## Later Phase B breadth
|
||||
|
||||
**INIT data-semantics side track started (2026-07-22).** Before naming more gameplay state, the static
|
||||
|
||||
@@ -18,19 +18,19 @@ The VM and content pipeline are already mostly platform-neutral:
|
||||
- Native-compatible persistence uses managed streams behind `INativeDatStore`; Godot redirects AGE's
|
||||
engine-owned root to `user://SAVE` while retaining its fixed portable DAT/STH filenames.
|
||||
|
||||
The selected movie path now uses the project-owned FFmpeg C ABI rather than a Windows multimedia API, but only a
|
||||
Windows-x64 native bundle is built and staged today. The retired-live DirectShow implementation remains in-tree
|
||||
until the corpus and manual gates pass. There are also softer Windows assumptions that should be tested or
|
||||
replaced before claiming portable exports.
|
||||
The sole movie path now uses the project-owned FFmpeg C ABI rather than a Windows multimedia API, but only a
|
||||
Windows-x64 native bundle is built and staged today. The accepted DirectShow fallback was deleted after the
|
||||
corpus and audible LOGO/OP/CHAPTER gates passed. There are also softer Windows assumptions that should be tested
|
||||
or replaced before claiming portable exports.
|
||||
|
||||
## Dependency inventory
|
||||
|
||||
| Area | Current dependency | Runtime impact | Portability status / future action |
|
||||
|---|---|---|---|
|
||||
| Retained graphics presentation | Backend-neutral `GfxState`; accepted default Godot `Sprite2D` GPU stage plus the retained software pixel oracle, using runtime `ImageTexture`, canvas transforms/materials, and no native graphics API | GPU backend caches static/color-key variants, updates dynamic surfaces, handles retained range transitions, and falls back whole-frame for the legacy host screen-transition path | Godot owns D3D/Vulkan/Metal/OpenGL selection; validate shader/blend/filter behavior per target rather than adding a platform renderer |
|
||||
| AGE movie decode (`0x236` scene movies; `0x20f` modal LOGO/OP/ED) | `FfmpegMovieDecoder` is the selected live factory over the project-owned `native/age_movie_ffmpeg` ABI; `DirectShowMovieDecoder` remains unselected pending deletion | Windows-x64 passes the complete 213-payload installed-corpus gate; other native targets and the remaining windowed CHAPTER/combat checks are pending | Run the windowed live gates, then delete DirectShow and add target-specific native builds |
|
||||
| Movie integration | Each surface owns a unique playback-instance id; `MovieRuntime` owns `IMovieDecoder` from an injected factory, and the FFmpeg worker paces PTS against a monotonic clock with cancellation/failure completion | Concurrent/restarted uses of one asset have independent frame/completion/teardown state; `Main` remains annotated Windows because only the win-x64 bundle is available | Recheck the corrected combat lifecycle, then add Linux/macOS builds and remove the Windows annotation after DirectShow is deleted |
|
||||
| Movie audio | FFmpeg detects the audio stream but the current ABI returns video frames only | MPEG movie audio remains intentionally silent | Extend the ABI with timestamped PCM and select an audio/presentation clock; separate feature slice |
|
||||
| AGE movie decode (`0x236` scene movies; `0x20f` modal LOGO/OP/ED) | `FfmpegMovieDecoder` is the sole factory over the project-owned `native/age_movie_ffmpeg` ABI | Windows-x64 passes the complete 213-payload installed video/audio corpus gate plus audible LOGO/OP/CHAPTER playback | Add target-specific native builds and export packaging |
|
||||
| Movie integration | Each surface owns a unique playback-instance id; `MovieRuntime` owns `IMovieDecoder` from an injected factory; video-only streams use monotonic pacing while audio-bearing streams use the Godot output clock | Concurrent/restarted uses of one asset have independent frame/audio/completion/teardown state; managed code is no longer Windows-annotated, while only the win-x64 native bundle exists today | Add Linux/macOS native builds and smoke gates |
|
||||
| Movie audio | ABI v2 returns timestamped stereo float PCM; bounded managed buffering feeds a per-playback Godot `AudioStreamGenerator` and routes native movie flags to engine buses | All 29 installed audio-bearing streams decode with signal; synchronized LOGO/OP/CHAPTER playback is audibly accepted | Treat absent, distorted, or unsynchronized audio from an audio-bearing movie as a runtime bug |
|
||||
| ADV font discovery | `godot/Main.cs` probes `C:/Windows/Fonts` for Japanese fonts | Harmless fallback today, but appearance depends on host fonts | Bundle/configure a redistributable font or add platform-specific discovery |
|
||||
| Filesystem semantics | Several filename and containment comparisons use `OrdinalIgnoreCase`; installed assets are conventionally uppercase | Needs validation on case-sensitive filesystems; may hide casing or containment mistakes | Add Linux/macOS tests with mixed-case synthetic roots and use filesystem-appropriate containment rules |
|
||||
| Save/profile storage | Managed `DirectoryNativeDatStore` under Godot `user://SAVE`; native S3SD/S4SD/S3RT files and 24-bit BMP thumbnails | No Win32 path API at runtime; port saves remain isolated from the original installation | Validate replace/flush, case, permissions, and interrupted-write behavior on each export target |
|
||||
@@ -53,41 +53,43 @@ VM op 0x236 (non-modal) / op 0x20f (modal)
|
||||
-> IHost.PlayMovieToSurface / PlayModalMovieToSurface
|
||||
-> VFS-owned MoviePayload bytes
|
||||
-> IMovieDecoderFactory
|
||||
-> FfmpegMovieDecoder (current live selection)
|
||||
-> FfmpegMovieDecoder
|
||||
-> FfmpegMovieSession -> age_movie C ABI
|
||||
-> DirectShowMovieDecoder (unselected; retained only through acceptance)
|
||||
-> newest RGBA frame
|
||||
-> retained movie surface
|
||||
-> Godot compositor
|
||||
-> newest due RGBA frame -> retained movie surface -> Godot compositor
|
||||
-> timestamped stereo float PCM -> per-playback AudioStreamGenerator
|
||||
```
|
||||
|
||||
Everything before and after the selected decoder is portable. The replacement decision is an in-process
|
||||
FFmpeg backend behind a project-owned C ABI, not raw FFmpeg structs in Godot/C# and not a subprocess. FFmpeg
|
||||
`n8.1.2-29-g703dcc25b9` is pinned by immutable release URL and SHA-256 in
|
||||
`native/age_movie_ffmpeg/dependency-win64.json`; changing that pin requires rerunning the full installed-movie
|
||||
gate. The shim dynamically links an LGPL build made without GPL or nonfree components and
|
||||
uses only `libavformat`, `libavcodec`, `libavutil`, and `libswscale` for the video slice. `libswresample` and the
|
||||
MPEG audio decoders may be packaged now, but PCM delivery remains a separate slice. Release artifacts must carry
|
||||
gate. The shim dynamically links an LGPL build made without GPL or nonfree components and uses
|
||||
`libavformat`, `libavcodec`, `libavutil`, `libswscale`, and `libswresample`. Release artifacts must carry
|
||||
the matching FFmpeg source/configuration and notices required by FFmpeg's
|
||||
[license checklist](https://ffmpeg.org/legal.html).
|
||||
|
||||
The boundary has two layers:
|
||||
|
||||
1. A small native `age_movie` ABI owns all FFmpeg objects and version-sensitive calls. It accepts a borrowed
|
||||
byte span only for the duration of `open`, copies it into native-owned memory, creates a seekable custom
|
||||
`AVIOContext`, probes the MPEG program stream, and returns immutable video metadata. Sequential decode returns
|
||||
one top-down tightly packed RGBA8 frame plus its normalized presentation timestamp. Conversion occurs in an
|
||||
byte span only for the duration of `open`, copies it once into native-owned memory, and creates independent
|
||||
seekable custom `AVIOContext` views for video and audio so either stream can be pulled without duplicating the
|
||||
payload or sharing a demux cursor. ABI v2 returns immutable video/audio metadata. Sequential video decode
|
||||
returns one top-down tightly packed RGBA8 frame; sequential audio decode returns one interleaved stereo float32
|
||||
PCM block. Both carry timestamps normalized to a common media origin. Video conversion occurs in an
|
||||
FFmpeg-owned aligned frame; only exact visible row bytes are copied into the caller's tightly packed buffer.
|
||||
`libswresample` converts the source channel/sample layout to the Godot generator contract.
|
||||
Return statuses distinguish frame, EOF, invalid arguments, undersized output, and decoder failure; the open
|
||||
call returns a bounded UTF-8 diagnostic and an opened handle retains its last decode error. Close accepts a
|
||||
null handle, and managed `SafeHandle` ownership guarantees one close for each successfully opened handle. No
|
||||
FFmpeg pointer crosses the ABI.
|
||||
2. Managed `IMovieDecoder` owns the native handle and the paced worker. Synchronous construction provides
|
||||
`StopTimeMs` before `0x236` returns. The worker uses a monotonic playback origin, decodes ahead by at most one
|
||||
frame, publishes frames when their timestamps become due, and retains newest-frame-wins behavior if Godot is
|
||||
late. EOF becomes completion only after the final frame's presentation interval/stop time, so surface cleanup
|
||||
cannot erase the last frame immediately. Disposal interrupts waits and joins without depending on Godot's main
|
||||
thread.
|
||||
2. Managed `IMovieDecoder` owns the native handle and cancellable video/audio workers. Synchronous construction
|
||||
provides `StopTimeMs` and audio format before `0x236` returns. Video remains at most one due frame ahead; PCM
|
||||
uses a bounded roughly half-second queue, so the 263-second ending never becomes a whole-file allocation.
|
||||
Video-only movies use the monotonic playback origin. Audio-bearing movies feed a per-instance
|
||||
`AudioStreamGenerator`; the monotonic-clamped Godot sound-hardware position
|
||||
(`playback + time-since-mix - cached output latency`) advances due video frames. Completion requires video
|
||||
stop time, audio decode EOF, and PCM submission. Disposal interrupts waits, joins both workers, and stops the
|
||||
generator without depending on native default-device playback.
|
||||
|
||||
`StopTimeMs` performs a bounded packet scan over the seekable in-memory payload and takes the longest usable
|
||||
FFmpeg format duration, video-stream duration, timestamp span, or constant-frame-rate packet-count duration. This
|
||||
@@ -99,12 +101,11 @@ safety rule still converts any backend initialization failure into an
|
||||
explicitly completed zero-duration movie identity, and the presentation watchdog remains a last-resort guard for
|
||||
an initialized backend that never reaches EOF.
|
||||
|
||||
The factory is injection for tests and future decoder replacement, not runtime codec roulette. FFmpeg is now the
|
||||
selected Windows-x64 live backend; once its corpus and live gates pass, DirectShow is deleted rather than shipped
|
||||
as a fallback. Modal
|
||||
completion/cancel remains owned above the decoder by the existing `0x20f` host path. A future audio implementation
|
||||
will consume timestamped PCM and may become the presentation clock; it must not change the VM-facing stop-time,
|
||||
surface, or cancellation contracts.
|
||||
The factory is injection for tests and future decoder replacement, not runtime codec roulette. FFmpeg is the
|
||||
sole backend; DirectShow was deleted rather than shipped as a fallback after the live gate passed. Modal
|
||||
completion/cancel remains owned above the decoder by the existing `0x20f` host path and tears down its generator
|
||||
with the video instance. Native movie flag overrides select mute/music/SE/voice audio buses; the ordinary path
|
||||
uses the movie bus.
|
||||
|
||||
This replacement is now also required for Windows gameplay parity. Archive-backed probes on 2026-07-21
|
||||
show that the current DirectShow graph accepts tested MPEG widths divisible by 16 (208, 288, 304, 400, and
|
||||
@@ -120,6 +121,18 @@ but only decodes Layer II audio and explicitly ignores program-stream PTS in its
|
||||
FFmpeg covers the complete installed codec set and leaves the mod/profile boundary open without selecting a
|
||||
different decoder per effect.
|
||||
|
||||
ABI v2 and MPEG audio playback landed on 2026-07-25. The expanded installed-corpus gate passes all 213 assets:
|
||||
184 remain strictly video-only and all 29 audio-bearing streams decode to finite 44.1 kHz stereo PCM with
|
||||
monotonic timestamps and clean EOF/teardown. The gate decoded 17,537 audio blocks / 18,185,856 stereo PCM
|
||||
frames in the same eight-second unpaced run. A headless natural boot completed audio-clocked `LOGO.AGF` at
|
||||
7,288 ms, released it without generator errors, and opened `OP.AGF`. A first windowed run confirmed audible
|
||||
LOGO/OP output but exposed crackle/warble. The decoded PCM was continuous; the sink had treated each whole-
|
||||
millisecond PTS rounding error (up to 44 samples at 44.1 kHz) as a real boundary and inserted or removed samples
|
||||
from every MPEG block. The sink now ignores established-timeline discrepancies within 2 ms while still honoring
|
||||
the first timestamp and material later gaps/overlaps. A 4,093-block OP regression pins splice-free continuity.
|
||||
The clean audible recheck passed for LOGO/OP and CHAPTER. DirectShow and the managed Windows platform annotations
|
||||
were then deleted; missing, distorted, or unsynchronized audio on any of the 29 audio-bearing assets is now a bug.
|
||||
|
||||
Native deliverables are RID-specific and bundled with the Godot export; the runtime must not discover an
|
||||
arbitrary system FFmpeg. The first implementation gate is Windows x64 because that is the current runnable
|
||||
target, but the C ABI and loader paths must reserve Windows x64, Linux x64, macOS x64, and macOS arm64 from the
|
||||
@@ -137,7 +150,8 @@ nondecreasing timestamps. Malformed input and repeated teardown are covered. Thi
|
||||
packaging. `FfmpegMovieDecoder` now adds cancellable timestamp pacing and is the live selection. The Godot build
|
||||
copies the shim, five required shared libraries, and FFmpeg license beside `Himegari.dll`; a natural SYSTEM4
|
||||
smoke completed 7288 ms `LOGO.AGF`, opened 106919 ms `OP.AGF`, and published frames from both without loading
|
||||
DirectShow; the user subsequently confirmed both opening movies work in normal windowed playback. Full
|
||||
DirectShow; the user subsequently confirmed both opening movies work in normal windowed playback. DirectShow
|
||||
was deleted after the later synchronized-audio gate. Full
|
||||
export/source-offer packaging and non-Windows builds remain outstanding.
|
||||
|
||||
The Windows-x64 installed-corpus gate passed on 2026-07-22 with the pinned dependency: all 213 signature-selected
|
||||
@@ -145,7 +159,8 @@ MPEG payloads opened and reached EOF under a 30-second per-item bound. It decode
|
||||
display sizes (120x120 through 800x600), independently matched sequence-header dimensions, found positive
|
||||
duration and frame-rate metadata, verified tightly packed RGBA size and nondecreasing timestamps, observed
|
||||
changing imagery in every asset, and closed every session. The set includes all 184 video-only and 29 audio-bearing
|
||||
streams; audio presence is detected but PCM remains intentionally undelivered. The 263-second `ED.AGF` was the
|
||||
streams; at that video-only milestone audio presence was detected but PCM remained intentionally undelivered.
|
||||
The later ABI-v2 gate above supersedes that limitation. The 263-second `ED.AGF` was the
|
||||
slowest decode at 4.4 seconds in the deliberately unpaced gate, so its prior 30-second failure was test-harness
|
||||
whole-frame hashing overhead rather than a decoder incompatibility. The disposable machine report is
|
||||
`build/movie-corpus-ffmpeg.json`; invocation and report semantics are canonical in `docs/tools-reference.md`.
|
||||
|
||||
@@ -219,7 +219,7 @@ immutable LGPL shared FFmpeg archive and SHA-256.
|
||||
|---|---|---|---|
|
||||
| `native/age_movie_ffmpeg/bootstrap-win64.ps1` | Download, hash-check, extract, and version-check the pinned FFmpeg SDK. Returns the resolved SDK root. | `.\\native\\age_movie_ffmpeg\\bootstrap-win64.ps1 [-Destination <dir>]` | `dependency-win64.json`, network/archive cache → disposable `build/downloads/`, `build/ffmpeg-sdk/` |
|
||||
| `native/age_movie_ffmpeg/build-win64.ps1` | Discover the MSVC x64 toolchain, build `age_movie_ffmpeg.dll`, and stage its exact shared-library/license dependencies. | `.\\native\\age_movie_ffmpeg\\build-win64.ps1 -SdkRoot <bootstrap-output> [-OutputDirectory <dir>]` | C ABI source + FFmpeg SDK → disposable `build/native/win-x64/` by default |
|
||||
| `tools/movie-corpus-gate` | Discover every MPEG program stream stored under an `.AGF` catalog entry, decode every video frame through the unpaced FFmpeg session, validate independent sequence dimensions, metadata, RGBA size, timestamps, EOF, timeout, and teardown, then emit a per-asset JSON report. Exit 0 means the expected corpus count and every asset passed; exit 1 is a gate failure; exit 2 means the native shim is absent. | `dotnet run --project tools/movie-corpus-gate -- --output build/movie-corpus-ffmpeg.json --expected-count 213 --max-item-ms 30000` · optional `--native-dir <dir>` | `SYS4INI.BIN` + loose/ALF VFS assets + staged FFmpeg shim → stdout progress + disposable `build/movie-corpus-ffmpeg.json` |
|
||||
| `tools/movie-corpus-gate` | Discover every MPEG program stream stored under an `.AGF` catalog entry; decode every video frame and, when present, every audio block through the unpaced FFmpeg session; validate independent sequence dimensions, metadata, RGBA size, stereo finite PCM, monotonic video/audio timestamps, EOF, timeout, and teardown; then emit a per-asset JSON report. Audio fields include sample rate/channels, block and PCM-frame counts, first/last PTS, signal presence, and decode time. Exit 0 means the expected corpus count and every asset passed; exit 1 is a gate failure; exit 2 means the native shim is absent. | `dotnet run --project tools/movie-corpus-gate -- --output build/movie-corpus-ffmpeg.json --expected-count 213 --max-item-ms 30000` · optional `--native-dir <dir>` | `SYS4INI.BIN` + loose/ALF VFS assets + staged FFmpeg shim → stdout progress + disposable `build/movie-corpus-ffmpeg.json` |
|
||||
|
||||
The managed isolated probes load from `AGE_FFMPEG_NATIVE_DIR` when set, then application-local and
|
||||
`runtimes/win-x64/native` locations. Build the shim before running the `FfmpegShim*` tests; no original-game
|
||||
@@ -227,8 +227,8 @@ movie is copied into the repository or native output. When `build/native/win-x64
|
||||
`dotnet build godot/Himegari.csproj` also stages the shim, its five DLL dependencies, and `FFmpeg-LICENSE.txt`
|
||||
beside `Himegari.dll` for development playback.
|
||||
|
||||
The corpus gate intentionally bypasses presentation waits: it validates decode compatibility and lifecycle,
|
||||
not wall-clock playback pacing. `--expected-count` makes additions, omissions, or profile changes explicit;
|
||||
The corpus gate intentionally bypasses presentation waits: it validates video/audio decode compatibility and
|
||||
lifecycle, not wall-clock playback pacing or audible output. `--expected-count` makes additions, omissions, or profile changes explicit;
|
||||
changing the pinned FFmpeg dependency requires rerunning this gate.
|
||||
|
||||
## Engine (C#) — VM core, CLI, Godot frontend
|
||||
|
||||
Reference in New Issue
Block a user