Adds an LGPL FFmpeg-backed video renderer that slots ahead of Media3's
MediaCodecVideoRenderer via EXTENSION_RENDERER_MODE_PREFER. Resolves
playback failures on Huawei EMUI 11 (Mate 20, Kirin 980): the Codec2
HiSilicon AVC decoder initialises cleanly on iOS High@3.1 streams with
deep DPB + full-range yuvj420p, then errors on the first sample inside
MediaCodecVideoRenderer (init-failure fallback can't catch this).
Google's C2 SW AVC decoder hits its 8-frame output-delay cap on the
same shape and stalls on dequeueOutputBuffer.
Media3's own decoder-ffmpeg ships only an audio renderer;
ExperimentalFfmpegVideoRenderer has been a stub since 2020 (returns
FORMAT_UNSUPPORTED_TYPE, createDecoder returns null). NextLib is
GPL-3.0. So we vendor our own Apache-licensed JNI on top of LGPL
FFmpeg, dynamically linked at runtime.
Build flow:
- android/ffmpeg/ holds the JNI source + CMakeLists + orchestrator
script + LGPL notice. No native binaries in git.
- :ux:buildFfmpegJni Gradle task (wired to preBuild) clones
Media3 1.9.2 + FFmpeg release/6.0 into build/ffmpeg-work/ on
first run, builds h264-only static libs per ABI, links
libffmpegJNI.so per ABI into build/jniLibs/<abi>/. AGP picks
them up via sourceSets.main.jniLibs.srcDirs +=. Gradle
UP-TO-DATE skips the task when ffmpeg_jni.cc / CMakeLists /
build_ffmpeg.sh are unchanged.
Renderer:
- FfmpegVideoDecoder (SimpleDecoder) sends each packet with its
inputBuffer.timeUs as pkt->pts; the JNI overwrites
outputBuffer.timeUs with f->pts on receive so frames emitted in
display order carry their true display PTS (input PTS in decode
order scrambles ExoPlayer's drop logic and halves the render
rate on B-frame streams).
- FfmpegOutputSurface does YUV->RGB in one GLES2 pass against an
EGL window surface sized to display orientation. Y plane uses
GL_NEAREST (1:1 sized, sampling at exact texel centres
preserves luma detail); chroma uses GL_LINEAR. Pre-rotated quad
UVs (0/90/180/270) keep the YUV sampling correct when the
coded frame needs rotation for display.
- FfmpegVideoRenderer swaps the output buffer's width/height for
90/270 streams before super.renderOutputBuffer notifies size,
matching MediaCodecVideoRenderer's post-rotation reporting.
Decoder fallback:
- Renderers.kt selects FfmpegVideoRenderer first when
libffmpegJNI.so is loaded; falls through to the platform path
for formats FFmpeg doesn't handle or ABIs without the .so.
- MediaCodec selector deprioritises every HiSilicon decoder
(OMX.hisi.* and c2.hisi.*) so the platform path picks
c2.android.avc.decoder ahead of the C2 Hisi variant when FFmpeg
isn't available. Required because the C2 Hisi failure is
post-init, which Media3's setEnableDecoderFallback(true) can't
intercept.
Compositor:
- VideoCompositor.setInputSurfaceSize lets the renderer resize the
codec-input SurfaceTexture before eglCreateWindowSurface so the
EGL surface inherits matching buffer dimensions on creation
(MediaCodec sizes natively; EGL doesn't).
- VideoPlayerInstance wires Renderers.build with a sizer callback
that calls into compositor.setInputSurfaceSize from the FFmpeg
renderer thread.
Adds docs/architecture.md with the layered video pipeline diagram,
file map, renderer-selection rationale, build flow, and LGPL
boundary notes.
- video_player: ExoPlayer (Android) / AVPlayer (iOS/macOS) backend with
PixelBufferSink, method-channel adapter, Dart-side XVideoPlayer +
testing fake.
- insets: XInsets singleton + XAnimatedInsets widget lerp the system
viewPadding over 220ms so OS bar visibility toggles
(immersiveSticky <-> edgeToEdge) slide bottom-/top-anchored UI into
place instead of snapping by the nav-bar / status-bar height.
Camera page kept the session running while the host app was
backgrounded — wastes battery, holds the hardware, and blocks
other apps from grabbing the camera. Add per-platform observers
that pause/resume the session on app foreground/background, with
a uniform `pauseForBackground` / `resumeForForeground` pair on the
shared CameraInstance.
Behaviour:
- On background: any in-flight recording is hard-cancelled
(matches every messaging app — the take ends with the app
switch). The session stops so the OS can release the camera.
- On foreground: session restarts iff it had been running.
Emits `sessionInterrupted` (`reason: appBackgrounded`) and
`sessionResumed` events so the Dart side can surface UX
affordances if needed.
iOS — `ios/Classes/Camera/CameraInstance+iOS.swift`:
Subscribes to UIApplication.{willResignActive, didBecomeActive}
notifications. Work hops onto sessionQueue so AV mutations stay
serialised. Storage uses the shared
`CameraInstance.lifecycleCleanup` closure slot — extension
doesn't need to add stored properties.
Android — added `androidx.lifecycle:lifecycle-process:2.7.0`,
observes `ProcessLifecycleOwner.get().lifecycle`. ON_STOP →
`pauseForBackground` (cancels recording + drops
CustomLifecycleOwner to CREATED → CameraX releases camera).
ON_START → `resumeForForeground`. Observer add/remove on main
thread per `ProcessLifecycleOwner` contract.
macOS — `macos/Classes/Camera/CameraInstance+macOS.swift`:
Intentional no-op. macOS desktop background semantics are
softer; the chat composer's Card dialog typically stays
foregrounded. Slot is wired so the shared
`observeLifecycle()` call still compiles.
Verified: all four platforms (iOS / Android / macOS / app tests)
build clean. Pod install picks up the new iOS extension file
once Pods/ is fresh — `flutter clean` if mid-iteration.
CameraX 1.3.4 (May 2024) ships `libimage_processing_util_jni.so`
with 0x1000 (4 KB) ELF LOAD alignment. Android 15's 16-KB page
requirement rejects that — the user hit "elf alignment check failed"
on device. 1.4.0+ corrected the linker flags; 1.4.2 is the current
stable.
Also adds `camera-video` to the dep set so Phase 4b can use
`VideoCapture<Recorder>` without another bump.
Verified post-bump:
$ zipalign -v -c -p 16 app-release.apk → all lib/*/*.so (OK)
$ llvm-readelf -l libimage_processing_util_jni.so →
LOAD … 0x4000 (16 KB) on all four ABIs.
