7243ef7de4
Adds an LGPL FFmpeg-backed video renderer that slots ahead of Media3's
MediaCodecVideoRenderer via EXTENSION_RENDERER_MODE_PREFER. Resolves
playback failures on Huawei EMUI 11 (Mate 20, Kirin 980): the Codec2
HiSilicon AVC decoder initialises cleanly on iOS High@3.1 streams with
deep DPB + full-range yuvj420p, then errors on the first sample inside
MediaCodecVideoRenderer (init-failure fallback can't catch this).
Google's C2 SW AVC decoder hits its 8-frame output-delay cap on the
same shape and stalls on dequeueOutputBuffer.
Media3's own decoder-ffmpeg ships only an audio renderer;
ExperimentalFfmpegVideoRenderer has been a stub since 2020 (returns
FORMAT_UNSUPPORTED_TYPE, createDecoder returns null). NextLib is
GPL-3.0. So we vendor our own Apache-licensed JNI on top of LGPL
FFmpeg, dynamically linked at runtime.
Build flow:
- android/ffmpeg/ holds the JNI source + CMakeLists + orchestrator
script + LGPL notice. No native binaries in git.
- :ux:buildFfmpegJni Gradle task (wired to preBuild) clones
Media3 1.9.2 + FFmpeg release/6.0 into build/ffmpeg-work/ on
first run, builds h264-only static libs per ABI, links
libffmpegJNI.so per ABI into build/jniLibs/<abi>/. AGP picks
them up via sourceSets.main.jniLibs.srcDirs +=. Gradle
UP-TO-DATE skips the task when ffmpeg_jni.cc / CMakeLists /
build_ffmpeg.sh are unchanged.
Renderer:
- FfmpegVideoDecoder (SimpleDecoder) sends each packet with its
inputBuffer.timeUs as pkt->pts; the JNI overwrites
outputBuffer.timeUs with f->pts on receive so frames emitted in
display order carry their true display PTS (input PTS in decode
order scrambles ExoPlayer's drop logic and halves the render
rate on B-frame streams).
- FfmpegOutputSurface does YUV->RGB in one GLES2 pass against an
EGL window surface sized to display orientation. Y plane uses
GL_NEAREST (1:1 sized, sampling at exact texel centres
preserves luma detail); chroma uses GL_LINEAR. Pre-rotated quad
UVs (0/90/180/270) keep the YUV sampling correct when the
coded frame needs rotation for display.
- FfmpegVideoRenderer swaps the output buffer's width/height for
90/270 streams before super.renderOutputBuffer notifies size,
matching MediaCodecVideoRenderer's post-rotation reporting.
Decoder fallback:
- Renderers.kt selects FfmpegVideoRenderer first when
libffmpegJNI.so is loaded; falls through to the platform path
for formats FFmpeg doesn't handle or ABIs without the .so.
- MediaCodec selector deprioritises every HiSilicon decoder
(OMX.hisi.* and c2.hisi.*) so the platform path picks
c2.android.avc.decoder ahead of the C2 Hisi variant when FFmpeg
isn't available. Required because the C2 Hisi failure is
post-init, which Media3's setEnableDecoderFallback(true) can't
intercept.
Compositor:
- VideoCompositor.setInputSurfaceSize lets the renderer resize the
codec-input SurfaceTexture before eglCreateWindowSurface so the
EGL surface inherits matching buffer dimensions on creation
(MediaCodec sizes natively; EGL doesn't).
- VideoPlayerInstance wires Renderers.build with a sizer callback
that calls into compositor.setInputSurfaceSize from the FFmpeg
renderer thread.
Adds docs/architecture.md with the layered video pipeline diagram,
file map, renderer-selection rationale, build flow, and LGPL
boundary notes.
82 lines
3.7 KiB
Markdown
82 lines
3.7 KiB
Markdown
# Vendored FFmpeg video decoder for `ux`
|
|
|
|
This directory contains the JNI source + Gradle build wiring for
|
|
`libffmpegJNI.so`, the LGPL-licensed FFmpeg shared library that backs
|
|
`io.swipelab.ux.video.ffmpeg.FfmpegVideoRenderer`. The renderer is
|
|
slotted ahead of `MediaCodecVideoRenderer` so iOS H.264 streams with
|
|
deep DPB (`has_b_frames > 8`) and full-range YUV play on devices where
|
|
the platform decoder fails (notably Huawei Mate 20 on EMUI 11).
|
|
|
|
## How it builds
|
|
|
|
The native library is produced by the `:ux:buildFfmpegJni` Gradle task,
|
|
wired as a dependency of `preBuild`. On any consumer build
|
|
(`flutter build apk`, `./gradlew assembleRelease`, IDE sync) the task
|
|
runs automatically; Gradle's UP-TO-DATE checking skips it when nothing
|
|
relevant changed.
|
|
|
|
What the task does:
|
|
|
|
1. Clones upstream Media3 (`1.9.2`) and FFmpeg (`release/6.0`) into
|
|
`<ux-android-build>/ffmpeg-work/` if missing — once per checkout.
|
|
2. Drops the vendored `ffmpeg_jni.cc` + `CMakeLists.txt` over the
|
|
upstream Media3 copies so the build produces a video-capable JNI.
|
|
3. Builds FFmpeg static libs (`libavcodec`, `libavutil`,
|
|
`libswresample`) with H.264 decoder enabled for `armeabi-v7a`,
|
|
`arm64-v8a`, `x86`, `x86_64`. Slow part — first build only
|
|
(~30 min on a typical x86 host, ~2 min on Apple Silicon). Static
|
|
libs are cached in `ffmpeg-work/` and reused on subsequent runs.
|
|
4. Cross-compiles `libffmpegJNI.so` per ABI via CMake + Ninja and
|
|
writes the result into `<ux-android-build>/jniLibs/<abi>/`. AGP
|
|
picks them up via the `jniLibs.srcDirs +=` line in
|
|
[build.gradle](../build.gradle) and bundles them in the AAR.
|
|
|
|
What ships in git:
|
|
|
|
* `ffmpeg_jni.cc` — JNI bridge exposing the five entry points
|
|
`FfmpegVideoDecoder.java` calls: initialize / sendPacket /
|
|
receiveFrame / flush / release. Adapted from Media3's audio-only
|
|
template; the audio path was dropped (MediaCodec AAC works
|
|
everywhere we ship to).
|
|
* `CMakeLists.txt` — links the FFmpeg static libs into
|
|
`libffmpegJNI.so`.
|
|
* `build_ffmpeg.sh` — orchestrates the clone + static-lib build +
|
|
JNI link. Env-driven, invoked only by the Gradle task — not
|
|
intended to be run by hand.
|
|
* `LICENSE-FFMPEG.txt` — LGPL v2.1 text. Required attribution.
|
|
|
|
**No `.so` files in git.** Everything under `<build>/jniLibs/` is
|
|
generated on demand; `/build` is already in [.gitignore](../.gitignore).
|
|
|
|
## License
|
|
|
|
FFmpeg is LGPL v2.1. We link to it dynamically (consumer apps load
|
|
`libffmpegJNI.so` at runtime via `System.loadLibrary`), which keeps the
|
|
LGPL boundary intact and does not impose copyleft on the consuming
|
|
app. The build configuration in upstream Media3's `build_ffmpeg.sh`
|
|
intentionally omits `--enable-gpl` and `--enable-nonfree` to keep the
|
|
binaries LGPL-only. Do not add codecs that require GPL configuration
|
|
(e.g. x264) — that would taint the artifact.
|
|
|
|
## Pinned versions
|
|
|
|
* Media3 `1.9.2` (matches the version used by the rest of `ux`).
|
|
* FFmpeg `release/6.0` (matches Media3's tested compatibility window).
|
|
* Android NDK is resolved via `android.ndkDirectory` from AGP — the
|
|
NDK version your Android project pins, normally r27 on a recent
|
|
AGP. Older NDKs may fail the 16-KB page-size alignment check
|
|
enforced for Android 15.
|
|
|
|
Bumping either version: edit the `MEDIA3_TAG` / `FFMPEG_TAG`
|
|
constants near the top of `build_ffmpeg.sh`, then `./gradlew
|
|
:ux:buildFfmpegJni --rerun-tasks` to force a rebuild.
|
|
|
|
## Why we vendor this instead of using `media3-decoder-ffmpeg`
|
|
|
|
Media3's published FFmpeg extension is audio-only.
|
|
`ExperimentalFfmpegVideoRenderer` in the same library has been a stub
|
|
since 2020 — `createDecoder()` returns null, `supportsFormat()` returns
|
|
`FORMAT_UNSUPPORTED_TYPE`. The community alternative (NextLib) is
|
|
GPL-3.0, which would impose copyleft on consumers. So we built our own
|
|
on top of the same JNI skeleton.
|