lab/ux

Files

agra 7243ef7de4 video: vendor FFmpeg software AVC renderer

Adds an LGPL FFmpeg-backed video renderer that slots ahead of Media3's
MediaCodecVideoRenderer via EXTENSION_RENDERER_MODE_PREFER. Resolves
playback failures on Huawei EMUI 11 (Mate 20, Kirin 980): the Codec2
HiSilicon AVC decoder initialises cleanly on iOS High@3.1 streams with
deep DPB + full-range yuvj420p, then errors on the first sample inside
MediaCodecVideoRenderer (init-failure fallback can't catch this).
Google's C2 SW AVC decoder hits its 8-frame output-delay cap on the
same shape and stalls on dequeueOutputBuffer.

Media3's own decoder-ffmpeg ships only an audio renderer;
ExperimentalFfmpegVideoRenderer has been a stub since 2020 (returns
FORMAT_UNSUPPORTED_TYPE, createDecoder returns null). NextLib is
GPL-3.0. So we vendor our own Apache-licensed JNI on top of LGPL
FFmpeg, dynamically linked at runtime.

Build flow:
  - android/ffmpeg/ holds the JNI source + CMakeLists + orchestrator
    script + LGPL notice. No native binaries in git.
  - :ux:buildFfmpegJni Gradle task (wired to preBuild) clones
    Media3 1.9.2 + FFmpeg release/6.0 into build/ffmpeg-work/ on
    first run, builds h264-only static libs per ABI, links
    libffmpegJNI.so per ABI into build/jniLibs/<abi>/. AGP picks
    them up via sourceSets.main.jniLibs.srcDirs +=. Gradle
    UP-TO-DATE skips the task when ffmpeg_jni.cc / CMakeLists /
    build_ffmpeg.sh are unchanged.

Renderer:
  - FfmpegVideoDecoder (SimpleDecoder) sends each packet with its
    inputBuffer.timeUs as pkt->pts; the JNI overwrites
    outputBuffer.timeUs with f->pts on receive so frames emitted in
    display order carry their true display PTS (input PTS in decode
    order scrambles ExoPlayer's drop logic and halves the render
    rate on B-frame streams).
  - FfmpegOutputSurface does YUV->RGB in one GLES2 pass against an
    EGL window surface sized to display orientation. Y plane uses
    GL_NEAREST (1:1 sized, sampling at exact texel centres
    preserves luma detail); chroma uses GL_LINEAR. Pre-rotated quad
    UVs (0/90/180/270) keep the YUV sampling correct when the
    coded frame needs rotation for display.
  - FfmpegVideoRenderer swaps the output buffer's width/height for
    90/270 streams before super.renderOutputBuffer notifies size,
    matching MediaCodecVideoRenderer's post-rotation reporting.

Decoder fallback:
  - Renderers.kt selects FfmpegVideoRenderer first when
    libffmpegJNI.so is loaded; falls through to the platform path
    for formats FFmpeg doesn't handle or ABIs without the .so.
  - MediaCodec selector deprioritises every HiSilicon decoder
    (OMX.hisi.* and c2.hisi.*) so the platform path picks
    c2.android.avc.decoder ahead of the C2 Hisi variant when FFmpeg
    isn't available. Required because the C2 Hisi failure is
    post-init, which Media3's setEnableDecoderFallback(true) can't
    intercept.

Compositor:
  - VideoCompositor.setInputSurfaceSize lets the renderer resize the
    codec-input SurfaceTexture before eglCreateWindowSurface so the
    EGL surface inherits matching buffer dimensions on creation
    (MediaCodec sizes natively; EGL doesn't).
  - VideoPlayerInstance wires Renderers.build with a sizer callback
    that calls into compositor.setInputSurfaceSize from the FFmpeg
    renderer thread.

Adds docs/architecture.md with the layered video pipeline diagram,
file map, renderer-selection rationale, build flow, and LGPL
boundary notes.

2026-05-28 19:24:17 +03:00

3.7 KiB

Raw Blame History

Vendored FFmpeg video decoder for `ux`

This directory contains the JNI source + Gradle build wiring for libffmpegJNI.so, the LGPL-licensed FFmpeg shared library that backs io.swipelab.ux.video.ffmpeg.FfmpegVideoRenderer. The renderer is slotted ahead of MediaCodecVideoRenderer so iOS H.264 streams with deep DPB (has_b_frames > 8) and full-range YUV play on devices where the platform decoder fails (notably Huawei Mate 20 on EMUI 11).

How it builds

The native library is produced by the :ux:buildFfmpegJni Gradle task, wired as a dependency of preBuild. On any consumer build (flutter build apk, ./gradlew assembleRelease, IDE sync) the task runs automatically; Gradle's UP-TO-DATE checking skips it when nothing relevant changed.

What the task does:

Clones upstream Media3 (1.9.2) and FFmpeg (release/6.0) into <ux-android-build>/ffmpeg-work/ if missing — once per checkout.
Drops the vendored ffmpeg_jni.cc + CMakeLists.txt over the upstream Media3 copies so the build produces a video-capable JNI.
Builds FFmpeg static libs (libavcodec, libavutil, libswresample) with H.264 decoder enabled for armeabi-v7a, arm64-v8a, x86, x86_64. Slow part — first build only (~30 min on a typical x86 host, ~2 min on Apple Silicon). Static libs are cached in ffmpeg-work/ and reused on subsequent runs.
Cross-compiles libffmpegJNI.so per ABI via CMake + Ninja and writes the result into <ux-android-build>/jniLibs/<abi>/. AGP picks them up via the jniLibs.srcDirs += line in build.gradle and bundles them in the AAR.

What ships in git:

ffmpeg_jni.cc — JNI bridge exposing the five entry points FfmpegVideoDecoder.java calls: initialize / sendPacket / receiveFrame / flush / release. Adapted from Media3's audio-only template; the audio path was dropped (MediaCodec AAC works everywhere we ship to).
CMakeLists.txt — links the FFmpeg static libs into libffmpegJNI.so.
build_ffmpeg.sh — orchestrates the clone + static-lib build + JNI link. Env-driven, invoked only by the Gradle task — not intended to be run by hand.
LICENSE-FFMPEG.txt — LGPL v2.1 text. Required attribution.

No .so files in git. Everything under <build>/jniLibs/ is generated on demand; /build is already in .gitignore.

License

FFmpeg is LGPL v2.1. We link to it dynamically (consumer apps load libffmpegJNI.so at runtime via System.loadLibrary), which keeps the LGPL boundary intact and does not impose copyleft on the consuming app. The build configuration in upstream Media3's build_ffmpeg.sh intentionally omits --enable-gpl and --enable-nonfree to keep the binaries LGPL-only. Do not add codecs that require GPL configuration (e.g. x264) — that would taint the artifact.

Pinned versions

Media3 1.9.2 (matches the version used by the rest of ux).
FFmpeg release/6.0 (matches Media3's tested compatibility window).
Android NDK is resolved via android.ndkDirectory from AGP — the NDK version your Android project pins, normally r27 on a recent AGP. Older NDKs may fail the 16-KB page-size alignment check enforced for Android 15.

Bumping either version: edit the MEDIA3_TAG / FFMPEG_TAG constants near the top of build_ffmpeg.sh, then ./gradlew :ux:buildFfmpegJni --rerun-tasks to force a rebuild.

Why we vendor this instead of using `media3-decoder-ffmpeg`

Media3's published FFmpeg extension is audio-only. ExperimentalFfmpegVideoRenderer in the same library has been a stub since 2020 — createDecoder() returns null, supportsFormat() returns FORMAT_UNSUPPORTED_TYPE. The community alternative (NextLib) is GPL-3.0, which would impose copyleft on consumers. So we built our own on top of the same JNI skeleton.

3.7 KiB Raw Blame History

Vendored FFmpeg video decoder for ux