lab/ux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

video: vendor FFmpeg software AVC renderer

Browse Source 
Adds an LGPL FFmpeg-backed video renderer that slots ahead of Media3's
MediaCodecVideoRenderer via EXTENSION_RENDERER_MODE_PREFER. Resolves
playback failures on Huawei EMUI 11 (Mate 20, Kirin 980): the Codec2
HiSilicon AVC decoder initialises cleanly on iOS High@3.1 streams with
deep DPB + full-range yuvj420p, then errors on the first sample inside
MediaCodecVideoRenderer (init-failure fallback can't catch this).
Google's C2 SW AVC decoder hits its 8-frame output-delay cap on the
same shape and stalls on dequeueOutputBuffer.

Media3's own decoder-ffmpeg ships only an audio renderer;
ExperimentalFfmpegVideoRenderer has been a stub since 2020 (returns
FORMAT_UNSUPPORTED_TYPE, createDecoder returns null). NextLib is
GPL-3.0. So we vendor our own Apache-licensed JNI on top of LGPL
FFmpeg, dynamically linked at runtime.

Build flow:
  - android/ffmpeg/ holds the JNI source + CMakeLists + orchestrator
    script + LGPL notice. No native binaries in git.
  - :ux:buildFfmpegJni Gradle task (wired to preBuild) clones
    Media3 1.9.2 + FFmpeg release/6.0 into build/ffmpeg-work/ on
    first run, builds h264-only static libs per ABI, links
    libffmpegJNI.so per ABI into build/jniLibs/<abi>/. AGP picks
    them up via sourceSets.main.jniLibs.srcDirs +=. Gradle
    UP-TO-DATE skips the task when ffmpeg_jni.cc / CMakeLists /
    build_ffmpeg.sh are unchanged.

Renderer:
  - FfmpegVideoDecoder (SimpleDecoder) sends each packet with its
    inputBuffer.timeUs as pkt->pts; the JNI overwrites
    outputBuffer.timeUs with f->pts on receive so frames emitted in
    display order carry their true display PTS (input PTS in decode
    order scrambles ExoPlayer's drop logic and halves the render
    rate on B-frame streams).
  - FfmpegOutputSurface does YUV->RGB in one GLES2 pass against an
    EGL window surface sized to display orientation. Y plane uses
    GL_NEAREST (1:1 sized, sampling at exact texel centres
    preserves luma detail); chroma uses GL_LINEAR. Pre-rotated quad
    UVs (0/90/180/270) keep the YUV sampling correct when the
    coded frame needs rotation for display.
  - FfmpegVideoRenderer swaps the output buffer's width/height for
    90/270 streams before super.renderOutputBuffer notifies size,
    matching MediaCodecVideoRenderer's post-rotation reporting.

Decoder fallback:
  - Renderers.kt selects FfmpegVideoRenderer first when
    libffmpegJNI.so is loaded; falls through to the platform path
    for formats FFmpeg doesn't handle or ABIs without the .so.
  - MediaCodec selector deprioritises every HiSilicon decoder
    (OMX.hisi.* and c2.hisi.*) so the platform path picks
    c2.android.avc.decoder ahead of the C2 Hisi variant when FFmpeg
    isn't available. Required because the C2 Hisi failure is
    post-init, which Media3's setEnableDecoderFallback(true) can't
    intercept.

Compositor:
  - VideoCompositor.setInputSurfaceSize lets the renderer resize the
    codec-input SurfaceTexture before eglCreateWindowSurface so the
    EGL surface inherits matching buffer dimensions on creation
    (MediaCodec sizes natively; EGL doesn't).
  - VideoPlayerInstance wires Renderers.build with a sizer callback
    that calls into compositor.setInputSurfaceSize from the FFmpeg
    renderer thread.

Adds docs/architecture.md with the layered video pipeline diagram,
file map, renderer-selection rationale, build flow, and LGPL
boundary notes.
This commit is contained in:
agra
2026-05-28 19:24:17 +03:00
parent 7ad3a38d38
commit 7243ef7de4
16 changed files with 2439 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
android/build.gradle
View File

@@ -51,6 +51,53 @@ android {
kotlinOptions {
jvmTarget = '1.8'
}
sourceSets {
main {
// libffmpegJNI.so is built by the buildFfmpegJni task into
// build/jniLibs/<abi>/ on first build (and any time the
// vendored ffmpeg_jni.cc / CMakeLists.txt change). Adding
// the directory here lets AGP package the .so into the
// AAR without committing native binaries to the repo.
jniLibs.srcDirs += "$buildDir/jniLibs"
}
}
}
// FFmpeg video decoder build — runs as part of the normal Android
// build. On first build for a given checkout it clones Media3 + FFmpeg
// into build/ffmpeg-work/ and produces libffmpegJNI.so per ABI (~30 min
// for the FFmpeg static-lib step the first time, fast after). Gradle
// UP-TO-DATE checking skips the task whenever the vendored JNI source
// + CMakeLists are unchanged. See android/ffmpeg/README.md.
def ffmpegSrcDir = file("$projectDir/ffmpeg")
def ffmpegWorkDir = file("$buildDir/ffmpeg-work")
def ffmpegOutDir = file("$buildDir/jniLibs")
def ndkCmakeBin = "${android.sdkDirectory}/cmake/3.22.1/bin"
def supportedAbis = ['armeabi-v7a', 'arm64-v8a', 'x86', 'x86_64']
task buildFfmpegJni(type: Exec) {
group = 'build'
description = 'Clones Media3 + FFmpeg if needed, builds libffmpegJNI.so per Android ABI'
inputs.file "$ffmpegSrcDir/ffmpeg_jni.cc"
inputs.file "$ffmpegSrcDir/CMakeLists.txt"
inputs.file "$ffmpegSrcDir/build_ffmpeg.sh"
supportedAbis.each { abi ->
outputs.file "$ffmpegOutDir/$abi/libffmpegJNI.so"
}
workingDir ffmpegSrcDir
commandLine 'bash', "$ffmpegSrcDir/build_ffmpeg.sh"
environment 'JNI_SRC', ffmpegSrcDir.absolutePath
environment 'NDK_PATH', android.ndkDirectory.absolutePath
environment 'CMAKE_PATH', ndkCmakeBin
environment 'OUTPUT_DIR', ffmpegOutDir.absolutePath
environment 'WORK_DIR', ffmpegWorkDir.absolutePath
}
afterEvaluate {
preBuild.dependsOn buildFfmpegJni
}
dependencies {