ux/docs/architecture.md

ux — architecture

This doc maps the load-bearing modules in the ux Flutter package: where each lives, what it owns, and which native pieces back it on each platform. Banlu is the primary consumer; everything here is meant to be reusable across other Flutter apps that depend on ux.

Conventions:

• Source paths are anchored at the package root (e.g. lib/src/...).
• Native paths use android/... and darwin/... (shared iOS+macOS sources) or ios/ / macos/ (platform-specific).
• "Banned" means: don't reintroduce. Lints and tests exist to catch regressions where applicable.

---

Module map

Module	Dart	Android	Apple
XKeyboard — frame-accurate keyboard tracking + interactive dismiss	lib/src/keyboard/	android/src/main/jni/keyboard_bridge.c	darwin/Keyboard/
XSensors — gyro / accelerometer over JNI/FFI	lib/src/sensors/	android/src/main/jni/sensor_bridge.c	darwin/Sensors/
XCamera — CameraX (Android) / AVCaptureSession (Apple)	lib/src/camera/	android/src/main/kotlin/io/swipelab/ux/camera/	darwin/Camera/
XScanner — ZXing QR scanner (Android) / VNDetect (Apple)	lib/src/scanner/	android/src/main/kotlin/io/swipelab/ux/scanner/	darwin/Scanner/
XVideoPlayer — ExoPlayer (Android) / AVPlayer (Apple)	lib/src/video/	android/src/main/kotlin/io/swipelab/ux/video/	darwin/Video/
XNotifications — OS notifications: show / cancel / tap / authorization	lib/src/notifications/	android/.../NotificationsPlugin.kt	macos/Classes/NotificationsPlugin.swift (macOS only)
XWindow — host focus state (focused)	lib/src/window/	android/.../WindowPlugin.kt	macos/Classes/WindowPlugin.swift (macOS only)
XFile, navi — see source	lib/src/	mixed	mixed

---

Video pipeline (Android)

The Android video stack is the most involved piece in ux because it spans four cooperating layers (Dart controller → Kotlin player instance → Media3 ExoPlayer → GL compositor → Flutter texture). It also carries the most production-load-bearing fix: a vendored FFmpeg video decoder that fronts the platform MediaCodec path for files the OEM hardware decoder can't handle.

Layers

Dart  XVideoPlayerController (lib/src/video/x_video_player.dart)
       │   MethodChannel `ux/video`
       ▼
Kotlin VideoPlayerInstance (one per Dart controller)
       │   owns:
       │   ├── ExoPlayer  (Media3 1.9.2; renderers wired via Renderers.kt)
       │   ├── VideoCompositor (codec output → Flutter texture; GL blit)
       │   └── Flutter SurfaceTextureEntry (the texture id Dart Texture() samples)
       ▼
Native Media3 video renderer
       │   tier 1: FfmpegVideoRenderer  (libavcodec, SW decode, GL YUV→RGB)
       │   tier 2: MediaCodecVideoRenderer (HW path; HiSilicon C2 decoders
       │           deprioritised so EMUI 10/11 falls through to c2.android.avc.decoder)
       ▼
GL    SurfaceTexture (compositor input)
       │   compositor reads + applies SurfaceTexture transform
       │   blits to Flutter SurfaceTexture (output)
       ▼
Flutter Texture widget samples the SurfaceTexture, scales to widget size

Files

What	Where
Dart controller	lib/src/video/x_video_player.dart
MethodChannel binding	lib/src/video/x_video_player_channel.dart
Texture widget	lib/src/video/x_video_player_view.dart
Per-controller native player	android/src/main/kotlin/io/swipelab/ux/video/VideoPlayerInstance.kt
ExoPlayer renderer factory	android/src/main/kotlin/io/swipelab/ux/video/Renderers.kt
GL blit between codec and Flutter texture	android/src/main/kotlin/io/swipelab/ux/video/VideoCompositor.kt
FFmpeg renderer (Java)	android/src/main/java/io/swipelab/ux/video/ffmpeg/
FFmpeg JNI source	android/ffmpeg/ffmpeg_jni.cc
FFmpeg build orchestrator	android/ffmpeg/build_ffmpeg.sh

Renderer selection (Renderers.kt)

Renderers.build(context, surfaceSizer) returns a DefaultRenderersFactory subclass that builds the video renderer list with FFmpeg first and MediaCodec second:

1. FfmpegVideoRenderer — picked when FfmpegLibrary.isAvailable() returns true (i.e. libffmpegJNI.so is loaded for the device's ABI) and the format is H.264. Software decode via libavcodec, no DPB cap, handles iOS yuvj420p (full-range) and deep B-frame reorder structures that defeat platform decoders on older Huawei silicon. YUV→RGB happens in a one-pass GLES2 shader inside FfmpegOutputSurface. Y plane uses GL_NEAREST filtering (1:1 sized to the EGL surface, sampling at exact texel centres preserves luma detail); chroma uses GL_LINEAR for clean 4:2:0 upsampling.
2. MediaCodecVideoRenderer (Media3 default) — picked for any other format, or if FFmpeg is unavailable. Two non-default tweaks:
   • setEnableDecoderFallback(true) so init-time failures bounce to the next decoder.
   • Custom MediaCodecSelector deprioritises every HiSilicon decoder (OMX.hisi.* and c2.hisi.*). The EMUI 10 OMX variant fails codec start (caught by fallback); the EMUI 11 C2 variant initialises cleanly but errors on the first sample, which fallback can't catch — name-based deprioritisation is the only way past it when FFmpeg is unavailable for that ABI.

The two-tier design is intentional: FFmpeg adds a slight battery cost vs. hardware decoding, but it's the only path that reliably plays the full range of producer-side content (iOS recordings in particular). Hardware stays as the fallback for the formats and devices where it works.

Why we vendor an FFmpeg video renderer

Media3's published decoder-ffmpeg is audio-only. ExperimentalFfmpegVideoRenderer in the same library has been a stub since 2020 (createDecoder returns null, supportsFormat returns FORMAT_UNSUPPORTED_TYPE). The only well-maintained community alternative (NextLib) is GPL-3.0 — would impose copyleft on every consumer. So we ship our own Apache-licensed JNI on top of the LGPL FFmpeg, vendored under android/ffmpeg/.

How the FFmpeg .so gets built

There are no native binaries in the repo. The :ux:buildFfmpegJni Gradle task (wired to preBuild in android/build.gradle) handles everything:

1. Clones Media3 (1.9.2) and FFmpeg (release/6.0) into the consumer's build/ffmpeg-work/ on first run.
2. Drops the vendored ffmpeg_jni.cc + CMakeLists.txt over the upstream copies (so the produced JNI exposes the video entry points FfmpegVideoDecoder.java calls).
3. Cross-builds FFmpeg static libs (H.264 only) for all 4 Android ABIs.
4. Links libffmpegJNI.so per ABI and writes them into <ux-build>/jniLibs/<abi>/, which AGP picks up via sourceSets.main.jniLibs.srcDirs +=.

Up-to-date checking on ffmpeg_jni.cc + CMakeLists.txt + build_ffmpeg.sh means subsequent builds skip the task entirely. The slow part (FFmpeg static-lib build, ~30 min on an x86 host, ~2 min on Apple Silicon) only happens on the first build for a given checkout, or after a gradle clean.

License: FFmpeg is LGPL v2.1, linked dynamically at runtime via System.loadLibrary("ffmpegJNI") — the LGPL boundary stays intact. The build config omits --enable-gpl and --enable-nonfree. See android/ffmpeg/LICENSE-FFMPEG.txt and android/ffmpeg/README.md.

Looping and end-of-stream

XVideoPlayerController.setLooping(true) maps to ExoPlayer's REPEAT_MODE_ONE (the entire seek-to-0-and-resume cycle is handled in-engine). Non-looping playback emits a completed event when STATE_ENDED is reached without repeat — see VideoPlayerInstance.kt. Consumers that need tap-play-at-end-restarts behavior (Telegram parity) check position against duration on the Dart side and call seekTo(Duration.zero) before play() when within 2 seconds of the end.

---

Video pipeline (Apple)

AVPlayer + AVPlayerItem + AVPlayerItemVideoOutput. No FFmpeg fallback needed — AVFoundation handles iOS-produced H.264 (and HEVC) directly without the DPB-cap / full-range quirks the Android platform decoders trip over. See darwin/Video/VideoPlayerInstance.swift.

Notifications + window focus

XNotifications and XWindow share a two-channel shape: a MethodChannel for commands and an EventChannel for native-pushed events. Each Dart constructor only subscribes to its EventChannel on platforms that register a native handler (defaultTargetPlatform gate) — otherwise activating the stream throws MissingPluginException, which Flutter reports straight to FlutterError.onError.

Handler coverage:

	macOS	Android	iOS
XWindow (ux/window/events)	NSApplication active/resign	ProcessLifecycleOwner ON_START/ON_STOP	none — focused stays true
XNotifications (ux/notifications + …/events)	UNUserNotificationCenter	NotificationManagerCompat + POST_NOTIFICATIONS	none

On Android the two are coupled: XWindow.focused flipping to false when the app backgrounds is what lets a consumer (Banlu's MessageNotifier) post a local notification for a live socket message while the process is alive but unfocused — the gap FCM intentionally skips for socket-connected devices. The notification's tap PendingIntent relaunches the app's launcher activity (resolved generically via getLaunchIntentForPackage, no app class hard-coded) carrying the data payload, which the plugin re-emits as a tap event; a tap that cold-starts the process is buffered until Dart subscribes.