Files
Oxicloud/src/infrastructure/services/ffmpeg_video_frame_service.rs
T
DioCrafts 5722481c4a feat(thumbnails): server-side video thumbnails via ffmpeg
Videos now get a thumbnail generated eagerly server-side on upload, through
the same WebP/blob-hash pipeline as photos — instead of the old browser path
that only ran when the Photos grid first rendered a video tile, re-downloaded
the whole video to seek a frame, and PUT 3 JPEGs back (and produced nothing at
all for HEVC/.mov, which a browser <video> cannot decode).

- New VideoFramePort (application) + FfmpegVideoFrameService / NoopVideoFrameService
  (infrastructure): shell out to the system ffmpeg (no compile-time libav dep),
  extract one representative frame as PNG, bounded by its own semaphore + a
  per-process timeout + kill_on_drop. Noop when ffmpeg is absent/disabled, so
  videos degrade gracefully to no thumbnail.
- ThumbnailRefreshHook.on_file_created routes video/* to
  generate_video_thumbnails_background: stream the (decrypted, reassembled) blob
  to a size- and time-bounded temp file on the data volume, extract a frame, and
  reuse the shared render_and_persist_all_webp helper — so video thumbnails are
  WebP, blob-hash keyed (dedup'd) and content-negotiated, exactly like photos.
- GET thumbnail serves the video's WebP to every client (byte-sniffed
  Content-Type); a genuine miss returns 204.
- Config: OXICLOUD_ENABLE_VIDEO_THUMBNAILS (default true, needs ffmpeg detected
  at startup) + OXICLOUD_FFMPEG_PATH / _CONCURRENCY / _TIMEOUT_SECS / _MAX_MB.
- Dockerfile installs ffmpeg in the runtime image.
- Frontend: drop the client-side generateVideoThumb/frameFromVideo re-download
  path; the server is now the source of truth.

Benchmark (examples/bench_video_thumbnails.rs, needs ffmpeg): 4/4 codecs incl.
HEVC/.mov produce a thumbnail server-side (was 0% for HEVC); ~50-70 ms/frame in
the background; ~3.9 KB preview WebP; up to ~23x less per-first-view transfer on
the test corpus (far more on real multi-MB clips). Methodology in
benches/VIDEO-THUMB.md.

Hardening from an adversarial review: video render holds the decode_semaphore
like the image path; the ffmpeg scale filter bounds both dimensions; the blob
stream has a timeout; the temp file lives on the data volume; the size cap uses
saturating_mul.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 23:23:04 +02:00

151 lines
5.7 KiB
Rust
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
//! ffmpeg-backed video frame extractor (and a no-op fallback).
//!
//! Shells out to the system `ffmpeg` binary rather than linking libav*: no
//! compile-time dependency, no binary bloat, and it decodes every container the
//! browser `<video>` element cannot (HEVC/MOV, ProRes, mkv/avi/wmv…). The
//! extracted still frame is handed to the existing image thumbnail pipeline, so
//! video thumbnails become first-class: WebP, blob-hash keyed (dedup'd), and
//! served through the same content negotiation as photos.
use async_trait::async_trait;
use bytes::Bytes;
use std::path::Path;
use std::process::Stdio;
use std::sync::Arc;
use std::time::Duration;
use tokio::process::Command;
use tokio::sync::Semaphore;
use tokio::time::timeout;
use crate::application::ports::video_frame_ports::VideoFramePort;
use crate::common::errors::DomainError;
/// PNG file signature — guards against feeding a non-image (or empty) ffmpeg
/// output into the renderer.
const PNG_MAGIC: &[u8; 8] = b"\x89PNG\r\n\x1a\n";
/// Extracts a representative frame by invoking `ffmpeg`.
pub struct FfmpegVideoFrameService {
ffmpeg_path: String,
/// Bounds concurrent ffmpeg processes — video decode is CPU-heavy and runs
/// outside the async runtime, so it gets its own (smaller) limit rather than
/// sharing the image decode semaphore.
semaphore: Arc<Semaphore>,
/// Per-extraction wall-clock cap; the child is killed on overrun.
timeout: Duration,
}
impl FfmpegVideoFrameService {
pub fn new(ffmpeg_path: String, concurrency: usize, timeout: Duration) -> Self {
Self {
ffmpeg_path,
semaphore: Arc::new(Semaphore::new(concurrency.max(1))),
timeout,
}
}
/// Best-effort startup probe: true if `<ffmpeg_path> -version` runs and exits
/// 0. Synchronous so the composition root can decide — register the real
/// extractor or fall back to [`NoopVideoFrameService`] — without an async
/// context, and so a misconfigured path is logged once at boot instead of
/// failing per upload.
pub fn is_available(ffmpeg_path: &str) -> bool {
std::process::Command::new(ffmpeg_path)
.arg("-version")
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null())
.status()
.map(|s| s.success())
.unwrap_or(false)
}
}
#[async_trait]
impl VideoFramePort for FfmpegVideoFrameService {
fn is_supported_video(&self, mime_type: &str) -> bool {
// ffmpeg is the arbiter of what actually decodes; gate broadly on video/*
// and let extraction fail gracefully for the rare unsupported container.
mime_type.starts_with("video/")
}
async fn extract_frame(&self, path: &Path) -> Result<Bytes, DomainError> {
let _permit =
self.semaphore.acquire().await.map_err(|_| {
DomainError::internal_error("VideoFrame", "extractor semaphore closed")
})?;
// One representative still → PNG on stdout. The `thumbnail` filter scans a
// window of frames and picks the most representative one (skipping black
// intros) without needing a separate duration probe. Fit within a
// 1024×1024 box (preserving aspect) — enough for the 800px `large`
// thumbnail, and bounding BOTH dimensions caps the emitted PNG size so a
// hostile/extreme geometry can't balloon the buffered output. Arguments
// are passed individually (never through a shell), so a hostile file name
// cannot inject anything.
let run = Command::new(&self.ffmpeg_path)
.arg("-nostdin")
.arg("-loglevel")
.arg("error")
.arg("-i")
.arg(path)
.arg("-vf")
.arg("thumbnail,scale=w='min(1024,iw)':h='min(1024,ih)':force_original_aspect_ratio=decrease")
.arg("-frames:v")
.arg("1")
.arg("-f")
.arg("image2pipe")
.arg("-vcodec")
.arg("png")
.arg("pipe:1")
.stdin(Stdio::null())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.kill_on_drop(true)
.output();
let output = timeout(self.timeout, run)
.await
.map_err(|_| DomainError::internal_error("VideoFrame", "ffmpeg timed out"))?
.map_err(|e| {
DomainError::internal_error("VideoFrame", format!("ffmpeg spawn failed: {e}"))
})?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(DomainError::internal_error(
"VideoFrame",
format!("ffmpeg exited {}: {}", output.status, stderr.trim()),
));
}
let png = output.stdout;
if png.len() < PNG_MAGIC.len() || &png[..PNG_MAGIC.len()] != PNG_MAGIC {
return Err(DomainError::internal_error(
"VideoFrame",
"ffmpeg produced no decodable frame",
));
}
Ok(Bytes::from(png))
}
}
/// No-op extractor used when `ffmpeg` is unavailable or video thumbnails are
/// disabled. `is_supported_video` returns false so the lifecycle hook never
/// attempts generation; videos simply have no thumbnail (the prior behaviour).
pub struct NoopVideoFrameService;
#[async_trait]
impl VideoFramePort for NoopVideoFrameService {
fn is_supported_video(&self, _mime_type: &str) -> bool {
false
}
async fn extract_frame(&self, _path: &Path) -> Result<Bytes, DomainError> {
Err(DomainError::internal_error(
"VideoFrame",
"video thumbnail extraction is disabled (no ffmpeg)",
))
}
}