FFmpeg 9.1
Since* 4.3
#

Do image processing with deep neural networks. It works together with another filter which converts the pixel format of the Frame to what the dnn network requires.

The filter accepts the following options:

dnn_backend

Specify which DNN backend to use for model loading and execution. This option accepts the following values:

tensorflow

TensorFlow backend. To enable this backend you need to install the TensorFlow for C library (see https://www.tensorflow.org/install/lang_c) and configure FFmpeg with --enable-libtensorflow

openvino

OpenVINO backend. To enable this backend you need to build and install the OpenVINO for C library (see https://github.com/openvinotoolkit/openvino/blob/master/build-instruction.md) and configure FFmpeg with --enable-libopenvino (–extra-cflags=-I... –extra-ldflags=-L... might be needed if the header files and libraries are not installed into system path)

torch

Libtorch backend. To enable this backend you need to build and install Libtorch for C++ library. Please download cxx11 ABI version (see https://pytorch.org/get-started/locally) and configure FFmpeg with --enable-libtorch --extra-cflags=-I/libtorch_root/libtorch/include --extra-cflags=-I/libtorch_root/libtorch/include/torch/csrc/api/include --extra-ldflags=-L/libtorch_root/libtorch/lib/

When FFmpeg is built with CUDA support and the input frames use AV_PIX_FMT_CUDA with an underlying pixel format of rgb0, bgr0, rgb24, or bgr24, the Torch backend maps GPU frame buffers directly into LibTorch CUDA tensors using zero-copy memory mapping. This avoids host-device PCIe transfers and can significantly reduce inference latency. The zero-copy path is selected automatically when the input format qualifies; no additional options are required. When the input format is not AV_PIX_FMT_CUDA, the backend uses the standard CPU tensor path.

onnx

ONNX Runtime backend. To enable this backend you need to install the ONNX Runtime library (see https://onnxruntime.ai/) and configure FFmpeg with --enable-libonnxruntime.

The current ONNX Runtime backend expects 4-D input and output tensors with NCHW layout and 32-bit floating-point element type (ONNX FLOAT); models with integer or other element types (e.g. UINT8) are not supported and will be rejected at load time. Models using NHWC layout or other ranks are not yet supported. Only single-input models are supported; the backend binds exactly one input tensor when running the model.

The input and output options are optional for the ONNX Runtime backend; when they are omitted the backend resolves the tensor names from the session. Multiple output names may be supplied separated by & (e.g. output=out_a&out_b); however for dnn_processing only the first output tensor is used for frame post-processing.

The ONNX Runtime backend runs inference synchronously using a single inference request. The shared async and nireq options therefore have no effect for dnn_backend=onnx; inference always runs synchronously regardless of their values.

When FFmpeg is built with Direct3D 12 support and this build’s ONNX Runtime provides a working DirectML execution provider, setting device=dml together with AV_PIX_FMT_D3D12 NV12 hardware frames as input (e.g. via -hwaccel d3d12va -hwaccel_output_format d3d12) keeps frames in VRAM for the whole filter, from the decoder’s output through DNN inference and back out, without a host round-trip. Software-frame input retains the host I/O path. D3D12 input requires successful DirectML negotiation; unsupported hardware-input configurations fail during configuration. At runtime, DirectML.dll and d3dcompiler_47.dll must be loadable from the application directory or System32. Currently, only single-channel (luma-only), FLOAT/NCHW models with a [1,1,H,W] output tensor and even input/output frame dimensions are supported; the chroma plane is copied or bilinearly upscaled on the GPU without going through the model. The output frames are D3D12 hardware frames, so they must either be consumed by a D3D12 encoder or downloaded explicitly with hwdownload,format=nv12.

model

Set path to model file specifying network architecture and its parameters. Note that different backends use different file formats. TensorFlow, OpenVINO, Libtorch, and ONNX Runtime backends can load files for only their respective formats.

input

Set the input name of the dnn network. Required for the TensorFlow backend; optional for the ONNX Runtime backend.

output

Set the output name of the dnn network. Required for the TensorFlow backend; optional for the ONNX Runtime backend.

backend_configs

Set the configs to be passed into backend. To use async execution, set async (default: set). Roll back to sync execution if the backend does not support async.

For tensorflow backend, you can set its configs with sess_config options, please use tools/python/tf_sess_config.py to get the configs of TensorFlow backend for your system.

async

Use DNN async inference (default: 1). When enabled, frames are submitted to the backend without blocking the caller. Supported by the Torch, OpenVINO, and TensorFlow backends; the ONNX Runtime backend ignores this option.

nireq

Number of concurrent inference requests (default: 0, meaning auto). When set to 0, the backend automatically determines an appropriate number of requests. Higher values increase throughput by allowing multiple frames to be in flight simultaneously.

batch_size

Number of frames to batch into a single inference request (default: 1). Values greater than 1 cause the backend to accumulate frames and concatenate them along the batch dimension before running the model. This can significantly increase throughput on GPUs at the cost of slightly higher per-frame latency. Currently supported by the OpenVINO and Torch backends.

device

Set the device to run the model. For the ONNX Runtime backend this selects the execution provider: cpu (default), cuda (NVIDIA GPU), dml (DirectML, Windows only) or vitisai (AMD Ryzen AI NPU).

device_id

Set the device index used by GPU execution providers (e.g. cuda or dml) for the ONNX Runtime backend. Default is 0.

threads_per_operation

ONNX Runtime backend only. Set the number of CPU threads used per ONNX Runtime operator when running with device=cpu. Default is 0 (let ONNX Runtime choose automatically). Has no effect for GPU/NPU providers.

#

Examples

  • Remove rain in rgb24 frame with can.pb (see derain filter):

    ./ffmpeg -i rain.jpg -vf format=rgb24,dnn_processing=dnn_backend=tensorflow:model=can.pb:input=x:output=y derain.jpg
  • Handle the Y channel with srcnn.pb (see sr filter) for frame with yuv420p (planar YUV formats supported):

    ./ffmpeg -i 480p.jpg -vf format=yuv420p,scale=w=iw*2:h=ih*2,dnn_processing=dnn_backend=tensorflow:model=srcnn.pb:input=x:output=y -y srcnn.jpg
  • Handle the Y channel with espcn.pb (see sr filter), which changes frame size, for format yuv420p (planar YUV formats supported), please use tools/python/tf_sess_config.py to get the configs of TensorFlow backend for your system.

    ./ffmpeg -i 480p.jpg -vf format=yuv420p,dnn_processing=dnn_backend=tensorflow:model=espcn.pb:input=x:output=y:backend_configs=sess_config=0x10022805320e09cdccccccccccec3f20012a01303801 -y tmp.espcn.jpg
  • Perform zero-copy GPU inference with the Libtorch backend. Because there is currently no GPU-side filter to convert from YUV to RGB, hwupload is used to upload CPU RGB frames into device memory. Once a GPU-side YUV-to-RGB filter becomes available, the hwupload step will no longer be necessary and all frames can stay in device memory throughout the pipeline:

    ./ffmpeg -init_hw_device cuda=cuda -filter_hw_device cuda -i input.mp4 \
      -vf "format=rgb24,hwupload,dnn_processing=dnn_backend=torch:model=my_model.pt,hwdownload,format=rgb0" \
      -y output.mp4
  • Run a single-channel (luma-only) super-resolution model on D3D12 hardware frames with the ONNX Runtime backend, keeping frames in VRAM from decode through inference (Windows only, requires a build with Direct3D 12 support and --enable-libonnxruntime against a DirectML-capable ONNX Runtime). The D3D12 output frames are encoded directly by a D3D12 hardware encoder:

    ./ffmpeg -hwaccel d3d12va -hwaccel_output_format d3d12 -i input.mp4 \
      -vf dnn_processing=dnn_backend=onnx:model=sr_model.onnx:device=dml \
      -c:v h264_d3d12va -bf 0 -y output.mp4
  • Same as above, but download the result to software frames after inference so that a software encoder can be used:

    ./ffmpeg -hwaccel d3d12va -hwaccel_output_format d3d12 -i input.mp4 \
      -vf dnn_processing=dnn_backend=onnx:model=sr_model.onnx:device=dml,hwdownload,format=nv12 \
      -y output.mp4
  • Process rgb24 frames with a TorchScript model using the Libtorch backend:

    ./ffmpeg -i input.jpg -vf format=rgb24,dnn_processing=dnn_backend=torch:model=my_model.pt output.jpg
  • Process rgb24 frames with the Libtorch backend using batch inference (four frames per request):

    ./ffmpeg -i input.mp4 -vf format=rgb24,dnn_processing=dnn_backend=torch:model=my_model.pt:batch_size=4 -y output.mp4