Do image processing with deep neural networks. It works together with another filter which converts the pixel format of the Frame to what the dnn network requires.
The filter accepts the following options:
- dnn_backend
-
Specify which DNN backend to use for model loading and execution. This option accepts the following values:
- tensorflow
-
TensorFlow backend. To enable this backend you need to install the TensorFlow for C library (see https://www.tensorflow.org/install/lang_c) and configure FFmpeg with
--enable-libtensorflow - openvino
-
OpenVINO backend. To enable this backend you need to build and install the OpenVINO for C library (see https://github.com/openvinotoolkit/openvino/blob/master/build-instruction.md) and configure FFmpeg with
--enable-libopenvino(–extra-cflags=-I... –extra-ldflags=-L... might be needed if the header files and libraries are not installed into system path) - torch
-
Libtorch backend. To enable this backend you need to build and install Libtorch for C++ library. Please download cxx11 ABI version (see https://pytorch.org/get-started/locally) and configure FFmpeg with
--enable-libtorch --extra-cflags=-I/libtorch_root/libtorch/include --extra-cflags=-I/libtorch_root/libtorch/include/torch/csrc/api/include --extra-ldflags=-L/libtorch_root/libtorch/lib/When FFmpeg is built with CUDA support and the input frames use
AV_PIX_FMT_CUDAwith an underlying pixel format ofrgb0,bgr0,rgb24, orbgr24, the Torch backend maps GPU frame buffers directly into LibTorch CUDA tensors using zero-copy memory mapping. This avoids host-device PCIe transfers and can significantly reduce inference latency. The zero-copy path is selected automatically when the input format qualifies; no additional options are required. When the input format is notAV_PIX_FMT_CUDA, the backend uses the standard CPU tensor path. - onnx
-
ONNX Runtime backend. To enable this backend you need to install the ONNX Runtime library (see https://onnxruntime.ai/) and configure FFmpeg with
--enable-libonnxruntime.The current ONNX Runtime backend expects 4-D input and output tensors with NCHW layout and 32-bit floating-point element type (ONNX
FLOAT); models with integer or other element types (e.g.UINT8) are not supported and will be rejected at load time. Models using NHWC layout or other ranks are not yet supported. Only single-input models are supported; the backend binds exactly one input tensor when running the model.The input and output options are optional for the ONNX Runtime backend; when they are omitted the backend resolves the tensor names from the session. Multiple output names may be supplied separated by & (e.g. output=out_a&out_b); however for
dnn_processingonly the first output tensor is used for frame post-processing.The ONNX Runtime backend runs inference synchronously using a single inference request. The shared async and nireq options therefore have no effect for
dnn_backend=onnx; inference always runs synchronously regardless of their values.When FFmpeg is built with Direct3D 12 support and this build’s ONNX Runtime provides a working DirectML execution provider, setting device=dml together with
AV_PIX_FMT_D3D12NV12 hardware frames as input (e.g. via-hwaccel d3d12va -hwaccel_output_format d3d12) keeps frames in VRAM for the whole filter, from the decoder’s output through DNN inference and back out, without a host round-trip. Software-frame input retains the host I/O path. D3D12 input requires successful DirectML negotiation; unsupported hardware-input configurations fail during configuration. At runtime, DirectML.dll and d3dcompiler_47.dll must be loadable from the application directory or System32. Currently, only single-channel (luma-only),FLOAT/NCHW models with a[1,1,H,W]output tensor and even input/output frame dimensions are supported; the chroma plane is copied or bilinearly upscaled on the GPU without going through the model. The output frames are D3D12 hardware frames, so they must either be consumed by a D3D12 encoder or downloaded explicitly withhwdownload,format=nv12.
- model
-
Set path to model file specifying network architecture and its parameters. Note that different backends use different file formats. TensorFlow, OpenVINO, Libtorch, and ONNX Runtime backends can load files for only their respective formats.
- input
-
Set the input name of the dnn network. Required for the TensorFlow backend; optional for the ONNX Runtime backend.
- output
-
Set the output name of the dnn network. Required for the TensorFlow backend; optional for the ONNX Runtime backend.
- backend_configs
-
Set the configs to be passed into backend. To use async execution, set async (default: set). Roll back to sync execution if the backend does not support async.
For tensorflow backend, you can set its configs with sess_config options, please use tools/python/tf_sess_config.py to get the configs of TensorFlow backend for your system.
- async
-
Use DNN async inference (default: 1). When enabled, frames are submitted to the backend without blocking the caller. Supported by the Torch, OpenVINO, and TensorFlow backends; the ONNX Runtime backend ignores this option.
- nireq
-
Number of concurrent inference requests (default: 0, meaning auto). When set to 0, the backend automatically determines an appropriate number of requests. Higher values increase throughput by allowing multiple frames to be in flight simultaneously.
- batch_size
-
Number of frames to batch into a single inference request (default: 1). Values greater than 1 cause the backend to accumulate frames and concatenate them along the batch dimension before running the model. This can significantly increase throughput on GPUs at the cost of slightly higher per-frame latency. Currently supported by the OpenVINO and Torch backends.
- device
-
Set the device to run the model. For the ONNX Runtime backend this selects the execution provider:
cpu(default),cuda(NVIDIA GPU),dml(DirectML, Windows only) orvitisai(AMD Ryzen AI NPU). - device_id
-
Set the device index used by GPU execution providers (e.g.
cudaordml) for the ONNX Runtime backend. Default is 0. - threads_per_operation
-
ONNX Runtime backend only. Set the number of CPU threads used per ONNX Runtime operator when running with
device=cpu. Default is 0 (let ONNX Runtime choose automatically). Has no effect for GPU/NPU providers.
Examples
-
Remove rain in rgb24 frame with can.pb (see derain filter):
./ffmpeg -i rain.jpg -vf format=rgb24,dnn_processing=dnn_backend=tensorflow:model=can.pb:input=x:output=y derain.jpg
-
Handle the Y channel with srcnn.pb (see sr filter) for frame with yuv420p (planar YUV formats supported):
./ffmpeg -i 480p.jpg -vf format=yuv420p,scale=w=iw*2:h=ih*2,dnn_processing=dnn_backend=tensorflow:model=srcnn.pb:input=x:output=y -y srcnn.jpg
-
Handle the Y channel with espcn.pb (see sr filter), which changes frame size, for format yuv420p (planar YUV formats supported), please use tools/python/tf_sess_config.py to get the configs of TensorFlow backend for your system.
./ffmpeg -i 480p.jpg -vf format=yuv420p,dnn_processing=dnn_backend=tensorflow:model=espcn.pb:input=x:output=y:backend_configs=sess_config=0x10022805320e09cdccccccccccec3f20012a01303801 -y tmp.espcn.jpg
-
Perform zero-copy GPU inference with the Libtorch backend. Because there is currently no GPU-side filter to convert from YUV to RGB,
hwuploadis used to upload CPU RGB frames into device memory. Once a GPU-side YUV-to-RGB filter becomes available, thehwuploadstep will no longer be necessary and all frames can stay in device memory throughout the pipeline:./ffmpeg -init_hw_device cuda=cuda -filter_hw_device cuda -i input.mp4 \ -vf "format=rgb24,hwupload,dnn_processing=dnn_backend=torch:model=my_model.pt,hwdownload,format=rgb0" \ -y output.mp4
-
Run a single-channel (luma-only) super-resolution model on D3D12 hardware frames with the ONNX Runtime backend, keeping frames in VRAM from decode through inference (Windows only, requires a build with Direct3D 12 support and
--enable-libonnxruntimeagainst a DirectML-capable ONNX Runtime). The D3D12 output frames are encoded directly by a D3D12 hardware encoder:./ffmpeg -hwaccel d3d12va -hwaccel_output_format d3d12 -i input.mp4 \ -vf dnn_processing=dnn_backend=onnx:model=sr_model.onnx:device=dml \ -c:v h264_d3d12va -bf 0 -y output.mp4
-
Same as above, but download the result to software frames after inference so that a software encoder can be used:
./ffmpeg -hwaccel d3d12va -hwaccel_output_format d3d12 -i input.mp4 \ -vf dnn_processing=dnn_backend=onnx:model=sr_model.onnx:device=dml,hwdownload,format=nv12 \ -y output.mp4
-
Process rgb24 frames with a TorchScript model using the Libtorch backend:
./ffmpeg -i input.jpg -vf format=rgb24,dnn_processing=dnn_backend=torch:model=my_model.pt output.jpg
-
Process rgb24 frames with the Libtorch backend using batch inference (four frames per request):
./ffmpeg -i input.mp4 -vf format=rgb24,dnn_processing=dnn_backend=torch:model=my_model.pt:batch_size=4 -y output.mp4