From 43602d36a1576b6c237d1439aa6b4ee89c6379d4 Mon Sep 17 00:00:00 2001 From: Viktor Sokolov Date: Mon, 14 Sep 2026 12:32:17 +0200 Subject: [PATCH] IMG-140: hw accel + TIFF_UNLIMITED --- docs/configuration/options.mdx | 29 +++- docs/features/hardware_acceleration.mdx | 152 ++++++++++++++++++ docs/features/video_thumbnails.mdx | 131 +++++++++++++++ docs/image_formats_support.mdx | 5 +- sidebars.ts | 16 ++ .../version-4.0.x/configuration/options.mdx | 6 + 6 files changed, 328 insertions(+), 11 deletions(-) create mode 100644 docs/features/hardware_acceleration.mdx create mode 100644 docs/features/video_thumbnails.mdx diff --git a/docs/configuration/options.mdx b/docs/configuration/options.mdx index 7dc9183..8bf8def 100644 --- a/docs/configuration/options.mdx +++ b/docs/configuration/options.mdx @@ -159,6 +159,12 @@ You can limit allowed source URLs with the following variable: Disabling the limit on the SVG file size can lead to memory exhaustion and DoS attacks. ::: +* [`IMGPROXY_TIFF_UNLIMITED`]: when `true`, disables the limit on the TIFF file size (50MB). Default: `false` + + :::warning + Disabling the limit on the TIFF file size can lead to memory exhaustion and DoS attacks. + ::: + * [`IMGPROXY_SANITIZE_SVG`]: when `true`, imgproxy will remove scripts from SVG images to prevent XSS attacks. Defaut: `true` When using imgproxy in a development environment, it can be useful to ignore SSL verification: @@ -383,6 +389,13 @@ imgproxy Pro can extract specific video frames to create thumbnails. This featur Though using `IMGPROXY_VIDEO_THUMBNAIL_PROBE_SIZE` and `IMGPROXY_VIDEO_THUMBNAIL_MAX_ANALYZE_DURATION` can lower the memory footprint of video thumbnail generation, they should be used in production only when you know what you're doing. ::: +Check out the [Video thumbnails](../features/video_thumbnails.mdx) guide to learn more. + +imgproxy Pro can also use VA-API to hardware-accelerate video frame decoding (and, optionally, scaling) on Intel and AMD GPUs. See the [Hardware acceleration](../features/hardware_acceleration.mdx) page for setup instructions. + +* [`IMGPROXY_VIDEO_THUMBNAIL_VAAPI_DEVICE`]: ((pro)) the path to a DRM render node (e.g., `/dev/dri/renderD128`) to use for VA-API hardware-accelerated video decoding. When empty, VA-API is disabled. Default: blank +* [`IMGPROXY_VIDEO_THUMBNAIL_VAAPI_SCALE`]: ((pro)) when `true`, also uses VA-API hardware scaling in the ffmpeg filter graph instead of only hardware decoding. Default: `false` + ## RAW formats imgproxy Pro can process digital camera RAW images. Since loading RAW images is pretty resource intensive, their processing is disabled by default. You can enable it with the following variable: @@ -452,10 +465,10 @@ imgproxy can detect objects on the image and use them to perform smart cropping, * `one_best`: in this mode, imgproxy will focus on the object with the highest score based on its area, confidence, and class weight. * `one_best_centermost`: the same as `one_best,` but imgproxy will add the object's proximity to the image center to its score. -* [`IMGPROXY_OBJECT_DETECTION_INFERENCE_NUM_STREAMS`]: ((pro)) when set, overrides the [IMGPROXY_ML_NUM_STREAMS](#IMGPROXY_ML_NUM_STREAMS) value for the object detection model. -* [`IMGPROXY_OBJECT_DETECTION_INFERENCE_NUM_THREADS`]: ((pro)) when set, overrides the [IMGPROXY_ML_INFERENCE_NUM_THREADS](#IMGPROXY_ML_INFERENCE_NUM_THREADS) value for the object detection model. -* [`IMGPROXY_OBJECT_DETECTION_EXECUTION_MODE`]: ((pro)) when set, overrides the [IMGPROXY_ML_EXECUTION_MODE](#IMGPROXY_ML_EXECUTION_MODE) value for the object detection model. -* [`IMGPROXY_OBJECT_DETECTION_CPU_PINNING`]: ((pro)) when set, overrides the [IMGPROXY_ML_CPU_PINNING](#IMGPROXY_ML_CPU_PINNING) value for the object detection model. +* [`IMGPROXY_OBJECT_DETECTION_ML_INFERENCE_NUM_STREAMS`]: ((pro)) when set, overrides the [IMGPROXY_ML_NUM_STREAMS](#IMGPROXY_ML_NUM_STREAMS) value for the object detection model. +* [`IMGPROXY_OBJECT_DETECTION_ML_INFERENCE_NUM_THREADS`]: ((pro)) when set, overrides the [IMGPROXY_ML_INFERENCE_NUM_THREADS](#IMGPROXY_ML_INFERENCE_NUM_THREADS) value for the object detection model. +* [`IMGPROXY_OBJECT_DETECTION_ML_EXECUTION_MODE`]: ((pro)) when set, overrides the [IMGPROXY_ML_EXECUTION_MODE](#IMGPROXY_ML_EXECUTION_MODE) value for the object detection model. +* [`IMGPROXY_OBJECT_DETECTION_ML_CPU_PINNING`]: ((pro)) when set, overrides the [IMGPROXY_ML_CPU_PINNING](#IMGPROXY_ML_CPU_PINNING) value for the object detection model. Read the [Object Detection guide](../features/object_detection.mdx) for more info. @@ -476,10 +489,10 @@ imgproxy can classify images by assigning them to predefined categories based on * `nhwc`: _(default)_ channels last (TensorFlow default) * `nchw`: channels first (PyTorch default) -* [`IMGPROXY_CLASSIFICATION_INFERENCE_NUM_STREAMS`]: ((pro)) when set, overrides the [IMGPROXY_ML_NUM_STREAMS](#IMGPROXY_ML_NUM_STREAMS) value for the classification model. -* [`IMGPROXY_CLASSIFICATION_INFERENCE_NUM_THREADS`]: ((pro)) when set, overrides the [IMGPROXY_ML_INFERENCE_NUM_THREADS](#IMGPROXY_ML_INFERENCE_NUM_THREADS) value for the classification model. -* [`IMGPROXY_CLASSIFICATION_EXECUTION_MODE`]: ((pro)) when set, overrides the [IMGPROXY_ML_EXECUTION_MODE](#IMGPROXY_ML_EXECUTION_MODE) value for the classification model. -* [`IMGPROXY_CLASSIFICATION_CPU_PINNING`]: ((pro)) when set, overrides the [IMGPROXY_ML_CPU_PINNING](#IMGPROXY_ML_CPU_PINNING) value for the classification model. +* [`IMGPROXY_CLASSIFICATION_ML_INFERENCE_NUM_STREAMS`]: ((pro)) when set, overrides the [IMGPROXY_ML_NUM_STREAMS](#IMGPROXY_ML_NUM_STREAMS) value for the classification model. +* [`IMGPROXY_CLASSIFICATION_ML_INFERENCE_NUM_THREADS`]: ((pro)) when set, overrides the [IMGPROXY_ML_INFERENCE_NUM_THREADS](#IMGPROXY_ML_INFERENCE_NUM_THREADS) value for the classification model. +* [`IMGPROXY_CLASSIFICATION_ML_EXECUTION_MODE`]: ((pro)) when set, overrides the [IMGPROXY_ML_EXECUTION_MODE](#IMGPROXY_ML_EXECUTION_MODE) value for the classification model. +* [`IMGPROXY_CLASSIFICATION_ML_CPU_PINNING`]: ((pro)) when set, overrides the [IMGPROXY_ML_CPU_PINNING](#IMGPROXY_ML_CPU_PINNING) value for the classification model. Read the [Classification guide](../features/classification.mdx) for more info. diff --git a/docs/features/hardware_acceleration.mdx b/docs/features/hardware_acceleration.mdx new file mode 100644 index 0000000..33e816a --- /dev/null +++ b/docs/features/hardware_acceleration.mdx @@ -0,0 +1,152 @@ +--- +title: Hardware acceleration +description: Learn about how to use VA-API hardware acceleration to speed up video thumbnail generation +--- + +# Hardware acceleration ((pro)) + +imgproxy Pro can use [VA-API](https://en.wikipedia.org/wiki/Video_Acceleration_API) to +hardware-accelerate video frame decoding (and, optionally, scaling) when +[generating video thumbnails](../configuration/options.mdx#video-thumbnails). VA-API is supported on both Intel and AMD GPUs +through the Linux DRM render-node interface, offloading decode/scale work from the CPU to the GPU +and reducing CPU usage in deployments that process a lot of video. + +:::note +This page covers standard VA-API acceleration on Intel and AMD GPUs. It doesn't apply to +purpose-built media-accelerator hardware such as Amazon EC2 [VT1 instances](https://aws.amazon.com/ec2/instance-types/vt1/) +(Xilinx Alveo U30 accelerators). Those rely on the Xilinx Video SDK, which ships its own +pre-compiled FFmpeg build with proprietary Xilinx plugins instead of VA-API — imgproxy's VA-API +support can't make use of that hardware. +::: + +## How it works + +imgproxy Pro's official Docker images already ship an `ffmpeg` build compiled with VA-API support. +To actually use hardware acceleration at runtime, you need three things: + +1. A vendor VA-API userspace driver installed in the image (imgproxy's official image doesn't + include one, since it's vendor-specific and has a significant impact on the resulting image size — see + [Building a custom image](#building-a-custom-image) below). +2. The host's DRM render node (`/dev/dri/renderD128` or similar) exposed to the container, with + permissions allowing imgproxy to open it. +3. The [`IMGPROXY_VIDEO_THUMBNAIL_VAAPI_DEVICE`](../configuration/options.mdx#IMGPROXY_VIDEO_THUMBNAIL_VAAPI_DEVICE) + (and, optionally, [`IMGPROXY_VIDEO_THUMBNAIL_VAAPI_SCALE`](../configuration/options.mdx#IMGPROXY_VIDEO_THUMBNAIL_VAAPI_SCALE)) + config variables are set. + +## Building a custom image + +You need to build a custom image based on the official one that adds the driver for your GPU vendor. + +### Intel GPUs + +```dockerfile +FROM darthsim/imgproxy-pro:v4 # or v4-ml + +RUN apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + intel-media-va-driver-non-free \ + && rm -rf /var/lib/apt/lists/* +``` + +### AMD GPUs + +```dockerfile +FROM darthsim/imgproxy-pro:v4 # or v4-ml + +RUN apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + mesa-va-drivers \ + && rm -rf /var/lib/apt/lists/* +``` + +In both cases, no rebuild of `ffmpeg`/`libva` is needed — VA-API support is already built into the +official image; only the vendor driver package is missing. + +Build the image: + +```bash +docker build . -t imgproxy-pro-vaapi +``` + +## Host requirements + +Before running a hardware-accelerated container, the host itself needs a working VA-API render +node: + +1. **A GPU with its kernel driver loaded.** On Linux, AMD GPUs use the `amdgpu` driver. Intel GPUs + use either `i915` (the long-standing driver, still the default for Arc Alchemist, Meteor Lake, + and older integrated GPUs) or `xe` (Intel's newer driver, the default since Lunar Lake and Arc + Battlemage, and optional/experimental on earlier Xe-based GPUs). Both ship with the kernel and + load automatically when the GPU is detected — you don't need to install anything for this on + bare metal. Verify with: + + ```bash + lsmod | grep -E 'i915|xe|amdgpu' + ls /dev/dri + ``` + + You should see device nodes such as `/dev/dri/renderD128` (the render node) and + `/dev/dri/card0` (the primary/display node). + +2. **You only need the render node, not `card0`.** ffmpeg's VA-API decode/scale path only opens + the render node (`/dev/dri/renderD128` or similar). `/dev/dri/card0` is used for + mode-setting/display and is unrelated to hardware-accelerated video decoding — there's no need + to expose it to the container. + +3. **Group permissions.** The render node is owned by the host's `render` group (its gid varies + by distro/host). Add the user that runs the container to that group: + + ```bash + sudo usermod -aG render $USER + ``` + +4. **Running under a hypervisor or cloud VM.** If imgproxy runs inside a VM rather than on bare + metal, the VM also needs the GPU exposed to it (PCI passthrough, SR-IOV, or a vendor virtual-GPU + mechanism) before any of the above applies — that's hypervisor/cloud-provider specific and + outside the scope of this page. + +## Running the container + +Expose the host's DRM render node to the container and make sure imgproxy's process can open it. +If imgproxy's Docker image runs as a non-root user, you typically need to add that user's group to +the host's `render` (or, on some distros, `video`) group that owns the device node: + +```bash +# Find the gid of the host group that owns the render node +getent group render + +# Run the container with the device exposed and the matching group added +docker run \ + --device=/dev/dri \ + --group-add \ + -e IMGPROXY_ENABLE_VIDEO_THUMBNAILS=true \ + -e IMGPROXY_VIDEO_THUMBNAIL_VAAPI_DEVICE=/dev/dri/renderD128 \ + -e IMGPROXY_VIDEO_THUMBNAIL_VAAPI_SCALE=true \ + imgproxy-pro-vaapi:latest +``` + +:::tip +If you're running Docker in **rootless** mode, the container's user namespace doesn't automatically +see the host's `render` group — you need to map it explicitly via `/etc/subgid` and restart the +rootless daemon, then pass the *namespace-mapped* gid (not the raw host gid) to `--group-add`: + +```bash +# Find the host gid of the render group (the 3rd field, e.g. "993") +getent group render + +# Map that gid into the rootless user's subordinate gid range +echo "your-user:993:1" | sudo tee -a /etc/subgid +systemctl --user restart docker +``` +::: + +## Verifying / troubleshooting + +If `IMGPROXY_VIDEO_THUMBNAIL_VAAPI_DEVICE` is set but imgproxy can't open the device, it will fail +fast on startup with an error along the lines of "failed to ensure availability of the hardware +acceleration device". If you see this: + +* Double check the device path matches an actual render node on the host (`ls /dev/dri`). +* Double check the container's `--device=/dev/dri` mount and that the container process has + permission to open the render node (see [Running the container](#running-the-container) above). +* Make sure the vendor VA-API driver package is installed in the image and matches your GPU. diff --git a/docs/features/video_thumbnails.mdx b/docs/features/video_thumbnails.mdx new file mode 100644 index 0000000..868bed7 --- /dev/null +++ b/docs/features/video_thumbnails.mdx @@ -0,0 +1,131 @@ +--- +title: Video thumbnails +description: Learn about how imgproxy Pro generates thumbnails, sprites, and animated previews from video sources +--- + +# Video thumbnails ((pro)) + +imgproxy Pro can use a video as a source and extract still frames: a single +thumbnail, a tiled sprite sheet, or an animated preview. imgproxy downloads only as +much of the video as it needs to reach the requested frame(s), instead of requiring +the whole file upfront. + +Since even partial video downloads and decoding are considerably more expensive than +processing a regular image, video thumbnail generation is **turned off by default**. +Enable it with [`IMGPROXY_ENABLE_VIDEO_THUMBNAILS`](../configuration/options.mdx#IMGPROXY_ENABLE_VIDEO_THUMBNAILS). + +:::info +imgproxy Pro can also use VA-API to hardware-accelerate video frame decoding. See the +[Hardware acceleration](hardware_acceleration.mdx) page. +::: + +## How it works + +imgproxy doesn't rely on a fixed file-extension allowlist to decide whether a source +is a video. It first checks the source against all supported image formats and, only if +none match, hands the data to FFmpeg's container prober as a last resort. + +The only video source format currently supported is MP4 (H.264). See the +[format support table](../image_formats_support.mdx) for the current source/result matrix. + +The video is fetched through imgproxy's normal streaming/ranged HTTP fetcher rather +than downloaded in full, which is why `Content-Length` isn't pre-validated for video +sources the way it is for images — imgproxy reads only as far into the stream as it +needs to. + +[`IMGPROXY_MAX_SRC_RESOLUTION`](../configuration/options.mdx#IMGPROXY_MAX_SRC_RESOLUTION) +applies to the decoded video frame's dimensions exactly as it does to still images. + +imgproxy favors returning something over failing outright: if a requested timestamp +falls beyond the video's actual duration, imgproxy returns the last successfully +decoded frame instead of an error. Sprites and animations behave the same way — if the +source stream ends before all the requested frames were collected, imgproxy returns +whatever frames it managed to decode. + +## Single-frame thumbnails + +Use the [video_thumbnail_second](../usage/processing.mdx#video-thumbnail-second) +(`vts`) processing option, or the +[IMGPROXY_VIDEO_THUMBNAIL_SECOND](../configuration/options.mdx#IMGPROXY_VIDEO_THUMBNAIL_SECOND) +config default, to pick which second of the video should be used for the thumbnail: + +```imgproxy_url +.../resize:fill:300:200/video_thumbnail_second:30/plain/https://example.com/video.mp4@jpg +``` + +By default, imgproxy decodes forward to the exact requested second. Setting +[video_thumbnail_keyframes](../usage/processing.mdx#video-thumbnail-keyframes) (`vtk`), +or the [IMGPROXY_VIDEO_THUMBNAIL_KEYFRAMES](../configuration/options.mdx#IMGPROXY_VIDEO_THUMBNAIL_KEYFRAMES) +config, makes imgproxy seek to the nearest preceding keyframe and use the first frame +it decodes from there. This is much faster on videos with long gaps between keyframes, but the returned +frame's timestamp may not exactly match the one you requested. + +As mentioned in [How it works](#how-it-works), if the requested second is past the end +of the video, imgproxy returns the last frame it could decode instead of an error. + +## Tiled sprite sheets + +The [video_thumbnail_tile](../usage/processing.mdx#video-thumbnail-tile) (`vtt`) +processing option generates a grid of frames sampled across the video — handy for +building scrubber-preview UIs like the ones used by video players: + +```imgproxy_url +.../video_thumbnail_tile:-1:5:2:160:90/plain/https://example.com/video.mp4@jpg +``` + +This example generates a 5x2 grid of 160x90 tiles. When the `step` argument (the first +one) is `0` or negative, imgproxy spreads the tiles evenly across the remaining +duration of the video, calculating the actual step as +`(video_duration - start) / (columns * rows)`. + +For videos with long steps between tiles, the +[IMGPROXY_VIDEO_THUMBNAIL_TILE_AUTO_KEYFRAMES](../configuration/options.mdx#IMGPROXY_VIDEO_THUMBNAIL_TILE_AUTO_KEYFRAMES) +config makes imgproxy automatically switch to keyframe-only seeking once the step +exceeds the keyframe interval, trading a bit of precision for a significant +speedup. + +## Animated previews + +The [video_thumbnail_animation](../usage/processing.mdx#video-thumbnail-animation) +(`vta`) processing option generates an animated preview instead of a static grid, +sampling frames across the video the same way tiles do: + +```imgproxy_url +.../video_thumbnail_animation:-1:500:10:320:180/plain/https://example.com/video.mp4@gif +``` + +This example generates a 10-frame, 320x180 animation with a 500ms delay between +frames, spread evenly across the video's duration. + +[video_thumbnail_tile](../usage/processing.mdx#video-thumbnail-tile) and +[video_thumbnail_animation](../usage/processing.mdx#video-thumbnail-animation) are +mutually exclusive — setting one clears any previously set value of the other so that +a single request can produce only a tile sheet or an animation, not both. + +## Fetching video metadata + +The [`/info` endpoint](../usage/getting_info.mdx)'s +[video_meta](../usage/getting_info.mdx#video-meta) (`vm`) option returns per-stream +metadata for video sources — codec, duration, bitrate, frame rate, and more — which is +useful for building a UI that picks a sensible thumbnail timestamp before requesting +it: + +```imgproxy_url +.../info/video_meta:1/plain/https://example.com/video.mp4 +``` + +## Configuration + +* [`IMGPROXY_ENABLE_VIDEO_THUMBNAILS`]: when `true`, enables video thumbnail generation. Default: `false` +* [`IMGPROXY_VIDEO_THUMBNAIL_SECOND`]: the timestamp (in seconds) used for a thumbnail. Default: `1` +* [`IMGPROXY_VIDEO_THUMBNAIL_KEYFRAMES`]: when `true`, uses the nearest preceding keyframe instead of the exact requested frame. Default: `false` +* [`IMGPROXY_VIDEO_THUMBNAIL_TILE_AUTO_KEYFRAMES`]: when `true`, automatically switches tiles to keyframes-only seeking once the `step` exceeds the interval between keyframes. Default: `false` +* [`IMGPROXY_VIDEO_THUMBNAIL_PROBE_SIZE`]: the maximum number of bytes used to probe the container format. Default: `5000000` +* [`IMGPROXY_VIDEO_THUMBNAIL_MAX_ANALYZE_DURATION`]: the maximum number of milliseconds spent analyzing streams. When `0`, a heuristic is used. Default: `0` + +imgproxy Pro can also use VA-API hardware acceleration to speed up video frame +decoding — see the VA-API-specific config variables on the +[Hardware acceleration](hardware_acceleration.mdx) page. + +Check out the [Video thumbnails](../configuration/options.mdx#video-thumbnails) +config reference for the complete list of related config variables. diff --git a/docs/image_formats_support.mdx b/docs/image_formats_support.mdx index d6b8cc7..1a39cfc 100644 --- a/docs/image_formats_support.mdx +++ b/docs/image_formats_support.mdx @@ -85,12 +85,11 @@ Since MP4 requires use of a `