Torch Tweak is an inference toolkit designed for tuning and deploying Deep Learning models with a focus on Intel XPU (GPU) accelerators. It provides model tuning capabilities through compilation and conversion paths that can significantly improve inference speed and efficiency across various AI workloads including Computer Vision, Natural Language Processing, Speech Recognition, and Generative AI.
The toolkit tunes PyTorch models and pipelines through one Python API. Four backends ship with the package:
- TorchAO quantizes weights through PyTorch AO
- Torch Inductor compiles with
torch.compile - OpenVINO converts a module to an OpenVINO model and runs it on an Intel CPU or GPU
- Torch Eager runs the module with no compilation, as the baseline other backends are compared against
The guides under docs/learn/ are the maintained documentation.
This page is the landing summary. The same file is the PyPI long description.
Note: This toolkit is currently in pre-release (pre-1.0) and not yet production-quality. The API is subject to change in future updates.
| Feature | Description |
|---|---|
| Ease-of-use | Single line of code to run all possible tuning paths directly from your source code |
| Wide Backend Support | Compatible with TorchAO, Torch Inductor, OpenVINO, and eager PyTorch |
| Model Tuning | Enhance the performance of models such as ResNET and BERT for efficient inference deployment |
| Pipeline Tuning | Streamline Python code pipelines for models such as Stable Diffusion and Flux using seamless model wrapping and tuning |
| Correctness Testing | Ensures tuned models produce correct outputs by validating on provided data samples |
| Performance Profiling | Profiles models to select the optimal backend based on performance metrics such as latency and throughput |
| Model Persistence | Save and load tuned models for production deployment with flexible storage options |
Torch Tweak provides compute graph optimizations for PyTorch models at the nn.Module level. Use Torch Tweak when you want automated inference optimization with minimal code changes.
If your model is supported by a dedicated serving framework and benefits from runtime optimizations (e.g. continuous batching, speculative decoding), use frameworks like vLLM or SGLang for best performance. Use Torch Tweak for general PyTorch models and pipelines that lack such specialized tooling.
from torch_tweak.torch.backend import (
TorchAOBackend,
TorchInductorBackend,
OpenVINOBackend,
TorchEagerBackend,
)
from torch_tweak.torch.tune_strategy import HighestThroughputStrategy
# The strategy measures every backend and keeps the fastest.
strategy = HighestThroughputStrategy(
backends=[
TorchAOBackend(),
TorchInductorBackend(),
OpenVINOBackend(),
TorchEagerBackend(),
]
)TorchAO, Torch Inductor, and OpenVINO configuration is in the backends guide. Torch Eager accepts an optional autocast dtype and otherwise runs the module unchanged. Tune strategies are in core functionalities.
PyTorch XPU wheels are not on PyPI. Stable is the default stack. Nightly is uv-only. The full rules for switching stacks are in the install guide.
# uv, stable (recommended)
uv sync --extra dev
# pip, editable install with the XPU extra
pip install \
--index-url https://download.pytorch.org/whl/xpu \
--extra-index-url https://pypi.org/simple \
-e ".[xpu,dev]"- Overview
- Install
- Quick start
- Core functionalities: inspect, tune, save, load, and tune strategies
- Backends
- Examples
Intel is committed to respecting human rights and avoiding complicity in human rights abuses. See Intel's Global Human Rights Principles. Intel's products and software are intended only to be used in applications that do not cause or contribute to a violation of an internationally recognized human right.
Please be aware that using this toolkit for very large model optimizations (greater than 10^23 FLOPs) may require additional compliance obligations under the EU AI Act (https://ec.europa.eu/newsroom/dae/redirection/document/118340), depending on usecase and setting.
This project is a port of NVIDIA's AITune library, adapted for Intel hardware. Most of the core logic still belongs to the original authors. Huge credit to the NVIDIA team for their incredible work.