Add offline SpinQuant/QuaRot rotation folding and learning (R1/R2)

.pre-commit-config.yaml

@@ -102,6 +102,7 @@ repos:
         # Instead, we should manually add the license header to those files *after* the original header.
         exclude: >
           (?x)^(
+              modelopt/torch/quantization/rotation/sgdg.py|
               modelopt/torch/quantization/utils/calib_utils.py|
               modelopt/onnx/quantization/operators.py|
               modelopt/onnx/quantization/ort_patching.py|

CHANGELOG.rst

@@ -8,6 +8,7 @@ Changelog
 
 *Quantization*
 
+- Add ``modelopt.torch.quantization.rotation`` — offline SpinQuant/QuaRot rotation folding (``fold_rotations``) and Cayley-SGD rotation learning (``learn_rotations``) of a global R1 plus per-layer R2 as a pre-quantization checkpoint transform for HF RMSNorm decoder LMs (Llama family, Qwen3). The rotated model is functionally identical (fp32 logits agree to ~3e-7) and remains a vanilla HF checkpoint, so every existing quant config, calibrator, exporter, and runtime works on it unchanged; online R3/R4 Hadamard transforms are out of scope.
 - Add the ``nvfp4_act_headroom`` calibration algorithm for NVFP4 **activation** global scales. Instead of setting the global scale from the largest per-block amax seen during calibration (plain ``max``, which leaves no room above it so any larger activation saturates), it anchors the scale to a low percentile of the per-block amax distribution, leaving the rest of the FP8 block-scale range as headroom: ``amax = max(rho * anchor, upper)``, where ``anchor`` and ``upper`` are the per-block amaxes at ``anchor_percentile`` (default 1) and ``upper_percentile`` (default 99.99; set to 100 to never clip calibration data), and ``rho`` (default 16384) is the headroom factor. Applies only to NVFP4 dynamic-block input quantizers; ``SequentialQuantizer`` activation quantizers raise. Weight scales are an orthogonal axis selected by a nested ``weight_scale_algorithm`` (``max`` by default, or ``mse`` / ``local_hessian``), so one recipe can combine a weight calibration with this activation policy in a single pass. Ships ``modelopt_recipes/general/ptq/nvfp4_act_headroom-kv_fp8_cast.yaml``, which mirrors ``nvfp4_default-kv_fp8_cast`` with only the calibration algorithm swapped and exports a standard NVFP4 checkpoint.
 
 *Megatron Framework (M-LM / M-Bridge)*

LICENSE

@@ -244,6 +244,7 @@ the following copyright holders, licensed under the MIT License:
   Copyright (c) Andrei Panferov
   Copyright (c) Microsoft Corporation
   Copyright (c) 2020 EleutherAI
+  Copyright (c) 2020 Jun Li
   Copyright (c) 2020 Dan Hendrycks
   Copyright (c) 2023 Deep Cognition and Language Research (DeCLaRe) Lab
   Copyright (c) 2023 DeepSeek

modelopt/torch/quantization/rotation/__init__.py

@@ -0,0 +1,19 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Rotation folding + learning (SpinQuant/QuaRot R1 + R2) as pre-quantization transforms."""
+
+from .fold import *
+from .learn import *