Fix NVFP4 multi-GPU export device handling - #2197
Conversation
Signed-off-by: realAsma <akuriparambi@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2197 +/- ##
==========================================
- Coverage 78.97% 78.96% -0.01%
==========================================
Files 522 522
Lines 60606 60612 +6
==========================================
Hits 47862 47862
- Misses 12744 12750 +6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Signed-off-by: realAsma <akuriparambi@nvidia.com>
What does this PR do?
Type of change: Bug fix
NVFP4 weight export could allocate intermediate tensors on the process's current CUDA device instead of the input tensor's device. This caused export failures when a model shard lived on a different GPU. This change scopes NVFP4 quantization to the input device while restoring the caller's current device afterward.
The export path also now uses the quantizer's public
is_enabledstate instead of inferring that state from its formatted representation.Usage
N/A — this fixes existing multi-GPU export behavior without changing the public API.
Testing
pre-commit run --files modelopt/torch/export/unified_export_hf.py modelopt/torch/quantization/qtensor/nvfp4_tensor.py tests/gpu/torch/quantization/test_qtensor_cuda.py tests/unit/torch/export/test_export_weight.pypytest tests/unit/torch/export/test_export_weight.py::test_export_quantized_weight_does_not_repr_input_quantizer -q -xCUDA_VISIBLE_DEVICES=1,2 pytest tests/gpu/torch/quantization/test_qtensor_cuda.py::TestQTensor::test_nvfp4_export_uses_input_device -q -xBefore your PR is "Ready for review"
Make sure you read and follow Contributor guidelines and your commits are signed (
git commit -s -S).Make sure you read and follow the Security Best Practices (e.g. avoiding hardcoded
trust_remote_code=True,torch.load(..., weights_only=False),pickle, etc.).CONTRIBUTING.md: N/AAdditional Information
No new dependency or public API is introduced.