Repository navigation
Loosen the late-iteration gradient tolerance in the checkpointing test - #8648
Merged
Merged
Conversation
The multiple-backward checkpointing test compares DDP and engine gradients over three optimizer iterations at dtype-default tolerances. The engine keeps fp32 accounting while the DDP reference steps the bf16 weights directly, so the two paths diverge by roughly one bf16 ulp of weight per step even from exactly equal gradients; by iteration 2 that crosses a reduction/relu rounding boundary and produces ~8e-3 absolute gradient differences on O(1) gradients. Whether the difference survives bf16 quantization depends on the CPU bf16 kernel's rounding, so the test flips between pass and fail across torch versions (fails on 2.10/2.11, passes on 2.13) without any behavioral change. Give iterations >= 2 an absolute tolerance floor (atol=1e-2 alongside the bf16-default rtol): the test already documents that late iterations drift, and the floor absorbs quantization-boundary chaos while still catching real gradient errors. Verified on both kernel families: 6/6 fail -> 6/6 pass on torch 2.11, unchanged 6/6 pass on torch 2.13. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
pengdurice
approved these changes
Sep 24, 2026
Merged
via the queue into
deepspeedai:master
with commit Sep 24, 2026
a4490b2
13 of 15 checks passed
yermakoffivan
pushed a commit
to yermakoffivan/deepspeed
that referenced
this pull request
Sep 28, 2026
…dai#8684) ## Description Every fp16-config test that reaches `deepspeed.initialize` crashes its sanity check (`Type fp16 is not supported on your device.`) on accelerators whose `is_fp16_supported()` is false. On CPU that maps to the AVX512-FP16 capability of the host, and GitHub's `ubuntu-24.04` runners are hardware-heterogeneous, so these tests flip between failure and skip depending on which runner they land on. deepspeedai#8398 added the first skipifs; the multi-rank CPU run in deepspeedai#8381 flushed out six more files: | File | Shape of the gap | |---|---| | `checkpoint/test_universal_checkpoint.py` | fp16 parametrizations fail **inside the baseline DistributedFixture's distributed run** — pytest reports a setup **ERROR** for every dependent test (48 on the multi-rank run) instead of a skip | | `v1/zero/test_zero_coalesce_grad_reduction.py` | `TestCoalesceFP16` forces an fp16 config | | `runtime/test_no_sync_ctxt.py` | dtype=float16 parametrizations of three methods; stages 2/3 additionally never reach their expected no_sync AssertionError on such hosts | | `checkpoint/test_moe_checkpoint.py` | whole class hardcodes fp16 | | `runtime/zero/test_zero_offloadpp.py` | `TestZeroPartialOffloadConfigSweep` hardcodes fp16 | | `checkpoint/test_pipeline.py` | fp16 enabled for zero_stage > 0; only that parametrization skips, zero_stage=0 keeps running | With this PR, **every fp16-config test in the suite guards on `is_fp16_supported()`** — the capability gap is a skip, not a failure, on any accelerator. ## Validation (executed on real hardware) - 20-core x86_64 CPU without AVX512-FP16, gloo, 2-4 ranks (`LOCAL_SIZE=2/4`) - Before: 18 failures + 48 setup ERRORs across these files on the multi-rank CPU run - After: every affected parametrization skips; adjacent non-fp16 parametrizations keep passing (e.g. pipeline zero_stage=0 runs to completion) - Full-suite evidence in deepspeedai#8381: the multi-rank CPU run went 8 failures -> 0 with these guards Sibling PRs from the same series: deepspeedai#8397, deepspeedai#8398, deepspeedai#8399, deepspeedai#8407, deepspeedai#8409, deepspeedai#8559, deepspeedai#8648. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
meefs
pushed a commit
to meefs/DeepSpeed
that referenced
this pull request
Oct 2, 2026
…es (deepspeedai#8702) ## Description Two latent bugs in Ulysses SP, both backend-independent (neither is CPU-specific — the CPU/gloo multi-rank run in deepspeedai#8381 is simply what surfaced them), plus test adaptations so the file can also run on non-CUDA accelerators. ### 1. `register_with_transformers` rejected every string model path `deepspeed/runtime/sequence_parallel/ulysses_sp.py` compared `core_attn_implementation` against the config's `_attn_implementation` even when the caller passed a **model path string**: ```python hf_model_config = AutoConfig.from_pretrained(model_name_or_path) # string branch model_attn_implementation = getattr(hf_model_config, "_attn_implementation", None) if model_attn_implementation is not None and model_attn_implementation != core_attn_implementation: raise ValueError(...) ``` In transformers, `_attn_implementation` is a **property resolved at model load time** — `from_pretrained` resolves which implementation the runtime can actually use and writes it back to the config. A bare `AutoConfig` never goes through that resolution, so the property returns its fallback value for "unresolved": ``` AutoConfig.from_pretrained(path)._attn_implementation -> eager (fallback, not a choice) AutoModelForCausalLM.from_pretrained(path).config._attn... -> sdpa (resolved) ``` So every string-path caller asking for `sdpa`/`flex_attention`/FA2 was rejected with a spurious mismatch against `'eager'`. The check is only meaningful for a config that was resolved, i.e. when the caller handed us a constructed model (or a PEFT wrapper exposing `.config`); scope it to that case. The comparison expression itself is unchanged, and `TestUlyssesSPHFAttnImplMismatch` (which passes a model object) still raises as intended. ### 2. `UlyssesSPDataLoaderAdapter` used a 0-dim tensor in `all_gather` ```python seqlen = torch.tensor(n, ...) # 0-dim seqlens = [torch.zeros(1, ...) for _ in range(sp_world_size)] # shape (1,) dist.all_gather(seqlens, seqlen, group=self.sp_group) ``` `c10d.all_gather` requires each receive tensor to have the same size as the send tensor. NCCL moves flattened bytes and tolerates the mismatch; gloo validates shapes and rejects it (`ProcessGroupGloo::allgather: invalid tensor size at index 0 (expected (), got (1))`). Send a 1-element tensor to match. ### 3. Tests - `TestUlyssesSPHFFlexAttention` needs the CUDA flex kernel -> skip on non-CUDA accelerators (same guard idiom as `tests/unit/util.py`, the nvme/triton CUDA-only tests). - `TestUlyssesSPHFPEFT`: the mock attached a **bare** `AutoConfig`, whose `_attn_implementation` is the unresolved `'eager'` — so this test failed on every backend, not just CPU. Give the mock the load-resolved value a real PEFT wrapper carries (the sibling `TestUlyssesSPHFAttnImplMismatch` in the same file already sets it the same way). - `TestUlyssesSPHFDisableInEval` pinned its tensors/models to `cuda:<rank>` although it runs sdpa and nothing in it is CUDA-specific. Use `get_accelerator().current_device_name()` like the rest of the file (on CUDA that is exactly `cuda:<rank>`, since the test harness sets the device per rank in `tests/unit/common.py`), so the test runs on any backend instead of erroring. ## Validation Hardware: 20-core x86_64 CPU without AVX512-FP16, gloo backend, 2 ranks (`LOCAL_SIZE=2`), transformers 4.51.3, torch 2.13. - `tests/unit/ulysses_alst/test_ulysses_sp_hf.py`: **5 passed, 2 skipped** (flex skipped by the new guard). The sdpa classes pass end to end including their numerical-parity assertions; before the fix they died at registration. - `TestUlyssesSPHFDisableInEval` now actually **executes** on CPU rather than being skipped. - `TestUlyssesSPHFAttnImplMismatch` still raises the expected `ValueError` (the mismatch check is preserved where a resolved config exists). - Full multi-rank CPU CI in deepspeedai#8381: this file went from 5 failures to 0, with the overall job green. (`TestUlyssesSPHFHubKernel` is not exercised by the version pin above — it needs the `lazy_import_flash_attention` API from a newer transformers; it passes in CI, which installs transformers from git main.) Sibling PRs from the same series: deepspeedai#8397, deepspeedai#8398, deepspeedai#8399, deepspeedai#8407, deepspeedai#8409, deepspeedai#8559, deepspeedai#8648, deepspeedai#8684. --------- Signed-off-by: Guokai Ma <guokai.ma@intel.com> Co-authored-by: pengdurice <pengduhit@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
TestZeroUserBackwardWithCheckpointing::test_checkpointed_multiple_backwardcompares DDP and engine gradients over three optimizer iterations at dtype-default tolerances, and flips between pass and fail across torch versions on cpu (fails on 2.10/2.11 kernels, passes on 2.13) without any behavioral change.A minimal two-rank repro isolates the mechanism:
The engine keeps fp32 accounting while the DDP reference steps the bf16 weights directly, so the two mathematically-equivalent paths diverge by ~one bf16 ulp of weight per optimizer step even from bit-identical gradients; by iteration 2 that crosses a reduction/relu rounding boundary. Whether the difference survives bf16 quantization depends on the CPU bf16 kernel rounding — hence the version dependence.
This PR gives iterations >= 2 an absolute tolerance floor (
atol=1e-2alongside the bf16-defaultrtol=1.6e-2). The test already documents "small differences at later iterations are expected due to bfloat16 precision"; the floor absorbs quantization-boundary chaos while still catching real gradient errors.Validation (executed on real hardware)
LOCAL_SIZE=2)Exposed by the
LOCAL_SIZE=4multi-rank CPU run in #8381.