examples: add Qwen3-8B streaming DSpark launcher example + walkthrough - #2164
examples: add Qwen3-8B streaming DSpark launcher example + walkthrough#2164h-guo18 wants to merge 2 commits into
Conversation
Adds the missing Qwen3-8B streaming DSpark pipeline. DSpark examples so far covered only large MoE targets (Kimi-K2.6, MiniMax-M3, both multi-node); this is the small-scale entry point that runs on 2 nodes x 1 GPU, so the pipeline can be exercised end-to-end before scaling up. - tools/launcher/examples/Qwen/Qwen3-8B/hf_streaming_dspark.yaml Combines the MiniMax-M3 DSpark streaming settings with the Qwen3-8B DFlash streaming topology. Keeps batch size and LR at the dspark.yaml defaults rather than the M3 warm-start values, which assume 8 GPUs/node. - examples/speculative_decoding/QWEN3_8B_DSPARK_WALKTHROUGH.md End-to-end handoff doc: data synthesis (hf_synth.yaml) -> streaming training -> evaluation, plus the failure modes that are silent (capture-id off-by-one, non-inherited draft dims, synthesis yield loss). Pre-commit hooks applicable to these two files were run and pass (yamlfmt, check-launcher-yaml, markdownlint-cli2, insert-license). Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughThe PR adds a Qwen3-8B DSpark walkthrough and launcher pipeline. It covers target-data synthesis, streaming hidden-state training, configuration, scaling, exported-drafter evaluation, smoke testing, and troubleshooting. ChangesQwen3-8B DSpark workflow
Estimated code review effort: 3 (Moderate) | ~20 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 6✅ Passed checks (6 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Warning
CodeRabbit couldn't request changes on this pull request because it doesn't have sufficient GitHub permissions.
Please grant CodeRabbit Pull requests: Read and write permission and re-run the review.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tools/launcher/examples/Qwen/Qwen3-8B/hf_streaming_dspark.yaml`:
- Around line 134-136: In hf_streaming_dspark.yaml, replace the mutable image
tags for both vLLM containers at lines 134-136 and 142-156 with immutable image
digests. Preserve a digest that includes the auxiliary-capture fix from vLLM PR
`#46788` for task_1 and a digest providing DSpark support for task_2; update both
container declarations accordingly.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f3865e82-c88d-4575-91d7-a736fab26b97
📒 Files selected for processing (2)
examples/speculative_decoding/QWEN3_8B_DSPARK_WALKTHROUGH.mdtools/launcher/examples/Qwen/Qwen3-8B/hf_streaming_dspark.yaml
| # Needs the aux-capture fix (vllm#46788), in-tree in recent nightlies; | ||
| # without it the final capture id is off by one and caps acceptance length. | ||
| container: vllm/vllm-openai:latest |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Record the merge commit required by the training task.
curl -fsSL https://api.github.com/repos/vllm-project/vllm/pulls/46788 |
jq '{state, merged_at, merge_commit_sha: .merge_commit_sha.sha}'
# Record the current mutable-tag digests before selecting immutable replacements.
token="$(
curl -fsSL \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:vllm/vllm-openai:pull' |
jq -r '.token'
)"
for tag in latest nightly; do
echo "vllm/vllm-openai:${tag}"
curl -fsSI \
-H "Authorization: Bearer ${token}" \
-H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
"https://registry-1.docker.io/v2/vllm/vllm-openai/manifests/${tag}" |
tr -d '\r' |
awk 'tolower($1) == "docker-content-digest:" { print $2 }'
doneRepository: NVIDIA/Model-Optimizer
Length of output: 211
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="tools/launcher/examples/Qwen/Qwen3-8B/hf_streaming_dspark.yaml"
sed -n '120,165p' "$file"
printf '\nAll vLLM image references in the file:\n'
rg -n 'container:|vllm|speculative|dspark|46788' "$file"Repository: NVIDIA/Model-Optimizer
Length of output: 3486
🌐 Web query:
vLLM pull request 46788 auxiliary capture DSpark speculative decoding vllm-openai Docker image tags
💡 Result:
The information regarding the items in your query is synthesized below based on recent vLLM developments: Pull Request #46788 Pull request #46788 in the vllm-project/vllm repository is titled "benchmark refactor" [1]. It is a development-related PR focused on infrastructure improvements rather than the DSpark speculative decoding feature [1]. DSpark Speculative Decoding and Auxiliary Capture DSpark is a state-of-the-art parallel drafting algorithm recently integrated into vLLM [2]. Key details include: - Functionality: DSpark builds upon a DFlash backbone by adding a lightweight autoregressive correction head and a confidence estimator to selectively forward tokens, improving verification efficiency [2]. - Implementation: It utilizes non-causal sliding-window attention by repurposing SparseMLA backends [3]. - Auxiliary Capture: DSpark requires capturing auxiliary hidden states from the target model's layers to inform the drafter [4][5]. While initial implementations were restricted to specific configurations (e.g., pipeline parallel size 1) [4], recent efforts have focused on enabling "aux-state forwarding" to support speculative decoding under pipeline parallelism [5]. vLLM-OpenAI Docker Image Tags vLLM does not maintain a specific "vllm-openai" Docker image tag. Official vLLM Docker images are typically hosted on platforms like Docker Hub (under vllm/vllm). The OpenAI-compatible API server is a core component of the standard vLLM image, enabled by default when running the vLLM entrypoint (e.g., vllm.entrypoints.openai.api_server) [2]. Users should refer to the official vLLM documentation or the vLLM Docker Hub repository for the latest versioned tags (e.g., latest, 0.x.x). For tracking specific features across releases, you can use the official vLLM PR release lookup tool at https://vllm.ai/pr-lookup [6].
Citations:
- 1: https://github.com/vllm-project/vllm/actions/runs/21125154011
- 2: https://vllm.ai/blog/2026-07-28-speculators-parallel-drafting
- 3: [Spec Decode] DSpark vllm-project/vllm#46995
- 4: [Spec Decode] Add DeepSeek V4 DSpark speculative decoding vllm-project/vllm#46965
- 5: Feat/spec decode under pipeline parallel vllm-project/vllm#50514
- 6: https://vllm.ai/pr-lookup
🌐 Web query:
site:docs.vllm.ai DSpark speculative decoding vLLM Docker image nightly
💡 Result:
DSpark is a speculative decoding algorithm supported by the vLLM ecosystem [1][2]. It is part of the "Speculators" project—a dedicated component within the vLLM ecosystem for speculative decoding algorithms—rather than a feature directly embedded in the main vLLM core engine's base configuration in the same manner as methods like MTP or EAGLE [3][1][2]. There is no specific "DSpark nightly" Docker image. Instead, vLLM provides official nightly Docker images, such as vllm/vllm-openai:nightly, which contain the latest development branch of the vLLM codebase [4]. Because DSpark is managed within the broader vLLM speculative decoding infrastructure (specifically the speculators project), users testing DSpark typically use the latest vLLM nightly builds to ensure they have the most recent implementation, features, and fixes [4][5]. To use DSpark or other speculative decoding methods, you should: 1. Pull the standard vLLM nightly Docker image: docker pull vllm/vllm-openai:nightly [4]. 2. Refer to the vLLM speculative decoding documentation to configure your serving parameters [6][7]. 3. Ensure your environment meets the requirements for the specific algorithm, as DSpark and other speculative methods are actively evolving [1][2]. If you are developing or testing specific experimental features from the speculators project, you may need to build the Docker image from source using the vLLM Dockerfile to include custom dependencies or specific commits if they have not yet been merged into the main nightly release [4][5][8].
Citations:
- 1: https://docs.vllm.ai/projects/speculators/en/latest/reference/speculators/models/dspark/
- 2: https://docs.vllm.ai/projects/speculators/en/latest/user_guide/algorithms/
- 3: https://docs.vllm.ai/en/stable/api/vllm/config/speculative/
- 4: https://docs.vllm.ai/en/latest/deployment/docker/
- 5: https://docs.vllm.ai/en/stable/getting_started/installation/cpu/
- 6: https://docs.vllm.ai/en/stable/features/speculative_decoding/#speculative-decoding
- 7: https://docs.vllm.ai/en/stable/features/speculative_decoding/?q=
- 8: https://docs.vllm.ai/en/stable/contributing/dockerfile/dockerfile/
🏁 Script executed:
#!/bin/bash
set -euo pipefail
for pr in 46788 46965 46995 50514; do
url="https://github.com/vllm-project/vllm/pull/$pr"
printf '\nPR %s:\n' "$pr"
curl -LfsS -A 'Mozilla/5.0' "$url" |
tr '\n' ' ' |
sed -n 's/.*<title>\([^<]*\)<\/title>.*/\1/p'
done
printf '\nDocker Hub repository lookup:\n'
curl -LfsS -A 'Mozilla/5.0' \
'https://hub.docker.com/v2/repositories/vllm/vllm-openai/tags?page_size=5' |
jq -r '.name // .message // empty' 2>/dev/null || trueRepository: NVIDIA/Model-Optimizer
Length of output: 711
Pin both vLLM containers to immutable image digests.
task_1 requires the auxiliary-capture fix from vLLM PR #46788. task_2 requires DSpark support. The mutable latest and nightly tags can change and cause incorrect capture behavior or startup rejection.
📍 Affects 1 file
tools/launcher/examples/Qwen/Qwen3-8B/hf_streaming_dspark.yaml#L134-L136(this comment)tools/launcher/examples/Qwen/Qwen3-8B/hf_streaming_dspark.yaml#L142-L156
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tools/launcher/examples/Qwen/Qwen3-8B/hf_streaming_dspark.yaml` around lines
134 - 136, In hf_streaming_dspark.yaml, replace the mutable image tags for both
vLLM containers at lines 134-136 and 142-156 with immutable image digests.
Preserve a digest that includes the auxiliary-capture fix from vLLM PR `#46788`
for task_1 and a digest providing DSpark support for task_2; update both
container declarations accordingly.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2164 +/- ##
==========================================
- Coverage 78.73% 78.54% -0.20%
==========================================
Files 522 522
Lines 60357 60692 +335
==========================================
+ Hits 47523 47670 +147
- Misses 12834 13022 +188
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Validated the walkthrough end-to-end on a Slurm cluster (build -> streaming
train -> export). Training and export work as described; the corrections below
are things that only surface when you actually follow the doc.
Blocking setup gaps:
- HF_TOKEN was unlisted, but the prompt corpus is gated; without it Step 1 dies
as DatasetNotFoundError, which reads like a wrong dataset name.
- SLURM_HOST=localhost does not work: the launcher stages artifacts over SSH
even when submitting locally, so sbatch reads from a path nothing populated.
- SLURM_JOB_DIR must pre-exist, and launch.py needs 'uv pip install -e .'
first; both fail with errors that name neither cause.
- Note the identity=<ssh_key> argument, and the nvcr.io pull failure mode.
Content corrections:
- Synthesis writes shard_{id}.jsonl + a .done sentinel; resume needs both and
is whole-shard only, so an interrupted shard restarts from zero.
- Added a runtime estimate (hours per shard) - the most useful thing to know
before launching Step 1.
- The shipped --max-model-len leaves no headroom for this corpus, the exact
failure the doc warns about; say so and give the grep to size it.
- Capture ids are tied to dflash_architecture_config.num_hidden_layers, not a
fixed count of 5.
- Dropped dflash_loss_decay_factor: dflash_loss_objective defaults to 'dpace',
which ignores it outright (the trainer logs a warning saying so).
- data.mode= is inert; streaming is selected by data.streaming_server_url,
injected at runtime by the launcher script.
- On an older vLLM, task_2 fails as AttributeError on 'hc_mult' from the
DeepSeek-V4 DSpark loader, not as a clean method-name rejection.
- Corrected the stale prompt-only-messages hang entry: hf_streaming_dataset
now prefers 'conversations' and skips bad entries rather than stalling.
- Point at /scratchspace/vllm_serve.<n>.log, where serve errors actually land.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>
1 participant
What does this PR do?
Type of change: new example
Adds the missing Qwen3-8B streaming DSpark launcher example, plus an end-to-end walkthrough doc covering data synthesis → streaming training → evaluation.
DSpark examples so far only covered large MoE targets (Kimi-K2.6, MiniMax-M3 — both multi-node, 8 GPU/node). There was no small-scale entry point, so anyone new to the pipeline had to start from a config tuned for a very large warm-start run. This one runs on 2 nodes × 1 GPU, which makes it practical to exercise the whole flow before scaling up.
Two files, both additive — no existing behavior changes:
tools/launcher/examples/Qwen/Qwen3-8B/hf_streaming_dspark.yaml— combines the MiniMax-M3 DSpark streaming settings with the Qwen3-8B DFlash streaming topology. Notably it does not copy M3'sper_device_train_batch_size=4/learning_rate=1e-4/warmup_steps=2000: those assume 8 GPUs per node and a warm-started backbone, and attraining_seq_len=4096on one GPU a batch of 4 will OOM. Batch/LR stay at thedspark.yamldefaults, matching the sibling DFlash example.examples/speculative_decoding/QWEN3_8B_DSPARK_WALKTHROUGH.md— the walkthrough. Covers why synthesis matters for acceptance length, the fields that actually matter, a Qwen3-8B-vs-large-MoE scaling table, and the failure modes that are silent rather than loud (capture-id off-by-one, non-inherited draft dims, synthesis yield loss, prompt-only-corpus hang).Usage
Testing
common/eagle3/train_eagle_streaming.sh(every env knob used here is one the script actually reads).yamlfmt,check-launcher-yaml(validates the recipe/template references resolve),markdownlint-cli2,insert-license.Not yet run as a live Slurm job — worth a
--dryrunon a login node before merge. Flagging that rather than implying end-to-end validation.Before your PR is "Ready for review"
CONTRIBUTING.md: N/Acheck-launcher-yamlhook)Note on
trust_remote_code(re: Security Best Practices):EXPORT_EXTRA_ARGS: "--trust_remote_code"is set intask_1. This is required because DSpark exports a custom modeling file that the exporter must load — it is not applied to the base model, which is stock Qwen3. This matches the existing DFlash/DSpark examples already onmain.Additional Information
The synthesis step uses
tools/launcher/common/query.py, already in-tree — no external tooling needed.Summary by CodeRabbit