Skip to content

Add Qwen3-TTS data and forward-loss contracts (1/4) - #350

Open
Wallbreazzz wants to merge 2 commits into
modelscope:mainfrom
Wallbreazzz:cookbook/qwen3-tts-01-contracts
Open

Wallbreazzz wants to merge 2 commits into
modelscope:mainfrom
Wallbreazzz:cookbook/qwen3-tts-01-contracts

Conversation

@Wallbreazzz

@Wallbreazzz Wallbreazzz commented Oct 28, 2026 •

Copy link
Copy Markdown

Problem and scope

This first dependency layer adds Qwen3-TTS 12Hz tensor preparation, two-stage forward/loss contracts and CPU tests for local single-Ascend-NPU bf16 LoRA. It is part of RFC #279 and does not close it. Model loading, the training CLI and generation are later layers; no Twinkle core or default dependency changes are included.

The residual Code Predictor must receive the Talker state that predicted the current frame's first codebook, before that frame was consumed. Selecting the hidden state at the target frame exposes its target codebook embeddings during training but not generation. The corrected selection uses the preceding position. Talker still applies its own temporal label shift; residual-codebook heads retain matching per-head labels without an extra codebook-axis shift.

A behavioral regression changes the current frame's residual codes and verifies that its conditioning is unchanged while the following frame's conditioning changes. It fails on the previous implementation. Earlier gradient/update/reload tests did not establish this causal equivalence.

Validation

  • This layer: 11 isolated CPU tests, 9 passed and 2 official-package checks skipped. Run python tests/batch1/test_contracts.py --isolated-twinkle from the cookbook. Infrastructure doubles are not native Twinkle/NPU validation.
  • Cumulative layers 1-5: 11/18/24/38/57 CPU tests, with two official-package skips in each layer. All remaining tests passed.
  • The same corrected forward was exercised in the complete candidate on one Ascend NPU: targeted contracts passed, real-data LoRA training reached 105 updates through checkpoint recovery, and adapter reload plus paired generation completed. This is evidence for the corrected integrated forward, not independent NPU validation of each intermediate commit.
  • Earlier integrated 40-test NPU results predate this correction and are retained as historical evidence, not claimed as a rerun of the updated tree. The exact revised full candidate 791b488 subsequently passed 8 targeted contracts and real paired generation on the same single-NPU environment; both decoded waveforms matched the previously reviewed pair sample for sample. This is a targeted packaging/default-language check, not a rerun of the complete NPU training suite.
  • Small, labeled listening comparisons motivated the correction and subsequent configuration experiments. They do not establish general convergence or quality guarantees. No private data or audio is included in this public PR.

Contribution sequence

  1. This PR: data, forward and loss contracts.
  2. Twinkle model bridge and LoRA integration.
  3. Accumulation and update-boundary recovery.
  4. Manifest preparation and training CLI.
  5. Generation and human review.

The active dependent fork branches include this correction. Later upstream PRs follow prerequisite merges. The closed monolithic draft #349 and its old branch remain historical references and are not the corrected delivery candidate. CUDA, MOSS, multi-card and concurrent multi-LoRA are outside the validated scope.

@Wallbreazzz

Copy link
Copy Markdown
Author

@tastelikefeet 感谢此前在 RFC #279 中帮忙梳理方案,#350 已 ready,方便时请优先帮忙检视这一层的数据与前向/loss契约。验证方面,本层 CPU 契约 8 项通过、2 项官方依赖检查在隔离模式下跳过,完整训练树已通过单昇腾 NPU 的 40 项测试,拆分后的最终 Git tree 与该验收版本完全一致。后续 LoRA 接入、恢复、训练 CLI 已准备好,将在前置 PR 合入后依次提交并吸收您的意见,希望先尽快对齐这一层的接口,减少后续返工。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant