Add single-NPU Qwen3-TTS LoRA training cookbook

README.md

@@ -405,6 +405,7 @@ foundation for building customizable, enterprise-grade training services.
 
 | Component Type | Component Link                                                                                           | Component Function                                                                      | Author              |
 | -------------- | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | ------------------- |
+| Cookbook | [Qwen3-TTS LoRA](cookbook/qwen3_tts/README.md) | Single Ascend NPU training, explicit data splits and update-boundary resume | [Wallbreazzz](https://github.com/Wallbreazzz) |
 | Patch          | [qwen3_moe_transformers4_patch](https://www.modelscope.cn/models/twinkle-kit/qwen3_moe_transformers4_patch) | Fixes Qwen3 MoE model hang issue during FSDP2 training, effective for transformers==4.x | ModelScope Official |
 
 ## Contributions

cookbook/qwen3_tts/.gitignore

@@ -0,0 +1,3 @@
+outputs/
+__pycache__/
+*.py[cod]

cookbook/qwen3_tts/README.md

@@ -0,0 +1,162 @@
+# Qwen3-TTS LoRA training on one Ascend NPU
+
+This cookbook connects the Qwen3-TTS 12Hz Base model to Twinkle's local
+Transformers training API. It provides explicit train/dev preparation,
+single-sample micro-batches, LoRA on the Talker and Code Predictor, accumulation
+with a final short window, and restart at completed optimizer updates.
+
+It is a single-card reference implementation, not a quality-tuned training
+recipe. There are no changes to Twinkle core or its default dependencies.
+Generation, speaker-similarity evaluation and experimental EOS handling are
+outside this contribution.
+
+## Installation and supported scope
+
+Run from the repository root with an existing Ascend PyTorch environment:
+
+```bash
+pip install -e .
+pip install -r cookbook/qwen3_tts/requirements.txt
+cd cookbook/qwen3_tts
+export MODEL_PATH=/models/Qwen3-TTS-12Hz-1.7B-Base
+export ASCEND_RT_VISIBLE_DEVICES=0
+```
+
+The model directory must already contain the complete Base weights, config,
+text tokenizer, processor and `speech_tokenizer/`. Preparation and training are
+offline: missing assets cause an error, not a download. No virtual environment
+or model download is created by these commands. Select one physical card with
+`ASCEND_RT_VISIBLE_DEVICES`; `npu:0` is relative to that selection.
+
+The development baseline used Twinkle `5337cc3`, Qwen3-TTS `022e286`, Python
+3.12, torch 2.10.0, torch_npu 2.10.0.post2, CANN 9.1.0 and an Ascend 910B.
+The optional requirements pin the Qwen implementation and its Transformers
+4.57.3 / accelerate 1.12.0 dependencies. They are not installed for ordinary
+Twinkle users. Match torch/torch_npu to your Ascend image before installing
+these extras. Codec and reference mel extraction use CPU fp32; the TTS model
+uses NPU bf16 with eager attention. Multi-card, CUDA, padding-free execution
+and other Qwen3-TTS model variants are not covered by the current evidence.
+
+## Prepare an existing split
+
+Provide two UTF-8 JSONL files. Each row has `id`, `text`, `audio`, `group_id`
+and `speaker_id`. Paths are relative to the input JSONL file. For example:
+
+```json
+{"id":"train-001","text":"A training utterance.","audio":"audio/train-001.wav","group_id":"session-a","speaker_id":"speaker-a"}
+```
+
+Use a different held-out group and text in the dev file. Audio must be finite
+mono 16 or 24 kHz; 16 kHz is resampled to 24 kHz. The preparer requires one
+speaker, unique IDs, nonempty train/dev sets and a reference chosen from train.
+It rejects cross-split group, normalized text, file, decoded PCM and resampled
+PCM duplicates. Group identifiers still need to reflect the real recording
+source; these checks do not detect all near-duplicates.
+
+```bash
+python scripts/prepare_split_manifest.py \
+  --model_path "$MODEL_PATH" \
+  --train_jsonl /data/splits/train.jsonl \
+  --dev_jsonl /data/splits/dev.jsonl \
+  --reference_id train-001 \
+  --output_dir /data/derived/qwen3-tts
+```
+
+The output directory must be new. The script preserves the assigned split,
+text and order; it never silently resplits or edits source audio. Output rows
+contain `id`, `text`, integer `audio_codes` of shape `[T,16]` in `[0,2048)`,
+and `ref_mel_path`. One train-reference mel `[F,128]` is shared by both splits.
+`provenance.json` records input/output hashes and the source mapping.
+The dev manifest is prepared for downstream evaluation and is **not** consumed
+by this training loop. No dev loss, early stopping or quality score is implied.
+
+## Train and resume
+
+```bash
+python scripts/train_qwen3_tts_lora.py \
+  --model_path "$MODEL_PATH" \
+  --train_jsonl /data/derived/qwen3-tts/train.jsonl \
+  --output_dir outputs/run-1 \
+  --epochs 3 --grad_accum 4 --shuffle --seed 42 \
+  --lr 1e-4 --lr_decay 1.0 --lora_r 8 --lora_dropout 0.1 \
+  --save_interval 10
+```
+
+These are example hyperparameters, not a convergence recommendation. The
+default learning rate stays constant. `--lr_decay` optionally supplies StepLR
+gamma per optimizer update; the short resume regression explicitly uses 0.8
+to exercise scheduler restoration. LoRA targets `q_proj`/`v_proj` in both
+branches, with alpha equal to twice the rank. All base parameters are frozen.
+
+Micro-batch size is one. The loop owns accumulation windows and keeps
+Twinkle's group accumulation at one; `clip_grad_and_step` normalizes by the
+actual number of micro-batches, including an epoch's short final window.
+The speaker encoder runs without gradients. The Talker loss uses its internal
+temporal shift; residual-codebook logits are supervised against corresponding
+codebooks 1–15 without another shift along the codebook axis. Tests check
+last-head gradients and the official residual assembly interface. This loss
+choice is not a claim about how official pretrained weights were trained.
+
+For a controlled stop, add `--stop_after_updates 1`. Resume to a **new** output
+directory with the same training options:
+
+```bash
+python scripts/train_qwen3_tts_lora.py \
+  --model_path "$MODEL_PATH" \
+  --train_jsonl /data/derived/qwen3-tts/train.jsonl \
+  --output_dir outputs/run-2 --resume_from outputs/run-1/checkpoint-update-1 \
+  --epochs 3 --grad_accum 4 --shuffle --seed 42 \
+  --lr 1e-4 --lr_decay 1.0 --lora_r 8 --lora_dropout 0.1 \
+  --save_interval 10
+```
+
+Checkpoints include adapter, optimizer, scheduler, RNG and loop progress.
+Resume checks content fingerprints of model/manifest/reference mel, sample
+plan and settings; it fails on mismatch instead of silently starting over.
+Model hashing reads the local weight files and may take time. Only complete,
+trusted checkpoints produced by this workflow should be resumed. Saving is
+at update boundaries; recovery from arbitrary interruption during a checkpoint
+write is not guaranteed. Inspect `trace.jsonl`, `run_config.json` and
+`result.json` for the actual update count and checkpoint path.
+
+## Tests and evidence boundaries
+
+With the optional dependencies installed:
+
+```bash
+python tests/run.py
+MODEL_PATH="$MODEL_PATH" python tests/run.py --npu
+```
+
+The first command runs data, forward/loss, projection, manifest, preparation
+and accumulation contracts. It includes official mel/residual-interface tests
+but uses small CPU networks, not the pretrained TTS model. The second also
+loads the real local model for a LoRA update/save/reload, public split
+preparation-to-training, and continuous versus independent-process resume.
+The two tiny LibriSpeech fixtures are attributed under
+`tests/fixtures/librispeech/`; they are regression fixtures, not an independent
+quality benchmark. Tests use temporary outputs and never require user data.
+
+For local CPU development without Twinkle or Qwen installed, install torch,
+Transformers, NumPy, scipy and soundfile, then run `python tests/run.py
+--isolated`. This mode explicitly substitutes Twinkle infrastructure and skips
+two official-package checks. It is not NPU or native Twinkle validation.
+
+Before upstream packaging, real 1.7B single-NPU runs covered nonzero LoRA
+gradients/updates, unchanged frozen parameters, exact saved/reloaded adapter
+weights, accumulation tails and restart in separate processes. Final adapter
+weights matched exactly in the resume comparison; optimizer states were
+compared with rtol=1e-3/atol=1e-5. Those results do not establish long-training
+stability, speaker similarity or fine-tuning quality. The reorganized PR tree
+requires its own final NPU regression before the draft is marked ready.
+
+## Integration boundaries
+
+The local `tts/` package is added to the script path; it is not registered as
+a new global Twinkle task. The subclass reuses forward/backward, optimizer,
+clipping, saving and resume APIs. It currently overrides
+`_construct_default_optimizer_group` for consistent tokenizer/processor/loss
+defaults and calls `_lazy_wrap_model` when inspecting/loading wrapped weights.
+These internal dependencies are isolated here and should be reviewed when
+Twinkle changes. A public generic TTS abstraction should only be added if
+maintainers identify a reusable need; this cookbook does not add one prematurely.

cookbook/qwen3_tts/requirements.txt

@@ -0,0 +1,8 @@
+accelerate==1.12.0
+numpy>=2.0,<2.3
+peft==0.17.1
+qwen-tts @ git+https://github.com/QwenLM/Qwen3-TTS.git@022e286b98fbec7e1e916cb940cdf532cd9f488e
+safetensors
+scipy
+soundfile
+transformers==4.57.3

cookbook/qwen3_tts/scripts/audio_features.py

@@ -0,0 +1,42 @@
+"""Official 24 kHz mel extraction and 16-codebook audio encoding."""
+
+import numpy as np
+
+MEL_DIM = 128
+MEL_HOP = 256
+MEL_WIN = 1024
+MEL_FMAX = 12000.0
+TARGET_SR = 24000
+
+def extract_mel(wav_24k: np.ndarray) -> np.ndarray:
+    """使用官方 mel 算法，返回单条 [F,128]，不使用替代 fbank。"""
+    import torch
+    from qwen_tts.core.models.modeling_qwen3_tts import mel_spectrogram
+
+    wav = np.asarray(wav_24k, dtype=np.float32)
+    if wav.ndim != 1:
+        raise ValueError('Expected mono waveform [samples]')
+    with torch.no_grad():
+        mel = mel_spectrogram(
+            torch.from_numpy(wav).unsqueeze(0), n_fft=MEL_WIN,
+            num_mels=MEL_DIM, sampling_rate=TARGET_SR, hop_size=MEL_HOP,
+            win_size=MEL_WIN, fmin=0, fmax=MEL_FMAX,
+        ).transpose(1, 2)
+    return mel[0].cpu().numpy()
+
+def encode_audio_codes(wav_24k: np.ndarray, tokenizer) -> np.ndarray:
+    """单条输入必须返回一个 [T,16]；兼容列表、二维张量和单批张量。"""
+    result = tokenizer.encode(wav_24k, sr=TARGET_SR)   # wav_24k 已升采样到 24kHz
+    codes = result.audio_codes
+    if isinstance(codes, (list, tuple)):
+        if len(codes) != 1:
+            raise ValueError('Single waveform must return exactly one audio_codes entry')
+        codes = codes[0]
+    if hasattr(codes, 'cpu'):
+        codes = codes.cpu().numpy()
+    codes = np.asarray(codes)
+    if codes.ndim == 3 and codes.shape[0] == 1:
+        codes = codes[0]
+    if codes.ndim != 2 or codes.shape[0] == 0 or codes.shape[1] != 16:
+        raise ValueError(f'Expected nonempty audio_codes [T,16], got {codes.shape}')
+    return codes.astype(np.int64)

cookbook/qwen3_tts/scripts/prepare_split_manifest.py

@@ -0,0 +1,128 @@
+"""Preserve an explicitly assigned single-speaker train/dev split; never resplit."""
+import argparse
+import hashlib
+import json
+import math
+import os
+from pathlib import Path
+import unicodedata
+
+import numpy as np
+
+
+def sha(data):
+    return hashlib.sha256(data).hexdigest()
+
+
+def read_audio(path):
+    import soundfile as sf
+    from scipy.signal import resample_poly
+    wave, sr = sf.read(path, dtype='float32')
+    if sr not in (16000, 24000) or wave.ndim != 1 or not len(wave) or not np.isfinite(wave).all():
+        raise ValueError('Expected finite mono 16/24 kHz audio')
+    original = sha(np.asarray(wave, dtype='<f4').tobytes() + str(sr).encode())
+    if sr != 24000:
+        divisor = math.gcd(sr, 24000)
+        wave = resample_poly(wave, 24000 // divisor, sr // divisor).astype(np.float32)
+    return wave, original
+
+
+def audit_split(train_path, dev_path, reference_id):
+    splits, seen, speakers = {}, {}, set()
+    for split, source in [('train', train_path), ('dev', dev_path)]:
+        source = Path(source).resolve(strict=True)
+        rows = []
+        for line_no, line in enumerate(source.read_text(encoding='utf-8').splitlines(), 1):
+            if not line.strip():
+                continue
+            row = json.loads(line)
+            for key in ('id', 'text', 'group_id', 'speaker_id', 'audio'):
+                if not isinstance(row.get(key), str) or not row[key].strip():
+                    raise ValueError(f'Missing/non-string {key} at {source}:{line_no}')
+            audio = (source.parent / row['audio']).resolve(strict=True)
+            wave, pcm_hash = read_audio(audio)
+            digest = sha(audio.read_bytes())
+            text_key = ' '.join(unicodedata.normalize('NFKC', row['text']).casefold().split())
+            identities = dict(id=row['id'], group=row['group_id'], text=text_key,
+                              file=digest, pcm=pcm_hash,
+                              resampled=sha(np.asarray(wave, dtype='<f4').tobytes()))
+            for kind, value in identities.items():
+                previous = seen.get((kind, value))
+                if previous and (kind == 'id' or previous != split):
+                    raise ValueError(f'Duplicate ID or cross-split overlap: {kind}')
+                seen[kind, value] = split
+            speakers.add(row['speaker_id'])
+            rows.append(dict(row, audio=str(audio), audio_sha256=digest, pcm_sha256=pcm_hash,
+                             source_manifest=str(source), source_line=line_no))
+        if not rows:
+            raise ValueError(f'{split} must not be empty')
+        splits[split] = rows
+    if len(speakers) != 1:
+        raise ValueError('This preparation entry supports exactly one speaker')
+    references = [r for r in splits['train'] if r['id'] == reference_id]
+    if len(references) != 1:
+        raise ValueError('Reference must be an explicit training sample ID')
+    return splits, references[0]
+
+
+def prepare(train_path, dev_path, output_dir, reference_id, encode, extract_mel):
+    splits, reference = audit_split(train_path, dev_path, reference_id)
+    out = Path(output_dir)
+    out.mkdir(parents=True, exist_ok=False)
+    ref_wave, _ = read_audio(reference['audio'])
+    mel = np.asarray(extract_mel(ref_wave))
+    if mel.ndim != 2 or mel.shape[0] == 0 or mel.shape[1] != 128 or not np.isfinite(mel).all():
+        raise ValueError('Expected finite reference mel [F,128]')
+    np.save(out / 'reference.npy', mel)
+    mapping = []
+    for split, rows in splits.items():
+        encoded = []
+        for row in rows:
+            if sha(Path(row['audio']).read_bytes()) != row['audio_sha256']:
+                raise ValueError('Source audio changed after audit')
+            wave, _ = read_audio(row['audio'])
+            codes = np.asarray(encode(wave))
+            if (codes.ndim != 2 or codes.shape[0] == 0 or codes.shape[1] != 16 or
+                    codes.dtype.kind not in 'iu' or (codes < 0).any() or (codes >= 2048).any()):
+                raise ValueError('Expected integer 12Hz codes [T,16] in [0,2048)')
+            encoded.append(dict(id=row['id'], text=row['text'], audio_codes=codes.tolist(),
+                                ref_mel_path='reference.npy'))
+            mapping.append(dict(row, split=split, reference_id=reference_id,
+                                output_manifest=f'{split}.jsonl'))
+        (out / f'{split}.jsonl').write_text(''.join(json.dumps(r, ensure_ascii=False) + '\n' for r in encoded), encoding='utf-8')
+    report = dict(format_version=1, counts={k: len(v) for k, v in splits.items()},
+                  reference_id=reference_id, reference_split='train', mapping=mapping,
+                  source_manifest_sha256={k: sha(Path(p).read_bytes()) for k, p in
+                                          [('train', train_path), ('dev', dev_path)]},
+                  output_sha256={n: sha((out / n).read_bytes()) for n in
+                                 ('train.jsonl', 'dev.jsonl', 'reference.npy')})
+    (out / 'provenance.json').write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding='utf-8')
+    return report
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    for name in ('train_jsonl', 'dev_jsonl', 'output_dir', 'reference_id', 'model_path'):
+        parser.add_argument('--' + name, required=True)
+    args = parser.parse_args()
+    # Audit before loading weights or creating output. No automatic split/filter fallback.
+    audit_split(args.train_jsonl, args.dev_jsonl, args.reference_id)
+    os.environ['HF_HUB_OFFLINE'] = '1'
+    os.environ['TRANSFORMERS_OFFLINE'] = '1'
+    import torch
+    from qwen_tts.core.models.modeling_qwen3_tts import Qwen3TTSTokenizer
+    from audio_features import encode_audio_codes, extract_mel
+    codec = Qwen3TTSTokenizer.from_pretrained(str(Path(args.model_path) / 'speech_tokenizer'),
+        local_files_only=True, device_map='cpu', torch_dtype=torch.float32)
+    codec.model.float().eval()
+    with torch.no_grad():
+        report = prepare(args.train_jsonl, args.dev_jsonl, args.output_dir, args.reference_id,
+                         lambda wave: encode_audio_codes(wave, codec), extract_mel)
+    from train_qwen3_tts_lora import model_fingerprint
+    report['codec_content_sha256'] = model_fingerprint(Path(args.model_path) / 'speech_tokenizer')
+    (Path(args.output_dir) / 'provenance.json').write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding='utf-8')
+    print('准备完成 / Preparation completed: ' + json.dumps(report['counts']), flush=True)
+
+
+if __name__ == '__main__':
+    main()