ARCHITECTURE.md

ARCHITECTURE.md

Technical map of this repository: what each stage does, how data flows between them, and what's planned next. AGENTS.md is the conventions/how-to-run guide; this file is the "what is this and why is it shaped this way" guide.

Overview

This repository is a staged model-training workspace, split by pipeline stage rather than by model:

pre-training/   PDF corpus  -> OCR text, summaries, layout, synthetic QA (CSVs)
fine-tuning/    text/summary pairs -> trained LoRA adapters
serving/        trained adapters -> inference (FastAPI)
training/       raw datasets -> from-scratch / non-LoRA trained models

Each leaf folder (pre-training/, fine-tuning/<pipeline>/, serving/<pipeline>/) is an independent uv project: its own pyproject.toml, uv.lock, .python-version, and pinned dependency set (in particular, its own CUDA torch build). Nothing is shared at runtime between folders — a pipeline can be deleted or reworked without touching its siblings. One uv binary at the root drives all of them via uv run --directory <folder> ... (see the root README.md's uv Commands section for the current, verified list); there is deliberately no root-level Python project or shared virtualenv, since the folders pin conflicting dependency versions (e.g. different torch builds) that a single shared resolution would fight.

Repo-wide hygiene is intentionally centralized rather than per-folder: one .gitignore at the root (unanchored patterns match every project's drop-zone folders at any depth), and one AGENTS.md at the root covering conventions for the whole repo. No subfolder should have its own copy of any of these.

Why .python-version matters here

Every project pins .python-version to 3.12. Without it, uv run picks the newest CPython it can find (e.g. 3.14), and some pinned dependencies — pillow==10.4.0 in particular — have no prebuilt wheel for that new a version yet, so uv falls back to a from-source build that fails on Windows (missing zlib headers). Pinning 3.12 (already installed and known to have wheels for every pinned dependency across all four projects) is what makes uv run --directory <folder> ... work reproducibly from a clean checkout. This was verified by reproducing the failure and fixing it during this pass — see the Verified working list below.

Stage 1 — pre-training/

Turns a PDF corpus into training data. Local, GPU-first, Surya OCR + Gemma 3 (unsloth/gemma-3-4b-it, an ungated mirror — no HF_TOKEN needed). Five steps (exec_1.bat … exec_5.bat, or main.bat for an interactive menu): PDF → PNG pages → OCR CSV → summary CSV / layout CSV / synthetic-QA CSV, all written per-run to outputs/[timestamp]_[dataset]/.

Step 1 (PDF → PNG) is not a standalone Python entry point — it's scripts/convert_pdf_to_png.ps1, which shells out to poppler (pdftoppm/pdfinfo on PATH) and a small compress_png_max.py helper. Run it via exec_1.bat or the .ps1 directly, not uv run ... python scripts/convert_pdf_to_png.py — that file doesn't exist (an earlier version of this doc incorrectly assumed it did; fixed here). Steps 2–5 (ocr_detection_png.py, summarize_ocr_gemma.py, describe_layout_gemma.py, generate_qa_gemma.py) are genuine argparse Python scripts and do run via uv run --directory pre-training python scripts/<name>.py.

Status: out of scope for active work — left as-is, beyond the .gitignore/.python-version consolidation described above.

Stage 2 — fine-tuning/

Two example pipelines, same transformers+peft pattern, both LoRA-based, both sized for a single RTX 3090 (24GB). (A third, Axolotl-based axolotl-ocr-summary/ pipeline existed earlier but was removed by the repo owner — it only resolved its uv environment on Linux/WSL, never natively on Windows, since axolotl[deepspeed] depends on triton.)

Pipeline	Framework	Base model	Data shape
fine-tuning/vicuna-7b-lora/	transformers + peft (manual Trainer loop)	lmsys/vicuna-7b-v1.5 — LoRA on q_proj/v_proj, loaded directly via AutoModelForCausalLM/AutoTokenizer. No LLaVA checkpoint, no vision encoder, no multimodal projector anywhere in the dependency graph.	JSONL with text / summary fields
fine-tuning/qwen25-3b-lora/	transformers + peft (same pattern as vicuna-7b-lora/)	Qwen/Qwen2.5-3B-Instruct — LoRA on q_proj/v_proj (same target modules as Vicuna; Qwen2ForCausalLM uses the same separate Q/K/V/O naming, confirmed via peft's own default LoRA target-module table). ChatML prompt format instead of Vicuna's USER:/ASSISTANT: (verified against the tokenizer's chat_template/eos_token).	JSONL with text / summary fields

Naming/loading history: vicuna-7b-lora, previously llava15-lm-lora, originally llava15-lora. Two renames, each fixing a real overstatement:

1. llava15-lora → llava15-lm-lora: the pipeline only ever LoRA'd the language-model backbone (q_proj/v_proj) of llava-hf/llava-1.5-7b-hf — never the vision encoder or multimodal projector, never an image. The -lora name alone overstated that as a VLM fine-tune.
2. llava15-lm-lora → vicuna-7b-lora: even loading LLaVA's checkpoint at all was unnecessary once the vision half was never used — it still downloaded the full ~14 GB multimodal weights via LlavaForConditionalGeneration/AutoProcessor to get to a submodule that is, in substance, Vicuna-7B. This pass switched to loading lmsys/vicuna-7b-v1.5 directly via AutoModelForCausalLM + AutoTokenizer — ~13 GB instead of ~14 GB, no vision-related code path in the dependency graph at all, same LoRA config/target modules. Trade-off: lmsys/vicuna-7b-v1.5 is the checkpoint LLaVA 1.5 was later visually-instruction-tuned from, not the LLaVA-tuned weights themselves — different starting point, not directly comparable to the old llava15-lm-lora run's results, but a cleaner, smaller, honestly-named base for a language-model-only LoRA. serving/llava15-lora was renamed to serving/vicuna-7b-lora in step, with matching model-loading changes (functionally required: a Vicuna-7B-trained adapter's parameter names don't match LlavaForConditionalGeneration's language_model.* prefix, so serving would fail to load it otherwise) — that serving folder has since been removed, see Stage 3. A planned llava15-full-lora sibling, trained on image+text pairs and actually exercising the vision encoder/projector, remains the natural first real VLM fine-tune in this repo — that one should load the full LLaVA checkpoint. Full reasoning in fine-tuning/vicuna-7b-lora/README.md.

vicuna-7b-lora/ is a generic text-summarization LoRA, not OCR-specific — its interface, data source, and default prompt were all cleaned up this pass to reflect that:

• JSONL field is text (was ocr_text); build_vicuna7b_dataset.py and generate_vicuna7b_lora.py's flags are --source-csv/--text/--text-file (were --ocr-csv/--ocr-text/--ocr-text-file).
• build_vicuna7b_dataset.py only builds from a CNN/DailyMail Parquet dump now (--cnn-dailymail-dir, required) — the earlier dual-source mode that also read pre-training's image-linked OCR/SUMMARIES CSV pair (normalize_image_key/resolve_image_path/load_summaries) was removed entirely, not just renamed, since it's not needed for this pipeline's current use (generate_vicuna7b_lora.py's --source-csv batch-eval mode still accepts any generic CSV with a text column, unrelated to that removed ingestion path).
• train_vicuna7b_lora.py/generate_vicuna7b_lora.py's DEFAULT_INSTRUCTION is now the CNN/DailyMail news-article wording (was "Summarize this scanned document page... UAP-related content") — since that's the only source this pipeline builds from, --instruction no longer needs to be passed explicitly for the common case.

qwen25-3b-lora/ is a near-clone of vicuna-7b-lora/ — same dataset builder logic, same trainer/generator structure, same CLI shape. Two verified differences (not assumed): the ChatML prompt wrapper (see table above), and no protobuf/sentencepiece dependency needed (Qwen2.5-3B-Instruct ships a ready tokenizer.json, unlike Vicuna's raw SentencePiece tokenizer). No serving/qwen25-3b-lora/ — serving/ holds no pipelines at all now (see Stage 3), so a serving folder would have to be written from scratch if this adapter goes to production. It could not have been shared with the removed serving/vicuna-7b-lora/ in any case: that one was Vicuna-specific, and the ChatML wrapper differs.

CNN/DailyMail — wired in and verified

Downloaded locally, outside the repo and outside the root folder (gitignored regardless). Measured against the actual files:

Split	Rows	Size
train (3 shards)	287,113	~772 MB
validation	13,368	~35 MB
test	11,490	~30 MB
Total	311,971	~799 MB

article (avg ~3,950 chars) → text, highlights (avg ~260 chars) → summary. build_vicuna7b_dataset.py --cnn-dailymail-dir ... --max-samples 2000 was run end-to-end against the real files and produces valid JSONL records; full details and commands in fine-tuning/vicuna-7b-lora/README.md.

First real training run (superseded) and the reconstruction-test tool

Before the switch to loading Vicuna-7B directly, a 2,000-sample run on the old llava15-lm-lora pipeline (1,800 train / 200 val, 1 epoch, 450 steps, ~31 min on a single RTX 3090) showed loss dropping 1.66 → ~1.12 in the first ~50 steps then plateauing in a ~1.0–1.2 band, with eval_loss (1.11) tracking train loss closely (no overfitting) — a normal curve for a rank-16, 2-projection adapter on a small dataset, not evidence of a broken run. That adapter and its data/hf_cache were deleted as part of this pass's switch to lmsys/vicuna-7b-v1.5 (different base weights, not compatible with the old adapter) — the numbers above are illustrative of the expected curve shape, not a claim about the current pipeline's untrained state.

What carries forward: loss alone doesn't say whether summaries are actually good, so generate_vicuna7b_lora.py has a --jsonl-eval <path> --num-samples N reconstruction-test mode (added the same pass as the old run above) — it replicates the trainer's train/val split (same seed/ratio) and prints genuinely held-out source/reference/generated triples with token-F1, instead of requiring a manual --text string. Run against the old adapter it produced coherent, on-topic, correctly-bulleted CNN/DailyMail summaries (avg token-F1 0.357 across 5 samples) despite the plateaued loss — this is the tool to use to judge the next real run on the current pipeline, not the loss curve. See the pipeline README.

Stage 3 — serving/

One serving/<pipeline>/ folder per fine-tuning pipeline that has a serving story. Currently empty. serving/vicuna-7b-lora/ — a FastAPI service (app.py) that loaded the base Vicuna-7B model plus the trained adapter (or a fused/merged model) once and served a JSON API plus a dataset-browser front-end — was removed by the repo owner, the same way fine-tuning/axolotl-ocr-summary/ was.

The design it demonstrated is still the intended shape for this stage, and worth restating for whatever returns here: a serving folder is deliberately decoupled from its fine-tuning counterpart, reading only that pipeline's trained output directory (for Vicuna that was ../../fine-tuning/vicuna-7b-lora/runs/vicuna7b_lora/final_adapter) and never importing its training code. That boundary is what makes a serving folder independently deployable.

Stage 4 — training/

From-scratch / non-LoRA training of other models, as distinct from adapting an existing checkpoint (fine-tuning/). Fifteen pipelines so far, each an independent uv project and each writing out by hand whatever the usual library would hide.

training/adult-income-logreg/ is the first pipeline here: logistic regression on the UCI Adult / Census Income dataset, implemented with raw numpy rather than scikit-learn — the sigmoid, binary cross-entropy loss, gradient derivation, and gradient-descent update loop are all written out by hand in train_logreg.py so the math stays visible, and build_income_dataset.py parses the raw CSV files and does one-hot/z-score encoding without pandas. Own uv project like every other pipeline folder, but no torch/CUDA dependency at all — just numpy. Verified end-to-end against the real dataset: 30,162 train / 15,060 test rows after dropping "?" rows (matches the cleaned-variant counts in adult.names exactly), 300 epochs of batch gradient descent, 84.6% test accuracy (in line with the 84–86% published for tree-based methods on this same cleaned split — a from-scratch linear model landing close to that is the expected sanity-check result, not a target to beat).

training/cifar10-vqvae/ is the next pipeline here: a VQ-VAE (van den Oord et al., 2017) trained from scratch on CIFAR-10 (raw python-format pickles, parsed by hand with stdlib pickle — no torchvision/keras, no Pillow). It is the in-place successor of the former cifar10-vae (renamed/converted): a plain VAE's blur comes from Gaussian-posterior averaging in the ELBO, and VQ-VAE removes that mechanism — an 8x8 grid of D-dim encoder vectors is replaced by its nearest neighbors in a learned 512x64 codebook (straight-through estimator, EMA codebook updates, commitment loss), and reconstruction is driven purely by MSE. Same hand-written philosophy as the whole training/ folder: encoder/quantizer/decoder all plain torch.nn, torch only for tensor ops/autograd/GPU, numpy-permutation batching, no VQ-VAE library, no DataLoader. ~0.74M params (codebook included), ~40–60 min for 100 epochs on a single RTX 3090. Reconstruction-only by design (encode -> quantize -> decode a real image back; no learned prior over the discrete codes, so no sampling) — the discrete code grid is the substrate for a later learned prior (the planned cascade). Its evaluator computes the same metric suite used to judge the predecessor VAE (MAE/PSNR/SSIM/ high-frequency retention) plus a codebook-usage check, so the VQ-VAE vs VAE comparison is reproducible; measured numbers in the pipeline README's "Verified runs".

training/imdb-sentiment-cnn/ is the text-classification pipeline here: a Text CNN (Kim, 2014) trained from scratch on the Large Movie Review Dataset (25k train / 25k test binary sentiment; raw review .txt files at parsed by hand — no torchtext/datasets/nltk). Same hand-written philosophy as the whole training/ folder: a randomly-initialized trainable embedding (no GloVe — strictly IMDB-only data by design), three parallel 1D convs (widths 3/4/5 × 128 filters) + ReLU + 1-max-pool per filter, concat, dropout 0.5, linear → 2; torch only for tensor ops/autograd/GPU, numpy-permutation batching, no DataLoader. ~6.6M params (6.4M in the embedding), 20 epochs in ~31 s on a single RTX 3090, best checkpoint by val acc (peaks at epoch 3 before the model overfits — train acc → 100%). Measured on the held-out 25k test split: 89.2% accuracy (neg 88.96% / pos 89.43%) — well above Kim's published CNN-rand (82.7%), which the README attributes to full-length reviews plus val-based early stopping. A dropout-0.7 variant scored 87.9% on test and was discarded. Full numbers in the pipeline README's "Verified runs".

training/flow-matching-mnist/ is the newest pipeline here, and the first generative model family in this repo that is not an autoencoder: flow matching / rectified flow trained from scratch on MNIST. The model learns a velocity field v(x,t) whose ODE transports N(0,I) at t=0 into the data at t=1; training regresses that velocity on straight-line conditional paths with plain MSE — x_t = (1-(1-sigma_min)*t)*x0 + t*x1, target x1 - (1-sigma_min)*x0 — which is Lipman et al.'s conditional-OT path (2210.02747) and, at the default --sigma-min 0.0, exactly the rectified flow of Liu et al. (2209.03003). There is no noise schedule, no variance parameterization, and no ELBO — that absence is the point, and it is the difference between this and a DDPM. Same hand-written philosophy as the rest of training/: the UNet velocity field (three resolutions, one 7x7 self-attention block, sinusoidal time embedding as a per-channel bias), the EMA, and the Euler/Heun ODE samplers are all plain torch.nn; no diffusers/torchcfm/torchdiffeq/torchvision, no DataLoader, numpy-permutation batching. ~1.18M params, 40 epochs in ~5.3 min on a single RTX 3090.

It is the deliberate counterpart to training/mnist-vae: same dataset, same data/mnist.npz contract, same hand-written zlib PNG writer, so the two prior-sample grids are directly comparable — the flow model's are visibly sharper, the VAE's blur being the Gaussian-posterior averaging that cifar10-vqvae also exists to remove. That comparison is model-family vs model-family, not a controlled ablation: 1,175,841 params of UNet against 370,945 of plain conv encoder/decoder, and the two runs cannot separate objective from capacity. The honest cost difference they do show: the VAE generates in one forward pass, the flow model needs 20–50 network evaluations.

Judging it needed care, and two of this pipeline's guardrails came from getting it wrong first. (1) The evaluator's round-trip MAE/PSNR sweep measures ODE discretization error, not sample quality — the ODE is time-reversible, so a real digit can be integrated back to noise and forward again, but a near-zero velocity field round-trips perfectly since the identity is its own inverse; a 2-epoch smoke run really did post a better round-trip PSNR than the converged model. The sweep's actual use is choosing --num-steps, and it shows Heun winning per network evaluation, not just per step (10 Heun steps beat 50 Euler steps by ~2 dB at the same 100 evals). (2) The nearest-neighbour memorization check reports a distance that is meaningless without a scale, so real held-out test digits are measured against the training set the same way as a control. There is deliberately no FID — it would require a pretrained Inception network, against this folder's from-scratch rule, and a substitute number would be worse than none. Measured figures in the pipeline README's "Verified runs".

training/rvq-audio-codec/ is the newest pipeline here, and the first audio pipeline in the repo: a neural audio codec with residual vector quantization (the EnCodec/SoundStream/DAC architecture) trained from scratch on LJSpeech (13,100 wavs, 23.92 h, parsed by a hand-written RIFF/WAVE chunk walker - no torchaudio, no soundfile, no librosa, no scipy). A SEANet-style strided conv encoder maps the waveform to 68.9 frames/s, a stack of 8 codebooks x 1,024 entries quantizes each frame (each codebook quantizing the residual the previous one left), and a mirrored transposed-conv decoder reconstructs it - 5.51 kbps. 7,338,658 params, plus a 2,112,582-param multi-scale STFT discriminator that exists only during training.

It is the deliberate successor of training/cifar10-vqvae: that folder has one codebook of 512 entries looked up in the full 64-dim latent, this one stacks eight looked up in an 8-dim factorized projection under cosine distance, with EMA updates and dead-code re-initialization. 9 bits per latent position is enough for a 32x32 thumbnail and nowhere near enough for a waveform; RVQ is how the bit budget is bought without a K^N codebook. It is also the layer every modern audio LM (VALL-E, MusicGen, Moshi) sits on - those models generate codec tokens, not waveforms.

Three things are deliberate and worth not undoing. (1) LJSpeech is 22,050 Hz and is trained at that native rate - no resampler is written, so the frame rate is 68.9 Hz and the bitrate 5.51 kbps rather than EnCodec's published 24 kHz / 75 Hz / 6 kbps. (2) Quantizer dropout (a random n_q in [1, N] on half of each batch) is what makes one trained model serve the whole 1->8 codebook ladder; without it the 1->8 quality demo would need eight separate runs. (3) The discriminator is staged behind --adv-start-step, because a randomly-initialized generator fighting a randomly-initialized discriminator collapses a codec in the first thousand steps; --lambda-adv 0 turns it off entirely for a reconstruction-only A/B.

Judging it needs the same care as flow matching's ODE sweep. SI-SDR is a weak proxy for a GAN-trained codec - the adversarial loss trades exact waveform/phase alignment for perceptual realism, so a model that sounds better can post a worse SI-SDR than a reconstruction-only one. The real evaluation is the original_NN.wav / recon_nq{8,4,2,1}_NN.wav files the evaluator writes (hand-written 44-byte RIFF writer, the inverse of the builder's parser), plus the per-codebook usage table that says whether the 8th codebook is doing any work. There is deliberately no ViSQOL/PESQ/ NISQA - each needs an external binary or a pretrained network, the same rule that keeps FID out of flow-matching-mnist.

The discriminator is also where the run's cost lives, by a wide margin. Measured on the RTX 3090 at batch 32: reconstruction-only runs at 7.75 steps/s, and turning the discriminator on in fp32 drops that to 0.95 — 8x, because its spectrograms are much larger than the waveform they judge (173 x 257 positions at the 512-point resolution against 22,080 samples, three resolutions, three passes per step) and those conv shapes map badly onto fp32 tensor cores. cudnn.benchmark and TF32 matmul were both measured and change nothing (0.90-0.95 steps/s, inside the noise). What does work is bf16 autocast on the critic only (--disc-bf16, default on): 2.01 steps/s and 10.8 GiB peak instead of 16.8, turning a 7-hour 60-epoch run into a 3.3-hour one with no architecture change. The generator, the codebook lookup and every EMA update deliberately stay fp32 — bf16 EMA statistics would quietly stop accumulating small updates, which is exactly the mechanism dead-code revival exists to detect.

One guardrail came from getting it wrong first: the dead-code cutoff is a fraction of uniform codebook usage, not the absolute 2.0 that EnCodec and vector-quantize-pytorch use. One batch here is 32 x 69 = 2,208 vectors over 1,024 entries, so uniform usage is only 2.16 per entry and an absolute 2.0 condemns half a healthy codebook every sweep - the first smoke run reported 1,023 of 1,024 entries "revived" per codebook; after the fix, 0.

training/fashion-mnist-dcgan/ is the repo's first GAN pipeline: a DCGAN (Radford et al., ICLR 2016) trained from scratch on Fashion-MNIST (60k/10k, 28x28 grayscale, 10 classes, IDX ubyte files parsed by hand - the Kaggle CSVs are also accepted by the builder). Generator: z ~ N(0,I) (100-dim) -> linear -> 7x7x256 -> BN+ReLU -> deconv -> 14x14x128 -> BN+ReLU -> deconv -> 28x28x1 -> Tanh. Discriminator: strided convs 28 -> 14 -> 7 -> 3 -> 1 with LeakyReLU(0.2) and BN everywhere except the input layer, ending in a single logit. Both nets use the hand-written N(0, 0.02) initialization, one-sided label smoothing (real = 0.9), and Adam at 2e-4 with betas (0.5, 0.999) - the DCGAN tuning that makes the two-player game actually converge. Hand-written like the rest of training/: no torchvision, no kagglehub/ pytorch-gan-metrics, no DataLoader (numpy-permutation batching).

Two notes worth keeping. First, 28x28 does not divide cleanly down DCGAN's canonical 32x32 ladder: three stride-2 convs take 28 -> 14 -> 7 -> 3, so the discriminator's last feature map is 3x3 (a final 3x3 conv to one logit) and the generator must start from a 7x7 grid, not 4x4. The shapes in train_dcgan.py are the verified ones - "fixing" them to the paper's 32x32 numbers breaks the tensors. Second, a GAN is judged by its samples, not its loss: D/G losses move adversarially and say almost nothing about sample quality, so train_dcgan.py writes a fixed-z sample grid every --sample-every epochs (collapse becomes visible across training) and evaluate_dcgan.py emits samples_grid.png plus the same nearest-neighbour memorization guard as flow-matching-mnist (L2 to the closest training image, compared against a real-image control) and a pairwise-diversity probe. There is deliberately no FID/IS - both need a pretrained Inception, the same rule that keeps FID out of flow-matching-mnist and ViSQOL out of rvq-audio-codec. The pipeline is new; verified-run numbers go in the pipeline README's "Verified runs" once it has been run on the repo owner's RTX 3090.

training/vit-cifar10/ is the first attention-based vision model in the repo — and its first from-scratch transformer of any kind: a Vision Transformer (Dosovitskiy et al., 2021, in the pre-LN / norm-first layout popularized by DeiT) trained from scratch on CIFAR-10. Hand-written philosophy like the rest of training/: the patch embedding, learned CLS token + positional embeddings, the transformer blocks, and the multi-head self-attention (QKV projections, scaled dot-product, output projection) are all plain torch.nn — no transformers/timm/torchvision, no DataLoader (numpy-permutation batching). Flip+crop augmentation is plain torch ops (torch.flip, zero-pad + random crop, per-channel normalize with the hardcoded CIFAR-10 train statistics). ~10.7M params at the defaults (--dim 384 --depth 6 --heads 6 --mlp-ratio 4), AdamW with weight decay 0.05 and a hand-written linear-warmup-then-cosine LR schedule (warmup is the part ViTs need that the rest of training/'s plain-cosine trainers don't), best checkpoint by val acc, ~23 min for 60 epochs on the RTX 3090 fp32. The evaluator reports test top-1/top-5, per-class accuracy + confusion matrix, and writes a hand-written zlib RGB predictions_grid.png (first 32 correct, first 32 misclassified, green/red borders) — deliberately no pretrained-feature score, the same rule that keeps FID out of flow-matching-mnist and ViSQOL out of rvq-audio-codec. Verified on the RTX 3090 (repo owner's run): 66.82% test top-1 / 97.32% top-5 at the 60-epoch defaults in 1,367 s (~23 min), 10,695,562 params, best val 67.50% at epoch 60, frog best (81.4%) / cat worst (44.7%) with the classic cat↔dog and truck↔automobile confusions. That is below the ~80–86% figure this entry originally estimated — too optimistic for flip+crop-only at 60 epochs; the measured 66.8% is the record (the correction is documented in the pipeline README).

training/mae-cifar100/ is the repo's first representation-learning (self-supervised) pipeline: a Masked Autoencoder (He et al., 2022) trained from scratch on CIFAR-100 — patchify → mask 75% → encoder → lightweight decoder → MSE on the masked patches. The encoder is vit-cifar10's patch-embed/block stack reused (copied in by hand; the pipelines don't import each other), at a denser grid: default patch 2 → 256 patches (64 visible at 75% masking — the paper's regime at 32×32, unlike the sibling's 64-patch patch-4 config, which stays available via --patch-size). No CLS token (MAE doesn't use one), no head; the decoder is a separate ~1M-param transformer that exists only for pretraining. Masking is the paper's fixed-count per-sample permutation, not a Bernoulli; the loss is MSE on masked patches only with per-patch-normalized targets (--no-patch-norm is the documented A/B of that trick). ~11.8M params total (10,750,848 encoder + 1,015,692 decoder), AdamW + warmup/cosine like the ViT sibling, flip+crop on raw [0,1] pixels (no normalization in pretraining — the reconstruction targets are the pixels), best checkpoint by a deterministic full-image val reconstruction MSE. Judged by a hand-written linear probe (linear_probe.py): a linear head trained from scratch on the encoder's frozen, mean-pooled patch-token features (SGD momentum + cosine, the paper's protocol) — the features are the model's own, so this is a from-scratch evaluation that does not violate the no-pretrained- features rule the way a FID/Inception score would. Verified on the RTX 3090 (repo owner's run): 25.56% test top-1 / 53.41% top-5 (coarse top-1 37.9%) at the 60-epoch defaults in 2,166 s (~36 min). The final (epoch-60) checkpoint is the record — it probes +0.91 top-1 over the best-val (epoch-34) checkpoint (24.65% / 53.15% / 36.8%): val recon bottomed at epoch 34 while the masked-MSE kept improving, so the late features are better even though full-image recon drifted (a finding — don't assume the best-val checkpoint holds the best representation). Per-class oak_tree 69.0% best / bowl 1.0% worst — the classic CIFAR-100 pattern. Both figures are below the ~30–45% estimate this entry's earlier draft carried — too optimistic for 60 epochs on 45k images without probe-time augmentation; the measured 25.56% is the record (the correction is documented in the pipeline README).

training/dit-cifar100/ is the repo's first class-conditional generative transformer: a Diffusion Transformer (Peebles & Xie, 2022 — the architecture Sora is built on) trained from scratch on CIFAR-100, the natural big sibling of training/flow-matching-mnist (same conditional-OT flow-matching objective, now conditioned on the 100 real fine classes, with classifier-free guidance). Hand-written philosophy like the rest of training/: the patch embedding, the frozen 2D sincos positional embedding, the adaLN-Zero transformer blocks, the hand-written multi-head self-attention, the final unpatchify layer, the class embedding (+ null token), the conditional-OT probability path, the velocity-regression loss, the EMA, and the Euler ODE sampler are all plain torch.nn — no diffusers/torchcfm/torchdiffeq/transformers/timm/torchvision, no DataLoader (numpy-permutation batching). The objective is mse(v(x_t, t, y), x1 - (1-sigma_min)*x0) on the conditional-OT path x_t = (1-(1-sigma_min)*t)*x0 + t*x1 — byte-identical to flow-matching-mnist's loss, plus the class. CFG is trained by dropping the class to null token 100 with probability 0.1 (the DiT paper's value); sampling is v_uncond + cfg*(v_cond - v_uncond) integrated by Euler. Defaults --patch-size 2 --dim 256 --depth 8 --heads 8 = 9,828,876 params (256 tokens, the same density as DiT-S/4 at 256 px), AdamW + warmup/cosine like the ViT siblings, flip+crop on raw [0,1] pixels (then rescaled to [-1,1] like flow-matching-mnist), EMA weights for sampling, best checkpoint by a deterministic-seed val velocity MSE, the 10k test split stays unseen until evaluate_dit.py. Judged the flow-matching-mnist way (the user's stated rule): samples_grid.png (100 class-conditional samples, one per fine class), cfg_sweep.png (the same latents at CFG scales 1.0–5.0), the nearest-neighbour memorization check vs a real-image control, and the test velocity MSE — deliberately no FID (pretrained Inception, same rule as everywhere else in training/). Verified on the RTX 3090 (repo owner's run): 9,828,876 params, 60 epochs in 4,521 s (~75 min) — train velocity MSE 0.5174 → 0.1695, best val 0.1842 at epoch 55, test velocity MSE 0.1887 on the held-out 10k (EMA weights, ~75 s/epoch at batch 256). Nearest-neighbour check: generated samples sit ~47% farther from the training set (mean L2 12.105, min 9.572) than real unseen test images (8.210 / 5.152) — no memorization, the same direction flow-matching-mnist measured. The ~0.17–0.19 loss floor is expected (the velocity target is irreducibly random given (x_t, t, y)) — judge the sample grids, not the loss.

training/librispeech-speaker-id/ is the repo's first discriminative audio pipeline — every other audio work here reconstructs a waveform; this one maps audio to a label — and its first metric-learning objective: every trainer in training/ before it is cross-entropy classification or MSE reconstruction, while this one learns a space in which cosine distance between two 192-d vectors is the quantity of interest. An ECAPA-TDNN (Desplanques et al., Odyssey 2020) — with the x-vector / TDNN it superseded (Snyder et al., ICASSP 2018) available as --arch xvector for a controlled A/B — trained from scratch on LibriSpeech.

Hand-written like the rest of training/: the dilated conv blocks, the squeeze-excitation channel attention, the Res2Net multi-scale channel split, the attentive statistics pooling, the AAM-softmax head, the speed-perturbation resampler, the EER/minDCF metrics and both plot renderers are all plain torch.nn/numpy — no speechbrain/kaldi/sidekit/ pyannote, no torchaudio/librosa/soundfile/scipy, no DataLoader (numpy-permutation batching over utterance indices, one crop each). The log-mel filterbank is copied in by hand from training/rvq-audio-codec (pipelines never import each other's code) and works at 16 kHz unchanged because it takes sample_rate as an argument.

~2,939,616 params (--arch ecapa --channels 256 --mfa-dim 1536 --emb-dim 192, the AAM head's 44,352 more are discarded at inference); --arch xvector is 2,589,140 at the same --channels 256 width, or 4,454,868 at --channels 512 (the literal TDNN width, and no longer a like-for-like A/B). Both are trivial for 24 GB — the constraint is data and time, not memory. Cost is linear in frames because there is no attention over time and nothing recurrent, which is the whole reason a speaker encoder trains in minutes where a Conformer-CTC on the same corpus needs 6–19 h per epoch. Measured: 30 epochs in 7.0 min (ECAPA) / 5.3 min (x-vector) on train.clean.100, and open-set EER 0.0725 / 0.0989 on 20 held-out speakers.

The A/B's result is the finding, and it inverted the expected answer. On the val EER that selects the checkpoint, x-vector won (0.0095 vs 0.0162) and trained faster; on the held-out-speaker EER, ECAPA won (0.0725 vs 0.0989) — x-vector was fitting the 231 training speakers more tightly and transferring worse. A pipeline that selected and reported on the in-training split alone would have concluded the 2020 architecture is not worth its cost. This is the three-way split paying for itself, measured rather than asserted, and it implies the best-checkpoint criterion here is the wrong signal (it should be held-out speakers, not held-out utterances of seen ones). Don't "simplify" the split back to a boolean.

Things not to silently undo:

• The index carries a three-way split, not a boolean: 0 = train, 1 = val (held-out utterances of seen speakers), 2 = unseen (held-out speakers). Two different questions need two different held-out sets. LibriSpeech is partitioned by speaker and train.clean.100's 251 speakers have zero overlap with test.clean/dev.clean (40 speakers each, themselves disjoint — verified, not assumed), so a 251-way classifier cannot be evaluated on test-clean: the held-out set has to come from inside the training split. Collapsing this to one boolean silently makes one of the two numbers meaningless.
• The headline metric is the open-set EER on split 2, not the closed-set accuracy on split 1. The accuracy split also selected the checkpoint, so it is mildly optimistic; eval_metrics.txt says so in the file itself. Don't promote the accuracy to the headline number.
• AAM-softmax (--margin 0.2 --aam-scale 30) is what makes the embedding a metric. A plain softmax head can score well on 251-way identification while producing embeddings that are useless for anyone outside those 251 people, because nothing constrains the geometry. Don't swap it for plain cross-entropy to "simplify".
• The head runs in fp32, outside the bf16 autocast. Its sqrt/where/one_hot path is not autocast-safe, and at ~5k parameters it costs nothing. Don't extend the autocast over it.
• Evaluation crops are centred and deterministic, not random — rng=None in sample_batch. A random crop at eval time makes the val EER jitter by more than the training signal. (This was got wrong first: the first run crashed with rng.integers on a None, which is how the distinction got made explicit rather than implied.)
• Speed perturbation is a resampler, and it is documented as one. It is augmentation, not dataset resampling — rvq-audio-codec's "no resampler" rule is about not resampling LJSpeech off its native 22,050 Hz, which this does not do (LibriSpeech is used at its native 16 kHz). Don't apply it to the val/unseen paths.
• ffmpeg is a hard dependency of the builder, and pyarrow is a hard dependency of the pipeline. FLAC's residual coding is Rice-coded and bit-serial, so a pure-Python decoder over 57 GB is not slow, it is impossible; the precedent for shelling out to a format tool is already pre-training/exec_1.bat -> poppler. pyarrow reads the parquet container, the same category as the stdlib pickle the CIFAR pipelines parse. Don't hand-write either, and don't remove the ffmpeg check from uv_setup.bat.
• The FLAC decode is batched through ffmpeg's concat demuxer and sliced by FLAC STREAMINFO sample counts. One process per utterance would be ~28,500 spawns. The slice is asserted against ffmpeg's byte count per batch, so a mis-split fails loudly instead of silently shifting every offset after it.
• The builder verifies each split against its canonical LibriSpeech size (28,539 for train.clean.100, etc.) — the same guardrail as build_ljspeech_dataset.py's 13,100-wav check. Don't relax it.
• There is deliberately no pretrained speaker-verification score — no pyannote embedding, no VoxCeleb-trained model to compare EER against. Same rule that keeps FID out of flow-matching-mnist, LPIPS out of the 3DGS branch and ViSQOL out of rvq-audio-codec.

It is the sibling of training/rvq-audio-codec from the opposite direction: that one reconstructs a waveform and knows nothing about who is speaking, this one discards the waveform and keeps only identity. The natural join is speaker-conditioned codec training, which is what would make rvq-audio-codec multi-speaker. It is also the warm-up rung for a planned CTC-ASR pipeline: same corpus builder, same memmap contract, same log-mel front-end, same variable-length batching. Measured numbers in the pipeline README's "Verified runs".

training/mamba2-tinystories/ is the repo's first non-attention sequence model — every earlier sequence model here is attention (ViT/MAE/DiT) or convolution (TextCNN/ECAPA), and this is the first recurrence. A Mamba-2 selective state-space model in its SSD (state-space-duality) form, trained from scratch on TinyStories, with a parameter- and token-matched causal transformer as --arch transformer making the A/B a controlled comparison rather than two pipelines quoting numbers at each other.

Hand-written like the rest of training/: the input-dependent discretisation (dA = exp(dt*A), dB = dt*B), the chunked SSD scan, the causal depthwise conv, the SiLU-gated output projection (no RMSNorm on the gate — the paper's Mamba-2 block puts one there and this pipeline's does not, a documented departure, so the two are not interchangeable), the tied-embedding LM head and a byte-level tokenizer are all plain torch.nn — no mamba-ssm, no causal-conv1d, no transformers/tokenizers, no DataLoader (numpy-permutation batching). The corpus is a memmap (data/tinystories_train.u16 + an index .npz), the rvq-audio-codec contract rather than an .npz of tokens: 1,943,728,852 tokens = 3.887 GB as uint16.

Three things are deliberate. (1) The tokenizer is the byte-level identity (vocab 256), so one token is one byte, total_tokens equals the memmap size exactly, and bits-per-token == bits-per-byte — which is what makes the A/B a clean comparison. (2) The chunked scan is asserted against a naive per-timestep reference (--selftest, run at startup): the naive form is an oracle, not a second implementation to maintain, and the assertion is the point. Measured fp32 forward relative error 3.8e-06 and fp64 6.6e-15; independently reproduced against a clean-room oracle that agrees with mamba2_scan_reference to exactly 0.0. That assertion earned its keep — it exposed a transposed causal mask and an A/B/C argument swap during development. (3) The builder's count guardrail refused to build at first, and that is the most interesting result here: the canonical TinyStories figures (2,119,718 / 21,989) are separator counts, not story counts. Train has 2,119,719 spans, 230 of them empty, giving 2,119,489 stories; valid has 21,990 spans, none empty. The naive stories = separators + 1 assertion failed loudly rather than quietly dropping 230 entries, and the index now carries train_separators, train_empty_spans, train_stories, … so the accounting is auditable without rescanning 2 GB.

Measured smoke A/B at 300 steps / 2.46 M tokens (500,000 params vs 497,024, matched to 0.6%): mamba2 val 1.9208 vs transformer val 2.1656, both far below the uniform-prediction level ln(256) = 5.545. (Those counts are sum(p.numel() for p in model.parameters()), and they are also what the checkpoint records as num_params. Summing state_dict() instead gives 532,768 / 529,792 — exactly +32,768 = 256 x 128 — because the tied embedding / LM-head weight is stored under two keys. An earlier draft of this entry quoted the state_dict figure; use the deduplicated one.) That is a smoke A/B and the README says so — ~0.5 M params at that budget cannot separate two architecture families, which is exactly the lesson librispeech-speaker-id's split-dependent reversal taught.

The documented config has since been run to completion on an idle RTX 3090, both legs, 1,104 steps x 32,768 tokens = 36.2 M tokens (1.91% of one pass): mamba2 3,515,008 params, 379 s (0.343 s/step), best val loss 0.8004 (ppl 2.23, 1.1547 bits/byte); transformer 3,527,488 params (+0.355%), 235 s (0.213 s/step), val 1.0053 (ppl 2.73, 1.4503 bits/byte). The evaluator re-scores the checkpoints at 0.8197 vs 1.0002 — mamba2 ahead by 0.18 nats, with the transformer 1.6x faster per step — and the length-extrapolation table is the sharper result: past the trained L=512 the transformer's learned absolute positional embeddings (linearly interpolated) degrade to 1.63 nats at L=1024 and 2.49 at L=2048 while mamba2 stays at 0.91 / 0.85. The MQAR recall probe is at chance for both (≈0.002, chance 0.0020) and is reported that way. All of it remains not a finding: 1.91% of a pass cannot separate two architecture families, and the contended s/step figures the earlier draft extrapolated from (0.53–0.81 s/step) turned out 1.6–2.4x pessimistic against the idle measurement above.

training/meanflow-cifar10/ is the repo's first 1-NFE generative model: MeanFlow — average-velocity flow matching (Geng, Deng, Bai, Kolter, He 2025) — trained from scratch on CIFAR-10. It is the direct successor of flow-matching-mnist and dit-cifar100: the same conditional-OT path, but regressing the average velocity u(z,r,t) over an interval instead of the instantaneous one, so a single network evaluation generates a sample where its two siblings need 20–50. Hand-written: the class-conditional DiT block stack, the two time embeddings (t and r), the adaLN conditioning, the stop-gradient target, the EMA and the sampler; the JVP is torch.func.jvp, i.e. torch's own forward-mode autodiff, not a library method. No diffusers/torchcfm/torchdiffeq.

This entry exists mostly to record one bug, because the bug is the interesting part. The first version of the objective used the model's own output for the v in the target — v = u_theta(z_t, t, t) — which makes u_theta == 0 an exact global optimum: if u=0 then v=0, the JVP=0, the target=0 and the loss=0. It is a self-consistent fixed point that is not the answer. Measured on a 2-D toy: that version diverges at lr 2e-3, and at lr 1e-4 "converges" to a model whose sample MMD (0.0647) equals raw noise's (0.0666) — it learned nothing while the loss fell. The paper's Eq. 9–11 is explicit that the target's only ground-truth signal is the conditional velocity v_cond = x1 - x0, and with that the same toy trains and 1-NFE sampling works (MMD 0.0067, ~10x closer than noise). Two diagnostics are therefore kept as evidence rather than commentary: at initialization the raw MSE must be O(1) — 6.33 with v_cond against 6.3e-04 with the model's own output — and forcing r == t must reduce exactly to plain Flow Matching, measured identical to 1e-9 (both 3.19850540). The warning generalises beyond this pipeline: a wrong-but-smooth target still produces a falling loss, so "the loss went down" is not evidence that an objective is right. The same wrong v was left behind in evaluate_meanflow.py, where it made the reported "training objective" read 0.0000 — a self-referential zero that any model scores, including a random one, and which therefore reads as "solved". Corrected to v_cond, the same smoke checkpoint reports 1.2536, exactly ||v_cond||^2 for CIFAR-10 in [-1,1]: with u ~ 0 the Jacobian ~ 0, the JVP term vanishes and the average-velocity MSE must equal the instantaneous one, so the two metrics agreeing exactly is the predicted signature rather than a coincidence.

Judged the flow-matching-mnist way — the sample grid and the NFE sweep over SAME latents at each step count, plus the nearest-neighbour memorization check against a real-image control — with deliberately no FID. Smoke checkpoint (--dim 64 --depth 2, 176 steps = one full 45k-image epoch, 27.2 s on an idle 3090 against 64.4 s while the sibling pipelines held the GPU): train average-velocity MSE 1.333 → 1.036, test 1.0072 average-velocity / 0.7566 instantaneous, and the non-degeneracy check passing on real data — the at-init raw MSE came out at 1.99090 against ||v_cond||^2 1.99090, a ratio of exactly 1.0000, and r == t reduces to plain Flow Matching to 1e-9.

Two findings from the end of that work are worth carrying forward. First, the siblings' 1000x time-embedding scale is unusable here: flow-matching-mnist and dit-cifar100 both scale the sin/cos time argument by 1000, but MeanFlow's loss contains (t-r) * d/dt u, so that factor sits inside the objective and multiplies the very time-derivative being learned — measured after one epoch, scale 1000 diverges (1.32 → 35.09 while |(t-r)*dudt| grew 0.000 → 2.017 and |v_cond| held at 1.164), while 100/10/1 give 1.014/0.821/0.812. The default is now --time-scale 100.0, with the learning rate at 1e-4 rather than dit-cifar100's 1e-3. Second, and more important for how this entry is read: the 1-NFE claim is still unmeasured on CIFAR-10. The smoke checkpoint's NFE=1 vs NFE=50 agreement is 0.038 RMS, but its samples have a pixel standard deviation of only 0.315 against ~1.0 for real CIFAR-10 — a near-constant field makes one step and fifty steps agree trivially, so the evaluator prints that control beside the agreement figure. The pipeline's first real run (the fast --dim 128 --depth 4 config, 20 epochs / 7,040 steps in 1,258 s: train average-velocity MSE 1.2476 → 0.6458, test 0.6705 average-velocity / 0.2210 instantaneous, NN check 9.613 generated vs 9.080 real) does not change that: its NFE=1 samples still carry a pixel std of only 0.1746 against ~1.0, so its NFE=1 vs NFE=50 agreement of 0.15897 is not evidence either. 1-NFE holds on the 2-D toy (MMD² 0.0139 against a 0.0655 noise reference). Settling it on CIFAR-10 needs the --dim 256 --depth 6 60-epoch default, which is now measured at ~10.2 h (0.867 s/step at batch 64; batch 256 does not fit the 24 GB card — 24,259 MiB observed) and has not been run.

training/3dgs-nerf-synthetic/ is the repo's first 3D / novel-view-synthesis pipeline: 3D Gaussian Splatting (Kerbl et al., SIGGRAPH 2023) trained from scratch on the NeRF synthetic scenes, with a hand-written vanilla NeRF (Mildenhall et al. 2020, train_nerf.py) as the representation-vs-representation baseline — an explicit, rasterised scene against an implicit, ray-marched one. Hand-written like the rest of training/: the PNG decoder (stdlib zlib + the five filter types), the camera conversion, the tile-based differentiable rasterizer, SSIM, the densification/pruning and the plot renderers — no nerfstudio, gsplat, diff-gaussian-rasterization, tiny-cuda-nn, plyfile, Pillow, torchvision, no DataLoader.

The data contract has more sharp edges than any earlier pipeline, and each one was verified rather than assumed:

• The shipped PNGs are 8-bit RGBA (colour type 6), not RGB, and carry straight alpha. Verified bit-exact (max diff 0) against an independent decoder on 10 images across 5 scenes, exercising all five filter types.
• train, val and test are three different camera orbits that reuse the same filenames (train/r_0, val/r_0, test/r_0), so frames are keyed by (split, file_path) and never by filename. Verified at the split boundaries: frame i is bit-exactly frame i's image and carries the correct world-to-camera, so the keying is demonstrated rather than asserted.
• test/ holds 600 PNGs while transforms_test.json lists 200 — the other 400 are depth and normal maps. Read the JSON; never glob the directory.
• The camera chain is pinned by two invariants: the world origin projects to the image centre (< 0.05 px over all 8 scenes × 3 splits), and a silhouette carve of the alpha masks keeps a coherent object volume — the latter being what catches a roll error, which no geometric invariant sees.
• The tile rasterizer is asserted against a brute-force composite whenever the top-K candidate bound is inactive, forward and gradients: --self-check reports forward 0.000e+00 (bit-identical) and gradient 1.492e-13 in fp64. It was worth having: it caught a wrong tile-scatter reshape that produced a factor-scale 0.5 error. The top-K bound and the 3-sigma bounding-box cull are deliberate approximations, and the unculled difference is reported, not asserted on — inventing a pass/fail threshold for an algorithm property would be the same mistake as adding a substitute perceptual score.
• A pure-PyTorch rasterizer cannot run at 800×800, so the default is --downscale 2 (400×400); the published 3DGS figures are at 800×800 and are not comparable. There is deliberately no LPIPS, the rule this file already stated for the 3DGS branch before it existed.

The NeRF baseline reuses SceneData, psnr and ssim from train_3dgs.py, so both models are scored by identical code, and it is coarse sampling only — the paper's two-stage hierarchical sampler is not implemented, which is its biggest quality gap. Measured smoke run: 530,052 params, 300 steps in 0.5 min, loss 0.180 → 0.022, val PSNR 16.04 dB. That number is only meaningful against the trivial baselines, which were measured for the same reason the generative pipelines carry a nearest-neighbour control: a constant white image scores 9.27 dB and a constant per-pixel train-mean image 13.55 dB, so the NeRF beats the strongest deployable constant predictor by 2.5 dB after thirty seconds. The 3DGS leg reached 16.959 dB val at iteration 200 (5000 → 5116 Gaussians, densification firing) before a densification-index crash interrupted that smoke run.

The first completed 3DGS run (2026-09-23, idle RTX 3090) is lego, 5,000 steps at --downscale 4: 48.6 ms/step, 242.9 s, 5,544 Gaussians, best val PSNR 19.748 dB, and held-out test 19.449 dB / SSIM 0.7941 over all 200 test frames with a train-vs-test gap of −0.256 dB (no overfitting). It also found a real bug: reset_opacity() clamped opacities up to --min-opacity (0.005) — the same constant the prune pass tests — instead of the reference's min(opacity, 0.01), so every reset parked weak Gaussians exactly on the prune line and the next densify pass deleted them en masse (2,628 of 2,992 = 87.8% at the step-3,000 reset of a --downscale 2 run, which was abandoned; the pre-fix 5,000-step run lost 1,023 of 5,830 at step 1,600). Fixed with a lowering reset at --opacity-reset-value (default 0.01) plus a main() guard refusing --opacity-reset-value <= --min-opacity; on the identical config the fix is worth +0.67 dB test PSNR / +0.014 SSIM (18.779 → 19.449). Still not the paper's budget: 5,000 steps at 200x200 is not 30,000 at 800x800, there is no --downscale 1 run, and only lego has been trained.

Datasets

None of the fine-tuning pipelines ship data — DATASET/, data/, runs/, output/ etc. are all git-ignored, drop-zone folders (via the single root .gitignore). Both vicuna-7b-lora/ and qwen25-3b-lora/ train on CNN/DailyMail. Example small, permissively-licensed public datasets are listed in the root README.md's Datasets section.

Two more drops were added to the local dataset folder for the newest work, both downloads rather than checked-in data:

• TinyStories (roneneldan/TinyStories, both the V1 and the V2-GPT4 text corpora) for training/mamba2-tinystories. The V1 train file is 1,924,281,556 bytes and the valid file 19,447,282; the canonical published counts are separator counts (2,119,718 / 21,989), which is why the builder asserts spans and empty spans rather than the raw marker count — see the mamba2-tinystories section above.
• NeRF synthetic (the Blender-rendered nerf_synthetic drop, all 8 scenes) for training/3dgs-nerf-synthetic: 8 x (100 train / 100 val / 200 test) 800x800 RGBA PNGs plus depth and normal maps, built into a 5.72 GB uint8 memmap store (metadata.json + dataset_summary.txt) rather than an .npz — 3,200 x 800 x 800 x 3 is not something to hold in RAM. It ships no COLMAP points, which is why the 3DGS leg uses the paper's random-init fallback.
• PASCAL VOC 2007 (the official Oxford VOCtrainval_06-Nov-2007.tar, VOCtest_06-Nov-2007.tar and VOCdevkit_08-Jun-2007.tar), downloaded and count-verified — 9,963 JPEGImages, 9,963 Annotations, 632 segmentation masks, and ImageSets/Main/ train 2,501 / val 2,510 / trainval 5,011 / test 4,952. No pipeline uses it yet; it is staged for a detection branch, and VOCtest_06-Nov-2007.tar does include the test annotations, so a genuine held-out mAP on the real test split is computable when that branch lands.

training/rvq-audio-codec is the one pipeline whose prepared data is too large for the .npz contract the others share: LJSpeech is 3.80 GB of int16, which becomes 7.6 GB as float32. It writes a raw data/ljspeech_audio.i16 opened with np.memmap plus a small data/ljspeech_index.npz of offsets/lengths/ids/split, and converts crops to float one batch at a time. Both are covered by the root .gitignore's data/ rule like everything else.

vicuna-7b-lora's real 2-epoch run (repo owner's machine)

2,000-sample JSONL (1,800 train / 200 val), 2 epochs, 900 steps, ~61 min on a single RTX 3090. Train loss 1.76 → 0.99, eval_loss essentially flat across epochs (1.100 → 1.094 — a mild overfitting signal in isolation, train loss kept falling while eval_loss didn't). What matters: reconstruction-test avg token-F1 rose from 0.357 (an earlier 1-epoch/1,800-sample run on the predecessor llava15-lm-lora pipeline) to 0.467 on this run, and the generated summaries reproduced exact figures from source text correctly (e.g. "383-41" and "70-26" vote counts). Confirms the earlier lesson again: eval_loss plateauing is not itself a stop signal — the reconstruction test is what actually shows whether a further epoch helped.

Verified working (this pass)

uv run --directory <folder> python <script> --help, and further real executions where noted, actually run, not assumed:

• training/cifar10-vqvae (successor of the former cifar10-vae) — real runs on the RTX 3090 against the actual downloaded CIFAR-10 python pickles: build_cifar10_dataset.py wrote data/cifar10.npz (50k train / 10k test); train_vqvae.py and evaluate_vqvae.py run end-to-end (100 epochs, ~10–15 min, codebook perplexity ~404/512, 512/512 codes fired on test — no collapse). Measured reconstruction on the held-out test set: PSNR 25.2 dB, SSIM 0.884, MAE 0.042, high-frequency detail kept 73.8% — vs the predecessor VAE's best (PSNR 21.5 dB, SSIM 0.742, HF 51.4%), i.e. the discrete-codebook family removes the plain-VAE blur mechanism. Full numbers in the pipeline README's "Verified runs".

• training/flow-matching-mnist — real run on the RTX 3090 against the actual MNIST IDX files: build_mnist_dataset.py wrote data/mnist.npz (60k train / 10k test); train_flow.py trained 1,175,841 params for 40 epochs in 317.5 s (val velocity MSE 0.2263 → 0.1704, best at epoch 38 — train and val track each other throughout, no overfitting); evaluate_flow.py scored 0.1687 velocity MSE on the held-out 10k and wrote all three PNGs. Reproduced in a second independent run of the same commands (312.1 s, best val 0.1705, test 0.1687) — expect the third decimal to move. Samples are clean, readable digits with a handful of malformed glyphs per 64. The ~0.17 loss floor is expected, not a defect: u = x1 - x0 is irreducibly random given (x_t, t), so the MSE floors at that conditional variance and can never reach zero — judge the samples, the same lesson fine-tuning/vicuna-7b-lora taught about loss plateaus. Memorization check: generated samples sit farther from the training set (mean L2 4.029, min 1.858) than real unseen test digits do (3.611 / 1.169).

• training/rvq-audio-codec — full 60-epoch run on the RTX 3090 against the real LJSpeech drop. build_ljspeech_dataset.py verified all 13,100 wavs (22,050 Hz / 16-bit / mono PCM, confirmed by reading the RIFF headers, not assumed) and wrote data/ljspeech_audio.i16 — 1,898,881,532 samples, 23.92 h, 3.80 GB — plus the index (12,838 train / 262 val; shortest utterance 1.11 s, so nothing is dropped by the 1.0014 s crop). The model is 7,338,658 params (3,659,936 encoder / 3,660,162 decoder / 18,560 RVQ) plus a 2,112,582-param discriminator. Training took 2 h 54 min for 24,060 steps at 10.8 GiB peak, and validation mel fell 7.313 → 3.075, still improving at the end. Evaluated on 64 held-out utterances (398.5 s), the bitrate ladder is monotone on all three metrics — 0.69 kbps: −4.84 dB SI-SDR / 3.968 mel; 1.38: −0.87 / 3.528; 2.76: +0.86 / 3.249; 5.51 kbps: +1.81 dB / 3.083 mel — all four rungs served by the same model, which is what quantizer dropout is for. The headline result is the codebooks: all 1,024 entries of all eight are used, and the deepest codebook has the highest perplexity of the stack (903.6, against the first's 792.0). A naive RVQ usually leaves the last codebooks nearly dead; dead-code revival did its work early (6,072 revivals in epoch 1, 205 by epoch 3) and then went silent from epoch 7 on. Judged by ear it is a working codec with an audible metallic edge, not a transparent one — the very low SI-SDR says waveform phase is only loosely tracked, which is the trade the adversarial loss makes. Supporting checks: the hand-written WAV writer round-trips through the hand-written parser at the 16-bit quantization floor (5.32e-05 against a floor of 3.05e-05), and throughput was profiled rather than guessed (see Stage 4). Full tables in the pipeline README's "Verified runs".

• fine-tuning/qwen25-3b-lora — build_qwen3b_dataset.py, train_qwen3b_lora.py, generate_qwen3b_lora.py, plus a real 40-sample smoke train against the actual downloaded Qwen/Qwen2.5-3B-Instruct weights, confirming trainable params > 0 (LoRA genuinely attached to q_proj/v_proj) rather than trusting peft's target-module table alone.

• fine-tuning/vicuna-7b-lora — real 2,000-sample/2-epoch training run (see above), executed by the repo owner, not just a smoke test.

• training/vit-cifar10 — full 60-epoch run by the repo owner on the RTX 3090 against the real CIFAR-10 drop: build_cifar10_dataset.py wrote data/cifar10.npz (50k/10k); train_vit.py trained 10,695,562 params in 1,367 s (~23 min) (train acc 0.30 → 0.714, best val 67.50% at epoch 60); evaluate_vit.py scored 66.82% test top-1 / 97.32% top-5 on the held-out 10k and wrote test_metrics.txt + predictions_grid.png. The ~80–86% expectation this pipeline's docs originally carried was corrected to the measured 66.8% (too optimistic for flip+crop-only at 60 epochs) — see the pipeline README.

• training/mae-cifar100 — new pipeline (MAE on CIFAR-100, the repo's first self-supervised run), full 60-epoch run by the repo owner on the RTX 3090 against the local E:\datasets\cifar-100-python drop: build_cifar100_dataset.py wrote data/cifar100.npz (50k train / 10k test, verified exact counts, fine + coarse labels); train_mae.py (11,766,540 params: 10,750,848 encoder + 1,015,692 decoder, patch 2 → 256 patches, mask 75%) trained for 60 epochs in 2,166 s (~36 min) (train masked-MSE 0.666 → 0.257, best val recon 0.47463 at epoch 34); linear_probe.py scored 25.56% test top-1 / 53.41% top-5 (coarse 37.9%) on the held-out 10k using the final epoch-60 checkpoint — which probes better than the best-val epoch-34 one (24.65% / 53.15% / 36.8%): val recon bottomed mid-run while the masked-MSE kept improving, so the late features are the better representation (both probe metrics preserved in runs/; the comparison is deterministic — re-running the best-checkpoint probe reproduces it exactly) — and wrote test_metrics.txt + probe_grid.png. A 2-epoch smoke preceded the real run (loss decreasing, grids/checkpoints written, ~35 s/epoch).

• training/dit-cifar100 — new pipeline (class-conditional DiT on CIFAR-100, the repo's first class-conditional generative transformer), full 60-epoch run by the repo owner on the RTX 3090 against the same local E:\datasets\cifar-100-python drop: build_cifar100_dataset.py wrote data/cifar100.npz (50k train / 10k test, verified exact counts, fine + coarse labels); train_dit.py (9,828,876 params, patch 2 → 256 tokens, dim 256 / depth 8 / heads 8) trained for 60 epochs in 4,521 s (~75 min) (train velocity MSE 0.5174 → 0.1695, best val 0.1842 at epoch 55); evaluate_dit.py scored 0.1887 velocity MSE on the held-out 10k (EMA weights) and wrote samples_grid.png (100 classes, one per fine class, cfg 3.0), cfg_sweep.png and nearest_neighbours.png. Nearest-neighbour check: generated samples sit ~47% farther from the training set (mean L2 12.105, min 9.572) than real unseen test images (8.210 / 5.152) — no memorization, the same direction flow-matching-mnist measured. The ~0.17–0.19 loss floor is the irreducible conditional variance of the velocity target, not a defect. A 2-epoch smoke + a 1-epoch timing run preceded the real run.

• training/mamba2-tinystories — new pipeline (Mamba-2 SSD on TinyStories, the repo's first non-attention sequence model). build_tinystories_dataset.py verified the canonical separator counts and wrote the memmapped corpus (1,943,728,852 uint16 tokens = 3.89 GB; 2,119,489 train / 21,990 valid stories); the fp32 scan self-test passed at 4.349e-04 abs / 3.797e-06 relative (fp64 6.576e-15). Both legs of the documented config were run to completion on an idle RTX 3090: mamba2 (3,515,008 params) 1,104 steps in 379 s, best val loss 0.8004 (ppl 2.23, 1.1547 bits/byte); the matched transformer (3,527,488 params, +0.355%) 1,104 steps in 235 s, val 1.0053 (ppl 2.73, 1.4503 bits/byte). evaluate_mamba.py re-scored them at 0.8197 vs 1.0002 and showed the transformer's learned absolute positional embeddings collapsing past its trained length (1.63 nats at L=1024, 2.49 at L=2048, against mamba2's 0.91 / 0.85), while the MQAR recall probe stayed at chance for both legs (≈0.002 against 0.0020). 36.2 M tokens is 1.91% of one pass, so none of it is a finding about the two architectures — see the pipeline README's §3b.

• training/meanflow-cifar10 — new pipeline (MeanFlow, the repo's first 1-NFE generative model). train_meanflow.py --dim 128 --depth 4 --num-epochs 20 --batch-size 128 (1,356,556 params) trained 7,040 steps in 1,258 s (21.0 min); train average-velocity MSE 1.2476 → 0.6458, best val 0.6715 at epoch 19. evaluate_meanflow.py scored test 0.6705 average-velocity / 0.2210 instantaneous on the held-out 10k and wrote samples_grid.png (100 class-conditional 1-NFE samples), nfe_sweep.png (the same latents at NFE 1/2/4/10/50) and nearest_neighbours.png (generated mean L2 9.613 vs real test 9.080 — no memorization). The 1-NFE claim is still unmeasured on CIFAR-10: the samples' pixel std is 0.1746 against ~1.0 for real CIFAR-10, so the 0.15897 NFE=1-vs-50 agreement is not evidence. The documented 60-epoch --dim 256 --depth 6 default measures ~10.2 h (0.867 s/step at batch 64; batch 256 exceeds the 24 GB card) and was not run. Not executed this pass (no PDFs/poppler set up in this environment):

• pre-training/exec_1.bat / scripts/convert_pdf_to_png.ps1

Next steps

• fine-tuning/vicuna-7b-lora has a verified-good real run (see above) — reasonable next moves are more samples (the eval_loss plateau suggests more epochs on this same 1,800-row set has limited further upside), judged by the reconstruction test, not loss alone.
• fine-tuning/qwen25-3b-lora is smoke-tested but not yet trained for real — same next step as Vicuna's first run: build a few-thousand-sample JSONL, train, then judge with --jsonl-eval.
• training/ has a verified real cifar10-vqvae run (see above) — the planned next rung is stage 2 of its cascade: a learned prior over the discrete code grid (PixelCNN/transformer over code indices, or a latent DDPM), latent-diffusion style.
• training/imdb-sentiment-cnn is verified at 89.2% test acc with random embeddings in 31 s — natural next rungs: a GloVe variant (+1–3 pts expected), or the bigger from-scratch projects that use the 50k unlabeled reviews (AWD-LSTM LM-pretrain + fine-tune, or a small transformer with MLM pretraining, both ~91% territory).
• training/flow-matching-mnist is verified at ~5.3 min for 40 epochs and was still improving when it stopped — natural next rungs: more epochs or a wider --base-channels; class-conditioning plus classifier-free guidance (the smallest real upgrade, and what makes samples steerable); a second rectification pass (re-train on the model's own noise/sample pairs) to straighten the paths for 1–4-step sampling, which is the whole reason rectified flow is used in production; or the same objective on CIFAR-10 next to cifar10-vqvae, where the VAE-blur comparison has more room to show itself than at 28x28.
• training/rvq-audio-codec is trained and verified (see above) — the audible gap left is the metallic edge, which the very low SI-SDR (+1.81 dB) identifies as loose phase tracking, and which is a training-budget problem rather than an architecture one: EnCodec and DAC run several hundred thousand steps against this run's 24,060. Natural next rungs, in rough order of value: the reconstruction-only A/B (--lambda-adv 0) to measure what the discriminator is actually worth; the collapse-mitigation ablations (--code-dim 128, --vq-l2-normalize 0, --dead-code-threshold 0, --vq-mode loss), whose findings are meant to feed back into training/cifar10-vqvae's single codebook; and then the obvious sequel — an autoregressive prior over the RVQ code indices, which is what turns a codec into an audio LM and is the same "learned prior over discrete codes" rung already planned for cifar10-vqvae. Multi-speaker (LibriTTS/VCTK) is the fix for the single-speaker limitation, but only if generalization becomes the goal.
• training/vit-cifar10 is verified at 66.8% test top-1 / 97.3% top-5 (60-epoch defaults, ~23 min on the RTX 3090; best val 67.50% at epoch 60, still slowly improving at the end) — natural rungs, in rough order: the --no-augment A/B (still unmeasured; measures how much of the accuracy is the flip+crop); a longer run (--num-epochs 120, the cosine schedule is designed for the full budget); stronger hand-written augmentation (AutoAugment-style ops are the biggest known lever on CIFAR-10 ViTs); a deeper/wider variant (--depth 8 --dim 512, ~24M params, still comfortable on 24 GB). The planned sequel this pipeline's patch-embed/block stack was built for — self-supervised pretraining — has now landed as the sibling training/mae-cifar100 (mask-reconstruct MAE rung, same stack reused); the next rung after that is the I-JEPA-style pipeline (small ViT encoder + predictor + EMA target on STL-10/Tiny ImageNet, linear-probe eval), which shares MAE's mask-reconstruct DNA.
• training/mae-cifar100 is verified at 25.56% test top-1 / 53.41% top-5 (coarse 37.9%; ~36 min for the 60-epoch pretrain) using the final epoch-60 checkpoint — it probes better than the best-val epoch-34 one (24.65% / 53.15% / 36.8%), because val recon bottomed mid-run while the masked-MSE kept improving (don't assume the best-val checkpoint holds the best representation). Natural rungs, in rough order: the --no-patch-norm A/B (measures the MAE trick); the --patch-size 4 A/B (vit-cifar10's literal 64-patch grid at the same 75% mask — the denser patch-2 default is expected to win, but it's unmeasured); a --mask-ratio sweep (50/75/90 — the paper's 75% is tuned for 196 patches, not 256); more pretraining epochs (--num-epochs 120, the cosine is designed for the full budget); probe-time augmentation or a k-NN probe as an augmentation-free alternative; and then the I-JEPA rung above, which this pipeline's mask-reconstruct machinery sets up directly.
• training/dit-cifar100 is verified at ~75 min for the 60-epoch defaults (9,828,876 params, train velocity MSE 0.5174 → 0.1695, best val 0.1842 at epoch 55, test 0.1887; generated samples sit ~47% farther from the training set than real test images — no memorization). Natural rungs, in rough order: the --cfg-scale sweep on the final checkpoint (the evaluator's cfg_sweep.png shows 1.0–5.0; the trade-off between class fidelity and diversity is the obvious knob to tune per class group); a longer run (--num-epochs 120, the cosine is designed for the full budget — val was still improving at epoch 55–60); the --no-augment A/B (measures how much of the quality is the flip+crop); a bigger model (--dim 384 --depth 12, the literal DiT-S/2 sizing — ~29M params, still comfortable on 24 GB); and the natural sequel this pipeline's conditioning machinery sets up directly: class-conditional generation at higher resolution (CIFAR-10/STL-10 at 64×64, or latent-DiT on cifar10-vqvae's codes), which is what turns the Sora-style stack into a production-shaped generator.
• training/librispeech-speaker-id is a new pipeline with two verified runs (see above and the pipeline README). The --arch xvector A/B is done, and its result inverted the expectation — x-vector won the val EER that selects the checkpoint and lost the held-out-speaker EER — so the highest value next rung is not a model change at all: select the best checkpoint on held-out speakers rather than held-out utterances of seen speakers, which is what the reversal says the current criterion gets wrong. Close behind it: a fourth held-out utterance split the trainer never touches (the builder writes three splits; the closed-set number is optimistic without a fourth). Then the ordinary knobs: the --no-speed-perturb A/B (how much of the result is the augmentation), --channels 512 (the published ECAPA width, ~4x the compute), a --crop-seconds sweep (2 s is the common choice, 3 s the other), and ECAPA's successors, which are the same trainer with different blocks. The higher-value rung beyond the model is the one this pipeline was built to unlock: the same corpus builder and log-mel front-end are what a planned CTC-ASR pipeline needs, so its real return is retiring that data risk — see the pipeline README's "Why this pipeline belongs here". Beyond that, a speaker-conditioned rvq-audio-codec (the embedding as a conditioning vector rather than a classification target) is the direct fix for that codec's documented single-speaker limitation, and is the first place this repo's two audio pipelines would meet.
• fine-tuning/llava15-full-lora (planned, not started): the first real VLM fine-tune in this repo — image+text pairs, vision encoder/projector actually in the training graph, unlike vicuna-7b-lora/qwen25-3b-lora.
• A phi35-mini-lora sibling (discussed, not started) would need target_modules=["qkv_proj"] instead of ["q_proj", "v_proj"] — Phi-3 fuses Q/K/V into one linear layer (confirmed by reading Phi3Attention's source), so the vicuna-7b-lora/qwen25-3b-lora target-module config would silently attach to nothing on that model.