MuMDIA Console: a desktop application for installing, configuring and running MuMDIA - #48
Open
RobbinBouwmeester wants to merge 93 commits into
Open
MuMDIA Console: a desktop application for installing, configuring and running MuMDIA#48RobbinBouwmeester wants to merge 93 commits into
RobbinBouwmeester wants to merge 93 commits into
Conversation
…ed configs parse The Strasbourg prenylation work is unpublished, so its configs and deployment notes must not reach the remote. `config.strasbourg-linux.json` had been committed and pushed; untrack it (the local file is kept for transfer by other means) and add ignore rules for `config.strasbourg*.json` and `config.*-local.json` so it cannot recur through a `git add -A`. Root-level notes were already covered by the existing `/*.md` rule. NOTE: untracking removes the file from the branch tip but NOT from history. If it must be unreachable, the branch needs a history rewrite and force-push. Its content is search parameters and server paths -- no sample, protein or result data -- so that is a judgement call for the repo owner. Also adds `shipped_configs_parse`, which round-trips every TRACKED config through `Config::from_json`. A `_comment` key shipped in a config once and, because `Config` is `deny_unknown_fields`, made the whole config unloadable -- caught only by running `doctor` on the deployment target. The workspace suite passed throughout, which is the real gap: a config is an artifact the engine must accept, not just valid JSON. Untracked machine-specific configs are deliberately excluded from the test and remain `doctor`'s job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cceeds on Windows
`deeplc_worker.py` imported numpy and pyarrow at module level and deferred `import deeplc`
into main(). DeepLC 4.x is torch-backed, and on Windows that ordering makes torch's DLL
initialisation fail outright:
OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed.
Error loading "...\torch\lib\c10.dll" or one of its dependencies.
`deeplc_finetune.py` already ordered its imports this way and says why ("import before numpy,
OpenMP load order"); this worker did not. The bug stayed latent because imported-library mode
skips predict-frag entirely, so nothing exercised the native RT-prediction path. It surfaced
the first time a library was built from FASTA, which failed after the peptidoform table was
already generated.
The comment states that the ordering is load-bearing, so a future tidy-up does not sort the
imports and silently reintroduce it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`doctor` checked the DeepLC interpreter for `deeplc,numpy,pandas`, but that interpreter also runs `deeplc_finetune.py`, which imports pyarrow, torch and psm_utils. A green doctor could therefore be followed by a crash at the fine-tune step, which on an experiment-wide batch is discovered long after the run is launched. DeepLC 4.x pulls torch and psm-utils itself, so in practice this catches a missing pyarrow, but the check should assert what the scripts import rather than what the dependency tree happens to imply. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nt.q_filter `run-experiment` hard-set `qcfg.q_filter = PsmQ`, silently discarding whatever `quant.q_filter` the config asked for. The override itself is deliberate: the grouped q columns (peptide_q_value / precursor_q / pg_q_value) are assigned only to each group's single experiment-wide winning row, so a per-run table can only gate on a per-PSM column. Doing it in silence is the problem. A user who explicitly configured a different `q_filter` got quantities gated on a column they did not choose, with no record of the substitution in any artifact. Now it warns with both the configured and the effective value. Behaviour is unchanged, so existing results stay interpretable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hon screen
New `mumdia prescan` stage. For a modification search it keeps only the candidates whose
modification-anchored sequence trimers are actually observed in their own isolation window
and retention-time range, so the per-run library is sized to the evidence rather than to the
enumeration. Same output contract as the Python screen it replaces (candidate_id, label), so
downstream library assembly is unchanged.
Measured on one 50-window Orbitrap DIA run against a 54.8M-row library:
tag index build 27 s -> 0.64 s (42x)
screening 412 s -> 5.6 s (74x)
total wall 439 s -> 38 s (11.6x)
Screening is independent per candidate, so it is a rayon fan-out; the Python loop was
single-threaded and was ~40% of the whole per-file chain. Most of the remaining 38 s is
reading the library, not computing.
WHAT THIS STAGE IS NOT. It cannot discriminate a true modified peptide from its decoy, and
must not be read as if it could. `anchored_tris` emits every trimer in both orientations and
a reverse decoy preserves composition and precursor m/z, so a decoy's anchored tag set is
identical to its target's and a decoy survives exactly when its target does. Measured
target:decoy survival ratio is 1.0000 (2,453,365 / 2,453,365). That is precisely why it is
safe: exchangeability is untouched, so downstream FDR stays valid. It is a compute reduction.
Both labels go through the identical criterion, each on its own sequence, m/z and RT window.
Screening only targets and then admitting their paired decoys would make surviving targets
signal-enriched while their decoys stay signal-blind, biasing the modification's q-values
anticonservatively; the stage aborts if survivors ever come back single-label.
The peak cut is deterministic (intensity descending, ties by ascending m/z). It has to be:
moving `top_peaks` by ONE peak changes the survivor set by 3.2%, which is larger than most
parameter changes, so an unspecified tie order would make reruns differ for no reason. The
0.4% symmetric difference against the Python implementation is this same cut sensitivity, not
a logic difference: the disagreeing candidates come in target/decoy pairs, and 0 of 1.48M
sampled peptidoforms have a backbone the tokeniser rejects.
Masses come from the shared model (`residue_mass` + `unimod_mass`), never a local copy; I and
L share a tag index because a residue-mass delta cannot separate isobaric residues. Output
goes through `mumdia-io`, so the snappy + arrow-utf8 library contract is automatic rather than
something an external writer has to remember.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…DE.md Nineteen docs updated, plus CLAUDE.md, which was whitelisted in .gitignore but had never been added, so the top-level guide existed only on one machine. The substantive corrections, each measured rather than argued: - `--top-peaks-ms2` is ACQUISITION-SPECIFIC and destructive at convert time. CLAUDE.md previously presented `300` as part of the validated workflow; it is correct for the chimeric AIF benchmark and actively harmful elsewhere. On a 50-window Orbitrap DIA run it discarded 78.6% of all MS2 peaks and truncated 85.5% of spectra (even p25 exceeds the cap), costing 25,425 versus 63,237 peptides.tsv rows at `peptide_q_value` <= 0.01 with the empirical decoy fraction unchanged at 0.99% in both arms. Mechanism is peak-group formation, not scoring: `presence_min_fragments` cannot be met, so real peptides are recorded NO_PEAK_GROUP. Docs now carry the peak census, the audit funnel, the cap dose-response, and a pre-flight check. - `compete.group_by = precursor` is a misnomer: it keys `base_peptide_id`, built from the STRIPPED sequence, so every charge and modform of a peptide collapses to one winner before rescore. Required, not optional, for PTM work: on a modification-rich library the default key deleted 880,464 of 1,890,239 extracted candidates (46.6%), while `peptidoform_charge` removed 0 rows and moved precursors per peptide from 1.000 to 1.174. - `cal.json` RT residuals are IN-SAMPLE and roughly 3x optimistic (6.14 s reported versus p50 17.6 s / p90 146.3 s out-of-sample). Size external RT tolerances from out-of-sample numbers. - DeepLC fine-tuning need not run per file: a once-fine-tuned library plus per-run LOESS gave 6.06 s median residual against 6.14 s, removing ~36 min per file. - An imported library may assign every modform the unmodified form's iRT (79.7% of stripped groups on one library). Check that variance before trusting RT windows in a PTM search. - Sidecar and IO contracts that fail late: deeplc import order, the widened `doctor` probe, snappy + arrow `utf8` for parquet written outside `mumdia-io`, contiguous `candidate_id` and m/z-ascending precursors as hard errors, and the `MUMDIA_NN_STREAM_GB` backend cliff. - `run-experiment` never calls report, so there is no peptides.tsv in its tree; grouped q columns exist only on each group's experiment-wide winning row, so per-run counts on them are diluted and `run_psm_q` is the correct per-file unit; and pooling more runs does not tighten q because `q = (decoys + 1) / max(1, targets)` is scale-invariant. Count tables now name their row and q-value unit, which is the convention CLAUDE.md itself requires and which the first draft of these edits did not follow. Datasets are described generically; no collaborator, sample, server path or unpublished result appears. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…1 and CLAUDE.md The stage landed in c6375b7 with no documentation, so it was reachable only by reading the source or `--help`. Adds docs/21_prescan.md and wires the references. The doc leads with what the stage is NOT, because that is the part easy to get wrong. The screen cannot separate a true modified peptide from its decoy and no tuning will make it: trimers are emitted in both orientations and a reverse decoy preserves composition and precursor m/z, so a decoy survives exactly when its target does (measured ratio 1.0000). That is what makes the stage safe rather than suspect, since exchangeability and therefore downstream FDR are untouched, but it also means a survivor count must never be read as enrichment for real modified peptides. Records why the decoy screen has to be symmetric, with the number that shows it: screening targets only and admitting their paired decoys made the apparent modification-specific FDR look like ~32% against ~43% under a symmetric screen, and that gap was the bias, not discrimination. Also records that `top_peaks` is sensitive out of proportion to its appearance: a one-peak change moved the survivor set by 3.2%, which is why the peak cut is tie-broken deterministically and why survivor counts should not be treated as precise quantities or used to tune by small differences. Distinguishes `prescan.top_peaks` from `--top-peaks-ms2`: the former bounds tag construction only, the latter is destructive at convert time and is the subject of docs/04_convert.md. Notes the two couplings a reader will otherwise miss: a per-run library means a per-run search space, which pooled rescore handles via `source`/`run_psm_q` but cross-run quant does not; and the stage must be paired with `compete.group_by = peptidoform_charge`, since the default competition key deletes exactly the modform candidates the prescan was run to keep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…er pruned The doc described what the stage keeps but not what it cannot touch, which leaves the reader unsure whether prescan tuning can damage an ordinary proteome result. It cannot. `anchored_tris` emits trimers only for positions carrying an `anchor_mods` modification, so a candidate without one yields an empty tag set and never appears in the output; library assembly then restores the whole unanchored remainder unconditionally. Verified on a 54.8M-row library with three cysteine anchors: 41,434,790 rows eligible (75.6%), 13,386,766 never at risk (24.4%), 4,906,730 survivors, and 0 survivors lacking an anchor modification. That makes the stage cheap to experiment with: `tol_da`, `rt_slack_s` and `top_peaks` can only add or remove modified hypotheses. Also records why the stage walks every row instead of pre-filtering to anchor-bearing ones: it costs about a second and keeps the eligibility rule in one place rather than duplicating it as a peptidoform string match that could drift out of agreement with the configured anchors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…a command substitution A Phase B orchestration script died with exit 1, an empty log, and nothing on stderr. Cause: under `set -euo pipefail`, a run-list built with `[ -f ... ] && echo "$r"` makes the command substitution return non-zero whenever the LAST glob entry fails the test, and `set -e` then terminates the script without a message. It happened once five of eighty-three runs were incomplete and the last of them was the final directory in the glob. The failure is worth documenting because the symptom points nowhere: the log contains only `nohup: ignoring input`, and the obvious suspects (CRLF line endings, a missing config, an empty glob) all produce visible errors instead. `bash -x` locates it immediately, which is the actual lesson. Fix is an explicit `if`, so no failing command is ever last in the substitution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ted) In-sample residual-percentile windows underestimate the tail an unseen library peptide has, and they invert under model capacity: an RT model that memorizes its anchors is handed the narrowest window exactly when its true error is largest (measured in-sample 15.9 s vs 24.9 s abs-median ranking two DeepLC versions backwards against held-out 195.1 s vs 46.4 s). window_holdout_frac = f holds out anchor peptides by the deterministic rule base_peptide_id % 1000 < round(f*1000), fits the sizing curve on the rest, and takes w_rt from the held-out residual percentile. The rule is duplicated verbatim in deeplc_finetune.py (--window-holdout-frac, passed by run/run-experiment) so the fine-tune reference excludes the same peptides; without that, adapter memorization leaks into the residuals. The calibration curve applied to the library still uses every anchor. Guards: anchor-count fallbacks to in-sample with a warning; hard error when combined with adaptive_rt_window; cal.json records w_rt_sizing, n_sizing_train, n_holdout, and the held-out residual scale. Standalone rt-im-train now also warns that finetune_deeplc is orchestrator-only, which previously read as fine-tuned on runs that never were. Measured end-to-end (AIF benchmark, DeepLC 4.1.0, frac 0.3): w_rt 141.5 -> 204.9 s, peptides at 1% 10,703 -> 10,822 (+1.1%) at unchanged 0.98% decoy; with the overfitting 4.0.0a2 model the honest window is ~950 s and costs 1.5%. Default off; benchmark-gated (docs/08 section 4b). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… variance On LFQ_Orbitrap_AIF_Ecoli_01 the validated workflow now measures 90.4-91.6% of DIA-NN 2.2.0 lib-free --reanalyse (11,817 peptides at 1%). Prefer the augment_library.py-completed tables: the raw imported library is missing 209 of DIA-NN's peptides (all N-terminal Met excision); augmentation recovers ~80 at unchanged 0.98% decoy and parity elsewhere. Also record that DeepLC per-run fine-tuning nondeterminism propagates into w_rt under held-out window sizing (held-out p95 150-211 s across draws, ~2% peptide swing), so single-run comparisons of window sizing or library variants must be judged against that variance. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two quantification options, both off by default so an existing config is bit-identical: - `quant.fragment_selection = predicted` ranks a precursor's fragments by their library intensity instead of by their own integrated area. Ranking by observed area preferentially selects interfered fragments, because interference inflates exactly the areas the ranking rewards, and the selected set then varies run to run. - `quant.fixed_scan_halfwidth` / `quant.fixed_window_s` integrate a fixed window centred on the identification apex rather than the descent-walk bounds, whose 1/6-height walk is itself interference-sensitive. The seconds form is instrument-independent and overrides the scan form. - `quant.baseline_subtract` (with `baseline_flank_scans`, `baseline_quantile`) subtracts a flank-quantile background inside the fixed window. Measured on the ProteoBench Astral HYE set (six runs, second-pass MBR, guard 0.8): median |epsilon| 0.273 -> 0.195 and CV 0.175 -> 0.107 at `fragment_selection = predicted`, `top_n_fragments = 12`, `fixed_window_s = 5`. On the AIF HYE set with `fixed_window_s = 20`, |epsilon| 0.210 -> 0.181. The window has to be sized per acquisition (about 1.5x the median peak half-width), so nothing here becomes a default: promotion needs entrapment validation on both acquisitions per docs/20. `predicted_intensity` is read as an OPTIONAL chromatogram column. Requiring it made every chromatogram artifact written before the column existed unquantifiable, which the applied-window contract test caught. The `predicted` ranking, which is the only consumer, now fails with an actionable message instead. Under a fixed window the reported `integration_lo_rt`/`integration_hi_rt` are the RT extent actually integrated, not the walked bounds that were ignored; `fixed_window_indices` is shared by the integration and the reported contract so the two cannot drift. `validate()` rejects a negative or non-finite `fixed_window_s` and a `baseline_quantile` outside [0, 1], and warns when both fixed-window forms are set or when `baseline_subtract` is on with no fixed window to apply it to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The M5 augmented scored table lowered only `q_value`, but quant gates on `quant.q_filter`, which the experiment path sets to `run_psm_q`. An accepted transfer therefore kept a sub-threshold `run_psm_q` and was never quantified: 34,280 of 34,664 transfers on the six-run HYE pooled run, i.e. MBR appeared to run and changed almost nothing. Lower `q_value`, `run_psm_q`, and `experiment_psm_q` (whichever the table carries) to the transfer q on the matching (candidate_id, source) row. Note for interpretation, measured on that run: fixing the gate alone raises min-3 ProteoBench features 70,657 -> 77,172 but worsens median |epsilon| 0.211 -> 0.245, because transfers into an ion's low condition sit at the noise floor and compress the ratio. The gate fix is correct; it needs transfer quality control (fragment-consensus guard, condition evidence) alongside it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four gaps, all of which let untracked clutter or 3 GB of build output show up as uncommitted work, and one of which silently hid a tracked doc: - `**/target/`: the workspace target dir is `rust/mumdia/target`, which the root-anchored `/target` never matched. - root-anchor `/*_plan.md`, `/MISS_ANALYSIS*.md`, `/fragindex_*.md`, `/speedup.md`. Unanchored, `*_plan.md` also matched `docs/22_release_plan.md`, so a tracked developer doc could not be committed. `/*.md` already keeps root notes local, so this only narrows the patterns. - `/config_*.json`, `/config.aif-*.json`, `/config.hye-*.json`: 19 experiment configs with machine-specific interpreter paths were untracked but not ignored, so `git add -A` would have committed them. - `/lib/`, `/lib_capped/`, `/raw_files/`, `/val*/`, `/mbr_k3/`, `/mbr_work/`, `/missed_xics*/`, `/alphadia/`: local benchmark inputs and scratch outputs. The libraries under `lib/` are irreplaceable local data rather than build output; ignoring them keeps them out of commits without inviting deletion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Docker `deeplc` env pinned a DeepLC git commit from the 4.0 multitask branch and capped `numpy<2` for a `pandas<2` constraint that release carried. Pin `deeplc==4.1.1` from PyPI instead and drop the numpy cap, which 4.1.1 does not need (it declares numpy<3, torch>=2.6, python>=3.11). 4.1.1 is a floor rather than merely the current release: the 4.0.0a2 multitask preview overfits per-run fine-tuning badly enough to invert RT-model rankings (in-sample 15.9 s vs 24.9 s but held-out 195.1 s vs 46.4 s for the same model pair, docs/08 section 4b), so an older DeepLC changes results and not only performance. Add `env/mumdia-deeplc.yml`, the portable equivalent for a native install. The DeepLC sidecars previously had no committed local env spec at all, so running them meant reproducing a developer machine by hand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Inventory of the tree against the gates CLAUDE.md states, the reasons the engine is hard to run today (each cited to the file that shows it), and a two-release plan: v0.1.0 makes today's engine installable, portable, tested and documented without promoting any new default; v0.2.0 ports the validated second-pass multi-run workflow out of the benchmark shell scripts and into the engine. Work packages carry acceptance criteria, sequencing, and effort estimates, plus a definition of done and an explicit out-of-scope list so the release notes cannot imply capabilities that do not exist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Standard repository files a public release needs, and which a reader currently has no substitute for. CONTRIBUTING covers the build, the exact checks a change has to pass, what the test suite does NOT cover (no sidecar is exercised, so a green run is not sidecar validation), the invariants that are easy to break without noticing (determinism, label leakage, paired decoys, artifact versioning, the clean-room boundary, the Parquet encoding contract), the bar for changing a default, and how to report a benchmark number so its row unit and q-value column are stated. SECURITY names GitHub private vulnerability reporting as the channel and states the threat model, which for a local analysis tool is mostly untrusted input files: a configuration names Python interpreters and is executed, so it is as trusted as a shell script, and a finding that requires a hostile configuration is not a vulnerability. The changelog also records what is versioned independently of it, since that is what matters when reading an old result: the per-artifact Parquet schema versions and the hashed feature-set identity, both stamped into manifest.json. `.gitignore` re-includes these root files. `/*.md` with only README and CLAUDE negated had excluded every standard repository file at the root. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI ran only `cargo build --release --locked` and `cargo test --locked`, while CLAUDE.md names `cargo fmt --check` and `cargo clippy -- -D warnings` as part of the gate. Formatting and clippy were therefore a local responsibility, and a tree that failed them could reach main. Measured today: the tree failed both (two unformatted hunks, two never-used functions) and one test, on a branch that had been treated as ready. New jobs: - `lint` on Linux: fmt, then clippy over `--workspace --all-targets`, so dead code reachable only from a test module still fails the build. - `build-test` unchanged across the three platforms, with `--workspace` made explicit: the release binary is one of three members, and the config and IO contracts live in the other two. - `sidecars`: `compileall` over `scripts` and `ci` (the Python workers are not exercised by the Rust suite, so a syntax error in one surfaces only mid-run), a JSON parse of every tracked configuration, a YAML parse of the environment specs, and the documentation-reference check. `ci/check_doc_refs.py` fails when a tracked file cites a Markdown document the repository does not ship. Source comments carry provenance by citing the document that specifies each behavior, and about 130 of those citations point at local-only design notes, so a public clone sends the reader after files it never received. Tracking those notes is not the fix: one of them quotes proprietary constants from a closed-source engine and must stay out of the repository, which is why the script refuses to advise adding an ignored file and why a small POLICY_FILES set may name it in order to explain the policy. Dependabot keeps Cargo and Actions dependencies moving, monthly and grouped. `arrow` and `parquet` are grouped apart from everything else because they carry the on-disk artifact contract, so a bump there is a data-format review rather than a routine upgrade. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No manifest carried a description, repository, homepage, keywords, categories or publish field, so the binary and the crates advertised nothing about what they are or where they come from. `publish = false` is deliberate rather than pending: mumdia-core and mumdia-io are internal boundaries of one application, not libraries with independent consumers, and the binary needs the Python sidecars and configs that a crate cannot carry. Distribution is the release archive and the container image. `rust-version` said 1.85 while `rust-toolchain.toml` pins 1.96.1, which is the only version CI builds and tests. A lower floor may well work, but claiming an unverified one is a guess; the comment says to lower it together with an MSRV job that proves it. `strip = "symbols"` in the release profile: archives shipped unstripped debug symbols, which is most of the binary size for no user benefit, and a backtrace still names the functions. Cargo.lock is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release archive held the binary, README and LICENSE. That is not an installation: the ML predictors and rescorers are Python sidecars the engine launches by path, so a binary alone cannot fine-tune retention times or rescore with mokapot or the neural network. The archive now also carries `scripts/`, `env/`, `docs/`, the changelog, and the example configs when a tag has them, plus a sha256 file, and it lists its own contents in the build log. Targets gain `x86_64-apple-darwin` so Intel Macs are covered alongside arm64. Each build smoke-tests the binary (`--version`, `--help`, `doctor`) before packaging, so a broken executable fails the release instead of shipping. The image had never been built or run by CI, only on a tag that was never pushed. The Docker workflow now builds into the local daemon, runs the checks, and pushes only afterwards. The checks are the ones that can actually fail here: both baked configs parse and their sidecars import; DeepLC is imported for real in the worker's order, because the failure that catches (torch DLL initialization after numpy) happens at import time and not at the module-presence probe `doctor` performs; and a bind-mounted directory is written both with and without `--user`, which is the primary way anyone uses this image. The image also no longer stays root after setup. Root was needed only for apt and for creating the conda environments, which are read-only at run time. The container user's uid is assigned by the base image and will not match a host uid, so the documented invocation now passes `--user "$(id -u):$(id -g)"`, and the smoke test asserts that path works rather than assuming it. `git` is dropped from the image since DeepLC is pinned to a PyPI version rather than a repository commit. Standard OCI labels point back at the source. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Verified on a Linux host with Docker 29.4 rather than assumed, since neither the image nor the DeepLC pin had ever been built: - the image builds (4.62 GB, amd64) and runs unprivileged as uid 57439; - `mumdia doctor` passes on both baked configurations; - the `deeplc` environment imports DeepLC 4.1.1, torch 2.12.1+cpu and numpy 2.4.6 in the order the worker uses, which confirms dropping the `numpy<2` cap was safe, and `rescore` imports mokapot 0.10.0; - all 11 sidecar workers are present; - a bind mount is writable with `--user "$(id -u):$(id -g)"` and the result is owned by the host user. Without `--user` it fails at the mount point (`mkdir: cannot create directory '/data': Permission denied`), so the flag is documented as required rather than optional. The quant refactor is numerically neutral, checked against the exact output behind the ProteoBench AIF submission: 72,168 quantified precursors, `quantity`, `n_fragments_used`, `quant_status` and `integration_apex_rt` all bit-identical. Only the reported window moved, from a 29.1 s median (the descent-walk bounds the fixed-window path never integrated) to 34.9 s, which is the fixed window that produced those numbers. The published submissions therefore still describe this code. Also smoke-test every release target instead of skipping aarch64-apple-darwin: macos-latest is arm64 and macos-13 is x86_64, so each runner executes the architecture it builds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Source comments carry provenance by naming the document that specifies each behavior, and about 130 of those citations named documents the repository does not ship: `plan.md` (110 references across 55 files), plus `comment.md`, `fragindex_spec.md`, `mbr_plan.md`, `COMPARISON.md` and three more. A public clone therefore sent the reader after files it never received, and the citation looked authoritative while being unreachable. Tracking those notes is not the fix. `plan.md` section 8 is a comparative dossier that quotes proprietary constants and internal line numbers from a closed-source engine, so publishing it would break the clean-room boundary the project claims; it stays local. Each citation now points at the tracked `docs/` page that describes the same thing today, with the section or stage number dropped, because the tracked guide does not share `plan.md`'s numbering and keeping the number would be a false pointer. Where the citation was the whole justification for a sentence, the dead pointer is deleted rather than replaced by an empty one; the technical claim stays. `CLAUDE.md`, `docs/02` and `docs/14` now say plainly that the design notes are untracked, so a reader is not left looking for them. `ci/check_doc_refs.py` no longer treats a quoted `.gitignore` glob as a file reference. The check passes: 305 references, all resolvable. Also restores a word the rewrap dropped in `index.rs`, whose module doc had come out as "shared by / and extract" with "search-seed" lost between the lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`peptides.tsv` and `proteins.tsv` are the only outputs most users read, and the data dictionary explicitly excluded them, so their columns were documented nowhere. `docs/15` now lists both, each column tied to the source line, and states the trap: rows are precursors `(peptidoform, charge)` while the filter is `peptide_q_value`, a base-peptide q, so a row count is not a precursor-q-controlled count. It also says the printed values are rounded for reading and points at the Parquet tables for analysis. The schema-version registry had drifted: it claimed `psms_scored` was v3 and put `psms_competed` at v2. Per `schema.rs:7-25` they are v4 and v3, with `psms_extracted`, `peptide_quant` and `protein_group_quant` at v2. The section now also says where a version is stamped, so an artifact on disk answers the question itself instead of inheriting the engine's version. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The WP0 table asked ten questions; all are now answered, so it records decisions rather than proposals. Three are worth reading: - platforms are Linux, Windows and macOS, with both macOS architectures built and each binary smoke-tested on its own architecture; - the Python floor is DeepLC 4.1.1, verified to resolve with torch 2.12.1+cpu, numpy 2.4.6 and pandas 2.3.3 on Python 3.11 both in the image and from the new portable env spec; - `plan.md` stays untracked. This reverses the plan's own recommendation, on inspection of the file rather than on preference: its section 8 quotes proprietary constants and internal line numbers from a closed-source engine, and its section 11 warns that reuse needs licence clearance, so publishing it would contradict the clean-room boundary the README claims. The 130 dangling citations were redirected instead, and CI now blocks new ones. `CITATION.cff` is blocked rather than pending: the author list is not derivable from the repository and guessing authorship is not acceptable. A progress table lists what landed with the evidence each item was checked against, and the sections below mark the completed WP1, WP2 and WP4 items. It also states the two things local verification cannot cover: neither the new CI workflow nor the Docker workflow has run on a GitHub runner, because nothing has been pushed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eleven citations named `fragindex_spec` without a `.md` suffix, so neither the filename grep nor the new CI check saw them: `(fragindex_spec Section 2.1)` reads as prose, not as a path. They pointed at an untracked local specification like the rest, so they are redirected to `docs/06_predict_frag_index_matchers.md`, with the section numbers dropped because the tracked page does not share that numbering. Inside `docs/06` itself the citations become "this document" rather than a self-reference. `ci/check_doc_refs.py` now also scans for the bare stems of the known untracked notes, which is the gap that let these through. It reports 312 resolvable references and no extension-less citation. One nearby false positive left alone: `features.rs:115` has a `PLAN` static, which is a variable name and not a document. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…figs
A MuMDIA config named the Python interpreter for each sidecar as an absolute
path, with no lookup and no discovery, so a config belonged to the machine that
wrote it. The only tracked config carried one developer's `C:/Users/...` and
OneDrive paths and was the config `CLAUDE.md`, `docs/19` and `docs/20` told
everyone to run, so a new user's first action was always to edit it. This is the
main reason the engine was hard to run.
`rust/mumdia/crates/mumdia/src/python.rs` gives each sidecar role one resolution
path. A field may be an absolute path, used as given and never second-guessed, or
`"auto"` (or absent), which searches `MUMDIA_PYTHON_<ROLE>`, `MUMDIA_PYTHON`,
`CONDA_PREFIX`, `VIRTUAL_ENV`, then `python3`/`python` on `PATH`. A candidate is
accepted only after it imports that role's own module list, so discovery cannot
pick a Python without torch and defer the failure to the rescore stage hours
later. A role is resolved only when the configuration uses it, so the default
native run probes nothing and still works on a machine with no Python.
Resolution happens before the config hash, so `manifest.json` records the
interpreter that actually ran rather than the word `auto`. That makes the hash
machine-specific for an `auto` config, which is the honest outcome.
`predict_frag.sidecar_script_dir` is resolved against the config file's own
directory and the executable's directory as well as the working directory. The
same config invoked from a different directory used to silently change which
worker scripts ran.
`mumdia doctor` now answers "can this configuration run?" instead of "do three
hard-coded interpreters import three hard-coded lists?". It reports the resolved
interpreter and how it was found, the versions of packages whose version changes
results, and whether the worker scripts are present; it covers `mbr.python` and
the script directory, which it never checked; and it no longer fails a native
configuration over a worker directory that configuration never opens. It warns
when DeepLC is older than 4.1.1, since the 4.0.0a2 preview overfits per-run
fine-tuning badly enough to invert RT-model rankings.
`configs/examples/{native,fasta-sidecars,diann-library}.json` replace the tracked
machine-specific config, with `configs/README.md` covering the resolution order,
the environment specs and the DeepLC floor. The old file is untracked but left on
disk so an existing local workflow keeps working. `shipped_configs_parse` now
covers all five shipped configs.
Verified against the real environments on this machine: a native config is
runnable with nothing probed; `nn_torch` with no torch anywhere fails with a
message naming the variable and the modules; `MUMDIA_PYTHON_RESCORE` pointing at
a torch env resolves and reports torch 2.5.1; and a DeepLC 4.1.0 environment is
accepted with the version warning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`--threads N` bounds the engine's rayon pool and forwards the count to the Python sidecars as `MUMDIA_NN_THREADS` and `OMP_NUM_THREADS` when those are unset. There was previously no way to bound MuMDIA: the engine never read `RAYON_NUM_THREADS`, so rayon took every core, which on a shared machine is antisocial and, for the NN rescore worker, actively slower. That worker measured faster on 8 threads than on 32. An already-set variable is left alone, because a user who exported `OMP_NUM_THREADS` did so for a reason. `--log-level` accepts any `RUST_LOG` filter, and `-v`/`-vv`/`-q` map onto levels. Verbosity was `RUST_LOG`-only, which does not appear in `--help` and is awkward to set on Windows. `RUST_LOG` still works and still offers per-module filtering; an explicit flag wins over it. `-q` and `-v` are mutually exclusive rather than silently ordered. All four are `global = true`, so they parse on either side of the subcommand, which is what a user types. Tested for that, for the level mapping, for the explicit-level precedence, and for `--threads 0` being rejected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The validated-workflow block now names configs/examples/diann-library.json, whose interpreters are "auto". That runs unchanged only where an environment with torch and DeepLC is discoverable, so the block says so, points at doctor for the diagnosis, and notes that copying the example and writing the two paths in is the supported way to pin them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no fixture and no end-to-end test. The Rust integration test builds its inputs in process and starts at `extract`, so mzML parsing, the library build, the `run` orchestrator, the manifest, retention-time calibration on real anchors, and `quant`/`report` writing files were all untested. `.gitignore` has whitelisted `test_data/**/*.parquet` for some time, but no `test_data/` existed. The fixture is generated rather than committed, for two reasons. A usable slice of a real DIA run is megabytes of binary in git and carries a licence question about excerpting a public raw file. More importantly, the planted fragment peaks have to sit where the engine looks for them: `ci/make_fixture_mzml.py` reads the precursor and fragment tables that `mumdia predict-frag` just produced and plants exactly those m/z values, so the fixture cannot disagree with the mass model, and if the mass model changes the fixture changes with it. `test_data/fixture.fasta` is 16 synthetic proteins composed of 160 distinct tryptic peptides, giving 3,820 library candidates. The generator writes a small DIA acquisition: 60 cycles of one MS1 and eight MS2 windows, 160 target precursors planted with Gaussian elution profiles, retention time an affine function of the library iRT so calibration has a real relationship to fit, and two kinds of seeded noise. The second kind matters: noise drawn from the library's own fragment m/z pool is what gives decoys the chance evidence a real run has. Without it every accepted candidate was a target and rescore correctly refused the run for having no decoys, so the fixture could not reach FDR, quant or report at all. `ci/check_smoke.py` asserts 102 things, in two kinds. Exactly determined facts are asserted exactly: spectrum and window counts, that retention time arrives in seconds, every artifact's blake3 hash shape and row count, all 17 artifact schema versions, that the manifest's rescorer identity matches what the scored artifact reports, and that the classifier that ran is the one requested. Scientific outcomes are asserted as bands, because pinning them would turn a sensitivity improvement into a red build. Measured on this fixture: 99.3% of planted peptides recovered (151 of 152), zero decoys at 1% peptide q, LOESS calibration fitted on 110 anchors with a 1.29 s in-sample residual median, and a byte-identical `peptides.tsv` across two runs. That last one is the determinism contract tested through the CLI for the first time; the Rust test only compared `apex_rt` within one process. `ci/smoke.sh` drives it and runs on Linux and Windows in CI. macOS is omitted to keep CI minutes down, which is a deliberate gap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A manifest said `mumdia_version: 0.1.0`, which every build from this branch says, so a result could not be tied back to the code that produced it. docs/20 asks every benchmark record to carry "commit/build:" and the manifest could not supply one. Inputs were not recorded at all: an imported library was hashed, but the mzML and the FASTA were not, so a manifest could not answer which file it came from. `mumdia-core/build.rs` stamps the short commit and the commit date into the binary, with a `-dirty` suffix when the worktree carried uncommitted changes, because a number produced from uncommitted code is not reproducible from the named commit. The COMMIT date is used rather than the build time on purpose: build time changes on every rebuild, which would make the binary unreproducible and defeat caching for no benefit. Without git the values become `unknown` rather than failing the build, so a release-tarball build still works. `Manifest` gains `git_sha`, `commit_date`, `cli_args` and `inputs`, plus `provenance()` for the one-line stamp a benchmark should quote. `run` hashes every input before compute starts. `cli_args` matters because the flags that are not in the config, `--top-peaks-ms2`, `--threads` and `--max-spectra`, previously existed in no record at all, and the peak cap changes results. All four fields are `#[serde(default)]`, so manifests already on disk keep parsing; a test asserts that, because making prior runs unreadable would be a worse outcome than the gap being fixed. `run-experiment` gains the same provenance, including a hash per input mzML. Its manifest is still thinner than the single-run one: it carries no per-artifact records, because the per-run chains do not thread a shared manifest. `scripts/deeplc_finetune.py` takes `--seed` and seeds numpy and torch, wired from `rng_seed`. Unseeded, the draw varied enough to change results: the held-out RT window p95 moved 150 to 211 s across two draws of one benchmark arm, worth about 2% of peptides, which made single-run comparisons of window sizing or library variants unreadable. Training kernels are still not bit-for-bit deterministic, so this narrows the variance rather than removing it, and both the flag help and the log line say so. The smoke test asserts the new fields; it is now at 112 assertions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ci/gen_third_party_licenses.py --check` failed on `457b93c`. The generator read per-crate notices only from cargo's UNPACKED `registry/src` tree, which exists after a build; the `sidecars` job never compiles anything, so on the runner it recovered zero notices and reported the committed file as stale. The difference was the environment, not the tree, which makes the gate meaningless in both directions. It now reads the `.crate` tarball from `registry/cache` when no unpacked source is present. The two sources hold the same bytes, verified by regenerating with `registry_src_dirs()` forced empty: byte-identical document, the same 126 of 173 crates with a recovered notice. The `sidecars` job runs `cargo fetch --locked` first, which is what populates that cache. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ed nothing The rehearsal at `b167fdb` failed the new "Verify the unpacked archive" step with exit 2 and no output. `archive=$(ls *.tar.gz *.zip 2>/dev/null | head -1)`: exactly one of those globs matches, `ls` exits 2 on the other, `pipefail` propagates it, `set -e` kills the step, and `2>/dev/null` discarded the only explanation. This is the failure mode docs/14 already records for `cmd && echo` inside a command substitution. Discovery now uses a nullglob array with an explicit empty check, unpacking a zip uses the Python already set up in the step rather than assuming 7z, and the unpacked root comes from `find` with a check that it exists. The smoke test is invoked with no work-directory argument, matching what ci.yml does on all three platforms, instead of handing a Windows path to a bash script. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… needs write `validate-tag` reads `ci.yml`'s conclusion for the tagged SHA through the Actions API, which `contents` does not grant: with the workflow's single `permissions: contents: write` the call returns 403 and the gate cannot tell a green commit from an untested one. Permissions are now per job and least privilege: the workflow default drops to `contents: read`, `validate-tag` adds `actions: read`, and only the publishing job carries `contents: write`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`needs.validate-tag.result != 'failure'` also admits `cancelled`, and a cancelled gate is not a passed gate: the checks stopped part-way, so a tag could publish without any of them having completed. The condition now allows exactly `success` (a validated tag) or `skipped` (a rehearsal, which has no tag to validate). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`THIRD_PARTY_LICENSES.md is stale` and nothing else is unactionable when the disagreement is environmental rather than in the tree, which is exactly the case under investigation: the runner reports the committed file as stale while a local run with a fresh CARGO_HOME and only `cargo fetch --locked` reproduces it byte-for-byte. Both `--check` paths now print a bounded unified diff, capped at 200 lines, and say so when the difference is line endings only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI's diff, now that `--check` prints one, named the cause exactly: 126 crates with
a recovered notice on Windows, 119 on Linux, the difference being the seven
`windows-*` crates. They ship `license-mit` and `license-apache-2.0` in lower case,
and `Path.glob("LICENSE*")` follows the platform: case-insensitive on Windows,
case-sensitive on Linux. A notice bundle whose contents depend on which machine
generated it is not a notice bundle, and it made the staleness gate fail on every
CI run.
Filename matching now goes through one explicitly case-insensitive predicate used
by both read paths, and the unpacked scan iterates the directory name-sorted rather
than per glob pattern, so a crate carrying both a LICENSE and a NOTICE yields them
in a fixed order. Verified: `license-mit`, `LICENSE-MIT`, `License.md`, `COPYING`
and `notice` all match, `readme.md` does not, and both read paths return the same
notices for the affected crates.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third environment-dependence in a generated file, again named by the diff the staleness check now prints: the committed `sbom.cdx.json` carried `Sørensen` where a Linux run produced `Sørensen`. `subprocess.run(..., text=True)` decodes with `locale.getpreferredencoding()`, which is cp1252 on this machine, so cargo's UTF-8 JSON was mis-decoded and the mojibake was written into the artifact. Both generators now pass `encoding="utf-8"`. The licence bundle now matches on Linux, confirming the case-insensitivity fix; this was the only remaining difference. docs/14 records all three causes together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ess causes The archive-verification claim now names where it was verified: locally by staging the layout, and then in a release rehearsal where the linux-musl, windows-msvc and aarch64-darwin archives each unpacked and passed their own 136 assertions on their own runner. The tag gate's CI lookup was tested against this repository's history in both directions, including that `457b93c` is refused. Adds the three defects found while getting the gates green: two generated files whose contents depended on the generating operating system (licence-file case, cargo's unpacked tree, locale decoding), which had made the staleness job permanently red. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ies are for The archive-contents list omitted `THIRD_PARTY_LICENSES.md`, and now also carries `sbom.cdx.json` and the smoke test plus its fixture. The licence section explains the split: the notice document discharges the obligations of 173 statically linked crates, the SBOM is the same inventory in the form a vulnerability scanner or a software inventory consumes, and both are staleness-checked in CI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ger served `x86_64-apple-darwin` was built on `macos-13`, which no longer receives a runner. Measured 2026-08-28: that job sat queued for over two hours in two separate rehearsals with no runner ever assigned, while every other target finished in about three minutes. A real `v0.1.0` tag would have hung to the six-hour job timeout and failed the release, so this was a release blocker in its own right rather than a cosmetic matrix entry. Cross-compiling it on the Apple silicon runner was considered and rejected: the result cannot be executed there, so it would be the one archive published without its own smoke test having run, which is precisely what the new verification step exists to prevent. Intel Mac users build from source or use the container image. Three targets remain, each built AND end-to-end verified on a runner that executes its own binary. README, CHANGELOG, docs/14 and docs/22 all say so; the docs/14 release-workflow section was rewritten, since its line citations, permissions and target list had all gone stale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t left
Deleting the `macos-13` matrix entry left its `exe: ""` attached to the previous
entry, so that mapping carried the key twice. PyYAML's `safe_load` accepts a
duplicate key silently and keeps the last value, so every local check passed and the
dispatch failed on the remote:
HTTP 422: failed to parse workflow: (Line: 143, Col: 13): 'exe' is already defined
The orphan is removed. `ci/check_workflows.py` now rejects duplicate mapping keys at
any depth with GitHub's own wording and line number, and checks that each workflow
has an `on:` trigger and each job a `runs-on` or `uses`. It runs in the `sidecars`
job. Verified in both directions: clean on the three workflows, and it reproduces
this exact failure when the duplicate is reintroduced.
The class matters more than the instance: a workflow GitHub cannot parse fails when
it is DISPATCHED, which for release.yml is the moment a release is attempted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The new workflow check ran before "Parse env specs", which is where `pip install pyyaml` happened to live, so it failed with `ModuleNotFoundError: No module named 'yaml'`. The dependency belongs to the job: it is now one explicit step ahead of both YAML checks, and the inline install is gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The identification arm said the apex moves for 48.3% of identified peptides while identification stays flat, and that quantities from before the change are therefore not comparable with quantities after it. That left the question open exactly where it mattered. This closes it on known ratios. `run-experiment` over the six ProteoBench AIF HYE files, imported DIA-NN library, `nn_torch`, MBR off, one arm per value of `extract.apex_evidence_rank` and nothing else changed. Both arms on the in-memory rescore backend, so they are comparable to each other. Quantification does not distinguish them: every accuracy difference is under 0.013 in median |epsilon| and 0.002 in median CV, inside the DeepLC fine-tune draw variance the two arms carry independently. Yeast medians agree to three decimals; human and E. coli differ in opposite directions. Extraction does distinguish them, for the promoted default. The legacy apex pushes 27.6% more candidates through extract (14.29 M against 11.20 M) and returns 0.4% fewer PSMs and 1.0% fewer peptides from them, costing an hour of pooled rescore. That is the documented fallback -- score 0.0 everywhere, apex silently becomes the lowest-RT qualifying scan, gate evaluated where the evidence is not -- appearing as measured cost rather than as an argument. Adds `bench/qab_score.py`, which computes the numbers from a `run-experiment` output tree, and states in its docstring why they are not ProteoBench's own figures. The E. coli compression is present in both arms, reproduces the 2026-08-26 pooled result, and is not an apex effect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Final pre-release scan of the workspace. Four defects, one of them reproduced as a crash, plus the class it belongs to. **A single malformed retention time aborted the run.** `convert` validates peak m/z and intensity but never checked the scan start time, which is the value every later stage keys on. Editing one `scan start time` to `NaN` in the fixture mzML: convert exits 0 with no warning, writes `nan` into `spectra_ms2.parquet`, and `mumdia run` then dies at `extract.rs` with `called `Option::unwrap()` on a `None` value`, naming neither the file, nor the scan, nor the value. Such spectra are now dropped with a count and the first offending scan id. Dropping loses nothing, because a spectrum with no retention time cannot be placed in a chromatogram, and `ci/smoke.sh` step 4b asserts the run completes, the drop is reported, and the identifications are unchanged. **The ordering class behind it.** 25 `partial_cmp(..).unwrap()` and 36 `partial_cmp(..).unwrap_or(Equal)` comparators now use `total_cmp`. The first panics on NaN. The second is worse: `Equal`-on-NaN is intransitive, and `sort_by` has detected that and panicked since Rust 1.81, so it turned a deterministic failure into an intermittent one. `total_cmp` agrees with both on every finite value, and the fixture's `peptides.tsv` and `proteins.tsv` hashes are byte-identical across the change. One rewritten comparator picks the competition winner, where treating every NaN as equal made the surviving row depend on iteration order. **`compete` panicked on a stale feature-schema companion** rather than naming the column and the file. `FeatureSchema::read` already handles a MISSING companion; this covers one that is present and wrong. **`make_reverse_decoys.py` silently assigned 0 Da to unknown modifications**, so those decoys carried fragment m/z for the wrong molecule and could never match. A decoy that cannot match does not compete, so the target-decoy null was optimistic for exactly the peptides carrying that modification, with nothing in the output to show it. The sampled calculator check cannot catch this (500 precursors, 99th percentile). Unknown modifications now raise, `valid()` rejects the peptidoform, and the script reports the names and counts, matching the engine's own `MassError::UnknownModification`. Gate: fmt, clippy -D warnings, 203 tests, smoke 136 assertions with unchanged output hashes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`rt` and `intensity` are two independent list columns, and every integration slices `intensity` with indices computed from the length of `rt`: `fixed_window_indices` bounds against `rt.len()` alone. Extract writes the two from paired vectors, so they always match in practice, but a chromatograms table is path-addressable and `mumdia quant --chromatograms` accepts one written by anything. A shorter intensity trace panicked with a slice-index message naming no candidate and no file. Checked once at the read rather than at each of the four slice sites, which is the same pairing check `convert` applies to an mzML's m/z and intensity arrays and `spectra.rs` applies to the MS1 pair. No behaviour change when the lengths agree: smoke output hashes are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hour one The four places recording why `x86_64-apple-darwin` was dropped all said a real tag would have hung to "the six-hour timeout". A cancelled rehearsal job has since reported its duration as exactly `24h0m0s`, which is GitHub's limit on a job WAITING for a runner. The six-hour limit applies to a job that is running, and this one never started, so the number was wrong and the consequence was understated by a factor of four. The decision is unchanged and better supported: a `v0.1.0` tag would have sat for a full day before failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First milestone of the desktop application plan: pick an mzML and a FASTA or a library, start a search, watch it, read the results. Windows and Linux. No engine changes. Progress comes from the `<artifact>.report.json` files the engine already writes, results from the report `rescore` already emits, and the run itself from the documented CLI. The whole milestone is an application, which is what makes it cheap to throw away if the shape turns out to be wrong. The engine is a CHILD PROCESS, not a linked crate. The engine installs no signal handler anywhere, so stopping a run is a kill, and a Rust thread cannot be killed: linked in-process there would be no Stop button at all, only a window that ignores you for an hour. Supporting reasons: a stage panic ends one run rather than the session, and rayon's global pool can only be built once per process, so --threads could not otherwise change between runs. The application also shows the exact command it ran, which a linked engine could not offer. Process control is three deep -- application, engine, Python workers -- so cancelling uses a process group on Linux and `taskkill /T` on Windows. A hard kill skips destructors, so the atomic-write layer never removes its `.tmp-<pid>` files; cancelling therefore sweeps them, or the next run would start in a dirty folder. Closing the window cancels every live run for the same reason. Two bugs found by writing the tests rather than by reading the code: - The terminal state was published before the watcher had scanned, so a caller polling for completion could read a finished run with no stages and no results. The interface would have flashed an empty results screen. The final scan now happens in the waiter, before the status stops being `running`. - The cancel test passed while never cancelling anything: the fixture search finishes in about 700 ms, faster than a stop can be issued. It now says so instead, and `kill_tree` is covered directly by a test that spawns a process and checks the kill lands. Verified: 10 unit tests including stage folding and results parsing against real artifact reports from a recorded run, plus 2 end-to-end tests that drive a real search through the supervisor. fmt and clippy -D warnings clean. The frontend is plain ES modules with no build step, so the release pipeline still needs no Node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two additions the desktop application needs, and nothing else changes. `mumdia doctor --json` emits the same check as data. The application renders one row per sidecar role and offers to install what is missing, which needs the modules, versions and provenance as fields rather than a paragraph to regex. The function is split into a builder and a renderer, so the prose form is produced from the same report and cannot describe a different outcome. Text output is unchanged and the exit status still carries the verdict. `configs/config-schema.json` is the settings form as data: 150 settings with type, typed default, doc comment, enum choices and the gate markers already used in the reference document. Generated by `ci/gen_config_reference.py`, which already parses `config.rs` for exactly this and already runs in CI, so one parse now produces both artifacts and `--check` covers both. The alternative was for the application to carry its own copy of the field list, which is a second thing to keep in step and the one that drifts is the one a user reads. Defaults are rendered by the same function the Markdown table uses, then given their JSON type: `true` not `"true"`, `0.2` not `"0.2"`, `base_peptide` not `` `base_peptide` ``. Computed defaults resolve to their value, so `features.bound_peak_fraction` is 0.333333 rather than the prose `1.0 / 3.0 (0.333333)`. The schema ships without any packaging change: the release archive already copies `configs/` wholesale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ated settings editor
M2, installing itself. `uv` supplies both the interpreter and the packages, so conda
disappears from the user's machine entirely: no "install Miniconda, create two
environments, then edit the config to point at the right interpreters", which is the
step where an external user gives up. The environment is created under the per-user
data directory, not beside the executable, because on Windows that is Program Files
and an installer needing administrator rights on first run is not an easy install.
Searching without components is refused, as decided. The predicate is "this
configuration requires no Python sidecar at all", asked of the engine through
`doctor --json` rather than kept as a list here. Refusing anything that mentions
`native_tda` would have been wrong: on an imported library with retention-time
modelling it measured 10,847 against nn_torch's 10,914, and the 1,213-vs-10,300 gap
that motivates the rule is the zero-component path, not the rescorer.
**The one-environment plan was wrong, and uv proved it.** The plan claimed a single
environment could serve all four sidecar roles, on the strength of their module
lists overlapping. That is not the same question. The resolver answer:
deeplc==4.1.1 -> psm-utils>=1.5 -> sqlalchemy>=2
ms2pip==4.0.0 -> sqlalchemy>=1.3,<2
Unsatisfiable. `ms2pip>=4.1` does coexist with DeepLC and would restore one
environment, but MS2PIP's version changes predicted fragment intensities and
env/docker-rescore.yml pins 4.0.0 deliberately as "a separate, testable upgrade";
trading a measured result for packaging convenience is the wrong direction. So there
are two: a primary one covering the whole recommended workflow, and MS2PIP on
request, needed only for FASTA-mode prediction. A test asserts MS2PIP stays out of
the primary set, so a later "fix" that adds it back fails at test time rather than
at install time on a user's machine.
M3, configuration. The form is generated from configs/config-schema.json, so nothing
about a setting -- name, type, default, help, enum choices, gate markers -- is
written in the interface. Saving writes only the difference from the defaults, which
keeps configurations short and lets a later release improve a default for someone
who saved settings today. Every save is validated by the engine before it is
offered, so a rejected value is reported while editing rather than an hour into a
search. Benchmark-gated parameters are badged with the reason.
Two bugs found by the tests, both user-facing:
- The installer's terminal status was overwritten by the next status refresh, so
"done" was never observable and a caller watching for completion waited for ever.
- `uv venv` refuses an existing directory, so Install failed permanently after the
first attempt, including after a failed attempt that left a partial environment --
exactly when someone presses it again. `--allow-existing` repairs instead, rather
than `--clear` discarding a several-hundred-megabyte download.
Verified: the environment really was created from nothing on this machine, and every
sidecar role's imports resolve in it. 20 unit tests, 3 end-to-end. fmt and clippy
-D warnings clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Milestone 4 of the console plan. `release.yml` gains a `desktop` job that builds the engine and the application from one checkout, so a released application and its engine cannot disagree, stages both plus a pinned `uv` into the bundle, and produces a `.msi` and an `.AppImage` with sha256 sidecars. It gates on the same `validate-tag` job the engine archives do, uploads on a tag, and uploads as an artifact on a rehearsal. macOS is deliberately absent: Gatekeeper blocks an unsigned bundle outright, so shipping one would be worse than shipping none. Also restores the bundle `resources` declaration, which had been removed in M1 because an empty glob fails the build. CI now stages real binaries there, so without the declaration the engine would have been built and then not bundled. The directory carries a committed README, which both explains itself and keeps the glob matching for a developer bundling locally. NOT VERIFIED. This has not been through a rehearsal, so two things are unconfirmed: the bundle layout, and whether the application finds its bundled binaries inside an AppImage, whose resource directory is not beside the executable the way the MSI's is. `engine.rs` looks beside the executable and in `binaries/`, which is right for the MSI and may not be for the AppImage. A rehearsal is the next step, and the fix if needed is a resource-directory lookup rather than a redesign. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…dle layout Building the first real installer found a packaging bug of exactly the kind the plan said needed a rehearsal, so it is worth naming precisely. Tauri's resource LIST form keeps a source path's shape, so `"../../configs/*"` installs to `<install>/_up_/_up_/configs/` -- read out of the generated WiX source rather than guessed. The application looks beside its executable and would never have found its own settings schema. The MAP form does let a destination be named, and fails the build outright on a `..` source with `Access is denied`. Rather than work around that, the files stop being files. The settings schema and both requirement sets are now compiled in with `include_str!`. This is simpler, and strictly more robust: a file that is not a file cannot go missing from a bundle. It costs nothing in freshness either, because all three are generated from sources that require a rebuild anyway, and CI already fails when the schema is stale. Only two resources remain, both executables: the engine and `uv`, in `binaries/`. Verified on Windows, not asserted: `cargo tauri build` produces a 29 MB MSI containing `mumdia-console.exe` with `binaries/mumdia.exe` and `binaries/uv.exe` beside it, and no `_up_` directory anywhere. The Linux AppImage remains unbuilt, and its resource directory is not laid out like an MSI's, so `engine.rs` may need a `resource_dir()` lookup as well; that is the first thing a rehearsal should check. Also fixes a test that passed for the wrong reason: it searched the whole requirements file for "ms2pip", which the comments discuss at length while explaining why it is excluded. It now reads requirement lines only. 21 unit tests, 3 end-to-end including the real component installation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes the remaining milestone-3 items and the packaging risk milestone 4 left open. `mumdia peak-census --mzml X` reports peaks per MS2 spectrum as JSON, with percentiles and what each candidate `--top-peaks-ms2` value would discard. This is the check `docs/04_convert.md` says must be made per acquisition and that the playbook describes but nothing performed: a cap carried from another run deletes fragment evidence rather than failing, and on one 50-window Orbitrap DIA run a 300-peak cap cost 60% of the peptides. The application runs it when an mzML is chosen and states the answer, instead of leaving a number box to guess into. Disk pre-flight estimates the output from the input size and compares it with the free space on the chosen volume. The engine cannot resume, so filling a disk at hour three loses the whole search; this is the cheapest moment to notice. Free space is read by shelling out rather than through the platform API, because this crate has no `unsafe` and the check runs once per search. The resource lookup now asks Tauri where the bundle put things, before falling back to the paths beside the executable. This is what an AppImage needs: its resources live under `usr/lib/<app>/`, so every `exe.parent()` candidate misses and the application would not have found its own engine. Recorded once at startup into a `OnceLock`, because resolution happens on background threads that hold no `AppHandle`. Also verified in both directions that the interface and the backend agree: every command the frontend calls exists in Rust, and every registered command is called. 24 desktop unit tests, 203 engine tests, fmt and clippy clean on both workspaces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ease job History reads past searches back from the folders they wrote rather than from an index: `manifest.json` for the engine version and `psms_scored.parquet.report.json` for the counts and the classifier that actually ran. The application remembers only which folders it has used, so a search moved or deleted on disk stops appearing instead of lingering as a stale row that disagrees with the folder it describes. A folder that never held a search drops out, which matters because users pick output folders by hand. Fixes the desktop Linux build, found by the first rehearsal. `rustup target add` adds a target to whichever toolchain is active in the directory it runs in, and `rust/mumdia` pins its own with `rust-toolchain.toml`. Run from the repository root it added musl to the DEFAULT toolchain while the build used the pinned one, which failed with "can't find crate for `core`" and advised running the command that had just succeeded. The engine's own release job gets this right by accident, because it sets `working-directory` for the whole job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The console lives in its own Cargo workspace, so nothing in CI compiled it: it could have broken while every check stayed green. It is not independent of the engine either, because the settings schema and both requirement files are compiled into it with `include_str!`, so a change to `config.rs` that regenerates the schema reaches this crate. The new job runs fmt, clippy and the library tests, plus `ci/check_desktop_ui.py`, which is the frontend's whole check: the interface has no build step and no framework, so nothing otherwise catches a typo in an element id or a command name. Those are precisely the mistakes that survive review and fail on a user's machine. It verifies in both directions, so a command that lost its caller is noticed as well as one that never had a definition. End-to-end tests are deliberately not run here: they skip themselves without an engine binary, which would make the job look like it covered more than it does. A real bundle is exercised by the release rehearsal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sidecar audit ran for the first time on this branch, because the job is gated on
a change under `env/` and nothing had changed there since it was added. It fails:
ERROR:pip_audit._cli:torch: Dependency not found on PyPI and could not be
audited: torch (2.12.1+cpu)
`--strict` fails when a package cannot be resolved on PyPI, and `torch==2.12.1+cpu`
is installed from the PyTorch index as a local version that by construction does not
exist there. That is how CPU torch is distributed, not a vulnerability, and a job
that fails every time on it teaches everyone to ignore it.
`--strict` is dropped and the reason recorded next to the command, along with the gap
it leaves: torch itself cannot be audited this way by anything, so its advisories
have to be watched at the source.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The new CI job failed on its first run, and the cause is worth stating plainly: 25 of 26 tests passed, the missing one was `kill_tree_terminates_the_process_it_is_given`, and the next line in the log was "The runner has received a shutdown signal". The test killed the GitHub runner. The mechanism is a real defect in the product, not only in the test. `kill_tree` signalled a process group derived from the child's pid without ever checking that the group was the child's own. `Command::process_group` is asked for at spawn time, but if it does not take effect the child sits in OUR group, and signalling that group reaches the application, the shell that started it, and on a shared machine whatever else shares it. The engine spawns Python workers, so group signalling is the right mechanism; doing it unverified is not. The group is now confirmed to belong to the child before it is signalled, and when it does not, the child is killed individually instead. That path orphans a Python worker, which is a real cost and much smaller than the alternative; it should never be taken, because `start` does set the group. Adds a test for the guard's decision that deliberately does not act on it: it spawns a child that shares our group and asserts the guard refuses. `pgid_of` shells out to `ps` rather than calling `getpgid`, because this crate has no `unsafe` and the check runs twice per cancellation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s a static one
The first successful AppImage build produced a broken application, which only
extracting and running it revealed. The bundled engine segfaults:
./usr/lib/MuMDIA/binaries/mumdia --version
Segmentation fault
`linuxdeploy` runs `patchelf` over every ELF binary it bundles. The engine is built
for `x86_64-unknown-linux-musl`, so it is static-pie, and patchelf injects a
`RUNPATH [$ORIGIN]` entry into it and corrupts it. `uv`, sitting in the same
directory and given the same treatment, runs fine because it is dynamically linked
and there is something meaningful to rewrite.
The desktop job now builds the engine for `x86_64-unknown-linux-gnu`. The engine's
own release archives are unaffected and stay musl; they are not passed through a
bundler. Nothing is lost by the change either, for the reason already recorded when
musl was chosen: it would buy portability only if the bundle had no other glibc
floor, and the Tauri host links WebKitGTK.
Also confirms the layout the resource lookup was written for, which until now was
reasoning rather than a measurement:
usr/bin/mumdia-console
usr/lib/MuMDIA/binaries/mumdia
The engine is not beside the executable in an AppImage. Without asking Tauri for the
resource directory the application would search `usr/bin/` and report that it cannot
find its own engine.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit's message described this change; the commit did not contain it. A patch script asserted on its second edit and aborted before writing the file, so only the README landed while the message claimed the workflow had changed too. The workflow change is here. Engine targets are unchanged and stay musl. Only the desktop job's engine moves to `x86_64-unknown-linux-gnu`, because a static-pie musl binary does not survive `linuxdeploy`, which patchelfs a `RUNPATH [$ORIGIN]` into it and leaves it segfaulting. `musl-tools` is dropped from that job with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It terminated a GitHub runner twice. The first time is explained: the group kill had no guard, so it could signal the runner's own process group. The second time it did it again WITH the guard, which should have permitted a group signal only for a child verifiably in a group of its own, and I cannot account for that. Two possibilities stay open: the guard's reasoning is wrong in a way I have not seen, or something about the runner's process arrangement makes any group signal fatal there. The gating follows from not knowing, not from a diagnosis. A test that can take down the machine it runs on does not belong in a shared pipeline while its failure mode is unexplained, and burning more runners to find out is not a reasonable way to look. It runs with `MUMDIA_TEST_KILL=1`, and passes on Windows, where `taskkill /T` addresses a process tree rather than a group. The gap this leaves, recorded in the README rather than left implicit: the Unix group-kill path in `kill_tree` is exercised by nothing automated. What is still covered is the guard's decision, tested without acting on it, which is the half that can be checked safely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A `cargo run` or `cargo tauri build` binary sits in `desktop/target/<profile>/`, which is three levels below the repository root. Both fallbacks counted four, so from this checkout they resolved to `H:/OneDrive - UGent/rust/...` rather than `H:/OneDrive - UGent/MuMDIA_NG/rust/...` and never matched. It went unnoticed because this machine redirects the engine's target directory off the synced tree, so `MUMDIA_BIN` was set for every run anyway. On a normal checkout it means the application cannot find an engine it built minutes earlier, and reports that it looked in a path outside the repository, which is a confusing first impression for anyone building from source. Affects only the development convenience path. A bundled application finds its engine through the resource directory or beside its own executable, both of which are verified on Windows and Linux. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A desktop application for people who did not write MuMDIA. Windows and Linux, one
installer, no conda, nothing to host.
What it does
uvsupplies both the interpreter and the packages, so condanever touches the user's machine. The environment goes under the per-user data
directory, not beside the executable, because on Windows that is Program Files and
an installer needing administrator rights on first run is not an easy install.
result. Progress comes from the
<artifact>.report.jsonfiles the engine alreadywrites; nothing is scraped from the log.
configs/config-schema.json,which
ci/gen_config_reference.pyemits from the same parse ofconfig.rsthatproduces the reference document. Saving writes only the difference from the
defaults, and every save is validated by the engine before it is offered.
at all is the one the 1,213-vs-10,300 comparison describes.
Engine changes
Four, all small and all additive:
doctor --json,peak-census, the generatedconfigs/config-schema.json, and nothing else. Milestone 1 needed no engine changeat all.
Decisions worth reviewing
The engine is a child process, not a linked crate. It has no signal handling
anywhere, so stopping a run is a kill, and a Rust thread cannot be killed. Linked
in-process there would be no Stop button. Also: a stage panic ends one run rather
than the session,
--threadscan change between runs, and the application can showthe exact command it ran.
Two Python environments, not one.
deeplc==4.1.1needssqlalchemy>=2throughpsm-utils;ms2pip==4.0.0needs<2.uvreports the pair as unsatisfiable.ms2pip>=4.1would resolve, but it changes predicted fragment intensities andenv/docker-rescore.ymlpins 4.0.0 as "a separate, testable upgrade". So MS2PIP getsits own environment, installed on request.
macOS is absent on purpose. Gatekeeper blocks an unsigned bundle outright.
Verification
installation from nothing).
binaries/{mumdia.exe, uv.exe}beside it.ci/check_desktop_ui.pychecks the frontend against the backend in bothdirections, since the interface has no build step to catch a typo.
desktopCI job: fmt, clippy, unit tests, frontend check.Known gaps
.AppImageis built by the release job but has not been verified end toend. The resource lookup now asks Tauri for the bundle directory, which is what an
AppImage needs, but that path has not been exercised on a real bundle.
drives is covered by tests.