Add the cNF organoid dataset: drugs and experiments

[jjacobson95]

Pipeline Hardening/Debugging PR # 8

Add the cNF organoid dataset: drugs and experiments

Second half of the new cNF dataset, building on the samples and omics PR. Adds the drug table and the dose-response experiment results for the cNF organoid drug screen.

Drugs (03-drugs-cnf.py)

• Pulls the unique drug names from the cNF drug screen on Synapse, then produces cnf_drugs.tsv (drug identifiers and synonyms) and cnf_drug_descriptors.tsv (molecular descriptors) using the shared coderdata drug utilities.
• Because pubchem_retrieval.py has no command-line entry point, this script imports it and calls its function directly (running it as a subprocess would exit without doing anything).
• Keeps the drug table usable even when descriptor generation fails (the descriptor builder can call Ensembl/biomaRt, which is sometimes down): descriptor work is retried with exponential backoff under a generous timeout, and there are --skip-descriptors and --only-descriptors options so descriptors can be regenerated later without redoing the drug table.

Experiments (04-experiments-cnf.py)

• Discovers every drug-screen viability file under three Synapse parent folders and attaches a specimen ID to each.
• Specimen attribution uses a three-step policy: trust the file's specimenID annotation, fall back to parsing the filename (for example NF0017_T2_Viabilities.csv gives NF0017_T2), then run the result through the shared classify_specimen helper as a safety net. Files whose specimen can't be resolved are skipped with a warning.
• For each specimen/drug pair:
  • multi-concentration measurements are fit with the shared fit_curve.py utility, producing fit_auc, fit_ic50, fit_einf, fit_hs, and fit_r2;
  • single-concentration measurements at 1 μM are recorded with metric uM_viability and value = viability percentage / 100. (This metric name must exist in the schema's ResponseMetric enum, which is added in the later schema PR.)
• Handles one known case where a single viability file should be attributed to two specimens, via a SPECIMEN_DUAL_MAPPINGS table at the top of the script that can be extended as more cases appear.

Build entrypoints

• build_drugs.sh and build_exp.sh.

Dependencies

• Builds on the cNF samples/omics PR (shares cnf_utils.py and the sample table) and uses the shared drug-pipeline and curve-fitting utilities from PR 3.

Scope: 4 files, all new. Base: cnf-dataset-samples-and-omics.

[sgosline]

we are not including cNF in this build