Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .claude/development-notes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ One file per subject, not per source file — the reasoning crosses file boundar
| `gates-that-stopped-checking.md` | the failure this project is prone to: a change moves what a checker points at, and the checker keeps exiting 0 over nothing |
| `host-glibc-floor.md` | v3.1.1's analysis environment installed only on the machine that froze it: virtual packages, the four guards, and why a solve cannot answer it |
| `someone-elses-machine.md` | four defects a green suite could not see, because the suite runs where the assumption holds; and the second machine that found three of them in an hour |
| `macos-support.md` | the sizing, after the manual was found claiming a platform the pinned environments cannot solve on: three blockers, and why exporting is what makes it single-platform |
| `brainstorming.md` | ideas for later releases: what each would buy, what it would break, and where it sits |
| `module-queue.md` | the plan from v3.1.1: thirteen modules at one a week, published without a release, and the two shape questions the roster raised |

Expand Down
48 changes: 48 additions & 0 deletions .claude/development-notes/macos-support.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# What macOS support would actually take

**Written 2026-09-23, against the tree at `6e12735`, during the 3.1.3 cycle.** Z asked whether the new tab completion would work on a Mac. It would not, and neither does anything else, although the manual had claimed macOS since before v3.1.0. Z's call the same day: **drop it for now, correct the claim, and plan it properly** - *"It is real work. I'm still fixing the backend. I need to get this to become more stable before we start working on MacOS."*

This note is the sizing, so the next attempt starts from measurements rather than from a survey.

## The claim that was wrong

Five places said macOS was supported, including the Requirements table and `README.md`. One said more than that: the analysis-environment section claimed the compiler is *"GCC on Linux, clang on macOS"*, describing a per-platform behavior the shipped file cannot have. All six now say Linux, and `#### Why not Windows` became `#### Why Linux only` (keeping the `#why-not-windows` anchor, which one row links to).

**This is the glibc bug one level up.** That was a single pin excluding older Linux; this was a whole platform the documentation promised. Same cause, recorded in [[someone-elses-machine]]: it has only ever been installed on one machine.

## The three blockers, in the order they bite

### 1. The environments cannot solve at all

`install/environment.yml` pins `ld_impl_linux-64`. `install/environment-analysis.yml` carries **12** `linux-64` pins - `sysroot_linux-64`, `gcc_impl_linux-64`, `gxx_impl_linux-64`, `kernel-headers_linux-64` and the rest of the toolchain. Those package names exist for `linux-64` by construction and for no other platform, so `conda env create` fails before anything else is reached.

**This arrived with the pinned exports.** A hand-written spec names what you ask for and solves per platform; an export names exactly what one machine got. The reproducibility that makes the export right is the same property that makes it single-platform.

So macOS needs its own exported files - and **two of them**, because conda treats `osx-64` and `osx-arm64` as different platforms. Each has to be exported on that hardware and verified there. That is the part that cannot be done from here.

### 2. Three GNU-only idioms in the shipped wrapper

| | |
|---|---|
| `PoolSeqFlow:86` | `readlink -f` - `-f` is a GNU extension; it fails at startup, before dispatch |
| `PoolSeqFlow:178` | `sort -V` - BSD `sort` has no version sort |
| `PoolSeqFlow:359` | `sed -i "..."` - BSD requires a backup suffix, `sed -i ''`, and errors without one |

Only the completion was fixed, because it was being written that day: `lib/poolseqflow-completion.bash` follows symlinks a hop at a time with plain `readlink` and a loop guard. That is better code on Linux too, so it stayed.

### 3. The suite itself has to run there

Proving macOS support means running the suite on the Mac, and the suite is not portable either. Found without looking hard: `stat -c` in `run_tests.sh` and `02_launcher`, `find -printf` in `04_pipeline`, GNU `sed -i` in `04_pipeline`, `05_guards` and `06_dryrun`. There will be more - a full audit was started and stopped when the decision was made.

**And the filesystem differs.** APFS is case-insensitive by default, so any two fixtures differing only in case are the same file there.

## What else to think about before starting

- **`check-host-floor.sh` reasons about `__glibc`**, which does not exist on macOS - conda uses `__osx`. It needs to know which platform it is checking rather than assuming.
- **`export-environment.sh` and `prep-version.sh`** write and validate one pair of files. They would need to know about three platforms, and a release would not be exportable from one machine.
- **Every module compiles its hot path on the user's machine.** So the analysis environment needs a working clang toolchain pinned for each macOS platform, not just R.
- **`bash 3.2`.** macOS ships the 2007 GPLv2 bash as `/bin/bash`. The wrapper's shebang is `#!/usr/bin/env bash`, so it would take whatever is first on PATH - conda's newer bash if an environment is active, the system one otherwise. Worth deciding deliberately rather than discovering.

## The honest summary

Not a flag and not an afternoon. It is: three sets of pinned environment files instead of one, a release process that can export them, three wrapper fixes, an unknown number of suite fixes, and a machine to verify all of it on. Nothing here is hard; it is just genuinely a platform port, and the pinning that makes this project reproducible is exactly what makes it one.
18 changes: 17 additions & 1 deletion .claude/development-notes/someone-elses-machine.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,23 @@ fi

`rehash` is zsh's name for `hash -r`. The wrapper is `#!/usr/bin/env bash`, so wherever `ZSH_VERSION` reaches a bash script, conda picks a command bash does not have. Measured on the server: `bash -c 'echo $ZSH_VERSION'` printed `5.9`, so something there exports it - zsh does not, and the maintainer's machine does not, which is exactly why it had never appeared.

**Fixed by defining it rather than by silencing conda.** `lib/wrapper_lib.sh` carries `rehash() { hash -r; }`; the backslash in `\rehash` suppresses aliases, not functions, so that is what runs. Suppressing conda's stderr was the first instinct and was wrong: it would hide conda's real failures, and refreshing the command table is what conda was asking for.
**NOT FIXED, on purpose. Z's ruling, 2026-09-23.** Three fixes were written and all three were rejected, and the reason they were all wrong is the same: **the wrapper is not what is broken.**

`eval "$(conda shell.bash hook)"` has been line 17 of `PoolSeqFlow` since **v1.0.0**, written by Z alone, and it is unchanged through every release to 3.1.2 - only its line number moved as the header grew. It works, and it is the only way a script gets `conda activate`: a shell function does not cross a process boundary, so a user who has run `conda init` gives their *interactive* shell the function and gives a script nothing. Measured - parent reports `conda is a function`, the child one process later reports `conda is a file` and `conda activate` fails with `CondaError: Run 'conda init' before 'conda activate'`.

**And asking for the bash hook does not get a bash-only hook.** `conda shell.bash hook` emits `__conda_hashr` with the `ZSH_VERSION` branch still in it, at line 20 of its own output. That is conda's, not ours.

So on a machine where something exports `ZSH_VERSION` into a bash process, conda believes a false claim the environment made about itself and calls a command bash does not have. **The correct response is to do nothing**, because correcting another machine's environment is not this tool's business. The noise is cosmetic: the consequence of the zsh branch is that `hash -r` does not run, and the wrapper invokes nothing before it activates, so there is no stale entry to refresh.

The three rejected fixes, and what each got wrong:

- **Suppressing conda's stderr.** Would hide conda's real failures, and the command-table refresh is what conda was actually asking for.
- **`rehash() { hash -r; }` in `lib/wrapper_lib.sh`.** Made a bash script carry zsh's vocabulary to satisfy a claim that was not true. It was also in the wrong place - `wrapper_lib.sh` is sourced at line 96 and the hook is evaluated at line 44, so the function did not exist for the eval that first raised the error.
- **`unset ZSH_VERSION POSH_VERSION` before the hook.** Z: *"You are still making assumptions about the shell. We don't deal with that."* Process-local or not, it is the tool reaching into variables it does not own to compensate for someone else's misconfiguration.

**What this costs**: the `rehash: command not found` line comes back on that server. Z accepted that knowingly.

**The general form is still worth keeping**, and it is why this one sits oddly beside the other three in this note. Each of them is a string standing in for a real question, and here the real question is *which shell is this*. The difference is that the other three were our strings, in our code, answering questions about our own behavior - and this one is conda asking a question we have no standing to answer.

**It was visible in one arm and not another for a reason worth keeping.** `uninstall` called `conda deactivate` bare while `uninstall_all` had `2>/dev/null || true`, so the same noise was hidden in one place and shown in the other. That inconsistency is what made it findable - the user could say "during uninstall but not uninstall_all", which pointed straight at the difference. The `|| true` also closed a real hazard: under `set -e` a non-zero `conda deactivate` would have abandoned the uninstall with the environment half removed.

Expand Down
10 changes: 5 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,22 +106,22 @@ bash test/run_tests.sh --suite 07_analysis --case citation

| you changed | run |
|---|---|
| `bin/` | `05_helpers` — except the three `check_*.sh`, which are `02_launcher` |
| `bin/` | `03_helpers` — except the three `check_*.sh`, which are `02_launcher` |
| `PoolSeqFlow`, install/uninstall, the check scripts | `02_launcher` |
| `bin/config_migrate.sh`, the templates | `01_migrate` |
| step 0, parameter resolution, the change guards | `04_guards` |
| wiring, channels, promotion, a step's script | `03_pipeline` |
| step 0, parameter resolution, the change guards | `05_guards` |
| wiring, channels, promotion, a step's script | `04_pipeline` |
| `dryrun.nf`, `dryrun`/`dryclean` | `06_dryrun` |
| version strings, packaging, syntax | `00_static` |
| a module library under `modules/lib/` | `analysis_rlib` — no JVM, 3 seconds |
| `analysis/lib/nf/`, the frame | the analysis seam you touched: `analysis_frame`, `analysis_plan`, `analysis_verify`, `analysis_design`, `analysis_time`, `analysis_series`, `analysis_modules`, `analysis_results` |
| a module | `--suite <module name>`; its cases travel with it under `modules/<name>/test/` |

`--fast` runs everything that does not start a JVM; what it skips is `03_pipeline`, `04_guards`, and the pipeline halves of `06_dryrun` and the analysis suites.
`--fast` runs everything that does not start a JVM; what it skips is `04_pipeline`, `05_guards`, and the pipeline halves of `06_dryrun` and the analysis suites.

**`bash test/run_tests.sh --changed` picks the suites for you**, from what each suite declares it runs expanded through the include graph. `dev/scripts/select-tests.py <file>` shows the reasoning without running anything. It errs wide — a change to `test/lib/` or to the selector selects everything — so a narrow answer is trustworthy and a wide one is only expensive.

**Every suite declares what it may cost** — `static`, `jvm` or `pipeline` — in a `# cost:` line in its own header. `--cost static` is the set that completes with nothing installed: `00_static`, `01_migrate`, `02_launcher`, `05_helpers`, `08_analysis_rlib`. `--fast` is a different axis and still a case-level switch, so the two compose.
**Every suite declares what it may cost** — `static`, `jvm` or `pipeline` — in a `# cost:` line in its own header. `--cost static` is the set that completes with nothing installed: `00_static`, `01_migrate`, `02_launcher`, `03_helpers`, `07_analysis_rlib`. `--fast` is a different axis and still a case-level switch, so the two compose.

**`--suite` and `--case` accumulate and match by name, not by number.** `--suite analysis_time --suite analysis_series` runs both, and `--suite analysis` runs all nine. Every run prints the scope it selected, so a narrowed run cannot be mistaken for a full one; renumbering a suite therefore costs nothing, because nothing addresses one by its number.

Expand Down
63 changes: 51 additions & 12 deletions PoolSeqFlow
Original file line number Diff line number Diff line change
Expand Up @@ -401,6 +401,53 @@ deploy_payload() {
;;
esac
echo ""
install_completion "$dest"
}

# Where a user's own bash completions live, per the XDG base directory specification. The same
# path bash-completion searches for a file named after the command.
completion_dir() {
printf '%s/bash-completion/completions' "${XDG_DATA_HOME:-$HOME/.local/share}"
}

# Put the completion where bash will find it by name, and say what zsh needs.
#
# Copied rather than symlinked, so uninstalling one version does not leave the file pointing
# into a directory that is gone. A failure here does not fail the install: completion is a
# convenience and the tool works without it.
install_completion() {
local src="$1/lib/poolseqflow-completion.bash" dir
[ -f "$src" ] || return 0
dir=$(completion_dir)
mkdir -p "$dir" 2>/dev/null || { echo "Could not create $dir; tab completion not installed."; return 0; }
if ! cp "$src" "$dir/PoolSeqFlow" 2>/dev/null; then
echo "Could not write $dir/PoolSeqFlow; tab completion not installed."
return 0
fi
echo "Tab completion installed:"
echo " $dir/PoolSeqFlow"
echo ""
echo " bash picks it up in your next shell."
echo " zsh does not read that directory. Add this to ~/.zshrc:"
echo ""
echo " autoload -U +X bashcompinit && bashcompinit"
echo " . $dir/PoolSeqFlow"
echo ""
}

# Remove the completion, but only when no other installed version still provides one.
remove_completion() {
local prefix file
prefix=$(install_prefix)
file="$(completion_dir)/PoolSeqFlow"
[ -f "$file" ] || return 0
# Another version left installed keeps it: the file is the same for every version, and the
# command it completes still exists.
if ls -d "$prefix"/opt/PoolSeqFlow-*/ > /dev/null 2>&1; then
return 0
fi
rm -f "$file" 2>/dev/null || true
echo "Removed $file"
}

# Every PoolSeqFlow environment except this version's own and the legacy unversioned one.
Expand Down Expand Up @@ -1300,14 +1347,7 @@ case $COMMAND in
bash "$(install_prefix)/opt/PoolSeqFlow-${VERSION}/bin/check_install.sh"
echo "Installation complete."
echo ""
echo "The analysis layer was copied with everything else, so 'PoolSeqFlow analysis'"
echo "answers now. THAT DOES NOT MEAN THE ANALYSIS LAYER IS INSTALLED. The scripts"
echo "weigh nothing and travel with the release so that they can never be a version out"
echo "of step with the pipeline; the weight is the environment, which carries R and"
echo "which nothing here creates. Until you create it, every module refuses and says so."
echo ""
echo "The pipeline is complete without it. Add the analysis layer whenever you want it,"
echo "or never:"
echo "To optionally install the analysis layer, run:"
echo " $(basename "$0") analysis install"
;;

Expand All @@ -1324,12 +1364,9 @@ case $COMMAND in
check)
run_check "$CHECK_TARGET"
;;
run|resume)
run)
# Resuming is what `run` already does: every step skips itself when its outputs are
# already in storage. Nextflow's own -resume is not used.
if [ "$COMMAND" = "resume" ]; then
echo "Note: 'resume' is deprecated - 'run' already resumes automatically."
fi
require_install
require_project_config
require_migrated_config
Expand Down Expand Up @@ -2028,6 +2065,7 @@ EOF
fi
if [ "$HAVE_PAYLOAD" -eq 1 ]; then
remove_payload "$TARGET_VERSION"
remove_completion
elif [ -n "$TARGET_VERSION" ]; then
echo "No pipeline installed at $PREFIX/opt/PoolSeqFlow-${TARGET_VERSION}."
fi
Expand Down Expand Up @@ -2084,6 +2122,7 @@ EOF
for v in $PAYLOADS; do
remove_payload "$v" || true
done
remove_completion
echo ""
echo "All PoolSeqFlow environments and installations removed."
;;
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

### 📖 [Read the documentation →](https://ozankiratli.github.io/PoolSeqFlow/)

> **Platform note:** PoolSeqFlow is developed and tested on **Linux and macOS**. Windows is not supported — the resume logic relies on symbolic links and Unix-style paths that are not compatible with native Windows filesystems.
> **Platform note:** PoolSeqFlow is developed and tested on **Linux**. macOS and Windows are not supported: the shipped conda environments are pinned to `linux-64` builds, and the resume logic relies on symbolic links and Unix-style paths that are not compatible with native Windows filesystems.

---

Expand Down Expand Up @@ -55,7 +55,7 @@ Raw FASTQ reads

## Quick start

Requires Linux or macOS and [conda](https://docs.conda.io/en/miniconda.html). Every bioinformatics tool is installed for you into an isolated environment, pinned to an exact build.
Requires Linux and [conda](https://docs.conda.io/en/miniconda.html). Every bioinformatics tool is installed for you into an isolated environment, pinned to an exact build.

```bash
# 1. Download the latest release
Expand Down
Loading
Loading