test: execute notebook Colab setup cells against Colab's environment

[psschwei]

Pull Request

Issue

Part of #89

Description

#1649 got the example notebooks running in CI, but the Colab setup cells stay tagged skip-execution there, so that run exercises the working tree rather than the path a Colab user actually takes: installing ollama, and uv pip install mellea resolving against Colab's preinstalled package set. #1649 called that out as out of scope. This covers it. (The release-checklist half of #89 is still open.)

• Flips the tag without editing a notebook. test/colab/run_notebooks.py points nbclient's skip_cells_with_tag at a sentinel tag no cell carries, so the skip-execution cells execute, and sets UV_OVERRIDE so the notebooks' own unmodified !uv pip install mellea installs the checked-out tree instead of the PyPI release. Selection comes from each notebook's metadata.mellea.markers, the same block test/test_example_collection.py enforces, so a new notebook is picked up automatically and slow ones stay nightly-only. The executed copy is written to /tmp whether or not a cell raised, so the traceback survives as an artifact.
• Two jobs in colab-notebooks.yml, split by cost. colab-constraints runs per PR: no image pull, it installs mellea under -c test/colab/colab-constraints.txt and executes the notebooks under Colab's pins on a normal runner, using the Python version recorded in that file's header. colab-image is nightly and on demand, because Colab's published runtime image is 23 GB: it moves Docker's data-root to /mnt, pulls the image, and runs the same script inside it.
• The pin file is generated, not authored. colab-image extracts Colab's 700 packages and its Python version from the pristine image, uploads them as an artifact, and diffs them against the committed copy, warning on drift. Refreshing it stays a manual, reviewed gh run download step (see test/colab/README.md). The file is committed here so the per-PR job has something to resolve against from the start.
• Advisory, for the same reason notebooks.yml is. Deliberately not part of ci.yml's code-checks aggregate, since publish-release.yml depends on that aggregate and a Colab-side change must not be able to block a release. The per-PR job is also stricter than real Colab: uv pip install -c makes a version conflict an error, whereas Colab would upgrade the preinstalled package instead. A red X there means "mellea forces upgrades in Colab, check what moved", not "Colab is broken".
• One notebook fix. mcp_example.ipynb's setup cell is only ever executed by this harness, and it now installs mcp<2: 2.x renamed FastMCP to MCPServer, and the notebook uses the 1.x server API.
• Shared ollama staging. test/colab/prepare_ollama_model.sh carries the same pins and rationale as notebooks.yml (OLLAMA_VERSION pinned per bug(ci): llama-server segfault cascade in quality job #1388, granite4.2:3b clamped to num_ctx=8192 so the published 131072 default does not spend 6 GB of a 16 GB runner on KV cache). It is idempotent, so the local entry point uv run poe colabtest is safe to run on a developer machine; the two intentional side effects are documented in test/colab/README.md.

Testing

• Tests added to the respective file if code was changed
• New code has 100% coverage if code was added
• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

Attribution

• AI coding assistants used

---

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

• Component
• Requirement
• Sampling Strategy
• Tool

NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.

[psschwei]

Colab test failure is due to same issue reported in #1640

[psschwei]

Self-review notes

Went back over this with fresh eyes. Two problems mean the per-PR job isn't actually exercising the Colab install path yet, so I don't want this merged as-is. Tracking the fixes here.

Blocking

• colab-constraints can't install at all. mellea requires pillow>=12.0.0 and test/colab/colab-constraints.txt pins pillow==11.3.0, so uv pip install <repo> nbclient ipykernel -c test/colab/colab-constraints.txt is unresolvable. That means "Stage the ollama model" and "Execute notebooks" are both skipped and run_notebooks.py never runs on a PR. I need to decide between relaxing our pillow floor, dropping pillow from the constraints file, or marking the job advisory with a comment explaining that pin conflicts are expected until Colab catches up.

• UV_OVERRIDE is inert in colab-constraints. The venv is created as .venv-colab and invoked as .venv-colab/bin/python, which doesn't set VIRTUAL_ENV, and I don't pass --system, so the notebooks' own !uv pip install cells exit with "No virtual environment found." uv only walks up to a directory literally named .venv, so the fix is renaming it, exporting VIRTUAL_ENV, or setting UV_SYSTEM_PYTHON=1. Until then the notebooks run against the pre-installed mellea, which is the one thing this job is supposed to check.

• The file:// override drops extras. write_uv_override emits mellea @ file:///repo, and with that in place uv pip install --dry-run 'mellea[docling]' resolves to the bare tree with no docling. document_mobject.ipynb and georgia_tech.ipynb would fail at RichDocument.from_document_file. mellea[docling] @ file:///repo keeps the extra, so I should emit one override line per extra form the notebooks actually use. While I'm in there, switch to repo_root.resolve().as_uri() so a checkout path with a space doesn't produce an unparseable requirement.

Correctness gaps

• select_notebooks skips the packages gate. I only read metadata.mellea.markers, but docs/examples/conftest.py also skips on _missing_packages(entry["packages"]). So table_mobject.ipynb (needs pandas, not a dep in any extra) and mcp_example.ipynb (needs mcp, which lives in the tools extra) get selected into environments that can't run them, and they fail as notebook errors rather than being skipped. Better to import the conftest helpers than keep a second copy of the metadata contract.

• No whole-notebook timeout. NotebookClient(timeout=...) is per-cell only. A kernel that never starts burns the job's 45 minutes, GitHub cancels the job, and the if: always() artifact upload doesn't run, so I lose the executed copies and the traceback. notebooks.yml and the nbtest task pair a per-cell timeout with --timeout=1800 per notebook for exactly this reason; I should do the same.

• My README claim about broken setup cells is wrong. I wrote that a failing setup cell "is still caught," but the setup cells are all IPython !cmd, which records _exit_code and never raises. A failing command emits a stderr stream, not an error output, so allow_errors=False doesn't trip and the cell reports ok. Either the runner inspects stderr / _exit_code on the executed cells, or I soften the wording.

• The OLLAMA_VERSION pin doesn't reach the daemon under test. prepare_ollama_model.sh pins 0.33.1 on the host, but the workflow then stops that daemon and the container reinstalls from the unpinned upstream install.sh. So nightly runs against whatever ollama is latest, which is the bug(ci): llama-server segfault cascade in quality job #1388 failure mode the pin was meant to prevent. Needs the pin threaded into the container's install cell, or at minimum a note that it's a known gap.

• colab-constraints leaves the pre-staged ollama daemon running. colab-image deliberately stops the host daemon so there's no doubt which one answered, but colab-constraints doesn't, so the notebooks' nohup ollama serve fails to bind and the pre-existing daemon serves everything. I should make the two jobs consistent.

Coverage

• Nothing passes --include-slow. Both jobs call run_notebooks.py without it, so the six slow notebooks never run anywhere, including the only two that exercise mellea[docling] and mellea[all]. The PR description and test/colab/README.md both describe them as nightly-only, which reads as "nightly runs them." The nightly job should pass the flag.

Cleanups

• Add unit tests for select_notebooks, notebook_markers, write_uv_override, and _summarize_cell_error. All pure, and test/test_example_collection.py is the precedent for guarding this contract.
• UV_CONSTRAINT re-hardcodes the constraints path instead of reusing the workflow-level CONSTRAINTS_FILE, so the two can drift.
• --list raises ValueError from relative_to when --notebooks-dir points outside the repo.
• Add the new advisory per-PR job to the CI tier table in test/README.md, since AGENTS.md points there as the tier map.