Sum-of-Checks

Code and artifacts for Sum-of-Checks: Structured Reasoning for Surgical Safety with Large Vision-Language Models.

Pointers

Workflow	Files
Checks and weights	CVS rubric v3
Human labels	Original, filtered, corrected Endoscapes labels and SAGES labels
Paper exemplars	Four Endoscapes v3 examples
Inference	Paper launcher, frame runner and prompts
Tables	Evaluation notebook, exported results
Figure 2	Example viewer, video 184 / frame 36750
Provenance	Frozen prediction manifest, reproduction notes

Setup

Python 3.10 or newer, with venv support:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Alternatively, create a conda environment with conda create -n sumofchecks python=3.10, activate it, and install the requirements. requirements-lock.txt records the exact tested Python 3.10 environment.

The local .venv is already installed. Run commands from the repository root. Inference reads OPENAI_API_KEY or ANTHROPIC_API_KEY from the environment; evaluation and dry runs need no keys.

Data

The paper evaluates 20 Endoscapes test videos / 791 frames. test_dev is a subset of the official test split. Four validation-set exemplars have label combinations 000, 111, 110, and 001.

Raw data remains external. On the lab server, these local links are already configured:

ln -s /mnt/md0/weiqiuy/datasets/endoscapes data/endoscapes
ln -s /mnt/md0/weiqiuy/datasets/CVS_Challenge_SAGES_v1 data/CVS_Challenge_SAGES_v1
ln -s /mnt/md0/weiqiuy/lvlm_cvs_reasoning/data/frames data/frames

Elsewhere, place or link Endoscapes under data/endoscapes/, including val/, test/, their images, and annotation_ds_coco.json files. SAGES annotation requires its raw dataset and sampled frames. The copied predictions and four exemplar images are sufficient for offline table evaluation without raw datasets.

Run Experiments

Inspect all seven prompts and their images without API calls:

python scripts/run_paper.py --models gpt-4.1-mini --runs 1 --dryrun

Run all seven methods, three models, and three repetitions:

python scripts/run_paper.py

Models are GPT-4.1-mini, Claude Haiku 4.5, and Claude Opus 4.5. The launcher uses temperature 0.1, rubric v3, and four fsv3 exemplars. Jobs run sequentially and resume by run number. New predictions go to ignored outputs/generated/; the frozen artifacts are protected.

Run only Sum-of-Checks or the no-FS ablation:

python scripts/run_paper.py --models gpt-4.1-mini --methods sum_of_checks
python scripts/run_paper.py --models gpt-4.1-mini --methods no_fs

Sum-of-Checks uses weighted_score: yes=1, no/uncertain=0. pred is the separate LLM-aggregation output.

Reproduce Results

python scripts/evaluate_paper.py
python -m pytest -q
jupyter lab

Evaluation verifies the 70 frozen prediction checksums and exports metrics, frame counts, and LaTeX tables to results/paper/. Headline average mAP reproduces as 36.9%, 34.0%, and 37.0% for GPT, Haiku, and Opus respectively.

There are documented run-count and no-FS scoring discrepancies, plus an unresolved Opus SubQ+FS row. See reproduction notes before interpreting the exported tables as an exact reproduction of every PDF entry.

Annotation

• Endoscapes labeling and SAGES labeling use rubric v3; new labels go to ignored annotations/new/.
• Endoscapes exemplar selection shows the corrected labels and the four paper examples. Export is disabled by default.
• Historical Endoscapes filenames and label metadata retain rubrics_v1; the corrected file is the final exemplar-selection input. SAGES labels are retained as additional annotation assets, not as results reported in this Endoscapes paper.

To prepare SAGES frames on another machine, run extract_frames.py for both train_dev.jsonl and train_rest.jsonl under data/manifests/cvs_challenge_sages_v1/, using --root data/CVS_Challenge_SAGES_v1 --manifest <manifest> --out data/frames.

Hugging Face

BrachioLab/sum-of-checks packages the 53 corrected Endoscapes and 60 SAGES rubric-labeled images, their original CVS labels, all SAGES rater votes, and rubric v3. It belongs to the Laparoscopic Cholecystectomy collection.

Each source has disjoint few_shot and dev image splits: Endoscapes 4/49, SAGES 6/54. SAGES retains all six v5 examples in manifest order, not a four-example experimental subsample. See data/README.md and the executed example notebook. Rubric labels are development annotations from a surgeon-trained ML PhD student, not expert ground truth.

Original video filenames/IDs and frame IDs for all ten selected exemplars are listed in the few-shot reference table.

Build and validate the portable image dataset:

pip install -r requirements-hf.txt
python scripts/export_hf_sum_of_checks.py --validate

Publish with python scripts/export_hf_sum_of_checks.py --upload. Authentication uses HF_TOKEN or the Hugging Face login cache; --env-file <path> can supply an existing dotenv credential file. The script validates every image and label before upload and again from the uploaded revision, then adds the dataset to the collection. Export files under hf_repos/ are ignored by Git.

from datasets import load_dataset
endo = load_dataset("BrachioLab/sum-of-checks", "endoscapes", split="dev")
sages = load_dataset("BrachioLab/sum-of-checks", "sages", split="dev")
sages_few_shot = load_dataset("BrachioLab/sum-of-checks", "sages", split="few_shot")