Pin Falcon-OCR to v1 and add a falcon-ocr-1.5 config - #60
Draft
davanstrien wants to merge 2 commits into
Draft
davanstrien wants to merge 2 commits into
davanstrien wants to merge 2 commits into
Conversation
tiiuae/Falcon-OCR now holds two model versions in one repo: TII overwrote main with the v1.5 weights on 2026-09-11 without a tag or branch. The `falcon-ocr` slug loaded main, so it silently became v1.5 and stopped matching the v1 rows already judged. - `falcon-ocr` pins the last v1 commit via `--revision`. - New `falcon-ocr-1.5` slug pins the v1.5 release head. Same model_id; the dataset loader already relabels colliding configs by slug so both stay distinct leaderboard rows. - Tests: registry count 17; both slugs must pin distinct full shas. Depends on the `--revision` flag landing in uv-scripts/ocr falcon-ocr.py (davanstrien/uv-scripts-for-ai, branch ocr/falcon-ocr-revision) and the Hub mirror syncing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TWTqZnz7cp1TzvicRo6FkN
uv-scripts split the Falcon recipe by version identity (falcon-ocr.py pins v1, falcon-ocr-1.5.py pins the v1.5 release head). The slug now loads the matching script. The explicit `--revision` pins stay as belt-and-braces so the bench never depends on a script default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TWTqZnz7cp1TzvicRo6FkN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
tiiuae/Falcon-OCRnow holds two model versions in one repo. TII overwrotemainwith the v1.5 weights on 2026-09-11 (no tag, no branch). Thefalcon-ocrslug loadedmain, so it silently became v1.5 and no longer matches the v1 rows already judged.What
falcon-ocrnow pins the last v1 commit (42ec56b72a23984ac059e7c8a6d397a8529423fe) via--revision.falcon-ocr-1.5slug loads the splitfalcon-ocr-1.5.pyrecipe and pins the v1.5 release head (fe757d59ecd79d4d68760162306a70a015761ad9). Samemodel_id, same 0.3B, same l4x1 launch.load_unified_datasetalready relabels collidingmodel_ids by config name, so the two land as distinct leaderboard rows (Falcon-OCR (falcon-ocr)/Falcon-OCR (falcon-ocr-1.5)).test_run,test_cli,test_budget_checkpoint: 180 passed.Depends on
The
--revisionflag lives in the script this registry loads from the Hub: davanstrien/uv-scripts-for-ai#110. Merge that first and let theuv-scripts/ocrmirror sync; until then the pinneddefault_argsare an unknown argument for the Hub copy offalcon-ocr.py. Draft until that lands.Smoke evidence for the flag itself (both pins reproduce the outputs previously collected for each version on the same BHL pages) is in #110.
🤖 Generated with Claude Code
https://claude.ai/code/session_01TWTqZnz7cp1TzvicRo6FkN