Update Benchmarks #6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Update Benchmarks | |
| on: | |
| schedule: | |
| - cron: '30 2 * * 1' | |
| workflow_dispatch: | |
| concurrency: | |
| group: update-benchmarks | |
| cancel-in-progress: false | |
| permissions: | |
| contents: write | |
| pull-requests: write | |
| jobs: | |
| update-benchmarks: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - name: Checkout repository | |
| uses: actions/checkout@v7 | |
| - name: Setup pnpm | |
| uses: ./.github/actions/setup-pnpm | |
| - name: Update and validate exact benchmark mappings | |
| run: pnpm manifest-data:update -- --only=benchmarks | |
| - name: Check formatting | |
| run: | | |
| pnpm format:check | |
| pnpm biome:check | |
| git diff --check | |
| - name: Check for changes | |
| id: git-check | |
| run: | | |
| git diff --exit-code manifests data/data-health.json docs/DATA-HEALTH.md src/lib/generated || echo "changes=true" >> "$GITHUB_OUTPUT" | |
| - name: Create pull request | |
| if: steps.git-check.outputs.changes == 'true' | |
| uses: peter-evans/create-pull-request@v8 | |
| with: | |
| token: ${{ secrets.GITHUB_TOKEN }} | |
| commit-message: 'chore(data): refresh benchmark scores' | |
| title: 'chore(data): refresh benchmark scores' | |
| body: | | |
| ## Summary | |
| Automated refresh from exact entries in the official SWE-bench leaderboard data. | |
| Each model manifest declares the upstream result ID and exact displayed label. The | |
| refresh fails rather than applying fuzzy model-name matching. | |
| ## Validation | |
| - Manifest schemas | |
| - Generated manifest modules | |
| - Data health | |
| branch: auto-update-benchmarks | |
| delete-branch: true | |
| labels: | | |
| automated |