Skip to content

Add bingyu filtered ANN task submission - #3

Open
Cooki-e wants to merge 1 commit into
VectorSpaceLab:mainfrom
Cooki-e:add-bingyu-filtered-ann
Open

Cooki-e wants to merge 1 commit into
VectorSpaceLab:mainfrom
Cooki-e:add-bingyu-filtered-ann

Conversation

@Cooki-e

@Cooki-e Cooki-e commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Submission paths (new tasks): task-submissions/bingyu/1-x-1

Category / Implementation or Optimization / CPU or GPU: 1 / Implementation / CPU

Authors and coauthors: bingyu

Adds a Faiss metadata-filtered ANN retrieval task based on YFCC-10M.
The task includes 10 public queries and 10 hidden queries. Hidden queries are
scored individually, and their scores are averaged.
Index building has a 600-second deadline. Once the service reports ready,
each query has a 15 ms deadline.
The verifier checks result correctness, index persistence, and consistency
after index relocation and reloading, and performs an independent trajectory audit.

Validation evidence

Completed checks:

  • python -B scripts/check_submission.py task-submissions/bingyu/1-x-1
    Result: 0 errors; submission review-stage static checks passed.
  • python -B scripts/check_release.py
    Result: 0 errors, 0 warnings.
  • python -B scripts/download_assets.py --task-path task-submissions/bingyu/1-x-1 --verify-only
    Result: size and SHA-256 verification passed for all 9 data files.
  • git show --format= --check HEAD
    Result: no whitespace errors in the commit.
  • Grader regression tests: all 13 tests passed after removing the separate
    service loading timeout.
  • Real subprocess failure tests: timeouts, malformed JSON, and unexpected
    process exits were handled as expected. Injecting 3 failures among 10 queries
    produced a retrieval score of 0.7, with subsequent queries still evaluated.

Earlier full-corpus reference implementation validation:

  • All 10 hidden queries passed across both runs, including the run after index
    relocation and reloading: 20 requests in total against the full 10-million-vector corpus.
  • Index building took approximately 120.74 seconds. The slowest query took
    approximately 5.67 ms. The retrieval score was 1.0.
  • This run preceded the final change removing the separate service loading
    timeout. The 13 regression tests passed after that change, but the full-corpus
    reference implementation validation has not been rerun.
  • The runtime summary is available in the
    task README.

Development dataset: Cooki-e/search-swe-development

Contributor checklist

  • All new tasks use exactly one contributor first-name namespace; temporary ordinals are category-unique and never reused after promotion; no direct formal additions or guessed final IDs.
  • Actual authors in task.toml; commit email associated with my GitHub account.
  • Allow edits from maintainers; promotion stays in this same PR.
  • No secrets, downloaded inputs/models, job outputs, or author-only solutions.
  • Static package/submission checks and applicable runtime checks recorded.
  • Asset provenance, redistribution rights, licenses and pinned development SHA documented.

Maintainer merge checklist (new tasks)

  • Design, verifier isolation and runtime evidence reviewed; exceptions explicitly approved.
  • Updated to latest main; every final number assigned serially without collision.
  • Separate pure git mv and reviewed finalization commits per task, in this PR.
  • Per-task HF community PR or maintainer mirror merged into official dataset; manifest,
    SOURCES and license updated, all old assets preserved; official SHA pinned in assets.json.
  • Repository inventories/docs updated; full checks and merge-ready gate pass;
    every task is promoted and no submission task.toml remains.
  • Merge commit only (no squash/rebase); contributor history retained.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant