[yuejiaointel]

Idea

The dynamic index's search already marks every node it finds as valid or invalid: today "invalid" means deleted. Invalid nodes are still navigated through, but they do not count toward the search window, and cleanup() drops them before results are returned.

This PR widens "invalid" to "deleted or rejected by a user predicate". Filtered top-k search then becomes the normal greedy search with one extra check per node, instead of the post-filter loop over the batch iterator: one pass, no resume, no extra buffer capacity.

Core changes

1. ValidBuilder takes an optional predicate (plus the id translator). If a node is not deleted and a predicate is set, it is valid only if pred(external_id) is true. This is the only place filtering is decided. (include/svs/index/vamana/dynamic_index.h)
2. The predicate is passed down through the existing functions as an optional, defaulted argument: internal_search_builder(pred), greedy_search_closure(..., pred), search(results, queries, params, cancel, pred). With no predicate everything behaves as before. search() still makes the same per_thread_batch_search call; with a predicate the closure is wrapped in a small FilteredSearch struct so the steps below can see the predicate.
3. Filtered-only checks (include/svs/index/vamana/extensions.h):
   • check_and_supplement_search_buffer skips ids that fail the predicate (before computing their distance).
   • single_search passes the predicate to that top-up.
   • Copy-out pads rows with fewer than k passing entries with id size_t(-1) and the metric's worst score (+inf for L2, -inf for inner product / cosine).
4. Translation skips padded ids (translate_to_external).
5. Orchestrator: DynamicVamana::filtered_search(results, queries, params, filter), a new virtual appended last (ABI). Returns false for compressed data types.

Not changed: greedy_search.h, MutableBuffer, the static index, the concurrent index, the batch iterator, unfiltered search results.

Notes and limitations

• The predicate is called from all search threads and must be thread safe.
• If fewer than a search window's worth of entries pass, the existing top-up scans all ids and fills in id order (not nearest-first), as it already does for deletions.
• Static index: every filtered path there still uses the batch iterator. Its search buffer has no valid bit, so this approach does not apply to it yet.
• Concurrent dynamic index: has its own copy of the builder/search closure; the same small change applies but is not in this PR.
• LVQ / LeanVec: they use their own search and top-up path, which does not pass the predicate, so filtered_search returns false for them.
• The C API and C++ runtime filtered paths are not switched to this yet.

Benchmark

Single thread, 1M x 128 synthetic, L2, k=10, dynamic index, one index per process, mean of 2 processes, search window 10, this exact version (PR head c4d7396). 99 % pass rate:

case	recall	QPS
greedy search (no filter)	0.9892	19,763
current batch-iterator loop (100 extra slots)	0.9892	12,541
this change	0.9892	18,780

Lower pass rates at window 10 (recall / QPS), this change vs the batch-iterator loop:

• 50 %: 0.9981 / 13,224 vs 0.9970 / 9,385
• 10 %: 1.0000 / 6,203 vs 0.9613 / 3,980
• 1 %: 0.9878 / 732 vs 0.6341 / 374
• 0.5 %: 0.9851 / 321 vs 0.6713 / 150

Speed falls behind at <= 1 % pass and large windows (1 %, window 100: 43 QPS), where many failing nodes accumulate in the sorted buffer.

Tests

[filtered] test in tests/svs/index/vamana/dynamic_index.cpp: recall against brute force over passing ids, deleted ids never returned, fewer than k passing (padding), unfiltered search unchanged by the optional argument, inner-product padding uses -inf, orchestrator entry point. Run: [filtered], *Dynamic*, MutableVamana*, Concurrent*, Multi*, [dynamic_index], *Vamana*, [vamana] (Release, GCC 11.4), and the C API target svs_c_api builds. Not covered: the orchestrator returning false for a non-SimpleData type (no cheap compressed index in this test).

🤖 Generated with Claude Code