[yuejiaointel]

What

filtered_topk_search (C API, static and dynamic Vamana) now picks its search strategy from the filter pass rate it measures on a sample of ids.

1. Always sample (about 200 ids, MIN_SAMPLE_SIZE), also when filter_rate() is 0 or not provided. The measured pass rate is p = hits / sampled. The early exit is unchanged: it only applies when filter_rate() > 0 and the sampled hit rate is below it.
2. p >= 0.5: one plain greedy search per query for N = max(window, ceil(1.2 * K / p)) neighbors (search window and capacity both N). The results are filtered in distance order and the first K are returned. A query with fewer than K hits is redone with the batch iterator loop below.
3. p < 0.5: the batch iterator loop as before, but the iterator gets clamp(ceil(0.2 * K / p), 60, 200) extra search buffer slots instead of the default 100 (200 if no sampled id passed). That is 60 slots at 10 % and 200 at 1 % and below (K = 10).

The floor of 60 was first 25. A recall test at 1/10 selectivity (10000 vectors, window 100, dynamic index, from #437) lost recall with 25, 35 and 50 extra slots (0.940, 0.935, 0.945 against a 0.95 threshold) and passed with 55 and above, so the floor was raised to 60 (a small test index needed it).

The initial batch size is unchanged (still max(K, window) when no filter_rate is given).

The greedy searches of a query block run through the index's own search (one call per block of queries, so it uses the index thread pool); the filtering and the fallback run in the C API thread pool, as the batch loop did before.

Why

With a high pass rate almost every candidate passes, so the batch iterator (several next() calls, a larger search buffer, resumable state) costs more than one ordinary search that is a little wider. With a low pass rate the iterator has to resume several times, and the extra capacity decides how often.

Related, independent

Recall at very low selectivity (about 1 % and below) is also limited by the visited set not being reset when the iterator resumes. That is fixed separately in #437 and this PR does not include or depend on it. The tests below were written so they do not depend on it.

Tests

New C API tests (static index and dynamic index, 5000 random vectors, dimension 32, K = 10, native thread pool with 4 threads):

• pass rates 1 and 1/2 with no filter rate: every query returns K results, all pass the filter, recall within 0.05 of the unfiltered search;
• pass rate 1/10 (window 400): every query returns K results, recall >= 0.8; pass rate 1/100: returned ids pass the filter;
• filter_rate() = 0.5 with an actual pass rate of 1/20 returns an empty result; filter_rate() = 0.01 returns K results.

The whole C API test binary (25 test cases) passes locally, built in Release with GCC 11.

Benchmark results to follow.

🤖 Generated with Claude Code