[python] Vectorize raw vector scoring and refinement in bounded blocks

[TheR1sing3un]

Purpose

Raw vector fallback and refinement currently materialize Python vector lists and execute a Python loop for every vector dimension. Score regular FLOAT vectors in bounded NumPy blocks, covering Arrow list/large-list/fixed-size-list columns, candidate refinement, and primary-key raw/refined results. Candidate filtering still happens before scoring; unsupported and null-containing blocks retain scalar handling.

L2 and cosine use float64 left-to-right accumulation to preserve the scalar reduction order. Inner product applies Python sum to precomputed products, preserving its runtime-specific summation behavior. Existing Top-K tie-breaking rules remain unchanged. Blocks target at most 1024 rows and 8 MiB per float64 matrix (with at least one vector); the Arrow read itself is still materialized.

Tests

• python -m pytest pypaimon/tests/vector_scoring_test.py pypaimon/tests/vector_search_filter_test.py pypaimon/tests/primary_key_global_index_golden_test.py pypaimon/tests/primary_key_index_definitions_test.py -q: 101 passed, 1 skipped because paimon_ftindex is unavailable.
• Exact score checks cover random vectors, cancellation, close scores, zero norms, Arrow slices/chunks/list layouts, nulls, dimension failures, candidate filtering and ties. Java-generated primary-key data files exercise both raw scoring and native-index refinement against the scalar path.
• Flake8, license-header checks and git diff --check passed.

Benchmark

macOS arm64, Python 3.9. Four queries over 8,192 vectors x 384 dimensions, FLOAT fixed-size lists in Zstd Parquet, Top-K=10. Each variant runs in a fresh process; values are medians of three runs. Timing includes Parquet reads, Arrow conversion, scoring and Top-K. Arrow read machinery is warmed before timing. Manifest planning and ANN search are excluded.

The blocked-scalar ablation uses bounded conversion but disables vectorized arithmetic, separating its memory benefit from the arithmetic speedup.

Metric	Variant	Four-query time (s)	Peak RSS (MiB)
L2	Original scalar	2.198696	450.344
L2	Blocked scalar	2.109683	342.562
L2	Vectorized	0.062453	335.391
Cosine	Original scalar	2.635863	441.766
Cosine	Blocked scalar	2.573783	339.500
Cosine	Vectorized	0.089766	322.234
Inner product	Original scalar	2.134443	451.984
Inner product	Blocked scalar	2.031288	341.406
Inner product	Vectorized	0.187022	337.109

All 27 runs matched Top-K row IDs and the binary representation of their scores within each metric. These workloads show about 35x / 29x / 11x faster fallback read-and-score execution and about 25-27% lower process peak RSS. These are not whole-query speedups including ANN/planning. A 64-row x 8-dimension smoke comparison also preserved result bits across all variants.

Reproduce:

python -m pypaimon.benchmark.vector_scoring_bench --output /tmp/scoring.json
python -m pypaimon.benchmark.vector_scoring_bench --rows 64 --dimension 8 --repeats 1 --output /tmp/scoring-small.json

[JingsongLi]

Reviewed e12fb7b. Requirement fit: supported; no actionable implementation finding.

There is a concrete benefit for full-mode raw fallback and original-vector refinement: the change removes per-dimension Python arithmetic while retaining candidate filtering and existing Top-K tie-breaking. The Arrow read and refinement candidate map still materialize their input, so the bound correctly applies to scoring buffers, not the whole query.

I ran the focused scoring/filter/primary-key tests on Python 3.13 (99 passed, 3 native-dependent skips), then installed paimon-vindex 0.4.0 and reran the golden-fixture suite (4 passed, 1 unrelated full-text skip). I also independently compared scalar/vectorized scores, Arrow layouts and Top-K results, including cancellation, signed zero and overflow. A 1,024-row × 128-dimension smoke benchmark preserved all result checksums; read-and-score time improved for all three metrics. This is a local smoke check, not a remote-storage throughput claim.