Repository navigation
fix(embed): respect physical batch limits for non-causal embeddings - #2387
original4422 wants to merge 7 commits into
Conversation
|
AI-assisted CI-sync receipt: merged current main All owning Python, tests, native vendor Previous release-sync and native validation receipts (retained unchanged): AI-assisted release-sync receipt: merged current main The actual native build and 41 passed in 15.76s result below were reused, not rerun for this metadata-only synchronization. The current package imports as 0.3.37 with the existing native library; Ruff check/format of all 30 Python files and whitespace checks pass. No new hosted-CI or other-platform validation is claimed. Previous native validation receipt (2026-11-14, retained unchanged): AI-assisted maintenance update: merged current main Rebuilt the native library against actual vendor @abetlen, when convenient, could you review the non-causal embedding batch-limit change? |
Summary
Fixes #1762. A 513-token BGE-M3 input with
n_batch=513currently reaches a native assertion because the omittedn_ubatchstays at 512.Resolve defaults before native context creation, including
n_ctx=0and logical batch clamping. Keep logical truncation unchanged, preserve causal microbatch splitting, and cache the fixed long-context model for the native CI regressions. No vendor changes.Evidence
0c1e57098, real BGE-M3 F16, actual non-causal context: one 513-token sequence and two sequences totaling 256+257 tokens both terminate isolated processes with SIGABRT (encoder requires n_ubatch >= n_tokens). The initial regression set has 5 failures and 3 passing controls.python -m pytest --full-trace -v: 41 passed. This includes real long embeddings, aggregate batches, explicit limits/zero,n_ctx=0, clamping, causal generation/embedding controls, and the existing model tests. Long embeddings retain every token and have finite output values.python -m ruff check llama_cpp testsandpython -m ruff format --check llama_cpp tests: pass;git diff --check: pass.Local environment: macOS arm64, Python 3.12.14, original native build with Metal available; the new regression cases explicitly use CPU model layers (
n_gpu_layers=0). BGE-M3 F16 revision:9379ce497e8814b200f2dc0d18eb4045426dcb8c.Merge Danger
Door: two-way
Blast Radius: embeddings
Omitted embedding microbatches now follow the logical batch size. Explicit physical limits remain supported; a sequence that cannot fit raises a Python error instead of aborting the process. Ordinary generation retains its 512-token default microbatch.