Skip to content

fix(embed): respect physical batch limits for non-causal embeddings - #2387

Open
original4422 wants to merge 7 commits into
abetlen:mainfrom
original4422:fix/noncausal-embedding-microbatches
Open

original4422 wants to merge 7 commits into
abetlen:mainfrom
original4422:fix/noncausal-embedding-microbatches

Conversation

@original4422

Copy link
Copy Markdown

Summary

Fixes #1762. A 513-token BGE-M3 input with n_batch=513 currently reaches a native assertion because the omitted n_ubatch stays at 512.

omitted n_ubatch: embedding -> n_batch; generation -> 512
explicit n_ubatch: retain the requested limit
non-causal/encoder embedding: pack within the native physical batch
single sequence exceeding an explicit physical limit: raise ValueError

Resolve defaults before native context creation, including n_ctx=0 and logical batch clamping. Keep logical truncation unchanged, preserve causal microbatch splitting, and cache the fixed long-context model for the native CI regressions. No vendor changes.

Evidence

  • Before: original 0.3.36 / vendor 0c1e57098, real BGE-M3 F16, actual non-causal context: one 513-token sequence and two sequences totaling 256+257 tokens both terminate isolated processes with SIGABRT (encoder requires n_ubatch >= n_tokens). The initial regression set has 5 failures and 3 passing controls.
  • After: complete python -m pytest --full-trace -v: 41 passed. This includes real long embeddings, aggregate batches, explicit limits/zero, n_ctx=0, clamping, causal generation/embedding controls, and the existing model tests. Long embeddings retain every token and have finite output values.
  • python -m ruff check llama_cpp tests and python -m ruff format --check llama_cpp tests: pass; git diff --check: pass.

Local environment: macOS arm64, Python 3.12.14, original native build with Metal available; the new regression cases explicitly use CPU model layers (n_gpu_layers=0). BGE-M3 F16 revision: 9379ce497e8814b200f2dc0d18eb4045426dcb8c.

Merge Danger

Door: two-way

Blast Radius: embeddings

Omitted embedding microbatches now follow the logical batch size. Explicit physical limits remain supported; a sequence that cannot fit raises a Python error instead of aborting the process. Ordinary generation retains its 512-token default microbatch.

@original4422

original4422 commented Nov 8, 2026 •

Copy link
Copy Markdown
Author

AI-assisted CI-sync receipt: merged current main 0542f207802fcfd174167dac2b228abe2f12093d in c19d94e9ee55e3a7fc9cd1a221d5535a72aa59f7, retaining both Unreleased changelog entries and the upstream release-runner update. The inherited release workflows remove unsupported macOS 14 runners; their YAML parses and the resulting matrices retain macOS 15 (plus Ubuntu/Windows for the general wheel build). Whitespace checks pass. No release job or hosted CI run is claimed.

All owning Python, tests, native vendor a956a852, build files and dependencies remain unchanged. The actual native build / 41 passed in 15.76s and prior full Ruff checks below are reused, not rerun for this CI/changelog-only synchronization.

Previous release-sync and native validation receipts (retained unchanged):

AI-assisted release-sync receipt: merged current main 46816d22f55432ecd4b81e2b2ff99dc345578fe9 in 5fbd385dbad25287e033019d1dc83c1c76077662, retaining the embedding entry under Unreleased and the upstream 0.3.37 release section. This merge changes only the release heading and package version metadata; embedding source, tests, native vendor a956a852, CMake and dependencies are unchanged.

The actual native build and 41 passed in 15.76s result below were reused, not rerun for this metadata-only synchronization. The current package imports as 0.3.37 with the existing native library; Ruff check/format of all 30 Python files and whitespace checks pass. No new hosted-CI or other-platform validation is claimed.

Previous native validation receipt (2026-11-14, retained unchanged):

AI-assisted maintenance update: merged current main dd70c923 in 764ea76cabb7d7270c3d2f0590d6348529f4f982, retaining the embedding fix and both changelog entries alongside the upstream native bindings/vendor update.

Rebuilt the native library against actual vendor a956a852c9c8bc1fb07206364838ddef299178c0 on macOS arm64 / Python 3.12.14 with -DGGML_NATIVE=OFF -DGGML_ACCELERATE=ON (two build jobs). The complete configured local suite passed: 41 passed in 15.76s, including the CPU long-embedding, aggregate-batch, explicit-limit and causal controls. Models were reused from the local cache in offline mode. ruff check llama_cpp tests, ruff format --check llama_cpp tests and git diff --check also pass. This is fresh native verification for this vendor; it does not claim hosted CI or other-platform validation.

@abetlen, when convenient, could you review the non-causal embedding batch-limit change?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Llama.embed crashes when n_batch > 512

1 participant