Repository navigation
Python: fix(faiss): resolve the serialized key name on upsert - #14690
Open
哈基米 (BlueX888) wants to merge 3 commits into
Open
哈基米 (BlueX888) wants to merge 3 commits into
哈基米 (BlueX888) wants to merge 3 commits into
Conversation
FaissCollection._inner_upsert reads the record key by the declared key field name. Records arrive already serialized, and the key name differs by serializer: the dict path writes the key's storage_name, while a pydantic model_dump() or a custom to_dict() may write the logical name. The dict shape therefore failed every upsert with KeyError, and simply switching to the storage name would have broken the pydantic shape that works today. Mirror InMemoryCollection: use the key name the serialization recorded when it is available, otherwise fall back between the storage name and the logical name. Add regression tests for the dict, pydantic and custom to_dict shapes with a key field that declares storage_name.
The vector read in _inner_upsert assumed the field's storage_name is always present, but that is only true for the dict path: a pydantic model_dump() and a custom to_dict() keep the logical name, so a vector field with storage_name produced a one-element list and numpy raised "not enough values to unpack". Read the vector under either name, mirroring the key lookup in the previous commit, and cover the pydantic-model shape with a regression test.
_inner_search asked faiss for min(top, ntotal) neighbours without checking ntotal, so searching a collection that was just created, or one whose last record was deleted, failed with faiss' bare "assert k > 0" wrapped in a VectorSearchExecutionException. The in-memory collections return an empty result in the same situation. Return an empty result before touching faiss, and cover both the fresh-collection and the deleted-last-record paths.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
FaissCollection._inner_upsertreads the record key by the declared key field name, but records reach it already serialized. The key name in that serialized mapping depends on the serializer: the dict path writes the key'sstorage_name, while a pydanticmodel_dump()or a customto_dict()may write the logical name.So a definition whose key field declares
storage_namefails every upsert of a plain dict record:InMemoryCollectionalready handles both shapes by using the key name recorded during serialization;FaissCollectionoverrides_inner_upsertand dropped that logic.Description
getattr(record, "key_field_name", ...)), otherwise fall back between the storage name and the logical name, mirroringInMemoryCollection._inner_upsert.storage_nameworks when the record is serialized with logical names, asmodel_dump()does (second commit).min(top, 0)neighbours, which fails with a bareassert k > 0. This is the state right afterensure_collection_exists(), and after deleting the last record (third commit).storage_name, parametrized over dict and pydantic records, plus a customto_dict()that emits logical names, and one test for a pydantic model whose vector field declaresstorage_name. The parametrized test asserts the returned key, the index key map, inner storage and the search round trip; theto_dicttest asserts the returned key and the index key map; the vector test asserts the returned key, the index size, the indexed vector itself and the search round trip. Two more tests cover searching a fresh collection and searching after the last record is deleted; a third asserts that an invalid filter on an empty collection still raises.Verification
mainwithKeyError: 'id'atfaiss.py:183, and passes with this change.python/tests/unit/connectors/memory/test_faiss.py(+test_in_memory.py): 109 passed. Onmain, the key test fails withKeyError: 'id', the vector-alias test withnot enough values to unpack (expected 2, got 1), and both empty-index tests withVectorSearchExecutionExceptionfrom faiss'assert k > 0.ruff checkandruff format --checkare clean on both changed files.Known limitations, unchanged by this PR: a pydantic model that sets
serialize_by_alias=Truewith aserialization_aliasfor the vector field that matches neither the logical name norstorage_namestill fails, and a definition where one field's logical name equals another field'sstorage_nameresolves to the other field. Both predate this change. Also pre-existing and left alone:FaissCollection._inner_deletekeeps the original index offsets afterremove_idscompacts positions, so deleting records that are not at the tail can leave the key map misaligned with the index; the delete tests here only remove the last record.AI disclosure: analysis, implementation and test runs were assisted by Claude Code; I reviewed the change and can explain it.
Contribution Checklist