Skip to content

Python: fix(faiss): resolve the serialized key name on upsert - #14690

Open
哈基米 (BlueX888) wants to merge 3 commits into
microsoft:mainfrom
BlueX888:fix/faiss-key-storage-name
Open

哈基米 (BlueX888) wants to merge 3 commits into
microsoft:mainfrom
BlueX888:fix/faiss-key-storage-name

Conversation

@BlueX888

@BlueX888 哈基米 (BlueX888) commented Oct 30, 2026 •

Copy link
Copy Markdown

Motivation and Context

FaissCollection._inner_upsert reads the record key by the declared key field name, but records reach it already serialized. The key name in that serialized mapping depends on the serializer: the dict path writes the key's storage_name, while a pydantic model_dump() or a custom to_dict() may write the logical name.

So a definition whose key field declares storage_name fails every upsert of a plain dict record:

VectorStoreOperationException: Error upserting record(s) into collection 'test-alias': 'id'

InMemoryCollection already handles both shapes by using the key name recorded during serialization; FaissCollection overrides _inner_upsert and dropped that logic.

Description

  • Read the key name the serialization recorded when it is available (getattr(record, "key_field_name", ...)), otherwise fall back between the storage name and the logical name, mirroring InMemoryCollection._inner_upsert.
  • Read the vector under either its storage name or its logical name, so a vector field that declares storage_name works when the record is serialized with logical names, as model_dump() does (second commit).
  • Return an empty result when the target index holds no vectors, instead of asking faiss for min(top, 0) neighbours, which fails with a bare assert k > 0. This is the state right after ensure_collection_exists(), and after deleting the last record (third commit).
  • Keep filter validation on that empty path: the filter is still resolved before returning, so an invalid filter raises as it does for a populated index.
  • Add regression tests with a key field that declares storage_name, parametrized over dict and pydantic records, plus a custom to_dict() that emits logical names, and one test for a pydantic model whose vector field declares storage_name. The parametrized test asserts the returned key, the index key map, inner storage and the search round trip; the to_dict test asserts the returned key and the index key map; the vector test asserts the returned key, the index size, the indexed vector itself and the search round trip. Two more tests cover searching a fresh collection and searching after the last record is deleted; a third asserts that an invalid filter on an empty collection still raises.

Verification

  • The new key test fails on main with KeyError: 'id' at faiss.py:183, and passes with this change.
  • python/tests/unit/connectors/memory/test_faiss.py (+ test_in_memory.py): 109 passed. On main, the key test fails with KeyError: 'id', the vector-alias test with not enough values to unpack (expected 2, got 1), and both empty-index tests with VectorSearchExecutionException from faiss' assert k > 0.
  • ruff check and ruff format --check are clean on both changed files.
  • I ran the suite under pydantic 2 and faiss-cpu on Python 3.12 (Windows); the repo's other memory-connector tests are unaffected.

Known limitations, unchanged by this PR: a pydantic model that sets serialize_by_alias=True with a serialization_alias for the vector field that matches neither the logical name nor storage_name still fails, and a definition where one field's logical name equals another field's storage_name resolves to the other field. Both predate this change. Also pre-existing and left alone: FaissCollection._inner_delete keeps the original index offsets after remove_ids compacts positions, so deleting records that are not at the tail can leave the key map misaligned with the index; the delete tests here only remove the last record.

AI disclosure: analysis, implementation and test runs were assisted by Claude Code; I reviewed the change and can explain it.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
  • All unit tests pass, and I have added new tests where possible
  • I didn't break anyone 😄

FaissCollection._inner_upsert reads the record key by the declared key
field name. Records arrive already serialized, and the key name differs
by serializer: the dict path writes the key's storage_name, while a
pydantic model_dump() or a custom to_dict() may write the logical name.
The dict shape therefore failed every upsert with KeyError, and simply
switching to the storage name would have broken the pydantic shape that
works today.

Mirror InMemoryCollection: use the key name the serialization recorded
when it is available, otherwise fall back between the storage name and
the logical name. Add regression tests for the dict, pydantic and custom
to_dict shapes with a key field that declares storage_name.
Copilot AI balanced review requested due to automatic review settings October 30, 2026 12:20
@BlueX888
哈基米 (BlueX888) requested a review from a team as a code owner October 30, 2026 12:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@semantic-kernel-automation semantic-kernel-automation Bot added the python Pull requests for the Python Semantic Kernel label Oct 30, 2026
The vector read in _inner_upsert assumed the field's storage_name is
always present, but that is only true for the dict path: a pydantic
model_dump() and a custom to_dict() keep the logical name, so a vector
field with storage_name produced a one-element list and numpy raised
"not enough values to unpack".

Read the vector under either name, mirroring the key lookup in the
previous commit, and cover the pydantic-model shape with a regression
test.
_inner_search asked faiss for min(top, ntotal) neighbours without
checking ntotal, so searching a collection that was just created, or
one whose last record was deleted, failed with faiss' bare
"assert k > 0" wrapped in a VectorSearchExecutionException. The
in-memory collections return an empty result in the same situation.

Return an empty result before touching faiss, and cover both the
fresh-collection and the deleted-last-record paths.

This branch was successfully deployed

1 active deployment
github-app-auth — bd161f19 Deployed Oct 30, 2026 by BlueX888 via add_label #29509
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Pull requests for the Python Semantic Kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants