[ISSUE-830] Fix inverted embedding value filter that dropped digit-free text - #832
Open
E2ern1ty wants to merge 1 commit into
Open
[ISSUE-830] Fix inverted embedding value filter that dropped digit-free text#832E2ern1ty wants to merge 1 commit into
E2ern1ty wants to merge 1 commit into
Conversation
The value filter feeding the embedding store dropped ordinary prose and kept digit-only noise, so an embedding store silently produced nothing for any text that happened to contain no digit. No request was issued, no error was raised, and the entity was still registered as indexed with an empty vector list, which makes it unrecallable. SearchUtils.isAllAllowedChars returned on the first character *inside* the ignorable set instead of the first one outside it, which is the negation of both its name and its Javadoc. Reproduced on master with an unusable ModelConfig: "no digits here at all" completes without contacting a model and writes zero index lines while logging "Successfully added 1 new index items", whereas the same text with a digit appended does attempt the request. - Correct the predicate and rename it to isAllIgnorableChars. Renaming rather than fixing in place is deliberate: a caller depending on the old meaning now fails to compile instead of silently flipping behaviour. Null and empty become ignorable, since callers use this to decide what to skip and there is nothing in them to index. Both call sites are in SubgraphSemanticPromptFunction and read the same as before, `!isAllIgnorableChars(value)`. - Stop reporting entities with no embeddable text as indexed. indexBatch now warns with a count and an example key, and the summary line reports how many entities actually hold vectors rather than how many were queued. Silence was half of this defect. Measured on the LDBC test dataset before changing anything: of 168 entities, the set wanting vectors is the same before and after the fix, and none of them are absent from the committed index file. LDBC values mix letters and digits, so both the old and the new predicate keep them; the two differ only on values that are entirely digits or punctuation, and on values that are entirely letters. The committed embedding index therefore needs no regeneration and GraphMemoryTest passes unchanged. Tests in IgnorableTextFilterTest, 7 cases: the predicate over values that do and do not carry meaning, the property that keeping a value must not depend on it containing a digit, verbalization of digit free vertex and edge text, and the store either attempting a request or correctly declining to. All 7 fail against the previous behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #830.
What changes were proposed in this pull request?
The value filter feeding
EmbeddingIndexStorekept values containing a digit or one of. _ - @ + ! $ % & = ~and dropped everything else, so ordinary prose was never embedded while a bare id, a date or a run of punctuation was.SearchUtils.isAllAllowedCharsreturned on the first character inside the ignorable set rather than the first one outside it, which is the negation of both its name and its Javadoc, andSubgraphSemanticPromptFunction.verbalize(GraphEntity)keeps a value when that predicate is false.Two changes, matching the split agreed in the issue.
Correct the predicate, and rename it to
isAllIgnorableChars. It now returns true only when every character is ignorable, and treats null and empty as ignorable since callers use it to decide what to skip and there is nothing in them to index. The rename is deliberate rather than a fix in place: with the name unchanged, a caller depending on the previous meaning would silently flip behaviour, whereas a rename makes it fail to compile. Both call sites are inSubgraphSemanticPromptFunctionand read the same as before,!isAllIgnorableChars(value).IGNORE_CHARSbecomesIGNORABLE_CHARS, and the Javadoc now describes the set as characters that carry no meaning on their own.Stop reporting entities with no embeddable text as indexed. This is the half that held regardless of the predicate's intended semantics, and it was requested on the issue. A digit-free entity logged
Successfully added 1 new index items. Total indexed: 1while producing nothing: the entity was registered inindexStoreMapwith an empty vector list, so it could never be recalled, and nothing distinguished that from a successful build.indexBatchnow warns with a count and an example key when entities yield no text, and the summary line reports how many entities actually hold vectors rather than how many were queued.How was this PR tested?
Unit tests.
IgnorableTextFilterTest, 7 cases: the predicate over values that do and do not carry meaning, the property that whether a value is kept must not depend on it containing a digit, verbalization of digit-free vertex and edge text, verbalization dropping a value that is only a date, and the store either attempting a request or correctly declining to. All 7 fail against the previous behaviour. The two store cases run offline: an unusableModelConfigmakes "a request was attempted" observable without a service, since reaching the request at all fails on the null url. Fullgeaflow-aisuite passes, 13 tests, 0 checkstyle violations.End-to-end against a real embedding service. Both
master(3f73eb55) and this branch were run through the full production path against SiliconFlowBAAI/bge-m3: real HTTPS requests, real 1024-dimension vectors, real on-disk index file, real recall throughGraphMemoryServer. Corpus is the module's owntext/Confucius, 532 chunks of Chinese prose, of which 532 contain no digit. Same harness source compiled against both trees, so the numbers are comparable.Successfully added 532 new index items. Total indexed: 532Successfully added 532 new index items. Entities holding vectors: 532 of 532学习和思考的关系是什么?子曰:“学而时习之,不亦说乎?…”, cosine 0.5835So on
masterthe whole corpus was reported as indexed while nothing was embedded and no request was ever issued. Recall was exercised with an embedding vector only, no keyword vector and no other index store registered, so anything returned had to come from the embedding index. The written index file holds 548 records of 1024 real dimensions each, 11.35 MB, keyed by entity, with no zero vectors.The counterpart case on the same endpoint, 20 date-only values, which is what the old predicate selected for:
Successfully added 20 new index items. Total indexed: 20WARN 20 of 20 entities have no embeddable text and will hold no vectors, for example Vd1chunk, thenEntities holding vectors: 0 of 20The two tables together are the inversion the issue describes, measured against a live model rather than argued:
masterspends requests on dates and none on prose, this branch does the opposite and says so when there is nothing to embed.Blast radius, measured before changing anything. On the LDBC test dataset, of 168 entities the set wanting vectors is identical before and after the correction, and none of them are absent from the committed
LDBCEmbeddingIndexStorefile. The two predicates differ only on values that are entirely digits or punctuation, and on values that are entirely letters; LDBC entities have neither. The committed embedding index therefore needs no regeneration andGraphMemoryTestpasses unchanged, which is also why the defect was invisible until now. It does mean the module's own end-to-end scenario was affected:MemoryServerTestimports those same 532 prose chunks, none of which would previously have been embedded.