test(memory): cover opposite polarity relevance filtering

[mikemikimike]

Motivation and Context

Fixes #14295 by adding focused regression coverage for the Python TextMemoryPlugin.recall() relevance threshold. The issue identifies a test blind spot: high embedding similarity does not establish semantic agreement, and a fixed threshold can also miss semantically equivalent paraphrases.

Description

Adds deterministic, offline Python coverage using four recorded sentences in two independent pairs:

• A negated instruction pair whose real cosine similarity is 0.9748118, so the opposite meaning is recalled through the default 0.75 threshold.
• A semantically equivalent paraphrase pair whose real cosine similarity is 0.7302992, so it is filtered by the same threshold.

The vectors were recorded from nomic-ai/nomic-embed-text-v1.5 through the local LM Studio OpenAI-compatible endpoint, with 768 dimensions and float32 serialization. The fixture includes the model checkpoint, endpoint, generation date, dimensions, source texts, measured cosines, and the regeneration script. The model is Nomic; this does not use text-embedding-3-small.

No production behavior is changed. The tests document the limitation of treating cosine similarity as a relevance/semantic-agreement decision while remaining deterministic and offline.

Verification

• python tests/unit/memory/generate_recorded_nomic_relevance_embeddings.py passed with the local LM Studio Nomic endpoint.
• python -m pytest tests/unit/memory/test_text_memory_plugin_relevance.py -q passed: 5 tests.
• git diff --check passed.

The full repository test suite and .NET validation were not run; this patch adds Python unit tests only and does not change .NET production code.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All focused unit tests pass, and I have added regression tests
• I didn't break anyone 😄

Related Issues

Fixes #14295

[Copilot]

Pull request overview

Adds a new .NET unit test to explicitly cover how minRelevanceScore gating behaves in VolatileMemoryStore.GetNearestMatchesAsync when a stored record’s embedding is the opposite direction of the query embedding.

Changes:

• Adds GetNearestMatchesFiltersOppositePolarityAsync to validate relevance-threshold filtering behavior with an opposite-direction embedding.
• Asserts only the matching-direction record is returned when minRelevanceScore is set to 0.75.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[mikemikimike]

Addressed in the current head a3e1354a. The test now uses recorded high-positive embeddings for both the matching and opposite-polarity records, asserts both scores are at least 0.75, and documents that this is an offline reproduction of the reported embedding behavior rather than a model call.

[mikemikimike]

Addressed in the current head a3e1354a: the test now defines the relevance threshold once and reuses it for the search call and all assertions, avoiding duplicated magic numbers.

[github-actions]

MAF Automated Review — Iteration 1

Result: No findings
Scope: full PR (1 commit(s)): 7845fa02485a
Model: claude-opus-4.8

Overview

This PR adds a single xUnit [Fact] (GetNearestMatchesFiltersOppositePolarityAsync) to VolatileMemoryStoreTests.cs and changes no production code. It pins the existing cosine-similarity threshold semantics of VolatileMemoryStore.GetNearestMatchesAsync by asserting that an opposite-polarity embedding ({-1,0}, cosine -1.0) is excluded under minRelevanceScore: 0.75 while the matching embedding ({1,0}, cosine 1.0) is returned. The math is exact and deterministic with wide margins from the threshold, record keys are distinct, and per-[Fact] instance isolation (_db and _collectionNum reset per test) prevents cross-test contamination. The only residual caveat is that neither the author nor the review environment could execute the .NET test (no dotnet toolchain); correctness here rests on static analysis, which is strong.

Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
No publishable findings remained after source verification for this scope.

[mikemikimike]

Addressed the review feedback in commit dc2df3a: the polarity fixture now gives both matching and opposite-meaning text high positive similarity (~0.98), rather than only testing negative-cosine filtering; a complementary low-similarity semantic-equivalent case was added; and the threshold is shared via one variable. The vectors are explicitly synthetic fixtures modeling the audited range, not embeddings computed by the unit test.

[Scott Frias (poeticize)]

Mike,

Thanks for taking this on, and for iterating twice on the review feedback. I want to flag something
about the fixtures before this merges, because as written I do not think the two tests assert what
their names say.

In GetNearestMatchesUsesCosineSimilarityForOppositeMeaningAsync, the matching record and the
opposite record carry the same vector:

id: "matching",  ...  embedding: new float[] { 0.98f, 0.199f }
id: "opposite",  ...  embedding: new float[] { 0.98f, 0.199f }

Both are { 0.98f, 0.199f }, so "both clear 0.75" holds by construction. It would still hold if the
two sentences were swapped, or replaced with lorem ipsum. Nothing in the assertion touches the text.
GetNearestMatchesFiltersLowSimilarityParaphraseAsync has the same shape: { 0.6f, 0.8f } was
chosen to give cosine 0.6, so Assert.Empty confirms that choice. Between them the two tests pin
VolatileMemoryStore's cosine arithmetic, which GetNearestMatchAsyncReturnsExpectedAsync, directly
above in the same file, already covers.

That is fixable rather than wasted. This PR says
Fixes #14295, so a merge closes the issue, and SK then ships two tests named for polarity coverage.
Anyone who later asks whether SK tests this gate finds green tests with the right names and stops
looking. The paper behind the issue argues that shipped suites assert a gate is fine without
measuring whether it is, and I would rather the fix here not become an instance of it.

What I think does work, given that a unit test cannot call an embedding model. *Record the vectors
once from a real encoder for the four sentences and check them in, with provenance in the fixture:
model id, dimension, date, and the script that generated them. * The test then asserts something
falsifiable about the world, that the opposite-meaning record clears the shipped default while the
faithful paraphrase does not, and it stays deterministic and offline. Prefer an encoder the ecosystem
actually runs, such as text-embedding-3-small. The effect does not hinge on the choice; the
mutation class scored 0.83 to 0.9997 across all nine encoders I tested.

Two smaller notes. VolatileMemoryStore is a deprecated in-memory store, so it may not be the
placement you want for a test meant to outlive the migration, and any live-encoder assertion belongs
in the integration project behind its existing trait. Separately, the audit ran against the Python
side, where DEFAULT_RELEVANCE is 0.75, while the C# DefaultRelevance is 0.0. That does not make
C# the wrong target, it only changes which default the test should assert at.

I am glad to generate the recorded vectors and send them over so you keep authorship here. Say the
word and I will produce them.

[Scott Frias (poeticize)]

Mike,

I offered to generate the recorded vectors. Rather than wait on a yes, I built them.

Package: https://gist.github.com/poeticize/cfb230f9ae92ad018b649470047cadfb

Everything in it runs offline from open weights. No credential, no API budget, no GPU. The
README.md in the gist repeats all of this in more detail.

What is in it

1. MemoryRelevancePolarityTests.cs: four xUnit tests, drop-in for
   dotnet/src/Plugins/Plugins.UnitTests/Memory/, namespace SemanticKernel.UnitTests.Memory. The
   vectors are inline, so there is no build-file change and no embedded resource to wire up.
2. sk_polarity_fixture.json: the sentences, their recorded 256-dim vectors, provenance, measured
   cosines, and a nine-configuration cross-encoder table.
3. generate_sk_polarity_fixture.py: produces the fixture from the checkpoint.
4. emit_sk_dotnet_test.py: produces the .cs from the fixture, so the asserted numbers cannot
   drift from the measurement that produced them.
5. measure_sk_specimens.py: measures the pairs across every configuration. Writes nothing.

Encoder. nomic-ai/nomic-embed-text-v1.5 via plain sentence-transformers, prefix
search_document: , truncated to 256 dims and renormalized. I recommended text-embedding-3-small
in my last comment and then went the other way, for one reason: you can reproduce this one. An
OpenAI vector is a number a reviewer has to take on faith, while open weights let you regenerate
every figure below and check it against mine.

The specimens did change shape. My earlier comment said "the four sentences". While generating I
moved to a controlled 2x2 on a single anchor, because it isolates the comparison instead of leaning
on two unrelated pairs. One query, two stored memories:

role	text	meaning	wording	token-Jaccard	cosine
query	Retry the request at most three times.
memory A	Retry the request at most thirty times.	BROKEN	close	0.7500	0.9891
memory B	Give the call up to three attempts, then stop.	PRESERVED	distant	0.1429	0.7847

The gate ranks A above B, and A is the one that contradicts the query. Both memories come from a
balanced corpus authored before any measurement and blind to any encoder.

A caveat about one number. The issue quotes the clinical pair at 0.9608, a figure from the
audited production router in my audit repository, which cannot be installed by you. The router
applies processing that plain sentence-transformers does not reproduce. I shipped the reproducible
path and reported both. The reproducible path puts the same pair at 0.9195. Every conclusion holds
under either, and the production figures are in the fixture under audited_production_router_cosines.
I would rather hand you a generatable number than one that matches my paper. (FYI I am about to
publish another paper that enhances this whole result set, and will link it here once we publish.)

The finding. Python's DEFAULT_LIMIT is 1, so a defaulted recall() returns exactly one memory.
Under this ranking, that one memory is the one that breaks the meaning. Raising the relevance floor
does not rescue it, as both memories clear 0.75, and the more faithful is the lower of the two. The
four tests assert: 1. the ranking, 2. the defaulted single-result case, 3. the floor failing to
separate them, and 4. the clinical reversal clearing the floor.

Not one model's quirk. I measured the same three pairs across all nine encoder configurations in
the audit registry. The inversion held in 9 of 9. The table ships in the fixture under cross_encoder_table,
making it checkable.

Validation you can run

• Regenerate: python generate_sk_polarity_fixture.py. Offline, needs sentence-transformers and
  the checkpoint. It prints the cosine and Jaccard table before it writes anything.
• Confirm self-consistency: recompute cosine from the stored vectors and compare against the
  measured block. I get agreement to six decimal places, and every vector is unit norm at 256 dims.
• I could not run dotnet either, so I verified the .cs by parsing its own float arrays back out
  and simulating GetNearestMatchesAsync ordering and threshold filtering at float32. All four tests
  pass. Ranking margin is 0.204, and the tightest assertion is memory B sitting 0.0347 above the
  floor. That is static verification, so CI still has to be the judge.

Two more things: the Python side has no behavioral unit test for recall at all, so
a Python version would be greenfield rather than a modification of anything. And I am the issue
reporter here rather than an SK maintainer, so the acceptance call belongs to them and not to me.

Take any of this, change any of it, and keep authorship. If you would rather have the Python version,
say so and I will build that too.

Scott