fix(tests): isolate the LangCache integration suite from its shared cache - #725
Open
vishal-bala wants to merge 1 commit into
Open
fix(tests): isolate the LangCache integration suite from its shared cache#725vishal-bala wants to merge 1 commit into
vishal-bala wants to merge 1 commit into
Conversation
The LangCache integration suite flaked semi-randomly in the Service Tests job, a different test each run, on PRs touching nothing LangCache-related. Two tests flushed the entire managed cache while the suite runs under `pytest -n auto` against a single cache_id from repo secrets, so concurrent xdist workers and concurrent CI runs on different PRs wiped each other's entries. - Remove both whole-cache flushes, and add an autouse fixture that fails any test calling delete/adelete/clear/aclear. The flush wrappers keep their mocked unit coverage; the flush HTTP path is deliberately left untested, because no cache we share can safely be flushed. - Tag every write with a per-test scope token, so no test can observe or delete another's entries, and give every entry a TTL so the shared caches drain without anyone flushing them. - Filter the TTL-expiry tests on a scope-unique attribute, and widen their TTL so the pre-expiry assertion is not racing a two-second budget across two live round trips. num_results is a client-side slice only, since the service returns one result by default, so it cannot provide the isolation an attribute filter does. - Tighten the delete-by-attribute counts from >= 1 to the number actually stored, now that scope-unique attributes make the count knowable. - Join scope tokens onto punctuation-heavy attribute values with "_" rather than "-", which survives percent-encoding and is a text separator. - Autospec the unit-test SDK mock, so a renamed method or a changed signature fails there instead of being silently auto-vivified. - Fix two unit tests that were indented into another test and so were never collected.
vishal-bala
marked this pull request as ready for review
September 4, 2026 11:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The LangCache integration suite fails semi-randomly in the Service Tests job, on pull requests that touch nothing related to LangCache, and a different test fails each time. A reproducible defect fails the same test every run, so the pattern itself points at a shared resource, not at any one branch: #716 saw
test_store_and_check_asyncfail onassert [], its stored entry gone; #718 saw two different tests fail, and a third on re-run.Two things combine to cause it.
TestLangCacheSemanticCacheIntegrationWithAttributesflushed the entire managed cache, throughdelete(),clear()andaclear(), and the suite runs underpytest -n auto, so a flush on one xdist worker wiped entries another worker had stored moments earlier. The fixtures then bind to a singlecache_idfrom repo secrets, which every CI run reaches concurrently: pull requests, fork pull requests, pushes to main, and the nightly cron. Runs on separate branches therefore flushed each other.Changes
Whole-cache flushes are removed, and blocked from returning
test_delete_and_clear_aliasis deleted and the trailingaclear()is removed fromtest_async_delete_variants. An autouse fixture patchesdelete,adelete,clearandaclearto fail the test, so the rule is now mechanical, not a convention recorded in a docstring.This trades away live coverage of the flush endpoint, which is a constraint and not a preference: no cache shared with other runs can safely be flushed, and a dedicated flushable
cache_idwould collide identically once two runs used it. The wrappers themselves are four one-line calls into the SDK and keep their mocked unit coverage; the flush HTTP path is deliberately untested.Every write is scoped and expiring
A per-test
scopetoken, fromuuid4().hex[:12], is threaded into every prompt, response and attribute value, so no test can observe or delete another's entries. Every entry carries a 60-second TTL, which lets the shared caches drain now that nothing flushes them. The TTL is passed per call, not set once on the fixtures, becausestore()ignores a constructor TTL entirely. That defect is filed separately, and the module docstring records it so the repetition is not tidied away into the fixtures.The TTL-expiry tests no longer depend on result ranking
num_resultsis a client-side slice:_build_search_kwargsnever sendsmax_results, and the installed SDK documents its own default as one result. Raising it therefore buys no headroom against a concurrent run's semantically identical prompt competing for that single slot. Filtering on a scope-unique attribute does, because the service can only return entries matching it.Both tests also move from a two-second TTL to five, with the sleep from three seconds to six. Two seconds had to cover a store round trip and a search round trip against a shared managed service, so the assertion that the entry exists was racing its own TTL.
Secondary changes:
>= 1to the number of entries actually stored, which scope-unique attributes make knowable._rather than-, which survives percent-encoding and is a RediSearch text separator.Note
Low Risk
Test-only and documentation changes; no production LangCache client behavior is modified.
Overview
Stops flaky LangCache integration failures caused by many workers and CI jobs sharing the same managed
cache_id.Whole-cache flush is removed from live tests and enforced with an autouse fixture that makes
delete/clear(sync and async) fail if called. Integration cases that flushed the cache are dropped; flush behavior stays covered only in unit mocks (now autospecced against the real SDK). Two previously nested unit tests are dedented so they actually run.Writes are isolated: per-test
scopetokens in prompts/responses/attributes, 60s TTL on routine stores, and assertions that look for your scoped row inhits(nothits[0]). TTL-expiry tests use scoped attribute filters, longer TTL/sleep, and tighter delete-by-attribute counts.CONTRIBUTING.md adds guidance to namespace data and never flush shared external services, pointing at the integration module docstring.
Reviewed by Cursor Bugbot for commit 68d378d. Bugbot is set up for automated code reviews on this repo. Configure here.