Skip to content

fix(tests): isolate the LangCache integration suite from its shared cache - #725

Open
vishal-bala wants to merge 1 commit into
mainfrom
test/langcache-shared-cache-isolation
Open

fix(tests): isolate the LangCache integration suite from its shared cache#725
vishal-bala wants to merge 1 commit into
mainfrom
test/langcache-shared-cache-isolation

Conversation

@vishal-bala

@vishal-bala vishal-bala commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Motivation

The LangCache integration suite fails semi-randomly in the Service Tests job, on pull requests that touch nothing related to LangCache, and a different test fails each time. A reproducible defect fails the same test every run, so the pattern itself points at a shared resource, not at any one branch: #716 saw test_store_and_check_async fail on assert [], its stored entry gone; #718 saw two different tests fail, and a third on re-run.

Two things combine to cause it. TestLangCacheSemanticCacheIntegrationWithAttributes flushed the entire managed cache, through delete(), clear() and aclear(), and the suite runs under pytest -n auto, so a flush on one xdist worker wiped entries another worker had stored moments earlier. The fixtures then bind to a single cache_id from repo secrets, which every CI run reaches concurrently: pull requests, fork pull requests, pushes to main, and the nightly cron. Runs on separate branches therefore flushed each other.

Changes

Whole-cache flushes are removed, and blocked from returning

test_delete_and_clear_alias is deleted and the trailing aclear() is removed from test_async_delete_variants. An autouse fixture patches delete, adelete, clear and aclear to fail the test, so the rule is now mechanical, not a convention recorded in a docstring.

This trades away live coverage of the flush endpoint, which is a constraint and not a preference: no cache shared with other runs can safely be flushed, and a dedicated flushable cache_id would collide identically once two runs used it. The wrappers themselves are four one-line calls into the SDK and keep their mocked unit coverage; the flush HTTP path is deliberately untested.

Every write is scoped and expiring

A per-test scope token, from uuid4().hex[:12], is threaded into every prompt, response and attribute value, so no test can observe or delete another's entries. Every entry carries a 60-second TTL, which lets the shared caches drain now that nothing flushes them. The TTL is passed per call, not set once on the fixtures, because store() ignores a constructor TTL entirely. That defect is filed separately, and the module docstring records it so the repetition is not tidied away into the fixtures.

The TTL-expiry tests no longer depend on result ranking

num_results is a client-side slice: _build_search_kwargs never sends max_results, and the installed SDK documents its own default as one result. Raising it therefore buys no headroom against a concurrent run's semantically identical prompt competing for that single slot. Filtering on a scope-unique attribute does, because the service can only return entries matching it.

# Before: reads back whichever single entry the service ranks first.
hits = langcache_with_attrs.check(prompt=prompt, num_results=5)

# After: the result set is provably this test's own.
hits = langcache_with_attrs.check(prompt=prompt, attributes=metadata)

Both tests also move from a two-second TTL to five, with the sleep from three seconds to six. Two seconds had to cover a store round trip and a search round trip against a shared managed service, so the assertion that the entry exists was racing its own TTL.

Secondary changes:

  • Delete-by-attribute assertions tighten from >= 1 to the number of entries actually stored, which scope-unique attributes make knowable.
  • Scope tokens join punctuation-heavy attribute values with _ rather than -, which survives percent-encoding and is a RediSearch text separator.
  • The unit-test SDK mock is autospecced, so a renamed method or a changed signature fails there instead of being auto-vivified.
  • Two unit tests were indented into another test and so were never collected. Dedenting them adds both to the run, and the file now collects 26, having also lost a name-checking test that autospec subsumes.
  • CONTRIBUTING.md gains a paragraph on namespacing writes to shared, stateful external services.

Note

Low Risk
Test-only and documentation changes; no production LangCache client behavior is modified.

Overview
Stops flaky LangCache integration failures caused by many workers and CI jobs sharing the same managed cache_id.

Whole-cache flush is removed from live tests and enforced with an autouse fixture that makes delete/clear (sync and async) fail if called. Integration cases that flushed the cache are dropped; flush behavior stays covered only in unit mocks (now autospecced against the real SDK). Two previously nested unit tests are dedented so they actually run.

Writes are isolated: per-test scope tokens in prompts/responses/attributes, 60s TTL on routine stores, and assertions that look for your scoped row in hits (not hits[0]). TTL-expiry tests use scoped attribute filters, longer TTL/sleep, and tighter delete-by-attribute counts.

CONTRIBUTING.md adds guidance to namespace data and never flush shared external services, pointing at the integration module docstring.

Reviewed by Cursor Bugbot for commit 68d378d. Bugbot is set up for automated code reviews on this repo. Configure here.

The LangCache integration suite flaked semi-randomly in the Service Tests
job, a different test each run, on PRs touching nothing LangCache-related.
Two tests flushed the entire managed cache while the suite runs under
`pytest -n auto` against a single cache_id from repo secrets, so concurrent
xdist workers and concurrent CI runs on different PRs wiped each other's
entries.

- Remove both whole-cache flushes, and add an autouse fixture that fails any
  test calling delete/adelete/clear/aclear. The flush wrappers keep their
  mocked unit coverage; the flush HTTP path is deliberately left untested,
  because no cache we share can safely be flushed.
- Tag every write with a per-test scope token, so no test can observe or
  delete another's entries, and give every entry a TTL so the shared caches
  drain without anyone flushing them.
- Filter the TTL-expiry tests on a scope-unique attribute, and widen their
  TTL so the pre-expiry assertion is not racing a two-second budget across
  two live round trips. num_results is a client-side slice only, since the
  service returns one result by default, so it cannot provide the isolation
  an attribute filter does.
- Tighten the delete-by-attribute counts from >= 1 to the number actually
  stored, now that scope-unique attributes make the count knowable.
- Join scope tokens onto punctuation-heavy attribute values with "_" rather
  than "-", which survives percent-encoding and is a text separator.
- Autospec the unit-test SDK mock, so a renamed method or a changed
  signature fails there instead of being silently auto-vivified.
- Fix two unit tests that were indented into another test and so were never
  collected.
@vishal-bala vishal-bala added the auto:tests Add or improve existing tests label Sep 4, 2026
@vishal-bala
vishal-bala marked this pull request as ready for review September 4, 2026 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto:tests Add or improve existing tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant