Repository navigation
fix: views read through their position only under the cache lock - #116
Merged
Merged
Conversation
Eight threads mix get/set/pop with walking keys()/values()/items() and calling len()/bool() on the views, in a child process so a fault fails the test instead of killing pytest. items() must yield key -> key pairs without duplicates in one pass, which also catches stale reads that don't fault. On free-threaded builds this crashes with SIGSEGV on main (FIFO, LFU, LRU and TTL in every local run, VTTL in some). faulthandler dumps all threads if the child hangs.
Views (keys/values/items, LFU items_with_frequency, TTL items_with_expire) hold raw positions into the policy storage and rely on the generation version to notice changes. They compared the generation without holding the cache lock and then read, so on free-threaded builds another thread could free or move entries in between (awolverp#94 reported this crash; awolverp#98 made replace() bump the generation, but the check-then-read race remained). - implement_view_guard! now generates lock_and_check(): take the cache lock, then compare the generation, and return the guard so it is held while reading. The unlocked check_generation() is removed, so every view method goes through the lock (__next__, __len__, and TTL/VTTL __bool__). - View fields hold the typed cache (Py<PyXCache>) to reach its lock. - TTL view constructors created the iterator and read the generation under two separate locks; a mutation in between made stale positions look current. Both now happen under one lock, as VTTL already did. - docs: iterator methods take the cache lock; calling them from code that runs inside an operation on the same cache (a key's __eq__, a value's __del__) blocks, like cache[key] there. Refs awolverp#94
Owner
|
Thanks ❤️ This PR will be better with a few changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #94 / #98.
Problem
#98 made
replace()bump the generation version, so iterating while another thread replaces a value now raisesRuntimeErrorinstead of reading a freed value. The underlying check-then-read race is still there, though: a view's__next__compares the generation without holding the cache lock and then reads through its raw position. On free-threaded builds another thread can free or move entries between the check and the read (an insert that grows the table,pop, eviction, …).Two related spots have the same race:
__len__and__bool__read entries (to skip expired ones) after the same unlocked check.On current
mainwith CPython 3.14t, eight threads mixingget/set/popwith iterating views crash with SIGSEGV forFIFOCache,LFUCache,LRUCacheandTTLCachein 5/5 local runs andVTTLCachein 2/5. The idea of holding the cache lock in the views comes from #94.Fix
implement_view_guard!now generateslock_and_check(): take the cache lock, then compare the generation, and return the guard so it stays held while the view reads. The unlockedcheck_generation()is removed, so every view method has to go through the lock and the compiler enforces it:__next__and__len__of all views,__bool__of TTL/VTTL views (16 call sites).Py<PyXCache>instead ofPy<PyAny>) to reach its lock.Behaviour change
next()/len()/bool()on a view now wait for the cache lock instead of reading unsynchronised. Calling them from Python code that runs inside an operation on the same cache (a key's__eq__, a value's__del__during eviction) now blocks, likecache[key]from those places already does. I added one paragraph about this todocs/docs/tips.md(Iteration Safety).Tests
New
IterationMixin.test_iterating_while_other_threads_mutateruns for every cache class. A child process runs 8 threads that mixget/set/popwith walkingkeys()/values()/items()and callinglen()/bool()on the views.items()must yieldkey -> keypairs without duplicates in one pass, which also catches stale reads that don't fault (LRUCachereuses freed list nodes, so a stale read returns another entry instead of crashing). It runs in a child process so a fault fails the test instead of killing pytest, like the existingEQ_TRIGGERING_GCtests, andfaulthandlerdumps all threads if the child hangs.mainwith only the test added, GitHub's ubuntu runner: 3.14t (FIFO, LFU, RR, TTL fail) and 3.15t (FIFO, LFU, TTL fail); every GIL build passes.What the test catches, reverting one part of the fix at a time (3 runs each, 3.14t):
__next__lock__len__/__bool__lockCachefollows the same pattern but didn't crash in this mix, because its table rarely reallocates under it;RRCachecrashed only on the x86 runner. Those are covered by the shared macro rather than by a reliable red run.Related, not in this PR
While testing I found that methods taking the lock with plain
inner.policy()instead ofpolicy_py_attached(clear,shrink_to_fit,popitem,len(cache),bool(cache),capacity, …) can hang free-threaded builds under contention when another thread needs a stop-the-world (GC, starting a thread). It reproduces onmainwithout any iteration. I'll open a separate issue or PR for it, since it's a different fix.