feat(index): owns_client for explicit Redis client ownership handover - #726
Draft
vishal-bala wants to merge 7 commits into
Draft
feat(index): owns_client for explicit Redis client ownership handover#726vishal-bala wants to merge 7 commits into
vishal-bala wants to merge 7 commits into
Conversation
BaseCache._get_async_redis_client called get_async_redis_connection, which warns unconditionally, so anyone who built a cache from a redis_url and awaited a cache method saw a DeprecationWarning for an API they never called. A suite-wide filter in pyproject.toml hid it. Point the method at _get_aredis_connection, the async form already used everywhere else, and drop the filter. Cache clients now also report their library name via CLIENT SETINFO like every other RedisVL client, and a connection failure surfaces when the client is created rather than at the first command. The regression guard lives in tests/unit/test_connection_normalization.py and lands with the next commit, which is the first to touch that file.
An index closed only a client it created itself, and callers who needed to override that wrote to the private _owns_redis_client keyword or poked the attribute afterwards. The MCP server did the latter, which never worked as intended: _register_client_finalizer gates on the flag, so a post-construction flip lands after registration has already declined and no finalizer is ever created. owns_client states ownership once, at construction, before the finalizer is registered. It replaces the private keyword rather than sitting alongside it, so there is one spelling and no precedence question. An explicit value also wins over the ownership from_existing would otherwise assume for a client it created, which is why the assignment there uses setdefault. Also documents the accessor asymmetry between the two classes: _redis_client lazily creates on SearchIndex but is a plain nullable attribute on AsyncSearchIndex, whose lazy getter is _get_client.
The four migration modules read index.client, which is None until the client is lazily created, and each handled that differently: one raised, two recorded an error, one dereferenced unguarded, and the planner degraded to an empty key sample that then made the key-sample check vacuously true. All four build their index through from_existing, which always yields a client, so none of this was reachable in practice. Reading through _redis_client and _get_client removes the disagreement and the dead guards with it. The test doubles are renamed to match the real accessors so they still stand in for an index.
The deprecation decorators told users each deprecated argument, function and class would be removed "in the next major release". Every release so far has been 0.x and breaking changes ship on minor bumps per project convention, so that promise has never been accurate and each removal would otherwise need a release note explaining the mismatch. Say "in a future release" instead. Warning text only; no behaviour change.
Moving this method onto _get_aredis_connection introduced an await between its "is the client None" check and the assignment, because the async factory issues a CLIENT SETINFO round trip. BaseCache has no lock, so two concurrent callers each built a client and the first was left unreachable: adisconnect only closes the client currently on the instance, so the orphan's connection pool was never released. Wrap the lazy path in a double-checked lock, the same shape AsyncSearchIndex._get_client already uses over the same factory. The regression test fails without the lock and passes with it.
SemanticRouter.from_existing merges {**init_kwargs, **index_kwargs}
with index_kwargs second, and set owns_client there unconditionally on
the branch where it creates the client. A caller's owns_client=False
was therefore discarded in silence, while SearchIndex.from_existing
honoured the same argument via setdefault. Same keyword, same verb,
opposite answer.
Claim ownership only when the caller has not already answered, so all
three public from_existing entry points agree.
Review of the owns_client work turned up several statements that were wrong or missing rather than merely terse. The owns_client entry said the index owns a client it created "from redis_url", but get_redis_connection falls back to REDIS_URL, so an index built with no connection arguments at all still creates and owns one. It also left the caller's obligation unstated: declining ownership of a client the index created means closing it yourself. disconnect was documented as "Disconnect from the Redis database" on the base and sync classes and not at all on the async one, which now misleads: it is a no-op for an unowned client, and with owns_client public that is a state callers choose. Its log line claimed the index did not own the client even when the index had created it. A test docstring asserted set_client() was already gone. It is not, until the next branch removes it, and pointing readers away from it hides the ownership footgun owns_client exists to fix. Also: coerce owns_client with bool(), since the finalizer gate tests truthiness while disconnect tested "is False", so a falsy non-bool made the two paths disagree; reject the retired private _owns_redis_client keyword loudly, because underscore-prefixed keywords are forwarded verbatim and it would otherwise be dropped in silence; hoist a lazily created client out of a per-key loop; drop the last dead client-is-None guard in the migration package; and stop one more warning promising removal in the next major release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
#660 reports that
SearchIndex.set_client()keeps client ownership, so an index closes a client the caller still owns. Two earlier pull requests tried to repair the ownership logic insideset_client()and both were closed: the method has carried aDeprecationWarningsince v0.4.0 and the project has shipped 23 further minor versions without removing it, so patching it adds machinery to a code path nobody should be on. The agreed fix is removal.This is the first of three stacked pull requests doing that. It removes nothing. It puts in place the one thing callers lose when
set_client()goes — a supported way to hand an index a client it should close — and clears the internal callers that would otherwise complicate the removal diffs. The two branches above it delete the deprecated index API and the deprecated connection-factory API.Changes
A public
owns_client, replacing a private keywordAn index closed only a client it created itself, and callers needing to override that passed a private
_owns_redis_clientkeyword or wrote the attribute afterwards. The MCP server did the latter, which never worked as intended:_register_client_finalizergates on the flag, so a post-construction flip lands after registration has already declined and no finalizer is ever created. That server was relying entirely on its explicitdisconnect(), with no safety net if a binding runtime were ever dropped.owns_clientstates ownership once, at construction, before the finalizer is registered. It replaces the private keyword rather than sitting beside it, so there is one spelling and no precedence question.The default is unchanged: an index owns a client it created and never closes one you passed. Note the default follows from who created the client, not from which argument you used —
get_redis_connectionfalls back toREDIS_URL, so an index built with no connection arguments at all also creates and owns one.Async cache clients no longer warn, and no longer race
BaseCache._get_async_redis_clientcalledget_async_redis_connection, which warns unconditionally, so anyone who built a cache from aredis_urland awaited a cache method saw aDeprecationWarningfor an API they never called. A suite-widefilterwarningsentry inpyproject.tomlhad been hiding it. The method now uses_get_aredis_connectionand the filter is gone.That move introduced an
awaitinside the lazy-init guard, which made check-then-set non-atomic and let two concurrent callers each build a client, orphaning the first with its pool never released. The lazy path is now wrapped in the same double-checked lockAsyncSearchIndex._get_clientalready uses over the same factory. The regression test fails without the lock.Migration modules read the client that actually exists
The four modules under
redisvl/migration/readindex.client, which isNoneuntil the client is lazily created, and each handled that differently: one raised, two recorded an error, one degraded to an empty key sample that then made the key-sample check vacuously true. All four build their index throughfrom_existing, which always yields a client, so none of it was reachable. They now read through the lazy accessors and the dead guards are gone.Smaller changes
disconnect()was documented as "Disconnect from the Redis database" on two classes and not at all on the async one, and its log line claimed the index did not own a client it had in fact created. Both corrected, andowns_clientis documented onfrom_existingtoo.owns_clientis coerced withbool(): the finalizer gate tests truthiness whiledisconnecttestedis False, so a falsy non-boolmade the two paths disagree._owns_redis_clientkeyword is now rejected with aTypeError. Underscore-prefixed keywords are forwarded verbatim by_split_from_existing_kwargs, so it would otherwise be dropped in silence and leak the connection it used to control.Notes
SemanticRouter.from_existingmerged{**init_kwargs, **index_kwargs}withindex_kwargssecond, so it discarded an explicitowns_client=FalsewhileSearchIndex.from_existinghonoured it. All three publicfrom_existingentry points now agree, and the rule is that an explicit value beats the ownership the library would otherwise infer. The consequence worth knowing:from_existing(name, redis_url=..., owns_client=False)produces a client nobody closes. The caller keeps a handle via.client, so it is an escape hatch rather than a leak, but there is no good reason to ask for it.Cache clients now issue
CLIENT SETINFOwhen they are created, so they finally report their library name like every other RedisVL client. The same change means an unreachable server surfacesConnectionErrorat client creation rather than at the first command. AResponseErrorthere is still swallowed, so a restricted ACL that forbidsCLIENT SETINFOis unaffected.Handing the same client to two indexes with
owns_client=Trueregisters two finalizers on it, so collecting the first closes the pool under the second. Do not do this. The redis-py documentation is explicit that closing one client which shares aConnectionPoolsilently invalidates the connections every other client using it holds, so the consequence is silent failure rather than reconnection. Ownership belongs to exactly one holder.The migration modules now reach into another package's private accessors. That is deliberate staging: the root cause is
.clientbeingOptional, which cannot be fixed untilconnect()andset_client()are gone. Branch 03 makes.clientnon-optional and reverts these call sites to the public property.BaseCachestill derives its own ownership flag with no keyword override and registers no finalizer, so there is no cache equivalent of this handover. That asymmetry is intentional here and recorded in a comment; nothing in the library needs to hand a cache a client it should close.Release Notes
SearchIndexandAsyncSearchIndexaccept a newowns_clientargument controlling whether the index closes the Redis client when it is disconnected or garbage collected. By default an index closes only a client it created itself and never closes one passed asredis_client, which is unchanged. Passowns_client=Trueto hand over a client you created, orowns_client=Falseto keep one the index would otherwise close.A cache built from a
redis_urlnow raisesConnectionErrorwhen its async client is created rather than at the first command, so an unreachable server surfaces earlier than before.