fix(cachekitio): build the sync client on HTTP/1.1 with a 32-connection pool (LAB-7062)

docs/backends/README.md

@@ -361,7 +361,7 @@ Other changes you may notice:
 | **L1 (In-Memory)** | ~50ns | Repeated calls in same process | Process-local only |
 | **File** | set = fsync + per-entry scan; get waits on a concurrent set | Single-process local caching | Development, scripts, CLI tools |
 | **Redis** | 1-7ms | Shared cache across pods | Production default |
-| **CachekitIO** | 42–45ms p50 hit, measured from MEL | Managed SaaS, zero-ops | HTTP/2; depends on vantage, store region and edge hits ([measured](cachekitio.md#characteristics)); closed beta |
+| **CachekitIO** | 42–45ms p50 hit, measured from MEL | Managed SaaS, zero-ops | HTTPS (sync HTTP/1.1, async HTTP/2); depends on vantage, store region and edge hits ([measured](cachekitio.md#characteristics)); closed beta |
 | **HTTP API** | 10-100ms | Custom cloud services | Network dependent |
 | **DynamoDB** | 100-500ms | Serverless, low-traffic | High availability |
 | **Memcached** | 1-5ms | Alternative to Redis | No persistence |

docs/backends/cachekitio.md

@@ -4,7 +4,7 @@
 
 > *cachekit.io is in closed beta — [request access](https://cachekit.io)*
 
-`CachekitIOBackend` connects to the cachekit.io managed cache API over HTTP/2. It implements the full `BaseBackend` protocol plus distributed locking (`LockableBackend`) and TTL inspection (`TTLInspectableBackend`).
+`CachekitIOBackend` connects to the cachekit.io managed cache API over HTTPS: HTTP/1.1 for sync calls, HTTP/2 for async ones. It implements the full `BaseBackend` protocol plus distributed locking (`LockableBackend`) and TTL inspection (`TTLInspectableBackend`).
 
 ## Setup
 
@@ -188,7 +188,6 @@ CACHEKIT_TIMEOUT=5.0                  # Optional — request timeout in seconds
 **When NOT to use**:
 - Sub-millisecond latency requirements — use Redis or L1 cache
 - Fully offline/air-gapped environments
-- Applications that cannot tolerate HTTP/2 dependency
 
 ## Characteristics
 
@@ -198,7 +197,17 @@ CACHEKIT_TIMEOUT=5.0                  # Optional — request timeout in seconds
   (n=50 per run). Your numbers depend on where your client enters Cloudflare, the store's region and
   how many reads the edge serves. No p95 is published yet: these samples are too few to claim one.
 - Sync and async support (hybrid client architecture)
-- Connection pooling built-in (default: 10 connections). Backends used on the same thread with the
+- Connection pooling built-in (default: 32 connections). The sync client speaks HTTP/1.1, one request
+  per connection, so threads sharing a backend never share a connection: a thread pool, and every
+  async-decorator L2 operation, which runs on the default executor's threads (at most 32). Over HTTP/2
+  those threads would multiplex one connection, which httpx's sync HTTP/2 support does not make
+  thread-safe, and 1–4% of operations failed as cache misses. Each extra connection costs one TCP and
+  TLS handshake on first use, then stays pooled. A request that finds every connection in use waits for
+  one to free, and fails if it gets none within the request timeout. The waits are not served in
+  order (a thread starting a new request can take a freed connection first), so with more threads than
+  connections a few requests wait many round trips: keep `connection_pool_size` at least as large as
+  the number of threads that share one backend. The async client keeps HTTP/2: one event loop drives
+  it, so its requests share one connection safely. Backends used on the same thread with the
   same key, URL, timeout and pool size share one pool while any of them is alive. The sync pool is closed
   when the last one is released; the async pool is not, and Python reclaims its sockets with a
   `ResourceWarning` each. Create one backend per key and reuse it
@@ -207,8 +216,8 @@ CACHEKIT_TIMEOUT=5.0                  # Optional — request timeout in seconds
   (about 25–30 ms from a client entering at MEL). Each pooled connection sends TCP keepalive probes
   after 60 s idle (every 10 s, 3 probes), which keeps NAT gateway mappings alive (AWS NAT Gateway drops
   idle flows at 350 s, Azure at 4 min). If a network path does die, the probes find it in about 90 s, and
-  the next request no longer waits out the 5 s timeout: over HTTP/1.1 it reconnects, and over HTTP/2 it
-  fails at once and that call is a cache miss. Probes cannot run while a process is suspended (a frozen
+  the next request no longer waits out the 5 s timeout: a sync call (HTTP/1.1) reconnects, and an async
+  call (HTTP/2) fails at once and is a cache miss. Probes cannot run while a process is suspended (a frozen
   serverless runtime, a stopped container), so the first request after such a pause can still wait out
   the timeout on a dead connection and miss. With any proxy setting in the
   environment (`HTTPS_PROXY`, `ALL_PROXY`, `NO_PROXY` and the rest), requests go through httpx's own

docs/configuration.md

@@ -131,9 +131,9 @@ CACHEKIT_API_URL=https://api.cachekit.io
 # Optional: Request timeout in seconds (default: 5.0, must be > 0)
 CACHEKIT_TIMEOUT=5.0
 
-# Optional: HTTP connection pool size (default: 10, must be > 0)
+# Optional: HTTP connection pool size (default: 32, must be > 0)
 # The same variable sizes the Redis pool, whose default is 50
-CACHEKIT_CONNECTION_POOL_SIZE=10
+CACHEKIT_CONNECTION_POOL_SIZE=32
 
 # Optional: Allow custom API hostname - disables SSRF hostname allowlist (default: false)
 # Only set to true when pointing at a private test server
@@ -147,7 +147,7 @@ CACHEKIT_ALLOW_CUSTOM_HOST=false
 | `CACHEKIT_API_KEY` | `SecretStr` | — | Unless `api_key=` is passed | API key (`ck_live_...`) for authentication. Required from one source: this variable or the `api_key=` argument to `CachekitIOBackend` / `@cache.io` |
 | `CACHEKIT_API_URL` | `str` | `https://api.cachekit.io` | No | API endpoint URL (must use HTTPS) |
 | `CACHEKIT_TIMEOUT` | `float` | `5.0` | No | Per-request timeout in seconds |
-| `CACHEKIT_CONNECTION_POOL_SIZE` | `int` | `10` | No | Max HTTP connections in pool. The same variable sizes the Redis pool, whose default is 50 |
+| `CACHEKIT_CONNECTION_POOL_SIZE` | `int` | `32` | No | Max HTTP connections in pool. The sync client uses one per concurrent request (HTTP/1.1). The same variable sizes the Redis pool, whose default is 50 |
 | `CACHEKIT_ALLOW_CUSTOM_HOST` | `bool` | `false` | No | Disable hostname allowlist (testing only) |
 
 **Security notes:**
@@ -629,11 +629,11 @@ export CACHEKIT_ARROW_COMPRESSION=zstd
 
 ### Connection Pooling
 
-One variable sizes the pool of whichever backend is in use: Redis defaults to 50 connections, CachekitIO to 10. A Redis operation that finds every connection in use waits up to the socket timeout for one, then fails as a cache miss. That timeout is `CACHEKIT_SOCKET_TIMEOUT`, unless the Redis URL sets `?socket_timeout=`, which wins.
+One variable sizes the pool of whichever backend is in use: Redis defaults to 50 connections, CachekitIO to 32. A Redis operation that finds every connection in use waits up to the socket timeout for one, then fails as a cache miss. That timeout is `CACHEKIT_SOCKET_TIMEOUT`, unless the Redis URL sets `?socket_timeout=`, which wins. A CachekitIO sync request holds a connection for its whole round trip (HTTP/1.1), so the pool caps how many run at once: size it to at least the number of threads that share one backend ([details](backends/cachekitio.md#characteristics)).
 
 ```bash
 # Tune connection pool size based on concurrency
-export CACHEKIT_CONNECTION_POOL_SIZE=100  # Defaults: Redis 50, CachekitIO 10
+export CACHEKIT_CONNECTION_POOL_SIZE=100  # Defaults: Redis 50, CachekitIO 32
 
 # Higher for:
 # - Many concurrent requests

docs/data-flow-architecture.md

@@ -112,7 +112,7 @@ cachekit uses a hybrid Python-Rust architecture to provide production caching wi
 │                                                                             │
 │  Backend types:                                                             │
 │  • RedisBackend: Distributed Redis storage (default)                        │
-│  • CachekitIOBackend: Managed cloud storage via HTTP/2 (api.cachekit.io)    │
+│  • CachekitIOBackend: Managed cloud storage via HTTPS (api.cachekit.io)     │
 │  • Future: DynamoDBBackend, etc.                                            │
 │                                                                             │
 │  RedisBackend internal flow:                                                │
@@ -448,7 +448,7 @@ def _blake2b_pickle_hash(args, kwargs):
 ┌─────────────────────────────────────────┐
 │ L2 Backend (Pluggable Storage Layer)    │
 │ • RedisBackend (default)                │
-│ • CachekitIOBackend (HTTP/2, managed)   │
+│ • CachekitIOBackend (HTTPS, managed)    │
 │ • Network call (~1-2ms latency)         │
 │ • Bytes-based protocol (BaseBackend)    │
 │ • Persistent across process restarts    │

docs/free-threading.md

@@ -225,12 +225,19 @@ not a free-threaded build, or if `--gil-python` is one. It refuses an existing
 that repeats an arm's repetition. The summary also drops a process whose GIL
 state changed while the cells ran.
 
-Today the CachekitIO cell is tainted under no-GIL. Threads that share one
-backend share one HTTP/2 connection, and httpcore's sync HTTP/2 send path
-allocates stream ids and HPACK-encodes headers outside its locks
-([encode/httpcore#1118](https://github.com/encode/httpcore/pull/1118)). The
-peer then rejects the connection and every request in flight on it fails. The
-race is rare under the GIL and frequent without it.
+The CachekitIO cell can still be tainted under no-GIL, far more rarely than
+before. The sync client is HTTP/1.1, so threads that share one backend no longer
+share one HTTP/2 connection, whose send path allocates stream ids and
+HPACK-encodes headers outside httpcore's locks
+([encode/httpcore#1118](https://github.com/encode/httpcore/pull/1118)): with 8
+threads on 3.14t that failed 23–32% of raw httpx requests, and 1–4% with the
+GIL. httpcore's HTTP/1.1 pool has a smaller race of its own without the GIL:
+`has_expired()` reads a connection's `_expire_at` twice while a thread starting
+a request on it sets it to `None`, and the comparison raises `TypeError`. That
+failed 0.15–0.23% of the same requests, and none with the GIL. There is no
+upstream fix yet. `tests/unit/backends/test_cachekitio_thread_share.py`
+expects zero failures with the GIL and marks the no-GIL case as an expected
+failure.
 
 ## Deferred: declared support + free-threaded wheels
 

src/cachekit/backends/cachekitio/backend.py

@@ -76,7 +76,7 @@
 _MAX_RETRY_AFTER_S = 2
 
 # The SaaS has no bulk delete, so whole-function invalidation sends this many DELETEs at once over
-# the sync client's HTTP/2 connection (LAB-7070). Each still takes its own limiter verdict.
+# the sync client's pool, one HTTP/1.1 connection each (LAB-7070). Each still takes its own limiter verdict.
 _DELETE_FANOUT = 16
 # A longer rate-limit hint than this is treated as a deny, not waited out. The tenant limiter's
 # window is 60 s, so a real hint is far below it; the cap only bounds the parse.

src/cachekit/backends/cachekitio/client.py

@@ -202,7 +202,7 @@ def _client_key(config: CachekitIOBackendConfig) -> _ClientKey:
 
 
 def _pin_hpack_logger() -> None:
-    # hpack (httpx's HTTP/2 header encoder) logs every header block it encodes at DEBUG, and that block
+    # hpack (the async client's HTTP/2 header encoder) logs every header block it encodes at DEBUG, and that block
     # decodes back to the Authorization bearer key and X-CacheKit-Lock-Id (CWE-532). A root logger at
     # DEBUG would publish the key, so hold hpack at INFO while its level is unset. A level the application
     # sets, before or after a client is built, wins: setting DEBUG is an explicit opt-in (SECURITY.md).
@@ -248,11 +248,17 @@ def _client_kwargs(
         max_keepalive_connections=config.connection_pool_size,
         keepalive_expiry=_KEEPALIVE_EXPIRY if probes else _KEEPALIVE_EXPIRY_NO_PROBES,
     )
-    mounts = {"all://": transport_cls(http2=True, limits=limits, socket_options=_KEEPALIVE_SOCKET_OPTIONS)} if probes else None
+    # The sync client is HTTP/1.1. Every thread that calls a backend sends on its one sync client, and over HTTP/2
+    # they would share one connection, which httpcore's sync HTTP/2 path does not lock (encode/httpcore#1118): a
+    # ReadError or RemoteProtocolError fails every request in flight on it, and the cache reads a miss (LAB-7062).
+    # HTTP/1.1 gives each concurrent request its own pooled connection, and h11 costs less CPU per request than h2.
+    # The async client stays on HTTP/2: one event loop drives it.
+    http2 = transport_cls is httpx.AsyncHTTPTransport
+    mounts = {"all://": transport_cls(http2=http2, limits=limits, socket_options=_KEEPALIVE_SOCKET_OPTIONS)} if probes else None
     return {
         "base_url": config.api_url,
         "timeout": config.timeout,
-        "http2": True,
+        "http2": http2,
         "limits": limits,
         "mounts": mounts,
         "headers": {

src/cachekit/backends/cachekitio/config.py

@@ -124,8 +124,10 @@ class CachekitIOBackendConfig(BaseBackendConfig):
         gt=0,
         description="Request timeout in seconds",
     )
+    # 32 = the most threads the default executor runs (min(32, cpu + 4)). The sync client is HTTP/1.1, one request
+    # per connection, so async-decorator L2 ops (asyncio.to_thread) never wait on the pool for a connection.
     connection_pool_size: int = Field(
-        default=10,
+        default=32,
         gt=0,
         description="Maximum HTTP connections in pool",
     )

tests/unit/backends/conftest.py

@@ -0,0 +1,39 @@
+"""Fixtures shared by the CachekitIO backend tests that need a real TLS peer."""
+
+from __future__ import annotations
+
+import subprocess
+import sys
+from collections.abc import Iterator
+from pathlib import Path
+
+import pytest
+
+_FAKE = Path(__file__).parents[2] / "performance" / "loopback_saas.py"
+
+
+@pytest.fixture(scope="module")
+def fake_saas(tmp_path_factory: pytest.TempPathFactory) -> Iterator[tuple[int, Path]]:
+    """tests/performance/loopback_saas.py on a free loopback port, HTTP/1.1 and HTTP/2 by ALPN: (port, CA cert).
+
+    Needs ``openssl`` on PATH; a module using it skips without one.
+    """
+    tmp = tmp_path_factory.mktemp("loopback-tls")
+    cert, key = tmp / "cert.pem", tmp / "key.pem"
+    subprocess.run(  # noqa: S603 (trusted: literal openssl argv)
+        ["openssl", "req", "-x509", "-newkey", "ec", "-pkeyopt", "ec_paramgen_curve:prime256v1", "-nodes", "-days", "1"]
+        + ["-subj", "/CN=127.0.0.1", "-addext", "subjectAltName=IP:127.0.0.1", "-keyout", str(key), "-out", str(cert)],
+        check=True,
+        capture_output=True,
+    )
+    proc = subprocess.Popen(  # noqa: S603 (trusted: this interpreter and a repo script)
+        [sys.executable, str(_FAKE), "0", str(cert), str(key), "2"], stdout=subprocess.PIPE, text=True
+    )
+    assert proc.stdout is not None
+    line = proc.stdout.readline().split()
+    if line[:1] != ["ready"]:
+        proc.kill()
+        pytest.fail(f"loopback fake failed to start (exit {proc.wait()})")
+    yield int(line[1]), cert
+    proc.kill()
+    proc.wait()

tests/unit/backends/test_cachekitio_fork.py

@@ -18,12 +18,11 @@
 import shutil
 import signal
 import ssl
-import subprocess
 import sys
 import threading
 import traceback
 import uuid
-from collections.abc import Callable, Iterator
+from collections.abc import Callable
 from pathlib import Path
 from typing import Any
 
@@ -41,33 +40,9 @@
 ]
 
 _REQUESTS = 5
-_FAKE = Path(__file__).parents[2] / "performance" / "loopback_saas.py"
 _PROTOCOLS = [pytest.param(False, id="h1"), pytest.param(True, id="h2")]
 
 
-@pytest.fixture(scope="module")
-def fake_saas(tmp_path_factory: pytest.TempPathFactory) -> Iterator[tuple[int, Path]]:
-    tmp = tmp_path_factory.mktemp("fork-tls")
-    cert, key = tmp / "cert.pem", tmp / "key.pem"
-    subprocess.run(  # noqa: S603 (trusted: literal openssl argv)
-        ["openssl", "req", "-x509", "-newkey", "ec", "-pkeyopt", "ec_paramgen_curve:prime256v1", "-nodes", "-days", "1"]
-        + ["-subj", "/CN=127.0.0.1", "-addext", "subjectAltName=IP:127.0.0.1", "-keyout", str(key), "-out", str(cert)],
-        check=True,
-        capture_output=True,
-    )
-    proc = subprocess.Popen(  # noqa: S603 (trusted: this interpreter and a repo script)
-        [sys.executable, str(_FAKE), "0", str(cert), str(key), "2"], stdout=subprocess.PIPE, text=True
-    )
-    assert proc.stdout is not None
-    line = proc.stdout.readline().split()
-    if line[:1] != ["ready"]:
-        proc.kill()
-        pytest.fail(f"loopback fake failed to start (exit {proc.wait()})")
-    yield int(line[1]), cert
-    proc.kill()
-    proc.wait()
-
-
 def _backend(monkeypatch: pytest.MonkeyPatch, fake_saas: tuple[int, Path], http2: bool) -> CachekitIOBackend:
     # Config validation (rightly) refuses a loopback URL, so the client kwargs are redirected instead; the
     # backend, its lease and httpx's real connection pool run unchanged. A long keepalive keeps the parent's