Repository navigation
[Bug] S3ClientFactory caches one S3Client per bucket forever with no eviction — unbounded native memory growth with per-bucket catalogs (e.g. S3 Tables) #68508
Description
What's Wrong?
S3ClientFactory::_cache (be/src/util/s3_util.h) is a plain std::unordered_map<S3ClientConf, ..., S3ClientConfHash> with no eviction, TTL, or size cap:
auto it = _cache.find(s3_conf);
if (it != _cache.end()) { ... }
...
auto [it, _] = _cache.emplace(s3_conf, std::move(obj_client));(s3_util.cpp, S3ClientFactory::create — identical on master and branch-4.1/4.1.4.)
S3ClientConf's identity includes the bucket, so every distinct bucket a BE process ever accesses creates and permanently retains a new cached client. This is a real problem for any catalog where storage is bucket-per-table rather than bucket-per-warehouse — e.g. AWS S3 Tables, which auto-provisions one bucket per Iceberg table. A long-lived BE process that scans N distinct tables accumulates N permanently-cached Aws::S3::S3Client objects, each with its own libcurl connection pool (CA bundle parsed per pooled handle) and, when using STS/WebIdentity-style credentials, an auxiliary STS credentials-provider sub-client built via a bare default constructor — bypassing Doris's own caFile/connection settings entirely.
Confirmed via jemalloc heap profiling (prof_active) that this is genuinely live allocated memory (jemalloc allocated, not just retained/fragmented resident), invisible to Doris's own MemTracker/workload-group accounting — it shows up purely as UntrackedMemory while workload groups report ~0 the whole time. One measured instance attributed roughly 700 MiB to these allocation sites after a routine ETL window, in a steadily-climbing, never-plateauing pattern that eventually leads to the process being killed under memory pressure.
How to Reproduce
- Configure an external catalog (e.g. Iceberg REST) where table storage maps to a distinct S3 bucket per table (S3 Tables does this automatically).
- From a single long-lived BE process, scan many distinct tables over time, no restarts between them.
- Observe RSS / jemalloc
allocatedclimbing roughly linearly with the number of distinct buckets ever touched, never plateauing, independent of query concurrency or workload-group memory settings. - Workload-group memory metrics stay near zero throughout — the growth is invisible from Doris's own accounting.
What You Expected
Some bound on S3ClientFactory's cache — idle-TTL and/or LRU eviction, similar to the bounded/expiring cache (_azure_cache, capacity 256, LRU + expiry pruning) added for Azure clients in #68117. That PR explicitly left the non-Azure _cache untouched.
Version
Reproduced on the official 4.1.4 release (branch-4.1). Same unbounded _cache design confirmed present on master by source inspection.
Anything Else?
Happy to contribute a patch — fix direction: capacity-bounded LRU + idle-TTL eviction on _cache (mirroring #68117's _azure_cache); share one credentials-provider/STS client across cache entries that share the same credential config instead of one per bucket; and separately, the maxConnections fallback to 102400 in _create_s3_client when no value is supplied looks worth revisiting too.
Related but distinct: #66300 (different trigger, same "unbounded S3 connection" smell); #66997/#65416/#67056 (separate BE thread-pool leak, unrelated to this cache).
- Yes, willing to submit a PR.
This issue was drafted by an AI coding assistant (Claude) based on source-code review and a heap-profiling investigation carried out by our engineering team, and was reviewed and submitted by a human maintainer of this report.
Activity
Doris-Breakwater commentedon Sep 25, 2026
Initial maintainer analysis
Assessment: confirmed BE client-cache lifetime/design bug; keep this issue open. The issue currently has no labels or assignee. I recommend triaging it as an object-storage/BE resource-lifetime bug and considering a 4.1 backport once the fix is proven.
Verified facts
4.1.4tag,S3ClientConffull-field equality and its hash includebucket, while URI conversion always copies the URI bucket intoclient_conf.bucket(source).S3ClientFactory::create()only looks up and inserts into the process-wide singleton's_cache; there is no erase, capacity, TTL, or lifecycle reset. Therefore every distinct full configuration is retained until BE shutdown, and otherwise-identical configurations with different buckets necessarily create distinct entries.master(573c93c63b88fb68ac1d318dcd3e1672dd2361e6) and currentbranch-4.1(5e1ee245e23adedc0e9fff8b6006f067653e9155) retain the same behavior. This is not limited to the reported release.Aws::S3::S3Clientand credentials provider. Importantly,_create_s3_client()does not consumes3_conf.bucket; bucket is supplied on each object request. The bucket is construction identity for Azure's container client, but not for the non-Azure S3-compatible client. This makes bucket-sensitive identity avoidable for the reported non-Azure case.Scope corrections
masteror 4.1. Its proposed Azure-only cache is a useful implementation reference (capacity-bounded LRU plus credential-expiry pruning), but it should not be described as an already-landed Doris cache.maxConnections = 102400fallback is real, but it is a ceiling, not an eager allocation of 102400 connections. It may amplify per-client transport use under concurrency, but it is separate from the cache-cardinality/lifetime defect and should be handled independently.STSAssumeRoleWebIdentityCredentialsProvider, while explicitrole_arnAssumeRole paths construct an STS client with Doris's selected CA configuration. Currentmasterlikewise passessts_client_configto explicit AssumeRole, but still default-constructs the WebIdentity base provider. Thus the bypass concern applies to the WebIdentity-default path, not to every STS-style provider.Recommended fix direction
bucket; retain it for Azure/container-bound clients. Do this in a cache-key type or normalization step rather than changing globalS3ClientConf::operator==, which is also used byObjClientHolderreset logic. This directly collapses bucket-per-table workloads to one client when transport and credential configuration are otherwise identical.shared_ptr, so active readers remain valid. Preserve construction outside the mutex, double-check on publication, and avoid destroying an evicted/losing AWS client while holding the factory lock.Suggested tests: two non-Azure buckets with identical transport/credentials reuse one fake client; Azure buckets remain distinct; capacity and idle expiry evict the expected entry while an externally held
shared_ptrremains usable; credential/token changes still create a new identity; concurrent misses publish one retained entry; and destructor counters prove evicted clients are released.Evidence requested to quantify impact and validate the regression
Please attach, with credentials/endpoints/bucket names redacted:
jeprof/flamegraph outputs) for a known number of newly touched buckets, including resolved cumulative stacks for S3 client, curl, TLS/CA, and credential-provider allocations.create one s3 clientlog count, jemallocallocated, RSS, and DorisUntrackedMemory; include a control that repeatedly scans already-seen buckets and the drop after BE restart.aws_credentials_provider_version, provider type, whetherrole_arnis set, and whether AK/SK/token values rotate per table) and effectivemax_connections, without secret values. In particular, confirm whether the per-table configurations differ only by bucket or also by temporary credentials.These artifacts are not required to establish the unbounded-ownership bug, but they are needed to validate the claimed allocation sites, choose a safe default bound/TTL, and create a meaningful memory regression test.
Breakwater-GitHub-Analysis-Slot: slot_2220f6d6b149