Repository navigation
feat: make SharedCoalescer do memory accounting for buffered batches #25904
Description
Activity
The nature of the query (many join operators, many partitions)
If it's a RSS and memory-pool limit inconcsistency, another possible trigger might be the 2X memory amplification in the buffering phase of NLJ/HJ
NLJ fix would be available in the next release
Reacted by Marko Grujichi @2010YOUY01 and @gruuya, I would like to work on this issue, if no one is working on this
Reacted by Marko Grujictake
Is your feature request related to a problem or challenge?
We ran into a case where a DataFusion process hit memory resource limits (and got OOMed) before it actually got near the set memory pool limit.
The nature of the query (many join operators, many partitions), led claude to believe a significant part of it was down to unaccounted memory in the
SharedCoalescer, which was coincidentally also discussed in the original PR #22010 (comment)Describe the solution you'd like
Thread a memory reservation down to
SharedCoalescerand grow/shrink it as the batches are pushed/drained from it.Even though an individual
SharedCoalescerby default buffers only a modest number of rows, this problem can compound otherwise, sinceSharedCoalescerare per partition and per operator.Describe alternatives you've considered
No response
Additional context
No response