You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native memory requests treat an error from Spark as a refusal #7244
Two native memory paths treat an error from Spark during a memory request as if the pool had refused it, so the task carries on instead of failing.
The multi-partition shuffle repartitioner grows its batch reservation with self.reservation.try_grow(mem_growth).is_err() and spills on any error (multi_partition.rs#L577). When a Spark-backed pool fails because another consumer's spill() threw, the repartitioner spills and keeps going, where only a ResourcesExhausted refusal should lead to a spill.
An infallible MemoryReservation::grow or resize on the Spark-backed pools goes through SparkMemory::acquire, which logs a failed call to Spark as a warning and books the whole size as overcommit (spark_memory.rs#L135-L151). The next try_grow asks Spark for that debt too, so a repeated exception does surface, but a single one is only logged.
Steps to reproduce
Run a native shuffle with more than one output partition on a Spark-backed pool where another memory consumer's spill() throws while the repartitioner asks for memory. The shuffle spills and finishes rather than failing with that exception.
Expected behavior
An error other than ResourcesExhausted fails the task, the way #6336 handles the shuffle writer's own buffer requests. For the infallible grow path, the exception should at least reach the task instead of only the log.
Additional context
#6336 fixes the shuffle writer's buffer requests and leaves these two paths as they are on main.
Activity