Buffer Allocator

[gatesn]

Problem or motivation

Array operations, canonicalization, and compute functions can allocate memory through the vortex-buffer crate. Today this memory uses the system allocator.

This makes Vortex memory invisible to the host engine. The host engine may need to track, limit, or account for that memory.

This FR does not cover back-pressure or failed allocation. It should provide a way for the host engine to intercept and manage all Vortex buffer allocations.

Proposed solution

RFC: vortex-data/rfcs#65

A BufferAllocator adapter can connect Vortex to memory management primitives from the host engine. For example:

• A DataFusion adapter can reserve bytes from its memory pool before allocation, then release the reservation when the buffer is dropped.
• A DuckDB adapter can allocate or account for buffers through DuckDB memory management primitives.
• Other integrations can wrap the equivalent allocation or accounting API from their host engine.

The allocator is carried through I/O, builders, and execution. This keeps Vortex buffer memory visible to the host engine.

Additional context

PRs

• feat(buffer): add allocator-backed storage #9668 - provide xxx_in(..., BufferAllocator) functions in vortex-buffer
• refactor(array): use buffer allocator in execution context #9669 - carry BufferAllocator through ExecutionCtx
• refactor(array): propagate allocators through builders #9670 - pass allocator into array builders
• refactor(array): allocate execution outputs through context #9671 - execution uses ctx.allocator()
• ci: reject implicit engine buffer allocation #9672 - CI check to reject use of system allocator