dynamic_pool_allocator

256-Byte Hardware-Aligned User-Space Memory Pool for C++17 LLM Inference Runtimes

Engineered by Ayushmann (@dev-erebus / Erebus007) — Systems & ML Runtime Engineering

---

Architectural Motivation • Memory Pipeline & Invariants • Integration API • Verification Suite • Citation & Authorship

---

1. Architectural Motivation (Why This Exists)

During token-by-token autoregressive decoding in local inference engines like llama.cpp, scratchpad buffers, intermediate activations, and KV-cache views undergo continuous resizing. Relying on raw OS-level malloc/free or driver allocations inside the hot loop introduces two severe hardware bottlenecks:

1. Kernel-Space Page-Table Thrashing & Heap Fragmentation: Minor sequence-length fluctuations ($N \to N+1$ tokens) force the OS virtual memory manager to repeatedly unmap and remap physical pages, spiking tail latency.
2. Unaligned SIMD & DMA Transfers: Standard heap allocators only guarantee 8- or 16-byte alignment. Vectorised AVX-512 loads and zero-copy GPU/ROCm/HIP DMA transfers require strict 256-byte cache-line boundaries (uintptr_t(ptr) % 256 == 0) to avoid split-load penalties.

dynamic_pool_allocator is a zero-dependency, header-only C++17 runtime memory engine that intercepts freed tensor buffers in user space. By combining +12.5% amortized geometric headroom with smallest-first budget eviction, it absorbs token-to-token growth without touching the OS kernel on cache hits while strictly honouring hard VRAM/RAM ceilings.

  Caller Request: alloc(requested_size, &actual_size)
                         │
                         ▼
      ┌──────────────────────────────────────┐
      │  requested_size > MAX_REQUEST_SIZE?  │──[YES]──► return nullptr (Overflow Guard)
      └──────────────────┬───────────────────┘
                        [NO]
                         ▼
      ┌──────────────────────────────────────┐
      │  PHASE 1: Best-Fit Pool Scan (N=256) │
      │  argmin_i (pool_[i].size >= req)     │──[HIT]──► Detach Slot & Return (0 Syscalls)
      └──────────────────┬───────────────────┘
                       [MISS]
                         ▼
      ┌──────────────────────────────────────┐
      │  PHASE 2: +12.5% Growth & Bit-Mask   │
      │  grown = req + (req >> 3)            │
      │  new_size = (grown + 255) & ~255     │
      └──────────────────┬───────────────────┘
                         ▼
      ┌──────────────────────────────────────┐
      │  OS Aligned Alloc (256B Boundary)    │◄──┐
      │  _aligned_malloc / std::aligned_alloc│   │ (OOM Retry Loop:
      └──────────────────┬───────────────────┘   │  evict_smallest() frees
                         │                       │  smallest/LRU cached block)
                         └───[ptr == nullptr?]───┘

---

2. Memory Pipeline & Core Invariants

PHASE 1: Bit-Mask Hardware Alignment (ALIGNMENT = 256)

Every address returned by the allocator satisfies the hardware invariant:

$$\text{reinterpret_cast}\langle\text{uintptr_t}\rangle(\text{ptr}) \equiv 0 \pmod{256}$$

Compile-time power-of-two enforcement is locked in via static_assert((ALIGNMENT & (ALIGNMENT - 1)) == 0). Instead of expensive integer division on the allocation path, rounding is resolved via bitwise AND-NOT masking:

static constexpr size_t align_up(size_t size, size_t alignment) noexcept {
    return (size + alignment - 1) & ~(alignment - 1);
}

PHASE 2: +12.5% Amortized Headroom (compute_alloc_size)

If an inference layer requests 200 bytes at step $t$ and 220 bytes at step $t+1$, an exact-size allocator suffers a cache miss on every step. Conversely, standard $2\times$ geometric doubling wastes up to $50%$ of precious edge RAM. compute_alloc_size applies a 12.5% growth factor via a single bitwise right-shift (requested_size >> 3) prior to 256-byte rounding:

$$\text{grown_size} = \text{requested_size} + \left\lfloor \frac{\text{requested_size}}{8} \right\rfloor, \quad \text{actual_size} = \text{align_up}\big(\max(\text{grown_size}, 1),, 256\big)$$

Input Condition	Mathematical Resolution	Output (*actual_size)	Architectural Purpose
requested_size = 0	grown_size != 0 ? grown_size : 1	256 B ($1 \times \text{block}$)	Prevents zero-byte UB while guaranteeing a valid aligned pointer
requested_size = 200	$200 + (200 \gg 3) = 225 \to 256$	256 B ($1 \times \text{block}$)	Absorbs up to $+28%$ growth within the initial 256B brick
requested_size = 256	$256 + (256 \gg 3) = 288 \to 512$	512 B ($2 \times \text{blocks}$)	Pre-allocates next cache line to prevent immediate step-$t+1$ thrashing
requested_size = 1024	$1024 + (1024 \gg 3) = 1152 \to 1280$	1280 B ($5 \times \text{blocks}$)	$+12.5%$ exact brick expansion without $2\times$ memory bloat
> MAX_REQUEST_SIZE	> (SIZE_MAX - 256) / 9 * 8	0 B (nullptr)	Pre-shift overflow guard; blocks integer wrap-around before bit-masking

PHASE 3: Best-Fit Lookup & Out-Parameter Accounting

Because C++ functions return a single value (void *), alloc(size_t requested_size, size_t * actual_size) writes the true rounded capacity into the caller's bucket via the actual_size pointer.

• Best-Fit Selection: Scans MAX_BUFFERS (256) slots to find the smallest cached buffer satisfying pool_[i].size >= requested_size.
• Zero-Syscall Reuse: On a hit, the slot is detached, total_cached_bytes_ is decremented by pool_[best_idx].size, and the existing pointer is returned directly to the caller.

PHASE 4: Smallest-First Budget Eviction (evict_smallest)

Contiguous multi-megabyte tensor blocks are the hardest allocations for the OS/driver to satisfy under memory fragmentation. When free_buffer(ptr, size) would push total_cached_bytes_ + size > max_cached_bytes_ (or when aligned_malloc fails and triggers the OOM retry loop), evict_smallest() purges cached buffers using a two-tier comparator:

1. Primary Key (min_size): Evicts the smallest cached buffer first, keeping large, expensive-to-allocate blocks resident in the pool.
2. Tie-Breaker (last_used_tick): Breaks size ties by evicting the oldest monotonic tick (LRU).

---

3. Quick Integration & API Contract

Drop pool_allocator.hpp directly into your project include tree:

#include "pool_allocator.hpp"

// 1. Initialise pool with a strict 64 MiB cache ceiling
dynamic_pool_allocator pool(64ULL * 1024 * 1024);

// 2. Request a tensor buffer; `actual` receives the 256B-rounded capacity
size_t actual = 0;
void * tensor_buf = pool.alloc(1000, &actual); // actual == 1280, uintptr_t(tensor_buf) % 256 == 0

// 3. Return buffer to pool after forward pass (cached in user-space, 0 OS syscalls)
pool.free_buffer(tensor_buf, actual);

Strict Runtime Contracts:

• size passed to free_buffer(ptr, size) must be the *actual_size returned by alloc().
• Copy construction and copy assignment are explicitly deleted (= delete) to prevent double-free corruption across pool instances.
• Designed for single-thread/per-device stream ownership (zero mutex contention in the hot path).

---

4. Build & Verification Matrix

The repository stays deliberately minimal—tracking only the header engine and the deterministic 4-phase verification suite:

POOL_ALLOCATION/
├── pool_allocator.hpp   # Header-only 256B aligned dynamic memory pool engine
├── pool_allocator.cpp   # 4-Phase deterministic verification & boundary test suite
└── README.md            # Architectural specification & invariants

Verification Coverage (pool_allocator.cpp)

Test Phase	Function	Verified Invariants
Phase 1	test_alignment_and_growth()	Bit-mask rounding boundaries (0, 1, 256, 257 $\to$ 512)
Phase 2	test_compute_alloc_size()	$+12.5%$ right-shift growth math and zero-size safety fallback
Phase 3	test_live_alloc_and_caching()	Live OS allocation, reinterpret_cast<uintptr_t>(ptr) % 256 == 0, 0xAA memset poison check, cache meter accounting
Phase 4	test_reuse_and_eviction()	Exact pointer address reuse (p2 == p1), smallest-first budget eviction under 1024 B cap, MAX_REQUEST_SIZE + 1 overflow rejection

Compile & Execute

Linux / WSL2 / macOS (g++ or clang++ with AddressSanitizer & UBSan):

g++ -std=c++17 -O2 -Wall -Wextra -Werror -fsanitize=address,undefined pool_allocator.cpp -o test_pool
./test_pool

Windows (MSVC Developer PowerShell):

cl /std:c++17 /O2 /W4 pool_allocator.cpp /Fe:test_pool.exe
.\test_pool.exe

Expected Output:

[PASS] All pool allocator tests succeeded.

---

5. Authorship & Attribution

Architected and implemented by Ayushmann (Erebus).

• GitHub: @dev-erebus
• Hugging Face: Erebus007
• Systems Engineering Breakdowns: Solutions.Inc (@nah.justsolving)

@software{ayushmann2026dynamicpool,
  author = {Ayushmann},
  title  = {dynamic_pool_allocator: 256-Byte Hardware-Aligned Memory Pool for C++17 Inference Runtimes},
  year   = {2026},
  url    = {https://github.com/dev-erebus}
}