Skip to content

Copy transient HTTP and handshake buffers before queuing TCP writes - #510

Merged
mathieucarbou merged 1 commit into
ESP32Async:mainfrom
labodj:codex/fix-response-buffer-lifetime
Nov 11, 2026
Merged

mathieucarbou merged 1 commit into
ESP32Async:mainfrom
labodj:codex/fix-response-buffer-lifetime

Conversation

@labodj

@labodj labodj commented Nov 10, 2026

Copy link
Copy Markdown

ESP8266's ESPAsyncTCP defaults to borrowing the buffers passed to add() and write(). Several response paths pass storage that is cleared, reused or destroyed before TCP has finished reading it: assembled HTTP headers, basic and buffered response bodies, and the local strings used for WebSocket and SSE handshakes. A delayed TCP read can therefore access freed memory or transmit overwritten data.

Request ASYNC_WRITE_FLAG_COPY explicitly at those six call sites. TCP then owns the queued bytes when the response clears its header, releases a body buffer or returns from a handshake callback. Transports that already copy by default retain that behavior.

The regression test executes the actual production methods with a TCP peer that delays reading until after the temporary buffer or response has been destroyed. It covers basic headers, basic bodies, abstract-response headers, abstract-response bodies, WebSocket handshakes and SSE handshakes. All six scenarios report heap-use-after-free under AddressSanitizer against unchanged main with borrowing enabled; all six pass with this fix. The same scenarios also pass when the transport copies by default.

Native tests pass with GCC and Clang under AddressSanitizer and UBSan. Run them with:

python3 -m unittest discover -s tests -p 'test_response_*.py' -v
CXX=clang++ python3 -m unittest discover -s tests -p 'test_response_*.py' -v

The included workflow runs both compilers. These tests exercise buffer ownership deterministically; they do not emulate lwIP or prove retransmission behavior on every target.

On borrowing transports, these writes now allocate a TCP-owned copy of the queued bytes.

Build verification

Local builds passed for:

Target Board Example
ESP8266 / Arduino 3.1.2 d1_mini PerfTests
ESP32 / Arduino 2.0.17 esp32dev PerfTests
ESP32 / Arduino 3.3.11 esp32dev PerfTests
ESP32-C3 (RISC-V) / Arduino 3.3.11 esp32-c3-devkitm-1 PerfTests
RP2040 / Arduino-Pico 4.4.4 rpipicow PerfTests
RP2350 / Arduino-Pico 4.4.4 rpipico2w PerfTests
BK7231N / LibreTiny 1.13.0 generic-bk7231n-qfn32-tuya AsyncResponseStream
RTL8710BN / LibreTiny 1.13.0 generic-rtl8710bn-2mb-788k AsyncResponseStream

ESP32 Arduino 3.3.11 was built from the official git tag with the matching IDF b774170f libraries. LibreTiny was built in an isolated Python 3.12 / PlatformIO 6.1.19 environment. These are compilation checks; no boards were flashed.

@mathieucarbou mathieucarbou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@labodj : we won't accept an AI-generated PR fixing a maybe issue (because it does not take into account how lwip / async_tcp works behind).
Please get rid of all this AI crap, and work on a real reproductible test case like we have in our examples that can prove the issue is real and reproductible with an ESP32.

@mathieucarbou

Copy link
Copy Markdown
Member

@willmmiles : I will let you verify - this relates to how lwip works so you'll be better at answering this PR ;-) Thanks!

@labodj

labodj commented Nov 11, 2026

Copy link
Copy Markdown
Author

Thanks for pointing out the lwIP behavior.
I tested buffer ownership on ESP32 with Arduino cores 2.x and 3.3.11, and on the ESP8266 configuration that motivated this change.
In each test, I overwrote the source buffer immediately after add().
The peer still received the original bytes with and without ASYNC_WRITE_FLAG_COPY, including under packet loss on ESP32.
I cannot reproduce the claimed lifetime issue on these configurations, so I’m closing this PR.
Sorry for submitting it without verifying the transport behavior first.

@labodj labodj closed this Nov 11, 2026
@mathieucarbou

Copy link
Copy Markdown
Member

tested buffer ownership on ESP32 with Arduino cores 2.x and 3.3.11, and on the ESP8266 configuration that motivated this change.

Arduino Core 2 is not supported anymore and same goes for ESP8266 (see: #511 (comment))

@mathieucarbou

Copy link
Copy Markdown
Member

@labodj fyi @willmmiles did some versification and the issue is valid also on esp32 - we will fix it through another pr. Thanks for letting us know about it!

@mathieucarbou mathieucarbou reopened this Nov 11, 2026
@mathieucarbou
mathieucarbou marked this pull request as ready for review November 11, 2026 21:03
@mathieucarbou
mathieucarbou force-pushed the codex/fix-response-buffer-lifetime branch 2 times, most recently from ebe25a1 to 37bba2a Compare November 11, 2026 21:09
@mathieucarbou

Copy link
Copy Markdown
Member

Conclusion: ASYNC_WRITE_FLAG_COPY is used by default in AsyncTCP so this issue cannot be seen on ESP32. But the lib does not always enforced ASYNC_WRITE_FLAG_COPY at some places, so if some AsyncTCP implementations do not use ASYNC_WRITE_FLAG_COPY by default, the problem can indeed appear.
So it is safe anyway to add ASYNC_WRITE_FLAG_COPY: this is a no-op for esp32 and will make sure the enforcement to copy the data is passed to the async tcp implementation.

@mathieucarbou

Copy link
Copy Markdown
Member

@copilot resolve the merge conflicts in this pull request

@mathieucarbou
mathieucarbou force-pushed the codex/fix-response-buffer-lifetime branch from be29fe7 to 7a71f47 Compare November 11, 2026 23:37
@mathieucarbou
mathieucarbou merged commit d2c2b43 into ESP32Async:main Nov 11, 2026
10 of 33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants