Skip to content

Hold the supported groups extension in one allocation - #11756

Open
Frauschi wants to merge 5 commits into
wolfSSL:masterfrom
Frauschi:tls_mem_2
Open

Frauschi wants to merge 5 commits into
wolfSSL:masterfrom
Frauschi:tls_mem_2

Conversation

@Frauschi

@Frauschi Frauschi commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

The supported groups extension used one heap node per group, on both client and server, and a TLS 1.3 server built and freed a full throwaway list on every handshake. Part of the TLS extension memory series (after #11353, #11345, #11342 and #11343; #11730 is still open).

  1. One allocation per list: the linked list becomes a flat array. The first allocation holds the larger of WOLFSSL_MAX_GROUP_COUNT (the wolfSSL_set_groups() limit) and the build's default group table, so a default or set_groups() list takes one allocation; a longer one, such as a peer's, grows with XREALLOC.
  2. One allocation per intersection: the server builds the groups it has in common with the peer in a single list sized from its own and swaps it in on success, where master allocated one node per common group.
  3. Iterate instead of a throwaway list: TLSX_NextSupportedGroup() returns this build's groups in preference order from the existing preferredGroup[] table, so TLSX_SupportedCurve_CheckPriority() and TLSX_SupportedFFDHE_Set() no longer allocate.

No public API or wire changes. An encapsulate-only ML-KEM server now ranks the hybrid groups first in its default list, as the key share ranking already did. TLSX_SupportedCurve_CheckPriority() and TLSX_SupportedFFDHE_Set() become WOLFSSL_TEST_VIS.

One in-memory TLS 1.3 handshake on macOS (arm64, static library), client and server allocations combined, counted from wolfSSL_new():

Build master this branch
default 146 allocs, peak 50043 B 126 allocs, peak 49971 B
--enable-all 477 allocs, peak 193664 B 430 allocs, peak 193304 B

@Frauschi Frauschi self-assigned this Oct 15, 2026
Copilot AI balanced review requested due to automatic review settings October 15, 2026 07:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@Frauschi
Frauschi requested a balanced review from Copilot October 15, 2026 07:38
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

MemBrowse Memory Report · Explore what changed →

gcc-arm-cortex-m3

  • FLASH: .text +144 B (+0.1%, 131,569 B / 262,144 B, total: 50% used)

gcc-arm-cortex-m4

  • FLASH: .text +348 B (+0.2%, 208,780 B / 262,144 B, total: 80% used)

gcc-arm-cortex-m4-dtls13

  • FLASH: .text +384 B (+0.2%, 195,480 B / 1,048,576 B, total: 19% used)

gcc-arm-cortex-m4-openssl-compat

  • FLASH: .rodata +16 B, .text +192 B (+0.0%, 797,260 B / 1,048,576 B, total: 76% used)

gcc-arm-cortex-m4-pq

  • FLASH: .text +256 B (+0.1%, 312,080 B / 1,048,576 B, total: 30% used)

gcc-arm-cortex-m4-rsa-only

  • FLASH: .rodata +8 B, .text +320 B (+0.1%, 342,992 B / 1,048,576 B, total: 33% used)

gcc-arm-cortex-m4-tls12

  • FLASH: .text +192 B (+0.1%, 132,369 B / 262,144 B, total: 50% used)

gcc-arm-cortex-m4-tls13

  • FLASH: .text +384 B (+0.2%, 249,122 B / 262,144 B, total: 95% used)

gcc-arm-cortex-m7

  • FLASH: .text +348 B (+0.2%, 208,780 B / 262,144 B, total: 80% used)

gcc-arm-cortex-m7-pq

  • FLASH: .text +256 B (+0.1%, 312,912 B / 1,048,576 B, total: 30% used)

gcc-arm-cortex-m7-tls13

  • FLASH: .text +320 B (+0.1%, 249,122 B / 262,144 B, total: 95% used)

zephyr-frdm_rw612

  • FLASH: log_const_area -8 B, rodata -8 B, text +1,600 B (+0.6%, 284,048 B / 67,108,864 B, total: 0% used)

No memory changes detected for:


Catch memory regressions in CI with MemBrowse. Free for open source.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Feature-dependent TLS negotiation and memory-ownership changes require human validation across supported configurations.

1 open finding

🧠 Review effort: Balanced

Comment thread tests/api/test_tls_ext.c Resolved
@Frauschi

Copy link
Copy Markdown
Member Author

Jenkins retest this please

Comment thread src/tls.c Resolved
Comment thread src/tls.c Resolved
Comment thread src/tls.c Resolved
Comment thread src/tls.c Resolved
Comment thread src/tls.c Resolved
Comment thread src/tls.c Resolved
Comment thread src/tls.c Resolved
Comment thread src/tls.c Resolved
Comment thread src/tls.c Resolved
Comment thread src/tls.c Resolved
@Frauschi
Frauschi force-pushed the tls_mem_2 branch 4 times, most recently from a67e3bb to 8aafcde Compare October 21, 2026 19:40
@Frauschi

Copy link
Copy Markdown
Member Author

Jenkins retest this please

Comment thread src/tls.c Resolved
julek-wolfssl
julek-wolfssl previously approved these changes Oct 22, 2026
@Frauschi Frauschi assigned wolfSSL-Bot and unassigned Frauschi Oct 22, 2026
@philljj philljj self-assigned this Oct 22, 2026

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #11756

Scan targets checked: wolfssl-src, wolfssl-bugs
Not reviewed (no scan target covers them): tests/api/test_tls_ext.h
Coverage: 2 of 5 in-scope changed file(s) opened by the reviewer; not opened: tests/api.c, tests/api/test_tls_ext.c, wolfssl/internal.h

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Review tier: Lite

@philljj philljj assigned Frauschi and unassigned wolfSSL-Bot Oct 30, 2026
@philljj

philljj commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

tests good, just rebase please to be safe

The supported groups extension stored its named groups as a linked list with
one node per two byte group identifier. A client advertising 16 groups made 16
allocations for 32 bytes of wire data, and a server made a second set of nodes
for the intersection with the client's list. Every insert also walked the list
to check for a duplicate.

Replace the list with a single block holding a count, a capacity and a flat
array of names. The block starts with room for WOLFSSL_MAX_GROUP_COUNT names,
the most wolfSSL_set_groups() accepts, so a handshake makes one allocation per
list; it doubles only for a peer that sends more. Inserts and lookups index the
array instead of chasing pointers, and freeing the extension is a single XFREE.
A server with its own supported groups list built a second list holding the
groups it has in common with the client, swapped that into the extension and
freed the original. Now that the groups are a flat array, the common ones are
moved to the front of the server's own list instead, so the intersection needs
no allocation.

The kept groups stay in the client's order, as before. A name that has already
been moved sits outside the range still searched, so a group repeated in the
client list is added once.

Adds a test for the resulting list and for the TLS 1.2 handshake failure when
no group is common.
Before sending EncryptedExtensions a TLS 1.3 server checked whether the client
offered the group the server prefers most. It answered that by populating a
whole temporary extension list with every group this build offers, reading the
first name out of it, and freeing the list again. The TLS 1.2 FFDHE selection
did the same to find the highest priority group both sides have. Both cost a
TLSX node, a groups block and a walk of the full preference chain on every
handshake.

Turn the preference chain into TLSX_WalkSupportedGroups(), which calls a
callback for each group in the same order. Populating an extension list is now
one caller of the walk, finding the preferred group is another that stops at
the first group, and the FFDHE selection is a third that stops at the first
group the client also has. The server builds a list only in the case that
sends it back to the client.
Walk the groups with TLSX_NextSupportedGroup(), an index into the existing
preferredGroup[] table filtered at run time, instead of a callback per group.
Populating the extension, CheckPriority() and the FFDHE selection now loop
over it directly. The FFDHE selection is back to master's shape: find the
group in the loop and set it once afterwards. The table now decides the ML-KEM
groups per side for the walk as it already did for the key share ranking,
which only differs in encapsulate-only server builds.

Build the server's intersection in a separate list instead of moving names
around inside its own one, and swap it in only on success, so an error leaves
the list as it was. The list is allocated once with room for the shorter of
the two lists plus one, where master made one allocation per common group. It
is not capped: after an HRR the second ClientHello is intersected with the
client's whole list from the first, which can be longer than
WOLFSSL_MAX_GROUP_COUNT. Only the first unsupported FFDHE codepoint is kept,
since TLSX_SupportedFFDHE_Set() only needs to see one.

Grow a live list with XREALLOC, keeping malloc and copy for builds without it,
such as FreeRTOS without OPENSSL_EXTRA. A new list still comes from XMALLOC:
the uITRON4 and uT-Kernel reallocs return NULL for a NULL pointer, and with
no realloc callback installed wolfSSL_Realloc() falls back to the native one.
The first allocation now holds the larger of WOLFSSL_MAX_GROUP_COUNT and the
build's default group table, plus the three old hybrid codepoints when
WOLFSSL_ML_KEM_USE_OLD_IDS adds them, so a default list no longer grows in
builds with many groups but no ML-KEM, where WOLFSSL_MAX_GROUP_COUNT is only
10. TLSX_SupportedCurve_Copy() makes one
allocation and copy instead of re-adding every group. Append through a typed
local rather than casting &extension->data, which GCC flags with
-Wstrict-aliasing=2. Drop two casts and some comments.

Adds tests for an intersection longer than WOLFSSL_MAX_GROUP_COUNT, for the
list surviving a failed allocation or a failed grow, for creating a list when
realloc always fails, and for which brainpool and ML-KEM groups the default list holds for
TLS 1.2, TLS 1.3 and version-flexible clients.
The comment in the server's intersection loop pointed back at the one in the
parse loop. State the RFC 7919 Section 4 requirement directly instead.
@philljj

philljj commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Retest this please.

(generic config PRB timeout or something)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants