Send unexpected message alert for UNKNOWN_RECORD_TYPE

[rlm2002]

Description

DoProcessReplyEx() rejected records with an undefined ContentType but returned UNKNOWN_RECORD_TYPE without putting an alert on the wire. RFC 8446 section 5 and RFC 9846 section 5 both require that a TLS implementation receiving an unexpected record type MUST terminate the connection with an unexpected_message alert. RFC 5246 section 6 states the same for TLS 1.2.

Two rejection sites were affected:

1. The error mapping switch in case getRecordLayerHeader, which handled VERSION_ERROR and LENGTH_ERROR, but let UNKNOWN_RECORD_TYPE fall through to default: break.
2. The record dispatch default: arm, reached by types the whitelist admits but nothing on a TLS connection handles. dtls12_cid (25) and ack (26) pass the header check whenever WOLFSSL_DTLS_CID or WOLFSSL_DTLS13 are built, with no ssl->options.dtls guard. It is also reached with the inner ContentType recovered by removeMsgInnerPadding().

Both now send a fatal unexpected_message before returning.

Fixes F-10722

Testing

Added new unit test: test_tls_unknown_record_type_alert
./configure --enable-all && make check

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[wolfSSL-Bot]

Can one of the admins verify this patch?

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m3

• FLASH: .text +8 B (+0.0%, 125,783 B / 262,144 B, total: 48% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +64 B (+0.0%, 242,191 B / 262,144 B, total: 92% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text +64 B (+0.0%, 242,255 B / 262,144 B, total: 92% used)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[rlm2002]

Retest this please Jenkins: history lost

[night1rider]

Jenkins retest this please

[rlm2002]

Retest this please Jenkins