Increase REALITY target TLS record buffer to 17 KiB

[fanyangCS]

Fixes XTLS/Xray-core#6356.

Summary

Increase REALITY's target TLS record buffer from 8192 bytes to 16384 bytes.

This fixes a reproducible REALITY failure when the legitimate target server returns a TLS Certificate record slightly larger than 8192 bytes. In the linked Xray-core issue, www.microsoft.com can return a Certificate record with total record length 8273 bytes when OCSP/status is included:

RecordHeader: 17 03 03 20 4c
Certificate handshake length: 0x203b
0x204c + 5 = 8273

The current REALITY code rejects it because:

size = 8192
...
if handshakeLen > size { // too long
    break f
}

The resulting user-facing/server log error is only:

REALITY: processed invalid connection ... handshake did not complete successfully

Validation

I reproduced the failure with Xray-core v26.3.27 using:

• VLESS + TCP + REALITY + Vision
• REALITY dest: www.microsoft.com:443
• serverNames: www.microsoft.com
• client fingerprint: chrome

Unpatched Xray failed locally with:

curl: (35) Recv failure: Connection reset by peer
Certificate: 8273
hs.c.isHandshakeComplete.Load(): false
handshake did not complete successfully

After this patch, the same localhost REALITY server/client setup succeeds:

https://www.google.com/generate_204 -> HTTP 204

A production deployment using the patched binary was also verified by the reporter.

Notes

TLS records can be up to around 16 KiB, so 8192 is too tight for real-world OCSP-stapled Certificate records from some large sites/CDN edges. This patch keeps the change minimal and avoids changing protocol behavior beyond allowing larger legitimate target handshake records.

[fanyangCS]

Update: changed the proposed bound from 16384 to 17 * 1024.

Reason: TLS plaintext records are commonly described as 16 KiB, but TLS 1.3 encrypted records can exceed 16 KiB slightly due to overhead (RFC allows TLSCiphertext length up to 2^14 + 256). A 17 KiB bounded buffer covers valid ~16 KiB TLS records plus overhead while still remaining small and bounded.

The original reproducer (www.microsoft.com Certificate record total length 8273) is still covered; this just makes the fix less edge-case-prone.

[jcdong98]

I met the same issue. LGTM👍

Nit: Please update the PR title accordingly to align with your change.

[papka1974]

Is this fix included in v26.7.11? As i remember it was promised to include it to next release...

[Szer]

Really interested in merging this, otherwise clients (which I can't change) are broken for dest with big certs

[corykiser]

I hit this too, so I set out to confirm the 8273 number before adding another "please merge" to the pile. I came away with two things the thread has not covered, including a reason this patch has to be written exactly the way it is.

Setup was Xray-core cd4ce97 (v26.7.28) with REALITY 9234c77, which is the commit Xray-core currently pins in go.mod. Go 1.27, darwin/arm64, a real xray server and client on loopback, curl through the client's SOCKS inbound.

Reproducing the original report

Unpatched, with dest set to www.microsoft.com:443:

REALITY remoteAddr: 127.0.0.1:64347	len(s2cSaved): 5544	Server Hello: 1215
REALITY remoteAddr: 127.0.0.1:64347	len(s2cSaved): 4329	Change Cipher Spec: 6
REALITY remoteAddr: 127.0.0.1:64347	len(s2cSaved): 4323	Encrypted Extensions: 41
REALITY remoteAddr: 127.0.0.1:64347	len(s2cSaved): 4282	Certificate: 8273
REALITY remoteAddr: 127.0.0.1:64347	hs.c.isHandshakeComplete.Load(): false
[Info] transport/internet/tcp: REALITY: processed invalid connection from 127.0.0.1:64347: handshake did not complete successfully

curl returns http_code=000. Same Certificate: 8273 the issue reports.

With this patch applied and nothing else changed:

REALITY remoteAddr: 127.0.0.1:64368	len(s2cSaved): 4282	Certificate: 8273
REALITY remoteAddr: 127.0.0.1:64368	len(s2cSaved): 8633	Certificate: 8273
REALITY remoteAddr: 127.0.0.1:64368	len(s2cSaved): 360	Certificate Verify: 286
REALITY remoteAddr: 127.0.0.1:64368	len(s2cSaved): 74	Finished: 74
REALITY remoteAddr: 127.0.0.1:64368	hs.handshake() err: <nil>
REALITY remoteAddr: 127.0.0.1:64368	hs.readClientFinished() err: <nil>
REALITY remoteAddr: 127.0.0.1:64368	hs.c.isHandshakeComplete.Load(): true

curl returns 204.

Why the patch has to change size itself

size gates four things in the handshake path, not one.

s2cSaved := make([]byte, 0, size)   // accumulator capacity
buf      := make([]byte, size)      // read buffer
if len(s2cSaved) > size { break }   // cumulative cap
if handshakeLen > size { break f }  // the check quoted in the issue

There is a fifth use that matters more. empty = make([]byte, size) is the padding source conn.go draws on when REALITY mimics the target's record sizes.

case typeCertificate:
    padding = hc.handshakeLen[3]
...
record = append(record, empty[:padding]...)

padding is the target's observed Certificate record length, so len(empty) has to stay at or above the largest handshakeLen the parser will accept. Bumping the single size var keeps that true for free. Anyone reviewing this and reaching for a narrower change should know that.

I built the narrower change to see what happens. Relax only the comparison to if handshakeLen > 17*1024, leave size = 8192:

RESULT dest=www.microsoft.com binary=naive http_code=000
REALITY remoteAddr: 127.0.0.1:64397	len(s2cSaved): 4282	Certificate: 8273
REALITY remoteAddr: 127.0.0.1:64397	hs.c.isHandshakeComplete.Load(): false

Still broken. len(s2cSaved) climbs to 8633 and trips the cumulative cap first. The one line in this PR is the fix, and a more surgical version of it is not.

A second failure mode, under the 8192 limit

This is the part I did not expect. azure.microsoft.com sends a Certificate record of 8179 bytes, comfortably under 8192, and fails anyway on the unpatched build:

REALITY remoteAddr: 127.0.0.1:64407	len(s2cSaved): 4323	Encrypted Extensions: 47
REALITY remoteAddr: 127.0.0.1:64407	len(s2cSaved): 4276	Certificate: 8179
REALITY remoteAddr: 127.0.0.1:64407	hs.c.isHandshakeComplete.Load(): false
[Info] REALITY: processed invalid connection ...: handshake did not complete successfully

The arithmetic is 8179 + 286 + 74 = 8539. Azure flushes CertificateVerify and Finished in the same segment as the tail of the Certificate, so the cumulative cap fires before the parser ever consumes the Certificate. It reproduced on 10 of 10 runs here. Patched, it returns 204.

So 8192 is not the number users can reason about. The usable headroom is lower than the constant suggests and it moves depending on what the target coalesces into one segment. That matters for triage, because someone testing a dest with an 8 KiB certificate can hit this and reasonably conclude they are looking at a different bug.

Probe table

I wrote a probe that replays REALITY's record classification loop against live targets and scores each one at both bounds, no server required. Source and instructions are at https://gist.github.com/corykiser/29b049d0497483dc82c555dc21b47e65

target	chain	chain bytes	OCSP	Certificate record	size=8192	size=17408
www.microsoft.com	3	5879	2341	8273	REJECT, handshakeLen=8273 > 8192	PASS
azure.microsoft.com	3	5785	2341	8179	REJECT, len(s2cSaved)=8539 > 8192	PASS
www.entrust.com	4	7741	599	7002	PASS	PASS
www.cisco.com	3	4934	1493	6480	PASS	PASS
www.bbc.com	3	4757	1429	6239	PASS	PASS
www.paypal.com	2	4882	471	5401	PASS	PASS
www.yahoo.com	2	4584	471	5103	PASS	PASS
www.bing.com	3	3888	1081	5022	PASS	PASS
www.apple.com	2	3231	1459	4738	PASS	PASS
gateway.icloud.com	3	3195	0	3240	PASS	PASS
www.google.com	3	2454	0	2821	PASS	PASS
www.cloudflare.com	4	3426	0	2809	PASS	PASS

I ran 35 targets in total and these are the representative ones. OCSP stapling is what separates the two failures from the rest. Both pair a chain of roughly 5.8 KB with a 2341 byte stapled OCSP response, and nothing else I probed came close.

End to end

Real server and client, curl https://www.google.com/generate_204 through the SOCKS inbound.

dest	unpatched	patched
www.microsoft.com	000	204
azure.microsoft.com	000	204
www.apple.com	204	204
gateway.icloud.com	204	204
www.yahoo.com	204	204
www.cisco.com	204	204

Nothing that worked before regressed.

Why 17 KiB is the right constant

REALITY already defines the ceiling in common.go:66 and enforces it at conn.go:707.

maxCiphertextTLS13 = 16384 + 256  // 16640

handshakeLen is recordHeaderLen + length, so the largest legal value is 5 + 16640 = 16645.

17 * 1024 is 17408, which covers that with 763 bytes to spare. The 16384 in the first revision of this PR falls 261 bytes short of it. Moving to 17 KiB was not caution, it was the difference between covering the protocol limit and missing it, so the revised constant is correct and about as tight as it can be.

The cost is small. empty grows once by 9216 bytes. Per connection the two handshake buffers go from 2x8192 to 2x17408, so 18 KiB more per connection, and only while the handshake goroutine is alive.

One thing that is not this bug

While probing I found that microsoft.com (the apex), www.office.com, teams.microsoft.com and login.microsoftonline.com answer a Chrome fingerprint ClientHello with a HelloRetryRequest. REALITY's parser rejects those whatever size is set to, so they are unusable as dest for a reason that predates this patch and is untouched by it. I left them out of the counts above. Flagging it only so it does not get folded into this report by mistake, and I can open a separate issue if that is useful.

Follow up

Every failure above reaches the operator as the same line.

REALITY: processed invalid connection from <addr>: handshake did not complete successfully

That opacity is most of why this one burned so much of people's time. Putting the offending length into failureReason, something like target handshake record too long: 8273 > 17408, would let the log diagnose itself. I would rather not add anything to this PR's diff while it is waiting to land. Happy to send it separately once this merges.

The patch looks right to me as written.