Skip to content

Decode into the remaining buffer after incomplete_utf8 bytes in console read - #159716

Open
Joel-Wwalker wants to merge 1 commit into
rust-lang:mainfrom
Joel-Wwalker:142847-stdin-incomplete-utf8
Open

Joel-Wwalker wants to merge 1 commit into
rust-lang:mainfrom
Joel-Wwalker:142847-stdin-incomplete-utf8

Conversation

@Joel-Wwalker

@Joel-Wwalker Joel-Wwalker commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

When reading from a Windows console, Stdin::read first copies any leftover bytes of a partially read code point (incomplete_utf8) into the start of buf. The large-buffer path then decodes into buf starting at position 0, overwriting those bytes, and returns bytes_copied + value, so the clobbered bytes still count toward the length. The result is corrupted output with a stray byte at the end. The small-buffer path already decodes into &mut buf[bytes_copied..]; this change makes the large path do the same. @Ramla-I diagnosed the root cause in #142847 (comment).

I asked in the issue last week whether a reader-side fix was welcome alongside #142872 and figured I'd open it since it was sitting ready. Verified with a pseudoconsole feeding real console input through read_to_end (the report's repro) across 13 input alignments: 7 corrupted before the change, including the exact string from the issue (ж loses its second byte and a stray 0 gets appended), all 13 clean after. Console input can't be driven from CI, so there is no test.

#142872 keeps read_to_end from handing the reader a too-small buffer and is complementary: even with it, this path stays reachable through other callers, e.g. read_vectored with a 1-byte first buffer.

Fixes #142847

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Jul 22, 2026
@rustbot

rustbot commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

  • Owners of files modified in this PR: @ChrisDenton, libs
  • @ChrisDenton, libs expanded to 13 candidates
  • Random selection from 7 candidates

@Mark-Simulacrum

Copy link
Copy Markdown
Member

r? @the8472 / @ChrisDenton

@rustbot rustbot assigned the8472 and unassigned Mark-Simulacrum Aug 8, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

io::stdin().read_to_end() drops a byte on certain Unicode input (Windows only)

4 participants