nvme-print: fix unintended sign extension in json_eom_printable_eye() - #3822
Merged
igaw merged 1 commit intoAug 13, 2026
Merged
Conversation
The json_eom_printable_eye() function uses @nrows and @NCols, both of type uint16_t, to compute the allocation size for the printable eye string buffer. Because uint16_t operands are promoted to int (32-bit signed) before multiplication, the expression @nrows * @NCols + @nrows + 1 is evaluated as a signed 32-bit value. When passed to malloc(), the compiler widens it to size_t (64-bit unsigned) via sign extension. If the product exceeds 0x7FFFFFFF the sign bit is set, the result sign-extends to a very large 64-bit value, and malloc() will either fail or trigger an out-of-memory condition. Cast @nrows to size_t before the multiplication so the entire expression is evaluated in 64-bit unsigned arithmetic, preventing the sign extension. Signed-off-by: Sarah Ahmed <sarah.ahmed@ibm.com>
Collaborator
|
Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The json_eom_printable_eye() function uses nrows and ncols, both of type uint16_t, to compute the allocation size for the printable eye string buffer.
Because uint16_t operands are promoted to int (32-bit signed) before multiplication, the expression nrows * ncols + nrows + 1 is evaluated as a signed 32-bit value. When passed to malloc(), the compiler widens it to size_t (64-bit unsigned) via sign extension. If the product exceeds 0x7FFFFFFF the sign bit is set, the result sign-extends to a very large 64-bit value, and malloc() will either fail or trigger an out-of-memory condition.
Cast nrows to size_t before the multiplication so the entire expression is evaluated in 64-bit unsigned arithmetic, preventing the sign extension.