gh-158316: Avoid second buffer fetch on bytearray_decode

gh-158316: Avoid second buffer fetch on bytearray_decode #303020