Skip to content

fix: encode binary parameters with unhex() instead of UTF-8 decoding - #82

Merged
hantmac merged 1 commit into
databendlabs:mainfrom
dwaddle:fix/binary-param-unhex
Oct 19, 2026
Merged

hantmac merged 1 commit into
databendlabs:mainfrom
dwaddle:fix/binary-param-unhex

Conversation

@dwaddle

@dwaddle dwaddle commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

Fixes #80

Problem

ParamEscaper.escape_item previously fell through to escape_string when given bytes parameters (such as LargeBinary / BINARY column binds). escape_string attempted item.decode("utf-8"), causing any non-UTF-8 byte sequence (e.g. b'\x00\xff', b'\x80', b'7\xe7\x9f') to raise UnicodeDecodeError.

Solution

  • Add ParamEscaper.escape_binary and handle (bytes, bytearray, memoryview) in ParamEscaper.escape_item by emitting unhex('<hex>').
  • Add unit tests in tests/unit/test_param_escaper.py covering arbitrary byte sequences (both valid and invalid UTF-8), bytearray, memoryview, and Cursor.mogrify.

Backward Compatibility

100% backward compatible; DatabendDialect uses supports_unicode_binds = True so string binds are passed as str, while bytes binds for BINARY / LargeBinary columns now reliably bind regardless of byte content.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-17T14:01:57.066589Z 654ab98 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@hantmac hantmac left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the parameter-dispatch path, SQLAlchemy binary bind processors, and Cursor.execute/executemany. No blocking findings. Hex encoding is byte-preserving, avoids UTF-8 decoding, and limits the interpolated payload to hex digits. The new bytes/bytearray/memoryview branch does not affect str parameters.

Validation on Python 3.11 with databend-driver 0.34.2:

  • All 11 added tests pass on SQLAlchemy 1.4.54 and 2.0.55.
  • Extra deterministic checks pass for 777 bytes-like cases per version, including empty data, all byte values, quote/backslash/percent/NUL bytes, execute, executemany, and LargeBinary/BINARY bind processing.
  • Combined with #76: 20 unit tests pass; the two failures (test_view_names and test_columns) also reproduce on unchanged main and are pre-existing.

I approved the pending fork workflows. Package build is green. The fresh CI TEST run fails before pytest: the MinIO download returns HTTP 410, /usr/bin/minio contains the error page, and Databend cannot start its S3 backend (StorageUnavailable / health check reset): https://github.com/databendlabs/databend-sqlalchemy/actions/runs/39038774957 . This is an infrastructure failure, not a binary-parameter test failure. A live-server binary round-trip was not validated in this review; recommend adding that coverage as a follow-up.

Approving and merging this targeted fix. Please address the MinIO/bootstrap CI breakage and stale unit-test expectations separately.

@hantmac
hantmac merged commit 04c2b26 into databendlabs:main Oct 19, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LargeBinary binds or raises depending on whether the value happens to be valid UTF-8

2 participants