Repository navigation
fix: encode binary parameters with unhex() instead of UTF-8 decoding - #82
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
hantmac
left a comment
There was a problem hiding this comment.
Reviewed the parameter-dispatch path, SQLAlchemy binary bind processors, and Cursor.execute/executemany. No blocking findings. Hex encoding is byte-preserving, avoids UTF-8 decoding, and limits the interpolated payload to hex digits. The new bytes/bytearray/memoryview branch does not affect str parameters.
Validation on Python 3.11 with databend-driver 0.34.2:
- All 11 added tests pass on SQLAlchemy 1.4.54 and 2.0.55.
- Extra deterministic checks pass for 777 bytes-like cases per version, including empty data, all byte values, quote/backslash/percent/NUL bytes, execute, executemany, and LargeBinary/BINARY bind processing.
- Combined with #76: 20 unit tests pass; the two failures (test_view_names and test_columns) also reproduce on unchanged main and are pre-existing.
I approved the pending fork workflows. Package build is green. The fresh CI TEST run fails before pytest: the MinIO download returns HTTP 410, /usr/bin/minio contains the error page, and Databend cannot start its S3 backend (StorageUnavailable / health check reset): https://github.com/databendlabs/databend-sqlalchemy/actions/runs/39038774957 . This is an infrastructure failure, not a binary-parameter test failure. A live-server binary round-trip was not validated in this review; recommend adding that coverage as a follow-up.
Approving and merging this targeted fix. Please address the MinIO/bootstrap CI breakage and stale unit-test expectations separately.
Fixes #80
Problem
ParamEscaper.escape_itempreviously fell through toescape_stringwhen givenbytesparameters (such asLargeBinary/BINARYcolumn binds).escape_stringattempteditem.decode("utf-8"), causing any non-UTF-8 byte sequence (e.g.b'\x00\xff',b'\x80',b'7\xe7\x9f') to raiseUnicodeDecodeError.Solution
ParamEscaper.escape_binaryand handle(bytes, bytearray, memoryview)inParamEscaper.escape_itemby emittingunhex('<hex>').tests/unit/test_param_escaper.pycovering arbitrary byte sequences (both valid and invalid UTF-8),bytearray,memoryview, andCursor.mogrify.Backward Compatibility
100% backward compatible;
DatabendDialectusessupports_unicode_binds = Trueso string binds are passed asstr, whilebytesbinds forBINARY/LargeBinarycolumns now reliably bind regardless of byte content.