-
Notifications
You must be signed in to change notification settings - Fork 55
FEAT: conda recipe for mssql-python (self-contained repackage of the signed wheel) #734
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
6cce9af
89544fd
1af01b3
f24b366
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # Keep shell build scripts LF so conda-build works on Linux/macOS agents, | ||
| # regardless of the checkout host's core.autocrlf setting. | ||
| *.sh text eol=lf | ||
|
|
||
| # Keep .bat scripts CRLF for Windows conda-build agents, regardless of the checkout | ||
| # host's core.autocrlf. cmd.exe tolerates LF here (no goto/labels), but pinning CRLF | ||
| # removes any doubt on odd checkouts. | ||
| *.bat text eol=crlf |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| @echo on | ||
| REM Repackage the prebuilt, signed mssql-python wheel into a conda package (offline) and | ||
| REM vendor the ODBC Driver 18 payload inside it. conda-build exports PKG_NAME / | ||
| REM PKG_VERSION / CONDA_PY; the pipeline exports WHEELS_DIR + MSSQL_ODBC_VERSION. | ||
| setlocal enabledelayedexpansion | ||
|
|
||
| set "SP=%PREFIX%\Lib\site-packages" | ||
| if not exist "%SP%" mkdir "%SP%" | ||
| set "PKG_UNDERSCORE=%PKG_NAME:-=_%" | ||
|
|
||
| REM Native leg: pip installs the matching wheel. Cross win-arm64 leg: the arm64 host | ||
| REM Python can't run on the x64 agent, so extract the cp%CONDA_PY% wheel (a zip) with tar. | ||
| "%PYTHON%" -c "import sys" >nul 2>&1 | ||
| if errorlevel 1 ( | ||
| echo Host Python "%PYTHON%" not executable here ^(non-emulated cross-build^); extracting cp%CONDA_PY% win_arm64 code wheel without Python. | ||
| set "CODE_WHL=" | ||
| for %%W in ("%WHEELS_DIR%\!PKG_UNDERSCORE!-%PKG_VERSION%-cp%CONDA_PY%-*-win_arm64.whl") do if exist "%%~fW" set "CODE_WHL=%%~fW" | ||
| if not defined CODE_WHL ( | ||
| echo ERROR: no %PKG_NAME%==%PKG_VERSION% cp%CONDA_PY% win_arm64 wheel in "%WHEELS_DIR%" | ||
| exit /b 1 | ||
| ) | ||
| echo Extracting "!CODE_WHL!" into "%SP%" | ||
| tar -xf "!CODE_WHL!" -C "%SP%" | ||
| if errorlevel 1 exit /b 1 | ||
| REM win-arm64 cross can't run the arm64 Python, so statically prove the extracted | ||
| REM binding is for THIS interpreter (the osx-arm64 twin bug shipped a cp310 .so in | ||
| REM every build); a wrong-Python .pyd would only fail at the user's import. | ||
| if not exist "%SP%\mssql_python\ddbc_bindings.cp%CONDA_PY%-*.pyd" ( | ||
| echo ERROR: extracted "!CODE_WHL!" has no mssql_python\ddbc_bindings.cp%CONDA_PY% pyd ^(wrong-Python binding^). | ||
| exit /b 1 | ||
| ) | ||
| REM The win-arm64 wheel bundles the x64 mssql_py_core (no arm64 build exists yet); an | ||
| REM x64 .pyd can't load on arm64 and fails the PE-arch assert, so strip it. Bulk copy | ||
| REM then raises a clean "not available" error (lazy import); the rest of the DBAPI works. | ||
| echo Removing x64 mssql_py_core from the win-arm64 package; bulk copy is unavailable on win-arm64 until an arm64 build ships. | ||
| if exist "%SP%\mssql_py_core" rmdir /s /q "%SP%\mssql_py_core" | ||
| if exist "%SP%\mssql_py_core.libs" rmdir /s /q "%SP%\mssql_py_core.libs" | ||
| ) else ( | ||
| "%PYTHON%" -m pip install --no-deps --no-index --find-links "%WHEELS_DIR%" %PKG_NAME%==%PKG_VERSION% -vv | ||
| if errorlevel 1 exit /b 1 | ||
| ) | ||
|
|
||
| REM Extract the py3-none-win odbc wheel into the SAME site-packages so | ||
| REM mssql_python_odbc\libs\ sits beside mssql_python\ and the loader finds the driver. | ||
| set "ODBC_WHL=" | ||
| for %%W in ("%WHEELS_DIR%\mssql_python_odbc-%MSSQL_ODBC_VERSION%-py3-none-win_*.whl") do if exist "%%~fW" set "ODBC_WHL=%%~fW" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The code wheel is matched arch-specifically ( This currently works only because of the implicit "one wheel per target Consider making this glob arch-specific to mirror the code-wheel handling (e.g. select |
||
| if not defined ODBC_WHL ( | ||
| echo ERROR: no mssql_python_odbc==%MSSQL_ODBC_VERSION% py3-none-win wheel in "%WHEELS_DIR%" | ||
| exit /b 1 | ||
| ) | ||
| echo Extracting "!ODBC_WHL!" into "%SP%" | ||
| tar -xf "!ODBC_WHL!" -C "%SP%" | ||
| if errorlevel 1 exit /b 1 | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| #!/bin/bash | ||
| # Repackage the prebuilt, signed mssql-python wheel into a conda package (offline) and | ||
| # vendor the ODBC Driver 18 payload inside it -- no separate mssql-python-odbc conda | ||
| # package. Both the code wheel and the py3-none-<plat> odbc wheel land in the SAME | ||
| # site-packages, so mssql_python_odbc/libs/ sits beside mssql_python/ and the C++ | ||
| # loader finds the driver. conda-build exports PKG_NAME / PKG_VERSION / CONDA_PY; | ||
| # WHEELS_DIR (one wheel per target) + MSSQL_ODBC_VERSION come from the pipeline. | ||
| set -euo pipefail | ||
| # Emulated aarch64 leg: $PYTHON is the aarch64 interpreter under qemu-user; point it at | ||
| # the aarch64 glibc loader so it doesn't abort. Harmless no-op on every other leg. | ||
| [ -d /usr/aarch64-linux-gnu ] && export QEMU_LD_PREFIX="${QEMU_LD_PREFIX:-/usr/aarch64-linux-gnu}" | ||
|
|
||
| odbc_ver="${MSSQL_ODBC_VERSION:?MSSQL_ODBC_VERSION not set}" | ||
|
|
||
| # Native / QEMU-emulated legs: the host Python runs, so pip installs both wheels. Cross | ||
| # osx-arm64 (built on Intel): the arm64 Python can't execute, so extract both wheels | ||
| # (zips) with unzip instead -- same approach as the Windows bld.bat. | ||
| if "$PYTHON" -c "import sys" >/dev/null 2>&1; then | ||
| "$PYTHON" -m pip install --no-deps --no-index --find-links "$WHEELS_DIR" "$PKG_NAME==$PKG_VERSION" -vv | ||
| "$PYTHON" -m pip install --no-deps --no-index --find-links "$WHEELS_DIR" "mssql-python-odbc==$odbc_ver" -vv | ||
| else | ||
| echo "Host Python '$PYTHON' is not executable on this agent (non-emulated cross-build);" | ||
| echo "extracting both wheels into \$SP_DIR without running Python." | ||
| mkdir -p "$SP_DIR" | ||
| pkg_underscore="${PKG_NAME//-/_}" | ||
| # universal2 wheels are cpXY-specific (compiled ddbc_bindings), so filter on the | ||
| # target CONDA_PY to never grab another interpreter's wheel (mirrors bld.bat). | ||
| code_whl="" | ||
| for w in "$WHEELS_DIR/${pkg_underscore}-${PKG_VERSION}-cp${CONDA_PY}-"*.whl; do | ||
| [ -e "$w" ] && { code_whl="$w"; break; } | ||
| done | ||
| [ -n "$code_whl" ] || { echo "ERROR: no ${PKG_NAME}==${PKG_VERSION} cp${CONDA_PY} wheel in '$WHEELS_DIR'" >&2; exit 1; } | ||
| odbc_whl="" | ||
| for w in "$WHEELS_DIR"/mssql_python_odbc-"$odbc_ver"-py3-none-*.whl; do | ||
| [ -e "$w" ] && { odbc_whl="$w"; break; } | ||
| done | ||
| [ -n "$odbc_whl" ] || { echo "ERROR: no mssql_python_odbc==$odbc_ver py3-none wheel in '$WHEELS_DIR'" >&2; exit 1; } | ||
| echo "Extracting '$code_whl' -> '$SP_DIR'" | ||
| unzip -oq "$code_whl" -d "$SP_DIR" | ||
| # osx-arm64 cross can't run the arm64 Python, so statically prove the extracted | ||
| # binding is for THIS interpreter -- a cpXY ddbc_bindings for another Python (the bug | ||
| # where every osx-arm64 build shipped the cp310 .so) would only fail at the user's | ||
| # import. The python-tag twin of the win-arm64 PE-arch assert. | ||
| ls "$SP_DIR"/mssql_python/ddbc_bindings.cp${CONDA_PY}-*.so >/dev/null 2>&1 || { | ||
| echo "ERROR: '$code_whl' has no mssql_python/ddbc_bindings.cp${CONDA_PY}-*.so (wrong-Python binding)." >&2 | ||
| exit 1 | ||
| } | ||
| echo "Extracting '$odbc_whl' -> '$SP_DIR'" | ||
| unzip -oq "$odbc_whl" -d "$SP_DIR" | ||
| fi | ||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Linux driver reachability (#563) -- the core fix. | ||
| # --------------------------------------------------------------------------- | ||
| # Declaring krb5/openssl/libltdl as conda deps drops them in $PREFIX/lib, but that is | ||
| # INERT: the vendored ODBC .so ship with a bare DT_RUNPATH=$ORIGIN (no climb), so the | ||
| # loader never looks in $PREFIX/lib and falls through to SYSTEM krb5 (#563 crash) or | ||
| # can't find libltdl.so.7. Fix: stamp a relative "$ORIGIN:$ORIGIN/<climb>" onto | ||
| # libmsodbcsql* + libodbcinst.so.2 so they resolve THIS env's $PREFIX/lib. Safe to | ||
| # patch -- the Linux .so are malware-scanned, not code-signed (only Windows .dll / | ||
| # macOS .dylib are, and those are never touched). Linux-only: the glob is a no-op on | ||
| # macOS. audit_bundled_binaries.py asserts the same exact climb. | ||
| prefix_lib="$PREFIX/lib" | ||
| shopt -s nullglob | ||
| have_linux_payload=0 | ||
| [ -d "$SP_DIR/mssql_python_odbc/libs/linux" ] && have_linux_payload=1 | ||
| # Count the driver (libmsodbcsql) and the driver manager (libodbcinst) separately so a | ||
| # payload missing EITHER fails loudly -- a lone libodbcinst would ship no SQL driver. | ||
| msodbc_seen=0 | ||
| odbcinst_seen=0 | ||
| for libdir in "$SP_DIR"/mssql_python_odbc/libs/linux/*/*/lib; do | ||
| # Exact climb from this driver dir up to $PREFIX/lib (from the real layout, never a | ||
| # hard-coded ../ count). | ||
| climb="$("$PYTHON" -c 'import os,sys; print(os.path.relpath(sys.argv[1], sys.argv[2]))' "$prefix_lib" "$libdir")" | ||
| want="\$ORIGIN:\$ORIGIN/$climb" | ||
| for so in "$libdir"/libmsodbcsql-*.so.* "$libdir"/libodbcinst.so.2; do | ||
| [ -e "$so" ] || continue | ||
| case "$(basename "$so")" in | ||
| libmsodbcsql-*.so.*) msodbc_seen=$((msodbc_seen + 1)) ;; | ||
| libodbcinst.so.2) odbcinst_seen=$((odbcinst_seen + 1)) ;; | ||
| esac | ||
| got="$(patchelf --print-rpath "$so" 2>/dev/null || true)" | ||
| if [ "$got" = "$want" ]; then | ||
| echo "RPATH-OK (already baked) $(basename "$so") -> $got" | ||
| continue | ||
| fi | ||
| patchelf --set-rpath "$want" "$so" | ||
| got="$(patchelf --print-rpath "$so")" | ||
| # Assert the EXACT intended RUNPATH, not just "no absolute entry". | ||
| if [ "$got" != "$want" ]; then | ||
| echo "ERROR: patch did not yield the exact expected RUNPATH ('$got' != '$want')." >&2 | ||
| exit 1 | ||
| fi | ||
| echo "RPATH-PATCHED $(basename "$so") -> $got" | ||
| done | ||
| done | ||
| shopt -u nullglob | ||
| # A Linux payload missing the driver OR the driver manager is a bypass hole (a bare | ||
| # `conda build` skipping the orchestrator audit would ship un-asserted binaries). | ||
| if [ "$have_linux_payload" = "1" ] && { [ "$msodbc_seen" = "0" ] || [ "$odbcinst_seen" = "0" ]; }; then | ||
| echo "ERROR: Linux payload present but incomplete (libmsodbcsql=$msodbc_seen, libodbcinst.so.2=$odbcinst_seen)." >&2 | ||
| exit 1 | ||
| fi | ||
| [ "$msodbc_seen" -gt 0 ] && echo "LINUX_RPATH_CLIMB_OK" || true |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| {% set version = environ.get('MSSQL_PYTHON_VERSION', '1.14.0') %} | ||
|
|
||
| package: | ||
| name: mssql-python | ||
| version: "{{ version }}" | ||
|
|
||
| build: | ||
| number: 0 | ||
| # This recipe REPACKAGES a prebuilt, signed wheel -- it compiles nothing -- so | ||
| # conda-build's from-source checks mis-fire on the vendored binaries. Downgrade | ||
| # overlinking/overdepending to warnings, and disable the relocation / prefix scans | ||
| # that would corrupt the signature or fail the pure-repackage packaging step. | ||
| error_overlinking: false | ||
| error_overdepending: false | ||
| binary_relocation: false | ||
| detect_binary_files_with_prefix: false | ||
| # .pyc byte-compilation runs the TARGET Python, which can't execute on the cross legs | ||
| # (osx-arm64 built on Intel, win-arm64 on x64). Skip pyc there; Python regenerates it. | ||
| skip_compile_pyc: | ||
| - "**/*.py" # [osx or win] | ||
| # Passed into the isolated conda-build env so build.sh / bld.bat can install the | ||
| # prebuilt wheel from --find-links offline (WHEELS_DIR) and locate the odbc wheel to | ||
| # vendor in (MSSQL_ODBC_VERSION). | ||
| script_env: | ||
| - WHEELS_DIR | ||
| - MSSQL_ODBC_VERSION | ||
|
|
||
| requirements: | ||
| build: | ||
| # patchelf stamps the $ORIGIN RPATH climb (#563) so the declared conda | ||
| # krb5/openssl/libltdl below are actually reachable. Linux-only, build-time only. | ||
| - patchelf # [linux] | ||
| host: | ||
| # Bare `python` (NO version range): conda-build binds the per-Python variant from the | ||
| # --python matrix here, so each build emits the matching py3XX package. A range such | ||
| # as `python >=3.10` collapses ALL --python builds into ONE cp<newest> package (build | ||
| # string "py>=310") whose python_abi is pinned to 3.14, so `conda create python=3.10 | ||
| # mssql-python` can't solve. The >=3.10 floor is enforced implicitly: only py310-py314 | ||
| # are built and each package's python_abi pins its exact minor. | ||
| - python | ||
| - pip | ||
| run: | ||
| # Bare `python` for the same reason as host (the floor is the per-build python_abi, | ||
| # NOT a range on python -- a range breaks the per-Python solve). | ||
| - python | ||
| # conda does NOT inherit the wheel's install_requires, so pin azure-identity here. | ||
| - azure-identity >=1.12.0 | ||
| # ODBC Driver 18 payload deps (the driver ships inside this package, so its | ||
| # security-serviced deps are declared here). OpenSSL for TLS is dlopen'd, so | ||
| # overlinking can't see it; pinned <4 (Driver 18 supports the OpenSSL 1.1/3.0 ABI | ||
| # only). macOS uses Homebrew OpenSSL, Windows SChannel. | ||
| - openssl >=3,<4 # [linux] | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. just double checked this one - On macOS this package is not self-contained for TLS and still points to homebrew installation instead of conda env concrete break scenario: end user on a conda-only Mac (no Homebrew) runs repro'd on the shipped 1.14.0 osx-arm64 package, one real SQL Server,
why the recipe misses it:
conda's own openssl does land in Why CI and manual runs look green:
direction: give the macOS driver a way to resolve the env's own openssl (the |
||
| # libmsodbcsql NEEDs libkrb5 / libgssapi_krb5 on Linux (macOS framework, Windows SSPI). | ||
| - krb5 # [linux] | ||
| # libodbcinst.so.2 NEEDs libltdl.so.7 (not bundled); libtool provides it and the | ||
| # #563 climb makes the driver resolve THIS env's copy. | ||
| - libtool # [linux] | ||
| # msodbcsql18.dll imports VCRUNTIME140.dll but the vendored vcredist ships only | ||
| # msvcp140.dll; declare the serviced conda runtime. | ||
| - vc14_runtime # [win] | ||
| # Re-assert the wheel's platform floor (conda drops the wheel tag); never stricter. | ||
| - __glibc >=2.28 # [linux] | ||
| - __osx >=15.0 # [osx] | ||
|
|
||
| test: | ||
| imports: | ||
| - mssql_python | ||
|
|
||
| about: | ||
| home: https://github.com/microsoft/mssql-python | ||
| # Ships MIT code AND the proprietary ODBC Driver 18 payload, so the license is both. | ||
| license: MIT AND LicenseRef-Microsoft-Proprietary | ||
| license_file: | ||
| - ../../LICENSE | ||
| # The proprietary EULAs live with the ODBC payload; reference them there rather than | ||
| # duplicating. conda-build embeds them in info/licenses/. | ||
| - ../../mssql_python_odbc/licenses/MICROSOFT_ODBC_DRIVER_FOR_SQL_SERVER_LICENSE.txt | ||
| - ../../mssql_python_odbc/licenses/MICROSOFT_VISUAL_STUDIO_LICENSE.txt | ||
| summary: Microsoft driver for Python to interact with SQL Server and Azure SQL. | ||
| description: | | ||
| mssql-python is a DB API 2.0 (PEP 249) compliant driver for SQL Server, | ||
| Azure SQL, and Azure Synapse. This conda package is self-contained: the | ||
| proprietary Microsoft ODBC Driver 18 payload ships inside it (the same model as | ||
| the v1.11.0 wheel), so no separate driver package is required. | ||
|
|
||
| extra: | ||
| recipe-maintainers: | ||
| - jahnvithakkar | ||
Uh oh!
There was an error while loading. Please reload this page.