FEAT: Add opt-in/opt-out ODBC provider selection (msodbcsql18 / mssql-odbc)

[gargsaumya]

Linked work item: AB#47445

Summary

Adds a shutdown-safe, process-wide opt-in/opt-out selector for the existing Microsoft ODBC Driver 18 and the future Rust ODBC provider. Existing applications remain unchanged: msodbcsql18 / mssql-python-odbc is still the default.

Behavior

• Selection precedence: MSSQL_PYTHON_ODBC_PROVIDER environment variable → mssql_python.odbc_provider module property → msodbcsql18 default.
• Supported values: msodbcsql18 and mssql-odbc; unknown values fail closed at first connection.
• Selection freezes at the first valid connection. Calling pooling() alone does not freeze it.
• If the selected provider package is absent or incomplete, the error names the package, expected native path, and install/reinstall command.
• get_odbc_provider_info() reports the effective provider, source, package, frozen state, and invalid configuration without itself freezing selection.

Native loader

• Driver loading is deferred until the first native connection, after Python pushes the selected provider.
• Process-lifetime provider/loader state avoids C++ static-destruction ordering and retained-Python-exception crashes during interpreter shutdown.
• Native setter is internal, validated, and cannot change the provider after load.
• Rust package/import names are mssql-python-rust-odbc / mssql_python_rust_odbc.
• Rust driver artifacts are resolved as mssqlodbc.dll, mssqlodbc.so, or mssqlodbc.dylib, with a Linux distro/libc path segment.
• On Windows, the classic provider preloads its co-located mssql-auth.dll; the Rust provider resolves that library lazily from the system DLL search path for interactive Entra authentication.

Validation

• Native x64/cp313 extension builds successfully.
• Provider/dependency/module-property coverage: 64 passed, 3 skipped.
• Rust load-order/shutdown subprocess regression repeated successfully across 20 fresh processes.
• Full suite collection/execution completed without a shutdown crash; live-database tests cannot pass on this workstation because DB_CONNECTION_STRING is not configured.

Follow-up PR sequence

1. mssql-rs artifact contract: produce signed mssqlodbc artifacts for Windows, macOS, glibc Linux, and musl Linux across supported architectures; publish versioned artifacts to Azure Artifacts with checksums and licenses.
2. Rust provider wheel: add the mssql-python-rust-odbc distribution and platform wheel pipeline, download pinned Azure artifacts, enforce the loader’s libs/<platform>/<distro>/<arch>/... layout, and publish the companion package independently.
3. Dependency and matrix wiring: pin mssql-python-rust-odbc from mssql-python, validate both providers across the supported OS/Python/architecture matrix, and add provider-specific auth/connection/packaging smoke tests.
4. Release rollout: document support boundaries and telemetry/diagnostics, ship Rust as opt-in, then consider changing the default only in a separately reviewed release after parity and reliability gates pass.

This PR intentionally does not add or publish Rust binaries and does not change the default provider.

[Copilot]

Pull request overview

Adds a process-wide, resolve-once ODBC provider selection mechanism so mssql-python can switch at runtime between the classic ODBC Driver 18 provider (msodbcsql18) and a future Rust provider (mssql-odbc), while keeping the existing public connection API unchanged.

Changes:

• Introduces ProviderManager to resolve provider selection (env var → module property → default), freeze it at first resolution, and fail closed on invalid selections.
• Wires provider resolution into Connection.__init__ and pushes the selected provider into the native loader via ddbc_bindings.set_odbc_provider.
• Adds a new unit test module covering precedence/normalization/freeze behavior and missing-provider fail-closed behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/test_026_odbc_provider.py	Adds unit tests for provider precedence, normalization, freeze semantics, and fail-closed behavior.
mssql_python/pybind/ddbc_bindings.cpp	Adds native-side provider selection plumbing and uses provider-specific package/dist names during driver resolution.
mssql_python/odbc_provider.py	Implements the Python-side provider selection engine (ProviderManager).
mssql_python/connection.py	Freezes/verifies provider selection and pushes it into the native layer before driver load.
mssql_python/init.py	Exposes mssql_python.odbc_provider and get_odbc_provider_info() as public diagnostics/surface area.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 90%

🎯 Overall Coverage 82%

📈 Total Lines Covered: 7950 out of 9650 📁 Project: mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

• mssql_python/init.py (100%)
• mssql_python/connection.py (100%)
• mssql_python/odbc_provider.py (100%)
• mssql_python/pybind/ddbc_bindings.cpp (75.8%): Missing lines 994,1002-1004,1017-1018,1086-1088,1090,1137,1139-1140,1473-1475
• mssql_python/pybind/logger_bridge.cpp (71.4%): Missing lines 166-167

Summary

• Total: 185 lines
• Missing: 18 lines
• Coverage: 90%

mssql_python/pybind/ddbc_bindings.cpp

Lines 990-998

  990 // (`GetDriverPathCpp` is defined further below; forward-declared here so we can
  991 // verify the external package actually ships this platform's driver binary.)
  992 std::string GetDriverPathCpp(const std::string& moduleDir);
  993 
! 994 // -----------------------------------------------------------------------------
  995 // ODBC provider selection
  996 //
  997 // Two providers are supported: the classic Microsoft ODBC Driver 18
  998 // ("msodbcsql18", shipped by mssql_python_odbc) and the Rust driver

---

Lines 998-1008

   998 // ("msodbcsql18", shipped by mssql_python_odbc) and the Rust driver
   999 // ("mssql-odbc", shipped by mssql_python_rust_odbc). Python is the sole
  1000 // resolver (env var -> module property -> default) and pushes the chosen id
  1001 // here via _set_odbc_provider() before the driver loads. The native side does
! 1002 // not read the environment itself; if the push has not happened yet, it falls
! 1003 // back to the hardcoded classic default.
! 1004 // -----------------------------------------------------------------------------
  1005 
  1006 namespace {
  1007 constexpr const char* kProviderMsodbcsql18 = "msodbcsql18";
  1008 constexpr const char* kProviderMssqlOdbc = "mssql-odbc";

---

Lines 1013-1022

  1013 std::string NormalizeProviderId(const std::string& id) {
  1014     // Mirror Python's ProviderManager._normalize(): trim surrounding whitespace
  1015     // and lowercase, leaving interior characters intact so both sides agree.
  1016     size_t start = 0;
! 1017     size_t end = id.size();
! 1018     while (start < end && std::isspace(static_cast<unsigned char>(id[start]))) {
  1019         ++start;
  1020     }
  1021     while (end > start && std::isspace(static_cast<unsigned char>(id[end - 1]))) {
  1022         --end;

---

Lines 1082-1094

  1082     // module initialization — but acquiring it here self-documents the C-API
  1083     // dependency and keeps a future GIL-released caller from turning this into a
  1084     // hard crash.
  1085     py::gil_scoped_acquire gil;
! 1086     const std::string providerId = GetSelectedProviderId();
! 1087     const std::string packageName = ProviderPackageForId(providerId);
! 1088     const std::string distName = ProviderDistForId(providerId);
  1089     try {
! 1090         py::object module = py::module::import(packageName.c_str());
  1091         py::object module_path = module.attr("__file__");
  1092         std::string module_file = module_path.cast<std::string>();
  1093 
  1094         fs::path parentDir = fs::path(module_file).parent_path();

---

Lines 1133-1144

  1133         if (e.matches(PyExc_ModuleNotFoundError)) {
  1134             // Phase 2: the standalone package is required. Turn the missing
  1135             // dependency into a clear, actionable error instead of a fallback.
  1136             LOG("GetOdbcLibsBaseDir: required package %s is not installed (%s)",
! 1137                 packageName.c_str(), e.what());
  1138             ThrowStdException(
! 1139                 "The required '" + distName + "' package (which ships the ODBC driver "
! 1140                 "binaries) is not installed. Install it with: pip install " + distName);
  1141         }
  1142         // A different import-time error means the package is installed but
  1143         // broken; surface it instead of silently masking the real problem.
  1144         LOG("GetOdbcLibsBaseDir: importing %s failed unexpectedly (%s); "

---

Lines 1469-1479

  1469         std::rethrow_exception(m_loadError);
  1470     }
  1471 }
  1472 
! 1473 bool DriverLoader::isDriverLoaded() const {
! 1474     return m_driverLoaded.load();
! 1475 }
  1476 
  1477 // SqlHandle definition
  1478 SqlHandle::SqlHandle(SQLSMALLINT type, SQLHANDLE rawHandle) : _type(type), _handle(rawHandle) {}

---

mssql_python/pybind/logger_bridge.cpp

Lines 162-171

  162     // gilstate_tss_set: failed to set current tstate (TSS)" once the
  163     // thread-state TSS key has been torn down. Skipping a log line is
  164     // harmless; crashing the process is not.
  165     if (Py_IsInitialized() == 0) {
! 166         return;
! 167     }
  168 #if PY_VERSION_HEX >= 0x030D0000
  169     if (Py_IsFinalizing()) {
  170         return;
  171     }

---

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.logger_bridge.cpp: 59.6%
mssql_python.pybind.ddbc_bindings.h: 61.5%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 75.2%
mssql_python.row.py: 77.6%
mssql_python.pybind.connection.connection_pool.cpp: 81.6%
mssql_python.__init__.py: 83.1%
mssql_python.pybind.connection.connection.cpp: 84.4%
mssql_python.logging.py: 85.5%
mssql_python.helpers.py: 89.3%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

[Vahid (Vahid-b)]

Summary

Adds a process-wide, resolve-once selector between msodbcsql18 and the Rust mssql-odbc driver, with msodbcsql18 as the Phase 1 default and the Rust path failing closed until its wheel ships. The design is sound — Python as the sole resolver, the native side purely a receiver, freeze at first connect, no public API change. One finding I'd treat as blocking; the rest are suggestions and nits.

I verified the load ordering the whole design rests on: loadDriver() is lazy behind std::call_once, and the only entry into it from a fresh process is the Connection constructor at pybind/connection/connection.cpp:25. connection.py:375-376 pushes the provider well before ddbc_bindings.Connection(...) at line 741. That part is correct.

All of copilot-pull-request-reviewer's findings and the DevSkim alert look addressed in e9372ba7 / 094dc2f3 / f854f8dc; I am not re-filing any of them. The one that is only half-closed is the Optional[str] stub, noted inline.

Blocking

The pooling hook (pooling.py:66-71) — its stated reason is false, and its only real effect is an unwanted early freeze. Detail inline. enable_pooling does not load the native driver, so the hook protects nothing, but it does make mssql_python.pooling() freeze the provider and become able to raise ImportError.

Suggestions

Six inline: the source field that is always None before the freeze, effective() raising on a bad env var, the native side's silent coercion plus unguarded set_odbc_provider, the mssql-auth.dll justification, the Linux path layout versus the libc split, and the Optional[str] stub.

One that has no line to sit on: no test covers the pooling freeze path. That is where the blocking finding lives, and a test asserting PoolingManager.enable() does not freeze the selection would have caught it. Worth adding alongside the fix.

Nits

Three inline: the mid-file #include, the recwarn assertion, and the resolve-before-validate ordering in Connection.__init__. Plus one that spans the change rather than a line: NormalizeProviderId (ddbc_bindings.cpp:993-1003) strips interior whitespace while Python's _normalize (odbc_provider.py:44-54) only does .strip(). Harmless while Python is the sole resolver, but the two should agree if the native one ever becomes authoritative.

What I ran

pytest isn't installed here and the package needs the compiled ddbc_bindings extension, so I could not run the suite — CI is the gate for that. The two runtime observations below came from loading odbc_provider.py standalone against a stubbed mssql_python.logging:

A. get_info BEFORE resolve, env=mssql-odbc: {'id': 'mssql-odbc', 'package': 'mssql_python_rust_odbc', 'source': None, 'frozen': False}
C. get_info AFTER  resolve:                 {'id': 'mssql-odbc', 'package': 'mssql_python_rust_odbc', 'source': 'environment', 'frozen': True}
D. get_info    with bad env RAISED: ValueError Unknown ODBC provider 'bogus-value'. ...
E. effective() with bad env RAISED: ValueError Unknown ODBC provider 'bogus-value'. ...

The cross-repo claims (mssql-auth.dll, artifact filenames, the Linux libc split) were checked against microsoft/mssql-rs source rather than from memory; file and line are cited in each comment.

---

Reviewed with GitHub Copilot on behalf of Vahid (@Vahid-b), then checked by hand. Not an approval — push back on anything that looks wrong.

[Saurabh Singh (saurabh500)]

The native load-order issue below blocks the Rust opt-in. I also agree with the existing PoolingManager.enable() thread: native enable_pooling() only configures pool state, so it should not freeze provider selection when the documented boundary is the first connection. I did not duplicate that thread.

The authentication comment below reflects the clarified requirement that the Rust provider also requires mssql-auth.dll for its Windows interactive-authentication path.