Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
15e4092
FIX: Validate caller-controlled native buffer sizes
gargsaumya Sep 21, 2026
39dea1b
FIX: Update negative Arrow batch size test
gargsaumya Sep 21, 2026
0110fef
Merge branch 'main' into saumya/native-size-validation
gargsaumya Sep 21, 2026
a02fdf4
FIX: Address native size review findings
gargsaumya Sep 21, 2026
44982a9
Merge remote-tracking branch 'origin/saumya/native-size-validation' i…
gargsaumya Sep 21, 2026
7519aa4
Merge branch 'main' into saumya/native-size-validation
gargsaumya Sep 22, 2026
3038841
FIX: Bound native parameter and Arrow source buffers
gargsaumya Sep 22, 2026
710ff12
FIX: Address native size validation review
gargsaumya Sep 25, 2026
597d2c2
Merge remote-tracking branch 'origin/main' into saumya/native-size-va…
Copilot Sep 25, 2026
a096689
FIX: Close remaining native size validation gaps
gargsaumya Oct 7, 2026
231ae90
FIX: Measure override payloads in native units
gargsaumya Oct 7, 2026
f75aef7
Merge main into native size validation
gargsaumya Oct 7, 2026
444fb7f
FIX: Enforce peak native fetch memory limit
gargsaumya Oct 7, 2026
d5b4431
FIX: Bound streamed LOB native memory
gargsaumya Oct 7, 2026
3b60b8c
FIX: Validate executemany payload sizes
gargsaumya Oct 7, 2026
9f5c11a
FIX: Copy encoded wide text as bytes
gargsaumya Oct 7, 2026
5b26c36
FIX: Stream DAE payloads in bounded chunks
gargsaumya Oct 7, 2026
3a57124
FIX: Complete bounded DAE streaming
gargsaumya Oct 7, 2026
c64fbdd
FIX: Align DAE and array metadata
gargsaumya Oct 7, 2026
b8a8873
FIX: Satisfy strict DAE return analysis
gargsaumya Oct 7, 2026
9a55372
FIX: Close remaining native size gaps
gargsaumya Oct 7, 2026
7bb3a1a
FIX: Restore driver-compatible DAE binding
gargsaumya Oct 7, 2026
37ec527
FIX: Bound wide LOB conversion memory
gargsaumya Oct 7, 2026
7b5a3b5
FIX: Align streamed parameter metadata
gargsaumya Oct 7, 2026
190760f
FIX: Preserve MAX parameter type semantics
gargsaumya Oct 7, 2026
5d0a2e3
FIX: Size text by active C binding
gargsaumya Oct 7, 2026
e98d558
FIX: Bind NULL rows independently in DAE batches
gargsaumya Oct 7, 2026
99e837c
FIX: Validate row-local DAE tokens
gargsaumya Oct 7, 2026
a72841c
FIX: Use MAX sentinel for oversized DAE
gargsaumya Oct 7, 2026
5919670
FIX: Size DAE metadata through MAX boundary
gargsaumya Oct 7, 2026
db18317
FIX: align DAE metadata with wide bindings
gargsaumya Oct 7, 2026
400bd0f
FIX: preserve actual parameter metadata size
gargsaumya Oct 7, 2026
26c4763
FIX: address native validation review findings
gargsaumya Oct 7, 2026
f533230
FIX: reset DAE bindings after each row
gargsaumya Oct 7, 2026
48a2f23
FIX: restore geometric LOB buffer growth
gargsaumya Oct 8, 2026
d0912f6
FIX: bound LOB growth test helper
gargsaumya Oct 8, 2026
019a1cc
Merge branch 'main' into saumya/native-size-validation
gargsaumya Oct 12, 2026
3bd80c4
FIX: address remaining parameter sizing reviews
gargsaumya Oct 12, 2026
97100e0
FIX: initialize executemany DAE token
gargsaumya Oct 12, 2026
f63e905
FIX: complete DAE rows and bound numeric text
gargsaumya Oct 12, 2026
73ab6f8
FIX: use bounded Unicode encoding paths
gargsaumya Oct 12, 2026
8fdde26
FIX: size base Unicode contents in batches
gargsaumya Oct 12, 2026
d202fb6
Merge branch 'main' into saumya/native-size-validation
gargsaumya Oct 12, 2026
bf5e738
Merge branch 'main' into saumya/native-size-validation
gargsaumya Oct 16, 2026
748ceb2
FIX: Address native size validation review feedback
gargsaumya Oct 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,21 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
does not change the default provider or ship any Rust driver binaries.

### Changed
- Cursor fetch/Arrow batch sizes and bulk-copy batch/commit sizes are now capped
at 1,000,000 rows; `arraysize` must be positive. Negative row counts raise
`ValueError` and boolean row counts raise `TypeError`. Zero-size fetches remain
empty, zero-size Arrow requests are schema-only, and bulk-copy zero retains
native default batching. Native array-bound `executemany()` calls are also
capped at 1,000,000 parameter sets.
- Native fetch buffers and parameter-array buffers each have a cumulative
256 MiB allocation budget. Array limits depend on batch size and column widths,
including slots for NULLs; reduce batch sizes or oversized `setinputsizes()`
hints when a budget is exceeded. Non-numeric hints are capped at 268,435,456
size units, while numeric precision/scale normalization is preserved.
Negative/boolean size hints are rejected. These are native buffer limits, not
process-memory or total-result limits; see the README's cursor-size guidance.
- Oversized formatted DECIMAL/NUMERIC text is rejected with `ValueError` before
native binding on both scalar and array-bound execution paths.
- Fetches reuse owned native metadata for stable columns within a result set;
`fetchmany()` avoids the Python metadata-dictionary roundtrip. Re-execution,
result transitions and statement/connection cleanup invalidate this metadata.
Expand Down
46 changes: 46 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,52 @@ example, requesting ASCII with `SQL_CHAR` does not cause non-ASCII parameters to
raise encoding errors. Use `setencoding()` with no arguments to restore the
supported defaults. `setdecoding()` independently controls how result data is read.

### Cursor size and native buffer limits
Comment thread
gargsaumya marked this conversation as resolved.

Row counts must be integers, not booleans. The following limits are inclusive:

| API setting | Accepted row count | Meaning of zero |
| --- | --- | --- |
| `Cursor.arraysize` | 1 to 1,000,000 | Not accepted |
| `Cursor.fetchmany(size)` | 0 to 1,000,000; `None` uses `arraysize` | Returns an empty list without fetching |
| `Cursor.arrow_batch(batch_size)`, `arrow(batch_size)`, `arrow_reader(batch_size)` | 0 to 1,000,000 | Schema only; no data rows |
| `Cursor.bulkcopy(batch_size)`, `bulkcopy_arrow(batch_size)` | 0 to 1,000,000 | Uses the native core's default batch/commit behavior |

Native array-bound `executemany()` calls also accept at most 1,000,000 parameter
sets per call; an empty input remains a no-op. These are per-batch limits, not
limits on the total number of rows in a result or table. Bulk-copy `batch_size`
controls rows per batch/commit, not the total source size.

Negative row counts now raise `ValueError`, rather than acting like empty or
unbounded requests. Boolean and other non-integer row counts raise `TypeError`.
In particular, negative Arrow sizes no longer produce an empty result.

Native fetch buffers have a cumulative **256 MiB** allocation budget, including
Arrow data buffers and native rowset storage. Native parameter-array buffers have
a separate cumulative **256 MiB** budget across all columns in one array-bound
`executemany()` call. A row count within the allowed range can still exceed its
byte budget when columns are wide. Parameter-array storage grows with the number
of parameter sets times the sum of each column's native element width and length
indicator; text buffers also include terminators, and wide text uses two-byte
UTF-16 code units. NULL rows still reserve array slots.

For allocation-limit errors, reduce `fetchmany()`/Arrow batch sizes or split
`executemany()` inputs into smaller calls, and avoid unnecessarily large
`setinputsizes()` hints. Non-numeric size hints must be non-negative integers no
greater than 268,435,456 in the SQL type's size units; this does not guarantee that
the corresponding byte allocation fits. Numeric hints describe precision/scale
instead of buffer length and retain their normalization to supported SQL Server
precision (up to 38). Boolean or negative size/scale hints raise `ValueError`.
Formatted DECIMAL/NUMERIC parameter text exceeding 4000 UTF-16 code units is
rejected before native binding.

These budgets bound the native buffers checked by the driver, **not total process
memory or total result memory**. Python objects, driver/core allocations, input
collections, and retained result batches consume additional memory. `fetchall()`
and `arrow()` still materialize all remaining rows; process bounded
`fetchmany()` batches or iterate `arrow_reader()` without retaining every batch
when the complete result may not fit in memory.

## Getting Started Examples
Connect to SQL Server and execute a simple query:

Expand Down
Loading
Loading