Skip to content

Allow SQLITE_MAX_VARS to be customized via Database(sqlite_max_vars=...)#784

Open
AmadNaseem wants to merge 1 commit into
simonw:mainfrom
AmadNaseem:feature/sqlite-max-vars-configurable
Open

Allow SQLITE_MAX_VARS to be customized via Database(sqlite_max_vars=...)#784
AmadNaseem wants to merge 1 commit into
simonw:mainfrom
AmadNaseem:feature/sqlite-max-vars-configurable

Conversation

@AmadNaseem

@AmadNaseem AmadNaseem commented Jul 8, 2026

Copy link
Copy Markdown

Refs #147.

SQLITE_MAX_VARS is hard-coded to 999, which caps how many rows insert_all batches into a single INSERT. Many SQLite builds are compiled with a much higher SQLITE_MAX_VARIABLE_NUMBER (e.g. 250,000 on Debian/Ubuntu, 500,000 on recent macOS), so on those systems the 999 cap forces many more, smaller batches than necessary.

As suggested in the issue, this adds an optional sqlite_max_vars argument to the Database constructor:

db = Database("data.db", sqlite_max_vars=250_000)
  • Default behaviour is unchanged: when the argument is not given, the limit falls back to the module-level SQLITE_MAX_VARS (999).
  • A Database.sqlite_max_vars property exposes the effective value.
  • Table.insert_all uses it in both places that previously referenced the module global: the column-count guard and the batch-size calculation.

I deliberately kept this to the constructor argument only, and did not add automatic detection of the compiled limit — that would change default batching for everyone and is a larger, separate change.

Docs updated in docs/python-api.rst; tests added in tests/test_create.py (default stays 999, a raised value produces fewer INSERT batches as measured via the tracer hook, and the column-count error message reflects the custom value).


📚 Documentation preview 📚: https://sqlite-utils--784.org.readthedocs.build/en/784/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant