Allow SQLITE_MAX_VARS to be customized via Database(sqlite_max_vars=...)#784
Open
AmadNaseem wants to merge 1 commit into
Open
Allow SQLITE_MAX_VARS to be customized via Database(sqlite_max_vars=...)#784AmadNaseem wants to merge 1 commit into
AmadNaseem wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #147.
SQLITE_MAX_VARSis hard-coded to 999, which caps how many rowsinsert_allbatches into a single INSERT. Many SQLite builds are compiled with a much higherSQLITE_MAX_VARIABLE_NUMBER(e.g. 250,000 on Debian/Ubuntu, 500,000 on recent macOS), so on those systems the 999 cap forces many more, smaller batches than necessary.As suggested in the issue, this adds an optional
sqlite_max_varsargument to theDatabaseconstructor:SQLITE_MAX_VARS(999).Database.sqlite_max_varsproperty exposes the effective value.Table.insert_alluses it in both places that previously referenced the module global: the column-count guard and the batch-size calculation.I deliberately kept this to the constructor argument only, and did not add automatic detection of the compiled limit — that would change default batching for everyone and is a larger, separate change.
Docs updated in
docs/python-api.rst; tests added intests/test_create.py(default stays 999, a raised value produces fewer INSERT batches as measured via thetracerhook, and the column-count error message reflects the custom value).📚 Documentation preview 📚: https://sqlite-utils--784.org.readthedocs.build/en/784/