Skip to content

Require full unique indexes for lookup tables - #915

Open
workstonedai-collab wants to merge 1 commit into
simonw:mainfrom
workstonedai-collab:fix/lookup-full-unique-index
Open

workstonedai-collab wants to merge 1 commit into
simonw:mainfrom
workstonedai-collab:fix/lookup-full-unique-index

Conversation

@workstonedai-collab

@workstonedai-collab workstonedai-collab commented Oct 24, 2026 •

Copy link
Copy Markdown

Fixes #914.

Problem

Table.lookup() checked only whether an index covered the lookup columns. A non-unique index could therefore be mistaken for the required uniqueness constraint, and a partial unique index could be treated as equivalent to a full unique index. This could allow duplicate lookup rows.

Behavior

  • Before: an ordinary or partial index could prevent lookup() from creating the required full unique index.
  • After: lookup() reuses an existing full unique index on exactly the lookup columns, or creates one when needed.
  • Existing indexes are preserved.
  • If existing data violates the required uniqueness constraint, the unique-index creation fails with sqlite3.IntegrityError before any rows are changed.

This change is limited to Table.lookup() unique-index detection and creation. It makes no broad performance or indexing claims and is independent of #913; both are based on the same upstream commit.

Validation

Validation was run by Codex:

  • Existing lookup tests, new regression tests, and three standalone probes: 18 passed.
  • Repository lookup tests: 15 passed after the fix (4 failed before the fix).
  • Full suite: 1502 passed, 16 skipped, 33 warnings.
  • mypy: 10 source files passed.
  • Black, flake8, and diff checks passed.

Implementation and tests: MiniMax. Review and validation: Codex.


📚 Documentation preview 📚: https://sqlite-utils--915.org.readthedocs.build/en/915/

@Muguett-DBY Muguett-DBY left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ordinary-index duplicate case and partial-index case behave as described in my checks. I also tried a full expression index, which still makes lookup() fail:

db = sqlite_utils.Database(memory=True)
db.execute("CREATE TABLE species(id INTEGER PRIMARY KEY, name TEXT)")
db.execute("CREATE UNIQUE INDEX species_lower ON species(lower(name))")
db["species"].lookup({"name": "Palm"})
# AttributeError: 'NoneType' object has no attribute 'translate'

SQLite's PRAGMA index_info reports None for the expression's column name, and the new comprehension passes that to fold_identifier_case(). This also happens on the base, so it isn't a regression from this PR. A non-unique expression index and a partial unique expression index now work because they're filtered out before the comprehension.

Would it be reasonable to skip indexes containing expressions when looking for an exact column constraint, and add a test? I checked the exact base and PR lookup() methods with in-memory SQLite and the existing sqlite-utils support modules; I didn't run the full branch suite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lookup() treats ordinary and partial indexes as unique lookup constraints

2 participants