Add MariaDB Vector DocumentStore integration by SyedShahmeerAli12 · Pull Request #3565 · deepset-ai/haystack-core-integrations

SyedShahmeerAli12 · 2026-07-08T13:59:38Z

Summary

Part of #2340

Implements a complete MariaDB document store integration using MariaDB 11.7+ native VECTOR support.

MariaDBDocumentStore full DocumentStore protocol (write_documents, filter_documents, delete_documents, count_documents)
Vector similarity search using VEC_DISTANCE_COSINE / VEC_DISTANCE_EUCLIDEAN with MHNSW indexing
Full-text keyword search via MATCH ... AGAINST (IN NATURAL LANGUAGE MODE) on a FULLTEXT index
Haystack metadata filtering converted to JSON_UNQUOTE(JSON_EXTRACT(...)) SQL expressions with parameterized queries
MariaDBEmbeddingRetriever and MariaDBKeywordRetriever with FilterPolicy support
DuplicatePolicy support: FAIL (INSERT), OVERWRITE (upsert via ON DUPLICATE KEY UPDATE), SKIP (INSERT IGNORE)
Lazy connection with reconnect on ping failure

Tests

80 tests total all passing:

68 unit tests (mocked DB, no external dependency)
12 integration tests verified against a real MariaDB 11.7 Docker container

CI

Added .github/workflows/mariadb.yml with a MariaDB 11.7 service container, matching the pattern used by pgvector.

How to run locally

docker run -d --name mariadb-haystack \
  -e MARIADB_ROOT_PASSWORD=password \
  -e MARIADB_DATABASE=haystack \
  -p 3306:3306 mariadb:11.7

cd integrations/mariadb
pip install -e .
MARIADB_USER=root MARIADB_PASSWORD=password pytest tests/ -m integration

socket-security · 2026-07-08T14:01:37Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/mariadb@1.1.14

View full report

github-actions · 2026-07-08T14:02:41Z

Coverage report (tavily)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
integrations/tavily/src/haystack_integrations/components/fetchers/tavily
tavily_fetcher.py
Project Total

_{This report was generated by python-coverage-comment-action}

Implements MariaDBDocumentStore backed by MariaDB 11.7+ native VECTOR support with MHNSW indexing and full-text keyword search. - Full DocumentStore protocol: write_documents (FAIL/OVERWRITE/SKIP), filter_documents, delete_documents, count_documents - Vector similarity via VEC_DISTANCE_COSINE / VEC_DISTANCE_EUCLIDEAN - Full-text keyword search via MATCH ... AGAINST (NATURAL LANGUAGE MODE) - Haystack metadata filtering converted to JSON_EXTRACT SQL expressions - MariaDBEmbeddingRetriever and MariaDBKeywordRetriever with FilterPolicy - 80 tests: 68 unit + 12 integration (all verified against MariaDB 11.7) - GitHub Actions workflow with MariaDB 11.7 service container Closes deepset-ai#2340

- Make mariadb C extension import lazy so API reference builds without requiring libmariadb-dev in the docs environment - Fix Docker health check to use --su-mysql flag required by MariaDB 11.7 - Add mariadb (LGPL-2.1) to license compliance exclusion list

- Add skip-install=true to default hatch env so docs build does not attempt to compile the mariadb C extension (libmariadb-dev not available in the API reference runner). The pydoc search_path already points to src/ so modules are importable without installation. - Switch service container health check from healthcheck.sh (unreliable in some MariaDB 11.7 images) to mysqladmin ping which is more robust.

…THCHECK The mariadb:11.7 Docker image ships with a HEALTHCHECK instruction. GitHub Actions automatically waits for it when no custom options override it. Custom health-cmd variants (healthcheck.sh, mysqladmin) were failing because the slim runner environment handles them differently.

anakin87

I took a first look and found some points to address and some others to discuss.

SyedShahmeerAli12 · 2026-07-21T11:01:00Z

Addressed all the straightforward review comments. The HNSW index design (create_vector_index flag) and the lazy import approach are still open for discussion

anakin87

There are still some comments to address.

Please request my review when they are fixed. In the meantime, ask questions if needed.

SyedShahmeerAli12 · 2026-07-23T11:31:49Z

ruff auto-detects mariadb and haystack as separate import sections, so import mariadb must go in its own block after all from haystack imports.
Every attempt I made grouped them together, which ruff kept rejecting until I let --fix show me the exact layout it expected.

SyedShahmeerAli12 · 2026-08-03T10:11:53Z

I mistakently Referenced some PRs here , Well I have cleaned the mess the Code is opened for review

anakin87 · 2026-08-03T16:04:37Z

+        filters: dict[str, Any] | None = None,
+        top_k: int = 10,
+        score_threshold: float | None = None,
+        vector_function: str | None = None,


Searches using a different distance function will not be able to use a vector index

I'd simply not allow users to change the distance function at runtime, in other words by removing this parameter

Sorry for the misunderstanding. I meant:

let's make distance configurable in MariaDBDocumentStore.__init__ and take this parameter into account only when the table is created

do not expose distance in _embedding_retrieval

anakin87

There are still a few rough edges to address in this PR, including some points that were left unresolved from the previous review.

Since this is a new integration, I'd like to see a more thorough pass: try it yourself, refine the implementation, and make sure to address the existing feedback.
Otherwise, I may need to deprioritize further review.

SyedShahmeerAli12 · 2026-08-04T11:01:16Z

Hi @anakin87 , I've addressed all of your review comments:

Removed DocumentStore protocol inheritance
Fixed the import order
Updated the vector index SQL to use VECTOR INDEX (embedding) (removed COMMENT 'MHNSW(...)')
Added HAVING score > 0 to the keyword search to filter out zero-score results
Changed user/password to use Secret only (removed the plain string fallback)
Removed the vector_function parameter from _embedding_retrieval (and the retriever)
Moved packaging>=21.3 to the main dependencies
Removed all section separator comments from the tests
Updated all test fixtures to use Secret.from_token()

I also verified everything against a real MariaDB 11.7 Docker instance all 120 tests are passing (65 unit + 55 integration).

anakin87 · 2026-08-10T15:02:28Z

+# skip-install prevents hatch from installing the project (and thus the mariadb C extension)
+# in the docs environment. The pydoc search_path points to src/ so modules are found directly.
+skip-install = true
+dependencies = ["haystack-pydoc-tools", "ruff", "haystack-ai>=2.28.0"]


Suggested change

# skip-install prevents hatch from installing the project (and thus the mariadb C extension)

# in the docs environment. The pydoc search_path points to src/ so modules are found directly.

skip-install = true

dependencies = ["haystack-pydoc-tools", "ruff", "haystack-ai>=2.28.0"]

dependencies = ["haystack-pydoc-tools", "ruff"]

anakin87 · 2026-08-10T15:09:00Z

+        filters: dict[str, Any] | None = None,
+        top_k: int = 10,
+        score_threshold: float | None = None,
+        vector_function: str | None = None,


Sorry for the misunderstanding. I meant:

let's make distance configurable in MariaDBDocumentStore.__init__ and take this parameter into account only when the table is created

do not expose distance in _embedding_retrieval

anakin87 · 2026-08-10T15:43:32Z

+    value: Any = condition["value"]
+    sql_field = _build_field_expr(field, value)
+    clause, val = COMPARISON_OPERATORS[operator](sql_field, value)
+    # _in/_not_in return the value list directly; flatten it so params stay flat


currently boolean meta filters seem to match nothing.
Try for example the condition {"field": "meta.flag", "operator": "==", "value": True}

anakin87 · 2026-08-10T15:46:30Z

+        :param score_threshold: Minimum score to include a document. Documents below this score are excluded.
+        :returns: List of Documents ordered by similarity (most similar first).
+        """
+        _validate_filters(filters)


MariaDB does not raise an error if the query_embedding does not match the default embedding dimension but just returns records with null score. I propose raising an error ourselves.

SyedShahmeerAli12 · 2026-08-11T10:32:54Z

I did all the testing against MariaDB 11.7via Docker

SyedShahmeerAli12 requested a review from a team as a code owner July 8, 2026 13:59

SyedShahmeerAli12 requested review from anakin87 and removed request for a team July 8, 2026 13:59

github-actions Bot added topic:CI integration:tavily type:documentation Improvements or additions to documentation labels Jul 8, 2026

SyedShahmeerAli12 added 2 commits July 8, 2026 19:07

SyedShahmeerAli12 force-pushed the feat/mariadb-document-store-2340 branch from f71d1cd to ff12b6a Compare July 8, 2026 14:09

SyedShahmeerAli12 added 6 commits July 8, 2026 19:14

fix(mariadb): lint errors and test env setup

8afc968

fix(mariadb): remaining lint errors and add libmariadb-dev install

7bc8914

fix(mariadb): fix Python 3.14 distutils, S608/EM102/RUF100 lint

90aea75

fix(mariadb): use Any for cursor/connection to fix mypy union-attr

fd1ae83

anakin87 requested changes Jul 21, 2026

View reviewed changes

SyedShahmeerAli12 added 3 commits July 21, 2026 15:29

fix

b0a7f31

fix

b013ead

fix

0eca31c

SyedShahmeerAli12 requested a review from anakin87 July 21, 2026 11:01

add create_vector_index flag

269eeca

anakin87 requested changes Jul 23, 2026

View reviewed changes

fix imports and tests

8f91d92

SyedShahmeerAli12 force-pushed the feat/mariadb-document-store-2340 branch from 93e2728 to 8f91d92 Compare July 23, 2026 10:10

SyedShahmeerAli12 added 3 commits July 23, 2026 15:16

fix lint and MRO

f864931

fix lint and test ordering

4c0ece1

fix lint and embedding comparison

a8dc6bc

SyedShahmeerAli12 added 3 commits July 23, 2026 16:02

fix

97f394e

fix

21aa3ea

fix

c4457ac

SyedShahmeerAli12 requested a review from anakin87 July 23, 2026 11:31

Merge branch 'main' into feat/mariadb-document-store-2340

3fe0a05

This was referenced Jul 30, 2026

fix(docling): stringify binary_hash to prevent OpenSearch long overflow #3698

Closed

fix(docling)!: store binary_hash as a string to avoid 64-bit overflow #3699

Merged

feat(qdrant): expose RRF k and weights parameters in QdrantHybridRetriever #3702

Merged

github-actions Bot mentioned this pull request Aug 3, 2026

feat/microsoft-sharepoint: add region parameter to MSSharePointRetriever #3722

Merged

8 tasks

anakin87 reviewed Aug 3, 2026

View reviewed changes

Comment thread integrations/mariadb/src/haystack_integrations/document_stores/mariadb/document_store.py Outdated

anakin87 reviewed Aug 3, 2026

View reviewed changes

Comment thread integrations/mariadb/src/haystack_integrations/document_stores/mariadb/document_store.py

anakin87 reviewed Aug 3, 2026

View reviewed changes

Comment thread integrations/mariadb/src/haystack_integrations/document_stores/mariadb/document_store.py Outdated

anakin87 reviewed Aug 3, 2026

View reviewed changes

Comment thread integrations/mariadb/tests/test_document_store.py Outdated

anakin87 requested changes Aug 3, 2026

View reviewed changes

SyedShahmeerAli12 added 3 commits August 4, 2026 15:47

fix

76b729d

fix

074813a

fix

d7a7ca2

SyedShahmeerAli12 requested a review from anakin87 August 4, 2026 11:01

github-actions Bot mentioned this pull request Aug 6, 2026

feat: add Python 3.14 support for WatsonX integration #3744

Merged

SyedShahmeerAli12 added 3 commits August 10, 2026 15:39

fix

608f783

fix

c268227

fix

a8e4e3c

anakin87 requested changes Aug 10, 2026

View reviewed changes

fix

5b19b93

SyedShahmeerAli12 requested a review from anakin87 August 11, 2026 10:33

Uh oh!

Conversation

SyedShahmeerAli12 commented Jul 8, 2026 • edited by anakin87 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

CI

How to run locally

Uh oh!

socket-security Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026

Coverage report (tavily)

Uh oh!

anakin87 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SyedShahmeerAli12 commented Jul 21, 2026

Uh oh!

anakin87 left a comment

Choose a reason for hiding this comment

Uh oh!

SyedShahmeerAli12 commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SyedShahmeerAli12 commented Aug 3, 2026

Uh oh!

Uh oh!

Uh oh!

anakin87 Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

anakin87 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

anakin87 left a comment

Choose a reason for hiding this comment

Uh oh!

SyedShahmeerAli12 commented Aug 4, 2026

Uh oh!

anakin87 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

anakin87 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

anakin87 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

anakin87 Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SyedShahmeerAli12 commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SyedShahmeerAli12 commented Jul 8, 2026 •

edited by anakin87

Loading

socket-security Bot commented Jul 8, 2026 •

edited

Loading

SyedShahmeerAli12 commented Jul 23, 2026 •

edited

Loading

anakin87 Aug 10, 2026 •

edited

Loading