Skip to content

feat: add SerpBase web search integration - #3758

Draft
gefsikatsinelou wants to merge 1 commit into
deepset-ai:mainfrom
gefsikatsinelou:feat/add-serpbase-engine
Draft

feat: add SerpBase web search integration#3758
gefsikatsinelou wants to merge 1 commit into
deepset-ai:mainfrom
gefsikatsinelou:feat/add-serpbase-engine

Conversation

@gefsikatsinelou

Copy link
Copy Markdown

Summary

Adds a new SerpBase web search integration (serpbase-haystack). SerpBaseWebSearch is a Haystack component that runs a Google web search through the SerpBase REST API and returns Document objects + links, matching the interface of the existing SerperDevWebSearch / TavilyWebSearch / SearchApiWebSearch components.

Background

Haystack already ships web search integrations for Serper, Tavily, SearchAPI, Brave and DDGS. SerpBase is a Google SERP API (Google Search, Images, News, Videos and Maps) aimed at a lower price point ($0.30/1k queries, no subscription). Adding it gives Haystack users another Google-results backend choice with the same component contract — drop-in replaceable via component.set() or YAML.

The integration follows the repo's established layout exactly: it was scaffolded with scripts/create_new_integration.py and mirrors the serperdev integration structure (component, tests, workflow, labeler, coverage workflow, root README inventory row).

Changes

  • New: integrations/serpbase/ — full integration package
    • src/haystack_integrations/components/websearch/serpbase/websearch.pySerpBaseWebSearch component (sync run + async run_async, to_dict/from_dict, domain filtering with allowed_domains/exclude_subdomains, optional search_params passthrough)
    • src/haystack_integrations/components/websearch/serpbase/__init__.py — exports SerpBaseWebSearch and SerpBaseError
    • tests/test_websearch.py — unit tests (mocked HTTP) + integration tests (skipped unless SERPBASE_API_KEY is set)
    • pyproject.toml, README.md, LICENSE.txt, pydoc/config_docusaurus.yml, py.typed
  • New: .github/workflows/serpbase.yml — CI workflow mirroring the serperdev workflow
  • Updated: .github/labeler.ymlintegration:serpbase label rule
  • Updated: .github/workflows/CI_coverage_comment.yml — register Test / serpbase
  • Updated: README.md — serpbase row in the integration inventory table

Design decisions

Decision Rationale
API key via SERPBASE_API_KEY env var / Secret Follows existing convention (SERPERDEV_API_KEY, TAVILY_API_KEY, ...)
POST + X-API-Key header SerpBase API contract (https://api.serpbase.dev/google/search)
Parses organic, featured_snippet, people_also_ask Mirrors the SERP modules the serperdev component surfaces; Document.meta keeps rank/position/title/link
Same run/run_async signature + output types Drop-in compatible with the other web search components
Domain filtering (allowed_domains, exclude_subdomains) Same API surface as SerperDevWebSearch
No new dependencies Uses httpx, already used by the other web search integrations

Testing

  • Unit tests mock the HTTP layer and cover: top-k truncation (sync/async), missing-snippet handling, featured snippet + people-also-ask parsing, timeout → TimeoutError, HTTP/request errors → SerpBaseError, domain filtering.
  • Integration tests run only when SERPBASE_API_KEY is set (skipped otherwise, same as serperdev).
  • ruff and mypy pass on the integration (same config as the rest of the repo).

Notes for maintainers

  • If you'd like the integration tests to run in CI, add a SERPBASE_API_KEY secret to the repo — the workflow already references it (tests skip gracefully when it's absent).
  • Version tag will follow the integrations/serpbase-v* convention at release time.

- Add integrations/serpbase with SerpBaseWebSearch component
  (sync run + async run_async, domain filtering, search_params passthrough)
- Add unit tests with mocked HTTP layer and integration tests
  (skipped unless SERPBASE_API_KEY is set)
- Add CI workflow, labeler rule, coverage workflow registration
  and root README inventory row (via create_new_integration.py scaffold)
@gefsikatsinelou
gefsikatsinelou requested a review from a team as a code owner August 11, 2026 06:42
@gefsikatsinelou
gefsikatsinelou requested review from bogdankostic and removed request for a team August 11, 2026 06:42
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions github-actions Bot added the type:documentation Improvements or additions to documentation label Aug 11, 2026
@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @gefsikatsinelou, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from bogdankostic August 11, 2026 07:46
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Aug 11, 2026
@HaystackBot
HaystackBot marked this pull request as draft August 11, 2026 07:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-pending PR is in draft until the contributor signs the CLA topic:CI type:documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants