Repository navigation
SmartScraperGraph returns a confident false negative when the answer is on a linked page (no signal that DepthSearchGraph was needed) #1120
Description
Summary
When the answer to a prompt lives on a linked page (a privacy policy, terms, an "our team" page), SmartScraperGraph returns a confident negative rather than any signal that the answer might be elsewhere. The output is indistinguishable from a genuine "this fact is not present on the site."
I know DepthSearchGraph is the right tool for this (per #1008). The issue is that nothing tells you that you needed it — the single-page result looks like a complete answer.
Reproduction
Prompt (a compliance question — does the site forbid transferring contact data?):
Does this page, or a privacy policy it links to, state that email addresses or
personal information will NOT be given, sold, rented, traded or transferred to
third parties? Answer strictly as JSON:
{"no_transfer_notice": true|false, "evidence": "<=25 words"}
SmartScraperGraph(prompt=PROMPT, source=url, config={
"llm": {"api_key": KEY, "model": "google_genai/gemini-2.5-flash",
"model_tokens": 1000000},
"headless": True,
}).run()Two public sites where the notice is real but sits one link away:
| URL | Notice lives at | Actual text | SmartScraperGraph |
|---|---|---|---|
https://www.covenantcarepractices.com/contact/ |
/privacy/ |
"this information will not be shared with any third parties" | no_transfer_notice: False |
https://excelsiorhealthcare.com/contact-us/ |
/privacy-policy/ |
"does not and will never sell or rent any identifiable information to third party vendors" | no_transfer_notice: False |
A ~30-line baseline (fetch page → follow href matching privacy|terms → concatenate → same model, same prompt) returns True with the quoted evidence for both. Across 10 URLs the two approaches were the same speed (4.6 vs 4.9 s/url) and agreed on 6 of 8 comparable pages; both disagreements were this pattern.
Why it's worth a signal rather than just docs
For extraction tasks the failure is a wrong field. For compliance questions the failure has a direction: a false False means "no restriction found," which is precisely the answer that causes someone to act. Silent false negatives are the expensive kind.
Suggestions (any one would help)
- Have
SmartScraperGraphreturn an explicit "not found on this page" state distinct from a negative finding. - When the fetched page contains links whose text/href matches terms in the prompt and the answer is negative, surface a hint that a depth graph may be required.
- Document on the
SmartScraperGraphpage that link-dependent questions needDepthSearchGraph, with a worked example — the current docs make the single-page graph the obvious default.
Environment
scrapegraphai installed fresh from PyPI, Python 3.13, macOS, langchain-google-genai, Playwright chromium headless.
Happy to open a PR for (3) if that's the preferred shape.
Activity
NG-PR0JECT commentedon Aug 17, 2026
I hit this exact failure mode on a compliance-style extraction task recently, and I think the "silent false negative" framing is the right way to look at it. The single-page graph is answering about the content it was given, and an LLM asked "does this page, or a policy it links to, say X" with only the page text will produce a confident negative rather than "I can't know — I never saw the policy." That's the root of why your ~30-line baseline wins: it never asks the model to reason about pages it hasn't seen.
The generalizable pattern behind that baseline is separating deterministic retrieval (fetch the page, follow same-domain links whose href/text match intents in the prompt, concatenate) from the LLM judgment step. Once retrieval is deterministic, the model only ever answers about content that actually reached it, and this whole class of confident-but-wrong negatives disappears. It also makes your "agreed on 6 of 8" explainable: both disagreements were exactly the cases where retrieval didn't include the policy page, so the two approaches weren't really disagreeing — one had the evidence and the other didn't.
On the signal itself, suggestion (2) is the cheapest to ship and I'd sharpen it slightly: rather than matching links against the prompt broadly, match against intent-bearing targets — privacy, terms, policy, team, about, pricing. Those are the link targets that carry answers for compliance/claims questions, and they're also the ones where a wrong negative is expensive (a false "no restriction found" reads as a green light to whoever acts on it).
FWIW, this is the design my extraction API (Scrapiq — URL → clean text/JSON for LLM pipelines) follows: deterministic extraction first, LLM judgment second. Happy to share the link-following heuristics we ended up with if you want to build out a docs example for (3).