SmartScraperGraph returns "NA"/blank for all fields even on pages with clearly present target content

Describe the bug

Using SmartScraperGraph with a plain-language JSON-extraction prompt, the LLM consistently returns "NA" (or blank) for every requested field — even against pages where the requested information is unambiguously present (e.g. a Wikipedia infobox with an explicit founding year and employee count). Verbose logging shows Fetch Node → ParseNode → GenerateAnswer all execute without error, and the run completes successfully, but the answer content doesn't reflect the actual page.

To Reproduce

from scrapegraphai.graphs import SmartScraperGraph

graph_config = {
    "llm": {"api_key": "sk-...", "model": "openai/gpt-4o-mini"},
    "verbose": True,
    "headless": True,
}
scraper = SmartScraperGraph(
    prompt="Extract the company founding year and number of employees as JSON with keys years_established and team_size.",
    source="https://en.wikipedia.org/wiki/Timpson_(company)",
    config=graph_config,
)
print(scraper.run())

Actual output:

{'content': {'years_established': 'NA', 'team_size': 'NA'}}

Expected: something close to {'years_established': '1865', 'team_size': '...'} — the Wikipedia infobox states the founding year plainly.

Also noting for anyone hitting the same thing: the answer is returned nested under a content key ({'content': {...}}), not at the top level of the dict run() returns — that part isn't a bug, just worth documenting since it's easy to miss when reading the field values back out.

Environment

scrapegraphai 2.1.5
Python 3.14.6
macOS
Model: openai/gpt-4o-mini

Additional context
Same result across multiple real-world pages tested (business homepages, an "About Us" page, and the Wikipedia page above) — not isolated to one site. Suspect the page content isn't reaching the LLM intact, possibly lost/truncated during the parsing or chunking step, but haven't traced it further into the library internals.

@paulhiltonmarketing-gif

Hi @paulhiltonmarketing-gif — I looked into this. The short version: the page in the repro
doesn't exist, and ScrapeGraph is scraping Wikipedia's 404 page without telling you.

https://en.wikipedia.org/wiki/Timpson_(company) returns HTTP 404. The article lives at
Timpson_(retailer). So FetchNode fetches Wikipedia's "Wikipedia does not have an article
with this exact name" page (~49k chars of navigation chrome), hands that to the LLM, and the
LLM correctly answers NA — there is no founding year in what it was given.

I traced the pipeline with no LLM involved (Fetch → Parse only), since that half is
deterministic:

# the URL from the issue
FetchNode  -> doc length: 49,891 chars  | '1865' present: False
ParseNode  -> 1 chunk, 7,568 chars      | '1865' present: False
             (content is Wikipedia nav: "Jump to content / Main menu / Navigation ...")

# same code, corrected URL: en.wikipedia.org/wiki/Timpson_(retailer)
FetchNode  -> doc length: 256,880 chars | '1865' present: True
ParseNode  -> 5 chunks, 43,940 chars    | '1865' present: True

So the extraction chain is fine. Swapping the URL should fix your case:

source="https://en.wikipedia.org/wiki/Timpson_(retailer)"

That said, I think there's a real bug here, and it's the reason this was so confusing to
debug. FetchNode has no HTTP status awareness on its default path, so any error page —
404, 403, 500, a captcha wall, a login redirect — is silently accepted as valid content and
passed to the LLM. You get a confidently wrong answer with no signal that anything went wrong.

The library already knows how to handle this, just not on the path everyone uses:

use_soup=True (opt-in) checks response.status_code == 200 and logs a warning otherwise
(fetch_node.py ~line 290).

The default path (ChromiumLoader → ascrape_playwright) discards the response entirely:

await page.goto(url, wait_until="domcontentloaded")   # returns a Response; dropped
results = await page.content()
logger.info("Content scraped")                        # logged even for a 404

Playwright's page.goto() returns a Response with .status, so the default path could warn
exactly the way the use_soup path already does:

response = await page.goto(url, wait_until="domcontentloaded")
if response is not None and response.status >= 400:
    logger.warning(
        f"Received HTTP {response.status} for {url}; "
        "the scraped content is likely an error page, not the intended document."
    )

Non-breaking (a warning, not a raise), and it makes this class of confusion self-diagnosing.
I checked that the status is genuinely available at that line — goto() reports 404 for the
URL in this issue and 200 for the corrected one, and the content lengths it returns (49,891 /
256,880) match what FetchNode produces exactly, so it's the same path:

issue's URL   -> response.status = 404 | content 49,891 chars  | '1865' present: False
correct URL   -> response.status = 200 | content 256,880 chars | '1865' present: True

@VinciGit00 — happy to open a PR for that if you'd like it. Two questions before I do:

Warn, or raise? A warning keeps existing behaviour for anyone deliberately scraping error
pages; raising is louder but could break callers. use_soup warns today, so warning seems
the consistent choice.
Scope: just ascrape_playwright (the default), or the other page.goto call sites
(ascrape_playwright_scroll, ascrape_with_js_support) too?

On the last point in your issue — the answer nesting under content — that's worth documenting
separately; it tripped me up as well when reading the values back out.

@paulhiltonmarketing-gif

@paulhiltonmarketing-gif @sahilkanger — great diagnosis on the 404 path, @sahilkanger. One thing I'd add for the second failure mode in the report (the "same result across business homepages and an About Us page" claim, which the 404 repro doesn't explain):

The '1865' present: False check you used manually is exactly the guard worth automating — and it catches both cases at once. An HTTP-400 warning solves the 404 path, but a 200 page can still reach the LLM with the target absent: content behind JS that never rendered, the field living in a <script> blob that the parser drops, or the doc truncated beyond the model window. In all of those the LLM "correctly" returns NA and the run looks clean.

So a cheap generalization of your fix: after ParseNode, before GenerateAnswer, grep the parsed text for evidence of the requested fields (schema keys, or any tokens from the prompt's expected values), and warn when zero matches. That's ~10 lines, no LLM call, and it turns "NA for everything" from a silent failure into a self-diagnosing one — same spirit as the HTTP-status warning, just one layer further down.

On your two questions: warning (not raise) is the consistent choice, and I'd scope it to all page.goto call sites — a 404 is a 404 wherever it happens.

(Disclosure: I build a small open-source URL→clean-JSON extraction API that takes the deterministic route — a 404 surfaces as an error there, not as an "NA" field — so this silent-failure class is the exact problem I care about.)

@sahilkanger

@sahilkanger @NG-PR0JECT @paulhiltonmarketing-gif — thanks both, this is fixed. Implemented exactly as you two converged on it: warn, not raise, and at every page.goto call site.

PRs: #1135 (pre/beta) and #1136 (main).

1. HTTP status awareness in ChromiumLoader. The response from page.goto() is no longer discarded; ascrape_playwright, ascrape_playwright_scroll and ascrape_with_js_support all warn on status >= 400. As @sahilkanger noted, this is what the use_soup=True path has been doing all along — the default path just never did it.

Received HTTP 404 for https://en.wikipedia.org/wiki/Timpson_(company);
the scraped content is likely an error page, not the intended document.

2. Content-relevance check in ParseNode, for @NG-PR0JECT's point about the second failure mode — a 200 page can still reach the LLM with the target absent, and the HTTP check does nothing for that. After parsing, we now collect the terms the user asked about (schema field names + the significant words of the prompt) and warn when none of them appear:

None of the requested terms ['employee', 'employees', 'name', 'payroll', 'salary']
appear in the parsed content (167 chars). The source may be an error page, may
render its content with JavaScript, or the relevant section may have been dropped
while parsing; the model will most likely answer NA.

Kept at your suggested "zero matches" bar rather than a ratio, so it stays quiet when a page just phrases the answer differently from the question. Schema field names come from Pydantic v1/v2 models (including models nested in List[...]/Optional[...]), JSON Schema dicts, and lists of either; snake_case and camelCase are split into their parts. A prompt made only of generic scraping vocabulary yields no discriminative terms and is skipped rather than warned about.

Verified end to end on the two URLs from this issue:

https://en.wikipedia.org/wiki/Timpson_(company)     WARNING: Received HTTP 404 ...
                                                    RESULT: {'content': 'NA'}
https://en.wikipedia.org/wiki/Timpson_(retailer)    RESULT: {'content': 'The founding
                                                    year of Timpson is 1865.'}

No warning on the corrected URL from either guard. 19 new mock-based tests in tests/test_error_page_detection.py, wired into the CI workflow.

@paulhiltonmarketing-gif — for your immediate case, the fix is the URL: the article lives at Timpson_(retailer), not Timpson_(company). Once these land you would have seen that in the logs instead of a silent NA.

The content nesting you mentioned is a separate, documentation-level thing; leaving this issue open is not needed for it, I will track it on its own.

ci(release): 2.2.0-beta.8 [skip ci]

🎉 This issue has been resolved in version 2.2.0-beta.8 🎉

The release is available on:

v2.2.0-beta.8
GitHub release

Your semantic-release bot 📦🚀

ci(release): 2.2.2 [skip ci]

🎉 This issue has been resolved in version 2.2.2 🎉

The release is available on:

v2.2.2
GitHub release

Your semantic-release bot 📦🚀

fix(fetch): surface HTTP errors and missing content instead of answer…

ci(release): 2.2.4-beta.1 [skip ci]

🎉 This issue has been resolved in version 2.2.4-beta.1 🎉

The release is available on:

v2.2.4-beta.1
GitHub release

Your semantic-release bot 📦🚀

ci(release): 2.2.4 [skip ci]

🎉 This issue has been resolved in version 2.2.4 🎉

The release is available on:

v2.2.4
GitHub release

Your semantic-release bot 📦🚀

fix(fetch): surface HTTP errors and missing content instead of answer…

Navigation Menu

Repository navigation

SmartScraperGraph returns "NA"/blank for all fields even on pages with clearly present target content #1102

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SmartScraperGraph returns "NA"/blank for all fields even on pages with clearly present target content #1102

Description

Activity

sahilkanger commented on Jul 16, 2026

NG-PR0JECT commented on Aug 22, 2026

VinciGit00 commented on Aug 23, 2026

github-actions commented on Aug 23, 2026

github-actions commented on Aug 23, 2026

github-actions commented on Sep 7, 2026

github-actions commented on Sep 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions