Skip to content

feat(scrape): hint when an Alexandria provider covers the scraped site - #377

Open
nickscamara wants to merge 2 commits into
mainfrom
feat/scrape-alexandria-site-hint
Open

nickscamara wants to merge 2 commits into
mainfrom
feat/scrape-alexandria-site-hint

Conversation

@nickscamara

@nickscamara nickscamara commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

With an API key, firecrawl scrape <url> now checks whether an Alexandria provider covers the page's site and, when one does, prints one line to stderr:

Alexandria has a structured-data provider for fred.stlouisfed.org (fred). For typed records instead of page text, list its tools with `firecrawl list fred`, then run one with `firecrawl scrape fred/<capability> --options '{...}'`.

Until now the only way to learn this from a scrape was passing --domain-tools, which few callers do.

How the lookup behaves

  • It starts beside the scrape and posts only the host (https://<host>/) to /exchange/skills/resolve, a free read-only discovery route.
  • After a successful scrape it waits at most 500 ms for the answer; a late lookup is aborted so it never keeps the process alive. A failed or empty lookup prints nothing, and a failed scrape prints no hint.
  • Each host is looked up and hinted once per run, so scrape a b c and x download on one site make one request and print one line.
  • It is skipped in keyless mode, against a self-hosted --api-url, with --lockdown, and with --domain-tools (which already returns the tools).

Verification

  • New unit tests in scrape.test.ts cover the request shape, the hint text, once-per-host dedupe, failure and no-match silence, the skip cases and the late-lookup abort. The existing scrape tests now stub fetch.
  • Ran the built CLI against the live API: a FRED series URL prints the hint and an example.com scrape prints none.

Open in Capy


Summary by cubic

With an API key, firecrawl scrape now detects when an Alexandria provider covers the scraped site and prints a one-line hint to stderr naming the provider and how to run its tools. Until now the only way to learn this was --domain-tools, which few callers use.

Behavior

  • The lookup posts only the site's host to the free /exchange/skills/resolve route and never fails the scrape it accompanies.
  • Each host is checked once per run; concurrent scrapes of the same host share one lookup that stays alive until all of them finish, and a response arriving after 500 ms is aborted so it never keeps the process alive.
  • The check is skipped in keyless mode, against a self-hosted --api-url, with --lockdown, or when --domain-tools is passed. The cloud API URL is accepted with or without a trailing slash.

Written for commit 8aabc97. Summary will update on new commits.

View guided diff Turn on auto-fix

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found across 4 files

Confidence score: 3/5

  • In src/utils/alexandria-hint.ts, one failed same-host scrape can cancel discovery for other concurrent scrapes, so a successful URL may get no provider hint. Scope cancellation to each lookup consumer.
  • In src/utils/alexandria-hint.ts, a trailing slash on a cloud API URL makes the self-hosted check suppress the hint. Normalize the effective API URL before checking it.
  • In README.md, the description suggests custom API URLs still trigger the Alexandria lookup. Document the custom-URL exclusion and the required provider argument.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="README.md">

<violation number="1" location="README.md:350">
P2: This description incorrectly implies that an API-key scrape against a self-hosted/custom `--api-url` performs the Alexandria lookup. Document the custom-URL exclusion, and show the provider argument required by the actual `firecrawl list <provider>` command.</violation>
</file>

<file name="src/utils/alexandria-hint.ts">

<violation number="1" location="src/utils/alexandria-hint.ts:85">
P2: A trailing slash on the cloud `--api-url` or `FIRECRAWL_API_URL` disables this hint because `isCustomApiUrl` compares the raw string. Normalize the effective API URL before applying the self-hosted check so `https://api.firecrawl.dev/` still performs discovery.</violation>

<violation number="2" location="src/utils/alexandria-hint.ts:86">
P2: Sharing the controller makes one failed same-host scrape cancel discovery for all concurrent scrapes, so `scrape a b` can successfully scrape one URL but print no provider hint when the other fails. Track lookup consumers or defer aborting until no successful scrape can still settle the shared lookup.</violation>
</file>

Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread README.md Resolved
Comment thread src/utils/alexandria-hint.ts Resolved
Comment thread src/utils/alexandria-hint.ts Resolved
…ccept the cloud URL with a trailing slash

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 4 files (changes from recent commits).

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Shadow auto-approve: would require human review. Adds a best-effort Alexandria provider hint but enables a new /exchange/skills/resolve lookup by default for all keyed default-API scrapes; the acceptable cost/latency and route semantics need human sign-off.

View guided diff | Turn on auto-fix | Re-trigger cubic

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant