Repository navigation
feat(scrape): hint when an Alexandria provider covers the scraped site - #377
Open
nickscamara wants to merge 2 commits into
Open
nickscamara wants to merge 2 commits into
nickscamara wants to merge 2 commits into
Conversation
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
3 issues found across 4 files
Confidence score: 3/5
- In
src/utils/alexandria-hint.ts, one failed same-host scrape can cancel discovery for other concurrent scrapes, so a successful URL may get no provider hint. Scope cancellation to each lookup consumer. - In
src/utils/alexandria-hint.ts, a trailing slash on a cloud API URL makes the self-hosted check suppress the hint. Normalize the effective API URL before checking it. - In
README.md, the description suggests custom API URLs still trigger the Alexandria lookup. Document the custom-URL exclusion and the required provider argument.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="README.md">
<violation number="1" location="README.md:350">
P2: This description incorrectly implies that an API-key scrape against a self-hosted/custom `--api-url` performs the Alexandria lookup. Document the custom-URL exclusion, and show the provider argument required by the actual `firecrawl list <provider>` command.</violation>
</file>
<file name="src/utils/alexandria-hint.ts">
<violation number="1" location="src/utils/alexandria-hint.ts:85">
P2: A trailing slash on the cloud `--api-url` or `FIRECRAWL_API_URL` disables this hint because `isCustomApiUrl` compares the raw string. Normalize the effective API URL before applying the self-hosted check so `https://api.firecrawl.dev/` still performs discovery.</violation>
<violation number="2" location="src/utils/alexandria-hint.ts:86">
P2: Sharing the controller makes one failed same-host scrape cancel discovery for all concurrent scrapes, so `scrape a b` can successfully scrape one URL but print no provider hint when the other fails. Track lookup consumers or defer aborting until no successful scrape can still settle the shared lookup.</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
View guided diff | Turn on auto-fix | Re-trigger cubic
…ccept the cloud URL with a trailing slash Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
0 issues found across 4 files (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would require human review. Adds a best-effort Alexandria provider hint but enables a new /exchange/skills/resolve lookup by default for all keyed default-API scrapes; the acceptable cost/latency and route semantics need human sign-off.
View guided diff | Turn on auto-fix | Re-trigger cubic
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With an API key,
firecrawl scrape <url>now checks whether an Alexandria provider covers the page's site and, when one does, prints one line to stderr:Until now the only way to learn this from a scrape was passing
--domain-tools, which few callers do.How the lookup behaves
https://<host>/) to/exchange/skills/resolve, a free read-only discovery route.scrape a b candx downloadon one site make one request and print one line.--api-url, with--lockdown, and with--domain-tools(which already returns the tools).Verification
scrape.test.tscover the request shape, the hint text, once-per-host dedupe, failure and no-match silence, the skip cases and the late-lookup abort. The existing scrape tests now stubfetch.example.comscrape prints none.Summary by cubic
With an API key,
firecrawl scrapenow detects when an Alexandria provider covers the scraped site and prints a one-line hint to stderr naming the provider and how to run its tools. Until now the only way to learn this was--domain-tools, which few callers use.Behavior
/exchange/skills/resolveroute and never fails the scrape it accompanies.--api-url, with--lockdown, or when--domain-toolsis passed. The cloud API URL is accepted with or without a trailing slash.Written for commit 8aabc97. Summary will update on new commits.