Skip to content

Fix: preserve explicit configuration in amap_domain - #2560

Open
GoldArowana wants to merge 2 commits into
unclecode:developfrom
GoldArowana:fix/amap-domain-config-20261115
Open

GoldArowana wants to merge 2 commits into
unclecode:developfrom
GoldArowana:fix/amap-domain-config-20261115

Conversation

@GoldArowana

Copy link
Copy Markdown

Summary

AsyncWebCrawler.amap_domain(domain, config=...) currently discards an explicit DomainMapperConfig when no keyword overrides are passed. For example, source="sitemap", max_urls=3, and include_subdomains=False become the defaults, so discovery can use different sources and limits from those requested.

Preserve the supplied config, create defaults only when it is None, and clone it only when applying keyword overrides. Mapper initialization and scan result/error propagation stay as before.

List of files changed and why

  • crawl4ai/async_webcrawler.py: make the three config paths explicit.
  • tests/test_amap_domain_config.py: cover supplied config before and after mapper initialization, non-mutating overrides, kwargs-only calls, fresh defaults, and errors.

How Has This Been Tested?

Exact-source validation run, Python 3.12, source 4c09792296344570909eab2601e6aba98736baf7:

  • The same six regression cases against develop 2c0b9e9cc6a7918314fe53a63693d1c2c6115f5a: 3 failed, 3 passed, with package import checked inside the test process.
  • python -m pytest tests/test_amap_domain_config.py tests/regression/test_reg_domain_mapper.py -m 'not network' -q: 9 passed, 13 deselected.
  • python -m black --check --diff tests/test_amap_domain_config.py and compile checks passed. The changed production method follows Black formatting; unrelated existing whole-file formatting differences are left untouched.
  • Six independently authored public-call control groups passed, including zero/false overrides, reuse, exception identity, and cancellation propagation.

These tests import the real package and config; the mapper transport is mocked. Browser and live-network discovery tests were not run. Validation used an isolated workflow in the author fork; no workflow changes are included in this PR.

Checklist:

  • My code follows the style guidelines of this project (changed method and test).
  • I have performed a self-review of my own code (Codex author review, followed by a separate Codex review).
  • I have commented my code, particularly in hard-to-understand areas (explicit branching replaces the ambiguous expression).
  • I have made corresponding changes to the documentation (not needed for restoring the existing config argument).
  • I have added/updated unit tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes (the scoped tests above passed in the author fork CI; local dependency setup was resource-deferred).

Implementation, test execution, and independent review were assisted by OpenAI Codex. No personal human review or validation is claimed.

Honor a supplied DomainMapperConfig when amap_domain receives no keyword overrides. Cover public configuration routing and unchanged defaults.

Assisted-by: OpenAI Codex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant