Repository navigation

Add robots.txt rules from modules - #1355

Open

jwrosewell wants to merge 42 commits into

feature/13-removalsfrom

feature/14-robots-txt

Open

Add robots.txt rules from modules#1355
jwrosewell wants to merge 42 commits into
feature/13-removalsfrom
feature/14-robots-txt

Commits on Oct 9, 2026
- A module can add rules to robots.txt, and core writes the file
  jwrosewell
  committed
  31dd42d
- The seam probe supplies a robots.txt contributor
  jwrosewell
  committed
  0188f5d
- Every adapter serves /robots.txt and the refusal's response header
  jwrosewell
  committed
  3701fbc
- The guides describe [robots-txt]
  jwrosewell
  committed
  6d27af2
Commits on Oct 12, 2026
Commits on Oct 13, 2026
- Merge 13-removals into 14-robots-txt
  jwrosewell
  committed
  1f1e27a
- Merge split/13-removals into split/14-robots-txt
  Jamesr51d
  committed
  8038607
- Merge split/13-removals into split/14-robots-txt
  Jamesr51d
  committed
  35eb5ae
Commits on Oct 16, 2026
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  8e6c936
- Hold a robots.txt answer for the host the request was addressed to
  jwrosewell
  committed
  141ae9e
- Answer a robots.txt with no rule with 503
  jwrosewell
  committed
  a752abe
- Give a robots.txt contributor nothing of the crawler's connection
  jwrosewell
  committed
  5fbe25b
- Answer robots.txt on Fastly before the Edge Cookie steps and the filters
  jwrosewell
  committed
  4805bfc
- Test robots.txt through every adapter: HEAD, a held answer and the 503
  jwrosewell
  committed
  e8a2bf5
- The changelog and guides describe the robots.txt fixes
  jwrosewell
  committed
  f8dc375
- Say what the addressed host and the held key are made of
  jwrosewell
  committed
  86c07b8
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  f98707a
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  53f5e15
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  3e714f5
- Merge the review fixes into split/14-robots-txt
  jwrosewell
  committed
  02261b4
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  33c5826
Commits on Oct 17, 2026
- The changelog and guide say no adapter holds a robots.txt answer here
  jwrosewell
  committed
  c03143d
Commits on Oct 18, 2026
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  c0df9a6
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  8476fb1
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  9efa3bd
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  8fd5af9
Commits on Oct 19, 2026
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  314ce8c
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  062a9e9
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  c68fa2f
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  3427bc6
Commits on Oct 28, 2026
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  c4682a0
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  656ba5c
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  0738ea6
- The robots.txt route tests carry the sample policy
  jwrosewell
  committed
  5e79072
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  48a9659
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  8c9a87d
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  001b238
- Merge split/13-removals into split/14-robots-txt
  jwrosewell
  committed
  a3bda39

Commits

Commits on Oct 9, 2026

Commits on Oct 12, 2026

Commits on Oct 13, 2026

Commits on Oct 16, 2026

Commits on Oct 17, 2026

Commits on Oct 18, 2026

Commits on Oct 19, 2026

Commits on Oct 28, 2026