fix(citations): Google moved the crawler docs out of Search Central - #160
Draft
jdevalk wants to merge 1 commit into
Draft
fix(citations): Google moved the crawler docs out of Search Central#160jdevalk wants to merge 1 commit into
jdevalk wants to merge 1 commit into
Conversation
Rotating-slice citation check (this run: agent-readiness, plus the two SEO / resilience pages that share the same drifted source). Every URL below was verified by fetching it today. Google's crawling documentation was migrated off Search Central onto a standalone "Crawling infrastructure" site, because the crawlers are shared across Shopping, News, Gemini, AdSense and more, not just Search. The changelog at developers.google.com/crawling/docs/changelog records two waves: 20 November 2025 (crawler overview, common crawlers, robots.txt spec) and 18 December 2025 (HTTP status codes, network errors). The old Search Central URLs still return 200 and still serve the content, so nothing here would fail the external-link job — but they now declare a canonical in the new tree, and the publisher on the citation was wrong. - agent-readiness-overview: overview-google-crawlers moved to /crawling/docs/crawlers-fetchers/overview-google-crawlers. - robots-for-ai-crawlers: the #google-extended anchor no longer exists on the overview page at all. Google-Extended is now documented on /crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended — so this one was a genuinely dead anchor, not just a moved page. - robots-txt: "How Google interprets the robots.txt specification" moved to /crawling/docs/robots-txt/robots-txt-spec. The companion "Introduction to robots.txt" did *not* move and still sits under Search Central, so it is left alone. - soft-404 and error-pages: http-network-errors moved to /crawling/docs/troubleshooting/http-status-codes. Both pages cited the same URL. The new page still covers soft 404s and 4xx de-indexing, which is what both pages lean on. Titles dropped the redundant "Google Search Central —" prefix now that the publisher field carries the right site name. `publisher` changed from "Google Search Central" to "Google Crawling Infrastructure" on the five moved citations. No `updated` bump: source hygiene only, no page text changed — matching #93/#94/#95/#101/#112/#120. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deploying specification-website with
|
| Latest commit: |
0fb267d
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://7b5f9cd6.specification-website.pages.dev |
| Branch Preview URL: | https://fix-citations-google-crawlin.specification-website.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Five citations repointed. Google migrated its crawling documentation off Search Central onto a standalone Crawling infrastructure site (
developers.google.com/crawling/), on the grounds that the crawlers serve Shopping, News, Gemini and AdSense as well as Search. The crawling docs changelog records two waves — 20 November 2025 (crawler overview, common crawlers, robots.txt spec) and 18 December 2025 (HTTP status codes, network errors).agent-readiness/agent-readiness-overviewsearch/docs/crawling-indexing/overview-google-crawlerscrawling/docs/crawlers-fetchers/overview-google-crawlersagent-readiness/robots-for-ai-crawlers…/overview-google-crawlers#google-extendedcrawling/docs/crawlers-fetchers/google-common-crawlers#google-extendedseo/robots-txtsearch/docs/crawling-indexing/robots/robots_txtcrawling/docs/robots-txt/robots-txt-specseo/soft-404search/docs/crawling-indexing/http-network-errorscrawling/docs/troubleshooting/http-status-codesresilience/error-pagespublishermoves from "Google Search Central" to "Google Crawling Infrastructure" on all five, and the twohttp-network-errorscitations drop the now-redundant "Google Search Central — " title prefix.Why now
This is the rotating-slice citation check for today's standards scan (slice: agent-readiness; the SEO and resilience pages came along because they cite the same moved document).
The external-link job would never have caught this. Every old URL still returns 200 and still serves the article — Google kept them serving and set the canonical to the new tree. Two things were nonetheless wrong:
overview-google-crawlers#google-extendedno longer resolves to anything: Google-Extended was moved to the separate common crawlers page. The citation pointed at a section that is not on that page any more.robots-for-ai-crawlersandagent-readiness-overvieware leaning on.The companion "Introduction to robots.txt" (
search/docs/crawling-indexing/robots/intro) did not move — verified, still titled Google Search Central — so it is deliberately left untouched.Status
No status changes. No
updatedbumps: source hygiene only, no page text changed, matching #93 / #94 / #95 / #101 / #112 / #120.npm run buildpasses (166 pages indexed); lint, Prettier andcheck:skillall pass.🤖 Generated with Claude Code