Skip to content

Fix javadoc crawler reading a Maven Central index that stopped updating - #8796

Open
thswlsqls wants to merge 3 commits into
open-telemetry:mainfrom
thswlsqls:fix/javadoc-crawler-stale-maven-index
Open

Fix javadoc crawler reading a Maven Central index that stopped updating#8796
thswlsqls wants to merge 3 commits into
open-telemetry:mainfrom
thswlsqls:fix/javadoc-crawler-stale-maven-index

Conversation

@thswlsqls

Copy link
Copy Markdown
Contributor

Fixes #8792

Description

  • JavaDocsCrawler queries search.maven.org/solrsearch, whose index stopped updating on 2025-06-06 and reports latestVersion=1.51.0 for every io.opentelemetry artifact.
  • Switch to central.sonatype.com/solrsearch. It is absent from Sonatype's REST API guide, but their FAQ says that index refreshes within minutes of a release while search.maven.org's does not (https://central.sonatype.org/faq/what-happened-to-search-maven-org/), and Apache's maven-indexer ships the same host as SmoSearchBackendFactory.CSC_SMO_URI.
  • The hosts read start differently, so getArtifacts sends one request with rows=500 and no start, failing if it receives fewer documents than numFound.
  • Runs 34300382073 and 34177613796 each log 246 Skipping crawling lines and no Crawling line, yet exit success. The fix restores 168 crawl requests across five groups.
  • Triggering javadoc.io's lazy loading is the module's intent, not measured here.
  • Follow-up idea: fail the job when nothing is crawled.

Testing done

  • Replaced testGetArtifactsHandlesPagination with testGetArtifactsUsesASingleRequest (one request; URI has rows=500, no start=) and testGetArtifactsFailsOnIncompleteResponse (short response throws IOException naming the group).
  • ./gradlew :javadoc-crawler:check passes; both new tests fail against the unpatched code.
  • Unpublished module: no apidiff or API change.

@thswlsqls
thswlsqls marked this pull request as ready for review September 11, 2026 03:40
@thswlsqls
thswlsqls requested a review from a team as a code owner September 11, 2026 03:40
@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Sep 11, 2026

Copy link
Copy Markdown

Pull request dashboard status

Waiting on maintainers · refreshed 2026-09-12 12:19 UTC

Merge when ready.

Status above doesn't look right?
  • Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

@codecov

codecov Bot commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.26%. Comparing base (fb5023e) to head (e8cce37).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #8796      +/-   ##
============================================
+ Coverage     91.25%   91.26%   +0.01%     
- Complexity    10512    10525      +13     
============================================
  Files          1007     1008       +1     
  Lines         28386    28465      +79     
  Branches       3591     3617      +26     
============================================
+ Hits          25904    25980      +76     
+ Misses         1687     1685       -2     
- Partials        795      800       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread javadoc-crawler/src/main/java/io/opentelemetry/javadocs/JavaDocsCrawler.java Outdated
thswlsqls and others added 2 commits September 12, 2026 19:15
…csCrawler.java

Co-authored-by: Jay DeLuca <jaydeluca4@gmail.com>
getArtifacts now fails when a 200 response has no response.numFound,
so that a changed endpoint shape cannot silently skip the completeness
check. Cover that path with a test that expects an IOException naming
numFound and the group. The test fails against the previous code.

Generated-by: Claude Code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Javadoc crawler skips every artifact because its Maven Central index stopped updating

2 participants