Skip to content

Give a descriptive error when a DOI cannot be resolved to citation metadata - #1900

Open
adityasingh2400 wants to merge 1 commit into
dandi:masterfrom
adityasingh2400:fix-1855
Open

Give a descriptive error when a DOI cannot be resolved to citation metadata#1900
adityasingh2400 wants to merge 1 commit into
dandi:masterfrom
adityasingh2400:fix-1855

Conversation

@adityasingh2400

Copy link
Copy Markdown

Fixes #1855

update-dandiset-from-doi crashed with a bare json.JSONDecodeError: Expecting value: line 1 column 1 (char 0) whenever doi.org answered with anything other than JSON.

There are two separate problems behind that traceback.

The first is a header bug. The CSL Accept header was set on the RESTFullAPIClient session, but RESTFullAPIClient.request() sets accept: application/json on the request whenever json_resp is true, and in requests a per-request header wins over a session header. The resolver therefore never saw the citation format we meant to ask for. The checked-in VCR cassettes record this, every captured request carries accept: application/json and never the CSL type. Crossref happens to serve JSON for that anyway, because it redirects to api.crossref.org/.../transform, which is why the existing tests pass. A registration agency that does not will redirect to the landing page and return HTML with a 200, which is the reported failure on a DataCite 10.48324 DOI.

The second is that there was no error handling at all on that path, so the user got a JSONDecodeError raised from inside requests with nothing naming the DOI.

The fetch moves into fetch_doi_citation_metadata(), which requests the raw response so the intended CSL Accept header survives, and turns a 404, any other HTTP error, a non-JSON body, and a non-object body each into a click.ClickException naming the DOI, the URL, and the content type actually received. normalize_doi() now accepts a bare DOI, a doi: URI, or a resolver URL, and rejects anything else with a click.UsageError instead of a traceback. That also fixes the relatedResource record, whose url was built as https://doi.org/{doi} and came out doubled when the user passed a resolver URL. The lookup now happens before connecting to the archive, so a bad DOI fails fast without needing credentials.

Only title, abstract, and author[*].given/family/ORCID/affiliation are read, and all of those are present in both CSL JSON and the Crossref record the cassettes captured, so replaying the existing cassettes is unaffected. vcrpy matches on method and URI, not headers.

Verified against the base ref with doi.org mocked to serve HTML at 200. Before, the run ends in requests.exceptions.JSONDecodeError from dandiapi.py line 326. After, it reports that the DOI answered with text/html instead of CSL JSON and explains that the registration agency likely does not serve citation metadata. The Accept header actually sent went from application/json to application/vnd.citationstyles.csl+json; charset=utf-8.

New tests are marked @pytest.mark.ai_generated. They give 18 passed. The 6 deselected are the VCR test_update_dandiset_from_doi cases, which need the docker archive fixture and could not run locally.

AI assistance disclosure: this change was written with the help of Claude Code, and the added tests are marked ai_generated as CLAUDE.md asks. I reviewed and tested everything before submitting.

…n metadata

`dandi service-scripts update-dandiset-from-doi` crashed with a bare
`json.JSONDecodeError: Expecting value: line 1 column 1 (char 0)`
traceback whenever doi.org answered with anything other than JSON.

Two things were wrong.

The CSL Accept header was set on the `RESTFullAPIClient` session, but
`RESTFullAPIClient.request()` sets `accept: application/json` on the
request itself whenever `json_resp` is true, and per-request headers win
over session headers. The resolver therefore never saw the citation
format we meant to ask for. The checked-in VCR cassettes record this:
every captured request carries `accept: application/json`. Crossref
happens to serve JSON for that, which is why the existing tests pass,
but a resolver that does not will redirect to the landing page and
return HTML with a 200.

And when that happened there was no error handling at all, so the user
got a `JSONDecodeError` out of the requests internals with nothing
pointing at the DOI.

The fetch now moves into `fetch_doi_citation_metadata()`, which requests
the raw response so the intended CSL Accept header survives, and turns a
404, another HTTP error, a non-JSON body, and a non-object body each
into a `click.ClickException` naming the DOI, the URL, and the content
type received.

`normalize_doi()` additionally accepts the DOI as a bare DOI, a `doi:`
URI, or a resolver URL, and rejects anything else with a
`click.UsageError` instead of a traceback. That also fixes the
`relatedResource` record, whose url was built as
`https://doi.org/{doi}` and so came out doubled when the user passed a
resolver URL.

The lookup now happens before connecting to the archive, so a bad DOI
fails fast without needing credentials.

Closes dandi#1855
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

update-dandiset-from-doi errors non-descriptively

1 participant