Skip to content

Remove dandi-archive compatibility kludges that no known instance needs - #1973

Open
yarikoptic-gitmate wants to merge 6 commits into
masterfrom
claude/upbeat-babbage-fvwvgy
Open

yarikoptic-gitmate wants to merge 6 commits into
masterfrom
claude/upbeat-babbage-fvwvgy

Conversation

@yarikoptic-gitmate

Copy link
Copy Markdown
Collaborator

Summary

This follows up on the review of server-compatibility conditionals in #1963 ("Existing conditionals: removable now?"). Here is what each instance currently reports at /api/info/ (2026-10-28):

Instance version schema_version allowed_schema_versions
DANDI 1.1.1 0.7.0 reported
DANDI sandbox 1.1.0.post1.dev2+g4f8a26029 0.7.0 reported
EMBER 0.0.post1.dev7927+g9e4606812 0.7.0 reported
EMBER sandbox 0.0.post1.dev7926+g7614f3033 0.7.0 reported
LINC 0.0.post6586+g39387b8 0.6.9 not reported
LINC staging 0.0.post7100+g49fd9e7 0.6.9 not reported

LINC is still in use, so anything it needs stays.

Removed

  • The dandi-staging check in get_instance(). It could never run: known_instances[dandi_id] raises KeyError first, and on the command line -i only accepts known instances. The id itself was removed in 0.74.0 (Add support for all known instances in short URL format, removed dandi-staging and renamed EMBER* ones into EMBER-DANDI* #1750).
  • Retrying downloads on HTTP 400. This was added for Girder (download: we need to retry #87), which was retired in 2021. Retrying the same request does not help with a 400 from dandi-archive or S3.
  • The fallback for servers that do not report allowed_schema_versions. When the server's schema was newer than the client's, the client's own ALLOWED_INPUT_SCHEMAS was used in place of the missing field.
    • All DANDI and EMBER instances now report the field (Expose allowed schema versions at /info/ endpoint dandi-archive#2625, 0.19.0).
    • LINC does not report it, but it never reaches this code. That branch only runs when the server's schema is newer than the client's, and LINC's 0.6.9 is older than any schema dandi-cli supports (dandischema >= 0.12.0, i.e. schema >= 0.7.0).
    • A server without the field now accepts no older client schema. That is how dandi-cli behaved before the field existed.
    • The existing test only covered the fallback; new tests cover the field being present and absent.

Kept, with comments saying why

DEVELOPMENT.md

  • LINC is now named alongside DANDI and EMBER as an instance to stay compatible with.
  • A shell loop shows what each instance deploys.
  • A fallback can go once no known instance can reach it, not only once all instances support the feature.
  • EMBER's version cannot be compared with dandi-archive's any more than LINC's can, so deciding support by server version works only for DANDI's own instances.

Not addressed here

Since 0.74.0, dandi upload to LINC has failed in check_schema_version() with Server uses older incompatible schema version 0.6.9; client supports 0.7.0 (or 0.8.0 with dandischema 0.14). The cause is that 0.6.9 is not in ALLOWED_TARGET_SCHEMAS and is in a different MAJOR.MINOR band. Fixing it would mean dandischema downgrading metadata to 0.6.9 (today it goes no lower than 0.6.10), or LINC upgrading its deployment. This PR does not change that behaviour.

Testing

  • flake8, codespell and mypy dandi pass.
  • The unit tests for check_schema_version(), the metadata downgrade, _check_attempts_and_sleep(), dandi/tests/test_utils.py and dandi/cli/tests/test_instances.py pass locally.
    • test_utils.py::test_server_info fails there only because the environment cannot reach dandiarchive.org; it fails the same way on master.
  • The Docker integration tests were not run locally. None of the code changed here depends on them.

Generated by Claude Code

'dandi-staging' was removed from known_instances in 0.74.0 (#1750), and
the check meant to tell users so came after `known_instances[dandi_id]`,
which raises KeyError for it first.  On the command line `-i` is a choice
of known instances, so it never got there either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvaGRXmMFKsLrDKSpivgHD
The retry was added for Girder (#87), which was retired in 2021.  A 400 from
dandi-archive or S3 says the request is wrong, and repeating it does not
help.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvaGRXmMFKsLrDKSpivgHD
The x-amz-tagging header is still sent only if signed, without failing an
embargoed upload when it is not: LINC, which is still in use, runs a fork of
dandi-archive 0.3.115 that does not sign it (dandi/dandi-archive#2503 added it
in 0.12.14) and accepts embargoed Zarrs without it.  Name LINC as what the
TODO waits for, since DANDI and EMBER do sign it.

mkzarr() matches both error messages for a duplicate Zarr because current
dandi-archive sends both, "Zarr already exists" when it loses a race.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvaGRXmMFKsLrDKSpivgHD
When the server's schema is newer than the client's, check_schema_version()
looked the client's schema up in the server's `allowed_schema_versions`, or,
for servers before dandi-archive 0.19.0 (dandi/dandi-archive#2625), in the
client's own ALLOWED_INPUT_SCHEMAS.  DANDI and EMBER, production and sandbox,
all report the field now.  LINC (schema 0.6.9) does not, but its schema is
older than any client's (dandischema >= 0.12.0, schema >= 0.7.0), so it never
gets here.  A server without the field now allows no older client schema, as
before the field was introduced.

The existing test only exercised the fallback; give it the field, and test
both a server-provided list and its absence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvaGRXmMFKsLrDKSpivgHD
LINC, still in use, deploys a fork of dandi-archive 0.3.115 with schema 0.6.9,
so name it alongside DANDI and EMBER as an instance to stay compatible with,
and give the command showing what each instance deploys.  A fallback can go
when no known instance can reach it, not only when all support the feature,
as with `allowed_schema_versions`.  Replace the example comment quoting the
reworded Zarr tagging TODO.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvaGRXmMFKsLrDKSpivgHD
As reported by api-dandi.emberarchive.org/api/info/ on 2026-10-28.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PvaGRXmMFKsLrDKSpivgHD
@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.91%. Comparing base (e4647a6) to head (5213d34).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1973      +/-   ##
==========================================
+ Coverage   90.82%   90.91%   +0.09%     
==========================================
  Files          92       92              
  Lines       14190    14211      +21     
==========================================
+ Hits        12888    12920      +32     
+ Misses       1302     1291      -11     
Flag Coverage Δ
unittests 90.91% <100.00%> (+0.09%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants