Skip to content

Reject nonexistent requested paths during asset discovery - #1932

Merged
yarikoptic merged 9 commits into
dandi:masterfrom
AtomicGlance:fix/1094-missing-path
Nov 2, 2026
Merged

yarikoptic merged 9 commits into
dandi:masterfrom
AtomicGlance:fix/1094-missing-path

Conversation

@AtomicGlance

@AtomicGlance AtomicGlance commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

find_dandi_files() currently allows nonexistent paths to be silently ignored or represented as assets. This change checks every requested root before yielding any files and raises FileNotFoundError with the offending filename.
The check uses lstat() so broken symlinks remain discoverable. This preserves DANDI’s missing-content handling for unfetched DataLad files. Existing empty directories still return no assets, and no additional recursive walk is introduced.
Tests cover missing paths with recognized, unrecognized, and hidden names; both allow_all settings; multiple requested roots; empty directories; and broken symlinks. A CLI regression test also confirms that dandi validate retains its existing missing-path error.
Validation on Windows with Python 3.13:

  • File tests excluding upload integration tests: 28 passed, 1 skipped.
  • Focused discovery and CLI tests: 12 passed, 1 skipped.
  • Black, isort, flake8, and mypy dandi passed.

@codecov

codecov Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.91%. Comparing base (152e941) to head (c9efab4).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1932      +/-   ##
==========================================
+ Coverage   90.84%   90.91%   +0.07%     
==========================================
  Files          92       92              
  Lines       14190    14223      +33     
==========================================
+ Hits        12891    12931      +40     
+ Misses       1299     1292       -7     
Flag Coverage Δ
unittests 90.91% <100.00%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AtomicGlance

Copy link
Copy Markdown
Contributor Author

This PR is ready for review, but GitHub won't let me remove its draft status, through either the website or CLI. Could you please mark it ready and add the patch label? All other checks are passing. Thank you!

@AtomicGlance
AtomicGlance marked this pull request as ready for review October 1, 2026 15:48
@AtomicGlance

Copy link
Copy Markdown
Contributor Author

This PR is ready for review, but GitHub won't let me remove its draft status, through either the website or CLI. Could you please mark it ready and add the patch label? All other checks are passing. Thank you!

@yarikoptic Hi, can you have a look at this?

Comment thread dandi/files/__init__.py Resolved
@AtomicGlance

Copy link
Copy Markdown
Contributor Author

Hi @yarikoptic, thanks again for the review! Could you please rerun the failed and cancelled jobs in the latest Tests workflow, and the cancelled label check, when you have a chance? Both failed Windows jobs report that the hosted runner lost communication with GitHub; neither provides a test failure traceback. I tried requesting a rerun, but GitHub returned a 403 because it requires repository admin rights.

The explicit lexists() fix is pushed and the review conversation is addressed. Thanks for your help!

@AtomicGlance

Copy link
Copy Markdown
Contributor Author

Hi @yarikoptic, a gentle follow-up when you have a moment: could you please rerun the failed and cancelled jobs in the Tests workflow and the cancelled label check? The latest results are still from October 5. Both failed Windows jobs reported a runner disconnection; your requested lexists() change is in place. Once the rerun finishes, I'd appreciate another look at the PR.

My related Archive fix, dandi/dandi-archive#2914, is also waiting for review. Only the deployment-preview check is reported there, so I can't claim the backend suite is green. If someone else is better placed to review it or enable the backend tests, could you point me their way?

Thanks again for your time and guidance!

@yarikoptic

Copy link
Copy Markdown
Member

clicking "Update branch". Next time if you need rerun -- just make sure it is up to date with current master and/or rewrite/force push last commit.

@yarikoptic yarikoptic added UX patch Increment the patch version when merged labels Oct 12, 2026
@AtomicGlance

Copy link
Copy Markdown
Contributor Author

Hi @yarikoptic, a quick follow-up: all 37 checks are now passing, including patch coverage, and your requested change is in place. Could you take another look when you have a chance? Thanks again for your help getting this into shape!

@AtomicGlance

Copy link
Copy Markdown
Contributor Author

Hi @yarikoptic, a quick follow-up: all 37 checks are now passing, including patch coverage, and your requested change is in place. Could you take another look when you have a chance? Thanks again for your help getting this into shape!

Hi @yarikoptic, would you please take a look at this again.

@yarikoptic-gitmate yarikoptic-gitmate left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the fix is correct and minimal, and lexists() keeps unfetched DataLad/git-annex symlinks discoverable. Locally, all 8 test_find_dandi_files_missing_path cases fail against master's find_dandi_files and pass here. The related suites also pass (test_files, test_cmd_validate, test_dandiset, test_move, test_upload_discovery, validate/tests/test_core).

Two optional suggestions:

  1. test_validate_missing_path doesn't reach the changed code. dandi validate rejects the path in Click (LinkAwarePath(lexists=True)) before find_dandi_files() runs, so the test passes on master too. What changed is the Python API: dandi.validate._core.validate(".../missing.nwb") used to yield pynwb.GENERIC ("Could not find the file"), an NWB Inspector read error and DANDI.NON_DANDI_FILENAME, and now raises FileNotFoundError. A test on validate() would cover that.
  2. find_dandi_files() is a generator, so the error comes on the first next(), not at call time. It may be worth saying so in the docstring, along with a .. versionchanged:: note.

Not for this PR: dandi upload still ignores missing paths without saying so, because upload() filters dandiset.assets() with under_paths() instead of passing the paths to find_dandi_files(). I filed that as #1974, including how --sync affects what the right behavior is.


Generated by Claude Code

@yarikoptic

Copy link
Copy Markdown
Member

as overall I think it is getting us in the right direction, I will merge this PR as is, but feel welcome to follow up with the one to address the @yarikoptic-gitmate's comments

@yarikoptic yarikoptic left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

potentially with follow ups to improve

@yarikoptic
yarikoptic merged commit 6b531f5 into dandi:master Nov 2, 2026
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

patch Increment the patch version when merged UX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

make find_dandi_files raise FileNotFoundError exception if path is invalid

3 participants