Say why metadata failed to load and how to see the tracebacks - #1901
Open
adityasingh2400 wants to merge 1 commit into
Open
Say why metadata failed to load and how to see the tracebacks#1901adityasingh2400 wants to merge 1 commit into
adityasingh2400 wants to merge 1 commit into
Conversation
`dandi organize` warned "Failed to load metadata for N out of M files due to following types of exceptions: ConstructError. Details of the exceptions will be shown at DEBUG level" and stopped there. The user got an exception class name, no reason, and no way to find out how to reach DEBUG level. Most people running organize are not programmers and do not know about log levels or where the log file lives. The warning now names each offending path with its exception type and the exception's own message, capped at MAX_METADATA_ERRORS_SHOWN so a large batch does not flood the console, and it says that full tracebacks are in the log file and that 'dandi --log-level DEBUG organize ...' prints them to the console. The message is built by `format_metadata_load_failures()` so it can be tested without running a full organize. Closes dandi#1640
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1640
dandi organizewarnedFailed to load metadata for N out of M files due to following types of exceptions: ConstructError. Details of the exceptions will be shown at DEBUG leveland stopped there. The user got an exception class name, no reason, and no hint about how to reach DEBUG level or where the log file lives. As the reporter notes, most people running organize are not programmers.The warning now names each offending path with its exception type and the exception's own message, capped at
MAX_METADATA_ERRORS_SHOWN = 5so a large batch does not flood the console. It also says that full tracebacks are written to the log file, and thatdandi --log-level DEBUG organize ...prints them to the console.The message is built by
format_metadata_load_failures()so it can be tested without running a full organize.Verified against the base ref. On
masterthe integration test fails because the reason never appears in the warnings. On this branch it does. One note on that check: the two unit tests import the new symbols, so a plain revert fails at collection rather than on behavior. To get a genuinely behavioral before signal I revertedorganize.pyto the base ref and also removed the two new imports from the test file, then ran only the integration test, which fails on the old message content. Both were restored afterward.New tests are marked
@pytest.mark.ai_generated.dandi/tests/test_organize.pyexcluding integration gives 34 passed, 14 deselected.AI assistance disclosure: this change was written with the help of Claude Code, and the added tests are marked
ai_generatedas CLAUDE.md asks. I reviewed and tested everything before submitting.