Remove load_from/save_to folder_metadata in save/load operations#4689
Open
alejoe91 wants to merge 17 commits into
Open
Remove load_from/save_to folder_metadata in save/load operations#4689alejoe91 wants to merge 17 commits into
load_from/save_to folder_metadata in save/load operations#4689alejoe91 wants to merge 17 commits into
Conversation
alejoe91
added
core
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In the direction of #4671, this PR starts by rmeoving the unused mechanism to save to / load from metadata.
In addition, this PR also fixes saving timestamps in proper binary format.
For recording objects, the
_extra_metadata_to_folderwas saving timestamps tonpyfiles, but this is now handled by thewrite_binaryandBinaryRecordingExtractor, which writes in chunk to proper binary files. The ~binaryRecordingExtractoracceprs afile_timestamps_pathwith the location of the raw timestamps. Note that this is absolutely necessary for chronic data, since the old version was saving the entire timestamps array with numpy and this cannot be done with very long recordings. Old datasets are handled with the_handle_extractor_backward_compatibility`.To further cleanup, this PR adds
save_properties_to/load_properties_from_binary_folderto remove duplicated code for saving and loading properties to numpy arrays.The cross serialization check was extended to always run against major past versions, to test a recording with timestamps, and to save recordings in parallel for
zarr/binary_folder.