Conversation
ESMF_RegridWeightGen benefits from many MPI tasks (128-512) but the rest of forcing processing runs efficiently on a single node. This change separates weight-generation into BuildMappingFile steps that can request large node allocations while processing steps run at ntasks=1, avoiding waste of hundreds of node-hours. Additionally, mapping files are now reusable across scenarios since they depend only on ice_sheet + mali_mesh_name + method_remap. Users can set mapping_files_path in the config to reuse weights from previous runs, and all built weights are automatically copied to output_base_path/mapping_files/ for future reuse. Changes: - Add BuildMappingFile step to atmosphere, ocean_thermal, and fracture test cases as first step (runs at esmf_ntasks) - Update all processing steps to consume weights via symlinks from ../build_mapping_file/map_*.nc instead of building them inline - Add optional mapping_files_path config option to [ismip7] section for cross-run weight reuse - Remove build_mapping_file calls from all processing step run() methods and add weight files as input dependencies in setup() This enables allocating large node counts only for the 5-10 minute weight-generation step while running the hours-long processing steps on single nodes. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
trhille
force-pushed
the
landice/ismip7_forcing_weight_gen_step
branch
from
September 22, 2026 19:31
7456b4d to
435c11c
Compare
Update both user and developer guides to document the new BuildMappingFile steps and workflow optimization strategies for the ISMIP7 forcing test group. User's Guide changes: - Update test case descriptions to include build_mapping_file as first step - Add new "Workflow Optimization" section explaining resource allocation strategies (separate allocations for weight-gen vs processing) - Document weight reuse across scenarios via mapping_files_path config option - Add mapping_files_path to config options documentation - Update atmosphere, ocean_thermal, and fracture sections to describe the BuildMappingFile steps Developer's Guide changes: - Update framework section to clarify build_mapping_file is now called only from BuildMappingFile steps, not from processing steps - Document BuildMappingFile classes for atmosphere, ocean_thermal, and fracture test cases - Explain architectural separation: weight-generation runs at esmf_ntasks, processing steps run at ntasks=1 - Update step listings and descriptions for all three test cases Key documentation points: - ESMF_RegridWeightGen runs for 5-10 minutes on 128-512 tasks - Processing steps run for hours on single nodes - Mapping files depend only on ice_sheet + mali_mesh_name + method_remap - Weights built for one scenario can be reused for other scenarios with the same mesh and remapping method Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
trhille
force-pushed
the
landice/ismip7_forcing_weight_gen_step
branch
from
September 22, 2026 19:34
435c11c to
333849a
Compare
Collaborator
Author
|
Closing this in favor of #997, which contains a refactored version of these changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ESMF_RegridWeightGen benefits from many MPI tasks across multiple nodes but the rest of forcing processing often only requires a single node. This change separates weight-generation into BuildMappingFile steps that can request large node allocations while processing steps run at ntasks=1, avoiding waste of hundreds of node-hours.
Additionally, mapping files are now reusable across scenarios since they depend only on ice_sheet + mali_mesh_name + method_remap. Users can set mapping_files_path in the config to reuse weights from previous runs, and all built weights are automatically copied to output_base_path/mapping_files/ for future reuse.
Changes:
This enables allocating large node counts only for the 5-10 minute weight-generation step while running the hours-long processing steps on single nodes.
Checklist
Testingin this PR) any testing that was used to verify the changes