MOM_auto_mask_table problem for multi-instance cases

[iangrooms]

Trying to run an ensemble with 80 members and got the following error in the ocean log file:

FATAL from PE     0: READ_ASCII_FILE: number of lines in file MOM_auto_mask_table is greater than size(Content(:)).

The case root is

/glade/work/kboden/cases/g.e30_b06.G_JRA.TL319_t232_zstar_N75.2025.SKEB_DA.rr

The offending auto mask table can be found here

/glade/derecho/scratch/kboden/g.e30_b06.G_JRA.TL319_t232_zstar_N75.2025.SKEB_DA.rr/run/MOM_auto_mask_table

I haven't seen this kind of error before so I can't say much more. I'd appreciate any help.

[iangrooms]

Update: We tried again using a smaller ensemble size, and got a new error (still related to MOM_auto_mask_table):

WARNING from PE     0: open_ASCII_file: File MOM_auto_mask_table opened WRITEONLY already exists!

This is a multi-instance case, and most files have an instance index on them, e.g. MOM_input_0031. I wonder if maybe the MOM_auto_mask_table should also have an instance index?

[iangrooms]

We have worked around it by not using the auto mask table - specifically setting AUTO_MASKTABLE = False. This slows the model down, but allows us to proceed. In the long run this is still an issue worth fixing, in case anyone wants to use the auto mask table with a multi-instance run.

[alperaltuntas]

@iangrooms Not sure if the latest changes have fixed this issue (probably not), but I'm wondering if it's happening because each MOM6 instance gets only a very small number of cores, so no block can be eliminated. Do you remember how many ocean cores there are per MOM6 instance?

[iangrooms]

I believe that we're using the same number of cores per instance as in the single-instance cases, so 8 nodes = 1024 cores per instance. A 40 member ensemble takes up a lot of space on the machine.