BWSSP245cmip6 f19_g17 hangs on day 2

[Zhuyi]

What version of the code are you using?
CESM2.1.5

Have you made any changes to files in the source tree?
No

Describe every step you took leading up to the problem:
./create_newcase --case b.e21.f19_g17.BWSSP245cmip6 --res f19_g17 --compset BWSSP245cmip6 --run-unsupported
./case.setup
./case.build --skip-provenance-check
./case.submit

If this is a port to a new machine: Please attach any files you added or changed for the machine port (e.g., config_compilers.xml, config_machines.xml, and config_batch.xml) and tell us the compiler version you are using on this machine.
Please attach any log files showing error messages or other useful information.
This a port to a new machine and I've attached the config_compilers.xml, config_machines.xml, and config_batch.xml.
Compiler version: compiler/intel/2017.5.239

Describe your problem or question:
The model hangs on the second day of the simulation without producing any explicit error messages or updating on log files (attached). The last component that produces output before the hang is GLC. I can confirm that the machine port itself is likely not the issue, since I was able to successfully run the FC2010climo compset.

Also, since I intend to run the model at f19_g17 resolution, I have only been able to locate initial condition files for the BWSSP245cmip6 compset at f09_g17 resolution. Could anyone please advise where I can obtain the f19_g17 initial condition files for 2015-01-01?

 

[Zhuyi]

The log files and configuration files are attached here.

 

[sacks]

Thank you for your description and attaching the log files. One log file I don't see is the cesm log file. There is often information there that can be useful in diagnosing problems. You should have a cesm log file from your case; I suggest looking in there or attaching it. (If you don't, that could indicate some issue on your system.)

Without seeing what's in your cesm log file, there are a few things that I would try initially. The fact that it seems to be hanging at the end of the first day suggests there may be an issue with I/O or the GLC component.

• This may be an out-of-memory issue. Do you have the flexibility on your system to give the case more processors? If so, I would try that first. Try giving twice the number of processors or even more.
• Try removing the custom output from the user_nl_cice file and the user_nl_clm file, by removing the settings in those files (I don't think the user_nl_clm file should be responsible, since it looks like that is just setting additional monthly and annual output, not daily, but try it anyway to be safe). If you have made any other output changes in other user_nl files, remove those as well
• Try removing CISM (the ice sheet model) by creating a case using --compset SSP245_CAM60%WCTS_CLM50%BGC-CROP-CMIP6WACCMDECK_CICE%CMIP6_POP2%ECO%NDEP_MOSART_SGLC_WW3 (this compset long name is the same as the compset you started with, but uses SGLC - i.e., a stub glc model - instead of CISM).
• If none of these work, then you could try some other, slightly simpler configurations to see what works and what fails. For example, does a B1850 compset work?

Regarding initial conditions files at this resolution: I think there probably are no out-of-the-box initial conditions files for this compset at f19_g17 resolution. This is related to your need to specify --run-unsupported: this is not a scientifically supported configuration at this resolution, and so there may be no spun-up initial conditions for this resolution. But I'll reach out to a couple of other people to see if others have more information on this.