CESM2.1.5 BSSP126cmip6 initialization crash on Derecho (seq_infodata_mod.F90)

[usmanummuqulthum]

What version of the code are you using?

• release-cesm2.1.5

Have you made any changes to files in the source tree?

• No changes were made to the core source tree. Custom configurations were isolated strictly to the case directory via the setup script.

Describe every step you took leading up to the problem: (I have attached a .csh file)

1. Created a new case from the CIME scripts directory
2. Modified time and run settings via xmlchange:
3. Ran ./case.setup.
4. Appended daily history stream modifications to the user namelists.
5. Ran ./preview_namelists followed by ./case.build, which completed successfully with the message MODEL BUILD HAS FINISHED SUCCESSFULLY.
6. Verified that reference case restart pointer files (rpointer.*) were successfully prestaged via ./check_input_data.
7. Submitted the job with ./case.submit running on 1152 cores (9 nodes).

If this is a port to a new machine: Please attach any files you added or changed for the machine port and tell us the compiler version you are using on this machine.

• Not a new port. Running on the standard NCAR Derecho environment using the Intel compiler and cray-mpich.

Please attach any log files showing error messages or other useful information.

• Paths for reference: * Case Directory: ~/glade/derecho/GGW_cases/SSP_SCENERIOS/cesm2.1.5.BSSP126cmip6.f09_g17.BSSP126_002_Run_215/
• Run Directory: /glade/derecho/scratch/usman/cesm2.1.5.BSSP126cmip6.f09_g17.BSSP126_002_Run_215/run/
• Attached Files: SSP_03.csh, software.environment.txt,

• Describe your problem or question:
• I am running a transient hybrid/branch SSP1-2.6 simulation on Derecho using 1152 cores. The model builds perfectly but consistently crashes roughly 1.5 minutes after the run begins. The high-level output captures a generic launcher failure:

ERROR: RUN FAIL: Command 'mpiexec ... cesm.exe' failed

Looking deeper into cesm.log, all MPI processes are stalling or getting terminated with a SIGTERM during the initialization handshake loop inside the CIME driver. The failure points directly to an MPI_Bcast call. I have checked the PBS batch logs and found no evidence of an Out-Of-Memory (OOM) event.

The bottom of the stack trace is as follows:

shr_mpi_mod_mp_sh 568 shr_mpi_mod.F90<br>seq_infodata_mod_ 2525 seq_infodata_mod.F90<br>component_mod_mp_ 285 component_mod.F90<br>cime_comp_mod_mp_ 1237 cime_comp_mod.F90<br>MAIN__ 114 cime_driver.F90<br>...<br>forrtl: error (78): process killed (SIGTERM)<br>
Does this specific crash inside seq_infodata_mod.F90 indicate a known environment/compiler issue on Derecho, or does it point to an initialization mismatch with the hybrid reference case startup?

 

[oleson]

I see this in your lnd_in:

fsurdat = '/glade/u/home/usman/scratchdir/landuse.timeseries_0.9x1.25_SSP1-2.6_78pfts_CMIP6_simyr1850-2100_c190214.nc'

That is not a surface dataset file, it is a landuse timeseries file. You should be using a "surfdata" file for fsurdat.
This is correct in your lnd_in:

flanduse_timeseries = '/glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/landuse.timeseries_0.9x1.25_SSP1-2.6_78pfts_CMIP6_simyr1850-2100_c190214.nc'

 

[usmanummuqulthum]

Thank you for pointing that out. I see that I accidentally mixed up the variables and assigned a landuse timeseries file to fsurdat instead of a static baseline surface dataset.

I looked into the directory path you provided (/glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/). There are a lot of different surfdata files listed there. For my specific grid resolution and plant functional type setup (0.9x1.25and 78pfts), I see options for simyr1850, simyr2000, and simyr2014. Since my simulation is a transient future run that starts in the year 2015, how do I choose the correct baseline file from this list to ensure it remains compatible across different future SSP scenarios?

Thanks for your help!

 

[slevis]

If your simulation starts in 2015, the model should use the corresponding pft data found in the landuse file, so I think any of the above fsurdat files should work.

 

[usmanummuqulthum]

Hi a I am running a transient SSP1-2.6 simulation using CESM2.1.5 (compset BSSP126cmip6, resolution f09_g17) starting in year 2015. In my user_nl_clm, I have pointed fsurdat to the 2014 historical surface dataset (surfdata_0.9x1.25_hist_78pfts_CMIP6_simyr2014_c190516.nc) and configured flanduse_timeseries to use the corresponding SSP1-2.6 transient file (landuse.timeseries_0.9x1.25_SSP1-2.6_78pfts_CMIP6_simyr1850-2100_c190214.nc). However, the model crashes during initialization with a dynpft_check_consistency mismatch between PCT_NAT_PFT at initial time and that obtained from surface dataset error at global gridcell index 6606. While the log suggests I can bypass this by setting check_dynpft_consistency = .false. in user_nl_clm, I want to confirm if disabling this check is the standard, scientifically sound practice for starting an SSP run in 2015, or if there is a more compatible baseline fsurdat file I should be using instead to ensure physical consistency. Thanks for any advice!
Relevant Paths in my Case Configuration:

• fsurdat: /glade/u/home/usman/scratchdir/surfdata_0.9x1.25_hist_78pfts_CMIP6_simyr2014_c190516.nc
• flanduse_timeseries: /glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/landuse.timeseries_0.9x1.25_SSP1-2.6_78pfts_CMIP6_simyr1850-2100_c190214.nc
• Case Run Directory: /glade/derecho/scratch/usman/cesm2.1.5.BSSP126cmip6.f09_g17.BSSP126_002_Run_215_Test/run/

 

[oleson]

I believe the CMIP6 defaults for that compset and resolution are:

fsurdat = '/glade/p/cesmdata/cseg/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/surfdata_0.9x1.25_hist_78pfts_CMIP6_simyr1850_c190214.nc'

which is a link to:

/glade/campaign/collections/gdex/data/d651077/cesmdata/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/landuse.timeseries_0.9x1.25_SSP1-2.6_78pfts_CMIP6_simyr1850-2100_c190214.nc

and

flanduse_timeseries = '/glade/p/cesmdata/cseg/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/landuse.timeseries_0.9x1.25_SSP1-2.6_78pfts_CMIP6_simyr1850-2100_c190214.nc'

which is a link to:

/glade/campaign/collections/gdex/data/d651077/cesmdata/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/landuse.timeseries_0.9x1.25_SSP1-2.6_78pfts_CMIP6_simyr1850-2100_c190214.nc

I got this information from the CMIP6 case directory:

/glade/work/cmip6/cases/ScenarioMIP/b.e21.BSSP126cmip6.f09_g17.CMIP6-SSP1-2.6.101

I assume you would get the same thing out of the box for that compset and resolution with release-cesm2.1.5, but I'm unable to confirm that as Derecho is down today.