CESM2 crash without obvious error message

[James King]

Hi all,

I have a CESM2.2.0 run on Cheyenne which crashes after submitting, but I can't see an obvious error source. It's a land-atmosphere only run based on the FC2010climo compset. The error message in the cesm log is the usual:

0:MCT::m_Router::initp_: GSMap indices not increasing...Will correct
0:MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
0:MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
0:MCT::m_Router::initp_: GSMap indices not increasing...Will correct
0:MCT::m_Router::initp_: GSMap indices not increasing...Will correct
0:MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
0:MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
0:MCT::m_Router::initp_: GSMap indices not increasing...Will correct
-1:MPT ERROR: MPI_COMM_WORLD rank 383 has terminated without calling MPI_Finalize()
-1: aborting job
MPT: Received signal 11

There are no error messages in any of the component log files. A forum search suggests the lines beginning 0:MCT can be related to an issue with ocean models but as I am using prescribed SST via DOCN this seems unlikely. The atm log file is the last one to be output before the error appears and the last line is:

READ_NEXT_TRCDATA emiss_aircraft

For this run I am simulating a time slice of conditions in the year 2095. To create emissions files for this I have interpolated between the years 2090 and 2100 in the CMIP6 ScenarioMIP emissions files using CDO. I suspect this interpolation is the source of the error but without an error message I'm not certain. My questions are:

1 - why does the run fail?
2 - if it is because of my custom emissions files, is there an NCAR-approved way of interpolating missing years from existing emissions files?

Log files are attached (only the end of the atm log as it's too large for the upload limit).

Thanks,

James

 

[sacks]

I agree with your assessment of the likely cause. In particular, the "GSMap indices not increasing...Will correct" messages are common and I agree that those can probably be ignored. I am going to transfer this to the CAM-Chem forums in the hopes that someone there will have more insight than I do into how to create the appropriate emissions file.

 

[emmons]

James,
I am not entirely clear on what you have done - did you create a new file for just the year 2095? You can interpolate however you wish to create the values for that year. The format of the emissions file must match the original file - include altitude and altitude_int, date array for the corresponding dates (the time variable is not used by the model). If your file has only that year, you will need to run with ext_frc_type = 'CYCLICAL' and ext_frc_cycle_yr = 2095. But then all external forcing emissions files must have the year 2095 in them (most may have only each decade). See: https://wiki.ucar.edu/display/camchem/Change+emissions+input

 

[James King]

Hi Louisa,

I've created new emissions files which just contain 2095, based off the default files used when you run an SSP3 case with CAM-Chem. I've set ext_frc_type and ext_frc_cycle_year as above. I think the issue might be that CDO is removing 'altitude_int' as this variable appears in the source files but not the files I made. I'll have a look at other methods. Thanks for the advice!