CESM2.1.3 mpiexec_mpt error

[wxalpha]

Hello,

I am working with cesm2.1.3 on the F1850 compset. Attached is the script I used to create, setup, and build the experiment (attached; AllSpecies.csh.txt).

But when I ran the case.submit, I encounter some issue I do not really know how to solve. It seems like the issue is related to the mpiexec_mpt as posted here (https://bb.cgd.ucar.edu/cesm/threads/cheyenne-update-module-issues-for-all-cesm-versions.4681/). I removed the -p option in the env_mach_specific.xml (attached). But after resetting, cleaning, logging out, logging back again and again, the issue is still there.

CaseStatus:
2020-05-29 08:36:51: case.build success
---------------------------------------------------
2020-05-29 08:40:51: case.submit starting
---------------------------------------------------
2020-05-29 08:40:56: case.submit success case.run:2469378.chadmin1.ib0.cheyenne.ucar.edu, case.st_archive:2469379.chadmin1.ib0.cheyenne.ucar.edu
---------------------------------------------------
2020-05-29 08:40:59: case.run starting
---------------------------------------------------
2020-05-29 08:41:04: model execution starting
---------------------------------------------------
2020-05-29 08:41:07: model execution success
---------------------------------------------------
2020-05-29 08:41:07: case.run error
ERROR: RUN FAIL: Command 'mpiexec_mpt -np 72 omplace -tm open64 /glade/scratch/yujiayou/AllSpecies/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed
See log file for details: /glade/scratch/myname/AllSpecies/run/cesm.log.2469378.chadmin1.ib0.cheyenne.ucar.edu.200529-084059

cesm.log.2469378.chadmin1.ib0.cheyenne.ucar.edu.200529-084059 is attached. It seems like the issue is "MPT ERROR: MPI_COMM_WORLD rank 47 has terminated without calling MPI_Finalize() aborting job"

Does anyone have an idea on how to solve this issue? Any help is deeply appreciated!

 

[sacks]

It looks like the error here is in CAM, when trying to read the emissions file:

 airpl_src: NO and CO do not have external source --> no aircraft sources will b
 e applied
 chemini: after airpl_src on node            0
 srf_emis_inti: n_emis_files =            9
 (GETFIL): attempting to find local file aerocom_mam3_dms_surf_2000_c090129.nc
 (GETFIL): using 
 /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam
 3_dms_surf_2000_c090129.nc
 trcdata_init: data type: CYCLICAL file: /glade/p/cesmdata/cseg/inputdata/atm/ca
 m/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc
 (GETFIL): attempting to find local file aerocom_mam3_dms_surf_2000_c090129.nc
 (GETFIL): using 
 /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam
 3_dms_surf_2000_c090129.nc
 open_trc_datafile: 
 /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam
 3_dms_surf_2000_c090129.nc
 open_trc_datafile: cycle year not found :         1850
 ERROR: 
 open_trc_datafile: cycle year not found /glade/p/cesmdata/cseg/inputdata/atm/ca
 m/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc

I am going to move this to the CAM forums in hopes that someone there can address this better than I can.

 

[wxalpha]

It looks like the error here is in CAM, when trying to read the emissions file:

airpl_src: NO and CO do not have external source --> no aircraft sources will b
e applied
chemini: after airpl_src on node            0
srf_emis_inti: n_emis_files =            9
(GETFIL): attempting to find local file aerocom_mam3_dms_surf_2000_c090129.nc
(GETFIL): using
/glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam
3_dms_surf_2000_c090129.nc
trcdata_init: data type: CYCLICAL file: /glade/p/cesmdata/cseg/inputdata/atm/ca
m/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc
(GETFIL): attempting to find local file aerocom_mam3_dms_surf_2000_c090129.nc
(GETFIL): using
/glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam
3_dms_surf_2000_c090129.nc
open_trc_datafile:
/glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam
3_dms_surf_2000_c090129.nc
open_trc_datafile: cycle year not found :         1850
ERROR:
open_trc_datafile: cycle year not found /glade/p/cesmdata/cseg/inputdata/atm/ca
m/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc

I am going to move this to the CAM forums in hopes that someone there can address this better than I can.

I just checked. The emission file is in the directory. Maybe it is no longer used by CESM2.1? Thank you for your help!

 

[cacraig]

My simplistic answer is that the file you are using (aerocom_mam3_dms_surf_2000_c090129.nc) does not have the year 1850 in it. You can see this by running "ncdump -v date /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc". The file that is in the default F1850 compset (without your file overrides) contains dates from 17500116 through 20151216. The file you are using contains just 12 dates (monthly snapshots) with "no year" (116 through 1216).

If that is not enough information for you to solve your problem, it would probably help to know why you are changing the emissions files from their defaults. That way one of the scientists can help you get your settings correct.