Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

CESM2.1.3 mpiexec_mpt error

wxalpha

New Member
Hello,

I am working with cesm2.1.3 on the F1850 compset. Attached is the script I used to create, setup, and build the experiment (attached; AllSpecies.csh.txt).

But when I ran the case.submit, I encounter some issue I do not really know how to solve. It seems like the issue is related to the mpiexec_mpt as posted here (https://bb.cgd.ucar.edu/cesm/threads/cheyenne-update-module-issues-for-all-cesm-versions.4681/). I removed the -p option in the env_mach_specific.xml (attached). But after resetting, cleaning, logging out, logging back again and again, the issue is still there.

CaseStatus:
2020-05-29 08:36:51: case.build success
---------------------------------------------------
2020-05-29 08:40:51: case.submit starting
---------------------------------------------------
2020-05-29 08:40:56: case.submit success case.run:2469378.chadmin1.ib0.cheyenne.ucar.edu, case.st_archive:2469379.chadmin1.ib0.cheyenne.ucar.edu
---------------------------------------------------
2020-05-29 08:40:59: case.run starting
---------------------------------------------------
2020-05-29 08:41:04: model execution starting
---------------------------------------------------
2020-05-29 08:41:07: model execution success
---------------------------------------------------
2020-05-29 08:41:07: case.run error
ERROR: RUN FAIL: Command 'mpiexec_mpt -np 72 omplace -tm open64 /glade/scratch/yujiayou/AllSpecies/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed
See log file for details: /glade/scratch/myname/AllSpecies/run/cesm.log.2469378.chadmin1.ib0.cheyenne.ucar.edu.200529-084059

cesm.log.2469378.chadmin1.ib0.cheyenne.ucar.edu.200529-084059 is attached. It seems like the issue is "MPT ERROR: MPI_COMM_WORLD rank 47 has terminated without calling MPI_Finalize() aborting job"

Does anyone have an idea on how to solve this issue? Any help is deeply appreciated!
 

Attachments

  • AllSpecies.csh.txt
    5.4 KB · Views: 5
  • env_mach_specific.xml.txt
    5.1 KB · Views: 0
  • atm.log.2469378.chadmin1.ib0.cheyenne.ucar.edu.200529-084059.txt
    71.4 KB · Views: 2
  • cesm.log.2469378.chadmin1.ib0.cheyenne.ucar.edu.200529-084059.txt
    134.6 KB · Views: 4
  • cpl.log.2469378.chadmin1.ib0.cheyenne.ucar.edu.200529-084059.txt
    41.4 KB · Views: 1

sacks

Bill Sacks
CSEG and Liaisons
Staff member
It looks like the error here is in CAM, when trying to read the emissions file:

Code:
 airpl_src: NO and CO do not have external source --> no aircraft sources will b
 e applied
 chemini: after airpl_src on node            0
 srf_emis_inti: n_emis_files =            9
 (GETFIL): attempting to find local file aerocom_mam3_dms_surf_2000_c090129.nc
 (GETFIL): using 
 /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam
 3_dms_surf_2000_c090129.nc
 trcdata_init: data type: CYCLICAL file: /glade/p/cesmdata/cseg/inputdata/atm/ca
 m/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc
 (GETFIL): attempting to find local file aerocom_mam3_dms_surf_2000_c090129.nc
 (GETFIL): using 
 /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam
 3_dms_surf_2000_c090129.nc
 open_trc_datafile: 
 /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam
 3_dms_surf_2000_c090129.nc
 open_trc_datafile: cycle year not found :         1850
 ERROR: 
 open_trc_datafile: cycle year not found /glade/p/cesmdata/cseg/inputdata/atm/ca
 m/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc

I am going to move this to the CAM forums in hopes that someone there can address this better than I can.
 

wxalpha

New Member
It looks like the error here is in CAM, when trying to read the emissions file:

Code:
airpl_src: NO and CO do not have external source --> no aircraft sources will b
e applied
chemini: after airpl_src on node            0
srf_emis_inti: n_emis_files =            9
(GETFIL): attempting to find local file aerocom_mam3_dms_surf_2000_c090129.nc
(GETFIL): using
/glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam
3_dms_surf_2000_c090129.nc
trcdata_init: data type: CYCLICAL file: /glade/p/cesmdata/cseg/inputdata/atm/ca
m/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc
(GETFIL): attempting to find local file aerocom_mam3_dms_surf_2000_c090129.nc
(GETFIL): using
/glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam
3_dms_surf_2000_c090129.nc
open_trc_datafile:
/glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam
3_dms_surf_2000_c090129.nc
open_trc_datafile: cycle year not found :         1850
ERROR:
open_trc_datafile: cycle year not found /glade/p/cesmdata/cseg/inputdata/atm/ca
m/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc

I am going to move this to the CAM forums in hopes that someone there can address this better than I can.

I just checked. The emission file is in the directory. Maybe it is no longer used by CESM2.1? Thank you for your help!
 

cacraig

Cheryl Craig
CSEG and Liaisons
Staff member
My simplistic answer is that the file you are using (aerocom_mam3_dms_surf_2000_c090129.nc) does not have the year 1850 in it. You can see this by running "ncdump -v date /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart_aero/emis/aerocom_mam3_dms_surf_2000_c090129.nc". The file that is in the default F1850 compset (without your file overrides) contains dates from 17500116 through 20151216. The file you are using contains just 12 dates (monthly snapshots) with "no year" (116 through 1216).

If that is not enough information for you to solve your problem, it would probably help to know why you are changing the emissions files from their defaults. That way one of the scientists can help you get your settings correct.
 
Top