CESM 2.1.3-ERROR: RUN FAIL: Command mpiexec_mpt -p "%g:"

[hsalah]

Hello All,

I am running a year long simulation over on CESM 2.1.3


However I face a few issues when I submit,


These have been my steps:

Step 1
/glade/work/[username]/my_cesm_sandbox/cime/scripts




Step 2
./create_newcase --case ~/CEDS2015 --compset FCSD --res f09_g17 —mach cheyenne --project [Project Number] --run-unsupported

Step 3 ~/CEDS2015
./xmlchange RUN_STARTDATE=2014-01-01
./xmlchange STOP_OPTION=nyears


./xmlchange STOP_N=1
./xmlchange RESUBMIT=1





qcmd -- ./case.build





In my script directory I used the following script:

#!/bin/bash


### Job Name


#PBS -N CEDS2015:q!


### project code


#PBS -A [Project Number]





### queue


#PBS -q economy





### set walltime


#PBS -l walltime=12:00:00


#PBS -l select=32:ncpus=36:mpiprocs=36


#Send email on abort, begin and end


#PBS -m abe


###Specify mail recipient


#PBS -M *******.edu


###-----------------


### Go to the running directory


###-----------------


cd /glade/u/home/[username]/[CaseFile]





###--------------


#### submit job


###--------------


./case.submit > ./run.log
​

I edited the user_nl_cam file:

Edit Met_Data_File
! Users should add all user specific namelist changes below in the form of


! namelist_var = new_namelist_value


&metdata_nl


met_data_file = '2014/MERRA2_0.9x1.25_20140101.nc'

/​

In the script directory I use this command


qsub run_CAM_script.sh

It ran for 12 hours however when I check CaseStatus
vi CaseStatus

I received this error:

ERROR: RUN FAIL: Command 'mpiexec_mpt -p "%g:" -np 384 omplace -tm open64 /glade/scratch/hsalah/CEDS2015/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed

​

See log file for details: /glade/scratch/hsalah/CEDS2015/run/cesm.log.6894279.chadmin1.ib0.cheyenne.ucar.edu.210305-215921


I would like to know what I could do to avoid this ERROR in the future?


Thank You,

 

[hsalah]

Typo in the script I have typed out:
#PBS -N CEDS2015 not #PBS -N CEDS2015:q!

Thank you

In my script directory I used the following script:

#!/bin/bash


### Job Name


#PBS -N CEDS2015:q!​

 

[dbailey]

The relevant abort information is further up in cesm.log:

1: Opened existing file
1: /glade/p/cesmdata/cseg/inputdata/atm/cam/met/MERRA2/0.9x1.25/2015/MERRA2_0.9x1.
1: 25_20150114.nc 103
0: size=17017901 rss=182017 share=17396 text=16015 datastack=0
1: Opened existing file
1: /glade/p/cesmdata/cseg/inputdata/atm/cam/met/MERRA2/0.9x1.25/2015/MERRA2_0.9x1.
1: 25_20150115.nc 2
0: size=17017901 rss=182031 share=17396 text=16015 datastack=0
1: Opened existing file
1: /glade/p/cesmdata/cseg/inputdata/atm/cam/met/MERRA2/0.9x1.25/2015/MERRA2_0.9x1.
1: 25_20150116.nc 103
1: Opened existing file
1: /glade/p/cesmdata/cseg/inputdata/atm/waccm/lb/LBC_1750-2014_CMIP6_0p5degLat_c17
1: 0126.nc 2
0: ERROR: Unknown error submitted to shr_abort_abort.
8: flbc_set: interp time is out of bounds
3: flbc_set: interp time is out of bounds
160:Image PC Routine Line Source
160:cesm.exe 00000000036C823D Unknown Unknown Unknown
160:cesm.exe 0000000002E2B056 shr_abort_mod_mp_ 114 shr_abort_mod.F90
160:cesm.exe 0000000000618EBF mo_flbc_mp_get_de 748 mo_flbc.F90
160:cesm.exe 0000000000616733 mo_flbc_mp_flbc_g 792 mo_flbc.F90
160:cesm.exe 000000000054504E chem_surfvals_mp_ 530 chem_surfvals.F90
160:cesm.exe 00000000006CC865 physpkg_mp_phys_r 2345 physpkg.F90
160:cesm.exe 00000000004EB482 cam_comp_mp_cam_r 258 cam_comp.F90
160:cesm.exe 00000000004DF812 atm_comp_mct_mp_a 454 atm_comp_mct.F90
160:cesm.exe 00000000004252F4 component_mod_mp_ 728 component_mod.F90
160:cesm.exe 000000000040CBE8 cime_comp_mod_mp_ 3465 cime_comp_mod.F90
160:cesm.exe 0000000000424F87 MAIN__ 125 cime_driver.F90
160:cesm.exe 0000000000408F5E Unknown Unknown Unknown
160:libc.so.6 00002B216D8956E5 __libc_start_main Unknown Unknown
160:cesm.exe 0000000000408E69 Unknown Unknown Unknown

I am moving this over to the CAM groups.

 

[cacraig]

Sorry to move this thread one more time, but I am moving it to chemistry. The error "flbc_set: interp time is out of bounds" is coming from that section of the code.

 

[hsalah]

Sorry to move this thread one more time, but I am moving it to chemistry. The error "flbc_set: interp time is out of bounds" is coming from that section of the code.

Hello,

for the flbc_set error do I correct this by editing the user_nl_cam file?
example:
flbc_file = glade/p/cesmdata/cseg/inputdata/atm/waccm/lb/LBC_17500116-25001216_CMIP6_SSP585_0p5degLat_c20200824.nc

 

[tilmes]

Hope this worked for you!
Cheers, Simone