Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

CESM 2.1.3-ERROR: RUN FAIL: Command mpiexec_mpt -p "%g:"

hsalah

hsalah
New Member
Hello All,

I am running a year long simulation over on CESM 2.1.3


However I face a few issues when I submit,


These have been my steps:

Step 1
/glade/work/[username]/my_cesm_sandbox/cime/scripts




Step 2
./create_newcase --case ~/CEDS2015 --compset FCSD --res f09_g17 —mach cheyenne --project [Project Number] --run-unsupported

Step 3 ~/CEDS2015

./xmlchange RUN_STARTDATE=2014-01-01
./xmlchange STOP_OPTION=nyears


./xmlchange STOP_N=1
./xmlchange RESUBMIT=1






qcmd -- ./case.build





In my script directory I used the following script:





#!/bin/bash


### Job Name


#PBS -N CEDS2015:q!


### project code


#PBS -A [Project Number]





### queue


#PBS -q economy





### set walltime


#PBS -l walltime=12:00:00


#PBS -l select=32:ncpus=36:mpiprocs=36


#Send email on abort, begin and end


#PBS -m abe


###Specify mail recipient


#PBS -M *******.edu


###-----------------


### Go to the running directory


###-----------------


cd /glade/u/home/[username]/[CaseFile]





###--------------


#### submit job


###--------------


./case.submit > ./run.log











I edited the user_nl_cam file:





Edit Met_Data_File
! Users should add all user specific namelist changes below in the form of


! namelist_var = new_namelist_value


&metdata_nl


met_data_file = '2014/MERRA2_0.9x1.25_20140101.nc'

/​

In the script directory I use this command


qsub run_CAM_script.sh

It ran for 12 hours however when I check CaseStatus
vi CaseStatus

I received this error:
ERROR: RUN FAIL: Command 'mpiexec_mpt -p "%g:" -np 384 omplace -tm open64 /glade/scratch/hsalah/CEDS2015/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed

See log file for details: /glade/scratch/hsalah/CEDS2015/run/cesm.log.6894279.chadmin1.ib0.cheyenne.ucar.edu.210305-215921


I would like to know what I could do to avoid this ERROR in the future?


Thank You,
 

dbailey

CSEG and Liaisons
Staff member
The relevant abort information is further up in cesm.log:

1: Opened existing file
1: /glade/p/cesmdata/cseg/inputdata/atm/cam/met/MERRA2/0.9x1.25/2015/MERRA2_0.9x1.
1: 25_20150114.nc 103
0: size=17017901 rss=182017 share=17396 text=16015 datastack=0
1: Opened existing file
1: /glade/p/cesmdata/cseg/inputdata/atm/cam/met/MERRA2/0.9x1.25/2015/MERRA2_0.9x1.
1: 25_20150115.nc 2
0: size=17017901 rss=182031 share=17396 text=16015 datastack=0
1: Opened existing file
1: /glade/p/cesmdata/cseg/inputdata/atm/cam/met/MERRA2/0.9x1.25/2015/MERRA2_0.9x1.
1: 25_20150116.nc 103
1: Opened existing file
1: /glade/p/cesmdata/cseg/inputdata/atm/waccm/lb/LBC_1750-2014_CMIP6_0p5degLat_c17
1: 0126.nc 2
0: ERROR: Unknown error submitted to shr_abort_abort.
8: flbc_set: interp time is out of bounds
3: flbc_set: interp time is out of bounds
160:Image PC Routine Line Source
160:cesm.exe 00000000036C823D Unknown Unknown Unknown
160:cesm.exe 0000000002E2B056 shr_abort_mod_mp_ 114 shr_abort_mod.F90
160:cesm.exe 0000000000618EBF mo_flbc_mp_get_de 748 mo_flbc.F90
160:cesm.exe 0000000000616733 mo_flbc_mp_flbc_g 792 mo_flbc.F90
160:cesm.exe 000000000054504E chem_surfvals_mp_ 530 chem_surfvals.F90
160:cesm.exe 00000000006CC865 physpkg_mp_phys_r 2345 physpkg.F90
160:cesm.exe 00000000004EB482 cam_comp_mp_cam_r 258 cam_comp.F90
160:cesm.exe 00000000004DF812 atm_comp_mct_mp_a 454 atm_comp_mct.F90
160:cesm.exe 00000000004252F4 component_mod_mp_ 728 component_mod.F90
160:cesm.exe 000000000040CBE8 cime_comp_mod_mp_ 3465 cime_comp_mod.F90
160:cesm.exe 0000000000424F87 MAIN__ 125 cime_driver.F90
160:cesm.exe 0000000000408F5E Unknown Unknown Unknown
160:libc.so.6 00002B216D8956E5 __libc_start_main Unknown Unknown
160:cesm.exe 0000000000408E69 Unknown Unknown Unknown

I am moving this over to the CAM groups.
 

cacraig

Cheryl Craig
CSEG and Liaisons
Staff member
Sorry to move this thread one more time, but I am moving it to chemistry. The error "flbc_set: interp time is out of bounds" is coming from that section of the code.
 

hsalah

hsalah
New Member
Sorry to move this thread one more time, but I am moving it to chemistry. The error "flbc_set: interp time is out of bounds" is coming from that section of the code.
Hello,

for the flbc_set error do I correct this by editing the user_nl_cam file?
example:
flbc_file = glade/p/cesmdata/cseg/inputdata/atm/waccm/lb/LBC_17500116-25001216_CMIP6_SSP585_0p5degLat_c20200824.nc
 
Top