MPI_ABORT on submit

[william.wilson]

We are getting an MPI_ABORT when we do a case.submit. I am trying to set up a working test case to make sure cesm is functioning as expected. This is the newcase I am using:

scripts/create_newcase --case /scratch/$USER/cesmtest --res f09_g17 --compset I1850Clm50Sp --machine monsoon -i /scratch/$USER/cesminput

I have the monsoon machine set up in the xml files.

the case setup and case build appear to work fine but the submit ends with the following error:
(seq_comm_printcomms) 38 0 32 1 CPLIAC:
(t_initf) Read in prof_inparm namelist from: drv_in
(t_initf) Using profile_disable= F
(t_initf) profile_timer= 4
(t_initf) profile_depth_limit= 4
(t_initf) profile_detail_limit= 2
(t_initf) profile_barrier= F
(t_initf) profile_outpe_num= 1
(t_initf) profile_outpe_stride= 0
(t_initf) profile_single_file= F
(t_initf) profile_global_stats= T
(t_initf) profile_ovhd_measurement= F
(t_initf) profile_add_detail= F
(t_initf) profile_papi_enable= F
ERROR: (shr_stream_getCalendar) ERROR: nf90_open file /scratch/wew/cesminput/lmwg/atm_forcing.datm7.GSWP3.0.5d.v1.
c170516/Solar/clmforc.GSWP3.c2011.0.5x0.5.Solr.1901-01.nc
#0 0x145c651a7131 in ???
#1 0xc3218e in ???
#2 0xc3235b in ???
#3 0xd24143 in ???
#4 0xd2c9c4 in ???
#5 0xd0b9f4 in ???
#6 0x4b99a4 in ???
#7 0x4ae674 in ???
#8 0x423daa in ???
#9 0x413b08 in ???
#10 0x42150a in ???
#11 0x145c643ee6a2 in ???
#12 0x4065fd in ???
#13 0xffffffffffffffff in ???
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1001.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------



What might be causing this error?

 

[fischer]

Hi William,

It looks like you don't have the required inputdata needed to run your case. You'll need to run "check_input_data --download" from your
case directory.

Chris

 

[william.wilson]

If I run the new case followed by the case setup, build, and submit, shouldn't that have downloaded the data? Or am I missing a step? Is there a smalle test case that may be better?

 

[william.wilson]

The check_input_data ended with this:

Loading input file list: 'Buildconf/clm.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/cism.input_data_list'
Loading input file list: 'Buildconf/mosart.input_data_list'
Checking server None with protocol None
Setting Environment OMP_STACKSIZE=256M
Setting Environment NETCDF_C_PATH=/packages/gcc-8.3.1/netcdf-c/4.7.4-opdm2fw
Setting Environment NETCDF_FORTRAN_PATH=/packages/openmpi-3.1/netcdf-fortran/4.5.3-gtyy5o4
Setting Environment TMPDIR=/tmp/wew
Setting Environment JOBDIR=/scratch/wew
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Client protocol None not enabled
ERROR: Could not find all inputdata on any server

 

[fischer]

Yes, that should have downloaded the data. But I'm guessing that something didn't work right and you need to use the check_input_data script.

A smaller test would be.
create_newcase --case /scratch/$USER/cesmAtest --res f45_g37 --compset A --machine monsoon -i /scratch/$USER/cesminput --run-unsupported

You can also check to see if the file in your error message exists, and isn't empty.
ls -l /scratch/wew/cesminput/lmwg/atm_forcing.datm7.GSWP3.0.5d.v1.c170516/Solar/clmforc.GSWP3.c2011.0.5x0.5.Solr.1901-01.nc