Issue in submitting the case

[csp]

Dear experts,
I managed to create, set and build the case successfully in my machine using the following codes respectively:

./create_newcase --case aqua_case --compset QSC6 --res f09_f09_mg17

./case.setup

./case.build

But while submitting the case using:

./case.submit

I am getting the following output in my screen and nothing runs:

Creating component namelists
  2022-09-20 12:18:38 atm 
   Calling /home/csp/my_cesm_sandbox/components/cam//cime_config/buildnml
CAM namelist copy: file1 /home/csp/my_cesm_sandbox/cime/scripts/aqua_case/Buildconf/camconf/atm_in file2 /home/csp/OUTPUT/CESM/scratch/aqua_case/run/atm_in 
  2022-09-20 12:18:38 lnd 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/slnd/cime_config/buildnml
  2022-09-20 12:18:38 ice 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/sice/cime_config/buildnml
  2022-09-20 12:18:38 ocn 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/data_comps_mct/docn/cime_config/buildnml
  2022-09-20 12:18:38 rof 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/srof/cime_config/buildnml
  2022-09-20 12:18:38 glc 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/sglc/cime_config/buildnml
  2022-09-20 12:18:38 wav 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/swav/cime_config/buildnml
  2022-09-20 12:18:38 iac 
   Running /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/siac/cime_config/buildnml 
  2022-09-20 12:18:38 esp 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/sesp/cime_config/buildnml
  2022-09-20 12:18:38 cpl 
   Calling /home/csp/my_cesm_sandbox/cime/src/drivers/mct/cime_config/buildnml
Checking that inputdata is available as part of case submission
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/docn.input_data_list'
Loading input file list: 'Buildconf/cam.input_data_list'
Check case OK
submit_jobs case.run
Submit job case.run
Starting job script case.run
Generating namelists for /home/csp/my_cesm_sandbox/cime/scripts/aqua_case
Creating component namelists
  2022-09-20 12:18:39 atm 
   Calling /home/csp/my_cesm_sandbox/components/cam//cime_config/buildnml
CAM namelist copy: file1 /home/csp/my_cesm_sandbox/cime/scripts/aqua_case/Buildconf/camconf/atm_in file2 /home/csp/OUTPUT/CESM/scratch/aqua_case/run/atm_in 
  2022-09-20 12:18:39 lnd 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/slnd/cime_config/buildnml
  2022-09-20 12:18:39 ice 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/sice/cime_config/buildnml
  2022-09-20 12:18:39 ocn 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/data_comps_mct/docn/cime_config/buildnml
  2022-09-20 12:18:39 rof 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/srof/cime_config/buildnml
  2022-09-20 12:18:39 glc 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/sglc/cime_config/buildnml
  2022-09-20 12:18:39 wav 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/swav/cime_config/buildnml
  2022-09-20 12:18:39 iac 
   Running /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/siac/cime_config/buildnml 
  2022-09-20 12:18:39 esp 
   Calling /home/csp/my_cesm_sandbox/cime/src/components/stub_comps_mct/sesp/cime_config/buildnml
  2022-09-20 12:18:39 cpl 
   Calling /home/csp/my_cesm_sandbox/cime/src/drivers/mct/cime_config/buildnml
-------------------------------------------------------------------------
 - Prestage required restarts into /home/csp/OUTPUT/CESM/scratch/aqua_case/run
 - Case input data directory (DIN_LOC_ROOT) is /home/csp/OUTPUT/CESM/cesm-inputdata 
 - Checking for required input datasets in DIN_LOC_ROOT
-------------------------------------------------------------------------
run command is mpiexec_mpt  -np 16 --prepend-rank /home/csp/OUTPUT/CESM/scratch/aqua_case/bld/cesm.exe   >> cesm.log.$LID 2>&1  
Exception from case_run: ERROR: RUN FAIL: Command 'mpiexec_mpt  -np 16 --prepend-rank /home/csp/OUTPUT/CESM/scratch/aqua_case/bld/cesm.exe   >> cesm.log.$LID 2>&1 ' failed
See log file for details: /home/csp/OUTPUT/CESM/scratch/aqua_case/run/cesm.log.220920-121839
Submit job case.st_archive
Starting job script case.st_archive
st_archive starting
moving /home/csp/OUTPUT/CESM/scratch/aqua_case/run/cesm.log.220920-121839 to /home/csp/OUTPUT/CESM/scratch/archive/aqua_case/logs/cesm.log.220920-121839
Cannot find a aqua_case.cpl*.r.*.nc file in directory /home/csp/OUTPUT/CESM/scratch/aqua_case/run 
Archiving history files for cam (atm)
Archiving history files for docn (ocn)
Archiving history files for drv (cpl)
Archiving history files for dart (esp)
st_archive completed
Submitted job case.run with id None
Submitted job case.st_archive with id None

I am unable to figure out the issue, any comment/suggestion would be kind help, thanks in advance.

 

[katec]

This line:

moving /home/csp/OUTPUT/CESM/scratch/aqua_case/run/cesm.log.220920-121839 to /home/csp/OUTPUT/CESM/scratch/archive/aqua_case/logs/cesm.log.220920-121839

Suggests that the model did run and produced a cesm.log file. Please take a look at that file for the error that caused the model to fail to complete.

Also this line:

Cannot find a aqua_case.cpl*.r.*.nc file in directory /home/csp/OUTPUT/CESM/scratch/aqua_case/run

Suggests that you are missing the coupler restart file from your previous or reference case.

 

[csp]

Thanks a lot @katec for your kind help and for your time. I looked at the log file and found the following:

/bin/sh: 1: mpiexec_mpt: not found

but when I checked in my pc for mpiexec using

mpiexec --version

I got the output:

mpiexec (OpenRTE) 4.0.3

seems like mpiexec is installed, not sure about the error.

 

[csp]

Suggests that you are missing the coupler restart file from your previous or reference case.

@katec not sure about this point, what to do next.

 

[katec]

Thanks a lot @katec for your kind help and for your time. I looked at the log file and found the following:

/bin/sh: 1: mpiexec_mpt: not found

but when I checked in my pc for mpiexec using

mpiexec --version

I got the output:

mpiexec (OpenRTE) 4.0.3

seems like mpiexec is installed, not sure about the error.

So the error "mpiexec_mpt: not found" looks like the model is trying to run with the command "mpiexec_mpt" which is different than "mpiexec" because it has an extra "_mpt" at the end. If you want the model to run using "mpiexec" (no "_mpt") then you will need to update the "<executable>" flag in your config_machines.xml file.