In order for the cesm resubmit feature to work you must be able to submit tasks from system compute nodes. The error from openmpi about
> unable to find any relevant network interfaces:
suggests that there is a problem with doing this. As an alternative try using the --resubmit-immediate flag to ./case.submit this will submit all of the
model runs at once and use the queueing system to hold them until the previous run has completed. You might also want to consult with your systems support staff about the nature of that error.
Thanks Jim. Hard to say what is the issue from, hardware?. Each time run, it stops at different steps: sometime after first run, sometime after a few runs.
...
st_archive.sh: short-term archiving completed successfully
RESUBMIT is now 1
-------------------------------------------------------------------------
CESM BUILDNML SCRIPT STARTING
- To prestage restarts, untar a restart.tar file into /home/ssm-1/My_Projects/ucar_CESM/CESM1_2_2_1/projects/test2/run
infile is /home/ssm-1/My_Projects/ucar_CESM/CESM1_2_2_1/projects/test2/Buildconf/cplconf/cesm_namelist
CESM BUILDNML SCRIPT HAS FINISHED SUCCESSFULLY
-------------------------------------------------------------------------
-------------------------------------------------------------------------
CESM PRESTAGE SCRIPT STARTING
- Case input data directory, DIN_LOC_ROOT, is /home/ssm-1/My_Projects/ucar_CESM/CESM1_2_2_1/projects/inputdata
- Checking the existence of input datasets in DIN_LOC_ROOT
CESM PRESTAGE SCRIPT HAS FINISHED SUCCESSFULLY
-------------------------------------------------------------------------
Thu May 6 20:39:43 UTC 2021 -- CSM EXECUTION BEGINS HERE
Thu May 6 20:39:43 UTC 2021 -- CSM EXECUTION HAS FINISHED
grep: cpl.log.210506-203918: No such file or directory
Model did not complete - see /home/ssm-1/My_Projects/ucar_CESM/CESM1_2_2_1/projects/test2/run/cesm.log.210506-203918
ccsm_postrun error: problem sourcing tempres..