Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

CESM1.2.0 porting issues

Jerry

Cao
Member
What version of the code are you using?
CESM1.2.0


Have you made any changes to files in the source tree?
Not yet, just for porting now.

Describe every step you took leading up to the problem:

We want to porting CESM1.2.0 to a HPC machine.
We want to port CESM1 instead of CESM2 because most of our previous work was based on CESM1.


The config_machines.xml and config_compile.xml have been modified to define the necessary items:
<machine MACH="lenovo">
<DESC> RHEL7, batch system is LSF</DESC>
<OS>LINUX</OS>
<COMPILERS>intel</COMPILERS>
<MPILIBS>impi</MPILIBS>
<RUNDIR>/public/home/amduan/cesm1.2/cesm1_2_0/casedir/$CASE/run</RUNDIR>
<EXEROOT>/public/home/amduan/cesm1.2/cesm1_2_0/casedir/$CASE/bld</EXEROOT>
<DIN_LOC_ROOT>/public/home/amduan/cesm/new_inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/public/home/amduan/cesm/new_inputdata/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>/public/home/amduan/cesm1.2/cesm1_2_0/scratch/$CASE</DOUT_S_ROOT>
<BASELINE_ROOT>/public/home/amduan/cesm1.2/cesm1_2_0/ccsm_baselines</BASELINE_ROOT>
<CCSM_CPRNC>/public/home/amduan/cesm1.2/cesm1_2_0/tools/cprnc/</CCSM_CPRNC>
<GMAKE_J>4</GMAKE_J>
<BATCHQUERY>squeue</BATCHQUERY>
<BATCHSUBMIT>sbatch</BATCHSUBMIT>
<SUPPORTED_BY>lenovo</SUPPORTED_BY>
<MAX_TASKS_PER_NODE>64</MAX_TASKS_PER_NODE>
</machine>

<compiler MACH="lenovo" COMPILER="intel">
<NETCDF_PATH>/public/software/netcdf/netcdf-4.7.4-intel</NETCDF_PATH>
<PNETCDF_PATH>/public/home/hujun/soft/pnetcdf</PNETCDF_PATH>
<MPICC>mpiicc</MPICC>
<MPIFC>mpiifort</MPIFC>
<MPICXX>mpiicpc</MPICXX>
<SFC>ifort</SFC>
<SCC>icc</SCC>
<SCXX>icpc</SCXX>
<FFLAGS> -qno-opt-dynamic-align -convert big_endian -assume byterecl -ftz -traceback -assume realloc_lhs -fp-model source -no-wrap-margin </FFLAGS>
<ADD_FFLAGS DEBUG="TRUE"> -O0 -g -check uninit -check bounds -check pointers -fpe0 -check noarg_temp_created </ADD_FFLAGS>
<ADD_CFLAGS DEBUG="FALSE"> -O2 -debug minimal </ADD_CFLAGS>
<ADD_LDFLAGS> -L/public/software/netcdf/netcdf-4.7.4-intel/lib -lnetcdf -lnetcdff -L/public/home/hujun/soft/pnetcdf/lib -lpnetcdf </ADD_LDFLAGS>
</compiler>

We use intel compiler for Fortran, C and C++, the version is 2021.1.
We can successfully execute ${CASE}.build, but we encounter trouble when submitting the job with sbatch ${CASE}.run afterwards.
The submitted job fails after approximately 30 seconds, and the log file shows the error message "BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES KILLED BY SIGNAL: 9 (Killed)".
The cesm.log* file has been uploaded.

Describe your problem or question:
Can someone tell me what went wrong? It can be sure that our job queue system and software libraries are functioning correctly because we have successfully ported CESM2.1.3 on this machine as well.
 

Attachments

  • cesm.log.txt
    3 KB · Views: 3

jedwards

CSEG and Liaisons
Staff member
This error appears to be happening at or before the mpi_init call. Check that the number of tasks requested matches the number expected.
 
Top