Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Error Submiting Tutorial Case (BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES, EXIT CODE 9)

J

Jakub_Sypek

Jakub
New Member
What version of the code are you using?
CESM v2.1.5


Have you made any changes to files in the source tree?
config_compilers.xml
config_machines.xml
config_inputdata.xml


Describe every step you took leading up to the problem:
I am attempting to port and run CESM v2.1.5 on a new machine. I am attempting to follow the CESM WACCM tutorial case found at https://files.cesm.ucar.edu/events/tutorials/2022/CESM-Tutorial-2022-WACCM-Practical.pdf.

If this is a port to a new machine: Please attach any files you added or changed for the machine port (e.g., config_compilers.xml, config_machines.xml, and config_batch.xml) and tell us the compiler version you are using on this machine.
Please attach any log files showing error messages or other useful information.
Config_compilers.xml:
<compiler COMPILER="gnu" MACH="MKlabgroup2">
<!-- LINUX -->

<CFLAGS>
<base> -std=gnu99 </base>
<append compile_threaded="true"> -fopenmp </append>
<append DEBUG="TRUE"> -g -Wall -Og -fbacktrace -ffpe-trap=invalid,zero,overflow -fcheck=bounds </append>
<append DEBUG="FALSE"> -O </append>
</CFLAGS>
<CPPDEFS>
<append>-DFORTRANUNDERSCORE -DNO_R16</append>
</CPPDEFS>
<FFLAGS>
<append> -fallow-argument-mismatch -fallow-invalid-boz -I$ENV{NETCDFFDIR}/include</append>
</FFLAGS>
<LDFLAGS>
<append compile_threaded="true"> -fopenmp </append>
</LDFLAGS>
<SFC>gfortran</SFC>
<SCC>gcc</SCC>
<SCXX>g++</SCXX>
<MPIFC>mpif90</MPIFC>
<MPICC>mpicc</MPICC>
<MPICXX>mpicxx</MPICXX>
<CXX_LINKER>FORTRAN</CXX_LINKER>
<SUPPORTS_CXX>TRUE</SUPPORTS_CXX>
<NETCDF_C_PATH>$ENV{HOME}/CESM/Libs/netcdf-c-4.9.0</NETCDF_C_PATH>
<NETCDF_FORTRAN_PATH>$ENV{HOME}/CESM/Libs/netcdf-fortran-4.6.0</NETCDF_FORTRAN_PATH>
<PNETCDF_PATH>$ENV{PNETCDFDIR}</PNETCDF_PATH>
<SLIBS>
<append>-L${NETCDF_FORTRAN_PATH}/lib -L${NETCDF_C_PATH}/lib -lnetcdff -lnetcdf</append>
<append>-L/usr/lib64 -llapack -lblas</append>
</SLIBS>
</compiler>



config_machines.xml:
<machine MACH="MKlabgroup2">
<DESC>Server 2, OS is RHEL9, 24 pes/node</DESC>
<NODENAME_REGEX>uos-22770</NODENAME_REGEX>
<OS>LINUX</OS>
<COMPILERS>gnu</COMPILERS>
<MPILIBS>mpich</MPILIBS>
<CIME_OUTPUT_ROOT>$ENV{HOME}/projects/cesm/scratch</CIME_OUTPUT_ROOT>
<DIN_LOC_ROOT>$ENV{HOME}/projects/cesm/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>$ENV{HOME}/projects/cesm/inputdata/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>$ENV{HOME}/projects/cesm/archive/case</DOUT_S_ROOT>
<BASELINE_ROOT>$ENV{HOME}/projects/cesm/baselines</BASELINE_ROOT>
<CCSM_CPRNC>$ENV{HOME}/my_cesm_sandbox/cime/tools/cprnc/build/cprnc</CCSM_CPRNC>
<GMAKE_J>4</GMAKE_J>
<BATCH_SYSTEM>none</BATCH_SYSTEM>
<SUPPORTED_BY> Jakub</SUPPORTED_BY>
<MAX_TASKS_PER_NODE>4</MAX_TASKS_PER_NODE>
<MAX_MPITASKS_PER_NODE>4</MAX_MPITASKS_PER_NODE>
<mpirun mpilib="default">
<executable>mpiexec</executable>
<arguments>
<arg name="anum_tasks"> -np {{ total_tasks }}</arg>
</arguments>
</mpirun>
<module_system type="none"/>
</machine>



config_batch.xml: Not edited.

config_inputdata.xml:
<?xml version="1.0"?>

<inputdata>
<!-- server precidence is order in this file. Highest preference at top -->
<!-- If the client doesn't have the protocol it will be skipped -->
<!-- chksum verification of inputfiles is possible. If a file with name -->
<!-- inputdata_chksum.dat is found on the server in the directory above inputdata -->
<!-- it will be searched for filename and chksum of each downloaded file. -->
<!-- see the file ftp://ftp.cgd.ucar.edu/cesm/inputdata_chksum.dat for proper format. -->
<server>
<protocol>wget</protocol>
<address>Index of /cesm/inputdata</address>
<checksum>../inputdata_checksum.dat</checksum>
</server>

<server>
<comment> wget access to subversion </comment>
<protocol>wget</protocol>
<address>- Revision 70792: /trunk/inputdata</address>
<checksum>../inputdata_checksum.dat</checksum>
</server>

<server>
<protocol>svn</protocol>
<address>- Revision 70792: /trunk/inputdata</address>
<checksum>../inputdata_checksum.dat</checksum>
</server>

<server>
<protocol>wget</protocol>
<address>https://redoak.cs.toronto.edu/twitcher/ows/proxy/thredds/fileServer/datasets/CESM/inputdata/</address>
<checksum>../inputdata_checksum.dat</checksum>
</server>

<server CLM_USRDAT_NAME="NEON">
<comment> NEON Tower data for datm </comment>
<protocol>wget</protocol>
<address>https://storage.neonscience.org/neon-ncar/NEON/</address>
<!-- inventory is a CSV list of available data files and the valid date for each -->
<!-- expected format is pathtofile,YYYY-MM-DD HH:MM:SS -->
<!-- one per line where the date stamp indicates the last modification time of the file -->
<inventory>../listing.csv</inventory>

<server CLM_USRDAT_NAME="NEON.PRISM">
<comment> NEON Tower data for datm </comment>
<protocol>wget</protocol>
<address>https://storage.neonscience.org/neon-ncar/NEON/</address>
<!-- inventory is a CSV list of available data files and the valid date for each -->
<!-- expected format is pathtofile,YYYY-MM-DD HH:MM:SS -->
<!-- one per line where the date stamp indicates the last modification time of the file -->
<inventory>../listing.csv</inventory>
</server>

</inputdata>

Describe your problem or question:
The case sets up and builds fine but after I try to submit I get the following:
dnml
Calling /home/jjs1u25/my_cesm_sandbox/cime/src/components/stub_comps/sesp/cime_config/buil dnml
Calling /home/jjs1u25/my_cesm_sandbox/cime/src/drivers/mct/cime_config/buildnml
NOTE: ignoring setting of rof2ocn_ice_rmapname=idmap in seq_maps.rc
NOTE: ignoring setting of rof2ocn_liq_rmapname=idmap in seq_maps.rc
Finished creating component namelists
-------------------------------------------------------------------------
- Prestage required restarts into /home/jjs1u25/projects/cesm/scratch/case01/run
- Case input data directory (DIN_LOC_ROOT) is /home/jjs1u25/projects/cesm/inputdata
- Checking for required input datasets in DIN_LOC_ROOT
-------------------------------------------------------------------------
2025-10-20 16:09:50 MODEL EXECUTION BEGINS HERE
run command is mpiexec -np 16 /home/jjs1u25/projects/cesm/scratch/case01/bld/cesm.exe >> ce sm.log.$LID 2>&1
ERROR: RUN FAIL: Command 'mpiexec -np 16 /home/jjs1u25/projects/cesm/scratch/case01/bld/cesm .exe >> cesm.log.$LID 2>&1 ' failed
See log file for details: /home/jjs1u25/projects/cesm/scratch/case01/run/cesm.log.251020-1609 47
jjs1u25@uos-22770:~/projects/cesm/cases/case01 $ tail -n 20 /home/jjs1u25/projects/cesm/scratch/case01/run/cesm.log.251020-160947
#14 0x4b2aff in ???
#15 0x4afa0f in ???
#16 0x428cd3 in ???
#17 0x41fc97 in ???
#18 0x42645a in ???
#19 0x7f4ab46295cf in ???
#20 0x7f4ab462967f in ???
#21 0x40aea4 in ???
#22 0xffffffffffffffff in ???

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 3856175 RUNNING AT uos-22770
= EXIT CODE: 9
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
 
J

Jakub_Sypek

Jakub
New Member
Upon further inspection of the log file it looks like there is a segmentation fault that is causing the problem:

MCT::m_Router::initp_: GSMap indices not increasing...Will correct
MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
MCT::m_Router::initp_: GSMap indices not increasing...Will correct
Opened existing file /home/jjs1u25/projects/cesm/inputdata/atm/waccm/geomag/igrf_ceofs_c160412.nc 104
** On entry to DGBSV parameter number 1 had an illegal value
** On entry to DGBSV parameter number 1 had an illegal value
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
** On entry to DGBSV parameter number 1 had an illegal value
** On entry to DGBSV parameter number 1 had an illegal value
** On entry to DGBSV parameter number 1 had an illegal value

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
 
J

jedwards

CSEG and Liaisons
Staff member
I would recommend confirming your port of the model with some simpler compsets and cases prior to attempting a WACCM compset.
DGBSV is in the lapack library external to cesm. Since you have just opened a file, it may be due to a corrupted input. This file has
an md5sum value of 876dd65bf11cb79a6e72cc6eb979cd77 inputdata/atm/waccm/geomag/igrf_ceofs_c160412.nc

I also see there is an update of that file in igrf_ceofs_c230420.nc you can try adding this using user_nl_cam.
 
You must log in or register to reply here.
Top