Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

CESM 2.2 NETCDF issue!

Y

yelshorbany@gmail_com

New Member
Hi, newbie here and ai trying to install CESM2.2 on the HPC RC USF.
The case.setup works but the case.build fail as following:
elshorbany@itn2.rc.usf.edu:/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17$ ./case.build
Building case in directory /work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17
ERROR: /home/e/elshorbany/CESM/CESM_2.2.0/cime/src/build_scripts/buildlib.mct FAILED, cat /work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/mct.bldlog.210225-161514

The log file shows the following:
Make -f /work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17/Tools/Makefile -C /work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/mct CIME_MODEL=cesm SMP=FALSE CASEROOT="/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17" CASETOOLS="/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17/Tools" CIMEROOT="/home/e/elshorbany/CESM/CESM_2.2.0/cime" COMP_INTERFACE="mct" COMPILER="intel" DEBUG="FALSE" EXEROOT="/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld" INCROOT="/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/lib/include" LIBROOT="/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/lib" MACH="circe" MPILIB="impi" NINST_VALUE="c1a1l1i1o1r1g1w1i1e1" OS="LINUX" PIO_VERSION="1" SHAREDLIBROOT="/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld" SMP_PRESENT="FALSE" USE_ESMF_LIB="FALSE" USE_MOAB="FALSE" CAM_CONFIG_OPTS="-phys cam6" COMP_LND="clm" COMPARE_TO_NUOPC="FALSE" CISM_USE_TRILINOS="FALSE" USE_TRILINOS="FALSE" USE_ALBANY="FALSE" USE_PETSC="FALSE" MODEL=mct COMP_NAME=mct /work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/mct/Makefile.conf
make: Entering directory `/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/mct'
make: Leaving directory `/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/mct'
cat: Filepath: No such file or directory
/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17/Tools/Makefile:183: *** "NETCDF_FORTRAN_PATH specified without NETCDF_C_PATH". Stop.
ERROR: cat: Filepath: No such file or directory
/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17/Tools/Makefile:183: *** "NETCDF_FORTRAN_PATH specified without NETCDF_C_PATH". Stop.
~
I also ran "scripts_regression_tests.py" and got the attached "regressions_test"

Below are also the .cime/config_machines.xml and config_compilers.xml
Config_machines.xml
<?xml version="1.0"?>
<machine MACH="circe">
<DESC>USF CIRCE Cluster, OS is Linux, batch system is SLURM</DESC>
<NODENAME_REGEX>.*.rc.usf.edu</NODENAME_REGEX>
<OS>LINUX</OS>
<COMPILERS>intel</COMPILERS>
<MPILIBS>impi</MPILIBS>
<CIME_OUTPUT_ROOT>/work/e/elshorbany/CESM/output</CIME_OUTPUT_ROOT>
<DIN_LOC_ROOT>/shares/elshorbany_group/CESM/input_data</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/shares/elshorbany_group/CESM/input_data/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>$CIME_OUTPUT_ROOT/archive/$CASE</DOUT_S_ROOT>
<BASELINE_ROOT>$ENV{HOME}/CESM/baselines</BASELINE_ROOT>
<CCSM_CPRNC>$CIMEROOT/tools/cprnc/build/cprnc</CCSM_CPRNC>
<GMAKE>make</GMAKE>
<GMAKE_J>10</GMAKE_J>
<BATCH_SYSTEM>slurm</BATCH_SYSTEM>
<SUPPORTED_BY>elshorbany@usf.edu</SUPPORTED_BY>
<MAX_TASKS_PER_NODE>10</MAX_TASKS_PER_NODE>
<MAX_MPITASKS_PER_NODE>10</MAX_MPITASKS_PER_NODE>
<PROJECT_REQUIRED>FALSE</PROJECT_REQUIRED>
<mpirun mpilib="impi">
<executable>mpirun</executable>
<arguments>
<arg name="num_tasks">-n {{ total_tasks }}</arg>
</arguments>
</mpirun>
<module_system type="module">
<init_path lang="perl">/usr/share/Modules/init/perl.pm</init_path>
<init_path lang="csh">/etc/profile.d/modules.csh</init_path>
<init_path lang="sh">/etc/profile.d/modules.sh</init_path>
<init_path lang="python">/usr/share/Modules/init/python.py</init_path>
<cmd_path lang="perl">/usr/bin/modulecmd perl </cmd_path>
<cmd_path lang="sh">module</cmd_path>
<cmd_path lang="csh">module</cmd_path>
<cmd_path lang="python">/usr/bin/modulecmd python</cmd_path>
<modules>
<command name="purge"/>
</modules>
<modules>
<command name="load">apps/perl/5.28.0</command>
<command name="load">apps/cmake/3.7.1</command>
<command name="load">apps/python/2.7.11</command>
<command name="load">apps/netcdf/4.6.3</command>
<command name="load">apps/pnetcdf/1.11.0</command>
<command name="load">apps/netcdf-fortran/4.4.5</command>
</modules>
<modules compiler="intel">
<command name="load">compilers/intel/2017_cluster_xe</command>
</modules>
</module_system>
<environment_variables compiler="intel">
<env name="NETCDF_PATH">$ENV{NETCDF}</env>
<env name="NETCDF_HOME">$ENV{NETCDF}</env>
<env name="MKL_PATH">$ENV{MKLROOT}</env>
</environment_variables>
</machine>

config_compiler.xml
<?xml version="1.0" encoding="UTF-8"?>

<compiler MACH="circe" COMPILER="intel">
<CFLAGS>
<append>-I/apps/netcdf/4.6.3/include -I/apps/pnetcdf/1.11.0/include -I/apps/netcdf-fortran/4.4.5/include </append>
</CFLAGS>
<FFLAGS>
<append>-I/apps/netcdf/4.6.3/include -I/apps/pnetcdf/1.11.0/include -I/apps/netcdf-fortran/4.4.5/include </append>
</FFLAGS>
<NETCDF_PATH>/apps/netcdf/4.6.3</NETCDF_PATH>
<NETCDF_FORTRAN_PATH>/apps/netcdf-fortran/4.4.5/</NETCDF_FORTRAN_PATH>
<PNETCDF_PATH>/apps/pnetcdf/1.11.0</PNETCDF_PATH>
<SLIBS>
<append> -L/apps/netcdf/4.6.3/lib -L/apps/pnetcdf/1.11.0/lib -L/apps/netcdf-fortran/4.4.5/lib -lnetcdf -lnetcdff -lpmi -L$ENV{MKLROOT} -lmkl_rt </append>
</SLIBS>
</compiler>

Thanks!
 

Attachments

  • regressions_test.txt
    13 KB · Views: 2
J

jedwards

CSEG and Liaisons
Staff member
Because your netcdf c and fortran paths are different you need to set both
NETCDF_FORTRAN_PATH and NETCDF_C_PATH in config_compilers.xml

I think that you just need to change NETCDF_PATH to NETCDF_C_PATH -
but you shouldn't hardcode the paths in config_compilers.xml
use module show apps/netcdf-fortran/4.4.5
and module show apps/netcdf/4.6.3
to learn the env variable that modules set - it looks like one of them is $NETCDF
(for now lets assume the other is NETCDF_FORTRAN) then in config_compilers.xml use

<NETCDF_C_PATH>$ENV{NETCDF}</NETCDF_C_PATH>
<NETCDF_FORTRAN_PATH>$ENV{NETCDF_FORTRAN}</NETCDF_FORTRAN_PATH>

and you don't need the netcdf settings in the environment section of config_machines.xml
 
Y

yelshorbany@gmail_com

New Member
Here what I did:
elshorbany@itn1.rc.usf.edu:/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17$ module show apps/netcdf-fortran/4.4.5
-------------------------------------------------------------------
/etc/modulefiles/apps/netcdf-fortran/4.4.5:

module-whatis adds netcdf-fortran to your PATH environment variable
setenv NFDIR /apps/netcdf-fortran/4.4.5
prepend-path PATH /apps/netcdf-fortran/4.4.5/bin
append-path LD_LIBRARY_PATH /apps/netcdf-fortran/4.4.5/lib
module add apps/hdf5/1.10.1-intel_parallel
module add apps/zlib/1.2.11
module add apps/curl/7.54.0
module add apps/netcdf/4.6.3
module add apps/pnetcdf/1.11.0
-------------------------------------------------------------------

elshorbany@itn1.rc.usf.edu:/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17$ module show apps/netcdf/4.6.3
-------------------------------------------------------------------
/etc/modulefiles/apps/netcdf/4.6.3:

module-whatis adds netcdf to your PATH environment variable
setenv NETCDF /apps/netcdf/4.6.3
setenv NETCDF_HOME /apps/netcdf/4.6.3
setenv NETCDF_PATH /apps/netcdf/4.6.3
setenv NCDIR /apps/netcdf/4.6.3
prepend-path PATH /apps/netcdf/4.6.3/bin
append-path LD_LIBRARY_PATH /apps/netcdf/4.6.3/lib
module add apps/hdf5/1.10.1-intel_parallel
module add apps/zlib/1.2.11
module add apps/curl/7.54.0
—————————————————————————————————

I then set my config_compiler.xml as following:
<?xml version="1.0" encoding="UTF-8"?>

<compiler MACH="circe" COMPILER="intel">
<CFLAGS>
<append>-I/apps/netcdf/4.6.3/include -I/apps/pnetcdf/1.11.0/include -I/apps/netcdf-fortran/4.4.5/include </append>
</CFLAGS>
<FFLAGS>
<append>-I/apps/netcdf/4.6.3/include -I/apps/pnetcdf/1.11.0/include -I/apps/netcdf-fortran/4.4.5/include </append>
</FFLAGS>
<NETCDF_C_PATH>$ENV{NETCDF}</NETCDF_C_PATH>
<NETCDF_FORTRAN_PATH>$ENV{NFDIR}</NETCDF_FORTRAN_PATH>
<PNETCDF_PATH>/apps/pnetcdf/1.11.0</PNETCDF_PATH>
<SLIBS>
<append> -L$(NETCDF_C_PATH)/lib -L$(NETCDF_FORTRAN_PATH)/lib -lnetcdff -lnetcdf -L$(PNETCDF_PATH)/lib -lpnetcdf -lpmi -L$ENV{MKLROOT} -lmkl_rt </append>
</SLIBS>
</compiler>

Still got the same error:
elshorbany@itn0.rc.usf.edu:/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17$ ./case.build
Building case in directory /work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17
sharedlib_only is False
model_only is Falseedu:/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17$
Setting Environment NETCDF_PATH=/apps/netcdf/4.6.3
Setting Environment NETCDF_HOME=/apps/netcdf/4.6.3
Setting Environment MKL_PATH=/apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl
Generating component namelists as part of build
Creating component namelists
2021-02-28 11:29:20 atm
Calling /home/e/elshorbany/CESM/CESM_2.2.0/components/cam//cime_config/buildnml
...calling cam buildcpp to set build time options
CAM namelist copy: file1 /work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17/Buildconf/camconf/atm_in file2 /work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/run/atm_in
2021-02-28 11:29:21 lnd
Calling /home/e/elshorbany/CESM/CESM_2.2.0/components/clm//cime_config/buildnml
2021-02-28 11:29:21 ice
Calling /home/e/elshorbany/CESM/CESM_2.2.0/components/cice//cime_config/buildnml
...buildnml calling cice buildcpp to set build time options
2021-02-28 11:29:21 ocn
Calling /home/e/elshorbany/CESM/CESM_2.2.0/cime/src/components/data_comps_mct/docn/cime_config/buildnml
2021-02-28 11:29:21 rof
Calling /home/e/elshorbany/CESM/CESM_2.2.0/components/mosart//cime_config/buildnml
2021-02-28 11:29:21 glc
Running /home/e/elshorbany/CESM/CESM_2.2.0/components/cism//cime_config/buildnml
2021-02-28 11:29:23 wav
Calling /home/e/elshorbany/CESM/CESM_2.2.0/cime/src/components/stub_comps_mct/swav/cime_config/buildnml
2021-02-28 11:29:23 iac
Running /home/e/elshorbany/CESM/CESM_2.2.0/cime/src/components/stub_comps_mct/siac/cime_config/buildnml
2021-02-28 11:29:23 esp
Calling /home/e/elshorbany/CESM/CESM_2.2.0/cime/src/components/stub_comps_mct/sesp/cime_config/buildnml
2021-02-28 11:29:23 cpl
Calling /home/e/elshorbany/CESM/CESM_2.2.0/cime/src/drivers/mct/cime_config/buildnml
NOTE: ignoring setting of rof2ocn_liq_rmapname=idmap in seq_maps.rc
NOTE: ignoring setting of rof2ocn_fmapname=idmap in seq_maps.rc
NOTE: ignoring setting of rof2ocn_ice_rmapname=idmap in seq_maps.rc
Building gptl with output to file /work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/gptl.bldlog.210228-112920
Calling /home/e/elshorbany/CESM/CESM_2.2.0/cime/src/build_scripts/buildlib.gptl
Building mct with output to file /work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/mct.bldlog.210228-112920
Calling /home/e/elshorbany/CESM/CESM_2.2.0/cime/src/build_scripts/buildlib.mct
ERROR: /home/e/elshorbany/CESM/CESM_2.2.0/cime/src/build_scripts/buildlib.mct FAILED, cat /work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/mct.bldlog.210228-112920
 
J

jedwards

CSEG and Liaisons
Staff member
This is not the same error that you got before. Please show me the contents of mct.bldlog.210228-112920 and of config.log in the mct
directory under the bld path.
 
Y

yelshorbany@gmail_com

New Member
Here is the mct log file (higher number but same setting):
make -f /work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17/Tools/Makefile -C /work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/mct CIME_MODEL=cesm SMP=FALSE CASEROOT="/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17" CASETOOLS="/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17/Tools" CIMEROOT="/home/e/elshorbany/CESM/CESM_2.2.0/cime" COMP_INTERFACE="mct" COMPILER="intel" DEBUG="FALSE" EXEROOT="/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld" INCROOT="/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/lib/include" LIBROOT="/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/lib" MACH="circe" MPILIB="impi" NINST_VALUE="c1a1l1i1o1r1g1w1i1e1" OS="LINUX" PIO_VERSION="1" SHAREDLIBROOT="/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld" SMP_PRESENT="FALSE" USE_ESMF_LIB="FALSE" USE_MOAB="FALSE" CAM_CONFIG_OPTS="-phys cam6" COMP_LND="clm" COMPARE_TO_NUOPC="FALSE" CISM_USE_TRILINOS="FALSE" USE_TRILINOS="FALSE" USE_ALBANY="FALSE" USE_PETSC="FALSE" MODEL=mct COMP_NAME=mct /work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/mct/Makefile.conf
make: Entering directory `/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/mct'
make: Leaving directory `/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/mct'
cat: Filepath: No such file or directory
/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17/Tools/Makefile:183: *** "NETCDF_FORTRAN_PATH specified without NETCDF_C_PATH". Stop.
ERROR: cat: Filepath: No such file or directory
/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17/Tools/Makefile:183: *** "NETCDF_FORTRAN_PATH specified without NETCDF_C_PATH". Stop.

And gptl.bldlog.210228-121410 is here:
make -f /home/e/elshorbany/CESM/CESM_2.2.0/cime/src/share/timing/Makefile install -C /work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/gptl MACFILE=/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17/Macros.make MODEL=gptl COMP_NAME=gptl GPTL_DIR=/home/e/elshorbany/CESM/CESM_2.2.0/cime/src/share/timing GPTL_LIBDIR=/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/gptl SHAREDPATH=/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct CIME_MODEL=cesm SMP=FALSE CASEROOT="/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17" CASETOOLS="/work/e/elshorbany/CESM/cases/cesm220FHIST_f09_f09_mg17/Tools" CIMEROOT="/home/e/elshorbany/CESM/CESM_2.2.0/cime" COMP_INTERFACE="mct" COMPILER="intel" DEBUG="FALSE" EXEROOT="/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld" INCROOT="/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/lib/include" LIBROOT="/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/lib" MACH="circe" MPILIB="impi" NINST_VALUE="c1a1l1i1o1r1g1w1i1e1" OS="LINUX" PIO_VERSION="1" SHAREDLIBROOT="/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld" SMP_PRESENT="FALSE" USE_ESMF_LIB="FALSE" USE_MOAB="FALSE" CAM_CONFIG_OPTS="-phys cam6" COMP_LND="clm" COMPARE_TO_NUOPC="FALSE" CISM_USE_TRILINOS="FALSE" USE_TRILINOS="FALSE" USE_ALBANY="FALSE" USE_PETSC="FALSE"
make: Entering directory `/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/gptl'
cp -p /home/e/elshorbany/CESM/CESM_2.2.0/cime/src/share/timing/gptl.h /work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/include
cp -p *.mod /work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/include
cp -p libgptl.a /work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/lib
make: Leaving directory `/work/e/elshorbany/CESM/output/cesm220FHIST_f09_f09_mg17/bld/intel/impi/nodebug/nothreads/mct/gptl'
 
Y

yelshorbany@gmail_com

New Member
MODEL BUILD HAS FINISHED SUCCESSFULLY!
Here is what I did:
Update the config_compiler.xml as Jim outlined:
<compiler MACH="circe" COMPILER="intel">
<CFLAGS>
<append>-I$(NETCDF)/include -I$(PNETCDF)/include -I$(NFDIR)/include </append>
</CFLAGS>
<FFLAGS>
<append>-I$(NETCDF)/include -I$(PNETCDF_PATH)/include -I$(NFDIR)/include </append>
</FFLAGS>
<NETCDF_C_PATH>$ENV{NETCDF}</NETCDF_C_PATH>
<NETCDF_FORTRAN_PATH>$ENV{NFDIR}</NETCDF_FORTRAN_PATH>
<PNETCDF_PATH>$ENV{PNETCDF}</PNETCDF_PATH>
<SLIBS>
<append> -L$(NETCDF_C_PATH)/lib -L$(NETCDF_FORTRAN_PATH)/lib -lnetcdff -lnetcdf -L$(PNETCDF_PATH)/lib -lpnetcdf -lpmi -L$ENV{MKLROOT} -lmkl_rt </append>
</SLIBS>
</compiler>
Click to expand...
Load the required modules only: module add apps/perl/5.28.0 apps/cmake/3.7.1 apps/python/2.7.11 apps/netcdf-fortran/4.4.5
Make new case
elshorbany@itn1.rc.usf.edu:/home/e/elshorbany/CESM/CESM_2.2.0/cime/scripts$
./create_newcase --case /work/e/elshorbany/CESM/cases/cesm220_2nd_FHIST_f09_f09_mg17 --res f09_f09_mg17 --compset FHIST
cd /work/e/elshorbany/CESM/cases/cesm220_2nd_FHIST_f09_f09_mg17
./case.setup
./case.build
 
Y

yelshorbany@gmail_com

New Member
After submitting the job, I got the following error:
error while loading shared libraries: libhdf5_hl.so.6: cannot open shared object file: No such file or directory
LDD cesm.exe prdocuce the follwoing:
linux-vdso.so.1 => (0x00007fff9cac7000)

libnetcdff.so.6 => /apps/netcdf-fortran/4.4.5/lib/libnetcdff.so.6 (0x00007f4540685000)

libnetcdf.so.15 => /apps/netcdf/4.6.3/lib/libnetcdf.so.15 (0x00007f4540285000)

libpnetcdf.so.3 => /apps/pnetcdf/1.11.0/lib/libpnetcdf.so.3 (0x00007f453f74b000)

libpmi.so.0 => /usr/lib64/libpmi.so.0 (0x00007f453f523000)

libmkl_rt.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_rt.so (0x00007f453ef33000)

libhdf5_hl.so.8 => /usr/lib64/libhdf5_hl.so.8 (0x00007f453ecfe000)

libmkl_intel_lp64.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_intel_lp64.so (0x00007f453e1e0000)

libmkl_cdft_core.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_cdft_core.so (0x00007f453dfb6000)

libmkl_scalapack_lp64.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_scalapack_lp64.so (0x00007f453d6d2000)

libmkl_blacs_intelmpi_lp64.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so (0x00007f453d496000)

libmkl_sequential.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_sequential.so (0x00007f453c6db000)

libmkl_core.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_core.so (0x00007f453abeb000)

libiomp5.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libiomp5.so (0x00007f453a8a4000)

libmpifort.so.12 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mpi/lib64/libmpifort.so.12 (0x00007f453a4fb000)

libmpi.so.12 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mpi/lib64/libmpi.so.12 (0x00007f45397f3000)

libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f45395ef000)

librt.so.1 => /usr/lib64/librt.so.1 (0x00007f45393e7000)

libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f45391ca000)

libm.so.6 => /usr/lib64/libm.so.6 (0x00007f4538ec8000)

libc.so.6 => /usr/lib64/libc.so.6 (0x00007f4538afb000)

libgcc_s.so.1 => /apps/gcc/6.2.0/lib64/libgcc_s.so.1 (0x00007f45388e4000)

libhdf5_hl.so.6 => not found

libhdf5.so.6 => not found

libcurl.so.4 => /apps/curl/7.54.0/lib/libcurl.so.4 (0x00007f453864e000)

libifport.so.5 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libifport.so.5 (0x00007f453841f000)

libifcoremt.so.5 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libifcoremt.so.5 (0x00007f4538090000)

libimf.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libimf.so (0x00007f4537ba4000)

libsvml.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libsvml.so (0x00007f4536c8e000)

libintlc.so.5 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libintlc.so.5 (0x00007f4536a23000)

libirc.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libirc.so (0x00007f45367b9000)

libhdf5_hl.so.100 => /apps/hdf5/1.10.1-intel_parallel/lib/libhdf5_hl.so.100 (0x00007f4536596000)

libhdf5.so.101 => /apps/hdf5/1.10.1-intel_parallel/lib/libhdf5.so.101 (0x00007f4535fd8000)

libz.so.1 => /apps/zlib/1.2.11/lib/libz.so.1 (0x00007f4535dc0000)

libirng.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libirng.so (0x00007f4535a38000)

libmpicxx.so.12 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mpi/lib64/libmpicxx.so.12 (0x00007f4535818000)

libcilkrts.so.5 => /apps/gcc/6.2.0/lib64/libcilkrts.so.5 (0x00007f45355fb000)

libstdc++.so.6 => /apps/gcc/6.2.0/lib64/libstdc++.so.6 (0x00007f4535267000)

libslurm_pmi.so => /usr/lib64/slurm/libslurm_pmi.so (0x00007f4534ea4000)

libresolv.so.2 => /usr/lib64/libresolv.so.2 (0x00007f4534c8b000)

libhdf5.so.8 => /usr/lib64/libhdf5.so.8 (0x00007f4534694000)

libsz.so.2 => /usr/lib64/libsz.so.2 (0x00007f4534490000)

/lib64/ld-linux-x86-64.so.2 (0x000055b409166000)

libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f453421f000)

libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007f4533dbd000)

libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f4533b70000)

libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f4533888000)

libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f4533654000)

libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f4533450000)

libaec.so.0 => /usr/lib64/libaec.so.0 (0x00007f4533247000)

libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f4533039000)

libkeyutils.so.1 => /usr/lib64/libkeyutils.so.1 (0x00007f4532e35000)

libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f4532c0d000)

libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f45329ab000)
Follow below:
 
Y

yelshorbany@gmail_com

New Member
So, it seems that CESM is looking for these files (libhdf5_hl.so.6, libhdf5.so.6) but not finding them.
I will try to link these files in my home directory (/home/e/elshorbany/CESM/lib) to the existing libraries:
ln -sf /usr/lib64/libhdf5_hl.so.8.0.1 libhdf5_hl.so.6
add to the path:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/e/elshorbany/CESM/lib

Amend the config_compiler.xml:
<append> -L$(NETCDF_C_PATH)/lib -L$(NETCDF_FORTRAN_PATH)/lib -lnetcdff -lnetcdf -L$(PNETCDF_PATH)/lib -lpnetcdf -lpmi -L$ENV{MKLROOT} -lmkl_rt -L/home/e/elshorbany/CESM/lib -lhdf5_hl -lhdf5</append>

Remove and rebuild a new case!

rm -rf /work/e/elshorbany/CESM/output/cesm220_5_FHIST_f09_f09_mg17/

cd ../

rm -rf cesm220_5_FHIST_f09_f09_mg17

cd ~/CESM/CESM_2.2.0/cime/scripts/

./create_newcase --case /work/e/elshorbany/CESM/cases/cesm220_5_FHIST_f09_f09_mg17 --res f09_f09_mg17 --compset FHIST

cd /work/e/elshorbany/CESM/cases/cesm220_5_FHIST_f09_f09_mg17

./case.setup

./case.build
MODEL BUILD HAS FINISHED SUCCESSFULLY

Now ldd cesium.exe produce the following:
linux-vdso.so.1 => (0x00007fffe95fb000)

libnetcdff.so.6 => /apps/netcdf-fortran/4.4.5/lib/libnetcdff.so.6 (0x00007f011c71e000)

libnetcdf.so.15 => /apps/netcdf/4.6.3/lib/libnetcdf.so.15 (0x00007f011c31e000)

libpnetcdf.so.3 => /apps/pnetcdf/1.11.0/lib/libpnetcdf.so.3 (0x00007f011b7e4000)

libpmi.so.0 => /usr/lib64/libpmi.so.0 (0x00007f011b5bc000)

libmkl_rt.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_rt.so (0x00007f011afcc000)

libhdf5_hl.so.8 => /home/e/elshorbany/CESM/lib/libhdf5_hl.so.8 (0x00007f011ad97000)

libhdf5.so.8 => /usr/lib64/libhdf5.so.8 (0x00007f011a7a0000)

libmkl_intel_lp64.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_intel_lp64.so (0x00007f0119c82000)

libmkl_cdft_core.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_cdft_core.so (0x00007f0119a57000)

libmkl_scalapack_lp64.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_scalapack_lp64.so (0x00007f0119174000)

libmkl_blacs_intelmpi_lp64.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so (0x00007f0118f38000)

libmkl_sequential.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_sequential.so (0x00007f011817c000)

libmkl_core.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_core.so (0x00007f011668d000)

libiomp5.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libiomp5.so (0x00007f0116345000)

libmpifort.so.12 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mpi/lib64/libmpifort.so.12 (0x00007f0115f9c000)

libmpi.so.12 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mpi/lib64/libmpi.so.12 (0x00007f0115295000)

libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f0115091000)

librt.so.1 => /usr/lib64/librt.so.1 (0x00007f0114e88000)

libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f0114c6c000)

libm.so.6 => /usr/lib64/libm.so.6 (0x00007f011496a000)

libc.so.6 => /usr/lib64/libc.so.6 (0x00007f011459c000)

libgcc_s.so.1 => /apps/gcc/6.2.0/lib64/libgcc_s.so.1 (0x00007f0114386000)

libhdf5_hl.so.6 => /home/e/elshorbany/CESM/lib/libhdf5_hl.so.6 (0x00007f0114152000)

libcurl.so.4 => /apps/curl/7.54.0/lib/libcurl.so.4 (0x00007f0113ebc000)

libifport.so.5 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libifport.so.5 (0x00007f0113c8d000)

libifcoremt.so.5 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libifcoremt.so.5 (0x00007f01138ff000)

libimf.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libimf.so (0x00007f0113412000)

libsvml.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libsvml.so (0x00007f01124fc000)

libintlc.so.5 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libintlc.so.5 (0x00007f0112292000)

libirc.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libirc.so (0x00007f0112027000)

libhdf5_hl.so.100 => /apps/hdf5/1.10.1-intel_parallel/lib/libhdf5_hl.so.100 (0x00007f0111e05000)

libhdf5.so.101 => /apps/hdf5/1.10.1-intel_parallel/lib/libhdf5.so.101 (0x00007f0111846000)

libz.so.1 => /apps/zlib/1.2.11/lib/libz.so.1 (0x00007f011162e000)

libirng.so => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64/libirng.so (0x00007f01112a7000)

libmpicxx.so.12 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mpi/lib64/libmpicxx.so.12 (0x00007f0111086000)

libcilkrts.so.5 => /apps/gcc/6.2.0/lib64/libcilkrts.so.5 (0x00007f0110e6a000)

libstdc++.so.6 => /apps/gcc/6.2.0/lib64/libstdc++.so.6 (0x00007f0110ad5000)

libslurm_pmi.so => /usr/lib64/slurm/libslurm_pmi.so (0x00007f0110713000)

libresolv.so.2 => /usr/lib64/libresolv.so.2 (0x00007f01104f9000)

libsz.so.2 => /usr/lib64/libsz.so.2 (0x00007f01102f6000)

/lib64/ld-linux-x86-64.so.2 (0x00005572dc2a9000)

libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f0110084000)

libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007f010fc23000)

libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f010f9d6000)

libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f010f6ed000)

libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f010f4ba000)

libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f010f2b6000)

libaec.so.0 => /usr/lib64/libaec.so.0 (0x00007f010f0ad000)

libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f010ee9f000)

libkeyutils.so.1 => /usr/lib64/libkeyutils.so.1 (0x00007f010ec9a000)

libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f010ea73000)

libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f010e810000)

Now, since all libraries are linked, hope this works. I am not sure why need both libhdf5_hl.so.8 and libhdf5_hl.so.6. Will update this post once the case run.
 
Y

yelshorbany@gmail_com

New Member
I got the same error: "error while loading shared libraries: libhdf5_hl.so.6: cannot open shared object file: No such file or directory"

ldd cesm.exe will produce:
linux-vdso.so.1 => (0x00007ffcda3b6000)
libnetcdff.so.6 => not found
libnetcdf.so.15 => not found
libpnetcdf.so.3 => not found
libpmi.so.0 => /usr/lib64/libpmi.so.0 (0x00007f48aa545000)
libmkl_rt.so => not found
libhdf5_hl.so.8 => /home/e/elshorbany/CESM/lib/libhdf5_hl.so.8 (0x00007f48aa310000)
libhdf5.so.8 => /usr/lib64/libhdf5.so.8 (0x00007f48a9d19000)
libmkl_intel_lp64.so => not found
libmkl_cdft_core.so => not found
libmkl_scalapack_lp64.so => not found
libmkl_blacs_intelmpi_lp64.so => not found
libmkl_sequential.so => not found
libmkl_core.so => not found
libiomp5.so => not found
libmpifort.so.12 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/libmpifort.so.12 (0x00007f48a996d000)
libmpi.so.12 => /apps/intel/2017/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib/release_mt/libmpi.so.12 (0x00007f48a8c66000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f48a8a61000)
librt.so.1 => /usr/lib64/librt.so.1 (0x00007f48a8859000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f48a863d000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00007f48a833a000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00007f48a7f6d000)
libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f48a7d57000)
libslurm_pmi.so => /usr/lib64/slurm/libslurm_pmi.so (0x00007f48a7994000)
libresolv.so.2 => /usr/lib64/libresolv.so.2 (0x00007f48a777b000)
libsz.so.2 => /usr/lib64/libsz.so.2 (0x00007f48a7578000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00007f48a7361000)
/lib64/ld-linux-x86-64.so.2 (0x000055ba0c2e9000)
libaec.so.0 => /usr/lib64/libaec.so.0 (0x00007f48a7159000)

I also make echo $LD_LIBRARY_PATH and don't see any linked libraries, which were linked before executing the run (see my previous post)??

So the model does not see my libraries during the run??

I understand CESM will get the libraries from the environmental variables (but this requires the modules to loaded)? Does CESM requires loading the modules before running, if so where is this defined?

Should I define all the libraries in my ~/.bashrc via "export $LD_LIBRARY_PATH for all libraries"

Thanks,
 
Y

yelshorbany@gmail_com

New Member
By teh way, here is my config_batch.xml:
<?xml version="1.0"?>
<batch_system MACH="circe" type="slurm" >
<batch_submit>sbatch</batch_submit>
<submit_args>
<arg flag="--mail-user" name="elshorbany@usf.edu"/>
<arg flag="--mail-type" name="all"/>
<arg flag="--time" name="$JOB_WALLCLOCK_TIME"/>
<arg flag="--partition" name="devel"/>
</submit_args>
<queues>
<queue walltimemax="00:30:00" nodemin="1" default="true">circe</queue>
</queues>
</batch_system>

Maybe we can add "module load" here?
 
J

jedwards

CSEG and Liaisons
Staff member
CESM loads the modules as part of case.submit. If you would like to load the same modules in your login environment you may run
source .env_mach_specific.sh (or .csh if that's your shell). This issue of hdf so.6 vs so.8 is because of some inconsistency in the build of
modules on your system - you should not be required to have both and should not be doing flips and twists to accommodate both.
 
Y

yelshorbany@gmail_com

New Member
But the modules are apparently not loaded during the run and/or that my linked library (to account for so.6) in the home directory are not seen. How can I source my ~/.bashrc during the run, should I add “source ~/.bashrc” in the .come/config_batch.xml at the end? Should I also load the modules in .come/config_batch.xml?
 
J

jedwards

CSEG and Liaisons
Staff member
These are system issues and not cesm issues. No you should not attempt to load modules in config_batch.xml I don't even understand where you got that idea. Your .batchrc is also irrelevant here and should not be referenced from cesm scripts.
 
Y

yelshorbany@gmail_com

New Member
I wanted to source .batchrc because this is where I "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/e/elshorbany/CESM/lib" which contains a soft link to the so.6. I know we shouldn't need both so.6 and so.8 and I am trying to find a solution until we figure where this is coming from.
Just to confirm, modules are loaded during the run based on what is loaded in the .cim/config_machines.xml, right?

Also, what kind of inconsistency do you think could be in our modules?

Here is my config_machines.xml:
<?xml version="1.0"?>

<machine MACH="circe">
<DESC>USF CIRCE Cluster, OS is Linux, batch system is SLURM</DESC>
<NODENAME_REGEX>.*.rc.usf.edu</NODENAME_REGEX>
<OS>LINUX</OS>
<COMPILERS>intel</COMPILERS>
<MPILIBS>impi</MPILIBS>
<CIME_OUTPUT_ROOT>/work/e/elshorbany/CESM/output</CIME_OUTPUT_ROOT>
<DIN_LOC_ROOT>/shares/elshorbany_group/CESM/input_data</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/shares/elshorbany_group/CESM/input_data/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>$CIME_OUTPUT_ROOT/archive/$CASE</DOUT_S_ROOT>
<BASELINE_ROOT>$ENV{HOME}/CESM/baselines</BASELINE_ROOT>
<CCSM_CPRNC>$CIMEROOT/tools/cprnc/build/cprnc</CCSM_CPRNC>
<GMAKE>make</GMAKE>
<GMAKE_J>10</GMAKE_J>
<BATCH_SYSTEM>slurm</BATCH_SYSTEM>
<SUPPORTED_BY>elshorbany@usf.edu</SUPPORTED_BY>
<MAX_TASKS_PER_NODE>10</MAX_TASKS_PER_NODE>
<MAX_MPITASKS_PER_NODE>10</MAX_MPITASKS_PER_NODE>
<PROJECT_REQUIRED>FALSE</PROJECT_REQUIRED>
<mpirun mpilib="impi">
<executable>mpirun</executable>
<arguments>
<arg name="num_tasks">-n {{ total_tasks }}</arg>
</arguments>
</mpirun>
<module_system type="module">
<init_path lang="perl">/usr/share/Modules/init/perl.pm</init_path>
<init_path lang="csh">/etc/profile.d/modules.csh</init_path>
<init_path lang="sh">/etc/profile.d/modules.sh</init_path>
<init_path lang="python">/usr/share/Modules/init/python.py</init_path>
<cmd_path lang="perl">/usr/bin/modulecmd perl </cmd_path>
<cmd_path lang="sh">module</cmd_path>
<cmd_path lang="csh">module</cmd_path>
<cmd_path lang="python">/usr/bin/modulecmd python</cmd_path>
<modules>
<command name="purge"/>
</modules>
<modules>
<command name="load">apps/perl/5.28.0</command>
<command name="load">apps/cmake/3.7.1</command>
<command name="load">apps/python/2.7.17-intel</command>
<command name="load">apps/netcdf-fortran/4.4.5</command>
</modules>
<modules compiler="intel">
<command name="load">compilers/intel/2017_cluster_xe</command>
</modules>
</module_system>
<environment_variables compiler="intel">
<env name="NETCDF_PATH">$ENV{NETCDF}</env>
<env name="NETCDF_HOME">$ENV{NETCDF}</env>
<env name="MKL_PATH">$ENV{MKLROOT}</env>
<env name="NETCDF_FORTRAN_PATH">$ENV{NFDIR}</env>
</environment_variables>
</machine>
~
 
Y

yelshorbany@gmail_com

New Member
So, I got the model to work but only by the renaming the so.8 to so.6 in my $HOME directory as outlined above. We are also working to find out if there is any inconsistency in the cluster libraries. Thank you for your help so far.
 
You must log in or register to reply here.
Top