build failure on the cedar cluster

[wvsi3w]

Hello,
I tried porting and configuring the CESM2.1.3 model on cedar (one of computecanada clusters) and it failed in the build step.

This is the modules loaded for this configuration (also the config files are attached):
module load StdEnv/2016.4 perl/5.22.4 python/3.7.4 cmake/3.16.3 openmpi/2.1.1 hdf5-mpi/1.8.18 netcdf-mpi/4.4.1.1 netcdf-fortran-mpi/4.4.4

./create_newcase --case $CASES2_1_3/HISTCrop_f09g17 --compset IHistClm50BgcCrop --res f09_g17 --machine cedar --compiler intel --mpilib intelmpi --walltime 02:00:00 --run-unsupported

I must say that the xmllint validates for all 3 config files.

./case.build
ERROR: /project/6001010/my_cesm_sandbox/cime/src/build_scripts/buildlib.mct FAILED, cat /home/meisam/projects/def-hbeltram/cesm2_1_3_OUT/HISTCedarTest/bld/mct.bldlog.231122-145024

And when I open the log file it says:

gmake -f /project/6001010/cesm2_1_3_OUT/HISTCedarTest/Tools/Makefile -C /home/meisam/projects/def-hbeltram/cesm2_1_3_OUT/HISTCedarTest/bld/intel/intelmpi/nod
ebug/nothreads/mct CASEROOT=/project/6001010/cesm2_1_3_OUT/HISTCedarTest MODEL=mct /home/meisam/projects/def-hbeltram/cesm2_1_3_OUT/HISTCedarTest/bld/intel/i
ntelmpi/nodebug/nothreads/mct/Makefile.conf
gmake: Entering directory `/project/6001010/cesm2_1_3_OUT/HISTCedarTest/bld/intel/intelmpi/nodebug/nothreads/mct'
gmake: Leaving directory `/project/6001010/cesm2_1_3_OUT/HISTCedarTest/bld/intel/intelmpi/nodebug/nothreads/mct'
cat: Filepath: No such file or directory
/project/6001010/cesm2_1_3_OUT/HISTCedarTest/Tools/Makefile:199: *** NETCDF not found: Define NETCDF_PATH or NETCDF_C_PATH and NETCDF_FORTRAN_PATH in config_m
achines.xml or config_compilers.xml. Stop.
ERROR: cat: Filepath: No such file or directory
/project/6001010/cesm2_1_3_OUT/HISTCedarTest/Tools/Makefile:199: *** NETCDF not found: Define NETCDF_PATH or NETCDF_C_PATH and NETCDF_FORTRAN_PATH in config_m
achines.xml or config_compilers.xml. Stop.

I loaded all the modules in the config_machines and set them in the bash profile (also tried it without setting them in bash_profile), I don't know why it is having this error.

 

[oleson]

Moving this to the porting forum

 

[jedwards]

You are using modules to load the netcdf these module likely already set NETCDF env variables you can use - to find out what they are
run module show netcdf-mpi/4.4.1.1 and module show netcdf-fortran-mpi/4.4.4

You don't need this line: <env name="PATH">$ENV{PATH}</env>

For config_compilers.xml you might look at the system "grace" for an example except that you can use the env variables you got from module show - so

<NETCDF_C_PATH>$ENV{LOCAL_NETCDF_C_PATH_VAR}</NETCDF_C_PATH>
<NETCDF_FORTRAN_PATH>$ENV{LOCAL_NETCDF_FORTRAN_PATH_VAR}</NETCDF_FORTRAN_PATH>

 

[wvsi3w]

You are using modules to load the netcdf these module likely already set NETCDF env variables you can use - to find out what they are
run module show netcdf-mpi/4.4.1.1 and module show netcdf-fortran-mpi/4.4.4

You don't need this line: <env name="PATH">$ENV{PATH}</env>

For config_compilers.xml you might look at the system "grace" for an example except that you can use the env variables you got from module show - so

<NETCDF_C_PATH>$ENV{LOCAL_NETCDF_C_PATH_VAR}</NETCDF_C_PATH>
<NETCDF_FORTRAN_PATH>$ENV{LOCAL_NETCDF_FORTRAN_PATH_VAR}</NETCDF_FORTRAN_PATH>

Thank you, I removed that redundant line (<env name="PATH">$ENV{PATH}</env>).

However, when I used show commands this happened:

Failed to find the following module(s): "netcdf-mpi/4.4.1.1" in your MODULEPATH
Try:

$ module spider netcdf-mpi/4.4.1.1

to see if the module(s) are available across all compilers and MPI implementations.

So I tried module spider for both nedcdf-mpi and netcdf-fortran-mpi and these showed up:

You will need to load all module(s) on any one of the lines below before the "netcdf-mpi/4.4.1.1" module is available to load.

nixpkgs/16.09 gcc/5.4.0 openmpi/2.0.2
nixpkgs/16.09 gcc/5.4.0 openmpi/2.1.1
nixpkgs/16.09 gcc/6.4.0 openmpi/2.1.1
nixpkgs/16.09 intel/2014.6 openmpi/2.1.1
nixpkgs/16.09 intel/2016.4 openmpi/2.0.2
nixpkgs/16.09 intel/2016.4 openmpi/2.1.1
nixpkgs/16.09 intel/2017.1 openmpi/2.0.2
nixpkgs/16.09 intel/2017.1 openmpi/2.1.1
nixpkgs/16.09 intel/2018.3 impi/2018.3.222
nixpkgs/16.09 intel/2018.3 intelmpi/2018.3.222
nixpkgs/16.09 pgi/17.3 openmpi/2.1.1

You will need to load all module(s) on any one of the lines below before the "netcdf-fortran-mpi/4.4.4" module is available to load.

nixpkgs/16.09 gcc/5.4.0 openmpi/2.0.2
nixpkgs/16.09 gcc/5.4.0 openmpi/2.1.1
nixpkgs/16.09 gcc/6.4.0 openmpi/2.1.1
nixpkgs/16.09 gcc/7.3.0 cuda/9.2.148 openmpi/3.1.2
nixpkgs/16.09 gcc/7.3.0 openmpi/3.1.2
nixpkgs/16.09 gcc/7.3.0 openmpi/3.1.4
nixpkgs/16.09 intel/2014.6 openmpi/2.1.1
nixpkgs/16.09 intel/2016.4 openmpi/2.0.2
nixpkgs/16.09 intel/2016.4 openmpi/2.1.1
nixpkgs/16.09 intel/2017.1 openmpi/2.0.2
nixpkgs/16.09 intel/2017.1 openmpi/2.1.1
nixpkgs/16.09 intel/2018.3 impi/2018.3.222
nixpkgs/16.09 intel/2018.3 intelmpi/2018.3.222
nixpkgs/16.09 intel/2018.3 openmpi/3.1.2
nixpkgs/16.09 intel/2018.3 openmpi/3.1.4


Based on the configuration files that I attached before, do you think I should use newer versions of StdEnv (lets say 2020) so that the rest of the modules can be loaded without any issues? Or maybe I am wrong, and maybe I should contact the cedar cluster support team and ask them the correct module to be used in my case?

 

[wvsi3w]

I tried one of them:
module load nixpkgs/16.09 gcc/5.4.0 openmpi/2.1.1
and did these also:
module load netcdf-mpi/4.4.1.1
module load netcdf-fortran-mpi/4.4.4

then I used the show command for these two modules and added the new path to the config files:
Config_machines:
<environment_variables compiler="intel" mpilib="openmpi">
<env name="NETCDF_FORTRAN_PATH">/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx2/MPI/gcc5.4/openmpi2.1/netcdf-fortran-mpi/4.4.4</env>
<env name="NETCDF_C_PATH">/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx2/MPI/gcc5.4/openmpi2.1/netcdf-mpi/4.4.1.1</env>
<env name="NETCDF_PATH">/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx2/MPI/gcc5.4/openmpi2.1/netcdf-mpi/4.4.1.1</env>
<env name="INC_NETCDF">/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx2/MPI/gcc5.4/openmpi2.1/netcdf-mpi/4.4.1.1/include</env>
<env name="LIB_NETCDF">/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx2/MPI/gcc5.4/openmpi2.1/netcdf-mpi/4.4.1.1/lib64</env>
</environment_variables>

Config_compilers:
<config_compilers version="2.0">
<compiler COMPILER="intel" MACH="cedar">
<SLIBS>
<append MPILIB="openmpi"> -L/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/gcc5.4/openmpi2.1/netcdf-mpi/4.4.1.1/lib -lnetcdf </appe
nd>
<append MPILIB="openmpi"> -L/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/gcc5.4/openmpi2.1/netcdf-fortran-mpi/4.4.4/lib -lnetcdff
</append>
</SLIBS>
<FFLAGS>
<append MPILIB="openmpi"> -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/MPI/gcc5.4/openmpi2.1/netcdf-mpi/4.4.1.1/include </append>
</FFLAGS>
</compiler>
</config_compilers>

and loaded the remaining modules:

module load python/3.7.4
module load cmake/3.16.3
module load perl/5.22.4

but it failed again in build step:

ERROR: /project/6001010/my_cesm_sandbox/cime/src/build_scripts/buildlib.mct FAILED, cat /home/meisam/projects/def-hbeltram/cesm2_1_3_OUT/HISTCROP_f09g17/bld/mct.bldlog.231130-114108


Do you think I am using wrong format of writing the module paths in the config files? Because this is the only thing I can think of that could be the issue.

 

[jedwards]

When you did module show you should have found environment variables associated with the paths. You should use the
env variables in config_machines.xml and config_compilers.xml instead of the hardcoded paths - this will be much more resilient to future changes.

You need to look in the mct config file to figure out exactly what the error is. The path to that file should be something like:
/home/meisam/projects/def-hbeltram/cesm2_1_3_OUT/HISTCROP_f09g17/bld/intel/mpich/nodebug/nothreads/mct/config.log

 

[wvsi3w]

When you did module show you should have found environment variables associated with the paths. You should use the
env variables in config_machines.xml and config_compilers.xml instead of the hardcoded paths - this will be much more resilient to future changes.

You need to look in the mct config file to figure out exactly what the error is. The path to that file should be something like:
/home/meisam/projects/def-hbeltram/cesm2_1_3_OUT/HISTCROP_f09g17/bld/intel/mpich/nodebug/nothreads/mct/config.log

Dear @jedwards ,
Thanks for your response.

Sorry for my lack of knowledge in this field as it is totally a new thing for me.

I did look into the log file and it was showing the same error as before:
gmake -f /project/6001010/cesm2_1_3_OUT/HISTCROP_f09g17/Tools/Makefile -C /home/meisam/projects/def-hbeltram/cesm2_1_3_OUT/HISTCROP_f09g17/bld/intel/intelmpi
/nodebug/nothreads/mct CASEROOT=/project/6001010/cesm2_1_3_OUT/HISTCROP_f09g17 MODEL=mct /home/meisam/projects/def-hbeltram/cesm2_1_3_OUT/HISTCROP_f09g17/bld
/intel/intelmpi/nodebug/nothreads/mct/Makefile.conf
gmake: Entering directory `/project/6001010/cesm2_1_3_OUT/HISTCROP_f09g17/bld/intel/intelmpi/nodebug/nothreads/mct'
gmake: Leaving directory `/project/6001010/cesm2_1_3_OUT/HISTCROP_f09g17/bld/intel/intelmpi/nodebug/nothreads/mct'
cat: Filepath: No such file or directory
/project/6001010/cesm2_1_3_OUT/HISTCROP_f09g17/Tools/Makefile:199: *** NETCDF not found: Define NETCDF_PATH or NETCDF_C_PATH and NETCDF_FORTRAN_PATH in config
_machines.xml or config_compilers.xml. Stop.
ERROR: cat: Filepath: No such file or directory
/project/6001010/cesm2_1_3_OUT/HISTCROP_f09g17/Tools/Makefile:199: *** NETCDF not found: Define NETCDF_PATH or NETCDF_C_PATH and NETCDF_FORTRAN_PATH in config
_machines.xml or config_compilers.xml. Stop.

About the path you mentioned (sorry I think I didn't understand your previous suggestion due to my English): should I just copy the paths that I found and put them in the { }; I mean below is what you suggested:
<NETCDF_C_PATH>$ENV{LOCAL_NETCDF_C_PATH_VAR}</NETCDF_C_PATH>
<NETCDF_FORTRAN_PATH>$ENV{LOCAL_NETCDF_FORTRAN_PATH_VAR}</NETCDF_FORTRAN_PATH>
So, I should put the path inside {} and it would be something like this (in my config compilers and config machines):
<NETCDF_C_PATH>$ENV{cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx2/MPI/gcc5.4/openmpi2.1/netcdf-mpi/4.4.1.1}</NETCDF_C_PATH>
<NETCDF_FORTRAN_PATH>$ENV{/cvmfs/soft.computecanada.ca/easybuild/modules/2017/avx2/MPI/gcc5.4/openmpi2.1/netcdf-fortran-mpi/4.4.4}</NETCDF_FORTRAN_PATH>

If so, then I should not include the other three paths in my config machines (?) :
NETCDF_PATH
INC_NETCDF
LIB_NETCDF
And for my config compilers, instead of "SLIBS" and "FFLAGS" I should just add those two lines of "<NETCDF_FORTRAN_PATH>" and "<NETCDF_C_PATH>" ?

I think that I shouldn't put any path address in the { } of the suggested lines, but again I am sorry I should ask to know the answer.

I check the config files of other two clusters that I worked with before (narval and beluga, which I attached here in this message) and I realized the only place I have used your suggestion of paths to the modules, is in config_compilers of narval:
<compiler COMPILER="intel" MACH="narval">
<NETCDF_PATH>$(EBROOTNETCDF)</NETCDF_PATH>
<NETCDF_C_PATH>$(EBROOTNETCDF)</NETCDF_C_PATH>
<NETCDF_FORTRAN_PATH>$(EBROOTNETCDFMINFORTRAN)</NETCDF_FORTRAN_PATH>
<PNETCDF_PATH>$(EBROOTPNETCDF)</PNETCDF_PATH>
<PIO_FILESYSTEM_HINTS>lustre</PIO_FILESYSTEM_HINTS>
<CFLAGS>
<append> -march=core=avx2 </append>
</CFLAGS>
<FFLAGS>
<append> -march=core-avx2 </append>
</FFLAGS>
<SLIBS>
<append> $(shell $(NETCDF_PATH)/bin/nc-config --flibs) -mkl=sequential -limf </append>
</SLIBS>
</compiler>

I noticed that in other config files, I always used the other hardcoded paths (which worked and did some simulations though).

 

[jedwards]

When I do module show netcdf-mpi/4.9.2 on my system I see:

whatis("Name : netcdf")
whatis("Version : 4.9.2")
whatis("Target : x86_64_v3")
whatis("Short description : NetCDF (network Common Data Form) is a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. This is a 'meta' package that includes the C, C++, and Fortran distributions.")
help([[NetCDF (network Common Data Form) is a set of software libraries and
machine-independent data formats that support the creation, access, and
sharing of array-oriented scientific data. This is a "meta" package that
includes the C, C++, and Fortran distributions.]])
depends_on("hdf5-mpi/1.12.2")
prepend_path("PATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/bin")
prepend_path("NCAR_LDFLAGS_NETCDF","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/lib")
prepend_path("NCAR_INC_NETCDF","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/include")
prepend_path("MANPATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/share/man")
prepend_path("PKG_CONFIG_PATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/lib/pkgconfig")
prepend_path("PATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/bin")
prepend_path("MANPATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/share/man")
prepend_path("PKG_CONFIG_PATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/lib/pkgconfig")
prepend_path("CMAKE_PREFIX_PATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/.")
setenv("NETCDF","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx")
setenv("NCAR_ROOT_NETCDF","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx")
setenv("NCAR_SPACK_HASH_NETCDF","wplxi7yxrcxide4acdjkjrzeid7yvfn6")
conflict("netcdf")

So the path to netcdf on this system is set in an environment variable NCAR_ROOT_NETCDF and for cesm I just need to map that env variable to
NETCDF_PATH so in config_compilers.xml I have

<NETCDF_PATH>$ENV{NCAR_ROOT_NETCDF}</NETCDF_PATH>

Your Narval setup looks correct. I pointed you to the mct/config.log but the contents of that log are not here anywhere so I think that you haven't found it yet.

 

[wvsi3w]

When I do module show netcdf-mpi/4.9.2 on my system I see:

whatis("Name : netcdf")
whatis("Version : 4.9.2")
whatis("Target : x86_64_v3")
whatis("Short description : NetCDF (network Common Data Form) is a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. This is a 'meta' package that includes the C, C++, and Fortran distributions.")
help([[NetCDF (network Common Data Form) is a set of software libraries and
machine-independent data formats that support the creation, access, and
sharing of array-oriented scientific data. This is a "meta" package that
includes the C, C++, and Fortran distributions.]])
depends_on("hdf5-mpi/1.12.2")
prepend_path("PATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/bin")
prepend_path("NCAR_LDFLAGS_NETCDF","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/lib")
prepend_path("NCAR_INC_NETCDF","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/include")
prepend_path("MANPATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/share/man")
prepend_path("PKG_CONFIG_PATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/lib/pkgconfig")
prepend_path("PATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/bin")
prepend_path("MANPATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/share/man")
prepend_path("PKG_CONFIG_PATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/lib/pkgconfig")
prepend_path("CMAKE_PREFIX_PATH","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx/.")
setenv("NETCDF","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx")
setenv("NCAR_ROOT_NETCDF","/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/wplx")
setenv("NCAR_SPACK_HASH_NETCDF","wplxi7yxrcxide4acdjkjrzeid7yvfn6")
conflict("netcdf")

So the path to netcdf on this system is set in an environment variable NCAR_ROOT_NETCDF and for cesm I just need to map that env variable to
NETCDF_PATH so in config_compilers.xml I have

<NETCDF_PATH>$ENV{NCAR_ROOT_NETCDF}</NETCDF_PATH>

Your Narval setup looks correct. I pointed you to the mct/config.log but the contents of that log are not here anywhere so I think that you haven't found it yet.

Thanks for the hints.
I have loaded the newest version of StdEnv/2023 so that I could load those two netcdf modules. So, these are the Currently Loaded Modules on my cedar system:
1) CCconfig 4) gcc/12.3 (t) 7) libfabric/1.18.0 10) openmpi/4.1.5 (m) 13) StdEnv/2023 (S) 16) netcdf-mpi/4.9.2 (io)
2) gentoo/2023 (S) 5) hwloc/2.9.1 8) pmix/4.2.4 11) flexiblas/3.3.1 14) mii/1.1.2 17) netcdf-fortran-mpi/4.6.1 (io)
3) gcccore/.12.3 (H) 6) ucx/1.14.1 9) ucc/1.2.0 12) imkl/2023.2.0 (math) 15) hdf5-mpi/1.14.2 (io)

then when I used show command it worked and I have put what it showed in the two attached files. But basically the two paths are like:
/cvmfs/soft.computecanada.ca/easybuild/modules/2023/x86-64-v3/MPI/gcc12/openmpi4/netcdf-mpi/4.9.2.lua
/cvmfs/soft.computecanada.ca/easybuild/modules/2023/x86-64-v3/MPI/gcc12/openmpi4/netcdf-fortran-mpi/4.6.1.lua:


And for the log file, yes I couldn't find it, I attached the screenshot of the path you gave me and there is no log file nor this path "/home/meisam/projects/def-hbeltram/cesm2_1_3_OUT/HISTCROP_f09g17/bld/intel/mpich/nodebug/nothreads/mct/"