case.build fail for CLM (fail at clm.buildlib)

[hayesdev]

I am trying an out-of-box CLM run from the 2019 tutorial https://www.cgd.ucar.edu/events/2019/ctsm/files/practical1-lombardozzi.pdf
These are the exact steps I took
1. Downloaded CLM5.0 from git as instructed

git clone -b release-clm5.0 https://github.com/ESCOMP/ctsm.git clm5.0_2019tutorial
cd clm5.0_2019tutorial/
./manage_externals/checkout_externals

2. Edited my .bashrc in my home directory

nano ~/.bashrc

export CIME=$HOME/clm5.0_2019tutorial/cime
export CESMDATAROOT=/data/groups/feldl/hadevane/CESM_work

^X
Source .bashrc

3. Attempted CLM tutorial as instructed

cd clm5.0_2019tutorial/cime/scripts/
./create_newcase --case ~/cases/I1850CLM50_001 --res f19_g17 --compset I1850Clm50Sp --mach lux-cpuq
cd ~/cases/I1850CLM50_001/

4. Checked that fortran path correct in Makefile (instead of NETCDF_C_PATH)

LIB_NETCDF_FORTRAN:=$(NETCDF_FORTRAN_PATH)/lib

5. Proceeded with ./case.setup and ./case.build in I1850CLM50_001 case folder and received build error
"

Building case in directory /home/hadevane/cases/I1850CLM50_001
sharedlib_only is False
model_only is False

Generating component namelists as part of build
Creating component namelists

Calling /home/hadevane/clm5.0_2019tutorial/cime/src/components/data_comps/datm/cime_config/buildnml

File not found: domainfile = "/data/groups/feldl/hadevane/CESM_work/inputdata/share/domains/domain.lnd.fv1.9x2.5_gx1v7.181205.nc", will attempt to download in check_input_data phase

Calling /home/hadevane/clm5.0_2019tutorial/cime_config/buildnml
Calling /home/hadevane/clm5.0_2019tutorial/cime/src/components/stub_comps/sice/cime_config/buildnml
Calling /home/hadevane/clm5.0_2019tutorial/cime/src/components/stub_comps/socn/cime_config/buildnml
Calling /home/hadevane/clm5.0_2019tutorial/components/mosart//cime_config/buildnml

File not found: frivinp_rtm = "/data/groups/feldl/hadevane/CESM_work/inputdata/rof/mosart/MOSART_routing_Global_0.5x0.5_c170601.nc", will attempt to download in check_input_data phase

Running /home/hadevane/clm5.0_2019tutorial/components/cism//cime_config/buildnml
Calling /home/hadevane/clm5.0_2019tutorial/cime/src/components/stub_comps/swav/cime_config/buildnml
Calling /home/hadevane/clm5.0_2019tutorial/cime/src/components/stub_comps/sesp/cime_config/buildnml
Calling /home/hadevane/clm5.0_2019tutorial/cime/src/drivers/mct/cime_config/buildnml

File not found: lnd2rof_fmapname = "lnd/clm2/mappingdata/maps/1.9x2.5/map_1.9x2.5_nomask_to_0.5x0.5_nomask_aave_da_c120522.nc", will attempt to download in check_input_data phase
File not found: rof2lnd_fmapname = "lnd/clm2/mappingdata/maps/1.9x2.5/map_0.5x0.5_nomask_to_1.9x2.5_nomask_aave_da_c120709.nc", will attempt to download in check_input_data phase
File not found: lnd2glc_fmapname = "cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gland4km_aave.170429.nc", will attempt to download in check_input_data phase
File not found: lnd2glc_smapname = "cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gland4km_blin.170429.nc", will attempt to download in check_input_data phase
File not found: glc2lnd_fmapname = "cpl/gridmaps/gland4km/map_gland4km_TO_fv1.9x2.5_aave.170429.nc", will attempt to download in check_input_data phase
File not found: glc2lnd_smapname = "cpl/gridmaps/gland4km/map_gland4km_TO_fv1.9x2.5_aave.170429.nc", will attempt to download in check_input_data phase
Finished creating component namelists

Building gptl with output to file /data/users/hadevane/CESM_out/I1850CLM50_001/bld/gptl.bldlog.210916-133525
Calling /home/hadevane/clm5.0_2019tutorial/cime/src/build_scripts/buildlib.gptl

Building mct with output to file /data/users/hadevane/CESM_out/I1850CLM50_001/bld/mct.bldlog.210916-133525
Calling /home/hadevane/clm5.0_2019tutorial/cime/src/build_scripts/buildlib.mct
Component mct build complete with 55 warnings

Building pio with output to file /data/users/hadevane/CESM_out/I1850CLM50_001/bld/pio.bldlog.210916-133525
Calling /home/hadevane/clm5.0_2019tutorial/cime/src/build_scripts/buildlib.pio
Component pio build complete with 26 warnings

Building csm_share with output to file /data/users/hadevane/CESM_out/I1850CLM50_001/bld/csm_share.bldlog.210916-133525
Calling /home/hadevane/clm5.0_2019tutorial/cime/src/build_scripts/buildlib.csm_share
Component csm_share build complete with 140 warnings

- Building clm4_5/clm5_0 Library

Building lnd with output to /data/users/hadevane/CESM_out/I1850CLM50_001/bld/lnd.bldlog.210916-133525

Error: Deferred-length character component 'num_layers_name' at (1) is not yet supported
Error: Deferred-length character component 'num_layers_name' at (1) is not yet supported
Error: Deferred-length character component 'num_layers_name' at (1) is not yet supported
Error: Deferred-length character component 'num_layers_name' at (1) is not yet supported
Error: Deferred-length character component 'coord_varname' at (1) is not yet supported
Error: Deferred-length character component 'coord_varname' at (1) is not yet supported
Error: Deferred-length character component 'coord_varname' at (1) is not yet supported
Error: 'confirm_monotonically_increasing' at (1) should be a SUBROUTINE
Error: 'interp_onelevel' at (1) should be a SUBROUTINE
Error: Deferred-length character component 'coord_varname' at (1) is not yet supported
Error: 'check_coordinate_array' at (1) should be a SUBROUTINE
Error: 'check_coordinate_array' at (1) should be a SUBROUTINE
Error: Deferred-length character component 'coord_varname' at (1) is not yet supported

clm built in 3.946365 seconds
ERROR: BUILD FAIL: clm.buildlib failed, cat /data/users/hadevane/CESM_out/I1850CLM50_001/bld/lnd.bldlog.210916-133525"

Since this is occurring for out-of-box I'm inclined to believe this is a porting issue. I've attached the output logs, and my config_machines and config_batch. I don't have config_compilers in my $HOME/.cime. Thanks!

 

[jedwards]

It looks like you are using a gnu 4.5.2 compiler version which is too old to correctly compile some of the code in ctsm.
Please try with a newer compiler.

 

[hayesdev]

It looks like you are using a gnu 4.5.2 compiler version which is too old to correctly compile some of the code in ctsm.
Please try with a newer compiler.

Great thank you, I will try that

 

[ykp990521]

Great thank you, I will try that

have you solved that gptl problem? I encountered the same question, thanks a lot!

 

[hayesdev]

It looks like you are using a gnu 4.5.2 compiler version which is too old to correctly compile some of the code in ctsm.
Please try with a newer compiler.

I have not been able to solve the issue,
I am trying to compile CLM with the following:
* GNU Compiler Collection (GCC) 4.8.5
* OpenMPI 4.0.1
* NetCDF C 4.7.3
* NetCDF Fortran 4.5.2
This works successfully for the CAM6 aquaplanet, and the NetCDF versions are recent. What are the compiler requirements for CLM?
I've also tried using Intel compilers and OpenMPI 4.1.2.

 

[jedwards]

I don't understand - you tried which intel compiler version? openmpi version is new enough but you should use a modern gnu compiler version
gnu is now in version 11, I recommend 9 or newer.

 

[hayesdev]

I don't understand - you tried which intel compiler version? openmpi version is new enough but you should use a modern gnu compiler version
gnu is now in version 11, I recommend 9 or newer.

I first attempted using the compiler configuration in the bullet points, starting with gnu compiler. When that didn't work, I tried the intel compilers [icc (ICC) 19.0.5.281 20190815] c compiler and [ifort (IFORT) 19.0.5.281 20190815] fortran compiler. I also tried using OpenMPI 4.1.2.

 

[jedwards]

That version of the intel compiler is known to work correctly - what error are you encountering with that?

 

[hayesdev]

the same error. I either get a fail at build.gptl or it runs through with warnings and then fails at clm.buildlib. I think this may have something to do with the config_machines file or porting in general but I'm not exactly sure what

 

[jedwards]

Can you please post your build logs using the intel compiler.

 

[hayesdev]

With intel the build runs a bit farther than with gnu, but eventually fails at buildexe. The output from terminal is very long and attached as a separate text file. The main differences are different error warnings for each of the components, and then a long list of undefined references.

I altered the config_machines file, only lux-feldl, which is what I used for this run. Here is the config_machines, the log files, and terminal output as a text file

 

[jedwards]

/home/hadevane/my_cesm_sandbox/components/clm/src/biogeophys/SoilWaterMovementMod.F90:1278: undefined reference to `dgtsv_'

This error indicates that you are missing the lapack library, this library is a prerequisite for cesm.

 

[hayesdev]

I've added the lapack and blas libraries for intel onto my machine. My machine is configured as follows:

<machine MACH="lux-feldl">
    <DESC>lux cluster at UCSC, os is Linux, batch system is slurm</DESC>
    <NODENAME_REGEX>lux.*</NODENAME_REGEX>
    <OS>LINUX</OS>
    <COMPILERS>intel, gnu</COMPILERS>
    <MPILIBS>openmpi_intel, openmpi</MPILIBS>
    <CIME_OUTPUT_ROOT>/data/users/$USER/CESM_out</CIME_OUTPUT_ROOT>
    <DIN_LOC_ROOT>$ENV{CESMDATAROOT}/inputdata</DIN_LOC_ROOT>
    <DIN_LOC_ROOT_CLMFORC>$ENV{CESMDATAROOT}/lmwg</DIN_LOC_ROOT_CLMFORC>
    <DOUT_S_ROOT>$CIME_OUTPUT_ROOT/archive/$CASE</DOUT_S_ROOT>
    <BASELINE_ROOT>$ENV{CESMDATAROOT}/cesm_baselines</BASELINE_ROOT>
    <CCSM_CPRNC>$ENV{CESMDATAROOT}/tools/cprnc/cprnc</CCSM_CPRNC>
    <GMAKE_J>8</GMAKE_J>
    <BATCH_SYSTEM>slurm</BATCH_SYSTEM>
    <SUPPORTED_BY>cseg</SUPPORTED_BY>
    <MAX_TASKS_PER_NODE>40</MAX_TASKS_PER_NODE>
    <MAX_MPITASKS_PER_NODE>40</MAX_MPITASKS_PER_NODE>
    <PROJECT_REQUIRED>FALSE</PROJECT_REQUIRED>
    <mpirun mpilib="default">
      <executable>mpirun</executable>
      <arguments>
        <arg name="num_tasks"> -np {{ total_tasks }}</arg>
        <arg name="tasks_per_node"> --map-by ppr:$MAX_MPITASKS_PER_NODE:node </arg>
      </arguments>
    </mpirun>
    <mpirun mpilib="mpi-serial">
      <executable></executable>
    </mpirun>
    <module_system type="module">
      <init_path lang="sh">/etc/profile.d/modules.sh</init_path>
      <init_path lang="csh">/etc/profile.d/modules.csh</init_path>
      <init_path lang="perl">/cm/local/apps/environment-modules/4.2.1/init/perl.pm</init_path>
      <init_path lang="python">/cm/local/apps/environment-modules/4.2.1/init/python.py</init_path>
      <cmd_path lang="sh">module</cmd_path>
      <cmd_path lang="csh">module</cmd_path>
      <cmd_path lang="perl">/cm/local/apps/environment-modules/4.2.1/bin/modulecmd perl</cmd_path>
      <cmd_path lang="python">/cm/local/apps/environment-modules/4.2.1/bin/modulecmd python</cmd_path>
      <modules>
        <command name="purge"/>
        <command name="load">python/3.6.7</command>
        <command name="load">svn/1.9.12</command>
        <command name="load">slurm/18.08.4</command>
      </modules>
       <modules compiler="intel">
        <command name="load">intel/icc</command>
        <command name="load">intel/ifort</command>
        <command name="load">lapack/intel/64/3.10.0</command>
        <command name="load">blas/intel/64/3.10.0</command>
        <command name="load">netcdf-c_ompi_intel/4.7.3</command>
        <command name="load">netcdf-f_ompi_intel/4.5.2</command>
        <command name="load">hdf5_ompi_intel/1.10.6-parallel</command>
      </modules>
      <modules compiler="gnu">
        <command name="load">lapack/gcc/64/3.8.0</command>
        <command name="load">blas/gcc/64/3.8.0</command>
        <command name="load">netcdf-c_gnu/4.7.3</command>
        <command name="load">netcdf-f_gnu/4.5.2</command>
      </modules>
      <modules compiler="gnu" mpilib="!mpi-serial">
        <command name="load">openmpi</command>
      </modules>
      <modules compiler="intel" mpilib="!mpi-serial">
        <command name="load">openmpi_intel/4.1.2</command>
      </modules>
    </module_system>
    <environment_variables>
      <env name="NETCDF_C_PATH">/cm/shared/apps/netcdf-c_ompi_intel/4.7.3</env>
      <env name="NETCDF_FORTRAN_PATH">/cm/shared/apps/netcdf-f_ompi_intel/4.5.2</env>
      <env name="LAPACK_LIBDIR">/cm/shared/apps/lapack/intel/64/3.10.0</env>
    </environment_variables>
  </machine>

When I try to build under this configuration, I get the following output


In the log, it seems like it has something to do with gmake, which (I believe?) is specific to gnu compiler. I'm trying to use intel because other people's programs on the supercomputer I work with use the older gnu compiler. Trying to find a work-around with intel that doesn't mess with other people's work. Is there an intel equivalent for gmake? Additionally, are there other places to specify using intel besides just the config_machines?

 

[jedwards]

gmake is not specific to the gnu compiler. You are still missing the dnrm2 function used in the cism model.
Here it is in the intel documentation If you are using the intel compiler you should use intels mkl library and not lapack and blas.

 

[hayesdev]

we have lapack and blas built for intel specifically so unclear why it wouldn't be working. Based on the log it seems like the build is using mpif90 which is for gnu not intel, any ideas on how to fix this to use the intel mpi? Thanks

 

[jedwards]

I'm confident that you are using the intel compiler for fortran - try mpif90 --version if you aren't sure.

Make sure that lapack is providing dnrm2.

 

[hayesdev]

Okay thank you, you are correct that mpif90 is intel. How do I check that lapack has dnrm2? Still very new to all of this, I appreciate your prompt responses