Error building CESM1_2_2 on a local machine: buildlib.mct failed

HeraGud · May 31, 2022

Hello,

We are currently in the process of building CESM1_2_2 on a local HPC facility in Iceland but we are stranded in the ./test.build step (we are following chapter 5 in the CESM manual). It appears to be a linking problem but we would much appreciate your expertise on this. Attached are relevant files along with the mtc.bldlog file if needed.

many thanks,
Hera

jedwards · May 31, 2022

Hi Hera,

CESM 1.2.2 is no longer officially supported. I highly recommend using cesm2.1 if possible. From the error here:

/users/home/hera/cesm1_2_2/models/utils/mct/mpeu/m_String.F90:478:17:
457 | call MPI_bcast(ln,1,MP_INTEGER,root,comm,ier)
| 2
......
478 | call MPI_bcast(Str%c(1),ln,MP_CHARACTER,root,comm,ier)
| 1
Error: Type mismatch between actual argument at (1) and actual
argument at (2) (CHARACTER(1)/INTEGER(4)).
gmake[1]: *** [Makefile:71: m_String.o] Error 1
gmake[1]: Leaving directory
'/users/home/hera/cesm/test1_elja/bld/gcc/mpicc/nodebug/nothreads/mct/
mpeu'
gmake: *** [Makefile:10: subdirs] Error 2

I think that you need to add the compiler flags: " -fallow-argument-mismatch -fallow-invalid-boz"

HeraGud · Jun 1, 2022

Thanks !
For sure, next time I will use CESM2 - but for now I will have to work with CESM1.2.2. Now emerges another problem (the first one being solved), if you are willing to assist us further it would be greatly appreciated.
The problem is as follows:

$ ./test1.build
-------------------------------------------------------------------------
CESM BUILDNML SCRIPT STARTING
- To prestage restarts, untar a restart.tar file into /users/home/eojons/cesm/test1/run
infile is /users/home/eojons/cesm_v1_2_2/scripts/test1/Buildconf/cplconf/cesm_namelist
CESM BUILDNML SCRIPT HAS FINISHED SUCCESSFULLY
-------------------------------------------------------------------------
-------------------------------------------------------------------------
CESM PRESTAGE SCRIPT STARTING
- Case input data directory, DIN_LOC_ROOT, is /users/home/share/ccsm3data/inputdata
- Checking the existence of input datasets in DIN_LOC_ROOT
CESM PRESTAGE SCRIPT HAS FINISHED SUCCESSFULLY
-------------------------------------------------------------------------
-------------------------------------------------------------------------
CESM BUILDEXE SCRIPT STARTING
rm: No match.
COMPILER is gcc
- Build Libraries: mct gptl pio csm_share
þri maí 31 14:32:03 GMT 2022 /users/home/eojons/cesm/test1/bld/gcc/mpicc/nodebug/nothreads/mct.bldlog.220531-143155
þri maí 31 14:32:03 GMT 2022 /users/home/eojons/cesm/test1/bld/gcc/mpicc/nodebug/nothreads/gptl.bldlog.220531-143155
þri maí 31 14:32:03 GMT 2022 /users/home/eojons/cesm/test1/bld/gcc/mpicc/nodebug/nothreads/pio.bldlog.220531-143155
þri maí 31 14:32:04 GMT 2022 /users/home/eojons/cesm/test1/bld/gcc/mpicc/nodebug/nothreads/csm_share.bldlog.220531-143155
þri maí 31 14:32:04 GMT 2022 /users/home/eojons/cesm/test1/bld/atm.bldlog.220531-143155
þri maí 31 14:32:04 GMT 2022 /users/home/eojons/cesm/test1/bld/lnd.bldlog.220531-143155
þri maí 31 14:32:05 GMT 2022 /users/home/eojons/cesm/test1/bld/ice.bldlog.220531-143155
þri maí 31 14:32:05 GMT 2022 /users/home/eojons/cesm/test1/bld/ocn.bldlog.220531-143155
þri maí 31 14:32:06 GMT 2022 /users/home/eojons/cesm/test1/bld/glc.bldlog.220531-143155
þri maí 31 14:32:06 GMT 2022 /users/home/eojons/cesm/test1/bld/wav.bldlog.220531-143155
þri maí 31 14:32:06 GMT 2022 /users/home/eojons/cesm/test1/bld/rof.bldlog.220531-143155
þri maí 31 14:32:07 GMT 2022 /users/home/eojons/cesm/test1/bld/cesm.bldlog.220531-143155
ERROR: cesm.buildexe.csh failed, see /users/home/eojons/cesm/test1/bld/cesm.bldlog.220531-143155
ERROR: cat /users/home/eojons/cesm/test1/bld/cesm.bldlog.220531-143155

greetings,
Hera

jedwards · Jun 1, 2022

ERROR: cesm.buildexe.csh failed, see /users/home/eojons/cesm/test1/bld/cesm.bldlog.220531-143155
ERROR: cat /users/home/eojons/cesm/test1/bld/cesm.bldlog.220531-143155

The information you (or I) need to debug this problem is in the cesm.bldlog

HeraGud · Jun 2, 2022

Thanks - see attachment.

jedwards · Jun 2, 2022

You are not linking the netcdf c library which is needed by the netcdff library. In config_compilers where
you have
$(shell $(NETCDF_PATH)/bin/nf-config --flibs)

Try changing it to
$(shell $(NETCDF_PATH)/bin/nf-config --flibs) -lnetcdf

HeraGud · Jun 3, 2022

I have tried as you suggested - and at least the buildlog is not as long as before - but still the same error. The NetCDF library linking seems to be off but I can't see why.
When you refer to the NetCDF C lib should it then be:

$(shell $(NETCDF_C_PATH)/bin/nf-config --flibs) -lnetcdf ?

In the first bldlog file I sent, the first "user defined" part had not been filled out. That I also did. Here below I have pasted the two user defined parts in the config_compiler file - do you see an obvious error?

#1

<compiler MACH="elja">

<NETCDF_PATH>/users/home/eojons/easybuild/software/netCDF/4.8.1-gompi-2021b</NETCDF_PATH>

<PNETCDF_PATH>/users/home/eojons/WACCM/PnetCDF</PNETCDF_PATH>

<ADD_SLIBS>$(shell $(NETCDF_PATH)/bin/nf-config --flibs) -lnetcdf</ADD_SLIBS>

<ADD_CPPDEFS>-DFORTRANUNDERSCORE</ADD_CPPDEFS>

<CONFIG_ARGS></CONFIG_ARGS>

<ESMF_LIBDIR></ESMF_LIBDIR>

<MPI_LIB_NAME>mpich</MPI_LIB_NAME>

<MPI_PATH>/users/home/eojons/easybuild/software/OpenMPI/4.1.1-GCC-11.2.0</MPI_PATH>

</compiler>

_______
#2

<compiler MACH="elja">

<MPIFC>mpif90</MPIFC>

<SCC>gcc</SCC>

<MPICC>mpicc</MPICC>

<ADD_CPPDEFS>-DFORTRANUNDERSCORE</ADD_CPPDEFS>

<ADD_CFLAGS>-I/users/home/eojons/easybuild/software/OpenMPI/4.1.1-GCC-11.2.0/include</ADD_CFLAGS>

<ADD_FFLAGS>-fallow-argument-mismatch -fallow-invalid-boz -fno-underscoring</ADD_FFLAGS>

<NETCDF_PATH>/users/home/eojons/easybuild/software/netCDF/4.8.1-gompi-2021b</NETCDF_PATH>

<NETCDF_C_PATH>/users/home/eojons/easybuild/software/netCDF-C+4/4.3.1-gompi-2021b</NETCDF_C_PATH>

<NETCDF_Fortran_PATH>/users/home/eojons/easybuild/software/netCDF-Fortran/4.5.3-gompi-2021b</NETCDF_Fortran_PATH>

<NETCDF_Fortran_LIBRARY>/users/home/eojons/easybuild/software/netCDF-Fortran/lib/libnetcdff.so</NETCDF_Fortran_LIBRARY>

<MPI_PATH>/users/home/eojons/easybuild/software/OpenMPI/4.1.1-GCC-11.2.0</MPI_PATH>

<MPI_LIB_NAME>mpicc</MPI_LIB_NAME>

<ADD_SLIBS>$(shell /users/home/eojons/easybuild/software/netCDF-Fortran/4.5.3-gompi-2021b/bin/nf-config --flibs) -lnetcdff</ADD_SLIBS>

<ADD_LDFLAGS>-Wl,-rpath /users/home/eojons/easybuild/software/netCDF/4.8.1-gompi-2021b/lib</ADD_LDFLAGS>

<ADD_LDFLAGS>-Wl,-rpath /users/home/eojons/easybuild/software/OpenMPI/4.1.1-GCC-11.2.0/lib</ADD_LDFLAGS>

<ADD_LDFLAGS>-Wl,-rpath /users/home/eojons/easybuild/software/netCDF-Fortran/4.5.3-gompi-2021b/lib</ADD_LDFLAGS>

<CONFIG_SHELL>/usr/bin/bash</CONFIG_SHELL>

</compiler>

HeraGud · Jun 7, 2022

Still having trouble identifying the NetCDF lib problem. I added flags to the Macros file that now looks like this:

# Makefile Macros generated from /users/home/hera/cesm1_2_2/scripts/ccsm_utils/Machines/config_compilers.xml using

# COMPILER=gcc

# OS=LINUX

# MACH=elja

#

CFLAGS+= -I/users/home/eojons/easybuild/software/OpenMPI/4.1.1-GCC-11.2.0/include

CPPDEFS+= -DFORTRANUNDERSCORE -DLINUX -DCPRGCC

FFLAGS+= -fallow-argument-mismatch -fallow-invalid-boz -fno-underscoring

LDFLAGS+= -L/users/home/eojons/easybuild/software/netCDF/4.8.1-gompi-2021b/lib -L/users/home/eojons/easybuild/software/OpenMPI/4.1.1-GCC-11.2.0/lib -L/users/home/eojons/easybuild/software/netCDF-Fortran/4.5.3-gompi-2021b/lib -lnetcdff -L/users/home/eojons/easybuild/software/netCDF-C++4/4.3.1-gompi-2021b/lib -lnetcdf

SLIBS+=$(shell /users/home/eojons/easybuild/software/netCDF/4.8.1-gompi-2021b/bin/nf-config --flibs)

CONFIG_SHELL:= /usr/bin/bash

MPICC:= mpicc

MPIFC:= mpif90

MPI_LIB_NAME:=mpi

MPI_PATH:=/users/home/eojons/easybuild/software/OpenMPI/4.1.1-GCC-11.2.0

NETCDF_C_PATH:=/users/home/eojons/easybuild/software/netCDF-C++4/4.3.1-gompi-2021b

NETCDF_Fortran_LIBRARY:=/users/home/eojons/easybuild/software/netCDF-Fortran/lib/libnetcdff.so

NETCDF_Fortran_PATH:=/users/home/eojons/easybuild/software/netCDF-Fortran/4.5.3-gompi-2021b

NETCDF_PATH:=/users/home/eojons/easybuild/software/netCDF/4.8.1-gompi-2021b

And here is the content of env_mach_specific:

source /etc/profile.d/lmod.csh

module purge

module use /users/home/eojons/easybuild/modules/all

module load GCCcore

module load netCDF

module load netCDF-Fortran

module load netCDF-C++4

module load PnetCDF-fs

module load HDF5

module load Trilinos-fs

module load Perl

module load CMake

module load OpenMPI/4.1.1-GCC-11.2.0

setenv NETCDF_PATH /users/home/eojons/easybuild/software/netCDF/4.8.1-gompi-2021b

setenv MPI_PATH /users/home/eojons/easybuild/software/OpenMPI/4.1.2-GCC-11.2.0

setenv NETCDF_Fortran_PATH /users/home/eojons/easybuild/software/netCDF-Fortran/4.5.3-gompi-2021b

setenv NETCDF_Fortran_LIBRARY /users/home/eojons/easybuild/software/netCDF-Fortran/4.5.3-gompi-2021b/lib/libnetcdff.so

setenv NETCDF_Fortran_INCLUDE_DIR /users/home/eojons/easybuild/software/netCDF-Fortran/4.5.3-gompi-2021b/include

setenv NETCDF_C_PATH /users/home/eojons/easybuild/software/netCDF-C++4/4.3.1-gompi-2021b

Attached is the latest version of the config_compiler.xml. It seems like a flag is missing or there is a mismatch between files. Your help is much appreciated.

jedwards · Jun 7, 2022

I don't see the -lnetcdf in the link line here:

mpif90 -o /users/home/hera/cesm/test1_elja/bld/cesm.exe
ccsm_comp_mod.o ccsm_driver.o mrg_mod.o seq_avdata_mod.o
seq_diag_mct.o seq_domain_mct.o seq_flux_mct.o seq_frac_mct.o
seq_hist_mod.o seq_map_esmf.o seq_map_mod.o seq_mctext_mod.o
seq_rest_mod.o -L/users/home/hera/cesm/test1_elja/bld/lib/ -latm
-L/users/home/hera/cesm/test1_elja/bld/lib/ -lice -L/users/home/
hera/cesm/test1_elja/bld/lib/ -llnd -L/users/home/hera/cesm/
test1_elja/bld/lib/ -locn -L/users/home/hera/cesm/test1_elja/bld/
lib/ -lrof -L/users/home/hera/cesm/test1_elja/bld/lib/ -lglc -L/
users/home/hera/cesm/test1_elja/bld/lib/ -lwav -L/users/home/hera/
cesm/test1_elja/bld/gcc/mpicc/nodebug/nothreads/MCT/noesmf/
a1l1r1i1o1g1w1/csm_share -lcsm_share -L/users/home/hera/cesm/
test1_elja/bld/gcc/mpicc/nodebug/nothreads/lib -lpio -lgptl -lmct
-lmpeu -L/users/home/eojons/easybuild/software/netCDF-Fortran/4.5.3-
gompi-2021b/lib -lnetcdff -L/users/home/eojons/easybuild/software/
OpenMPI/4.1.1-GCC-11.2.0/lib -lmpi -Wl,-rpath /users/home/eojons/
easybuild/software/netCDF/4.8.1-gompi-2021b/lib -Wl,-rpath /users/
home/eojons/easybuild/software/OpenMPI/4.1.1-GCC-11.2.0/lib -Wl,-
rpath /users/home/eojons/easybuild/software/netCDF-Fortran/4.5.3-
gompi-2021b/lib

But I'm not sure that the first error listed can be explained by that.

Try a simple netcdf fortran test:

program nf_test
include 'netcdf.inc'
integer ierr, NCID
call nf_create("test.nc", NCID, ierr)
end program

gfortran nf_test.F -I/users/home/eojons/easybuild/software/netCDF-Fortran/4.5.3-
gompi-2021b/include -L/users/home/eojons/easybuild/software/netCDF-Fortran/4.5.3-
gompi-2021b/lib -lnetcdff -lnetcdf

HeraGud · Jun 7, 2022

Yes sorry, I did not include the latest cesmbldlog file, now it is:
mpif90 -o /users/home/hera/cesm/test1_elja/bld/cesm.exe ccsm_comp_mod.o ccsm_driver.o mrg_mod.o seq_avdata_mod.o seq_diag_mct.o seq_domain_mct.o seq_flux_mct.o seq_frac_mct.o seq_hist_mod.o seq_map_esmf.o seq_map_mod.o seq_mctext_mod.o seq_rest_mod.o -L/users/home/hera/cesm/test1_elja/bld/lib/ -latm -L/users/home/hera/cesm/test1_elja/bld/lib/ -lice -L/users/home/hera/cesm/test1_elja/bld/lib/ -llnd -L/users/home/hera/cesm/test1_elja/bld/lib/ -locn -L/users/home/hera/cesm/test1_elja/bld/lib/ -lrof -L/users/home/hera/cesm/test1_elja/bld/lib/ -lglc -L/users/home/hera/cesm/test1_elja/bld/lib/ -lwav -L/users/home/hera/cesm/test1_elja/bld/gcc/mpicc/nodebug/nothreads/MCT/noesmf/a1l1r1i1o1g1w1/csm_share -lcsm_share -L/users/home/hera/cesm/test1_elja/bld/gcc/mpicc/nodebug/nothreads/lib -lpio -lgptl -lmct -lmpeu -L/users/home/eojons/easybuild/software/OpenMPI/4.1.1-GCC-11.2.0/lib -lmpi -L/users/home/eojons/easybuild/software/netCDF/4.8.1-gompi-2021b/lib -L/users/home/eojons/easybuild/software/OpenMPI/4.1.1-GCC-11.2.0/lib -L/users/home/eojons/easybuild/software/netCDF-Fortran/4.5.3-gompi-2021b/lib -lnetcdff -L/users/home/eojons/easybuild/software/netCDF-C++4/4.3.1-gompi-2021b/lib -lnetcdf

When I do the test you suggest I get a warning that the include file can't be open/missing include dir. In the Macros file, the path to the include file is only defined for openMPI. Should it also be defined for NetCDF/netcdf fortran ?

HeraGud · Jun 7, 2022

By changing

SLIBS+=$(shell /users/home/eojons/easybuild/software/netCDF/4.8.1-gompi-2021b/bin/nf-config --flibs)

to

SLIBS+=$(shell /users/home/eojons/easybuild/software/netCDF/4.8.1-gompi-2021b/bin/nf-config --libs)

in the Macros finally solved the problem. Now I get:

cp: cannot stat '/users/home/hera/cesm/test1_elja/bld/cesm.exe': No such file or directory

- Locking file env_build.xml

CESM BUILDEXE SCRIPT HAS FINISHED SUCCESSFULLY

That should do it I hope. Thank you for your assistance!

AndrewLowry · Mar 8, 2023

Hera, sorry for posting on a slightly old thread. Would you mind replying with the final version of your env_mach_specific.elja, mkbatch.elja, config_compilers.xml, and config_machines.xml? It might help me and others who come across this thread in the future.
I would also be curious to see your EasyBuild script (.eb file) to build the dependencies if possible. I am curious as to how you got CESM to find both netCDF-c (not c++) and netCDF-fortran. Did you have them in the same directory? Lastly from the final above post on 7 June 2022 how did you access nf-config from the netCDF (I assume c) directory? In my case nf-config exists in my path_to/netCDF-fortran/bin directory and nc-config is in path_to/netCDF-c/bin directory. Hope you can help.

Error building CESM1_2_2 on a local machine: buildlib.mct failed

HeraGud

Hera Guðlaugsdóttir

New Member

Attachments

jedwards

CSEG and Liaisons

HeraGud

Hera Guðlaugsdóttir

New Member

jedwards

CSEG and Liaisons

HeraGud

Hera Guðlaugsdóttir

New Member

Attachments

jedwards

CSEG and Liaisons

HeraGud

Hera Guðlaugsdóttir

New Member

Attachments

HeraGud

Hera Guðlaugsdóttir

New Member

Attachments

jedwards

CSEG and Liaisons

HeraGud

Hera Guðlaugsdóttir

New Member

HeraGud

Hera Guðlaugsdóttir

New Member

AndrewLowry

Andrew Lowry

New Member