Porting to a Linux cluster

[kshedstrom@alaska_edu]

I have downloaded an alpha version of CESM on cheyenne and got it running with some help. I'm now trying to get the same code to compile on a system here at UAF. Due to the way they have installed netcdf, I am using the system C library and my own Fortran library:

ifeq ($(COMPILER),gnu)
  NETCDF_C_PATH := /usr/local/pkg/data/netCDF/4.4.1.1-foss-2016b
  NETCDF_FORTRAN_PATH := /import/home/kshedstrom
endif

The build is missing the -Ipath_to_netcdf.mod:

-ffixed-line-length-none -fcray-pointer   -O  -fdefault-real-8  -fcray-pointer -fdefault-double-8 -w -I  -DLINUX

which gives errors:

f951: Warning: Nonexistent include directory ‘-DLINUX’ [-Wmissing-include-dirs]

/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/diag_manager/diag_data.F90:60:6:

   USE netcdf, ONLY: NF_FILL_REAL => NF90_FILL_REAL
      1
Fatal Error: Can't open module file ‘netcdf.mod’ for reading at (1): No such file or directory
compilation terminated.

Can anyone tell me how to fix this?

 

[jedwards]

Add an additional variable: NetCDF_Fortran_INCLUDE_DIR, this should fix the problem.

 

[kshedstrom@alaska_edu]

Thanks, that helped. I also had to do this in libraries/FMS:

diff --git a/Makefile.cesm b/Makefile.cesm
index 649523a..1b13494 100644
--- a/Makefile.cesm
+++ b/Makefile.cesm
@@ -172,7 +172,11 @@ else
   endif
   endif
endif
-FFLAGS += -I$(INC_NETCDF)
+ifdef INC_NETCDF_FORTRAN
+  FFLAGS += -I$(INC_NETCDF_FORTRAN)
+else
+  FFLAGS += -I$(INC_NETCDF)
+endif
CFLAGS += -I$(INC_NETCDF)

# Set HAVE_SLASHPROC on LINUX systems which are not bluegene or Darwin (OSx)

 

[kshedstrom@alaska_edu]

I changed to a newer version of gfortran. I had to build my own set of netcdf libraries and such. Now I'm getting:

/usr/local/pkg/mpi/OpenMPI/1.10.3-GCC-8.3.0-2.32/bin/mpif90 -c -I. -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/include -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/mpp/include -I/center1/AKWATERS/kate/climate/kshedstrom/Arctic5_JRA.MOM6/bld/ocn/obj/FMS -I. -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/block_control -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/amip_interp -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/column_diagnostics -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/tracer_manager -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/memutils -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/monin_obukhov -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/topography -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/tridiagonal -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/exchange -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/time_interp -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/interpolator -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/horiz_interp -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/fms -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/station_data -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/time_manager -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/sat_vapor_pres -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/data_override -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/astronomy -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/mosaic -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/random_numbers -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/platform -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/diag_manager -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/mpp -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/fft -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/oda_tools -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/include -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/fv3gfs -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/coupler -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/drifters -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/field_manager -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/diag_integral -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/constants -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/axis_utils -I/center1/AKWATERS/kate/climate/kshedstrom/Arctic5_JRA.MOM6/bld/lib/include -fconvert=big-endian -ffree-line-length-none -ffixed-line-length-none -fcray-pointer  -O -fdefault-real-8 -fcray-pointer -fdefault-double-8 -w -I/import/home/kshedstrom/gnu_8.3/include  -DLINUX -Duse_libMPI -Duse_netCDF -Duse_LARGEFILE -DSPMD -D__IFC -DNDEBUG -DHAVE_MPI  -DCESMCOUPLED  -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DHAVE_SLASHPROC -ffree-form -DUSE_CONTIGUOUS= /import/c1/AKWATERS/kate/CESM/libraries/FMS/src/diag_manager/diag_data.F90
make: Leaving directory '/import/c1/AKWATERS/kate/climate/kshedstrom/Arctic5_JRA.MOM6/bld/ocn/obj/FMS'
/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/diag_manager/diag_data.F90:60:6:

   USE netcdf, ONLY: NF_FILL_REAL => NF90_FILL_REAL
      1   
Fatal Error: Cannot read module file ‘netcdf.mod’ opened at (1), because it was created by a different version of GNU Fortran
compilation terminated.

I've built other Fortran codes with this compiler and the files under /import/home/kshedstrom/gnu_8.3.
Is there any way to tell which netcdf.mod it is picking up?

 

[jedwards]

You have more than one in the include path, I don't know of an easy way to determine which it is using. Please post if you find one.

 

[kshedstrom@alaska_edu]

My husband was showing me how to use strace, but it's not so easy with the CESM build system. Anyway, we went to the build directory and executed the command I show above using strace and now it is the time_manager_mod.mod which is not of the same compiler. Doing gunzip on the time_manager_mod.mod shows:

GFORTRAN module version '14' created from /import/c1/AKWATERS/kate/CESM/libraries/FMS/src/time_manager/time_manager.F90

while gunzip on netcdf.mod shows:

GFORTRAN module version '15' created from netcdf4.f90

Somehow, the CESM build system is using some other gfortran on the machine even when I give the full path to the compiler in Macros.make? I find this very mysterious.

 

[kshedstrom@alaska_edu]

I should add that the time_manager_mod.mod was created yesterday during a clean case.build.

 

[jedwards]

I see from your post above that you have the full path to mpif90 specified - but what version of gfortran is it finding? You may need to adjust your PATH environment variable to make sure that you are finding the correct gfortran version.

 

[kshedstrom@alaska_edu]

The way I have it set up now, when I log in, the default gfortran it is finding is an older one from the foss 2016b environment. When I need something newer, I "module purge; module load slurm; module load toolchain/foss/2019b". If I switch completely to the new one, then some favorite tools like git and nco don't work because they aren't done making the 2019 toolchain viable. If I stay in the old one, CESM throws compile-time syntax errors. Do you know what I need to do differently?

 

[jedwards]

Load the modules needed for cesm in your machine section of config_machines.xml, this will make them available for building and running the model without setting them in your shell environment.

 

[kshedstrom@alaska_edu]

Thanks. Is there a way to add a path to the config_machines.xml? I'll have to build my own cmake and stash it somewhere.

 

[jedwards]

Yes there is an environment variables section in the machine definition. You could also define and install your own cmake module.

 

[kshedstrom@alaska_edu]

OK, now I have a different sort of error:

/usr/local/pkg/mpi/OpenMPI/1.10.3-GCC-8.3.0-2.32/bin/mpif90 -c -I. -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/include -I/import/c1/AKWATERS/kate/CESM/libraries/FMS/src/mpp/include -I/center1/AKWATERS/kate/climate/kshedstrom/Arctic5_JRA.MOM6/bld/ocn/obj/FMS -I/center1/AKWATERS/kate/climate/kshedstrom/Arctic5_JRA.MOM6/bld/gnu/openmpi/nodebug/nothreads/nuopc/include -I/center1/AKWATERS/kate/climate/kshedstrom/Arctic5_JRA.MOM6/bld/gnu/openmpi/nodebug/nothreads/nuopc/nuopc/esmf/c1a1i1o1/include -I/import/home/kshedstrom/gnu_8.3/include -I/center1/AKWATERS/kate/climate/kshedstrom/Arctic5_JRA.MOM6/bld/gnu/openmpi/nodebug/nothreads/nuopc/include -I/center1/AKWATERS/kate/climate/kshedstrom/Arctic5_JRA.MOM6/bld/lnd/obj -I/center1/AKWATERS/kate/climate/kshedstrom/Arctic5_JRA.MOM6/bld/lnd/obj -I. -I/import/c1/AKWATERS/kate/CESM/cime/scripts/Arctic5_JRA.MOM6/SourceMods/src.mom -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/config_src/nuopc_driver -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/config_src/dynamic_symmetric -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/ALE -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/core -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/diagnostics -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/equation_of_state -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/equation_of_state/TEOS10 -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/framework -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/ice_shelf -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/initialization -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/ocean_data_assim -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/ocean_data_assim/core -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/ocean_data_assim/geoKdTree -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/pkg/CVMix-src/src/shared -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/parameterizations/lateral -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/parameterizations/vertical -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/tracer -I/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/user -I/center1/AKWATERS/kate/climate/kshedstrom/Arctic5_JRA.MOM6/bld/lib/include -fconvert=big-endian -ffree-line-length-none -ffixed-line-length-none -fcray-pointer  -fdefault-real-8 -Duse_LARGEFILE  -O -I/import/AKWATERS/kshedstrom/src/esmf-esmf/mod/modO/Linux.gfortran.64.mpiuni.default -I/import/AKWATERS/kshedstrom/src/esmf-esmf/src/include   -DLINUX  -DCESMCOUPLED  -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DNDEBUG -DUSE_ESMF_LIB -DNUOPC_INTERFACE -DHAVE_MPI -DHAVE_SLASHPROC -DESMF_VERSION_MAJOR=8 -DESMF_VERSION_MINOR=1 -DATM_PRESENT -DICE_PRESENT -DLND_PRESENT -DOCN_PRESENT -DROF_PRESENT -DGLC_PRESENT -DWAV_PRESENT -DESP_PRESENT -ffree-form -DUSE_CONTIGUOUS= /import/c1/AKWATERS/kate/CESM/components/mom/MOM6/config_src/nuopc_driver/mom_cap.F90

 err:

ERROR:  No results found for variable CAM_DYCORE
/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/src/framework/MOM_file_parser.F90:1018:0:

           line = trim(line(iso+1:last)) ! Strip away the block name for subsequent processing

Warning: ‘__builtin_memmove’ reading 320 bytes from a region of size 319 [-Wstringop-overflow=]
/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/config_src/nuopc_driver/mom_cap.F90:351:19:

      read(value, '(i)', iostat=iostat) scalar_field_count
                   1
Error: Nonnegative width required in format string at (1)
/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/config_src/nuopc_driver/mom_cap.F90:374:19:

      read(value, '(i)', iostat=iostat) scalar_field_idx_grid_nx
                   1
Error: Nonnegative width required in format string at (1)
/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/config_src/nuopc_driver/mom_cap.F90:397:19:

      read(value, '(i)', iostat=iostat) scalar_field_idx_grid_ny
                   1
Error: Nonnegative width required in format string at (1)
/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/config_src/nuopc_driver/mom_cap.F90:1434:51:

          scalar_field_name, scalar_field_count, rc)
                                                   1
Error: Type mismatch in argument ‘value’ at (1); passed REAL(16) to REAL(8)
/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/config_src/nuopc_driver/mom_cap.F90:1441:52:

           scalar_field_name, scalar_field_count, rc)
                                                    1
Error: Type mismatch in argument ‘value’ at (1); passed REAL(16) to REAL(8)
/import/c1/AKWATERS/kate/CESM/components/mom/MOM6/config_src/nuopc_driver/mom_cap.F90:413:38:

        attrList=(/'RestartFileToRead', 'RestartFileToWrite'/), rc=rc)
                                      1
Error: Different CHARACTER lengths (17/18) in array constructor at (1)
make: *** [/import/c1/AKWATERS/kate/CESM/cime/scripts/Arctic5_JRA.MOM6/Tools/Makefile:981: mom_cap.o] Error 1

Any tips for these? Thanks!