Can I get some guidance on which version to port?

cdevaneprugh · Nov 19, 2024

Hello, my research group is looking to port CTSM or CLM. I have successfully ported CESM 2.1.5 to our HPC but it does not give us all the functionality we are looking for. We need the ability to do spin ups, as well as restarts in single point mode. Based on the demo found here, we need access to the tools for creating input datasets found in $CTSM/tools. Based on the README in that directory, it looks like any version after ctsm5.1.dev158 should be fine? So would the latest ctsm5.3.012 tag work for our purposes?

Any advice is appreciated. I should be able to make any version work with our HPC. It's just a matter of narrowing down the version I need to focus on so I can work with our HPC system administrators.

oleson · Nov 19, 2024

Yes, I think ctsm5.3.012 would work for your purposes. You would be able to do single-point, regional, and global land-only simulations with it. You can create surface datasets with the mksurfdata_esmf tool, as well as subset the global data for single-point and regional simulations using the subset_data tool.

cdevaneprugh · Nov 21, 2024

Thanks Keith!

xiaoxiaokuishu · Dec 9, 2024

oleson said:
Yes, I think ctsm5.3.012 would work for your purposes. You would be able to do single-point, regional, and global land-only simulations with it. You can create surface datasets with the mksurfdata_esmf tool, as well as subset the global data for single-point and regional simulations using the subset_data tool.

Hi, Oleson,

I have a question about the ./subset_data tools,
(1) is it only for CTSM, as I did not find anything under clm5.0.
(2) I have created a surface,domain(?) files from CTSM5.3 ./subset_data tools and try to use it under clm5.0,
I thought maybe there is a problem, the subset_data did not create domainfile file, but CLM5.0 require this file.

Do you have any suggestions for make domainfile easily under CLM5.0, as the tools under clm5.0 seems not easy to use
as the ./subset_data tools.. but ./subset_data did not create domainfile required by CLM5.0..

Best
Ru

cdevaneprugh · Dec 9, 2024

I've got a follow up question regarding building some of the CTSM tools. In $CTSM/tools/mksurfdata_esmf I'm trying to run ./gen_mksurfdata_build but get an error that:

Code:

The PIO directory for the PIO build is required and was not set in the configure
Make sure a PIO build is provided for gnu with openmpi in config_machines

I built the PIO library in $CTSM/libraries/parallelio and tried setting a PIO_LIBDIR environment variable in config_machines as well as the cmake file for our computer but have had no luck. I was going to start messing with the config_pio.xml in ccs_config/machines but figure I would ask here first. How should I proceed?

slevis · Dec 9, 2024

@xiaoxiaokuishu the subset_data tool was developed long after clm5, so its outputs may not be compatible with clm5. If from your experience that is the case, then you will need to use clm5-specific tools to accomplish your tasks.

@cdevaneprugh this seems like a porting issue, so I recommend asking for help in the "porting" forum.

slevis · Dec 9, 2024

I will move this thread to "porting"

Lumoss · Dec 10, 2024

cdevaneprugh said:
I've got a follow up question regarding building some of the CTSM tools. In $CTSM/tools/mksurfdata_esmf I'm trying to run ./gen_mksurfdata_build but get an error that:

Code:

The PIO directory for the PIO build is required and was not set in the configure Make sure a PIO build is provided for gnu with openmpi in config_machines

I built the PIO library in $CTSM/libraries/parallelio and tried setting a PIO_LIBDIR environment variable in config_machines as well as the cmake file for our computer but have had no luck. I was going to start messing with the config_pio.xml in ccs_config/machines but figure I would ask here first. How should I proceed?

I have encountered the same problem as you. Please let me know if you have a solution. Thank you!

Lumoss · Dec 13, 2024

Lumoss said:
I have encountered the same problem as you. Please let me know if you have a solution. Thank you!

add:
if(DEFINED ENV{PIO})
set(PIO_LIBDIR "$ENV{PIO}/lib")
set(PIO_INCDIR "$ENV{PIO}/include")
endif()
to cmake_macros/*.cmake

and add <env name="PIO">your path</env> to your config_machines.xml

cdevaneprugh · Dec 13, 2024

Lumoss said:
add:
if(DEFINED ENV{PIO})
set(PIO_LIBDIR "$ENV{PIO}/lib")
set(PIO_INCDIR "$ENV{PIO}/include")
endif()
to cmake_macros/*.cmake

and add <env name="PIO">your path</env> to your config_machines.xml

This definitely made some progress. When I run the script it looks like it detects the libraries it needs.

Code:

-- The Fortran compiler identification is GNU 12.2.0
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /apps/mpi/gcc/12.2.0/openmpi/4.1.6/bin/mpif90 - skipped
-- Found MPI_Fortran: /apps/mpi/gcc/12.2.0/openmpi/4.1.6/bin/mpif90 (found version "3.1")
-- Found MPI: TRUE (found version "3.1") 
-- Found NetCDF: /apps/gcc/12.2.0/openmpi/4.1.6/netcdf-c/4.9.2/include;/apps/gcc/12.2.0/openmpi/4.1.6/netcdf-f/4.6.1/include (found suitable version "4.9.2", minimum required is "4.7.4") found components: Fortran
-- FindNetCDF defines targets:
--   - NetCDF_VERSION [4.9.2]
--   - NetCDF_PARALLEL [TRUE]
--   - NetCDF_C_CONFIG_EXECUTABLE [/apps/gcc/12.2.0/openmpi/4.1.6/netcdf-c/4.9.2/bin/nc-config]
--   - NetCDF::NetCDF_C [SHARED] [Root: /apps/gcc/12.2.0/openmpi/4.1.6/netcdf-c/4.9.2] Lib: /apps/gcc/12.2.0/openmpi/4.1.6/netcdf-c/4.9.2/lib64/libnetcdf.so
--   - NetCDF_Fortran_CONFIG_EXECUTABLE [/apps/gcc/12.2.0/openmpi/4.1.6/netcdf-f/4.6.1/bin/nf-config]
--   - NetCDF::NetCDF_Fortran [SHARED] [Root: /apps/gcc/12.2.0/openmpi/4.1.6/netcdf-f/4.6.1] Lib: /apps/gcc/12.2.0/openmpi/4.1.6/netcdf-f/4.6.1/lib/libnetcdff.so
-- Found ESMF library: /apps/gcc/12.2.0/openmpi/4.1.6/esmf/8.7.0/lib/libO/Linux.gfortran.64.openmpi.default/libesmf.a
-- Found ESMF: /apps/gcc/12.2.0/openmpi/4.1.6/esmf/8.7.0/lib/libO/Linux.gfortran.64.openmpi.default (found suitable version "8.7.0", minimum required is "8.2.0") 
-- Configuring done (2.8s)
-- Generating done (0.0s)
-- Build files have been written to: /blue/gerber/earth_models/ctsm5.3/tools/mksurfdata_esmf/tool_bld

But then it also spits out a bunch of errors like this at me. Not sure if there is a specific way I need to build the PIO library, or if this is something else.

Code:

[  2%] Building Fortran object CMakeFiles/mksurfdata.dir/shr_kind_mod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[  5%] Building Fortran object CMakeFiles/mksurfdata.dir/mkchecksMod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[  7%] Building Fortran object CMakeFiles/mksurfdata.dir/shr_sys_mod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[ 10%] Building Fortran object CMakeFiles/mksurfdata.dir/mkutilsMod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[ 13%] Building Fortran object CMakeFiles/mksurfdata.dir/shr_const_mod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[ 15%] Building Fortran object CMakeFiles/mksurfdata.dir/mkvarpar.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[ 18%] Building Fortran object CMakeFiles/mksurfdata.dir/mkesmfMod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[ 21%] Building Fortran object CMakeFiles/mksurfdata.dir/mkvarctl.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[ 23%] Building Fortran object CMakeFiles/mksurfdata.dir/mkdiagnosticsMod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
/blue/gerber/earth_models/ctsm5.3/tools/mksurfdata_esmf/src/mkdiagnosticsMod.F90:238:20:

  238 |     call mpi_reduce(loc_gdata_i, gdata_i, 1, MPI_REAL8, MPI_SUM, 0, mpicom, ier)
      |                    1
......
  420 |     call mpi_reduce(loc_garea_i, garea_i, size(garea_i), MPI_REAL8, MPI_SUM, 0, mpicom, ier)
      |                    2
Error: Rank mismatch between actual argument at (1) and actual argument at (2) (rank-1 and scalar)

make[2]: *** [CMakeFiles/mksurfdata.dir/build.make:101: CMakeFiles/mksurfdata.dir/mkdiagnosticsMod.F90.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/mksurfdata.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
Error doing make for hipergator openmpi gnu

@Lumoss Were you able to build everything successfully?

Lumoss · Dec 13, 2024

cdevaneprugh said:

This definitely made some progress. When I run the script it looks like it detects the libraries it needs.

Code:

-- The Fortran compiler identification is GNU 12.2.0
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /apps/mpi/gcc/12.2.0/openmpi/4.1.6/bin/mpif90 - skipped
-- Found MPI_Fortran: /apps/mpi/gcc/12.2.0/openmpi/4.1.6/bin/mpif90 (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- Found NetCDF: /apps/gcc/12.2.0/openmpi/4.1.6/netcdf-c/4.9.2/include;/apps/gcc/12.2.0/openmpi/4.1.6/netcdf-f/4.6.1/include (found suitable version "4.9.2", minimum required is "4.7.4") found components: Fortran
-- FindNetCDF defines targets:
--   - NetCDF_VERSION [4.9.2]
--   - NetCDF_PARALLEL [TRUE]
--   - NetCDF_C_CONFIG_EXECUTABLE [/apps/gcc/12.2.0/openmpi/4.1.6/netcdf-c/4.9.2/bin/nc-config]
--   - NetCDF::NetCDF_C [SHARED] [Root: /apps/gcc/12.2.0/openmpi/4.1.6/netcdf-c/4.9.2] Lib: /apps/gcc/12.2.0/openmpi/4.1.6/netcdf-c/4.9.2/lib64/libnetcdf.so
--   - NetCDF_Fortran_CONFIG_EXECUTABLE [/apps/gcc/12.2.0/openmpi/4.1.6/netcdf-f/4.6.1/bin/nf-config]
--   - NetCDF::NetCDF_Fortran [SHARED] [Root: /apps/gcc/12.2.0/openmpi/4.1.6/netcdf-f/4.6.1] Lib: /apps/gcc/12.2.0/openmpi/4.1.6/netcdf-f/4.6.1/lib/libnetcdff.so
-- Found ESMF library: /apps/gcc/12.2.0/openmpi/4.1.6/esmf/8.7.0/lib/libO/Linux.gfortran.64.openmpi.default/libesmf.a
-- Found ESMF: /apps/gcc/12.2.0/openmpi/4.1.6/esmf/8.7.0/lib/libO/Linux.gfortran.64.openmpi.default (found suitable version "8.7.0", minimum required is "8.2.0")
-- Configuring done (2.8s)
-- Generating done (0.0s)
-- Build files have been written to: /blue/gerber/earth_models/ctsm5.3/tools/mksurfdata_esmf/tool_bld

But then it also spits out a bunch of errors like this at me. Not sure if there is a specific way I need to build the PIO library, or if this is something else.

Code:

[  2%] Building Fortran object CMakeFiles/mksurfdata.dir/shr_kind_mod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[  5%] Building Fortran object CMakeFiles/mksurfdata.dir/mkchecksMod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[  7%] Building Fortran object CMakeFiles/mksurfdata.dir/shr_sys_mod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[ 10%] Building Fortran object CMakeFiles/mksurfdata.dir/mkutilsMod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[ 13%] Building Fortran object CMakeFiles/mksurfdata.dir/shr_const_mod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[ 15%] Building Fortran object CMakeFiles/mksurfdata.dir/mkvarpar.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[ 18%] Building Fortran object CMakeFiles/mksurfdata.dir/mkesmfMod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[ 21%] Building Fortran object CMakeFiles/mksurfdata.dir/mkvarctl.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
[ 23%] Building Fortran object CMakeFiles/mksurfdata.dir/mkdiagnosticsMod.F90.o
f951: Warning: Nonexistent include directory ‘/include’ [-Wmissing-include-dirs]
/blue/gerber/earth_models/ctsm5.3/tools/mksurfdata_esmf/src/mkdiagnosticsMod.F90:238:20:

  238 |     call mpi_reduce(loc_gdata_i, gdata_i, 1, MPI_REAL8, MPI_SUM, 0, mpicom, ier)
      |                    1
......
  420 |     call mpi_reduce(loc_garea_i, garea_i, size(garea_i), MPI_REAL8, MPI_SUM, 0, mpicom, ier)
      |                    2
Error: Rank mismatch between actual argument at (1) and actual argument at (2) (rank-1 and scalar)

make[2]: *** [CMakeFiles/mksurfdata.dir/build.make:101: CMakeFiles/mksurfdata.dir/mkdiagnosticsMod.F90.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/mksurfdata.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
Error doing make for hipergator openmpi gnu

@Lumoss Were you able to build everything successfully?

The version I'm using is ctsm5.2. After modification, I was able to successfully build mksurfdata on my machine without encountering your problem.

Bash:

[100%] Linking Fortran executable mksurfdata
[100%] Built target mksurfdata






Successfully created mksurfdata_esmf executable for: Lumos_intel-oneapi for impi library

cdevaneprugh · Jan 13, 2025

It looks like you're using an intel compiler @Lumoss. Have you by chance tried it with GCC as well?

Looking at the `gen_mksurfdata_build` script, at line 170 when it runs `cmake` there are no Fortran flags for dealing with legacy code. I modified it by adding a few of the flags that I know are needed, giving me a line that looks like this:

Code:

CC=mpicc FC=mpif90 cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_Fortran_FLAGS=" -g -fallow-argument-mismatch -fallow-invalid-boz -ffree-line-length-none" $options $cwd/src

When I run the script now it gets all the way to 97% before failing, so at least that's some progress. It looks like it's failing at `$CTSMROOT/tools/mksurfdata_esmf/src/mksurfdata.F90` with the following errors.

Code:

/ctsm5.3/tools/mksurfdata_esmf/src/mksurfdata.F90:271:24:

  271 |      write(ndiag,'(2(a,I))') ' npes = ', npes, ' grid size = ', grid_size
      |                        1
Error: Nonnegative width required in format string at (1)

/ctsm5.3/tools/mksurfdata_esmf/src/mksurfdata.F90:295:19:

  295 |      read(nfpio, '(i)', iostat=ier) pio_iotype
      |                   1
Error: Nonnegative width required in format string at (1)

/ctsm5.3/tools/mksurfdata_esmf/src/mksurfdata.F90:328:27:

  328 |         write (ndiag,'(a, I, a, I)') ' node_count = ', node_count, ' grid_size = ', grid_size
      |                           1
Error: Nonnegative width required in format string at (1)

I'm confused because it looks like that is just a Fortran syntax error. Or is there a specific compiler flag I can use to ignore this?

Lumoss · Jan 13, 2025

cdevaneprugh said:
It looks like you're using an intel compiler @Lumoss. Have you by chance tried it with GCC as well?

Looking at the `gen_mksurfdata_build` script, at line 170 when it runs `cmake` there are no Fortran flags for dealing with legacy code. I modified it by adding a few of the flags that I know are needed, giving me a line that looks like this:

Code:

CC=mpicc FC=mpif90 cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_Fortran_FLAGS=" -g -fallow-argument-mismatch -fallow-invalid-boz -ffree-line-length-none" $options $cwd/src

When I run the script now it gets all the way to 97% before failing, so at least that's some progress. It looks like it's failing at `$CTSMROOT/tools/mksurfdata_esmf/src/mksurfdata.F90` with the following errors.

Code:

/ctsm5.3/tools/mksurfdata_esmf/src/mksurfdata.F90:271:24: 271 | write(ndiag,'(2(a,I))') ' npes = ', npes, ' grid size = ', grid_size | 1 Error: Nonnegative width required in format string at (1) /ctsm5.3/tools/mksurfdata_esmf/src/mksurfdata.F90:295:19: 295 | read(nfpio, '(i)', iostat=ier) pio_iotype | 1 Error: Nonnegative width required in format string at (1) /ctsm5.3/tools/mksurfdata_esmf/src/mksurfdata.F90:328:27: 328 | write (ndiag,'(a, I, a, I)') ' node_count = ', node_count, ' grid_size = ', grid_size | 1 Error: Nonnegative width required in format string at (1)

I'm confused because it looks like that is just a Fortran syntax error. Or is there a specific compiler flag I can use to ignore this?

Sorry, I didn't try to compile it with gcc, so I don't have any ideas about how to solve your problem. You can seek the help of other professionals. Or try using an intel compiler. Good luck with that.

Yuan Sun · Jan 14, 2025

cdevaneprugh said:
It looks like you're using an intel compiler @Lumoss. Have you by chance tried it with GCC as well?

Looking at the `gen_mksurfdata_build` script, at line 170 when it runs `cmake` there are no Fortran flags for dealing with legacy code. I modified it by adding a few of the flags that I know are needed, giving me a line that looks like this:

Code:

CC=mpicc FC=mpif90 cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_Fortran_FLAGS=" -g -fallow-argument-mismatch -fallow-invalid-boz -ffree-line-length-none" $options $cwd/src

When I run the script now it gets all the way to 97% before failing, so at least that's some progress. It looks like it's failing at `$CTSMROOT/tools/mksurfdata_esmf/src/mksurfdata.F90` with the following errors.

Code:

/ctsm5.3/tools/mksurfdata_esmf/src/mksurfdata.F90:271:24: 271 | write(ndiag,'(2(a,I))') ' npes = ', npes, ' grid size = ', grid_size | 1 Error: Nonnegative width required in format string at (1) /ctsm5.3/tools/mksurfdata_esmf/src/mksurfdata.F90:295:19: 295 | read(nfpio, '(i)', iostat=ier) pio_iotype | 1 Error: Nonnegative width required in format string at (1) /ctsm5.3/tools/mksurfdata_esmf/src/mksurfdata.F90:328:27: 328 | write (ndiag,'(a, I, a, I)') ' node_count = ', node_count, ' grid_size = ', grid_size | 1 Error: Nonnegative width required in format string at (1)

I'm confused because it looks like that is just a Fortran syntax error. Or is there a specific compiler flag I can use to ignore this?

Hi Cooper,

I built the tool with GCC and also met this error. I modified the code manually for your reference.

# modified below:
write(ndiag,'(2(a,I5))') ' npes = ', npes, ' grid size = ', grid_size

read(nfpio, '(I5)', iostat=ier) pio_iotype

write (ndiag,'(a, I5, a, I5)') ' node_count = ', node_count, ' grid_size = ', grid_size

Note that since you met an error when building the tool, you should delete the existing tool_bld folder before starting a new build. The new-version tool, as well as CTSM, worked well with GCC in the HPC machine I am using.

Hope it helps.

Best,
Yuan

cdevaneprugh · Mar 3, 2025

Thanks @Yuan Sun ! I got it to work by changing what you suggested. Additionally, I found I had to change a few lines in the CMakeLists file as well as add fortran specific compiler flags to the build script.

Lines 51-54 of $CTSMROOT/tools/mksurfdata_esmf/src/CMakeLists.txt look like they are asking for conflicting versions of the pio library files. It seems weird that line 51 and 52 specify static libraries, but 53 and 54 are looking for shared ones.

Code:

51    add_library(pioc STATIC IMPORTED)
52    add_library(piof STATIC IMPORTED)
53    set_property(TARGET pioc PROPERTY IMPORTED_LOCATION $ENV{PIO}/lib/libpioc.so)
54    set_property(TARGET piof PROPERTY IMPORTED_LOCATION $ENV{PIO}/lib/libpiof.so)

I chose shared libraries when I built pio, so I ended up changing STATIC to SHARED in lines 51 and 52, and it worked just fine.

For the gen_mksurfdata_build script I added the following to line 170.

Bash:

-DCMAKE_Fortran_FLAGS=" -fallow-argument-mismatch -fallow-invalid-boz -ffree-line-length-none"

We're using gcc 12.2.0 on our HPC for reference.

Hopefully this helps someone out in the future.

Can I get some guidance on which version to port?

cdevaneprugh

Cooper

Member

oleson

Keith Oleson

CSEG and Liaisons

cdevaneprugh

Cooper

Member

xiaoxiaokuishu

Ru Xu

Member

cdevaneprugh

Cooper

Member

slevis

Moderator

slevis

Moderator

Lumoss

Member

Lumoss

Member

cdevaneprugh

Cooper

Member

Lumoss

Member

cdevaneprugh

Cooper

Member

Lumoss

Member

Yuan Sun

Yuan Sun

Member

cdevaneprugh

Cooper

Member