Linking CESM to an external library with hdf5 and openmp dependencies on Derecho

[jshaw35]

Hi all,

I have had trouble linking CESM with an external library (the RTTOV radiative transfer model) when building on Derecho (I didn't have this issue on Cheyenne but am not sure how relevant that is). I have done my best to describe my issue and code changes in a useful way below, including code and details where most relevant. I'm hoping to eventually pull this new code into CESM/CAM and would love some help as I try to handle a few final bugs that I haven't been able to figure out on my own.

Problem: CESM build fails when RTTOV is built against the HDF5 library and/or RTTOV is built using openmp (error messages during the model build are attached).
A note: When RTTOV is not built with hdf5 or openmp I can link it to CESM on Derecho without errors, but unfortunately I need the hdf5 functionality (openmp is not as important).

My intuition: There are additional flags needed when compiling CESM against RTTOV with the openmp and hdf5 packages. I think that maybe I am trying but not correctly adding those flags ("-lz" and "-qopenmp") to the CESM build.

CESM source code here: /glade/u/home/jonahshaw/w/CESM_source/cesm2_3_alpha17b_rttovhdf (version_info.txt is attached)

Example case here: /glade/u/home/jonahshaw/Cases/CESM2/cesm2_3_alpha17b_rttovhdf/20240327_163959.FHIST.f19_f19_mg17.rttov_test

My code changes (happy to provide these in more detail if helpful):

Changes to the CESM Makefile (cime/CIME/Tools/Makefile):

• Add dependencies on several RTTOV libraries.

# Additionally include RTTOV if asked via CAM_CONFIG_OPTS.
ifeq ($(findstring -rttov,$(CAM_CONFIG_OPTS)),-rttov)
ULIBDEP += $(COSP_LIBDIR)/librttov_wrapper.a $(COSP_LIBDIR)/librttov_mw_scatt.a $(COSP_LIBDIR)/librttov_brdf_atlas.a $(COSP_LIBDIR)/librttov_emis_atlas.a $(COSP_LIBDIR)/librttov_other.a $(COSP_LIBDIR)/librttov_parallel.a $(COSP_LIBDIR)/librttov_coef_io.a $(COSP_LIBDIR)/librttov_hdf.a $(COSP_LIBDIR)/librttov_main.a
endif


Changes to files in components/cam/bld:

• components/cam/bld/configure
  • Update ldflags to include the RTTOV libraries. The RTTOV documentation says this is how to link the code and the normal cosp library is linked in this way. But the HDF5 libraries (-lhdf5_hl -lhdf5_fortran -lhdf5) do not sit in $cosp_libdir, so I am not sure I have done this correctly.

# Turn on RTTOV if passed the keyword. The cosp-rttov Makefile copies rttov libraries to the $cosp_libdir path
if ($rttov) {
$ldflags .= " -L$cosp_libdir -lrttov13_wrapper -lrttov13_mw_scatt -lrttov13_brdf_atlas -lrttov13_emis_atlas -lrttov13_other -lrttov13_parallel -lrttov13_coef_io -lrttov13_hdf -lrttov13_main -lhdf5_hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 ";
# Additional flags needed when RTTOV is compiled with HDF5 (-lz) and openmp (-qopenmp).
$ldflags .= " -lz -qopenmp";
$cfg_ref->set('ldflags', $ldflags);
print "Adding rttov libraries as dependencies in ldflags.\n";
}

• components/cam/bld/namelist_files/namelist_definitions.xml
  • Add namelist variables used in the new code.
• components/cam/bld/config_files/definition.xml
  • Add options so that RTTOV can be built like cosp using CAM_CONFIG_OPTS

Changes to cam source code:

• components/cam/src/physics/cam/cospsimulator_intr.F90
• Since my error is in the build step I do not think that these changes are relevant to this problem.

Changes to COSP source code:

• See git repository here: https://github.com/jshaw35/COSPv2.0/tree/cesm2.2.0_rel_cosp_rttov

For this issue, I think that the relevant changes are in the CESM Makefile and the cam configure file.

I have a few other questions related to building CESM with this new code but wanted to focus on this single issue here. It would be great to talk through this and other questions with a software engineer and I would be happy to set aside time for a call or come up to the Mesa Lab.

Thanks!
Jonah

 

[jedwards]

Hi Jonah,

Thank you for the thorough summary of the issue. I've copied your source and tried to reproduce the problem but I get
stuck earlier -
gmake: *** No rule to make target '/glade/derecho/scratch/jedwards/SMS_Ln9.ne30pg3_ne30pg3_mg17.FLTHIST.derecho_intel.cam-outfrq9s.20240401_130243_b9i9gf/bld/atm/obj/cosp/librttov_wrapper.a', needed by '/glade/derecho/scratch/jedwards/SMS_Ln9.ne30pg3_ne30pg3_mg17.FLTHIST.derecho_intel.cam-outfrq9s.20240401_130243_b9i9gf/bld/cesm.exe'. Stop.

 

[jshaw35]

Hi Jim,

Thanks for jumping in on this. I think that you will need to make two changes in order to reproduce the issue that I did not describe in the original post:
1. Set "RTTOV" as an environment variable before building(e.g. export RTTOV=TRUE). This triggers building against COSP-RTTOV code using "Makefile.rttov" in the cosp2 directory.
2. Add "-cosp" and "-rttov" to the CAM_CONFIG_OPTS setting for the case. This triggers the modifications to ULIBDEP in the CIME Makefile and "ldflags" in cam/bld/configure
^I would like to reduce these to a single step but have not figured out how yet.

Let me know if that helps!

Best,
Jonah

 

[jedwards]

It was the RTTOV env variable that I missed. Once I set that I got the errors that you are seeing - to build with SMP support:
./xmlchange FORCE_SMP_BUILD=TRUE

To add hdf5 libraries edit file ccs_config/machines/cmake_macros
+if(DEFINED ENV{RTTOV})
+ string(APPEND LDFLAGS " -L$(NCAR_LDFLAGS_HDF5) -lhdf5_fortran -lhdf5_hl_fortran -lhdf5")
+endif()

 

[jshaw35]

Hi Jim,

Great, this is really helpful! I have a few more questions.

1. ccs_config/machines/cmake_macros is a directory in my CESM source code (/glade/u/home/jonahshaw/w/CESM_source/cesm2_3_alpha17b_rttovhdf/ccs_config/machines/cmake_macros). What file(s) do I need to modify to add the hdf5 flags using the additional lines you shared?
Also, I see that the ccs_config/machines/cmake_macros path exists within model components (e.g. CAM) as well. Do I need to modify any additional files there?

2. I currently copy the RTTOV source libraries into the COSP library directory each time that I build a case with RTTOV. Then in the CIME Makefile I append the library paths to ULIBDEP. I would prefer to just point CESM to a single build of RTTOV rather than copying the libraries each time. Can I pass the RTTOV library path in via CAM_CONFIG_OPTS such that I can append the original RTTOV path to ULIBDEP? (e.g. ULIBDEP +=
$(RTTOV_LIBDIR)/librttov_wrapper.a $(RTTOV_LIBDIR)/librttov_mw_scatt.a ...) It seems that there is a limited ability to parse strings in the Makefile and my previous attempts to do this have not worked (I added a string field to CAM_CONFIG_OPTS but was unable to read it within the Makefile). Does this seem like a good idea or are there reasons to just copy the RTTOV libraries each time?

Thanks for your help on this!
Jonah

 

[jshaw35]

Hi @jedwards,

I wanted to check in and see if you had any responses on the questions from my last post.

Thanks again for all of your help on this!

Jonah

 

[jedwards]

You only need to modify the ones at the top level ccs_config/machines/cmake_macros.

The files in that directory form a hierarchy of importance
compiler
OS.cmake
compiler_OS.cmake
machine.cmake
compiler_machine.cmake

What I think that you want to do is to change the cosp2 sparse checkout in components/cam/Externals_cam.cfg so that it includes the rrtov subdirectory.
Are you not getting guidance from the cam developers on this?

 

[jshaw35]

Great, just one more question so I make sure I am understanding this. If I am compiling with intel then I just need to update the "intel.cmake" file since it is at the top of the hierarchy?


The RTTOV libraries are completely separate from the COSP source code and interface with RTTOV pulled in by checkout_externals. I'll keep thinking about this and make sure to raise it with the CAM developers when I submit a pull request.

Thanks!

 

[jedwards]

It really depends on how you do it - I assume that you want to commit this option and make it available as part of the distributed model? In that
case you may need to be more specific that using the intel.cmake file and you will want it to work with all of the compilers - so you will need to provide source code, not compiled libraries. I would be happy to participate in the conversation with the cam developers - from the sound of it you may want to initiate this before you submit a pull request.

 

[jshaw35]

I've talked briefly with CAM developers about the RTTOV source code and plan to address that when trying to add this to the distributed model. It is a bit of a conundrum because RTTOV is free to use but access to the source code requires going through NWP SAF. So making the linking with RTTOV optional and having a pre-built version of the libraries on the NCAR systems currently seems like a decent option, though there are certainly others. I'd be really happy to have your insight when I get to that step.

For now I am just aiming to get a functioning version of the code working on Derecho so I can do some additional testing and validation. For that, I think that I just need to handle the initial errors I was having with hdf5 and openmp. I'm planning to implement those changes as soon as I can.