How to produce dynamically linked component model libraries in CESM1.5

[koichi_sakaguchi@pnnl_gov]

Hi forum members,

I'd like to know if there are ways to build dynamically linked libraries for each model component (libatm.so, liblnd.so, ... etc.) instead of statically linked ones (libatm.a, liblnd.a).

I am trying to compile CESM1.5 with our custom modifications to CAM for the NERSC Cori system. With ifort version 19.0 or later, we get a compile-time error that has been reported in this forum (relocation truncated to fit: R_X86_64_PC32 against `.bss'). But after the March 2022 update at NERSC, older ifort versions are not available anymore.

A reported solution in this forum is either use certain versions of PIO1 (we are using version 1.9.10 according to the ChangeLog in our cesm1.5 code) or to build the model with '-mcmodel=medium' option
relocation truncated to fit error for compset C
CESM1.2 build problem on cheyenne, relocation truncated to fit
relocation truncated to fit: R_X86_64_32 against `.bss'


Focusing on the mcmodel flag, ifort compiler also requires the model being built as dynamically linked executable by adding '-shared -fpic' flags. But even with these flags, we still get the same error (but from different functions); relocation truncated to fit: R_X86_64_32 against symbol `rrlw_kg03_mp_kao_' defined in LARGE_COMMON section in ... /bld/lib//libatm.a(rrlw_kg03.o). The error happens when cesm.exe is being built by linking to component model libraries, e.g., libatm.a, libice.a.

According to the NERSC user support and the ifort manual,

"shared libraries must not be linked with archive libraries" (in the section "Creating Shared Libraries" in the ifort manual, https://cdrdv2.intel.com/v1/dl/getContent/671552?explicitVersion=true&wapkw=ifort 19.0).

So we like to try building each model component as shard library instead of static libraries, which are then linked to produce dynamically linked cesm.exe. We tried to enforce dynamic linking by setting environmental variable CRAYPE_LINK_TYPE=dynamic in env_mach_specific.xml, in addition to the '-shared' option in FFLAGS, CFLAGS, and LDFLAGS in "Macros.make" for a working case, and also to the fact that the default is to build dynamically linked libraries/executables in the current NERSC environment.

With those specifications, we still get statically linked model components (libatm.a, libice.a).

Could anyone tell me how to make dynamically linked model libraries? Or any other solutions to avoid the "relocation truncated to fit" error?

Please note that we need to use the CESM1.5 code with our modifications to finish on-going long-term climate simulations.

Best regards,

Koichi

 

[jedwards]

I'm sorry but cesm1.5 is no longer supported. The general nature of the error that you are seeing
is that some static allocations are too large for the memory model in use. I don't think that using
dynamically linked libraries is going to solve that issue for you. You can update pio to version 1.9.22 to solve that particular issue in the pio library.

 

[koichi_sakaguchi@pnnl_gov]

Hi Jim,

Too bad that cesm1.5 is not supported, but thanks for explanation and sharing your insight.

I'll report back your suggestions to the NERSC support desk. But just for testing/curiosity, so it is impossible to produce dynamically linked libraries for CESM model components?

I really wonder why some allocations suddenly become too big by just switching ifort version from 18 or older to 19 or newer, through which the mcmodel option seems to remain the same (but memory model changed?) and for the same model code.

I did not get such errors from pio so far; I see errors from zlib and mpich when statically building the model, and then from the cesm code (cam's rrtmg) when dynamically building the model.

Rapid (in domain scientists' timescale) & irreversible changes to both the model code and HPC environment make it very challenging to do experiments involving long-term climate simulation with custom-modifications in the code, as well as reproduce scientific results. For example, clm4 (which we use in the simulation) is not supported in the latest CESM2 model :(

Best,

Koichi