Failed to convert GOTPCREL relocation: building cesm1_2 on NERSC Cori

[njumingxuanwu@gmail_com]

After the recently system update on Cori, I can’t build the CESM1.2 on NERSC Cori any more. It gives the error message below:

Load a valid targeting module or set CRAY_CPU_TARGET

ccsm_comp_mod.o: in function `ccsm_comp_mod_mp_ccsm_run_':
ccsm_comp_mod.F90:(.text+0x28): relocation truncated to fit: R_X86_64_32 against symbol `ccsm_comp_mod_mp_infodata_' defined in COMMON section in ccsm_comp_mod.o
ccsm_comp_mod.F90:(.text+0x4e6): relocation truncated to fit: R_X86_64_32S against symbol `ccsm_comp_mod_mp_begstep_' defined in COMMON section in ccsm_comp_mod.o
ccsm_comp_mod.F90:(.text+0x520): relocation truncated to fit: R_X86_64_32S against symbol `ccsm_comp_mod_mp_dtime_' defined in COMMON section in ccsm_comp_mod.o
ccsm_comp_mod.F90:(.text+0x559): relocation truncated to fit: R_X86_64_32S against symbol `ccsm_comp_mod_mp_calendar_' defined in COMMON section in ccsm_comp_mod.o
ccsm_comp_mod.F90:(.text+0x5a3): relocation truncated to fit: R_X86_64_PC32 against symbol `ccsm_comp_mod_mp_dtime_' defined in COMMON section in ccsm_comp_mod.o
ccsm_comp_mod.F90:(.text+0x5c1): relocation truncated to fit: R_X86_64_PC32 against symbol `ccsm_comp_mod_mp_ncpl_' defined in COMMON section in ccsm_comp_mod.o
ccsm_comp_mod.F90:(.text+0x5c8): relocation truncated to fit: R_X86_64_PC32 against symbol `ccsm_comp_mod_mp_cktime_acc_' defined in COMMON section in ccsm_comp_mod.o
ccsm_comp_mod.F90:(.text+0x5cf): relocation truncated to fit: R_X86_64_PC32 against symbol `ccsm_comp_mod_mp_cktime_acc_' defined in COMMON section in ccsm_comp_mod.o
ccsm_comp_mod.F90:(.text+0x5d6): relocation truncated to fit: R_X86_64_PC32 against symbol `ccsm_comp_mod_mp_cktime_acc_' defined in COMMON section in ccsm_comp_mod.o
ccsm_comp_mod.F90:(.text+0x5dd): relocation truncated to fit: R_X86_64_PC32 against symbol `ccsm_comp_mod_mp_cktime_acc_' defined in COMMON section in ccsm_comp_mod.o
ccsm_comp_mod.F90:(.text+0x5e4): additional relocation overflows omitted from the output

/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: failed to convert GOTPCREL relocation; relink with --no-relax
gmake: *** [/global/cscratch1/sd/mwu1/CESM_cases/EmissMIP_CESM1_FAMIPC5_NUG_CMIP5_SO2/Tools/Makefile:636: /global/cscratch1/sd/mwu1/CESM_runs/EmissMIP_CESM1_FAMIPC5_NUG_CMIP5_SO2/bld/cesm.exe] Error 1

I saw similar issues posted on the CESM forum. One solution is to update PIO, but I already have pio1.8.14. I'm running F cases, so it does not need POP for ocean.

The machine configuration (env_mach_specific.cori-knl) is listed below:
source /opt/modules/default/init/csh

module purge

module load craype
module load PrgEnv-intel
module load cray-mpich
# module load craype-mic-knl

if ( $MPILIB == 'mpt' ) then
module swap cray-mpich cray-mpich/7.7.19
endif

if ( $MPILIB == 'impi' ) then
module swap cray-mpich impi/2020.up4
endif

if ( $COMPILER == 'intel' ) then
module load PrgEnv-intel/6.0.10
module rm intel
module load intel/19.1.2.254
endif

if ( $COMPILER == 'gnu' ) then
module swap PrgEnv-intel PrgEnv-gnu/6.0.10
module rm gcc
module load gcc/11.2.0
module rm cray-libsci
module load cray-libsci/20.09.1
endif

module swap craype craype/2.7.10
module rm pmi
module load pmi/5.0.17

if ( $MPILIB == "mpi-serial" ) then
module rm cray-netcdf-hdf5parallel
module rm cray-hdf5-parallel
module rm cray-parallel-netcdf
module load cray-netcdf/4.8.1.1
module load cray-hdf5/1.12.1.1
setenv MKL -mkl=sequential
endif

if ( $MPILIB != "mpi-serial" ) then
module rm cray-netcdf-hdf5parallel
module load cray-netcdf-hdf5parallel/4.8.1.1
module load cray-hdf5-parallel/1.12.1.1
module load cray-parallel-netcdf/1.12.2.1
setenv MKL -mkl
endif

module rm cmake
module load cmake/3.22.1
# module load perl5-extras

setenv OMP_STACKSIZE 128M
setenv OMP_PROC_BIND spread
setenv OMP_PLACES threads
setenv PERL5LIB /global/common/cori/software/perl5/lib/perl5

 

[sacks]

At this time, CESM1.2 and iCESM1.2 are only supported for deep time paleoclimate and water isotope applications, respectively. Are you using the model for one of those purposes? If so, I will direct this to the appropriate forum. If not, I apologize that we are no longer able to provide support for CESM1.2; we encourage you to update to the CESM2 series. See CESM Support Policy for support details.

 

[njumingxuanwu@gmail_com]

Unfortunately, I'm not running it for the two purposes you mentioned.

 

[jedwards]

The issue is that some static declaration in your code is exceeding the limits of memory under the current memory model.
At some point in the past netcdf removed some internal static definitions like NC_MAX_VARS and redefined to MAX_INT
this caused older versions of pio1 that used this variable to give the error you see, thus the need to update pio. Since you have already done
that your error is probably elsewhere. Another place I've seen this error is in the POP ocean model if you try to build on too few tasks, the solution in that case is to increase the pelayout. In this case it looks like maybe its something in the infodata structure.