Build issue running running cime/scripts/create_test

[gerardo]

I’ve been asked to set up and run CESM on the HPC-AI Advisory Council’s clusters (specifically, Iris from HPC Advisory Council - Cluster Center, a 32-node cluster with two sockets per node and 28-core Cascade Lake processors in each socket). I’m pretty sure that I have my XML machine, compiler and batch files set up correctly for "iris", but I’m having trouble getting even the libraries to build. However, if I execute the failing gmake commands outside of the Python scripts, they work fine. I even modified config_compilers.xml to ensure that the appropriate compiler and library modules were loaded before the failing compilation command was executed by gmake,

Specifically, I am trying to follow the verification step in section 6.5 of 6. Porting and validating CIME on a new platform — CIME master documentation, and became stuck at:

cime/scripts/create_test --xml-category prealpha --xml-machine cheyenne --xml-compiler intel --machine iris --compiler intel 2>&1 | tee create.log

The creation and set-up steps work but the first attempt to do a build in each test case that is set up fails. Here is a sample build log output:

[gerardo@login01 cesm]$ cat /global/scratch/users/gerardo/cesm/ERI.f09_g17.B1850.iris_intel.allactive-defaultio.20220927_134729_lso12z/bld/gptl.bldlog.220927-134747
gmake -f /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/Makefile install -C /global/scratch/users/gerardo/cesm/sharedlibroot.20220927_134729_lso12z/intel/openmpi/nodebug/nothreads/mct/gptl MACFILE=/global/scratch/users/gerardo/cesm/ERI.f09_g17.B1850.iris_intel.allactive-defaultio.20220927_134729_lso12z/Macros.make MODEL=gptl COMP_NAME=gptl GPTL_DIR=/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing GPTL_LIBDIR=/global/scratch/users/gerardo/cesm/sharedlibroot.20220927_134729_lso12z/intel/openmpi/nodebug/nothreads/mct/gptl SHAREDPATH=/global/scratch/users/gerardo/cesm/ERI.f09_g17.B1850.iris_intel.allactive-defaultio.20220927_134729_lso12z/bld/intel/openmpi/nodebug/nothreads/mct CIME_MODEL=cesm SMP=FALSE CASEROOT="/global/scratch/users/gerardo/cesm/ERI.f09_g17.B1850.iris_intel.allactive-defaultio.20220927_134729_lso12z" CASETOOLS="/global/scratch/users/gerardo/cesm/ERI.f09_g17.B1850.iris_intel.allactive-defaultio.20220927_134729_lso12z/Tools" CIMEROOT="/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime" COMP_INTERFACE="mct" COMPILER="intel" DEBUG="FALSE" EXEROOT="/global/scratch/users/gerardo/cesm/ERI.f09_g17.B1850.iris_intel.allactive-defaultio.20220927_134729_lso12z/bld" INCROOT="/global/scratch/users/gerardo/cesm/ERI.f09_g17.B1850.iris_intel.allactive-defaultio.20220927_134729_lso12z/bld/lib/include" LIBROOT="/global/scratch/users/gerardo/cesm/ERI.f09_g17.B1850.iris_intel.allactive-defaultio.20220927_134729_lso12z/bld/lib" MACH="iris" MPILIB="openmpi" NINST_VALUE="c1a1l1i1o1r1g1w1i1e1" OS="LINUX" PIO_VERSION="1" SHAREDLIBROOT="/global/scratch/users/gerardo/cesm/sharedlibroot.20220927_134729_lso12z" SMP_PRESENT="FALSE" USE_ESMF_LIB="FALSE" USE_MOAB="FALSE" CAM_CONFIG_OPTS="-phys cam6 -co2_cycle" COMP_LND="clm" COMPARE_TO_NUOPC="FALSE" CISM_USE_TRILINOS="FALSE" USE_TRILINOS="FALSE" USE_ALBANY="FALSE" USE_PETSC="FALSE"
gmake: Entering directory '/global/scratch/users/gerardo/cesm/sharedlibroot.20220927_134729_lso12z/intel/openmpi/nodebug/nothreads/mct/gptl'
module list; mpicc -c -I/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing -qno-opt-dynamic-align -fp-model precise -std=gnu99 -xCORE-AVX512 -O2 -debug minimal -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DMCT_INTERFACE -DHAVE_MPI /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/gptl.c
Currently Loaded Modulefiles:
1) cmake/3.21.4 3) intel/2022.1.2 5) compiler-rt/2022.1.0 7) mkl/2022.0.2 9) hdf5/1.10.4-i201h260
2) python/3.7 4) tbb/2021.6.0 6) compiler/2022.1.0 8) hpcx/2.12.0 10) netcdf/4.6.2-i201h260
gmake: Leaving directory '/global/scratch/users/gerardo/cesm/sharedlibroot.20220927_134729_lso12z/intel/openmpi/nodebug/nothreads/mct/gptl'
mpicc: error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory
gmake: *** [/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/Makefile:82: gptl.o] Error 127
ERROR: mpicc: error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory
gmake: *** [/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/Makefile:82: gptl.o] Error 127[gerardo@login01 cesm]$

As I wrote above, I modified config_compilers.xml so that instead of just “mpicc …”, the compilation command executes “module list ; mpicc …”, to verify that the expected modules are indeed loaded before 'mpicc' executes.

I don’t understand the error message (in red), because the mpicc command (in blue) has the “-c” option and shouldn’t be looking for any libraries. In addition, if I load the indicated modules and execute by hand the ‘gmake’ command from the first line of the log output, the corresponding library gets built with no errors whatsoever.

Please let me know which files I should attach.

 

[jedwards]

You neglected to say what version of cesm you are using. libimf.so is an intel compiler library.
You probably need the path to that library in LD_LIBRARY_FLAGS.

 

[jedwards]

Can you post the result of module show intel/2022.1.2?

 

[gerardo]

To answer your first reply, I'm trying to port CESM 2.2; I know that libimf.so is an Intel library, and LD_LIBRARY_FLAGS is set by module load ...

To answer your second reply:
[gerardo@login01 cesm]$ module load cmake/3.21.4 python/3.7 intel/2022.1.2 compiler/2022.1.0 mkl/2022.0.2 hpcx/2.12.0
Loading compiler version 2022.1.0
Loading tbb version 2021.6.0
Loading compiler-rt version 2022.1.0
Load "debugger" to debug DPC++ applications with the gdb-oneapi debugger.
Load "dpl" for additional DPC++ APIs: GitHub - oneapi-src/oneDPL: oneAPI DPC++ Library (oneDPL) https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/dpc-library.html

Loading compiler/2022.1.0
Loading requirement: tbb/2021.6.0 compiler-rt/2022.1.0

Loading mkl version 2022.0.2
[gerardo@login01 cesm]$ module list
Currently Loaded Modulefiles:
1) cmake/3.21.4 2) python/3.7 3) intel/2022.1.2 4) tbb/2021.6.0 5) compiler-rt/2022.1.0 6) compiler/2022.1.0 7) mkl/2022.0.2 8) hpcx/2.12.0
[gerardo@login01 cesm]$ module show intel/2022.1.2
-------------------------------------------------------------------
/global/software/centos-8.x86_64/modfiles/langs/intel/2022.1.2:

module-whatis {This module sets up INTEL 2022.1.2 in your environment.}
conflict intel
conflict gcc
append-path MODULEPATH /global/software/centos-8.x86_64/modfiles/intel/2022.1.2
prepend-path INTEL_LICENSE_FILE 27009@192.168.0.1
-------------------------------------------------------------------
[gerardo@login01 cesm]$ module show compiler-rt/2022.1.0
-------------------------------------------------------------------
/global/software/centos-8.x86_64/modfiles/intel/2022.1.2/compiler-rt/2022.1.0:

conflict compiler-rt32
conflict compiler-rt
unsetenv INTEL_TARGET_ARCH_IA32
prepend-path LIBRARY_PATH /global/software/centos-8.x86_64/modules/langs/intel/2022.1.2/compiler/2022.1.0/linux/lib
prepend-path LD_LIBRARY_PATH /global/software/centos-8.x86_64/modules/langs/intel/2022.1.2/compiler/2022.1.0/linux/compiler/lib/intel64_lin
prepend-path LD_LIBRARY_PATH /global/software/centos-8.x86_64/modules/langs/intel/2022.1.2/compiler/2022.1.0/linux/lib/x64
prepend-path LD_LIBRARY_PATH /global/software/centos-8.x86_64/modules/langs/intel/2022.1.2/compiler/2022.1.0/linux/lib
prepend-path DIAGUTIL_PATH /global/software/centos-8.x86_64/modules/langs/intel/2022.1.2/compiler/2022.1.0/sys_check/sys_check.sh
-------------------------------------------------------------------
[gerardo@login01 cesm]$ mpicc -V
Intel(R) C Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.6.0 Build 20220226_000000
Copyright (C) 1985-2022 Intel Corporation. All rights reserved.

Since loading the newer Intel modules results in undesirable output from the 'module' command, I had to add
allow_error="true" in the <module_system ...> clause of my MACH=iris description in config_machines.xml

Did I mention that if I load the modules and execute the 'gmake' command by hand the gptl library builds with no errors?

 

[jedwards]

In the case directory try running
source .env_mach_specific.sh
make sure that that script is loading all the modules you need without error - if it isn't then there
is something wrong in your config_machines.xml file.
Adding the module list to mpicc is pretty much the same thing, so I'm not sure we'll learn anything.

And I have no idea why you would get a link error when you are compiling that file.

 

[gerardo]

In the case directory try running
source .env_mach_specific.sh
make sure that that script is loading all the modules you need without error - if it isn't then there
is something wrong in your config_machines.xml file.
. . .

[gerardo@login01 cesm]$ cd PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z
[gerardo@login01 PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z]$ ls -a
. CaseDocs check_case env_mach_pes.xml LockedFiles run user_nl_cice xmlchange
.. case.qstatus check_input_data .env_mach_specific.csh Macros.cmake software_environment.txt user_nl_cism xmlquery
archive_metadata case.setup Depends.intel .env_mach_specific.sh Macros.make SourceMods user_nl_clm
bld case.st_archive env_archive.xml env_mach_specific.xml pelayout TestStatus user_nl_cpl
Buildconf CaseStatus env_batch.xml env_run.xml preview_namelists TestStatus.log user_nl_mosart
case.build case.submit env_build.xml env_test.xml preview_run Tools user_nl_pop
case.cmpgen_namelists .case.test env_case.xml env_workflow.xml README.case user_nl_cam user_nl_ww
[gerardo@login01 PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z]$ source .env_mach_specific.sh
[gerardo@login01 PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z]$ module list
Currently Loaded Modulefiles:
1) cmake/3.21.4 3) intel/2022.1.2 5) compiler-rt/2022.1.0 7) mkl/2022.0.2 9) hdf5/1.10.4-i201h260
2) python/3.7 4) tbb/2021.6.0 6) compiler/2022.1.0 8) hpcx/2.12.0 10) netcdf/4.6.2-i201h260
[gerardo@login01 PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z]$

If I now run the gmake command, this is what I get:
[gerardo@login01 PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z]$ gmake -f /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/
timing/Makefile install -C /global/scratch/users/gerardo/cesm/sharedlibroot.20220927_134729_lso12z/intel/openmpi/nodebug/nothreads/mct/gptl MACFILE=/global/scratch/users/gera
rdo/cesm/PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z/Macros.make MODEL=gptl COMP_NAME=gptl GPTL_DIR=/global/home/groups/hpcperf/centos-7/benchmarks/
CESM/cesm2_2/cime/src/share/timing GPTL_LIBDIR=/global/scratch/users/gerardo/cesm/sharedlibroot.20220927_134729_lso12z/intel/openmpi/nodebug/nothreads/mct/gptl SHAREDPATH=/gl
obal/scratch/users/gerardo/cesm/PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z/bld/intel/openmpi/nodebug/nothreads/mct CIME_MODEL=cesm SMP=FALSE CASER
OOT="/global/scratch/users/gerardo/cesm/PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z" CASETOOLS="/global/scratch/users/gerardo/cesm/PFS.f09_g17.B1850
.iris_intel.allactive-default.20220927_134729_lso12z/Tools" CIMEROOT="/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime" COMP_INTERFACE="mct" COMPILER="intel"
DEBUG="FALSE" EXEROOT="/global/scratch/users/gerardo/cesm/PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z/bld" INCROOT="/global/scratch/users/gerardo/c
esm/PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z/bld/lib/include" LIBROOT="/global/scratch/users/gerardo/cesm/PFS.f09_g17.B1850.iris_intel.allactive-
default.20220927_134729_lso12z/bld/lib" MACH="iris" MPILIB="openmpi" NINST_VALUE="c1a1l1i1o1r1g1w1i1e1" OS="LINUX" PIO_VERSION="1" SHAREDLIBROOT="/global/scratch/users/gerard
o/cesm/sharedlibroot.20220927_134729_lso12z" SMP_PRESENT="FALSE" USE_ESMF_LIB="FALSE" USE_MOAB="FALSE" CAM_CONFIG_OPTS="-phys cam6 -co2_cycle" COMP_LND="clm" COMPARE_TO_NUOPC
="FALSE" CISM_USE_TRILINOS="FALSE" USE_TRILINOS="FALSE" USE_ALBANY="FALSE" USE_PETSC="FALSE"
gmake: Entering directory '/global/scratch/users/gerardo/cesm/sharedlibroot.20220927_134729_lso12z/intel/openmpi/nodebug/nothreads/mct/gptl'
mpicc -c -I/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing -qno-opt-dynamic-align -fp-model precise -std=gnu99 -xCORE-AVX512 -O2 -debug minimal -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DMCT_INTERFACE -DHAVE_MPI /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/gptl.c
/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/gptl.c(3444): warning #810: conversion from "struct ompi_communicator_t *" to "int" may lose significant bits
if (((int) comm) == 0)
^
mpicc -c -I/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing -qno-opt-dynamic-align -fp-model precise -std=gnu99 -xCORE-AVX512 -O2 -debug minimal -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DMCT_INTERFACE -DHAVE_MPI /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/GPTLutil.c
mpicc -c -I/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing -qno-opt-dynamic-align -fp-model precise -std=gnu99 -xCORE-AVX512 -O2 -debug minimal -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DMCT_INTERFACE -DHAVE_MPI /global/home/groups/hpcperf/centos7/benchmarks/CESM/cesm2_2/cime/src/share/timing/GPTLget_memusage.c
mpicc -c -I/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing -qno-opt-dynamic-align -fp-model precise -std=gnu99 -xCORE-AVX512 -O2 -debug minimal -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DMCT_INTERFACE -DHAVE_MPI /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/GPTLprint_memusage.c
mpicc -c -I/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing -qno-opt-dynamic-align -fp-model precise -std=gnu99 -xCORE-AVX512 -O2 -debug minimal -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DMCT_INTERFACE -DHAVE_MPI /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/gptl_papi.c
mpicc -c -I/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing -qno-opt-dynamic-align -fp-model precise -std=gnu99 -xCORE-AVX512 -O2 -debug minimal -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DMCT_INTERFACE -DHAVE_MPI /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/f_wrappers.c
mpif90 -c -I/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing -qno-opt-dynamic-align -convert big_endian -assume byterecl -ftz -traceback -assume realloc_lhs -fp-model source -xCORE-AVX512 -O2 -debug minimal -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DMCT_INTERFACE -DHAVE_MPI -free /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/perf_utils.F90
mpif90 -c -I/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing -qno-opt-dynamic-align -convert big_endian -assume byterecl -ftz -traceback -assume realloc_lhs -fp-model source -xCORE-AVX512 -O2 -debug minimal -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DMCT_INTERFACE -DHAVE_MPI -free /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/perf_mod.F90
ar rv libgptl.a gptl.o GPTLutil.o GPTLget_memusage.o GPTLprint_memusage.o gptl_papi.o f_wrappers.o perf_mod.o perf_utils.o
ar: creating libgptl.a
a - gptl.o
a - GPTLutil.o
a - GPTLget_memusage.o
a - GPTLprint_memusage.o
a - gptl_papi.o
a - f_wrappers.o
a - perf_mod.o
a - perf_utils.o
cp -p /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/gptl.h /global/scratch/users/gerardo/cesm/PFS.f09_g17.B1850.iris_intel.allactive-defa
ult.20220927_134729_lso12z/bld/intel/openmpi/nodebug/nothreads/mct/include
cp -p *.mod /global/scratch/users/gerardo/cesm/PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z/bld/intel/openmpi/nodebug/nothreads/mct/include
cp -p libgptl.a /global/scratch/users/gerardo/cesm/PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z/bld/intel/openmpi/nodebug/nothreads/mct/lib
gmake: Leaving directory '/global/scratch/users/gerardo/cesm/sharedlibroot.20220927_134729_lso12z/intel/openmpi/nodebug/nothreads/mct/gptl'
[gerardo@login01 PFS.f09_g17.B1850.iris_intel.allactive-default.20220927_134729_lso12z]$

I wonder whether the warning message produced by the Intel C compiler when compiling gptl.c (the first source in the library build) is what's making gmake (or the python script driving it) fail. I also notice that the same case's bld subdirectory has another log file,
cprnc.bldlog.220927-134737, which is the log of the successful build (using cmake) of an executable, cprnc.

 

[jedwards]

If you take the compiled cprnc and move it to another directory and add CCSM_CPRNC=/path/to/cprnc in your
config_machines.xml it will bypass that step in the case build - maybe that step is somehow interfering. I would also recommend
a very simple case to work out what the problem is here. Please try

create_test SMS_D.f19_g17.X.iris_intel

 

[gerardo]

I did as suggested Re CCSM_CPRNC and tried the simpler test case. Got the same misbehavior:

[gerardo@iris032 cesm2_2]$ cime/scripts/create_test SMS_D.f19_g17.X.iris_intel 2>&1 | tee create.log
Testnames: ['SMS_D.f19_g17.X.iris_intel']
No project info available
create_test will do up to 1 tasks simultaneously
create_test will use up to 70 cores simultaneously
Creating test directory /global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9
RUNNING TESTS:
SMS_D.f19_g17.X.iris_intel
Starting CREATE_NEWCASE for test SMS_D.f19_g17.X.iris_intel with 1 procs
Finished CREATE_NEWCASE for test SMS_D.f19_g17.X.iris_intel in 0.955553 seconds (PASS)
Starting XML for test SMS_D.f19_g17.X.iris_intel with 1 procs
Finished XML for test SMS_D.f19_g17.X.iris_intel in 0.161280 seconds (PASS)
Starting SETUP for test SMS_D.f19_g17.X.iris_intel with 1 procs
Finished SETUP for test SMS_D.f19_g17.X.iris_intel in 1.133360 seconds (PASS)
Starting SHAREDLIB_BUILD for test SMS_D.f19_g17.X.iris_intel with 1 procs
Finished SHAREDLIB_BUILD for test SMS_D.f19_g17.X.iris_intel in 1.716877 seconds (FAIL). [COMPLETED 1 of 1]
Case dir: /global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9
Errors were:
Building test for SMS in directory /global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9
File not found: atm2ocn_fmapname = "cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gx1v7_aave.181205.nc", will attempt to download in check_input_data phase
. . . [several more "File not found" messages]
File not found: wav2ocn_smapname = "cpl/gridmaps/ww3a/map_ww3a_TO_gx1v7_splice_170214.nc", will attempt to download in check_input_data phase
ERROR: /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/build_scripts/buildlib.gptl FAILED, cat /global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9/bld/gptl.bldlog.220929-073432

Due to presence of batch system, create_test will exit before tests are complete.
To force create_test to wait for full completion, use --wait
At test-scheduler close, state is:
FAIL SMS_D.f19_g17.X.iris_intel (phase SHAREDLIB_BUILD)
Case dir: /global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9
test-scheduler took 4.216069459915161 seconds
[gerardo@iris032 cesm2_2]$

And

[gerardo@login01 cesm]$ cat /global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9/bld/gptl.bldlog.220929-073432
gmake -f /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/Makefile install -C /global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9/bld/intel/openmpi/debug/nothreads/mct/gptl MACFILE=/global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9/Macros.make MODEL=gptl COMP_NAME=gptl GPTL_DIR=/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing GPTL_LIBDIR=/global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9/bld/intel/openmpi/debug/nothreads/mct/gptl SHAREDPATH=/global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9/bld/intel/openmpi/debug/nothreads/mct CIME_MODEL=cesm SMP=FALSE CASEROOT="/global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9" CASETOOLS="/global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9/Tools" CIMEROOT="/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime" COMP_INTERFACE="mct" COMPILER="intel" DEBUG="TRUE" EXEROOT="/global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9/bld" INCROOT="/global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9/bld/lib/include" LIBROOT="/global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9/bld/lib" MACH="iris" MPILIB="openmpi" NINST_VALUE="c1a1l1i1o1r1g1w1i1e1" OS="LINUX" PIO_VERSION="1" SHAREDLIBROOT="/global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_
intel.20220929_073429_wsb6t9/bld" SMP_PRESENT="FALSE" USE_ESMF_LIB="FALSE" USE_MOAB="FALSE" COMP_LND="xlnd" COMPARE_TO_NUOPC="FALSE" USE_TRILINOS="FALSE" USE_ALBANY="FALSE" USE_PETSC="FALSE"
gmake: Entering directory '/global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9/bld/intel/openmpi/debug/nothreads/mct/gptl'
mpicc -c -I/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing -qno-opt-dynamic-align -fp-model precise -std=gnu99 -xCORE-AVX512 -diag-disable=all -O0 -g -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DMCT_INTERFACE -DDEBUG -DHAVE_MPI /global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/gptl.c
gmake: Leaving directory '/global/scratch/users/gerardo/cesm/SMS_D.f19_g17.X.iris_intel.20220929_073429_wsb6t9/bld/intel/openmpi/debug/nothreads/mct/gptl'
mpicc: error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory
gmake: *** [/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/Makefile:82: gptl.o] Error 127
ERROR: mpicc: error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory
gmake: *** [/global/home/groups/hpcperf/centos-7/benchmarks/CESM/cesm2_2/cime/src/share/timing/Makefile:82: gptl.o] Error 127[gerardo@login01 cesm]$
[gerardo@login01 cesm]$

Again, if I execute the mpicc command by hand, I get no errors and 'gptl.o' is duly generated. (I added -diag-disable=all to the compiler options to get rid of a warning, in the unlikely case it might be interfering.) It appears to me that something else is going on in the script that executes the gmake command.

 

[gerardo]

BTW, I also tried using Python 2.7 instead of Python 3.7 (modifying config_machines.xml accordingly), but got the same failure.