Porting cesm 3.0b02 on Fujitsu A64FX - "mpp_min" module or submodule not found from ocn_comp_nuopc.mod

[mike_scchen@gmail_com]

Hi,
I'm trying to port the cesm with Fujitsu compiler.
I've managed to build all components that B1850 (we did add the compset difinition since the git-ed source doesn't have this) compset needs, but got this error message in the last (maybe?) step:

mpifrtpx (......) cesm/components/cmeps/cime_config/../cesm/driver/esm.F90
jwd1384i-s "/IFS/nwpr/gfs/com132/cesm3_0_b02/components/cmeps/cime_config/../cesm/driver/esm.F90", line 826: The module or submodule 'mpp_min' that is used directly or indirectly in the module or submodule 'ocn_comp_nuopc' must be available.

This "mpp_min" seems to come from FMS.
But I did see some modules that contain the "mpp_min" strings in scratch/case/bld/....../FMS folder, so I have no idea why the compiler / linker cannnot find or use these modules.
I tried to compare the results from another working port on common x86_64 / Intel compiler platform; but the modules from different compilers cannot be compared directly.
Any ideas are welcome :p

Mike

 

[mike_scchen@gmail_com]

PS.
The fujitsu/(......) is NOT in the include path, so the only available module is:
ocn/obj/mom_coms_infra.mod

 

[jedwards]

Could it be that the path is too long for your compiler to handle? For example in my build of a specific case I see that this include path is quite long:

-I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-d\
efaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/FMS/

I didn't understand your PS about the only available module. Looking at my compile instruction for esm.F90 I see that there are quite a number of include paths listed and that the FMS directory is at the top of the list.

ftn -c -I. -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-d\
efaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/FMS/ -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_\
MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/CDEPS/fox/include -I/glade/derecho/scr\
atch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v\
/bld/intel/mpich/nodebug/nothreads/CDEPS/dshr -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE\
_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/include -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_w\
g37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/\
include -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defa\
ultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/finclude -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1\
/wplx/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/oneapi/2023.2.1/2qqp/include -I/glade/derecho/scratch/jedwards/\
PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/atm/obj -\
I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.202\
50122_160141_2uml0v/bld/ice/obj -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.dere\
cho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/ocn/obj -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_M\
OSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/glc/obj -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CA\
M70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/rof/obj -I/glade/derecho/scratch/jedwards\
/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/wav/obj \
-I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20\
250122_160141_2uml0v/bld/esp/obj -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.der\
echo_intel.allactive-defaultio.20250122_160141_2uml0v/bld/iac/obj -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/cray-mpich/8.1.27/oneapi/2023.2.1/z\
yhu/include -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-\
defaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/include -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_C\
ICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/include -I/glade/derecho/scratch/j\
edwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/i\
ntel/mpich/nodebug/nothreads/clm/obj -I. -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_\
SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/SourceMods/src.drv -I/glade/work/jedwards/sandboxes/cesm3_0_alphabranch/components/cmeps/cime_config/.\
./mediator -I/glade/work/jedwards/sandboxes/cesm3_0_alphabranch/components/cmeps/cime_config/../cesm/flux_atmocn -I/glade/work/jedwards/sandboxes/cesm3_0_alphabranc\
h/components/cmeps/cime_config/../cesm/driver -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE\
_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/lib/include -qno-opt-dynamic-align  -convert big_endian -assume byterecl -ftz -traceback -ass\
ume realloc_lhs -fp-model source -O2 -debug minimal -no-fma -qopt-report -march=core-avx2 -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27\
/oneapi/2023.2.1/7haa/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/oneapi/2023.2.1/7haa/include -I/glade/u/apps/derecho/23.09/\
spack/opt/spack/netcdf-c/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/njnx/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf-fortran/4.6.1/cray-mpich/8.1.27/o\
neapi/2023.2.1/w55c/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/oneapi/2023.2.1/2qqp/include -I/glade/u/apps/dere\
cho/23.09/spack/opt/spack/parallelio/2.6.2/cray-mpich/8.1.27/oneapi/2023.2.1/zyhu/include  -DCNL  -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DLINUX -DHAVE_GETTID\
 -DNDEBUG -DUSE_ESMF_LIB -DHAVE_MPI -DNUOPC_INTERFACE -DPIO2 -DHAVE_SLASHPROC -D_PNETCDF -DESMF_VERSION_MAJOR=8 -DESMF_VERSION_MINOR=6 -DATM_PRESENT -DICE_PRESENT -\
DLND_PRESENT -DOCN_PRESENT -DROF_PRESENT -DGLC_PRESENT -DWAV_PRESENT -DMED_PRESENT -DPIO2 -free -DUSE_CONTIGUOUS= /glade/work/jedwards/sandboxes/cesm3_0_alphabranch\
/components/cmeps/cime_config/../cesm/driver/esm.F90

 

[mike_scchen@gmail_com]

Huh that's really strange...
Neither the cesm.bldlog on x86_64 / Intel and A64fx has the FMS in the include path here.

This is the esm.F90 line of my porting attempt on x86_64 / Intel:

mpiifort -c -I. -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/CDEPS/fox/include -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/CDEPS/dshr -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/include -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/include -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/finclude -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/netcdf-4.9.2/include -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/pnetcdf-1.12.2/include -I/work1/u8935479/scratch/cesm3-f1-nox/bld/atm/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/ice/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/ocn/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/glc/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/rof/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/wav/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/esp/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/iac/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/include -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/include -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/clm/obj -I. -I/home/u8935479/case/cesm3-f1-nox/SourceMods/src.drv -I/home/u8935479/cesm3_0_b02/components/cmeps/cime_config/../mediator -I/home/u8935479/cesm3_0_b02/components/cmeps/cime_config/../cesm/flux_atmocn -I/home/u8935479/cesm3_0_b02/components/cmeps/cime_config/../cesm/driver -I/work1/u8935479/scratch/cesm3-f1-nox/bld/lib/include -qno-opt-dynamic-align -convert big_endian -assume byterecl -ftz -traceback -assume realloc_lhs -fp-model source -O2 -debug minimal -no-fma -qmkl=cluster -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/esmf-8.6.0/mod/modO/Linux.intel.64.intelmpi.default -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/esmf-8.6.0/include -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/netcdf-4.9.2/include -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/netcdf-4.9.2/include -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/pnetcdf-1.12.2/include -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/pio-2.6.2/include -DLinux -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DNDEBUG -DUSE_ESMF_LIB -DHAVE_MPI -DNUOPC_INTERFACE -DPIO2 -D_PNETCDF -DESMF_VERSION_MAJOR=8 -DESMF_VERSION_MINOR=6 -DATM_PRESENT -DICE_PRESENT -DLND_PRESENT -DOCN_PRESENT -DROF_PRESENT -DGLC_PRESENT -DMED_PRESENT -DPIO2 -free -DUSE_CONTIGUOUS= /home/u8935479/cesm3_0_b02/components/cmeps/cime_config/../cesm/driver/esm.F90

And on A64fx:

mpifrtpx -c -I. -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/CDEPS/fox/include -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/CDEPS/dshr -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/include -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/include -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/finclude -I/nwpr/gfs/com132/fx1000/netcdf-4.9.2/include -I/nwpr/gfs/com132/fx1000/pnetcdf-1.14.0/include -I/nwpr/gfs/com132/scratch/test/bld/atm/obj -I/nwpr/gfs/com132/scratch/test/bld/ice/obj -I/nwpr/gfs/com132/scratch/test/bld/ocn/obj -I/nwpr/gfs/com132/scratch/test/bld/glc/obj -I/nwpr/gfs/com132/scratch/test/bld/rof/obj -I/nwpr/gfs/com132/scratch/test/bld/wav/obj -I/nwpr/gfs/com132/scratch/test/bld/esp/obj -I/nwpr/gfs/com132/scratch/test/bld/iac/obj -I/nwpr/gfs/com132/fx1000/pio-2.6.2/include -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/include -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/include -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/clm/obj -I. -I/IFS/nwpr/gfs/com132/cases/test/SourceMods/src.drv -I/IFS/nwpr/gfs/com132/cesm3_0_b02/components/cmeps/cime_config/../mediator -I/IFS/nwpr/gfs/com132/cesm3_0_b02/components/cmeps/cime_config/../cesm/flux_atmocn -I/IFS/nwpr/gfs/com132/cesm3_0_b02/components/cmeps/cime_config/../cesm/driver -I/nwpr/gfs/com132/scratch/test/bld/lib/include -Nalloc_assign -O2 -fs -I/nwpr/gfs/com132/fx1000/esmf-8.6.1/mod/modO/Linux.fujitsu.64.mpi.default -I/nwpr/gfs/com132/fx1000/esmf-8.6.1/include -I/nwpr/gfs/com132/fx1000/pio-2.6.2/include -I/nwpr/gfs/com132/fx1000/netcdf-4.9.2/include -I/nwpr/gfs/com132/fx1000/netcdf-4.9.2/include -I/nwpr/gfs/com132/fx1000/pnetcdf-1.14.0/include -DLinux -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRFUJITSU -DLINUX -DNDEBUG -DUSE_ESMF_LIB -DHAVE_MPI -DNUOPC_INTERFACE -DPIO2 -DHAVE_SLASHPROC -D_PNETCDF -DESMF_VERSION_MAJOR=8 -DESMF_VERSION_MINOR=6 -DATM_PRESENT -DICE_PRESENT -DLND_PRESENT -DOCN_PRESENT -DROF_PRESENT -DGLC_PRESENT -DMED_PRESENT -DPIO2 -Free -Fwide -DUSE_CONTIGUOUS= /IFS/nwpr/gfs/com132/cesm3_0_b02/components/cmeps/cime_config/../cesm/driver/esm.F90

 

[mike_scchen@gmail_com]

And this is the modification in config_compsets.xml:

<compset>
<alias>B1850</alias>
<lname>1850_CAM70_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_SWAV</lname>
</compset>