Porting cesm 3.0b02 on Fujitsu A64FX - "mpp_min" module or submodule not found from ocn_comp_nuopc.mod

[mike_scchen@gmail_com]

Hi,
I'm trying to port the cesm with Fujitsu compiler.
I've managed to build all components that B1850 (we did add the compset difinition since the git-ed source doesn't have this) compset needs, but got this error message in the last (maybe?) step:

mpifrtpx (......) cesm/components/cmeps/cime_config/../cesm/driver/esm.F90
jwd1384i-s "/IFS/nwpr/gfs/com132/cesm3_0_b02/components/cmeps/cime_config/../cesm/driver/esm.F90", line 826: The module or submodule 'mpp_min' that is used directly or indirectly in the module or submodule 'ocn_comp_nuopc' must be available.

This "mpp_min" seems to come from FMS.
But I did see some modules that contain the "mpp_min" strings in scratch/case/bld/....../FMS folder, so I have no idea why the compiler / linker cannnot find or use these modules.
I tried to compare the results from another working port on common x86_64 / Intel compiler platform; but the modules from different compilers cannot be compared directly.
Any ideas are welcome :p

Mike

 

[mike_scchen@gmail_com]

PS.
The fujitsu/(......) is NOT in the include path, so the only available module is:
ocn/obj/mom_coms_infra.mod

 

[jedwards]

Could it be that the path is too long for your compiler to handle? For example in my build of a specific case I see that this include path is quite long:

-I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-d\
efaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/FMS/

I didn't understand your PS about the only available module. Looking at my compile instruction for esm.F90 I see that there are quite a number of include paths listed and that the FMS directory is at the top of the list.

ftn -c -I. -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-d\
efaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/FMS/ -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_\
MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/CDEPS/fox/include -I/glade/derecho/scr\
atch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v\
/bld/intel/mpich/nodebug/nothreads/CDEPS/dshr -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE\
_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/include -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_w\
g37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/\
include -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defa\
ultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/finclude -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1\
/wplx/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/oneapi/2023.2.1/2qqp/include -I/glade/derecho/scratch/jedwards/\
PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/atm/obj -\
I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.202\
50122_160141_2uml0v/bld/ice/obj -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.dere\
cho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/ocn/obj -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_M\
OSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/glc/obj -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CA\
M70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/rof/obj -I/glade/derecho/scratch/jedwards\
/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/wav/obj \
-I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20\
250122_160141_2uml0v/bld/esp/obj -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.der\
echo_intel.allactive-defaultio.20250122_160141_2uml0v/bld/iac/obj -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallelio/2.6.2/cray-mpich/8.1.27/oneapi/2023.2.1/z\
yhu/include -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-\
defaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/include -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_C\
ICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/intel/mpich/nodebug/nothreads/include -I/glade/derecho/scratch/j\
edwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/i\
ntel/mpich/nodebug/nothreads/clm/obj -I. -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_WW3_\
SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/SourceMods/src.drv -I/glade/work/jedwards/sandboxes/cesm3_0_alphabranch/components/cmeps/cime_config/.\
./mediator -I/glade/work/jedwards/sandboxes/cesm3_0_alphabranch/components/cmeps/cime_config/../cesm/flux_atmocn -I/glade/work/jedwards/sandboxes/cesm3_0_alphabranc\
h/components/cmeps/cime_config/../cesm/driver -I/glade/derecho/scratch/jedwards/PFS_PL.ne30pg3_t232_wg37.1850_CAM70%LT_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE\
_WW3_SESP.derecho_intel.allactive-defaultio.20250122_160141_2uml0v/bld/lib/include -qno-opt-dynamic-align  -convert big_endian -assume byterecl -ftz -traceback -ass\
ume realloc_lhs -fp-model source -O2 -debug minimal -no-fma -qopt-report -march=core-avx2 -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27\
/oneapi/2023.2.1/7haa/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/oneapi/2023.2.1/7haa/include -I/glade/u/apps/derecho/23.09/\
spack/opt/spack/netcdf-c/4.9.2/cray-mpich/8.1.27/oneapi/2023.2.1/njnx/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/netcdf-fortran/4.6.1/cray-mpich/8.1.27/o\
neapi/2023.2.1/w55c/include -I/glade/u/apps/derecho/23.09/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.27/oneapi/2023.2.1/2qqp/include -I/glade/u/apps/dere\
cho/23.09/spack/opt/spack/parallelio/2.6.2/cray-mpich/8.1.27/oneapi/2023.2.1/zyhu/include  -DCNL  -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DLINUX -DHAVE_GETTID\
 -DNDEBUG -DUSE_ESMF_LIB -DHAVE_MPI -DNUOPC_INTERFACE -DPIO2 -DHAVE_SLASHPROC -D_PNETCDF -DESMF_VERSION_MAJOR=8 -DESMF_VERSION_MINOR=6 -DATM_PRESENT -DICE_PRESENT -\
DLND_PRESENT -DOCN_PRESENT -DROF_PRESENT -DGLC_PRESENT -DWAV_PRESENT -DMED_PRESENT -DPIO2 -free -DUSE_CONTIGUOUS= /glade/work/jedwards/sandboxes/cesm3_0_alphabranch\
/components/cmeps/cime_config/../cesm/driver/esm.F90

 

[mike_scchen@gmail_com]

Huh that's really strange...
Neither the cesm.bldlog on x86_64 / Intel and A64fx has the FMS in the include path here.

This is the esm.F90 line of my porting attempt on x86_64 / Intel:

mpiifort -c -I. -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/CDEPS/fox/include -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/CDEPS/dshr -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/include -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/include -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/finclude -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/netcdf-4.9.2/include -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/pnetcdf-1.12.2/include -I/work1/u8935479/scratch/cesm3-f1-nox/bld/atm/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/ice/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/ocn/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/glc/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/rof/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/wav/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/esp/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/iac/obj -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/include -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/include -I/work1/u8935479/scratch/cesm3-f1-nox/bld/intel/intelmpi/nodebug/nothreads/clm/obj -I. -I/home/u8935479/case/cesm3-f1-nox/SourceMods/src.drv -I/home/u8935479/cesm3_0_b02/components/cmeps/cime_config/../mediator -I/home/u8935479/cesm3_0_b02/components/cmeps/cime_config/../cesm/flux_atmocn -I/home/u8935479/cesm3_0_b02/components/cmeps/cime_config/../cesm/driver -I/work1/u8935479/scratch/cesm3-f1-nox/bld/lib/include -qno-opt-dynamic-align -convert big_endian -assume byterecl -ftz -traceback -assume realloc_lhs -fp-model source -O2 -debug minimal -no-fma -qmkl=cluster -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/esmf-8.6.0/mod/modO/Linux.intel.64.intelmpi.default -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/esmf-8.6.0/include -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/netcdf-4.9.2/include -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/netcdf-4.9.2/include -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/pnetcdf-1.12.2/include -I/home/yhtseng00/coda/intel-2022.3.1/intelmpi/pio-2.6.2/include -DLinux -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRINTEL -DNDEBUG -DUSE_ESMF_LIB -DHAVE_MPI -DNUOPC_INTERFACE -DPIO2 -D_PNETCDF -DESMF_VERSION_MAJOR=8 -DESMF_VERSION_MINOR=6 -DATM_PRESENT -DICE_PRESENT -DLND_PRESENT -DOCN_PRESENT -DROF_PRESENT -DGLC_PRESENT -DMED_PRESENT -DPIO2 -free -DUSE_CONTIGUOUS= /home/u8935479/cesm3_0_b02/components/cmeps/cime_config/../cesm/driver/esm.F90

And on A64fx:

mpifrtpx -c -I. -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/CDEPS/fox/include -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/CDEPS/dshr -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/include -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/include -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/finclude -I/nwpr/gfs/com132/fx1000/netcdf-4.9.2/include -I/nwpr/gfs/com132/fx1000/pnetcdf-1.14.0/include -I/nwpr/gfs/com132/scratch/test/bld/atm/obj -I/nwpr/gfs/com132/scratch/test/bld/ice/obj -I/nwpr/gfs/com132/scratch/test/bld/ocn/obj -I/nwpr/gfs/com132/scratch/test/bld/glc/obj -I/nwpr/gfs/com132/scratch/test/bld/rof/obj -I/nwpr/gfs/com132/scratch/test/bld/wav/obj -I/nwpr/gfs/com132/scratch/test/bld/esp/obj -I/nwpr/gfs/com132/scratch/test/bld/iac/obj -I/nwpr/gfs/com132/fx1000/pio-2.6.2/include -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/include -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/include -I/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodebug/nothreads/clm/obj -I. -I/IFS/nwpr/gfs/com132/cases/test/SourceMods/src.drv -I/IFS/nwpr/gfs/com132/cesm3_0_b02/components/cmeps/cime_config/../mediator -I/IFS/nwpr/gfs/com132/cesm3_0_b02/components/cmeps/cime_config/../cesm/flux_atmocn -I/IFS/nwpr/gfs/com132/cesm3_0_b02/components/cmeps/cime_config/../cesm/driver -I/nwpr/gfs/com132/scratch/test/bld/lib/include -Nalloc_assign -O2 -fs -I/nwpr/gfs/com132/fx1000/esmf-8.6.1/mod/modO/Linux.fujitsu.64.mpi.default -I/nwpr/gfs/com132/fx1000/esmf-8.6.1/include -I/nwpr/gfs/com132/fx1000/pio-2.6.2/include -I/nwpr/gfs/com132/fx1000/netcdf-4.9.2/include -I/nwpr/gfs/com132/fx1000/netcdf-4.9.2/include -I/nwpr/gfs/com132/fx1000/pnetcdf-1.14.0/include -DLinux -DCESMCOUPLED -DFORTRANUNDERSCORE -DCPRFUJITSU -DLINUX -DNDEBUG -DUSE_ESMF_LIB -DHAVE_MPI -DNUOPC_INTERFACE -DPIO2 -DHAVE_SLASHPROC -D_PNETCDF -DESMF_VERSION_MAJOR=8 -DESMF_VERSION_MINOR=6 -DATM_PRESENT -DICE_PRESENT -DLND_PRESENT -DOCN_PRESENT -DROF_PRESENT -DGLC_PRESENT -DMED_PRESENT -DPIO2 -Free -Fwide -DUSE_CONTIGUOUS= /IFS/nwpr/gfs/com132/cesm3_0_b02/components/cmeps/cime_config/../cesm/driver/esm.F90

 

[mike_scchen@gmail_com]

And this is the modification in config_compsets.xml:

<compset>
<alias>B1850</alias>
<lname>1850_CAM70_CLM60%BGC-CROP_CICE_MOM6_MOSART_DGLC%NOEVOLVE_SWAV</lname>
</compset>

 

[mike_scchen@gmail_com]

I tried to add the FMS include path in the failed command like:

mpifrtpx -c -I. -I/IFS/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodefug/nothreads/FMS (......)
/IFS/nwpr/gfs/com132/cesm3_0_b02/components/cmeps/cime_config/../cesm/driver/esm.F90

It still fails, but the error message becomes:

jwd1370i-s "/IFS/nwpr/gfs/com132/cesm3_0_b02/components/cmeps/cime config/../cesm/driver/esm.F90", line 801: Module or submodule 'med_internalstate_mod' is not available.

This file is in bld/cpl/obj.
Trying again with:

mpifrtpx -c -I. -I/IFS/nwpr/gfs/com132/scratch/test/bld/cpl/obj -I/IFS/nwpr/gfs/com132/scratch/test/bld/fujitsu/openmpi/nodefug/nothreads/FMS (......)
/IFS/nwpr/gfs/com132/cesm3_0_b02/components/cmeps/cime_config/../cesm/driver/esm.F90

passed at least for this esm.F90.
Maybe there's some difference on the module file search for Fujitsu compiler and Intel compiler.
So is there some way to change the include paths set by the CESM build system...?

 

[mike_scchen@gmail_com]

Update:
The problem was solved.
Upon browsing the cime/CIME/Tools/Makefile, it was found that the FMS include lines is not working as it should.
The FMS lines in 3.0.beta02, line 69~71:

ifeq ($(strip $(USE_FMS)), TRUE)
SLIBS += -lfms
endif

The FMS lines in master branch, line 129~132:

ifeq ($(strip $(USE_FMS)), TRUE)
SLIBS += -lfms
INCLDIR += -I$(SHAREDLIBROOT)/$(SHAREDPATH)/FMS/
endif

The build process can complete after making corresponding changes to the Makefile.

 

[jedwards]

You might want to update to the latest tag - cesm3_0_beta05. I have confirmed that this FMS change is in that tag.

 

[mike_scchen@gmail_com]

I'm trying the beta05 tag and uses BHISTC_MTt4s / ne30pg3_t232, which is scientifically supported.
I applied some changes I made on beta02, but now I got this when I create the case:

You can now run './preview run' to get more info on how your case will be run
ERROR: Command: 'git -C /IFS/nwpr/gfs/com132/cases/test_b1850 commit -m "local git repository created" failed with error 'Author identity unknown
*** Please tell me who you are.
Run
git config --global user.email "you@example.com"
git config - -global user.name "Your Name"
to set your account's default identity.
Omit --global to set the identity only in this repository.
fatal: unable to auto-detect email address (got 'com132@h6ln16.(none)')' from dir '/IFS/nwpr/gfs/com132/cases/test_b1850'
ERROR: Command: 'git -C /IFS/nwpr/gfs/com132/cases/test b1850 commit -m " no message provided"' failed with error 'Author identity unknown
*** Please tell me who you are.
Run
git config --global user.email "you@example.com"
git config - -global user.name "Your Name"
to set your account's default identity.

This did not happen in beta02; and since the computing nodes of this system has no internet access, the git operation may fail if it attempts to do so.
Is it possible to disable this check? Can the model still work without it?
Thanks!