esm_init_pelayout failure "Component MED has mpi task mismatch"

[glemieux]

I'm trying to switch running ctsm from MCT to NUOPC (by removing the --driver=mct option, but I'm running into errors with the ctsm initialization. A fairly generic case will build successfully, but immediately crash upon run submission. Attached are the various bld and run logs.

Given the location of the crash, I'm assuming this is likely due to dependencies (mpi, hdf5, netcdf, and/or esmf) not being installed correctly for parallel operation. As such I've uploaded the config log output for the hdf5 and netcdf builds for review. Unfortunately I don't have the logs for the esmf gmake info setup, but it looks I think in the case I was using the Linux.gfortran.default build_conf (I have a support email into esmf_support@ucar.edu about a different build failure in pursuit of running with NUOPC). Openmpi was installed as pre-built binary via apt and is 4.0.3-0ubuntu1. This is running on an ubuntu-based distro workstation with 2 16 core xeon nodes.

git describe: ctsm5.1.dev084
./manage_externals/checkout_externals --status --verbose:

Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : Externals_CDEPS.cfg
Checking status of externals: clm, fates, cism, source_cism, rtm, mosart, mizuroute, ccs_config, cime, cmeps, cdeps, fox, genf90, cpl7, share, mct, parallelio, doc-builder,
    ./ccs_config
        clean sandbox, on ccs_config_cesm0.0.5
    ./cime
        clean sandbox, on cime6.0.13
    ./components/cdeps
        clean sandbox, on cdeps0.12.35
    ./components/cdeps/fox
        clean sandbox, on 4.1.2.1
    ./components/cdeps/share/genf90
        clean sandbox, on genf90_200608
    ./components/cism
        clean sandbox, on cismwrap_2_1_95
    ./components/cism/source_cism
        clean sandbox, on cism_main_2.01.011
    ./components/cmeps
        clean sandbox, on cmeps0.13.47
    ./components/cpl7
        clean sandbox, on cpl7.0.12
    ./components/mizuRoute
        clean sandbox, on 34723c2e4df7caa16812770f8d53ebc83fa22360
    ./components/mosart
        clean sandbox, on mosart1_0_45
    ./components/rtm
        clean sandbox, on rtm1_0_78
e-o ./doc/doc-builder
        -, not checked out --> v1.0.8
    ./libraries/mct
        clean sandbox, on MCT_2.11.0
    ./libraries/parallelio
        clean sandbox, on pio2_5_5
    ./share
        clean sandbox, on share1.0.10
    ./src/fates
        clean sandbox, on sci.1.55.4_api.22.1.0

 

[jedwards]

It looks like the number of mpi tasks in your mpirun argument does not match the number of tasks that were specified to the model.
If you run ./pelayout in your case and the ./preview_run script do the total number of pes match?