"MPT ERROR: invalid option -p" when submitting CESM2.0.0 jobs on cheyenne

[hyogase]

I am trying to run CESM 2.0.0 simulations on Cheyenne.
The running code is like this.
qcmd -- ./case.build
./case.submit
The script compiles well and can be submitted. However, I got ERROR in CaseStatus:
ERROR: RUN FAIL: Command 'mpiexec_mpt -np 72 -p "%g:" omplace -tm open64 /glade/scratch/***/cesm2_0_0_F_test/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed
And the Error in the log file is like this:
"MPT ERROR: invalid option -p
(HPE MPT 2.19 02/23/19 05:31:12)"

How can I solve this and get cesm2.0.0 simulation running on cheyenne?

Thank you very much.

 

[hyogase]

I just download the cesm2.0.0 by check out 'release-cesm2.0.0' and 'cesm2.0.0'. Even I use the default configuration, the error remains in both version of cesm2.0.0 on cheyenne.

 

[mlevy]

CESM 2.0 is no longer supported, so the official recommendation would be to switch to CESM 2.1. However, I'll also point out that there is a cesm2.0.1 release; I'm not sure if the MPT error you are seeing, but I would try cesm2.0.1 and if it doesn't work move to cesm2.1.3

 

[hyogase]

Thank you for the information. I have to reproduce some previous research from CESM 2.0.0. I tried release cesm 2.0.1. The same error occurred. Do you have any suggestions about how to solve this problem (MPT ERROR)?

 

[mlevy]

I think what is happening is that MPT 2.16 (the version used by CESM 2.0) is no longer available on cheyenne, but module load mpt/2.16 is actually loading MPT 2.19 instead and that is causing some weird side effects. There is a wiki page outlining the steps to take, but following that directly will make the following changes:

1. use ESMF 7.1.0 instead of 7.0.0
1. use netcdf 4.6.1 instead of 4.5.0
1. use pnetcdf 1.11.0 instead of 1.9.0

If you'd rather stay as close to CESM 2.0 as possible, you could manually edit the existing env_mach_specific.xml to explicitly update to MPT 2.19 and remove the -p option:

@@ line 57 @@
       <command name="load">netcdf-mpi/4.6.1</command>
     </modules>
     <modules compiler="intel" mpilib="mpt">
-      <command name="load">mpt/2.16</command>
+      <command name="load">mpt/2.19</command>
       <command name="load">netcdf-mpi/4.6.1</command>
       <command name="load">pnetcdf/1.11.0</command>
     </modules>
@@ line 101 @@
     <executable>mpiexec_mpt</executable>
     <arguments>
       <arg name="anum_tasks"> -np {{ total_tasks }}</arg>
-      <arg name="labelstdout">-p "%g:"</arg>
       <arg name="zthreadplacement"> omplace -tm open64 </arg>
     </arguments>
   </mpirun>

and then run

$ ./case.build --clean-all
$ ./case.setup --reset
$ qcmd -- ./case.build

Lastly, check to make sure the changes stuck by looking for mpt in .env_mach_specific.sh:

$ grep mpt .env_mach_specific.sh
module load ncarenv/1.2 intel/17.0.1 esmf_libs mkl esmf-7.0.0-defio-mpi-O mpt/2.16 netcdf-mpi/4.5.0 pnetcdf/1.9.0 ncarcompilers/0.4.1

If it worked, the mpt/2.16 line should change to mpt/2.19. You can make these changes apply to future cases by editing $CESM/cime/config/cesm/machines/config_machines.xml; search for "cheyenne" and then look for the lines similar to the ones you edited above.