CESM1_2_2_1 runtime fail if change the pes

[xlong@ncsu_edu]

I run the CESM1_2_2_1 in f19_g16 configuration.The default requested # of cores are 64.If I set this numer as 64, 128 or 256, the model runs fine. However, if I set it to 100, for example, the model fails and the error is something like: MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:aa area ->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:aa aream->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:ll area ->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:ll aream->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:rr area ->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:aa area ->MCT::m_AttrVect::indexRA_rr aream->MCT::m_AttrVect::indexRA_aa area ->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:aa aream->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:aa area ->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:aa aream->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:aa aream->MCT::m_AttrVect::indexRA_ MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:
a area ->MCT::m_AttrVect::indexRA_rr aream->MCT::m_AttrVect::indexRA_aa area ->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:aa aream->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:aa area ->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:aa aream->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:aa aream->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:ll area ->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:ll aream->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:rr area ->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:rr aream->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:ll area ->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:ll aream->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:rr area ->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:rr aream->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:ll area ->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:ll aream->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "area" Traceback:rr area ->MCT::m_AttrVect::indexRA_MCT::m_AttrVect::indexRA_:: ERROR--attribute not found: "aream" Traceback:rr aream->MCT::m_AttrVect::indexRA_MCT::m_Router::initp_: GSMap indices not increasing...Will correctMCT::m_Router::initp_: RGSMap indices not increasing...Will correctMCT::m_Router::initp_: RGSMap indices not increasing...Will correctMCT::m_Router::initp_: GSMap indices not increasing...Will correctMCT::m_Rearranger::Rearrange_: Number of attributes in SourceAV and TargetAV do not matchMCT::m_Rearranger::Rearrange_: error, nRAttr(SourceAV)=1, nRAttr(TargetAV)=6.063.MCT(MPEU)::die.: from MCT::m_Rearranger::Rearrange_()application called MPI_Abort(MPI_COMM_WORLD, 2) - process 99In: PMI_Abort(2, application called MPI_Abort(MPI_COMM_WORLD, 2) - process 99)MCT::m_Rearranger::Rearrange_: Number of attributes in SourceAV and TargetAV do not matchMCT::m_Rearranger::Rearrange_: Number of attributes in SourceAV and TargetAV do not matchMCT::m_Rearranger::Rearrange_: error, nRAttr(SourceAV)=1, nRAttr(TargetAV)=6.061.MCT(MPEU)::die.: from MCT::m_Rearranger::Rearrange_()MCT::m_Rearranger::Rearrange_: Number of attributes in SourceAV and TargetAV do not matchMCT::m_Rearranger::Rearrange_: error, nRAttr(SourceAV)=1, nRAttr(TargetAV)=6.062.MCT(MPEU)::die.: from MCT::m_Rearranger::Rearrange_()
application called MPI_Abort(MPI_COMM_WORLD, 2) - process 96application called MPI_Abort(MPI_COMM_WORLD, 2) - process 97In: PMI_Abort(2, application called MPI_Abort(MPI_COMM_WORLD, 2) - process 97)In: PMI_Abort(2, application called MPI_Abort(MPI_COMM_WORLD, 2) - process 96)application called MPI_Abort(MPI_COMM_WORLD, 2) - process 98In: PMI_Abort(2, application called MPI_Abort(MPI_COMM_WORLD, 2) - process 98)slurmstepd: error: *** STEP 9924.0 ON node-0086 CANCELLED AT 2019-05-20T23:52:49 ***srun: Job step aborted: Waiting up to 122 seconds for job step to finish.srun: error: node-0090: tasks 80-96,99: Killedsrun: Terminating job step 9924.0srun: error: node-0088: tasks 40-59: Killedsrun: error: node-0090: tasks 97-98: Exited with exit code 2srun: error: node-0087: tasks 20-39: Killedsrun: error: node-0089: tasks 60-79: Killed srun: error: node-0086: tasks 0-19: Killed
 

 

[xlong@ncsu_edu]

I out debug option on, there is more information about the error.forrtl: severe (408): fort: (3): Subscript #1 of the array RATTR has value 0 which is less than the lower bound of 1 Image              PC                Routine            Line        Sourcecesm.exe           00000000069F0110  Unknown               Unknown  Unknowncesm.exe           0000000000436079  ccsm_comp_mod_mp_        1428  ccsm_comp_mod.F90cesm.exe           00000000004C10F5  MAIN__                     90  ccsm_driver.F90cesm.exe           000000000041392E  Unknown               Unknown  Unknownlibc-2.17.so       00002B9694BDE3D5  __libc_start_main     Unknown  Unknowncesm.exe           0000000000413829  Unknown               Unknown  Unknown forrtl: severe (408): fort: (3): Subscript #1 of the array RATTR has value 0 which is less than the lower bound of 1
I have been running cesm1_2_2_1 on our older hpc system, and never had any issue.This issue occur when I tried to port the model a new system.
Thanks

 

[qiwen zhou]

The same problem happened when I run clm5. Do you have a solution?