MPT ERROR: Rank 52(g:52) is aborting with error code -1

[adamhb]

Hi there,

I am running a regional simulation with CLM-FATES on Cheyenne. After 152 years the model crashes with the following error message written to the cesm log:

MPT ERROR: Rank 52(g:52) is aborting with error code -1.

Typically the cesm log gives a little more information about where the error comes from, but in this case it seems to be coming from outside CLM itself. Any information about where to start to look to try to diagnose / fix this error would be greatly appreciated.

I have attached a bash script that was used to set up this case in case it is helpful.

Adam

 

[jedwards]

First - when you report a problem on an NCAR system please include the path to the case directory. I searched around
and found it but your script was of little value. The model failed with an IO error. There were many hardware failures on cheyenne
over the long weekend and we do not think that they will be repaired before the system is retired at the end of December.

If you repeat this error on derecho I would be happy to look into it further, but at this point we should put it down as a system issue.

 

[adamhb]

Ok thank you jedwards for the insights. I will be sure to provide a case directory next time! Hopefully this will work on Derecho.
Adam