When I submited one of the B1850 experiments, there was a special error. The solution in userguide is "Check the binary file for errors and resolution. The POP2 model may be reading bad data from the binary grid or topography file".
Error in cesm logs:
POP aborting...
ERROR: ANGLE is outside its expected range
------------------------------------------------------------------------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 102 in communicator MPI_COMM_WORLD
with errorcode 0.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
------------------------------------------------------------------------
POP aborting...
ERROR: ANGLE is outside its expected range
------------------------------------------------------------------------
--------------------------------------------------------------------------
Error in cesm logs:
POP aborting...
ERROR: ANGLE is outside its expected range
------------------------------------------------------------------------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 102 in communicator MPI_COMM_WORLD
with errorcode 0.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
------------------------------------------------------------------------
POP aborting...
ERROR: ANGLE is outside its expected range
------------------------------------------------------------------------
--------------------------------------------------------------------------