Previously successful run crashes after 18 years

[James King]

Hi all,

I'm running a coupled atmosphere-ocean compset in CESM2.2.0 on Cheyenne. I have successfully run this compset for 10 years - in the next run of 10 years it crashed at year 18. The error in the cesm.log is:

WHL, oc_tavg_helper is already associated; reset the tavg fields
0: sysmem size=80780.1 MB rss=906.4 MB share=77.2 MB text=79.1 MB datastack=0.0 MB
631:MPT ERROR: Assertion failed at ibdev_multirail.c:4331: "0 <= chan->queued"
631:MPT ERROR: Rank 631(g:631) is aborting with error code 0.
631: Process ID: 3080, Host: r1i5n32, Program: /glade/scratch/jamesking/maxforest_transient_test_02/bld/cesm.exe

There are no error messages in any of the component logs and a search for this error on these forums drew a blank. Any idea what might have gone wrong?

Thanks,

James

 

[dbailey]

This looks like a cheyenne MPI thing. Have you just tried resubmitting?

 

[James King]

Sent it back into the queue - fingers crossed!

 

[samrabin]

I'm experiencing the same error on the same node!

 

[James King]

I'm experiencing the same error on the same node!

Hi Sam,

Did your resubmits work? Mine failed with a clear MPI error:

-1:MPT: Launch error on r12i5n21.ib0.cheyenne.ucar.edu
-1:MPT ERROR: could not run executable.
-1: (HPE MPT 2.21 11/28/19 04:22:45)
MPT: Launch error on r11i1n1.ib0.cheyenne.ucar.edu

 

[samrabin]

Huh. Yes, both my resubmits worked, at least initially. The first resubmit ended up crashing with the same ibdev_multirail.c error. The second resubmit is running now, and it's not on r1i5n32, so I'm hopeful.

 

[James King]

Huh. Yes, both my resubmits worked, at least initially. The first resubmit ended up crashing with the same ibdev_multirail.c error. The second resubmit is running now, and it's not on r1i5n32, so I'm hopeful.

Both my resubmits are now running too, so hopefully this has now been resolved.