CCSM2.0.1 T42 fully coupled continue run crashes

[jane@gi_alaska_edu]

Hi,
I am doing a continue run of CCSM2.0.1 on bluesky using the T42 inputdata. My simulation crashes and gives the error message in stderr file as:

" ERROR: 0031-250 task 0: Segmentation fault
D1: Forcing all remote tasks to exit: rc = -11
D1: Sending SIGTERM to task 67: shutting down task
......
.......
D1: Forcing all remote tasks to exit: rc = -11
D1: Sending SIGTERM to task 67: shutting down task
........
........"

I also find that early in the same file there is message given like:

"D3: Message type 21 from source 62
ocn/ocn: A file or directory in the path name does not exist.
....
...."

This simulation was started in mid Aug, it ran well for eight years, and then I stopped forwarding it in time for about 2 months. When I started submitting the simulation again, I didn't change anything in inputdata or source code. Could you please take a look at me script and help me checking if there is any file missing somewhere?

my main script is at:
/home/bluesky/jane/ccsm2.0.1/scripts/newcase/newcase.run

the stderr file is at:
/home/bluesky/jane/ccsm2.0.1/scripts/newcase/poe.stderr.92414.0

the exedir is
/ptmp/jane/newcase/$MODLE

Thank you!

 

[njn01]

The reason your model run failed is because the ocean-model executable, /ptmp/jane/newcase/ocn/ocn, no longer exists. Most probably, the ocn/ocn file "aged off" the /ptmp directory, which is subject to "scrubbing." In a period of two months, on bluesky, this is not surprising.

What is surprising is that the scripts did not automatically re-compile the ocean model and create a new /ptmp/jane/newcase/ocn/ocn. I've looked through your output, and I'm unsure why a new ocean binary was not generated. I would recommend that you do the following:

cd /ptmp/jane/newcase/ocn
/rm -r obj
llsubmit /home/bluesky/jane/ccsm2.0.1/scripts/newcase/newcase.run

The first two steps will remove all traces of the previous ocean build. During the third step, a new ocean binary *should* be generated, which will eliminate your most recent probelm.

It is also possible that some of your restart files may have aged off /ptmp/jane as well, in which case you'll need to "prestage" them to your /ptmp/jane/newcase directories prior to a successful restart.

ocean-model liaison