Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

CCSM2.0.1 T42 fully coupled continue run crashes

J

jane@gi_alaska_edu

New Member
Hi,
I am doing a continue run of CCSM2.0.1 on bluesky using the T42 inputdata. My simulation crashes and gives the error message in stderr file as:

" ERROR: 0031-250 task 0: Segmentation fault
D1: Forcing all remote tasks to exit: rc = -11
D1: Sending SIGTERM to task 67: shutting down task
......
.......
D1: Forcing all remote tasks to exit: rc = -11
D1: Sending SIGTERM to task 67: shutting down task
........
........"

I also find that early in the same file there is message given like:

"D3: Message type 21 from source 62
ocn/ocn: A file or directory in the path name does not exist.
....
...."

This simulation was started in mid Aug, it ran well for eight years, and then I stopped forwarding it in time for about 2 months. When I started submitting the simulation again, I didn't change anything in inputdata or source code. Could you please take a look at me script and help me checking if there is any file missing somewhere?

my main script is at:
/home/bluesky/jane/ccsm2.0.1/scripts/newcase/newcase.run

the stderr file is at:
/home/bluesky/jane/ccsm2.0.1/scripts/newcase/poe.stderr.92414.0

the exedir is
/ptmp/jane/newcase/$MODLE

Thank you!
 
N

njn01

Member
The reason your model run failed is because the ocean-model executable, /ptmp/jane/newcase/ocn/ocn, no longer exists. Most probably, the ocn/ocn file "aged off" the /ptmp directory, which is subject to "scrubbing." In a period of two months, on bluesky, this is not surprising.

What is surprising is that the scripts did not automatically re-compile the ocean model and create a new /ptmp/jane/newcase/ocn/ocn. I've looked through your output, and I'm unsure why a new ocean binary was not generated. I would recommend that you do the following:

cd /ptmp/jane/newcase/ocn
/rm -r obj
llsubmit /home/bluesky/jane/ccsm2.0.1/scripts/newcase/newcase.run

The first two steps will remove all traces of the previous ocean build. During the third step, a new ocean binary *should* be generated, which will eliminate your most recent probelm.

It is also possible that some of your restart files may have aged off /ptmp/jane as well, in which case you'll need to "prestage" them to your /ptmp/jane/newcase directories prior to a successful restart.

ocean-model liaison
 
You must log in or register to reply here.
Top