crash in px_pgin

[klaus_wyser@smhi_se]

I am running CCSM in a IPCC SRES style experiment with T42 resolution. It all works fine from 1870 to 2027 (that is more than 150 years), and then suddenly it crashes (reproducable!) with the cryptical message:

cam-24(mpi:1249@n80): posixio.c:210: px_pgin: Assertion `*posp == ((off_t)(-1)) || *posp == lseek(nciop->fd, 0, 1)' failed.
forrtl: error (76): IOT trap signal
--- mpimon --- Aborting run after process-24 terminated abnormally Childprocess 25906 got signal SIGIOT( 6): IOT instruction ---

A quick google search gave a hint that this might be related to large (> 2Gb) netCDF files. Has anyone seen this error before when running CCSM? Any solution?

Klaus

 

[klaus_wyser@smhi_se]

The problem is gone...

The problem with running CCSM occurred during the weekend and I could reproduce the failure. A colleague also saw some strange effects with the output from a different model on the same machine. I set up a new experiment on Monday, and strangely enough everything worked fine. I resumed the old run, and it would proceed without any problem. No idea where the problem was, a glitch in the file system maybe?

Thanks anyway all those who tried to help.

Klaus