Problem with running CAM in restart mode

[bnarapus@gmu_edu]

Hi all,

Model runs well as initial run, but When I restart the model, it crashes with the following message:
#----------------------------------------------------------
DYCORE is EUL
scon set to fixed value of 1367000.00000000000
ERROR: 0031-250 task 6: Illegal instruction
ERROR: 0031-250 task 0: Illegal instruction
ERROR: 0031-250 task 2: Illegal instruction
ERROR: 0031-250 task 4: Illegal instruction
ERROR: 0031-250 task 5: Illegal instruction
ERROR: 0031-250 task 1: Illegal instruction
ERROR: 0031-250 task 3: Illegal instruction
ERROR: 0031-250 task 7: Illegal instruction
ATTENTION: 0031-386 MP_INSTANCES setting ignored when LoadLeveler is not being used.
#----------------------------------------------------------

However, if I keep submitting the job, the model runs after few attempts.
This is only happening from a week. I am running bluefire machine. Is it a bug? or Am I missing something.
(I saw a thread which says that there is a compiler bug on bluefire, and that can be avoided by using userlist variable set inithist='none'. I did that in the run script as well as setting inithist='NONE' in "/cam1/models/atm/cam/src/control/history.F90" file.)

Thank you very much,
Bala

 

[rneale]

The known bug you mention happens on the 1st day of January each year. Is that true for you run?

 

[bnarapus@gmu_edu]

No. It happens every time I do the restrart run. As I mentioned earlier, if i keep submitting my job for few times (sometimes works in the first attempt), the job will run eventually and the simulations are all fine.