WACCM crashing due to "forrtl error"

[lantao]

Hello, I am running CESM1.1.1 (WACCM4) with prescribed sea ice and SST forcing (F_2000_WACCM). "nspltvrm" has been set to be 2 to improve the model stability. However, the model crashes at the beginning of year 5. The error message in cesm.log.140612-200013 shows below right before the crashing (no error message in other log files):

" 15: filew failed, worst i, j, qtmp, q = 1 50
15: -5.526296541889161E-180 -5.353195846995510E-180
62: QNEG3 from TPHYSBCb:m= 64 lat/lchnk= 554 Min. mixing ratio violated at 2 points. Reset to 1.0E+00 Worst = 9.5E-01 at i,k= 1 28
INFO: 0031-251 task 62 exited: rc=-11
19:forrtl: error (78): process killed (SIGTERM)
19:Image PC Routine Line Source
19:libpthread.so.0 00002AD8BF8052A5 Unknown Unknown Unknown
19:libpoe.so 00002AD8C47EEAE2 Unknown Unknown Unknown
19:libpthread.so.0 00002AD8BF7FD851 Unknown Unknown Unknown
19:libc.so.6 00002AD8C1DB190D Unknown Unknown Unknown
19:INFO: 0031-306 pm_atexit: pm_exit_value is 1.
178:forrtl: error (78): process killed (SIGTERM)
178:Image PC Routine Line Source
178:libpthread.so.0 00002B0F381652A5 Unknown Unknown Unknown
178:libpoe.so 00002B0F3D14EAE2 Unknown Unknown Unknown
178:libpthread.so.0 00002B0F3815D851 Unknown Unknown Unknown
178:libc.so.6 00002B0F3A71190D Unknown Unknown Unknown
178:INFO: 0031-306 pm_atexit: pm_exit_value is 1.
...... "

It seems to me that the crashing does not come from either the "filew failing" or "QNEG3" since these warning messages appear very frequently in the model run. But I am not sure what kind of problem it is from the "INFO: 0031-251 task 62 exited: rc=-11" and "forrtl error". I recompile the model and rerun, but the model crashes at the same time.

Could someone help diagnose this ? I will appreciate the help very much.

Best,
Lantao

 

[santos]

This message with "rc=-11" is the one given on Yellowstone for a segmentation fault (out-of-bounds memory access). "forrtl" refers to the Fortran runtime library.I took a look in your run directory and found a core_lite file, which says that this is crashing in the gravity wave module. A crash in this module in CESM 1.1.1 usually is due to an unrealistic wind speed, which may be due to dynamical instability.I have the following recommendations:1) If at all possible, use CESM 1.2.2 instead. CESM 1.1.1 was a short term release, and no longer officially supported. Furthermore, CESM 1.2 is easier to debug.2) It seems that you are running an F compset on the grid f19_g16, with custom SSTs. We don't currently have the resources to support SSTs other than the defaults provided by the AMWG. However, there is a user's guide section and several posts in the CAM forums on this topic, which may be of interest:http://www.cesm.ucar.edu/models/cesm1.2/cesm/doc/usersguide/x2304.htmlhttps://bb.cgd.ucar.edu/how-do-i-use-b-compset-history-output-create-sstice-data-files-drive-f-compsethttps://bb.cgd.ucar.edu/ncl-scripts-create-sstice-data-files-drive-f-compset-using-b-compset-history-files

 

[santos]

This message with "rc=-11" is the one given on Yellowstone for a segmentation fault (out-of-bounds memory access). "forrtl" refers to the Fortran runtime library.I took a look in your run directory and found a core_lite file, which says that this is crashing in the gravity wave module. A crash in this module in CESM 1.1.1 usually is due to an unrealistic wind speed, which may be due to dynamical instability.I have the following recommendations:1) If at all possible, use CESM 1.2.2 instead. CESM 1.1.1 was a short term release, and no longer officially supported. Furthermore, CESM 1.2 is easier to debug.2) It seems that you are running an F compset on the grid f19_g16, with custom SSTs. We don't currently have the resources to support SSTs other than the defaults provided by the AMWG. However, there is a user's guide section and several posts in the CAM forums on this topic, which may be of interest:http://www.cesm.ucar.edu/models/cesm1.2/cesm/doc/usersguide/x2304.htmlhttps://bb.cgd.ucar.edu/how-do-i-use-b-compset-history-output-create-sstice-data-files-drive-f-compsethttps://bb.cgd.ucar.edu/ncl-scripts-create-sstice-data-files-drive-f-compset-using-b-compset-history-files

 

[lantao@ucar_edu]

Hi, Sean, Thank you so much for the valuable explanations and suggestions. We have used custom SSTs with CAM4 and CAM5, both working fine. So I guess the problem might come from the version 1.1.1. Best,Lantao

 

[lantao@ucar_edu]

Hi, Sean, Thank you so much for the valuable explanations and suggestions. We have used custom SSTs with CAM4 and CAM5, both working fine. So I guess the problem might come from the version 1.1.1. Best,Lantao