Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

WACCM crashed due to "DEPTH exceeding table value"

X

xiaokonglong666@gmail_com

New Member
Hello,I'm trying to run CESM1.2.2 with F1850W compset. And I want to increase the SST by 30 degrees centigrade by changing the SST_cpl and ice_cov data in "sst_HadOIBl_bc_1.9x2.5_clim_pi_c101028.nc".But when I want to run this model, it crashes with the following error(see log_file attached for more imformation):  DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value DEPTH exceeding table value  findsp not converging at i =           11  t, q, p    236.970523240216       3.735770877134584E-002   6152.05000000000       tsp, qsp    263.787360155026       2.971450329655875E-002(shr_sys_abort) ERROR: wv_saturation::FINDSP -- not converging(shr_sys_abort) WARNING: calling shr_mpi_abort() and stoppingapplication called MPI_Abort(MPI_COMM_WORLD, 1001) - process 37yhrun: error: cn7068: task 37: Exited with exit code 233yhrun: First task exited 60s agoyhrun: tasks 0-36,38-119: runningyhrun: task 37: exited abnormallyyhrun: Terminating job step 4588727.0yhrun: Job step aborted: Waiting up to 2 seconds for job step to finish.slurmd[cn7067]: *** STEP 4588727.0 KILLED AT 2018-01-17T12:27:46 WITH SIGNAL 9 ***slurmd[cn7067]: *** STEP 4588727.0 KILLED AT 2018-01-17T12:27:46 WITH SIGNAL 9 ***yhrun: error: cn7067: task 22: KilledCan anyone tell me why the model crashed with the error message "DEPTH exceeding table value", and I also run a model with SST increased by 10 degrees centigrade and it runs fine.

Any help is highly appreciated.  
 
mmills

mmills

CSEG and Liaisons
Staff member
The final error that appears to be crasing the model is at the end of the log file:

(shr_sys_abort) ERROR: wv_saturation::FINDSP -- not converging
The subroutine that is crashing is here: models/atm/cam/src/physics/cam/wv_saturation.F90 It seems likely that the very large surface temperatures you are forcing are causing the water vapor saturation routine not to converge. This may be a transient issue that you could get around by reducing the model time step.
 
mmills

mmills

CSEG and Liaisons
Staff member
The final error that appears to be crasing the model is at the end of the log file:

(shr_sys_abort) ERROR: wv_saturation::FINDSP -- not converging
The subroutine that is crashing is here: models/atm/cam/src/physics/cam/wv_saturation.F90 It seems likely that the very large surface temperatures you are forcing are causing the water vapor saturation routine not to converge. This may be a transient issue that you could get around by reducing the model time step.
 
B

bardeenc

New Member
Hi Shoazhi, I haven't looked in your exact code branch, but have some suggestions for you.1) I am not sure about the "DEPTH" error massage. I don't see where it is generated in the code (at least looking at the latest branch), but I suspect it is coming from the estbl lookup table used to calculate water saturation. I suspect you are going past the ends of this table because of your extreme conditions.2) If it wasn't for your extreme conditions and what I think are table lookup errors, you could try increasing the number of iterations to alow it to try to converge. However, once you hit the edge of the table, I don't think you will be able to converge.3) The fundamental problem seems like that your model state has gotten very wet. It looks like it failed at 61 hPa and a temperature of 263K with a specific humidity of 3.7e-2 kg/kg. This is a very large value for the stratosphere and for this temperature. Do you see the model state getting wetter and wetter?How abruptly did you change the surface temperature? If you did it quickly, you might try introducing the change more slowly to allow the model state to adjust. If the stratosphere is getting so much moisture that it is above saturation, you may need to increase the altitude at which cloud microphysics is applied. This is controlled by a namelist variable that I think is set to 1 hPa in your version. You might try setting it to a lower value (beyond where you see the relative humidity exceeding 0.8), to allow clouds and precipitation to remove some of the excess humidity. The varaible you should be looking for is trop_cloud_top_press. and it is set in Pa. So you might try adding the following to your user_nl_cam:trop_cloud_top_press = 1.d-4
 
B

bardeenc

New Member
Hi Shoazhi, I haven't looked in your exact code branch, but have some suggestions for you.1) I am not sure about the "DEPTH" error massage. I don't see where it is generated in the code (at least looking at the latest branch), but I suspect it is coming from the estbl lookup table used to calculate water saturation. I suspect you are going past the ends of this table because of your extreme conditions.2) If it wasn't for your extreme conditions and what I think are table lookup errors, you could try increasing the number of iterations to alow it to try to converge. However, once you hit the edge of the table, I don't think you will be able to converge.3) The fundamental problem seems like that your model state has gotten very wet. It looks like it failed at 61 hPa and a temperature of 263K with a specific humidity of 3.7e-2 kg/kg. This is a very large value for the stratosphere and for this temperature. Do you see the model state getting wetter and wetter?How abruptly did you change the surface temperature? If you did it quickly, you might try introducing the change more slowly to allow the model state to adjust. If the stratosphere is getting so much moisture that it is above saturation, you may need to increase the altitude at which cloud microphysics is applied. This is controlled by a namelist variable that I think is set to 1 hPa in your version. You might try setting it to a lower value (beyond where you see the relative humidity exceeding 0.8), to allow clouds and precipitation to remove some of the excess humidity. The varaible you should be looking for is trop_cloud_top_press. and it is set in Pa. So you might try adding the following to your user_nl_cam:trop_cloud_top_press = 1.d-4
 
You must log in or register to reply here.
Top