Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

run time issue in AMIP type run with soil moisture climatology

S

sanjiv_kumar@noaa_gov

New Member
Hello:I am running an AMIP run with specifeid soil mosituer climatology; model runs file for ~20-years then it stops. Below are some relevant error message from cesm.log file.Case Direcotry is: /glade/p/uola0001/DL_AMIP_OBS/DL_AMIP_OBS_e023run direcotry is: /glade/scratch/kumar34/DL_AMIP_OBS_e023/run /************************************190: QNEG3 from vertical diffusion/SO2:m=  8 lat/lchnk=   3283 Min. mixing ratio violated at    1 points.  Reset to  1.0E-36 Worst =-1.1E-12 at i,k=   5 30  18: imp_sol: Time step   1.8000000000000E+03 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Time step   9.0000000000000E+02 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Time step   4.5000000000000E+02 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Time step   2.2500000000000E+02 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Time step   1.1250000000000E+02 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Time step   1.1250000000000E+01 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Failed to converge @ (lchnk,lev,col,nstep,dt,time) =    535     5     7329240  1.1250000000000E+01  1.1250000000000E+01  18: DMS       0.000E+00  18: imp_sol: Time step   3.6000000000000E+02 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Failed to converge @ (lchnk,lev,col,nstep,dt,time) =    535     5     7329240  3.6000000000000E+02  1.0687500000000E+03  18: H2O2      1.000E+00  18: imp_sol: Time step   3.6000000000000E+02 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Failed to converge @ (lchnk,lev,col,nstep,dt,time) =    535     5     7329240  3.6000000000000E+02  1.7887500000000E+03  18: H2O2      1.000E+00  18: imp_sol: Time step   1.1250000000000E+01 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Failed to converge @ (lchnk,lev,col,nstep,dt,time) =    535     5     7329240  1.1250000000000E+01  1.8000000000000E+03  18: H2O2      0.000E+00  18: imp_sol : @ (lchnk,lev,col) =          535           5           7  failed   18:           9  times 216:  filew failed, worst i, j, qtmp, q =            1         121 216: -6.145432902153057E-012  0.000000000000000E+000 216:  filew failed, worst i, j, qtmp, q =            1         122 216: -8.552798841086867E-014  0.000000000000000E+000 181: dpcoup cant adjust           3        3145          12 -4.258287487216322E-015 181:  0.000000000000000E+000  3.633496165789148E-015  41: dpcoup cant adjust           3         897          12 -5.913762905605607E-017  41:  0.000000000000000E+000  5.043544824274600E-017INFO: 0031-251  task 75 exited: rc=-11  59:forrtl: error (78): process killed (SIGTERM)  59:  59:Stack trace terminated abnormally.  59:*** stack smashing detected ***: /glade/scratch/kumar34/DL_AMIP_OBS_e023/bld/cesm.exe terminated/**********************************I am not sure why the above run is failing. While another similar run with case direcotry /glade/p/uola0001/DL_AMIP_OBS/DL_AMIP_OBS_e024 is running fine for the same year (at least as of now).Any help is highly appreciated. -SK 
 
S

sanjiv_kumar@noaa_gov

New Member
OK.. I was able to overcome this issue.This is how: "I reduced the STOP_N = 8 (months), and increased the wall time to 8 hours"The model is running now. From this solution it seems to be an wall time issue. However, i could not see it any where in the LSF email that it was a wall time issue; because it was showing run time as: Run time :  12941 sec. that is ~ 4 hours which is way below i specified the wall time as 6 hours. May be this problem is for the software engineers to look into. 
-SK     
 
You must log in or register to reply here.
Top