run time issue in AMIP type run with soil moisture climatology

[sanjiv_kumar@noaa_gov]

Hello:I am running an AMIP run with specifeid soil mosituer climatology; model runs file for ~20-years then it stops. Below are some relevant error message from cesm.log file.Case Direcotry is: /glade/p/uola0001/DL_AMIP_OBS/DL_AMIP_OBS_e023run direcotry is: /glade/scratch/kumar34/DL_AMIP_OBS_e023/run /************************************190: QNEG3 from vertical diffusion/SO2:m=  8 lat/lchnk=   3283 Min. mixing ratio violated at    1 points.  Reset to  1.0E-36 Worst =-1.1E-12 at i,k=   5 30  18: imp_sol: Time step   1.8000000000000E+03 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Time step   9.0000000000000E+02 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Time step   4.5000000000000E+02 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Time step   2.2500000000000E+02 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Time step   1.1250000000000E+02 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Time step   1.1250000000000E+01 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Failed to converge @ (lchnk,lev,col,nstep,dt,time) =    535     5     7329240  1.1250000000000E+01  1.1250000000000E+01  18: DMS       0.000E+00  18: imp_sol: Time step   3.6000000000000E+02 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Failed to converge @ (lchnk,lev,col,nstep,dt,time) =    535     5     7329240  3.6000000000000E+02  1.0687500000000E+03  18: H2O2      1.000E+00  18: imp_sol: Time step   3.6000000000000E+02 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Failed to converge @ (lchnk,lev,col,nstep,dt,time) =    535     5     7329240  3.6000000000000E+02  1.7887500000000E+03  18: H2O2      1.000E+00  18: imp_sol: Time step   1.1250000000000E+01 failed to converge @ (lchnk,lev,col,nstep) =    535     5     7329240  18: imp_sol: Failed to converge @ (lchnk,lev,col,nstep,dt,time) =    535     5     7329240  1.1250000000000E+01  1.8000000000000E+03  18: H2O2      0.000E+00  18: imp_sol : @ (lchnk,lev,col) =          535           5           7  failed   18:           9  times 216:  filew failed, worst i, j, qtmp, q =            1         121 216: -6.145432902153057E-012  0.000000000000000E+000 216:  filew failed, worst i, j, qtmp, q =            1         122 216: -8.552798841086867E-014  0.000000000000000E+000 181: dpcoup cant adjust           3        3145          12 -4.258287487216322E-015 181:  0.000000000000000E+000  3.633496165789148E-015  41: dpcoup cant adjust           3         897          12 -5.913762905605607E-017  41:  0.000000000000000E+000  5.043544824274600E-017INFO: 0031-251  task 75 exited: rc=-11  59:forrtl: error (78): process killed (SIGTERM)  59:  59:Stack trace terminated abnormally.  59:*** stack smashing detected ***: /glade/scratch/kumar34/DL_AMIP_OBS_e023/bld/cesm.exe terminated/**********************************I am not sure why the above run is failing. While another similar run with case direcotry /glade/p/uola0001/DL_AMIP_OBS/DL_AMIP_OBS_e024 is running fine for the same year (at least as of now).Any help is highly appreciated. -SK 

 

[sanjiv_kumar@noaa_gov]

OK.. I was able to overcome this issue.This is how: "I reduced the STOP_N = 8 (months), and increased the wall time to 8 hours"The model is running now. From this solution it seems to be an wall time issue. However, i could not see it any where in the LSF email that it was a wall time issue; because it was showing run time as: Run time :  12941 sec. that is ~ 4 hours which is way below i specified the wall time as 6 hours. May be this problem is for the software engineers to look into. 
-SK