pft water state variables average error in iCESM

[littledddna@gmail_com]

Dear all,

I was trying to setup a LGM run with iCESM but the model fails at the very first step in CLM.
There was no specific error in the log file, just a general error floating invalid when CLM was calculating averaged pft water state variables for canopy water.
Seems the model crashed at the following lines:

! Note: some glacier_mec pfts may have zero weight
if (wtgcell(p) > 0._r8 .or. ltype(l)==istice_mec) colarr(c,j) = colarr(c,j) + pftarr(p,j) * wtcol(p)
and
call p2c(pwtrc, num_nolakec, filter_nolakec, pws%RCanopyWater, pws_a%RCanopyWater)

So, should I check the restart file for each pfts? Is it possible to print out the lat/long of error grid ?

Thanks very much

Xiao

 

[nusbaume]

Hi Xiao,

Before diving into the numbers, it would be good to make sure you are running with debug flags off (i.e. the variable DEBUG in env_build.xml is set to FALSE). If not then you'll get spurious floating point errors in iCLM4, which could be what you are running into.

Otherwise, I agree with your suggestion about checking the input (restart) file isotope values. In particular, I would check the variables named Rcanopy_XXX, where XXX is the name of the isotope species (H2OTR, HDO, or H218O). As a sanity check, it might also be worth trying a start-up simulation for iCESM1.2 and following the included instructions for creating your own isotopic finidat variable. Please note that after you build the model you can look in lnd_in in your run directory for the file the model wants to use, and simply use the NCL script to add water isotopes to it.

If that start-up run works as expected, then it likely indicates that the problem is with your CLM4 input files.

Hope that helps, and good luck!

Jesse

 

[littledddna@gmail_com]

Hi Xiao,

Before diving into the numbers, it would be good to make sure you are running with debug flags off (i.e. the variable DEBUG in env_build.xml is set to FALSE). If not then you'll get spurious floating point errors in iCLM4, which could be what you are running into.

Otherwise, I agree with your suggestion about checking the input (restart) file isotope values. In particular, I would check the variables named Rcanopy_XXX, where XXX is the name of the isotope species (H2OTR, HDO, or H218O). As a sanity check, it might also be worth trying a start-up simulation for iCESM1.2 and following the included instructions for creating your own isotopic finidat variable. Please note that after you build the model you can look in lnd_in in your run directory for the file the model wants to use, and simply use the NCL script to add water isotopes to it.

If that start-up run works as expected, then it likely indicates that the problem is with your CLM4 input files.

Hope that helps, and good luck!

Jesse

Thanks very much, Jesse,

After set DEBUG to FALSE, the model started to running. But the main log file shows the following errors with crashing: e.g.


wtrc q1q2 uqdiff error here! 1.010941327861600E-009 1.571008947931496E-016
ERROR: Isotopic deep-conv precip error! 5.574210046598391E-011
2.682776950452851E-011 0.000000000000000E+000 0.000000000000000E+000
and massive
QNEG4 WARNING from TPHYSAC Max possible LH flx exceeded at 1 points. , Worst excess = -8.7514E-08, lchnk = 82, i = 8, same as indices lat = 37, lon = 82
BalanceCheck: soil balance error nstep = 553 point = 2264 imbalance = -0.000001 W/m2
QNEG3 from pcwdetrain/HDOI:m= 26 lat/lchnk= 357 Min. mixing ratio violated at 1 points. Reset to 0.0E+00 Worst =-1.0E-08 at i,k= 9 18
QNEG3 from macrop/HDOI:m= 26 lat/lchnk= 357 Min. mixing ratio violated at 18 points. Reset to 0.0E+00 Worst =-1.0E-08 at i,k= 9 18

I thought these errors are due to CAM chemistry, so I set -chem none in env_build and i applied default bulk aerosol. But the errors are still there. Lets see how long can the model run survive...

Xiao

 

[littledddna@gmail_com]

And by the way, by turning off chemistry, the model speed doubles...interesting....

 

[nusbaume]

Hi Xiao,

All of those error messages are actually warnings that shouldn't kill the model (even the one that says ERROR: Isotopic deep-conv... is actually a warning, and should obviously be re-named). If the model dies again would it be possible to attach your cesm.log.XXX file (or at least the last few hundred lines of the file)? That log will hopefully point to the actual cause of the model crash.

Otherwise, if turning off chemistry works for you and the model runs to completion then that is great to hear!

Thanks, and have a good weekend!

Jesse

 

[littledddna@gmail_com]

Hi Xiao,

All of those error messages are actually warnings that shouldn't kill the model (even the one that says ERROR: Isotopic deep-conv... is actually a warning, and should obviously be re-named). If the model dies again would it be possible to attach your cesm.log.XXX file (or at least the last few hundred lines of the file)? That log will hopefully point to the actual cause of the model crash.

Otherwise, if turning off chemistry works for you and the model runs to completion then that is great to hear!

Thanks, and have a good weekend!

Jesse

Dear Jesse,

The model crashed with the ice vertical thermo error as indicated below and some grids has surface temp lower than -100 degree.
I already set ATM coupling frequency to 144 and changed xndt_dyn = 6 in ICE model and perturbed gamma_coeff as it was discussed in FAQ: ice thermo error. (FAQ: CICE Thermodynamic convergence errors.)

But the model still does not work. Any further suggestions?

Thanks very much

Xiao



istep1: 2828 idate: 110120 sec: 55200
Starting thermo, Tsn < Tmin
Tsn= -100.022290691646
Tmin= -100.000000000000
istep1, my_task, i, j: 2828 0 5 4
qsn -180006484.486192
istep1, my_task, iblk = 2828 0 4
Global block: 227
Global i and j: 64 91
Lat, Lon: 76.1025183375770 -147.078375069603
(shr_sys_abort) ERROR: ice: Vertical thermo error
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping

 

[nusbaume]

Hi Xiao,

Apologies for the slow response (again). Do you still need help with this issue? Sadly I am not an expert when it comes to the sea ice model. Maybe @jiangzhu can help? To summarize Xiao is using iCESM1.2, but running into the sea ice error listed above.

Thanks, and have a good weekend!

Jesse

 

[jiangzhu]

Hi Jesse and Xiao,

If possible, can you clarify whether the problem is in water isotope or in CESM? This can be tested by running a parallel simulation without isotopes from a regular CESM1.2 code base (e.g. CESM1_2_2_1). I think you want to carefully examine your initial condition files.

Best,
Jiang