Total mass conservation limit exceeded error in iCESM

[littledddna@gmail_com]

Dear all,

I was using iCESM to perform a LGM run with POP restart from iTrace. But for the atmosphere, I was using the iCESM instruction data.
The model crashed with 'Total mass conservation limit exceeded' error in CAM in water tracer module.
Does the error occur due to isotope values?

Here is the error:

after-shallow UW: Warning - Total mass conservation limit exceeded 1
3 1.00000000000000 26.5161440188953
26.3184036928522 7.513386007405492E-003
after-shallow UW: Warning - Total mass conservation limit exceeded 1
5 1.00000000000000 35.9796228611969
35.8202561373570 4.449067120814433E-003

Thanks very much

Xiao

 

[nusbaume]

Hi Xiao,

That error indicates that the tracer water has deviated from the bulk water in two locations. However, the deviations are fairly small, so I wouldn't worry about them unless something looks off in your history file output. You can prevent this error from popping up entirely by adding the line:

wtrc_check_total_h2o = .false.

to your user_nl_cam file. However, my suspicion is that this error is not what killed your model (otherwise it would say "Error" instead of "Warning"). I would thus recommend checking the other log files to see if another error exists somewhere.

Good luck!

Jesse

 

[littledddna@gmail_com]

Hi Xiao,

That error indicates that the tracer water has deviated from the bulk water in two locations. However, the deviations are fairly small, so I wouldn't worry about them unless something looks off in your history file output. You can prevent this error from popping up entirely by adding the line:

wtrc_check_total_h2o = .false.

to your user_nl_cam file. However, my suspicion is that this error is not what killed your model (otherwise it would say "Error" instead of "Warning"). I would thus recommend checking the other log files to see if another error exists somewhere.

Good luck!

Jesse

Thanks very much, Jesse,

The error disappeared by adding that line in CAM user specific file. Now, a new error showed up


ERROR: Isotopic deep-conv precip error!
9.843918322068904E-012 0.000000000000000E+000 0.000000000000000E+000
7 3

isotope tracer: 2
isotot-isotot0 2.254856176885024E+155
fiso_atm-fiso_ocnn 2.254856176885024E+155
isotope tracer: 1

I think the deviation is also very small. Let me try to figure this out...

 

[littledddna@gmail_com]

OK, I think I need to set the precip back to bulk water to avoid the error.
But I am not sure which parameter I should set to FALSE after I read https://github.com/NCAR/iCESM1.2/bl...am/bld/namelist_files/namelist_definition.xml

 

[nusbaume]

Hi Xiao,

The "deep-conv precip" error is sadly also not the reason your model is dying, as that error automatically adjusts the precip to the correct values instead of killing the model (it should have a "warning" label instead of an "error" label, that was my bad).

To find the error that actually killed the model, I would recommend looking at the end of the run log files for each component. In one of the components (including possibly cesm.log.XXX) you should see an error message that is different from any errors that came before it. That is likely the actual error that killed your simulation.

Also, the values right below that deep convection error ("isotot-isotot0") look really bad, and are likely related to whatever is causing your model to die. Those values are coming from the sea ice model. I am not sure what your model set-up is, but it might be worth trying first just a short (~1 month) F-compset to see if that works, and then trying a B-compset that is either a start-up run, or is configured exactly like the run described here:

iCESM1.2 restart file from "The Connected Isotopic Water Cycle in the Community Earth System Model Version 1"

iCESM1.2 restart file at year 1850 Citation: Brady, E. C., Stevenson, S., Bailey, D., Liu, Z., Noone, D., Nusbaumer, J., … Zhu, J. (2019). The Connected Isotopic Water Cycle in the Community Earth System Model Version 1. Journal of Advances in Modeling Earth Systems, 11, 2547–2566...

zenodo.org

and then branched off using those restart files. If all of those simulations work, then it indicates that there is something likely wrong with your isotopic boundary conditions that may be causing the model to blow-up, at least in the sea ice component.

Anyways, I hope that helps, and good luck with your test simulations!

Jesse

 

[Yingying]

Hi Xiao,

The "deep-conv precip" error is sadly also not the reason your model is dying, as that error automatically adjusts the precip to the correct values instead of killing the model (it should have a "warning" label instead of an "error" label, that was my bad).

To find the error that actually killed the model, I would recommend looking at the end of the run log files for each component. In one of the components (including possibly cesm.log.XXX) you should see an error message that is different from any errors that came before it. That is likely the actual error that killed your simulation.

Also, the values right below that deep convection error ("isotot-isotot0") look really bad, and are likely related to whatever is causing your model to die. Those values are coming from the sea ice model. I am not sure what your model set-up is, but it might be worth trying first just a short (~1 month) F-compset to see if that works, and then trying a B-compset that is either a start-up run, or is configured exactly like the run described here:

iCESM1.2 restart file from "The Connected Isotopic Water Cycle in the Community Earth System Model Version 1"

iCESM1.2 restart file at year 1850 Citation: Brady, E. C., Stevenson, S., Bailey, D., Liu, Z., Noone, D., Nusbaumer, J., … Zhu, J. (2019). The Connected Isotopic Water Cycle in the Community Earth System Model Version 1. Journal of Advances in Modeling Earth Systems, 11, 2547–2566...

zenodo.org

and then branched off using those restart files. If all of those simulations work, then it indicates that there is something likely wrong with your isotopic boundary conditions that may be causing the model to blow-up, at least in the sea ice component.

Anyways, I hope that helps, and good luck with your test simulations!

Jesse

Hi Jesst,

Thank you so much for offering the solution! Following this I successfully found where my actual error is. Appreciate it !

Best,
Yingying