Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Total mass conservation limit exceeded error in iCESM

Dear all,

I was using iCESM to perform a LGM run with POP restart from iTrace. But for the atmosphere, I was using the iCESM instruction data.
The model crashed with 'Total mass conservation limit exceeded' error in CAM in water tracer module.
Does the error occur due to isotope values?

Here is the error:

after-shallow UW: Warning - Total mass conservation limit exceeded 1
3 1.00000000000000 26.5161440188953
26.3184036928522 7.513386007405492E-003
after-shallow UW: Warning - Total mass conservation limit exceeded 1
5 1.00000000000000 35.9796228611969
35.8202561373570 4.449067120814433E-003


Thanks very much

Xiao
 

nusbaume

Jesse Nusbaumer
CSEG and Liaisons
Staff member
Hi Xiao,

That error indicates that the tracer water has deviated from the bulk water in two locations. However, the deviations are fairly small, so I wouldn't worry about them unless something looks off in your history file output. You can prevent this error from popping up entirely by adding the line:

wtrc_check_total_h2o = .false.

to your user_nl_cam file. However, my suspicion is that this error is not what killed your model (otherwise it would say "Error" instead of "Warning"). I would thus recommend checking the other log files to see if another error exists somewhere.

Good luck!

Jesse
 
Hi Xiao,

That error indicates that the tracer water has deviated from the bulk water in two locations. However, the deviations are fairly small, so I wouldn't worry about them unless something looks off in your history file output. You can prevent this error from popping up entirely by adding the line:

wtrc_check_total_h2o = .false.

to your user_nl_cam file. However, my suspicion is that this error is not what killed your model (otherwise it would say "Error" instead of "Warning"). I would thus recommend checking the other log files to see if another error exists somewhere.

Good luck!

Jesse
Thanks very much, Jesse,

The error disappeared by adding that line in CAM user specific file. Now, a new error showed up


ERROR: Isotopic deep-conv precip error!
9.843918322068904E-012 0.000000000000000E+000 0.000000000000000E+000
7 3

isotope tracer: 2
isotot-isotot0 2.254856176885024E+155
fiso_atm-fiso_ocnn 2.254856176885024E+155
isotope tracer: 1


I think the deviation is also very small. Let me try to figure this out...
 

nusbaume

Jesse Nusbaumer
CSEG and Liaisons
Staff member
Hi Xiao,

The "deep-conv precip" error is sadly also not the reason your model is dying, as that error automatically adjusts the precip to the correct values instead of killing the model (it should have a "warning" label instead of an "error" label, that was my bad).

To find the error that actually killed the model, I would recommend looking at the end of the run log files for each component. In one of the components (including possibly cesm.log.XXX) you should see an error message that is different from any errors that came before it. That is likely the actual error that killed your simulation.

Also, the values right below that deep convection error ("isotot-isotot0") look really bad, and are likely related to whatever is causing your model to die. Those values are coming from the sea ice model. I am not sure what your model set-up is, but it might be worth trying first just a short (~1 month) F-compset to see if that works, and then trying a B-compset that is either a start-up run, or is configured exactly like the run described here:


and then branched off using those restart files. If all of those simulations work, then it indicates that there is something likely wrong with your isotopic boundary conditions that may be causing the model to blow-up, at least in the sea ice component.

Anyways, I hope that helps, and good luck with your test simulations!

Jesse
 
Top