ERROR: In UpdateState_TopLayerFluxes, h2osoi_ice has gone significantly negative

[rneale]

HI,
I'm getting the following reproducible error in CAM while running a CAM6 f09 FHIST case with cam_cesm2_2_rel_03 (I think).
Oddly it is able to run over 10 years before this error. The modifications I make are 1) using coupled CESM2 SSTs and sea ice rather than observed and 2) masking out the maritime continent/Indonesia land areas and replacing with SSTs. Each of these changes work separately, but not together.
Are these fatal areas or cn they be remedied.

Run dir: /glade/scratch/rneale/f.e22.FHIST.f09_f09.cesm2_2.mcont_mjo_ocn_c2sst.001/run
Case dir: /glade/work/rneale/cesm_cases/f.e22.FHIST.f09_f09.cesm2_2.mcont_mjo_ocn_c2sst.001

Thanks!
Rich

------

99: ERROR: In UpdateState_TopLayerFluxes, h2osoi_ice has gone significantly negativ
99: e
99: Bulk/tracer name = bulk
99: c, lev_top(c) = 35449 0
99: h2osoi_ice_top_orig = 9.609349124611866E-007
99: h2osoi_ice = -3.306816626080789E-018
99: frac_sno_eff = 1.00000000000000
99: qflx_soliddew_to_top_layer*dtime = 0.000000000000000E+000
99: qflx_solidevap_from_top_layer*dtime = 9.609349124644934E-007
99: ENDRUN:
99: ERROR:
99: In UpdateState_TopLayerFluxes, h2osoi_ice has gone significantly negative
99:Image PC Routine Line Source
99:cesm.exe 0000000003AD0BFA Unknown Unknown Unknown
99:cesm.exe 00000000031BF707 shr_abort_mod_mp_ 114 shr_abort_mod.F90
99:cesm.exe 000000000259D5EF abortutils_mp_end 50 abortutils.F90
99:cesm.exe 000000000294CE4D snowhydrologymod_ 1232 SnowHydrologyMod.F90
99:cesm.exe 000000000294A2D1 snowhydrologymod_ 1025 SnowHydrologyMod.F90
99:cesm.exe 00000000028B52EC hydrologynodraina 287 HydrologyNoDrainageMod.F90
99:cesm.exe 00000000025A5608 clm_driver_mp_clm 845 clm_driver.F90
99:libiomp5.so 00002B2DFD32DD43 __kmp_invoke_micr Unknown Unknown
99:libiomp5.so 00002B2DFD2BD63F Unknown Unknown Unknown
99:libiomp5.so 00002B2DFD2BC65C Unknown Unknown Unknown
99:libiomp5.so 00002B2DFD32E2FB Unknown Unknown Unknown
99:libdplace.so.0.0. 00002B2DF5DCE1AD Unknown Unknown Unknown
99:libpthread-2.22.s 00002B2DFDCEF6DA Unknown Unknown Unknown
99:libc-2.22.so 00002B2DFEF2C27D clone Unknown Unknown
99:MPT ERROR: Rank 99(g:99) is aborting with error code 1001.
99: Process ID: 23417, Host: r10i4n27, Program: /glade/scratch/rneale/f.e22.FHIST.f09_f09.cesm2_2.mcont_mjo_ocn_c2sst.001/bld/cesm.exe
99: MPT Version: HPE MPT 2.21 11/28/19 04:21:40
99:

 

[oleson]

Hi Rich, try changing "custom_rel_epsilon" from 1.e-12_r8 to 1.e-11_r8 here in components/clm/src/biogeophys/SnowHydrologyMod.F90:

call truncate_small_values_one_lev( &
num_f = num_snowc, &
filter_f = filter_snowc, &
lb = bounds%begc, &
ub = bounds%endc, &
lev_lb = -nlevsno+1, &
lev = lev_top(bounds%begc:bounds%endc), &
data_baseline = h2osoi_ice_top_orig(bounds%begc:bounds%endc), &
data = h2osoi_ice(bounds%begc:bounds%endc, :), &
custom_rel_epsilon = 1.e-12_r8)

Note that there is one call for ice and one for liquid and you'll want to change the one for ice.
I encountered that error in my own simulation and this modification allowed me to continue the run.
My error was the same magnitude as yours (e-18).

 

[rneale]

Hi Rich, try changing "custom_rel_epsilon" from 1.e-12_r8 to 1.e-11_r8 here in components/clm/src/biogeophys/SnowHydrologyMod.F90:

call truncate_small_values_one_lev( &
num_f = num_snowc, &
filter_f = filter_snowc, &
lb = bounds%begc, &
ub = bounds%endc, &
lev_lb = -nlevsno+1, &
lev = lev_top(bounds%begc:bounds%endc), &
data_baseline = h2osoi_ice_top_orig(bounds%begc:bounds%endc), &
data = h2osoi_ice(bounds%begc:bounds%endc, :), &
custom_rel_epsilon = 1.e-12_r8)

Note that there is one call for ice and one for liquid and you'll want to change the one for ice.
I encountered that error in my own simulation and this modification allowed me to continue the run.
My error was the same magnitude as yours (e-18).

Thanks Keith, I'll give it a try!

 

[rneale]

Thanks Keith, I'll give it a try!

It worked! Thanks for the help Keith.