Water balance and h2o_soi_ice error with 0.5 deg IHistClm45SpGs

[steverarnold]

Hi Folks,

I am running the IHistClm45SpGs component set with CLM 2.2.2 on Derecho, but with a 0.47 x 0.63 deg grid.

I created the case with:
/glade/work/fvitt/cesm/cesm2.2.2/cime/scripts/create_newcase --case CLM_hires --res f05_g17 --project P93300043 --compset IHistClm45SpGs --run-unsupported

And made the following additions to user_nl_clm:
flanduse_timeseries = '/glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/surfdata_map/landuse.timeseries_0.47x0.63_hist_16pfts_Irrig_CMIP6_simyr1850-2015_c180508.nc'
fsurdat = '/glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/surfdata_map/surfdata_0.47x0.63_16pfts_Irrig_CMIP6_simyr1850_c170919.nc'

The model builds fine, and I submitted the run initially for a few days (successfully) and then I started a spin-up, first for 12 months initially.

During the first couple of months the model flags many water balance errors, seemingly across several columns e.g.:
dec2178.hsn.de.hpc.ucar.edu 221: WARNING: water balance error nstep= 5621 local indexc= 91293
dec2178.hsn.de.hpc.ucar.edu 221: errh2o= 1.124938420193899E-009
dec2178.hsn.de.hpc.ucar.edu 221: WARNING: water balance error nstep= 5628 local indexc= 91293
dec2178.hsn.de.hpc.ucar.edu 221: errh2o= -1.446340107142535E-009
dec2178.hsn.de.hpc.ucar.edu 221: WARNING: water balance error nstep= 5650 local indexc= 91293
dec2178.hsn.de.hpc.ucar.edu 221: errh2o= -1.111338243653393E-009

The model then fails with an error associated with a significant negative h2o_soi_ice value:
ERROR: In UpdateState_TopLayerFluxes, h2osoi_ice has gone significantly negativ
dec2177.hsn.de.hpc.ucar.edu 25: e
dec2177.hsn.de.hpc.ucar.edu 25: Bulk/tracer name = bulk
dec2177.hsn.de.hpc.ucar.edu 25: c, lev_top(c) = 10012 0
dec2177.hsn.de.hpc.ucar.edu 25: h2osoi_ice_top_orig = 1.163629733743448E-002
dec2177.hsn.de.hpc.ucar.edu 25: h2osoi_ice = -1.855342950660486E-003
dec2177.hsn.de.hpc.ucar.edu 25: frac_sno_eff = 0.310826954742182
dec2177.hsn.de.hpc.ucar.edu 25: qflx_soliddew_to_top_layer*dtime = 0.000000000000000E+000
dec2177.hsn.de.hpc.ucar.edu 25: qflx_solidevap_from_top_layer*dtime = 4.340563159744538E-002
dec2177.hsn.de.hpc.ucar.edu 25: ENDRUN:

I have specified different start dates in trying to investigate the problem (01 Jan 2003, 01 Jan 2010, 01 Jan 2000). The error occurs after a few days or after a few months depending on the start date.

Any advice on how to proceed would be appreciated. The component set is a scientifically supported option, but I guess the higher resolution grid with this comp set may have not been formally tested / evaluated.

The forum will not let me upload the 2.9 MB cesm.log* file, so I have copied the logs to here:

CESM_logs - Google Drive

drive.google.com

Thanks,
Steve.

 

[steverarnold]

Edit: I have made no changes to the default GSWP3v1 data forcing dataset for this simulation. Would you expect water conservation errors to be introduced by a potential mis-match in the resolution of the land grid and the atmospheric data? Does the model re-grid the datm data to the CLM grid?

 

[oleson]

It might be related to the issue described in this post, which also provides a possible solution:

ERROR: In UpdateState_TopLayerFluxes, h2osoi_ice has gone significantly negative

HI, I'm getting the following reproducible error in CAM while running a CAM6 f09 FHIST case with cam_cesm2_2_rel_03 (I think). Oddly it is able to run over 10 years before this error. The modifications I make are 1) using coupled CESM2 SSTs and sea ice rather than observed and 2) masking out the...

bb.cgd.ucar.edu

If that doesn't help, some code changes were made to address this, see these issues/pull requests here:

h2osoi_ice can go significantly negative · Issue #1253 · ESCOMP/CTSM

Brief summary of bug In some situations (I haven't determined exactly when), it's possible for h2osoi_ice to go significantly negative, triggering an abort due to an error check: "In UpdateState_To...

github.com

Need some changes to avoid negative h2osoi_ice in UpdateState_TopLayerFluxes · Issue #1979 · ESCOMP/CTSM

Brief summary of bug In UpdateState_TopLayerFluxes, there is a somewhat arbitrary epsilon used to determine if h2osoi_ice and h2osoi_liq are close enough to zero that they should be truncated to ze...

github.com

Rework handling of evaporation constraint in SoilFluxes by billsacks · Pull Request #1987 · ESCOMP/CTSM

Description of changes Occasionally, h2osoi_ice was going significantly negative in UpdateState_TopLayerFluxes - see #1979. As noted in that issue, this seems to be due to h2osoi_ice having a very ...

github.com

It would probably be easiest to first try setting the thresholds from 1.e-12 (which I think is the default used in your code) to e-11 or even e-10. If that doesn't work, then you'd have to implement the code changes that were made to provide a more robust fix.

 

[steverarnold]

Thanks for the rapid response. I will take a look and see if this helps.