water balance error-clm model is stopping

[VivianH]

Hi all. I am using CESM2.1.3 and the compset is BSSP585cmip6. I used the SP mode and set up soil_moisture_streams and LAI_streams to modify the soil moisture and LAI in SP mode from 2066 to 2100. The input soil moisture and LAI data comes from the BGC mode of the BSSP585cmip6 without any modification. The model went well from 2066 to March 2087, but it got an error and stopped on April 1st,2087. I checked the cesm.log and the error is:

WARNING: water balance error nstep= 1265761 local indexc= 147741
errh2o= 7.321385161531552E-004
clm model is stopping - error is greater than 1e-5 (mm)
nstep = 1265761
errh2o = 7.321385161531552E-004
forc_rain = 6.153849349878760E-004
forc_snow = 0.000000000000000E+000
total_plant_stored_h2o_col = 0.000000000000000E+000
endwb = 3045.52049665144
begwb = 3045.55373427568
qflx_evap_tot = -5.596342574981824E-003
qflx_irrig = 0.000000000000000E+000
qflx_surf = 4.018149027064356E-002
qflx_h2osfc_surf = 0.000000000000000E+000
qflx_qrgwl = 0.000000000000000E+000
qflx_drain = 0.000000000000000E+000
qflx_drain_perched = 0.000000000000000E+000
qflx_flood = 0.000000000000000E+000
qflx_ice_runoff_snwcp = 0.000000000000000E+000
qflx_ice_runoff_xs = 0.000000000000000E+000
qflx_glcice_dyn_water_flux = 0.000000000000000E+000
qflx_snwcp_discarded_ice = 0.000000000000000E+000
qflx_snwcp_discarded_liq = 0.000000000000000E+000
qflx_rootsoi_col(1:nlevsoil) = 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000
clm model is stopping
calling getglobalwrite with decomp_index= 147741 and clmlevel= column
local column index = 147741
global column index = 157594
global landunit index = 53672
global gridcell index = 17917
gridcell longitude = 238.750000000000
gridcell latitude = 63.6125654450261
column type = 215
landunit type = 2
ENDRUN:
ERROR: ERROR in BalanceCheckMod.F90 at line 432

I have attached the CESM log files in the attachment. I have done the exact same work in BHIST_BPRPcmip6 before, replacing the soil moisture and LAI of the SP mode by the BGC output from 1980 to 2014, and it worked quite well. But now it produces this error in BSSP585cmip6.
I would be really appreciated for any suggestion in solving this problem.

 

[oleson]

Generally, I would say it might be easier and cheaper to troubleshoot this in a land-only configuration rather than fully-coupled. Maybe you can setup an I-case as similar as possible to the coupled case and see if you encounter the error. If you don't, then I guess you'll have to troubleshoot the coupled case. Another thing to try is to increase the error threshold and see if the model will continue to run after that timestep, or if it indicates a larger problem than just an imbalance at a single timestep. If the error grows to some much larger value, it might be easier to see what the problem is.
The balance code is:

errh2o(c) = endwb(c) - begwb(c) &
- (forc_rain_col(c) &
+ forc_snow_col(c) &
+ qflx_floodc(c) &
+ qflx_irrig(c) &
+ qflx_glcice_dyn_water_flux(c) &
- qflx_evap_tot(c) &
- qflx_surf(c) &
- qflx_h2osfc_surf(c) &
- qflx_qrgwl(c) &
- qflx_drain(c) &
- qflx_drain_perched(c) &
- qflx_ice_runoff_snwcp(c) &
- qflx_ice_runoff_xs(c) &
- qflx_snwcp_discarded_liq(c) &
- qflx_snwcp_discarded_ice(c)) * dtime

In your case, the equation reduces to the non-zero terms:

errh2o(c) = endwb(c) - begwb(c) &
- (forc_rain_col(c) &
- qflx_evap_tot(c) &
- qflx_surf(c)

I think one thing I would examine is qflx_evap_tot, which being negative, means there is a flux of water from the atmosphere to the land. You could output it's components (qflx_evap_soi + qflx_evap_can + qflx_tran_veg). For example, since qflx_rootsoi_col is zero for all layers I would expect qflx_tran_veg to be zero, but maybe it's not.
You might also have to output the terms that go into endwb/begwb, e.g., see subroutine ComputeLiqIceMassNonLake in TotalWaterAndHeatMod.F90.

 

[VivianH]

Generally, I would say it might be easier and cheaper to troubleshoot this in a land-only configuration rather than fully-coupled. Maybe you can setup an I-case as similar as possible to the coupled case and see if you encounter the error. If you don't, then I guess you'll have to troubleshoot the coupled case. Another thing to try is to increase the error threshold and see if the model will continue to run after that timestep, or if it indicates a larger problem than just an imbalance at a single timestep. If the error grows to some much larger value, it might be easier to see what the problem is.
The balance code is:

errh2o(c) = endwb(c) - begwb(c) &
- (forc_rain_col(c) &
+ forc_snow_col(c) &
+ qflx_floodc(c) &
+ qflx_irrig(c) &
+ qflx_glcice_dyn_water_flux(c) &
- qflx_evap_tot(c) &
- qflx_surf(c) &
- qflx_h2osfc_surf(c) &
- qflx_qrgwl(c) &
- qflx_drain(c) &
- qflx_drain_perched(c) &
- qflx_ice_runoff_snwcp(c) &
- qflx_ice_runoff_xs(c) &
- qflx_snwcp_discarded_liq(c) &
- qflx_snwcp_discarded_ice(c)) * dtime

In your case, the equation reduces to the non-zero terms:

errh2o(c) = endwb(c) - begwb(c) &
- (forc_rain_col(c) &
- qflx_evap_tot(c) &
- qflx_surf(c)

I think one thing I would examine is qflx_evap_tot, which being negative, means there is a flux of water from the atmosphere to the land. You could output it's components (qflx_evap_soi + qflx_evap_can + qflx_tran_veg). For example, since qflx_rootsoi_col is zero for all layers I would expect qflx_tran_veg to be zero, but maybe it's not.
You might also have to output the terms that go into endwb/begwb, e.g., see subroutine ComputeLiqIceMassNonLake in TotalWaterAndHeatMod.F90.

Hi Oleson, thank you for your quick and detailed reply! May I ask if I want to increase the error threshold and output the qflx_evap_soi, qflx_evap_can, and qflx_tran_veg variables, is it correct that I only modify the codes (lines 406-434) in BalanceCheckMod.F90?

The codes I try to modify and add are:
else if (abs(errh2o(indexc)) > 1.e-3_r8 .and. (DAnstep > skip_steps) ) then

write(iulog,*)'clm model is stopping - error is greater than 1e-5 (mm)'
write(iulog,*)'nstep = ',nstep
write(iulog,*)'errh2o = ',errh2o(indexc)
write(iulog,*)'forc_rain = ',forc_rain_col(indexc)*dtime
write(iulog,*)'forc_snow = ',forc_snow_col(indexc)*dtime
write(iulog,*)'total_plant_stored_h2o_col = ',total_plant_stored_h2o_col(indexc)
write(iulog,*)'endwb = ',endwb(indexc)
write(iulog,*)'begwb = ',begwb(indexc)

write(iulog,*)'qflx_evap_tot = ',qflx_evap_tot(indexc)*dtime
write(iulog,*)'qflx_evap_soi = ',qflx_evap_soi(indexc)*dtime
write(iulog,*)'qflx_evap_can = ',qflx_evap_can(indexc)*dtime
write(iulog,*)'qflx_tran_veg = ',qflx_tran_veg(indexc)*dtime
write(iulog,*)'qflx_irrig = ',qflx_irrig(indexc)*dtime
write(iulog,*)'qflx_surf = ',qflx_surf(indexc)*dtime
write(iulog,*)'qflx_h2osfc_surf = ',qflx_h2osfc_surf(indexc)*dtime
write(iulog,*)'qflx_qrgwl = ',qflx_qrgwl(indexc)*dtime
write(iulog,*)'qflx_drain = ',qflx_drain(indexc)*dtime
write(iulog,*)'qflx_drain_perched = ',qflx_drain_perched(indexc)*dtime
write(iulog,*)'qflx_flood = ',qflx_floodc(indexc)*dtime
write(iulog,*)'qflx_ice_runoff_snwcp = ',qflx_ice_runoff_snwcp(indexc)*dtime
write(iulog,*)'qflx_ice_runoff_xs = ',qflx_ice_runoff_xs(indexc)*dtime
write(iulog,*)'qflx_glcice_dyn_water_flux = ', qflx_glcice_dyn_water_flux(indexc)*dtime
write(iulog,*)'qflx_snwcp_discarded_ice = ',qflx_snwcp_discarded_ice(indexc)*dtime
write(iulog,*)'qflx_snwcp_discarded_liq = ',qflx_snwcp_discarded_liq(indexc)*dtime
write(iulog,*)'qflx_rootsoi_col(1:nlevsoil) = ',qflx_rootsoi_col(indexc,:)*dtime
write(iulog,*)'clm model is stopping'
call endrun(decomp_index=indexc, clmlevel=namec, msg=errmsg(sourcefile, __LINE__))
end if
end if

If I am correct in modifying them, do I need to recompile the CESM model or just rebuild the case? If I am wrong, where should I modify them or where can I get this information? Sorry I am quite new in learning CESM and modifying the codes. I tried to search on the forum, but I am still confused. Any advice on these will be quite grateful.

 

[slevis]

That does seem like the right code to change. Whenever you change code, you need to run ./case.build, which compiles the code. After that you can run ./case.submit. Before that, you may wish to check some settings in env_run.xml, such as whether CONTINUE_RUN is TRUE.