Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

IHIST case crash after 30 years due to cbalance warning and "ERROR in CNBalanceCheckMod.F90"

K

KeerZ

Member
Hi all,

I'm running a 25 km land-only spinup using an IHIST compset (HIST_DATM%CPLHIST_CLM50%BGC-CROP_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV) driven by cpl history files from 2000 to 2009. After running the case for 30 years, the model crashed with the error message below.

I made sure all atm forcing was cycled from 2000 to 2009. The taxmode is "cycle" for all stream files except for the co2tseries.20tr, which uses "extend". I now realize that I should also use taxmode =cycle for co2tseries.

But I wonder if using taxmode = extend for co2tseries.20tr instead of cycle could be the main cause of the cbalance error. If not, what else should I check to avoid crashes after long runs?

Here is my case directory: /glade/work/leizhao/keer/IHistClm50BgcCrop.Spinup/; and log files: /glade/derecho/scratch/leizhao/IHistClm50BgcCrop.Spinup/run/.

Any suggestions are appreciated. Thank you!

Key error message
dec0068.hsn.de.hpc.ucar.edu 181: cbalance warning at c = 176 2.93618310530540
dec0068.hsn.de.hpc.ucar.edu 181: 4380.10585378379
dec0068.hsn.de.hpc.ucar.edu 181: column cbalance error = 2.93618310530540 176
dec0068.hsn.de.hpc.ucar.edu 181: is fates column? = F
dec0068.hsn.de.hpc.ucar.edu 181: Latdeg,Londeg= -8.09647979139504 280.625000000000
dec0068.hsn.de.hpc.ucar.edu 181: begcb = 4383.04874857540
dec0068.hsn.de.hpc.ucar.edu 181: endcb = 4380.10585378379
dec0068.hsn.de.hpc.ucar.edu 181: delta store = -2.94289479160852
dec0068.hsn.de.hpc.ucar.edu 181: --- Inputs ---
dec0068.hsn.de.hpc.ucar.edu 181: gpp = 0.000000000000000E+000
dec0068.hsn.de.hpc.ucar.edu 181: --- Outputs ---
dec0068.hsn.de.hpc.ucar.edu 181: er = 6.871720232064937E-003
dec0068.hsn.de.hpc.ucar.edu 181: col_fire_closs = 0.000000000000000E+000
dec0068.hsn.de.hpc.ucar.edu 181: col_hrv_xsmrpool_to_atm = 0.000000000000000E+000
dec0068.hsn.de.hpc.ucar.edu 181: col_xsmrpool_to_atm = -1.600339290017582E-004
dec0068.hsn.de.hpc.ucar.edu 181: wood_harvestc = 0.000000000000000E+000
dec0068.hsn.de.hpc.ucar.edu 181: crop_harvestc_to_cropprodc = 0.000000000000000E+000
dec0068.hsn.de.hpc.ucar.edu 181: -1*som_c_leached = -5.694475777585909E-014
dec0068.hsn.de.hpc.ucar.edu 181: iam = 181: local column index = 176
dec0068.hsn.de.hpc.ucar.edu 181: iam = 181: global column index = 394449
dec0068.hsn.de.hpc.ucar.edu 181: iam = 181: global landunit index = 252089
dec0068.hsn.de.hpc.ucar.edu 181: iam = 181: global gridcell index = 125882
dec0068.hsn.de.hpc.ucar.edu 181: iam = 181: gridcell longitude = 280.6250000
dec0068.hsn.de.hpc.ucar.edu 181: iam = 181: gridcell latitude = -8.0964798
dec0068.hsn.de.hpc.ucar.edu 181: iam = 181: column type = 268
dec0068.hsn.de.hpc.ucar.edu 181: iam = 181: landunit type = 2
dec0068.hsn.de.hpc.ucar.edu 181: ENDRUN:
dec0068.hsn.de.hpc.ucar.edu 181: ERROR: ERROR in CNBalanceCheckMod.F90 at line 385
dec0068.hsn.de.hpc.ucar.edu 181: Image PC Routine Line Source
dec0068.hsn.de.hpc.ucar.edu 181: cesm.exe 0000000001551CFD shr_abort_mod_mp_ 114 shr_abort_mod.F90
dec0068.hsn.de.hpc.ucar.edu 181: cesm.exe 00000000005DBEE1 abortutils_mp_end 98 abortutils.F90
dec0068.hsn.de.hpc.ucar.edu 181: cesm.exe 0000000000E50818 cnbalancecheckmod 385 CNBalanceCheckMod.F90
dec0068.hsn.de.hpc.ucar.edu 181: cesm.exe 00000000008722F8 cnvegetationfacad 1219 CNVegetationFacade.F90
dec0068.hsn.de.hpc.ucar.edu 181: cesm.exe 00000000005EBCCA clm_driver_mp_clm 1174 clm_driver.F90
dec0068.hsn.de.hpc.ucar.edu 181: cesm.exe 00000000005910AE lnd_comp_nuopc_mp 904 lnd_comp_nuopc.F90
Click to expand...

What version of the code are you using? CTSM5.2.001

Have you made any changes to files in the source tree? Yes, but only to output some variables. My code changes should not affect model results.

Describe every step you took leading up to the problem:
./create_newcase --case ~/CTSM5.2.001_BEMpres/cime/scripts/JoyceCase/IHistClm50BgcCrop.Spinup --compset HIST_DATM%CPLHIST_CLM50%BGC-CROP_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV --res f02_g17 --project UIUC0056 --run-unsupported

./case.setup

./xmlchange DATM_YR_ALIGN=2000,DATM_YR_START=2000,DATM_YR_END=2009,RUN_STARTDATE=2000-01-01

./xmlchange STOP_OPTION=nyears,STOP_N=4,RESUBMIT=4,JOB_WALLCLOCK_TIME=12:00:00

./xmlchange LND_DOMAIN_MESH='/glade/u/home/keerzhang/CTSM_mesh/Makemesh/lnd_mesh_0.23x0.31_c240912.nc',ATM_DOMAIN_MESH='/glade/u/home/keerzhang/CTSM_mesh/Makemesh/lnd_mesh_0.23x0.31_c240912.nc'

Modify user_nl_datm:
model_meshfile='/glade/u/home/keerzhang/CTSM_mesh/Makemesh/lnd_mesh_0.23x0.31_c240912.nc'
model_maskfile='/glade/u/home/keerzhang/CTSM_mesh/Makemesh/lnd_mesh_0.23x0.31_c240912.nc'

Modify user_nl_datm_stream:
use the 2000-2009 coupler history files from b.e21.BHIST.f09_g17.CMIP6-historical.011. Make sure all year_first, and year_align are 2000, and year_last is 2009.

Modify user_nl_clm:
use_init_interp = .true.
finidat='/glade/derecho/scratch/keerzhang/HistSSP37/restart/2000-01-01-00000/b.e21.BHIST.f09_g17.CMIP6-historical.011.clm2.r.2000-01-01-00000.nc'
fsurdat='/glade/work/yifanc17/02_data/cesmdata/surfdata/0.23x0.31/surfdata_0.23x0.31_hist_2015_78pfts_TEST_THR_modified_c240829.nc'
flanduse_timeseries=''

do_transient_crops=.false.
do_transient_pfts=.false.
do_transient_urban=.false.
urban_explicit_ac = .true.
stream_fldfilename_urbantv ='/glade/work/xinchang/02_Explicit_AC_Adoption/02_data_present_day/CTSM52_urbantv_Li_2024_0.9x1.25_simyr1849-2106_c20230621.nc'

stream_year_first_urbantv = 2000
stream_year_last_urbantv = 2009
model_year_align_urbantv = 2000
stream_year_first_cropcal = 2000
stream_year_last_cropcal = 2009
model_year_align_cropcal = 2000

stream_year_first_popdens = 2000
stream_year_last_popdens = 2009
model_year_align_popdens = 2000

stream_year_first_ndep = 2000
stream_year_last_ndep = 2009
model_year_align_ndep = 2000

urban_hac='ON_WASTEHEAT'
building_temp_method=1
calc_human_stress_indices='ALL'

Then I also added some monthly output
Click to expand...
 
K

KeerZ

Member
Also, since this is a spinup case, I turned off all transient behavior by setting year_first, year_last, and year_align to 2000 for all stream files. I also set do_transient_crops = .false., do_transient_pfts = .false., do_transient_urban = .false. and flanduse_timeseries='' in user_nl_clm.

If there's anything else I should do (or check) to fully disable transient behavior in the IHIST compset, please kindly let me know! Thanks!
 
slevis

slevis

Moderator
Staff member
I have no helpful guesses at this point, and I would not have expected a cbalance error from using 'extend' for the co2. Regardless, go ahead and start again since you have to redo the simulation.

My first guess is that you have hit a problem going from the f09 finidat to this high resolution, so you may hit the same problem again. The most likely way around that would be to do a spin-up for the high res case from scratch (i.e. starting with cold start).
 
You must log in or register to reply here.
Top