Soil Carbon spin up with crops

[rambhari01]

Dear CESM user/help,

We are planning to run 2 atmospheric only control runs for 1850, with coupled land (using F compset) using CESM2.2.2 and cases are created using the compset "

1850_CAM60_CLM50%BGC-CROP_CICE%PRES_DOCN%DOM_MOSART_SGLC_SWAV".

In these 2 runs, First would be the 1850 control with the crops as true (active) and in second, crops would be turned off using "crop_use=.false." and omitting the "-crop" in env_run.xml file.

In the both of these case, CLM model get this below file as clm initial conditions assigned to it.

"finidat = /scratch/rs9552/inputdata/lnd/clm2/initdata_map/clmi.B1850Clm50BgcCrop.0161-01-01.0.9x1.25_gx1v7_simyr1850_c200729.nc"

We have following questions for this experiment setup:

1.) Does the initial condition file for 1850CAM-CLMBGC_Crop include a carbon spinup for 1850 crops? In other words, are the soil and vegetation carbon pools equilibrated for pre-industrial conditions?
- We believe that this is true as the main B1850 run (control run from which clm initial file came) has used the crop_use=true. Please confirm.

2.) For the second 1850 control run, Is there any initial condition file 1850CAM-CLMBGC_Crop with carbon spinup that has crops turned OFF?
If not, how can we create a carbon spinup we start with using the same initial condition file mentioned above?
I guess, if we run use the CLM_ACCELERATED_SPINUP=on ? and then switch to a control run? How long we need to run these sequence in order to get a spun up control run for carbon with crops turned off?


Apart from these another question regarding the CLM vegetation,

3.) Are the vegetation types pre-assigned to a grid cell or is there changing biogeography of vegetation?


Thank you
-R

 

[oleson]

1) Yes
2) No. Ideally, you would do an I-case spinup (CLM_ACCELERATED_SPINUP=on, then a normal run using the initial file from the end of the accelerated spinup run) forced with coupler history (atmospheric) forcing taken from an F-case that has the same land configuration as the I-case. And then use the initial file from the end of that to start your F-case. But I assume you probably don't have coupler history forcing available. Alternatively, you could do an I-case spinup forced by reanalysis (GSWP3v1 is the default) with crops off and then use the initial file at the end of that to initialize your F-case. You'd want to run the F-case for 10-20 years before using data for analysis to allow the land to equilibrate to the different atmospheric model forcing.
There is some information on I-case spinup here:

1.5.5. Spinup of CLM5.0-BGC-Crop — ctsm release-clm5.0 documentation

To meet strict equilibrium criteria in an I-case spinup, AD usually takes about 300 years, and post-AD can take around 1000 years. But you should decide what's really important to spinup, maybe you don't need deep soil carbon at high latitudes to be spunup, which takes the longest amount of time.

3) Vegetation types are fixed in an 1850 configuration. They can change in a historical simulation, but the changes are prescribed using a landuse timeseries file.

 

[rambhari01]

Dear Keith Oleson,
Thank you very much for your response with precise details.

Regarding the point 2.: To achieve a initial condition with equilibrated soil and vegetation carbon pools. I am using the cesm code : cesm2.2.2

I have followed the following set of steps to perform the I compset runs. I used the compset with crops and manually turned the crops off by removing the "-crop" form env_run.xml and setting the clm namelist "use_crops=.false." Used the following line to create the case

A--> ./create_newcase --case /scratch/rs9552/cesm2.2.2_N/BGC_spinup_2 --res f09_g17 --compset 1850_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV --machine greene --run-unsupported

Then used the following xmlchanges
./xmlchange CLM_ACCELERATED_SPINUP="on"

removed "-crop" from "-bgc bgc -crop" and modified the user_nl_clm using the these line

&clm_inparm
use_crop=.false.
/

Then build the case and run the initial run of 5 days finished successfully. All the history files are saved properly.

After that i modified the .xmls using the following xmlchanges. (I submitted a 5 years run just to check the restart).

./xmlchange STOP_N=5,STOP_OPTION=nyears
./xmlchange CONTINUE_RUN=TRUE
./xmlchange JOB_WALLCLOCK_TIME=10:00:00

After that, I resubmitted the run which crashed on restart with the following error line in cesm.log file


ENDRUN:
ERROR: ERROR in CNBalanceCheckMod.F90 at line 621

nbalance warning at g = 9536 -2.951479051793528E+020
9.969197048290677E+036
nbalance warning at g = 9537 -2.951479051793528E+020

I have also attached the log file here. Please check this issue and help me on this issue.

Thank you very much.

-Ram

 

[oleson]

A simpler way to deselect crops would be to remove "CROP" from the compset longname, e.g.,

1850_DATM%GSWP3v1_CLM50%BGC_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV

I would expect the model to restart properly after a 5-day run, although I'm not sure we test that.
I'd try a 5 year run from the beginning.

 

[rambhari01]

HI Keith,

Thank you for very much for this information. I will use either of this in future to remove crops.

Meanwhile, Sticking with the same compset and removing crops from the case as described above. I ran a 5 year long run and use the RESUBMIT=1 to restart it from the 5th year onwards.
This run also get crashed on restarting from the 5th year onward with an error related with CH4 conservation as given below in red colored text and also see the cesm.log_ch4.txt.

CH4 Conservation Error in CH4Mod during diffusion, nstep, c, errch4 (mol /m^2.t
imestep) 87601 34625 NaN
Latdeg,Londeg= -77.7486910994765 95.0000000000000
ENDRUN:
ERROR:
ERROR: CH4 Conservation Error in CH4Mod during diffusionERROR in ch4Mod.F90 at
line 3998
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.

Then as suggested from one of the post, I tried wtih use_lch4=.false., but now it crashed due to another error given below (also check the cesm_bal.txt)

WARNING: water balance error nstep= 87601 local indexc= 28491
errh2o= 8.136861095693304E-002
clm urban model is stopping - error is greater than 1e-5 (mm)
nstep = 87601

It is all happening at the restart from 5th year onwards. Could you please suggest on this?

Thank you

-Ram

 

[oleson]

I'm not sure what could be going wrong. I tried the following with release-cesm2.2.2 and it ran for 5 years and then restarted and ran another 5 years.

./create_newcase --case BGC_spinup_2 --res f09_g17 --compset 1850_DATM%GSWP3v1_CLM50%BGC_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV --run-unsupported
cd BGC_spinup_2
./xmlchange CLM_ACCELERATED_SPINUP=on
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=5
./xmlchange RESUBMIT=1
./case.setup
./preview_namelists
./case.build
./case.submit

The initial file that comes out of the box is:

finidat = '/glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/initdata_map/clmi.I1850Clm50BgcCrop-ciso.1366-01-01.0.9x1.25_gx1v7_simyr1850_c200428.nc'

Is this the same as what you get in your I-case?

 

[rambhari01]

Hi Keith,

I also tried the exact run which you performed as mentioned in above post. This run runs well for 5 years and then it crashed with the error as given below and in attached log file. (The model code cesm2.2.2 which I am using is cloned on 1st April 2025, * (HEAD detached at release-cesm2.2.2))

ENDRUN:
ERROR: ERROR in BalanceCheckMod.F90 at line 470
clm model is stopping
calling getglobalwrite with decomp_index= 51624 and clmlevel= pft
local patch index = 51624
global patch index = 41332
global column index = 29983
global landunit index = 18739
global gridcell index = 8606
gridcell longitude = 152.500000000000
gridcell latitude = -4.24083769633508
pft type = 15
column type = 215
landunit type = 2
ENDRUN:
ERROR: ERROR in BalanceCheckMod.F90 at line 653

Yes, Initial file for this run is :

finidat = '/scratch/rs9552/inputdata/lnd/clm2/initdata_map/clmi.I1850Clm50BgcCrop-ciso.1366-01-01.0.9x1.25_gx1v7_simyr1850_c200428.nc'



Based on the model throughput for I-compset, I estimated that I can manage the initial BGC spin up to 400 years in single run submit without using the resubmit. But these files will be used to restart the BGC final spinup run. I tried to check if I can use these files to restart the final bgc running as given in your first response.
Again, here model is restarting from the previous run and it got crashed because of the same issue. The error as given below. I also have attached the cesm log file (cesm_BGCFinal.txt).

Creating variable usurf
WARNING: water balance error nstep= 0 local indexc= 28491
errh2o= 8.190679404727615E-002
CH4 Conservation Error in CH4Mod during diffusion, nstep, c, errch4 (mol /m^2.t
imestep) 0 34625 NaN
Latdeg,Londeg= -77.7486910994765 95.0000000000000
ENDRUN:
ERROR:
ERROR: CH4 Conservation Error in CH4Mod during diffusionERROR in ch4Mod.F90 at
line 3998
Image PC Routine Line Source
cesm.exe 000000000119E55D shr_abort_mod_mp_ 114 shr_abort_mod.F90
cesm.exe 0000000000515D6F abortutils_mp_end 50 abortutils.F90

Please help me figure out if I am missing anythings here.

Thank you.

-Ram

 

[oleson]

I'm not sure how to help here, since I can't replicate this behavior on our machine. Have you successfully run I-cases with restarts on your machine before? Or any kind of cases? I assume you've ported the model to your machine correctly?
If you think you've ported correctly and have had success with restarts in the past, then you'll have to troubleshoot the errors themselves. For example, the first problem in the InitialRestart run is that solar radiation is not conserved, by a large amount (-119.972778840090 W/m2), maybe suggesting a problem with albedos on the restart. The beginning and ending water balance quantities (begwb and endwb) after that are huge (e+70), which might indicate missing values are being used in the calculation.
You could try a DEBUG=True run from the beginning to see if maybe your run actually has problems from the beginning and the run is proceeding regardless. You can look at your history file output to see if there are any strange values in any of the fields.

 

[rambhari01]

Hi Keith,

Thank you very much for suggesting these important look throughs. It looks alike that the issue is related with I-compset restart and may be connected with the model diagnostics populated in restart file. In order to answer some of the question you have asked. I hope these will be useful to understand the cause of this problem.

1.) Yes, we ported the CESM on NYU HPC with the system admin's help. Also I am able to perform a successful restart run with F-Compsets and this is first time, I am trying do with the I compset.

2.) I tried to look into the output filed saved and I found things suspicious for variable TOTECOSYSC and CPOOL. I could not upload the entire ncfile due to size issue but extracted the variable for a year and uploaded here as .tar file. I am also sharing the verbose output of some command to check maximum and minimum of these variables as give below.

[rs9552@log-2 hist]$ ncks -s '%f\n' -H -C -v TOTECOSYSC BGC_spinup_N2.clm2.h0.0001-01-01-00000.nc | sort -n | awk 'NR==1{min=$1} {max=$1} END{print "Min:", min; print "Max:", max}'
Min:
Max: 9760.971680

[rs9552@log-2 hist]$ ncks -s '%f\n' -H -C -v CPOOL BGC_spinup_N2.clm2.h0.0001-01-01-00000.nc | sort -n | awk 'NR==1{min=$1} {max=$1} END{print "Min:", min; print "Max:", max}'
Min:
Max: 0.000000

It seems that this needs some netcdf flag or libabries need to connected for this.

3.) As you suggested, I also tried the initial run with DEBUG=TRUE, this run could not even start as it populated a lot of verbose output to log file. It seems that these are beyond my understanding to look into. I request you to please have look into the attached log file.
This gave some error which may be related to netcdf and may have resulted in to an error while restarting as a resulted of broken diagonostics.

Thank you ver much for your help and working on this ssiue.

-Ram

 

[oleson]

1) I'm not sure this constitutes a complete and proper port of the model, but I"m not an expert on that. Usually, there are a bunch of system tests that have to pass before the model can be declared ported.
2) There are some "-Infinityf" values in the TOTECOSYSC file. If this file represents output from before the restart then I can see why the restart isn't working.
3) This output is beyond my understanding as well, as it seems to be pointing to some kind of problem in cime, according to the traceback:

3 0x000000000004917e ucp_proto_perf_envelope_make() ???:0
4 0x000000000004a81d ucp_proto_init_parallel_stages() ???:0
5 0x0000000000051df7 ucp_proto_common_init_caps() ???:0
6 0x00000000000535f5 ucp_proto_multi_init() ???:0
7 0x0000000000078404 ucp_rndv_rtr_handler() ???:0
8 0x0000000000054d42 ucp_proto_select_elem_trace() ???:0
9 0x0000000000056561 ucp_proto_select_lookup_slow() ???:0
10 0x0000000000056a25 ucp_proto_select_short_init() ???:0
11 0x000000000004ba89 ucp_worker_get_ep_config() ???:0
12 0x00000000000a139c ucp_wireup_init_lanes() ???:0
13 0x00000000000339ce ucp_ep_create_to_worker_addr() ???:0
14 0x0000000000034b33 ucp_ep_create() ???:0
15 0x00000000000039fb mca_pml_ucx_add_procs() ???:0
16 0x00000000000f914a ompi_mpi_init() ???:0
17 0x0000000000096bfd PMPI_Init() ???:0
18 0x00000000000577a7 mpi_init__() ???:0
19 0x000000000041bc43 cime_comp_mod_mp_cime_pre_init1_() /scratch/rs9552/cesm2.2.2_N/cime/src/drivers/mct/main/cime_comp_mod.F90:678
20 0x000000000045e1ff MAIN__() /scratch/rs9552/cesm2.2.2_N/cime/src/drivers/mct/main/cime_driver.F90:61

where line 61 of cime_driver.F90, at least in my checkout of cesm2.2.2, is:

call cime_pre_init1(esmf_logfile_option)

I think your best bet would be to post on the infrastructure forum and describe the steps you took to port the model. Maybe they can help further with this. I am puzzled why you can supposedly run an F-case with restart successfully.