Spin-Up Issues in BGC Mode Using Plumber 2 Flux Tower Data

[ijaguirre]

Hi,

In the context of running the Plumber 2 flux towers using CLM5, I am trying to spin up the towers in BGC mode using AD (300 years) and post-ad (100 years). To conduct this, I am using the ./run_towers script:
# Run the script to set up the tower sites
export tower=AT-Neu
./run_tower --plumber-sites $tower --output-root ~/scratch/pl2_tower/v1/$tower/ --overwrite --run-type ad --setup-only

Then, I modify the length of the simulation:
./xmlchange STOP_OPTION="nyears”
./xmlchange STOP_N=300

# Currently,
./xmlquery CLM_ACCELERATED_SPINUP # returned on
./xmlquery CLM_FORCE_COLDSTART # returned on

Finally, I set up, built, and submitted the case.

However, the simulation fails after a few minutes.

The CaseStatus file mentions:
•-------------------------------------------------------------------------
•- Prestage required restarts into /glade/u/home/iaguirre/scratch/pl2_tower/v1/AT-Neu/AT-Neu.ad/run
•- Case input data directory (DIN_LOC_ROOT) is /glade/campaign/cesm/cesmdata/inputdata
•- Checking for required input datasets in DIN_LOC_ROOT
•-------------------------------------------------------------------------
•run command is /glade/u/home/iaguirre/scratch/pl2_tower/v1/AT-Neu/AT-Neu/bld/cesm.exe >> cesm.log.$LID 2>&1
•ERROR: RUN FAIL: Command ' /glade/u/home/iaguirre/scratch/pl2_tower/v1/AT-Neu/AT-Neu/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed
•See log file for details: /glade/u/home/iaguirre/scratch/pl2_tower/v1/AT-Neu/AT-Neu.ad/run/cesm.log.9642201.desched1.250526-171145

The log file, CaseStatus, and lnd log file are attached.

Therefore, I would like to know how to configure my simulations correctly to spin up the Plumber 2 flux towers using BGC mode.
As a reference, I was able to spin up the NEON towers using the same procedure (reaching equilibrium with around 200-300 years)

Thank you very much for the help,
Warm regards,

What version of the code are you using?
ctsm5.3.046


Describe every step you took leading up to the problem:
I have repeated manually with the traditional way of setting up CLM and got the same result:

cd /glade/work/iaguirre/CTSM/cime/scripts
export site="AT-Neu"
# then create a new case.
./create_newcase --case ~/scratch/pl2/v3/${site}_test2 \
--res CLM_USRDAT \
--compset I1PtClm60Bgc \
--output-root ~/scratch/pl2/v3/${site} \
--run-unsupported \
--user-mods-dir /glade/work/iaguirre/CTSM/cime_config/usermods_dirs/clm/PLUMBER2/${site} \
--handle-preexisting-dirs r

cd ~/scratch/pl2/v3/${site}_test2

 

[slevis]

@ijaguirre I reproduced your error on derecho with version ctsm5.3.048.

The relevant error appears in run/PET0.ESMF_LogFile
ERROR PET0 ESMCI_Calendar.C:1059 ESMCI::Calendar::convertToTime() Input argument out of range - ; Gregorian: for February 22, dd=29 > 28 days in the month.

...so I typed ./xmlquery CALENDAR and got NO_LEAP
and then typed grep CALENDAR *.xml and got env_build.xml
I used my editor to open env_build.xml and looked for CALENDAR and any associated documentation. I found that I could change CALENDAR to GREGORIAN, so I did. Next steps:
./case.setup --reset
./case.build
./case.submit

Unfortunately now I get a different error
forrtl: severe (61): format/variable-type mismatch, unit -131, file /glade/derecho/scratch/slevis/AT-Neu.ad/run/atm.log.9649510.desched1.250527-111504
Image PC Routine Line Source
cesm.exe 000000000137C933 Unknown Unknown Unknown
cesm.exe 00000000010FA8D3 dshr_strdata_mod_ 1050 dshr_strdata_mod.F90
cesm.exe 000000000057CD58 atm_comp_nuopc_mp 670 atm_comp_nuopc.F90
cesm.exe 000000000057C933 atm_comp_nuopc_mp 560 atm_comp_nuopc.F90

@erik @oleson @wwieder has all this been tested before?
@ijaguirre have you tried other PLUMBER sites in case the problem is with this site only?

 

[ijaguirre]

Thank you for the follow-up.
Yes, I have tested many different sites and observed the same problem.

 

[slevis]

A troubleshooting idea could be to open env_run.xml and modify DATM_YR_START and DATM_YR_END in a way that would avoid leap years.

 

[ijaguirre]

Hi,
Thank you for the suggestions.

I set DATM_YR_START to 2009 and DATM_YR_END to 2012 to avoid leap years on /glade/derecho/scratch/iaguirre/plumber2_learning/test3/AT-Neu/AT-Neu.ad

However, the simulation did not run correctly. I attached the cesm.logout.
Thank you again.

 

[slevis]

In your test I see the same error that I got earlier when I changed CALENDAR to GREGORIAN:
forrtl: severe (61): format/variable-type mismatch, unit -131, file /glade/derecho/scratch/iaguirre/plumber2_learning/test3/AT-Neu/AT-Neu.ad/run/atm.log.9650958.desched1.250527-135757
Image PC Routine Line Source
cesm.exe 000000000137C933 Unknown Unknown Unknown
cesm.exe 00000000010FA8D3 dshr_strdata_mod_ 1050 dshr_strdata_mod.F90
cesm.exe 000000000057CD58 atm_comp_nuopc_mp 670 atm_comp_nuopc.F90
cesm.exe 000000000057C933 atm_comp_nuopc_mp 560 atm_comp_nuopc.F90

I'm still interested in feedback from others about whether PLUMBER cases used to work or they had not been tested.

 

[slevis]

...though 2012 may be a leap year, right? Maybe change the end year to 2011.

 

[oleson]

I don't know the status of the testing on this. I did examine my old PLUMBER setup and made some changes to the AT-Neu case and it seemed to work for an AD spinup. The case directory is here:

/glade/work/oleson/ctsm5.3.048/tools/site_and_regional/AT-Neu.ad

I passed this info on to @ijaguirre

 

[slevis]

@ijaguirre it would be helpful to me and the community if you could summarize here the changes (based on @oleson's case) that got your case working. Thanks!