How to add water tracers to CAM6

[huazhen]

Hi there,

I am trying to run a CAM6 case with water tracers on our super computer. I am using FHIST_BGC compset. I have tried to add commands ./xmlchange --append CAM_CONFIG_OPTS='-nadv_tt 3' and test_tracer_names='WT1','WT2','WT3' in the case run. But I get the following error messages in $Case/run/atm.log.36678.h2o.200110-104926 (will attach full log file below). Based on the following messages in log file (line 689), I think the problem maybe connect with settings of "IC file" or "tracers_init_cnst", but I still don't know how to fix it. In addition, I still don't know how to define the boundary of each tracer. Do you have any suggestions? Any help is much appreciated. Thanks a lot.

---------------------------------------------------
CAM_PIO_CHECK_VAR INFO: variable WT1 is not on file
Field WT1 not found on initial dataset
tracers_init_cnst: ERROR: tracer WT1 should be on IC file
ERROR: tracers_init_cnst: ERROR: tracer missing from IC file

Kind regards,
Huazhen

 

[nusbaume]

Hi Huazhen,

The water tracer feature is still in development for CAM6, and thus is not present in the standard code. If you are ok with using CAM5 instead, then you can find a publicly-available version with a water tracer capability on Github here:

nusbaume/iCESM1.2

Sadly with this version turning on water tracers is not a trivial endeavor, and requires enabling water isotopes as well. Instructions for doing both can be found in the "instructions" directory, with the "water_tags" directory containing both instructions and the associated modifications required for two example water tracers (land sources and a North Pacific source). Please let me know if you run into any issues with anything.

Finally, if you absolutely need to have CAM6 instead of CAM5, then feel free to send me an email (nusbaume@ucar.edu) and we can discuss possible arrangements for doing the experiments you are interested in.

Thanks, and good luck!

 

[huazhen]

Hi Jesse Nusbaumer,

Thanks a lot for your reply. I have one more question. We have previously installed the CESM1.2.2 version. Is this iCESM1.2 different from the normal version CESM1.2? Do I need to reinstall iCESM1.2? Thanks a lot.

Kind regards,
Huazhen

 

[nusbaume]

Hi Huazhen,

Yes, you will need to install/port iCESM1.2 to whatever machine you are using, as this version of the model is the one that contains all of the water tracer code (and trying to add it to the CESM1.2.2 version you already have would involve transferring and modifying many different source code files).

That being said, the underlying code base for iCESM1.2 is CESM1.2.0.1, which is quite similar to CESM1.2.2. So you should be able to port iCESM1.2 to your machine in the same way you ported CESM1.2.2.

Hope that helps!

 

[huazhen]

Hi Jesse Nusbaumer,

Thanks for your help. I have installed iCESM1.2 successfully as you suggested.

When I was trying to use the "iclm4_IC_create.ncl" script as suggested in ' water_isotope_instructions.txt ', I got the following error messages. But I have no idea how to solve it. Do you have any suggestions? Thanks a lot. (I have changed file path in lines 23,24,29,672,676,793 of the "iclm4_IC_create.ncl" script.)

$ ncl iclm4_IC_create.ncl

Copyright (C) 1995-2012 - All Rights Reserved
University Corporation for Atmospheric Research
NCAR Command Language Version 6.1.0
The use of this software is governed by a License Agreement.
See NCAR Command Language (NCL) for more details.
(0) 1.031885
(0) 0.6078135
(0) 1.003541
(0) 0.949131
warning:VarVarWrite: lhs has dimension name and rhs doesn't, deleting name of lhs dimension number(0), use "(/../)" if this is not desired outcome
warning:["Execute.c":7743]:Execute: Error occurred at or near line 312 in file iclm4_IC_create.ncl

warning:VarVarWrite: lhs has dimension name and rhs doesn't, deleting name of lhs dimension number(0), use "(/../)" if this is not desired outcome
warning:["Execute.c":7743]:Execute: Error occurred at or near line 313 in file iclm4_IC_create.ncl

warning:VarVarWrite: lhs has dimension name and rhs doesn't, deleting name of lhs dimension number(0), use "(/../)" if this is not desired outcome
warning:["Execute.c":7743]:Execute: Error occurred at or near line 336 in file iclm4_IC_create.ncl

warning:VarVarWrite: lhs has dimension name and rhs doesn't, deleting name of lhs dimension number(0), use "(/../)" if this is not desired outcome
warning:["Execute.c":7743]:Execute: Error occurred at or near line 337 in file iclm4_IC_create.ncl

warning:VarVarWrite: lhs has dimension name and rhs doesn't, deleting name of lhs dimension number(0), use "(/../)" if this is not desired outcome
warning:["Execute.c":7743]:Execute: Error occurred at or near line 340 in file iclm4_IC_create.ncl

warning:VarVarWrite: lhs has dimension name and rhs doesn't, deleting name of lhs dimension number(0), use "(/../)" if this is not desired outcome
warning:["Execute.c":7743]:Execute: Error occurred at or near line 341 in file iclm4_IC_create.ncl

warning:VarVarWrite: lhs has dimension name and rhs doesn't, deleting name of lhs dimension number(0), use "(/../)" if this is not desired outcome
warning:["Execute.c":7743]:Execute: Error occurred at or near line 344 in file iclm4_IC_create.ncl
......


Kind regards,
Huazhen

 

[nusbaume]

Hi Huazhen,

those warnings shouldn't actually impact the final result, but just in case I have uploaded an updated version to github which should no longer have any warning messages. This new script also has slightly re-named and re-arranged path variables to hopefully prevent the need to change the paths in so many different locations. If for some reason this new script doesn't work for you please let me know.

Thanks, and have a great day!

Jesse

 

[nusbaume]

Hi Huazhen,

Just FYI, another user pointed out that there was a bug in the example water tracer "user_nl_cam" file. I have uploaded the correct file to github, but you might need to replace your current local copy with the new one, or else you'll potentially run into a bunch of errors during run time.

Sorry about that, and good luck with the simulation!

Jesse

 

[huazhen]

Hi Jesse,

I also discovered this problem yesterday, so I made some changes in the file. And the case is still being tested. Now you prove that I did it right. Thank you very much for your help.

Kind regards,
Huazhen

 

[huazhen]

Hi Jesse,

When I was trying to run the example water tracer case, I got the following error message (will attach full log file below). I have changed the second number in the line " sed 's/DPCNST=50/DPCNST=71/g' < CCSM_cppdefs > CCSM_cppdefs_tmp " as suggested in point 3 in ' water_tag_instructions.txt ', but the problem is still not solved. ( I have tried changing the second number from 71 to 85, 92 and 113. But all cases gave me the following error message. ) Do you have any suggestions? Thanks a lot.

---------------------------------------------

WTRC_REGISTER: done.

CNST_ADD: advected tracer index greater than pcnst = 57
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
forrtl: error (78): process killed (SIGTERM)

Kind regards,
Huazhen

 

[nusbaume]

Hi Huazhen,

It looks like there was some issue with the example "cam.buildnml.csh" file. So I have gone ahead and updated the github repo so that instead of copying over the csh file, you just need to add the line:

"-water_tag_num 2"

to the "CAM_CONFIG_OPTS" variable in your case's "env_build.xml" file (which will sadly require a re-build of your case). I have also updated the water_tag_instructions.txt file to describe this new method.

I tested this new method on Cheyenne and it fixed the issue for me, so it should hopefully work for you as well. Of course if you are still having issues just let me know.

Thanks, and have a good weekend!

Jesse

 

[huazhen]

Hi Jesse,

Thanks a lot for all your help. The iCESM1.2 works very well with F compset of using CLM4. But when I was trying to create a new case of using CLM45 (2000_CAM5_CLM45%SP_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV), I got the following error messages in the case build process (will attach full log file below). I haven't found a solution yet. Do you have any advice? Normally, the iCESM1.2 model can run with CLM45, right? Thanks a lot.

ERROR: clm.buildexe.csh failed, see /scratch/L.chou/cesm1/12_test/bld/lnd.bldlog.200312-074650
ERROR: cat /scratch/L.chou/cesm1/12_test/bld/lnd.bldlog.200312-074650

Kind regards,
Huazhen

 

[nusbaume]

Hi Huazhen

Sadly the isotopic tracer physics in CLM only works with version 4.0. However, if you are only interested in water tracers (not isotopes), and are only planning to do F-compset runs (i.e. no prognostic ocean), then you might be able to simply replace the CLM source code in your iCESM1.2 copy with the CLM code present in your CESM1.2.2 copy. You also will need to replace all references to

index_x2a_Faxx_evap_16O

index_x2a_Faxx_evap_HDO

index_x2a_Faxx_evap_18O

with just

index_x2a_Faxx_evap

in your "atm_comp_mct.F90" file.

If everything works then you should be able to run with CLM4.5 while still using water tracers in CAM5. Of course if not just let me know.

Thanks, and have a great day!

Jesse

 

[huazhen]

Hi Jesse,

Thanks for your suggestion. I have replaced the CLM source code (the location is /iCESM1.2/models/lnd/clm/src, right?) and all three references you list in "atm_comp_mct.F90" file. But the case crashed with the following error in log file at run time (will attach full log file below). Does this mean I need to update the input surface data or to use another surface data? Do you have any advice? Thanks a lot. Have a great day!

Attempting to read surface boundary data .....
(GETFIL): attempting to find local file surfdata_0.9x1.25_simyr2000_c120319.nc
(GETFIL): using
/data/cesm1/inputdata/lnd/clm2/surfdata/surfdata_0.9x1.25_simyr2000_c120319.nc
check_var: variable xc is not on dataset
surfrd_get_data lon_var = LONGXY lat_var =LATIXY
domain_clean: cleaning 288 192
PCT_URBAN is not multi-density, pcturb set to 0
surfrd_wtxy_veg_all ERROR: sum(pct) over numpft+1 is not = 100.
97.5322035197224 -2.46779648027756 105
ENDRUN: called without a message string
surfrd_wtxy_veg_all ERROR: sum(pct) over numpft+1 is not = 100.
99.6547511040790 -0.345248895920989 759
ENDRUN: called without a message string
surfrd_wtxy_veg_all ERROR: sum(pct) over numpft+1 is not = 100.
95.7836383298949 -4.21636167010512 1085
ENDRUN: called without a message string
surfrd_wtxy_veg_all ERROR: sum(pct) over numpft+1 is not = 100.
99.2935398845687 -0.706460115431341 2069
ENDRUN: called without a message string
surfrd_wtxy_veg_all ERROR: sum(pct) over numpft+1 is not = 100.
98.8585016698492 -1.14149833015082 1414
ENDRUN: called without a message string
surfrd_wtxy_veg_all ERROR: sum(pct) over numpft+1 is not = 100.
99.4343039275457 -0.565696072454330 2404
ENDRUN: called without a message string
surfrd_wtxy_veg_all ERROR: sum(pct) over numpft+1 is not = 100.
99.8800894372861 -0.119910562713926 429
ENDRUN: called without a message string
surfrd_wtxy_veg_all ERROR: sum(pct) over numpft+1 is not = 100.
99.7405914737836 -0.259408526216390 4042
ENDRUN: called without a message string
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI COMMUNICATOR 9 CREATE FROM 0
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

Kind regards,
Huazhen

 

[nusbaume]

Hi Huazhen,

Yes, this error indicates that the inputs to CLM4.5 are somehow incorrect, and thus resulting in a PFT percentage total that is not equal to 100%. So, if possible I would recommend updating your input files to ones that have worked for you before on your machine (using your original CESM1.2.2 version). You could also try replacing the entire "clm" or "lnd" directories with your CESM1.2.2 version as well, to make sure this issue isn't being caused by incorrect build and/or configure scripts.

Otherwise, I would recommend creating a post on the land model forum (https://xenforo.cgd.ucar.edu/cesm/forums/ctsm-clm-mosart-rtm.134/) to see if anyone there has any suggestions.

Sadly, beyond that I am not much help, as this is specifically a CLM4.5 issue, and not a CAM or water tracer issue. Sorry!

Good luck, and have a great day!

Jesse

 

[huazhen]

Hi Jesse,

Thanks for all your help. I finally get some progress in running iCESM1.2. But I still have one more question.

I submitted two sets of simulations, one with 16 water tracers and one without water tracer. When I compared the outputs of the two simulations, I found the outputs seem very strange (will attach outputs below). In my mind, for those variables not related to water tracers, the outputs of these two simulations should be identical. But the results are weird, they are very different. I haven't figured out why. Maybe there is a problem when I'm trying to modify the code to add water tracers (also included in attach file). Can you take a look of it? Do you have any ideas about possible causes? Thanks a lot.

Kind regards,
Huazhen

 

[nusbaume]

Hi Huazhen,

Apologies for the long delay! I was pulled away from work due to family issues, but everything should be back to normal now.

You are correct that adding water tracers shouldn't have an impact on the underlying physical climate, and I ran a test myself to make sure that was true.

So, I suspect the way the two simulations you ran were configured or modified is what is driving the difference. Sadly it doesn't look like anything is wrong based off the code-snippets you sent. Could you send me copies of all the source code files you modified, along with any XML or user_nl_XXX files you modified as well?

Finally, did you compare an iCESM1.2 run with water tracers to an iCESM1.2 run without water tracers, or did you compare an iCESM1.2 run to another, separate CESM1.2 model run? Depending on which model version(s) you use there could be underlying differences that are independent of the water tracers themselves.

Anyways, sorry I couldn't be of more immediate help, but hopefully with the additional files we can figure out what exactly is going on.

Thanks, and have a great day!

Jesse

 

[XinhaoSuo]

Hi Huazhen,

Apologies for the long delay! I was pulled away from work due to family issues, but everything should be back to normal now.

You are correct that adding water tracers shouldn't have an impact on the underlying physical climate, and I ran a test myself to make sure that was true.

So, I suspect the way the two simulations you ran were configured or modified is what is driving the difference. Sadly it doesn't look like anything is wrong based off the code-snippets you sent. Could you send me copies of all the source code files you modified, along with any XML or user_nl_XXX files you modified as well?

Finally, did you compare an iCESM1.2 run with water tracers to an iCESM1.2 run without water tracers, or did you compare an iCESM1.2 run to another, separate CESM1.2 model run? Depending on which model version(s) you use there could be underlying differences that are independent of the water tracers themselves.

Anyways, sorry I couldn't be of more immediate help, but hopefully with the additional files we can figure out what exactly is going on.

Thanks, and have a great day!

Jesse

Hi Jesse，

I am a beginner of CESM and I am learning about Water-Source-Tagging recently. I found you answered the question in 2020.May I ask whether The water tracer feature is still in development for CAM6？Looking forward to your reply.

 

[nusbaume]

Hi Xinhao,

The water-tracer version of CAM6 is still under development, which means that the code base is currently in a relatively old, non-validated state, and is still very likely to receive additional code modifications in the future.

However, people have been using it to simulate moisture sources with some success. If you are willing to accept the risks of using developmental code then feel free to try it yourself. You can acquire the code by running the following terminal commands:

git clone https://github.com/nusbaume/CAM.git water_tracer_CAM6
cd water_tracer_CAM6
git checkout -b geotrace_cam
./manage_externals/checkout_externals

After that you can go to the instructions/water_tags directory to find instructions and some examples on how to set-up water tracers yourself.

Finally, if you need to have a validated physical climate, and are not sure if you'll have the time to validate the simulated climate yourself, then I would recommend sticking with iCESM1.2, which has been much more thoroughly validated and tested.

Hope that helps, and have a great day!

Jesse

 

[XinhaoSuo]

Hi Jesse，

Thanks for your advice,that helped me a lot and I will use iCES1.2.

Xinhao

 

[XinhaoSuo]

Hi, Jesse
I have some problems about radiative forcing.In the CAM tutorial says:
Removing species from modes requires us to create new mode definitions. Using black carbon as a specific example, we see from the default definitions of the trop_mam4 modes that black carbon is contained in mam4_mode1 and mam4_mode4.The best way to create the definition of a new mode which doesn’t contain black carbon is to copy the definition of modes 1 and 4, change their names, and remove the black carbon from the definition.

However in the example of Black carbon radiative forcing the bc didn't remove from the mam4_mode4:
mode_defs =
'mam4_mode1:accum:=',
'A:num_a1:N:num_c1:num_mr:+',
'A:so4_a1:N:so4_c1:sulfate:/fs/cgd/csm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+',
'A:pom_a1:N:pom_c1:p-organic:/fs/cgd/csm/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+',
'A:soa_a1:N:soa_c1:s-organic:/fs/cgd/csm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+',
'A:bc_a1:N:bc_c1:black-c:/fs/cgd/csm/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc:+',
'A:dst_a1:N:dst_c1:dust:/fs/cgd/csm/inputdata/atm/cam/physprops/dust_aeronet_rrtmg_c141106.nc:+',
'A:ncl_a1:N:ncl_c1:seasalt:/fs/cgd/csm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc',
'mam4_mode2:aitken:=',
'A:num_a2:N:num_c2:num_mr:+',
'A:so4_a2:N:so4_c2:sulfate:/fs/cgd/csm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+',
'A:soa_a2:N:soa_c2:s-organic:/fs/cgd/csm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+',
'A:ncl_a2:N:ncl_c2:seasalt:/fs/cgd/csm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc:+',
'A:dst_a2:N:dst_c2:dust:/fs/cgd/csm/inputdata/atm/cam/physprops/dust_aeronet_rrtmg_c141106.nc',
'mam4_mode3:coarse:=',
'A:num_a3:N:num_c3:num_mr:+',
'A:dst_a3:N:dst_c3:dust:/fs/cgd/csm/inputdata/atm/cam/physprops/dust_aeronet_rrtmg_c141106.nc:+',
'A:ncl_a3:N:ncl_c3:seasalt:/fs/cgd/csm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc:+',
'A:so4_a3:N:so4_c3:sulfate:/fs/cgd/csm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc',
'mam4_mode4:primary_carbon:=',
'A:num_a4:N:num_c4:num_mr:+',
'A:pom_a4:N:pom_c4:p-organic:/fs/cgd/csm/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+',
'A:bc_a4:N:bc_c4:black-c:/fs/cgd/csm/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc',
'mam4_mode1_nobc:accum:=',
'A:num_a1:N:num_c1:num_mr:+',
'A:so4_a1:N:so4_c1:sulfate:/fs/cgd/csm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+',
'A:pom_a1:N:pom_c1:p-organic:/fs/cgd/csm/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+',
'A:soa_a1:N:soa_c1:s-organic:/fs/cgd/csm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+',
'A:dst_a1:N:dst_c1:dust:/fs/cgd/csm/inputdata/atm/cam/physprops/dust_aeronet_rrtmg_c141106.nc:+',
'A:ncl_a1:N:ncl_c1:seasalt:/fs/cgd/csm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc',
'mam4_mode4_nobc:primary_carbon:=',
'A:num_a4:N:num_c4:num_mr:+',
'A:pom_a4:N:pom_c4:p-organic:/fs/cgd/csm/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+',
'A:bc_a4:N:bc_c4:black-c:/fs/cgd/csm/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc'

So , Is there a problem with the code?

Xinhao