Running with your own atmosphere forcing

[MrIgnition]

Hi all,
I am trying to run CLM5 using my own atmospheric forcing. I followed the guide on http://www.cesm.ucar.edu/models/cesm1.2/clm/models/lnd/clm/doc/UsersGuide/x12979.html but somethings still appear unclear.
1. the file link_dirtree does not seem to exist in my scripts directory
2. the guide recommends data path as $MYCSMDATA/atm/datm7/CLM1PT_data/$MYUSRDAT but after case submission the directory checked was $MYCSMDATA/atm/datm7/$MYUSRDAT/CLM1PT_data/.
3. the file naming convention YYYY-MM.nc as instructed does not signify variable or field name. How will the files belonging to a stream be identified.

Please I will appreciate it if anyone who has used his/her own atmospheric forcing for a regional simulation or any other person who is able to please assist in this.
Thanks.

 

[oleson]

Those instructions are for an earlier version of the model, so some things will not work with the current version.
The file link_dirtree was removed quite a while ago. Alternatives are to either put your new data in the "inputdata" directory on your machine ($CSMDATA), or preferably set up a "mydatafiles" directory within your code directory (or elsewhere). For example, my atmospheric data for single point simulations is here:

$CTSMROOT/tools/PTCLM/mydatafiles/1x1pt_US-UMB/CLM1PT_data

For single point simulations, since all of the forcing data is at the same time resolution as the model timestep (e.g., 1/2 hourly), all the of the forcing data is contained within one file (e.g., 1980-01.nc) and hence is all in the above directory.

If your atmospheric data is similar to that in GSWP3 or CRUNCEP, etc., then there are separate streams depending on the nature of the variable. E.g,, solar radiation, precipitation, and other variables are treated differently. Thus, these files should be in separate subdirectories, e.g.,

Solar/1980-01.nc
Precip/1980-01.nc
TPHWL/1980-01.nc

And then you would point to these directories explicitly in your streams files:

datm.streams.txt.CLMGSWP3v1.Solar
datm.streams.txt.CLMGSWP3v1.Precip
datm.streams.txt.CLMGSWP3v1.TPQW

 

[MrIgnition]

Thanks for this guide. I will try this out. I have a question please regarding the stream files. Which of the streamfiles should be edited? The ones in the caseroot/Buildconf or the ones on the rundir? My experience is that the streamfiles get overwritten once the case is submitted.

 

[oleson]

Don't change the files in caseroot/Buildconf or the ones in the rundir.
You need to make "user" copies of the ones in CaseDocs and put them in your case directory, e.g., do this in your run directory:

cp CaseDocs/datm.streams.txt.CLMGSWP3v1.Precip user_datm.streams.txt.CLMGSWP3v1.Precip

and then edit the user_* files. Running preview_namelists will use those and generate new streams files for your run.

 

[MrIgnition]

Thank you so much for this. I'm grateful.

 

[MrIgnition]

I have now made the user copies and edited them as directed. The case submits successfully now. But I keep getting:
ERROR:
(shr_stream_getCalendar) ERROR: nf90_open file /p/scratch/cjicg41/bamidele/inp
utdata/atm/datm7/atm_forcing.datm7.ERA5BC.0.5d.bc.c200316/CLM1PT_data/TPHWL1Hrl
y/2000-01.nc

I have attempted converting the netcdf files to Classic and nc4 but the error keeps coming. I also used both cdo setcalendar and cdo setreftime to set the calendar to 365 days calendar and reference time to 0000-01-01 respectively. What could I have done wrong please?

 

[MrIgnition]

Here is my file header:
netcdf 2000-01 {
dimensions:
time = UNLIMITED ; // (744 currently)
lon = 720 ;
lat = 360 ;

variables:
double time(time) ;
time:standard_name = "time" ;
time:long_name = "Time" ;
time:units = "hours since 0-1-1 00:00:00" ;
time:calendar = "365_day" ;
time:axis = "T" ;
double lon(lon) ;
lon:standard_name = "longitude" ;
lon:long_name = "Longitude" ;
lon:units = "degrees_east" ;
lon:axis = "X" ;
double lat(lat) ;
lat:standard_name = "latitude" ;
lat:long_name = "Latitude" ;
lat:units = "degrees_north" ;
lat:axis = "Y" ;
float Tair(time, lat, lon) ;
Tair:standard_name = "air_temperature" ;
Tair:long_name = "Near-Surface Air Temperature" ;
Tair:units = "K" ;
Tair:_FillValue = 1.e+20f ;
Tair:missing_value = 1.e+20f ;
float PSurf(time, lat, lon) ;
PSurf:standard_name = "surface_air_pressure" ;
PSurf:long_name = "Surface Air Pressure" ;
PSurf:units = "Pa" ;
PSurf:_FillValue = 1.e+20f ;
PSurf:missing_value = 1.e+20f ;
float Qair(time, lat, lon) ;
Qair:standard_name = "specific_humidity" ;
Qair:long_name = "Near-Surface Specific Humidity" ;
Qair:units = "kg kg-1" ;
Qair:_FillValue = 1.e+20f ;
Qair:missing_value = 1.e+20f ;
float LWdown(time, lat, lon) ;
LWdown:standard_name = "surface_downwelling_longwave_flux_in_air" ;
LWdown:long_name = "Surface Downwelling Longwave Radiation" ;
LWdown:units = "W m-2" ;
LWdown:_FillValue = 1.e+20f ;
LWdown:missing_value = 1.e+20f ;
float Wind(time, lat, lon) ;
Wind:standard_name = "wind_speed" ;
Wind:long_name = "Near-Surface Wind Speed" ;
Wind:units = "m s-1" ;
Wind:_FillValue = 1.e+20f ;
Wind:missing_value = 1.e+20f ;

// global attributes:
:CDI = "Climate Data Interface version 1.9.5 (Overview - CDI - Project Management Service)" ;
:Conventions = "CF-1.7" ;
:history = "Sat Mar 28 07:12:56 2020: cdo setreftime,0000-01-01,0,1h -setcalendar,365_day n2000-01.nc mn2000-01.nc\n",
"Fri Mar 27 23:05:01 2020: cdo setreftime,0000-01-01,0,1h 2000-01.nc n2000-01.nc\n",
"Tue Mar 17 09:52:52 2020: cdo -P 10 merge Tair_WFDE5_CRU_200001_v1.0.nc PSurf_WFDE5_CRU_200001_v1.0.nc Qair_WFDE5_CRU_200001_v1.0.nc LWdown_WFDE5_CRU_200001_v1.0.nc Wind_WFDE5_CRU_200001_v1.0.nc TPQLW_WFDE5_CRU_200001_v1.0.nc" ;
:institution = "Copernicus Climate Change Service" ;
:title = "WATCH Forcing Data methodology applied to ERA5 data" ;
:contact = "http://copernicus-support.ecmwf.int" ;
:summary = "ERA5 data regridded to half degree regular lat-lon; Genuine land points from CRU grid and ERA5 land-sea mask only; Tair elevation & bias-corrected using CRU TS4.03 mean monthly temperature and mean diurnal temperature range" ;
:comment = "Methodology implementation for ERA5 and dataset production by B-Open Solutions for the Copernicus Climate Change Service in the context of contract C3S_25c" ;
:reference = "Weedon et al. 2014 Water Resources Res. 50, 7505-7514, doi:10.1002/2014WR015638; Harris et al. 2014 Int. J. Climatol. 34, 623-642, doi:10.1002/joc.3711; Cucchi et al. 2020 (in preparation)" ;
:licence = "The dataset is distributed under the Licence to Use Copernicus Products. The corrections applied are based upon CRU TS4.03, distributed under the Open Database License (ODbL)" ;
:cdo_openmp_thread_number = 10 ;
:CDO = "Climate Data Operators version 1.9.5 (Overview - CDO - Project Management Service)" ;

 

[MrIgnition]

This Error has now been resolved. There was a typo in my file path and my time units have been corrected accordingly. Thanks.

 

[kaeritai]

Those instructions are for an earlier version of the model, so some things will not work with the current version.
The file link_dirtree was removed quite a while ago. Alternatives are to either put your new data in the "inputdata" directory on your machine ($CSMDATA), or preferably set up a "mydatafiles" directory within your code directory (or elsewhere). For example, my atmospheric data for single point simulations is here:

$CTSMROOT/tools/PTCLM/mydatafiles/1x1pt_US-UMB/CLM1PT_data

For single point simulations, since all of the forcing data is at the same time resolution as the model timestep (e.g., 1/2 hourly), all the of the forcing data is contained within one file (e.g., 1980-01.nc) and hence is all in the above directory.

Hi dear Oleson,
I am now working on a flux tower site too and I have another two questions for this case, could you please help me out?
1, should the forcing data be in seprate months (such as 1980-01.nc, 1980-02.nc ...) or a single NC file? Is one single NC file for different meterological variables OK?
2, is there any requirement for the variable of 'Time' in the created NC file? I mean should it be "cdo setcalendar and cdo setreftime to set the calendar to 365 days calendar and reference time to 0000-01-01 " as MrIgnition mentioned above?

 

[oleson]

If all of the forcing data is at the same time resolution as the model timestep (e.g., 1/2 hourly), all of the forcing data can be contained within one file
If your atmospheric data is at a time resolution that is coarser than the model time step, e.g., the global GSWP3V1 data which is 3-hourly, then the forcing data will need to be divided into separate files because the needed interpolation methods are different.
In general, the units on the time coordinate should be of the form "days since YYYY-MM-DD HH:MM:SS". The YYYY could either be 0001, in which case you'll need to determine the number of days until your first time slice of data, or it could be the actual start year. If you don't have any leap day forcing, then you should set calendar = "noleap", otherwise set calendar = "gregorian".

 

[kaeritai]

If all of the forcing data is at the same time resolution as the model timestep (e.g., 1/2 hourly), all of the forcing data can be contained within one file
If your atmospheric data is at a time resolution that is coarser than the model time step, e.g., the global GSWP3V1 data which is 3-hourly, then the forcing data will need to be divided into separate files because the needed interpolation methods are different.
In general, the units on the time coordinate should be of the form "days since YYYY-MM-DD HH:MM:SS". The YYYY could either be 0001, in which case you'll need to determine the number of days until your first time slice of data, or it could be the actual start year. If you don't have any leap day forcing, then you should set calendar = "noleap", otherwise set calendar = "gregorian".

hi dear Oleson,
thank you very much for your detailed response; I undstand and try to run the model then.

 

[mengqi]

Those instructions are for an earlier version of the model, so some things will not work with the current version.
The file link_dirtree was removed quite a while ago. Alternatives are to either put your new data in the "inputdata" directory on your machine ($CSMDATA), or preferably set up a "mydatafiles" directory within your code directory (or elsewhere). For example, my atmospheric data for single point simulations is here:

$CTSMROOT/tools/PTCLM/mydatafiles/1x1pt_US-UMB/CLM1PT_data

For single point simulations, since all of the forcing data is at the same time resolution as the model timestep (e.g., 1/2 hourly), all the of the forcing data is contained within one file (e.g., 1980-01.nc) and hence is all in the above directory.

If your atmospheric data is similar to that in GSWP3 or CRUNCEP, etc., then there are separate streams depending on the nature of the variable. E.g,, solar radiation, precipitation, and other variables are treated differently. Thus, these files should be in separate subdirectories, e.g.,

Solar/1980-01.nc
Precip/1980-01.nc
TPHWL/1980-01.nc

And then you would point to these directories explicitly in your streams files:

datm.streams.txt.CLMGSWP3v1.Solar
datm.streams.txt.CLMGSWP3v1.Precip
datm.streams.txt.CLMGSWP3v1.TPQW

Dear Oleson,

Thanks for your clarification. Given that no detailed info and I am a rookie, however, I have a few questions here. I really appreciate it if you could give me some guidance.

(a) I noticed that there is a keyword "PTCLM" in your directory. Not sure if I need to run PTCLM mode when setting up a "mydatafiles" directory. Or I just set up the "mydatafiles" directory regardless of running PTCLM mode?

(b) You mentioned "You need to make "user" copies of the ones in CaseDocs and put them in your case directory. And then edit the user_* files. Running preview_namelists will use those and generate new streams files for your run" In my case, the default in CaseDocs/datm.streams.xml is GSWP3 data instead of my data high-resolution data. Does that mean that I should first revise CaseDocs and then put them in user_datm.streams.xml?

(c) Last, regarding GSWP3 data, their format in CaseDocs/datm.streams.xml is something like this below. However, I cannot find out the format of one file (e.g., 1980-01.nc)? Could you please show me some code format for "one file" in the datm.streams.xml and/or user_datm.streams.xml? In this case, I can easily understand and use them.

-----------------------------------
<?xml version="1.0"?>
<file id="stream" version="2.0">

<stream_info name="CLMGSWP3v1.Solar">
<taxmode>cycle</taxmode>
<tintalgo>coszen</tintalgo>
<readmode>single</readmode>
<mapalgo>none</mapalgo>
<dtlimit>1.5</dtlimit>
<year_first>2000</year_first>
<year_last>2005</year_last>
<year_align>2000</year_align>
<vectors>null</vectors>
<meshfile>none</meshfile>
<lev_dimname>null</lev_dimname>
<datafiles>
<file>/glade/scratch/mengqij/my_subset_data_Bo1_c/datmdata/clmforc.GSWP3.c2011.0.5x0.5.Solr.Bo1_point_c.2000-01.nc</file>
<file>/glade/scratch/mengqij/my_subset_data_Bo1_c/datmdata/clmforc.GSWP3.c2011.0.5x0.5.Solr.Bo1_point_c.2000-02.nc</file>

............

<stream_info name="CLMGSWP3v1.Precip">
<taxmode>cycle</taxmode>
<tintalgo>nearest</tintalgo>
<readmode>single</readmode>
<mapalgo>none</mapalgo>
<dtlimit>1.5</dtlimit>
<year_first>2000</year_first>
<year_last>2005</year_last>
<year_align>2000</year_align>
<vectors>null</vectors>
<meshfile>none</meshfile>
<lev_dimname>null</lev_dimname>
<datafiles>
<file>/glade/scratch/mengqij/my_subset_data_Bo1_c/datmdata/clmforc.GSWP3.c2011.0.5x0.5.Prec.Bo1_point_c.2000-01.nc</file>
<file>/glade/scratch/mengqij/my_subset_data_Bo1_c/datmdata/clmforc.GSWP3.c2011.0.5x0.5.Prec.Bo1_point_c.2000-02.nc</file>

..............

 

[oleson]

You didn't say what version of the model you are using but from your mention of datm.streams.xml I'm guessing you are using a very recent version of the model for which the instructions are different. There are some instructions here:

How do I create a single-point run with the NUOPC (default) coupler? · ESCOMP CTSM · Discussion #1833

We get this question a lot: I am trying to set up a case for a specific latitude and longitude. I want to use GSWP3 climate data, how do I set this up?

github.com

If you need to modify the datm streams further, then you would modify them in user_nl_datm_streams which is a file created in your case directory after you've run ./case.setup. That file has information in it that can serve as a template for changes you might wish to make.

 

[mengqi]

You didn't say what version of the model you are using but from your mention of datm.streams.xml I'm guessing you are using a very recent version of the model for which the instructions are different. There are some instructions here:

How do I create a single-point run with the NUOPC (default) coupler? · ESCOMP CTSM · Discussion #1833

We get this question a lot: I am trying to set up a case for a specific latitude and longitude. I want to use GSWP3 climate data, how do I set this up?

github.com

If you need to modify the datm streams further, then you would modify them in user_nl_datm_streams which is a file created in your case directory after you've run ./case.setup. That file has information in it that can serve as a template for changes you might wish to make.

Hi Oleson,

Thanks for your answer! But I still have a few issues.

First, I am using a recent version of the CLM. And I try to run with my own atmosphere forcing. Note that my forcing data is high resolution (e.g., 1/2 hourly), and thus, all of the forcing data is contained within one file (e.g., 1996-01.nc).

According to your advice, it seems that I just need to modify the datm streams in the user_nl_datm_streams. If so, my question is how to point to these directories explicitly in the user_nl_datm_streams. I am sure that there is ONLY a GSWP3 example (i.e., Solar, Precip, TPHWL) for the instructions you shared. I tried many times in my case but they failed. And I cannot find out any "one file" example online.

Here is my current format in the user_nl_datm_streams:

datafiles= '/glade/u/home/mengqij/my_ctsm_sandbox/tools/mydatafiles/1x1_US_Bo1/1996-01.nc,/glade/u/home/mengqij/my_ctsm_sandbox/tools/mydatafiles/1x1_US_Bo1/1996-02.nc,/glade/u/home/mengqij/my_ctsm_sandbox/tools/mydatafiles/1x1_US_Bo1/1996-03.nc,/glade/u/home/mengqij/my_ctsm_sandbox/tools/mydatafiles/1x1_US_Bo1/1996-04.nc,/glade/u/home/mengqij/my_ctsm_sandbox/tools/mydatafiles/1x1_US_Bo1/1996-05.nc,/glade/u/home/mengqij/my_ctsm_sandbox/tools/mydatafiles/1x1_US_Bo1/1996-06.nc,/glade/u/home/mengqij/my_ctsm_sandbox/tools/mydatafiles/1x1_US_Bo1/1996-07.nc,/glade/u/home/mengqij/my_ctsm_sandbox/tools/mydatafiles/1x1_US_Bo1/1996-08.nc,/glade/u/home/mengqij/my_ctsm_sandbox/tools/mydatafiles/1x1_US_Bo1/1996-09.nc,/glade/u/home/mengqij/my_ctsm_sandbox/tools/mydatafiles/1x1_US_Bo1/1996-10.nc,/glade/u/home/mengqij/my_ctsm_sandbox/tools/mydatafiles/1x1_US_Bo1/1996-11.nc,/glade/u/home/mengqij/my_ctsm_sandbox/tools/mydatafiles/1x1_US_Bo1/1996-12.nc

When I submitted, it states "ERROR: input stream mod can only be of the form streamname:var=value(s)"

Any chance if you could show me a template for "one file" in the user_nl_datm_streams?

Thank you so much!!

Mengqi

 

[oleson]

I've attached a sample file. As the error notes, you are missing "streamname:". In the example file attached it is "presaero.SSP3-7.0".

 

[kaeritai]

This Error has now been resolved. There was a typo in my file path and my time units have been corrected accordingly. Thanks.

hi dear Mrlgnition,
sorry for make this thread up again.
have you succeeded in simulation on single point? I want to simulate on multiple sites (say about 20 Fluxnet/Ameriflux sites), do we have any handy methods? Since I guess following the guidlines of CLM, I may make surface data/domain files and also set up the corresponding cases (20 cases) seperately. The setup of the cases may take much time.
Hope you have any suggestions? Thank you.