high resolution (0.1 degree) CLM5-BGC-crop regional simulation

[xgao304]

Dear Sir:

I am able to run the CLM5-BGC-crop at half degree resolution over the user-defined region (eg., USA) using the following command:

create_newcase --case BgcCrop2000_USA --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_SROF_SGLC_SWAV --res hcru_hcru --user-mods-dir USAdata --machine svante --compiler intel --run-unsupported

since half degree is one of the supported resolutions, it is fairly easy to use "getregional_datasets.pl" command for generating the domain file and surface dataset (included in "USAdata" above) for this case by subsetting from the existing global datasets.

I would like to run the same case but at a very high resolution (0.1 degree). I am wondering how I should generate the corresponding domain and surface datasets at 0.1 degree.
My understanding is to use

cime/tools/mapping/gen_domain_files/gen_domain for creating the domain file, which reads in a conservative ocean->atmosphere mapping file

components/clm/tools/mksurfdata_map/mksurfdata.pl, which requires a scrip grid file to be generated first.

My questions are:

1) for "gen_domain", how to obtain this conservative ocean->atmosphere mapping file at 0.1 degree?
2) for "mksurfdata.pl", how to obtain this scrip grid file at 0.1 degree?

3) Is it really meaningful to run the ctsm at this high resolution, if all the required input data to generate the domain and surface datasets are much coarser than 0.1 degree - I am not sure if this is the case, though. If so, the results are simply interpolated from coarse resolution into this 0.1 degree without much information added. Is this true?

Thanks a lot.

 

[oleson]

For 1) and 2), see the use of mknoocnmap.pl in this post:

Run CLM with my own atmosphere forcing data

Hello, everyone! I used the local atmospheric forcing dataset CMFD(China meteorological forcing dataset; 1978-2018; 0.1°×0.1°;3 hours) to replace GSWP3v1 in the CLM model (1901-2014; 0.5° x 0.5°) data. But I meet some questions. My test was successful if I ues GSWP3v1 data, the code as follows...

bb.cgd.ucar.edu

For 3), it is true that most of the input datasets are at coarser resolution. However, you can specify "-hirespft" in your mksurfdata.pl command to get 0.05 degree pft and lai data as input (only available for present day I bellieve).

 

[xgao304]

Dear Oleson,

Thanks for the information. I checked the post you pointed to, particularly your reply regarding how to use "mknoocnmap.pl" to generate the domain file of specific resolution. My understanding is that command is only used for the region without ocean points (based on the command name). If my area of interest is over Bangledash where the majority of the land points would be ocean, it will not work. Is that correct? If so, how should I generate the domain file? For my second question above, how to generate the corresponding scrip grid used for making the surface data?

Thanks,

Xiang

 

[oleson]

Right, only applicable for a land-only domain. The User's Guide has this to say (1.3.4. Creating input for surface dataset generation — ctsm CTSM master documentation):

If you want to create a regular latitude/longitude single-point or regional grid, we suggest you use mknoocnmap.pl in $CTSMROOT/tools/mkmapdata which will create both the SCRIP grid file you need (using $CTSMROOT/tools/mkmapgrids/mkscripgrid.ncl AND an identity mapping file assuming there is NO ocean in your grid domain. If you HAVE ocean in your domain you could modify the mask in the SCRIP grid file for ocean, and then use ESMF_RegridWeightGen to create the mapping file, and gen_domain to create the domain file.

I haven't tried this myself. Note that mknoocnmap.pl creates a SCRIP grid file as well as a mapping file because it runs mkscripgrid.ncl.
@erik

 

[xgao304]

Dear Oleson,

Thanks for the information. If my understanding is correct, here are the steps I should follow to generate the domain with ocean points.

1. run "mknoocnmap.pl", which generates three files:

map_40x30_BD_noocean_to_40x30_BD_nomask_aave_da_230510.nc
SCRIPgrid_40x30_BD_nomask_c230510.nc
SCRIPgrid_40x30_BD_noocean_c230510.nc

2. modify the mask (variable "grid_imask") in the two scrip grid files so that the values for this variable are a mix of 0 and 1. (not sure how to do that yet, but assume this is successfully done and the new scripgrid files are the following)

SCRIPgrid_40x30_BD_nomask_c230510_v2.nc
SCRIPgrid_40x30_BD_noocean_c230510_v2.nc

3. run "create_ESMF_map.sh (in tools/mapping/gen_mapping_files/gen_ESMF_mapping_file) to create the mapping file

I have questions for the required inputs to "create_ESMF_map.sh":

--filesrc|-fsrc input source grid_filename (required)
--filedst|-fdst input destination grid_filename (required)
--namesrc|-nsrc output source name in mapping file (required)
--namedst|-ndst output destination name in mapping file (required)
--maptype|-map type of mapping [aave|blin|bilin|patc|nearestdtos|neareststod] (required)

What files should I specify for "fsrc","fdst","nsrc","ndst"? For other variables:
maptype : aave
typesrc: regional
typedst: regional

Do I need to specify "serial" "machine"?

4. use "gen_domain" to create the domain file. Are the following commands correct?

export MAPFILE= "fdst" from step 3
export OCNDOM= ?
export ATMDOM= ?
./gen_domain -m $MAPFILE -o $OCNDOM -l $ATMDOM


Thanks.

 

[oleson]

Unfortunately, I'm not sure how to implement this. Maybe @erik or @slevis can help with these steps.
However, it occurred to me that another way to go would be to create global 0.1x0.1 surface dataset and domain files and then subset them using get_regional.pl.
The following steps using release-cesm2.1.3 worked for me:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TO create mapping file for gen_domain
-------------------------------------
qcmd -- ./create_ESMF_map.sh --maptype aave --filesrc /glade/p/cesmdata/cseg/inputdata/share/scripgrids/gx1v7_151008.nc --filedst /glade/p/cesmdata/cseg/inputdata/share/scripgrids/0.1x0.1_090917.nc --namesrc gx1v7 --namedst 0.1x0.1

OUTPUT IS map_gx1v7_TO_0.1x0.1_aave.230512.nc

TO create domain file
---------------------
./gen_domain -m /glade/work/oleson/release-cesm2.1.3/cime/tools/mapping/gen_mapping_files/gen_ESMF_mapping_file/map_gx1v7_TO_0.1x0.1_aave.230512.nc -o gx1v7 -l 0.1x0.1

OUTPUT IS domain.lnd.0.1x0.1_gx1v7.230512.nc

TO create mapping files
-----------------------
setenv RES 0.1x0.1
setenv GRIDFILE /glade/p/cesmdata/cseg/inputdata/lnd/clm2/mappingdata/grids/SCRIPgrid_0.1x0.1_nomask_c110712.nc
qsub ./regridbatch.sh

OUTPUT IS:

map_0.25x0.25_MODIS_to_0.1x0.1_nomask_aave_da_c230512.nc
map_0.5x0.5_AVHRR_to_0.1x0.1_nomask_aave_da_c230512.nc
map_0.5x0.5_MODIS_to_0.1x0.1_nomask_aave_da_c230512.nc
map_0.9x1.25_GRDC_to_0.1x0.1_nomask_aave_da_c230512.nc
map_10x10min_IGBPmergeICESatGIS_to_0.1x0.1_nomask_aave_da_c230512.nc
map_10x10min_nomask_to_0.1x0.1_nomask_aave_da_c230512.nc
map_1km-merge-10min_HYDRO1K-merge-nomask_to_0.1x0.1_nomask_aave_da_c230512.nc
map_360x720cru_cruncep_to_0.1x0.1_nomask_aave_da_c230512.nc
map_3x3min_GLOBE-Gardner-mergeGIS_to_0.1x0.1_nomask_aave_da_c230512.nc
map_3x3min_GLOBE-Gardner_to_0.1x0.1_nomask_aave_da_c230512.nc
map_3x3min_LandScan2004_to_0.1x0.1_nomask_aave_da_c230512.nc
map_3x3min_MODISv2_to_0.1x0.1_nomask_aave_da_c230512.nc
map_3x3min_MODIS-wCsp_to_0.1x0.1_nomask_aave_da_c230512.nc
map_3x3min_USGS_to_0.1x0.1_nomask_aave_da_c230512.nc
map_5x5min_IGBP-GSDP_to_0.1x0.1_nomask_aave_da_c230512.nc
map_5x5min_ISRIC-WISE_to_0.1x0.1_nomask_aave_da_c230512.nc
map_5x5min_nomask_to_0.1x0.1_nomask_aave_da_c230512.nc
map_5x5min_ORNL-Soil_to_0.1x0.1_nomask_aave_da_c230512.nc

TO create surface dataset
-------------------------
setenv CDATE `date +%y%m%d`
setenv GRIDNAME 0.1x0.1
qcmd -- ./mksurfdata.pl -r usrspec -usr_gname $GRIDNAME -usr_gdate $CDATE -hirespft -years 2005

OUTPUT IS:

surfdata_0.1x0.1_hist_78pfts_CMIP6_simyr2005_c230512.nc
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

qcmd is our method of submitting jobs to our compute nodes on cheyenne.
gx1v7 is CESM's standard land/ocean mask.
The required files should be available in inputdata.

I haven't tried the domain and surface datasets in a global simulation but they look reasonable.
Then you could subset them for your domain using get_regional.pl.
I might be missing something of course!

 

[xgao304]

Dear Oleson,

Thanks so much for the very detailed steps to create the domain and surface datasets at 0.1 degree. I will try your way. Xiang

 

[xgao304]

@oleson:

I have one question about the "regridbatch.sh" - the step for creating mappling file. In the downloaded cesm2.1.3 package, I am unable to identify where
this script is stored.

My downloaded package directory content is as follows (contains four files):

[xgao@svante-login gen_mapping_files]$ ll
total 40
-rwxr-xr-x 1 xgao users 15750 Mar 8 2021 gen_cesm_maps.sh
drwxr-xr-x 2 xgao users 172 May 18 17:28 gen_ESMF_mapping_file
-rw-r--r-- 1 xgao users 4763 Mar 8 2021 README
drwxr-xr-x 6 xgao users 4096 Mar 8 2021 runoff_to_ocn

[xgao@svante-login gen_ESMF_mapping_file]$ ll
total 24
-rwxr-xr-x 1 xgao users 11925 May 18 11:20 create_ESMF_map.sh
-rwxr-xr-x 1 xgao users 946 Mar 8 2021 nearestdtos.area_fix.sh
-rw-r--r-- 1 xgao users 1264 Mar 8 2021 overwrite_map_area.ncl
-rw-r--r-- 1 xgao users 3006 Mar 8 2021 README

Could you tell me where I could find this file or share the file with me?

Thanks,

Xiang

 

[oleson]

It should be in components/clm/tools/mkmapdata.

I will usually use the following to find files, running this in the top level directory:

find . -name regridbatch.sh

 

[xgao304]

@oleson:

Thanks for the information.
I ran the command "create_ESMF_map.sh", but get the following error message. I looked through the script. If I don’t specify source grid type –typesrc or -tsrc, the script uses default value “global”. I am not sure why it gives the error message. I found a thread and your reply on this forum Error when excute mkmapdata.sh :unable to detect the source grid file type
It uses different command but with the same error message. Our system Ad just installed ESMF module 8.3.0. So could the error be related to memory "> 400 GB)?

./create_ESMF_map.sh --machine svante --maptype aave --filesrc /net/fs01/data/cesm2/inputdata/share/scripgrids/gx1v7_151008.nc --filedst /net/fs01/data/cesm2/inputdata/share/scripgrids/0.1x0.1_090917.nc --namesrc gx1v7 --namedst 0.1x0.1

./create_ESMF_map.sh

Fri May 19 15:29:50 EDT 2023

map_type is aave

fsrc is /net/fs01/data/cesm2/inputdata/share/scripgrids/gx1v7_151008.nc

fdst is /net/fs01/data/cesm2/inputdata/share/scripgrids/0.1x0.1_090917.nc

Creating map_gx1v7_TO_0.1x0.1_aave.230519.nc...

cmd is /home/software/intel/intel-2017.0.1/pkg/esmf/8.3.0/bin/binO/Linux.intel.64.mpiuni.default//ESMF_RegridWeightGen --ignore_unmapped -m conserve -w map_gx1v7_TO_0.1x0.1_aave.230519.nc -s /net/fs01/data/cesm2/inputdata/share/scripgrids/gx1v7_151008.nc -d /net/fs01/data/cesm2/inputdata/share/scripgrids/0.1x0.1_090917.nc

ERROR: Unable to detect the source grid file type.

Error status returned from create_ESMF_map script


Thanks,

Xiang

 

[oleson]

The default global is what we want here, because we are creating global files that will be subset for your regional domain later.
Does this file exist and does it have non-zero size:

/net/fs01/data/cesm2/inputdata/share/scripgrids/gx1v7_151008.nc

The qcmd command we use submits the job to a compute node, so that we don't overwhelm a login node that is shared.
You could try adding --serial to the command.

I've put the mapping file on our ftp site if that helps:

ftp://ftp.cgd.ucar.edu/pub/oleson/map_gx1v7_TO_0.1x0.1_aave.230512.nc.gz

 

[xgao304]

@oleson:
The file exists and have non-zero size:

[xgao@svante-login scripgrids]$ ls -al
total 542963
-rw-r--r-- 1 xgao users 544320736 May 17 22:36 0.1x0.1_090917.nc
-rw-r--r-- 1 xgao users 11305792 May 17 11:10 gx1v7_151008.nc

I also tried to submit the job to a compute node with option "--mem=0", but gets the same error message.
I will download the mapping file so that I could continue to the next step. However, it is good if we could
figure out the reason just for any application in the future.

Thanks,

Xiang

 

[xgao304]

@oleson:

Somehow, the file is not accessible. I tried to use ftp on linux, it is just hanging there.

[xgao@svante-login scripgrids]$ ftp
ftp> open ftp.cgd.ucar.edu
ftp: connect: Connection timed out
ftp>

I noticed that accessing the inputdata via ftp (https://www.cesm.ucar.edu/models/cesm2 - "Anonymous FTP" link at the bottom of the page) does not work either (It workedbefore if I recall correctly).

The web browse (I tried firefox, chrome, and safari) does not work either. Could you check the link again or is there any other way?

Thanks,

Xiang

 

[oleson]

Can you get to it through a web browser from this?

Index of /pub/oleson

 

[xgao304]

@oleson:

That works. Thank you very much.

 

[pesieber]

Hi Keith,
I also needed 0.1° regional surface data with the -hirespft option, so I followed your instructions above on cheyenne and succesfully created a 0.1° global surface dataset for 2005 that I clipped to the regional domain. I have a few questions:

1) Is the -hirespft option only for the year 2005, not for other individual years and not for periods of the recent past? (in the docs and in mksurfdata.pl it says "hires only available for present-day [2000]", but it does not work for year 2000)

2) I tried using -years 2005-2015 (which I assume would create a time dimension on the PFT fractions, and nothing else?) but it failed with "Invalid simulation simulation year range: 2005-2015", although 2005-2015 is stated as the default in mksurfdata.pl by $opts{'years'}) - do you know why?

3) Is the only option of using LAI streams by interpolation from the 0.5° dataset? (i.e. there is no -hires alternative)

Thanks for your help!
Petra

 

[oleson]

1) Only available for year 2005. The year "2000" is used generically in the surface dataset notation to denote "present-day". It's possible that it was available for 2000 at some point but then got updated to 2005 and the mksurdata.pl --help was not updated.
2) I don't see "2005-2015" in mksurfdata.pl in either the supported release (release-cesm2.1.3) or development versions (release-cesm2.2.0)? Regardless, if you request 1850-2000, you will actually get 1850-2015. Again, the datasets were probably extended from 2000 to 2015 at some point and the mksurfdata.pl --help wasn't updated.
3) Right, there is no --hires alternative for the LAI streams.

 

[xgao304]

Dear @oleson,

I still have some questions about generating the relevant high-resolution (0.1x0.1) data.
For the step "creating mapping files" you listed on May 13 thread,

TO create mapping files
-----------------------
setenv RES 0.1x0.1
setenv GRIDFILE /glade/p/cesmdata/cseg/inputdata/lnd/clm2/mappingdata/grids/SCRIPgrid_0.1x0.1_nomask_c110712.nc
qsub ./regridbatch.sh

OUTPUT IS:

map_0.25x0.25_MODIS_to_0.1x0.1_nomask_aave_da_c230512.nc
map_0.5x0.5_AVHRR_to_0.1x0.1_nomask_aave_da_c230512.nc
map_0.5x0.5_MODIS_to_0.1x0.1_nomask_aave_da_c230512.nc
map_0.9x1.25_GRDC_to_0.1x0.1_nomask_aave_da_c230512.nc
map_10x10min_IGBPmergeICESatGIS_to_0.1x0.1_nomask_aave_da_c230512.nc
map_10x10min_nomask_to_0.1x0.1_nomask_aave_da_c230512.nc
map_1km-merge-10min_HYDRO1K-merge-nomask_to_0.1x0.1_nomask_aave_da_c230512.nc
map_360x720cru_cruncep_to_0.1x0.1_nomask_aave_da_c230512.nc
map_3x3min_GLOBE-Gardner-mergeGIS_to_0.1x0.1_nomask_aave_da_c230512.nc
map_3x3min_GLOBE-Gardner_to_0.1x0.1_nomask_aave_da_c230512.nc
map_3x3min_LandScan2004_to_0.1x0.1_nomask_aave_da_c230512.nc
map_3x3min_MODISv2_to_0.1x0.1_nomask_aave_da_c230512.nc
map_3x3min_MODIS-wCsp_to_0.1x0.1_nomask_aave_da_c230512.nc
map_3x3min_USGS_to_0.1x0.1_nomask_aave_da_c230512.nc
map_5x5min_IGBP-GSDP_to_0.1x0.1_nomask_aave_da_c230512.nc
map_5x5min_ISRIC-WISE_to_0.1x0.1_nomask_aave_da_c230512.nc
map_5x5min_nomask_to_0.1x0.1_nomask_aave_da_c230512.nc
map_5x5min_ORNL-Soil_to_0.1x0.1_nomask_aave_da_c230512.nc

I am able to configure the command in our system. However, running the command only gives me the first 3 output files of your above list:

map_0.25x0.25_MODIS_to_0.1x0.1_nomask_aave_da_c230719.nc
map_0.5x0.5_AVHRR_to_0.1x0.1_nomask_aave_da_c230719.nc
map_0.5x0.5_MODIS_to_0.1x0.1_nomask_aave_da_c230719.nc

I have not tried creating the surface data (the last step). I would like to check with you first if the command "regridbatch.sh" works correctly on my system.

Thanks,

Xiang

 

[oleson]

If those are the only three files you are getting, then I guess it's not working. What error messages are you getting?

 

[xgao304]

My bad. The command is working now. The error is because we have not downloaded all the input data.

Thanks,

Xiang