high resolution (0.1 degree) CLM5-BGC-crop regional simulation

[xgao304]

Dear Sir:

I would like to run the CLM5-BGC-crop at a high resolution (0.1x0.1) over the user-defined region (eg., USA). I am able to successfully create the relevant global surface dataset at 0.1x0.1 degree following the thread I posted before (high resolution (0.1 degree) CLM5-BGC-crop regional simulation). Then I used
"getregional_datasets.pl" to subset the data over my region of interest.

My create newcase command is as follows:

create_newcase --case BgcCrop2000_BANGhigh --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_SROF_SGLC_SWAV \
--res CLM_USRDAT --user-mods-dir $MYDATA_DIR --machine svante --compiler intel \
--run-unsupported

I am able to compile and submit the case. But I got the following running error message:

"surfdata/fatmgrid lon/lat mismatch error",

I went to the log file (see the attachment) and looked at the relevant source code related to the error message:

eal(r8) :: rmaxlon,rmaxlat ! local min/max vars

rmaxlon = 0.0_r8
rmaxlat = 0.0_r8
do n = begg,endg
if (ldomain%lonc(n)-surfdata_domain%lonc(n) > 300.) then
rmaxlon = max(rmaxlon,abs(ldomain%lonc(n)-surfdata_domain%lonc(n)-360._r8))
elseif (ldomain%lonc(n)-surfdata_domain%lonc(n) < -300.) then
rmaxlon = max(rmaxlon,abs(ldomain%lonc(n)-surfdata_domain%lonc(n)+360._r8))
else
rmaxlon = max(rmaxlon,abs(ldomain%lonc(n)-surfdata_domain%lonc(n)))
endif
rmaxlat = max(rmaxlat,abs(ldomain%latc(n)-surfdata_domain%latc(n)))
enddo
if (rmaxlon > 0.001_r8 .or. rmaxlat > 0.001_r8) then
write(iulog,*)' ERROR: surfdata/fatmgrid lon/lat mismatch error', rmaxlon,rmaxlat
call endrun(msg=errMsg(sourcefile, __LINE__))
end if

I don't think I should relax the criterion "if (rmaxlon > 0.001_r8 .or. rmaxlat > 0.001_r8)" to prevent the error. If not, how to solve the problem?

Thanks for your help.

Xiang

 

[slevis]

I agree, do not relax the if-statement.

The error suggests that your fsurdat and fatmgrid are inconsistent. My first thought is to change --res to one of the available global grids and see whether it runs. You can see a list of available grids in /cime/scripts by typing
./query_config --grids

Once you have that working, you can compare the working and failing cases to look for differences.

 

[xgao304]

I can run the same compset on 0.5 degree. My main objective is to run the same compset on a very high resolution (0.1) degree. Dr. Oleson instructed me how to create the corresponding surface data at 0.1 degree (https://bb.cgd.ucar.edu/cesm/threads/high-resolution-0-1-degree-clm5-bgc-crop-regional-simulation.8263/). The surface data is successfully created. But running ctsm gives me the error. Is there any way to fix the issue or other alternative for running ctsm on 0.1 degree?

Thanks.

Xiang

 

[slevis]

@xgao304
I find it helpful to type
diff <case1> <case2>
when I have a case that works and a case that fails.
This way I look for differences that may explain the failure.

 

[xgao304]

What file for "case1" and "case2" should be used? Our system is down now so I do not have access to the run directory. But it is good to know which file to compare once the system is up. Thanks. Xiang

 

[slevis]

- case 1 is BgcCrop2000_BANGhigh (the directory where you type ./case_setup, ./case_build, ./case_submit).
- case 2 is the one that works.

 

[xgao304]

Sorry for the confusion. I knew what "case1" and "case2" represent. I am wondering if I directly compare the entire directory or I compare each file under the directory for case 1 and case 2? Thanks, Xiang

 

[xgao304]

@slevis:

I compared the two cases and do not see any major difference. The only question I have is the specification of grid when creating the case:
"--res hcru_hcru" versus "--res CLM_USRDAT"

The 0.5 degree (works)

create_newcase --case BgcCrop2000_USA --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_SROF_SGLC_SWAV --res hcru_hcru --user-mods-dir USAdata --machine svante --compiler intel --run-unsupported

The 0.1 degree (with error):

create_newcase --case BgcCrop2000_BANGhigh --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_SROF_SGLC_SWAV \
--res CLM_USRDAT --user-mods-dir $MYDATA_DIR --machine svante --compiler intel \
--run-unsupported

This causes some differences in env_build.xml (but not sure this is the reason to cause the error):

< <entry id="GRID" value="a%360x720cru_l%360x720cru_oi%null_r%null_g%null_w%null_m%null">
---
> <entry id="GRID" value="a%CLM_USRDAT_l%CLM_USRDAT_oi%null_r%null_g%null_w%null_m%null">
14c14
< <entry id="ATM_GRID" value="360x720cru">
---
> <entry id="ATM_GRID" value="CLM_USRDAT">
18c18
< <entry id="ATM_NX" value="720">
---
> <entry id="ATM_NX" value="1">
22c22
< <entry id="ATM_NY" value="360">
---
> <entry id="ATM_NY" value="1">
26c26
< <entry id="LND_GRID" value="360x720cru">
---
> <entry id="LND_GRID" value="CLM_USRDAT">
30c30
< <entry id="LND_NX" value="720">
---
> <entry id="LND_NX" value="1">
34c34
< <entry id="LND_NY" value="360">
---
> <entry id="LND_NY" value="1">

Other than that, there are certainly differences in surface dataset and domain file. I attached the detail comparisons. Do you have any idea where goes wrong with the 0.1x0.1 case? Thanks. Xiang

 

[oleson]

The error in longitude is quite large, almost 180 degrees. I wonder if one of either the domain or surface datasets has west longitude coordinates in negative values (e.g., -180 to 180 on a global grid) and the other dataset has west coordinates in positive values (e.g., 0 to 360 on a global grid). The latitude coordinates are within e-6.

 

[xgao304]

@oleson:

Thanks for the information. I checked and you are right. The surface dataset in the 0.1 degree starts from west. I assume it has to be consistent with
domain file. I have two questions:

1. Since the domain file is much smaller, can I process the domain file to make it consistent with surface data (starting from -179.95)? Or I have to make
the surface data to start from 0? (I am wondering if CLM requires the data to start from east instead of west).

2. LONGXY in surface data does not align exactly on the center grid with some marginal error (eg, -179.95 versus -179.949996948242).
Also, the domain file grid seems to start from the edge (0) instead of the center(0.05). What is the best way to make them consistent with
each other?


for 0.5 degree case:

global domain file (domain.lnd.360x720_cruncep.100429.nc) - downloaded from inputdata

xc:0.25 ~ 359.75
yc: -89.75 ~ 89.75

surfdata: (surfdata_360x720cru_hist_78pfts_CMIP6_simyr2000_c210423) -downloaded from inputdata
longxy: 0.25 ~ 359.75
LATIXY: -89.75 ~ 89.75


For 0.1 degree case:
global domain file (domain.lnd.0.1x0.1_gx1v7.230522) - generated

xc: 0, 0.1, ...., 359.9
yc: -89.95, -89.85, ...., 89.95

surfdata: surfdata_0.1x0.1_hist_78pfts_CMIP6_simyr2005_c230725.nc - generated
LONGXY: -179.949996948242, -179.850006103516,..., 179.950012207031


Thanks,

Xiang

 

[oleson]

The surface dataset should be 0 to 360, which would then agree with the 0.1 degree domain file you are using.
I think we went through the process of creating the surface dataset in the previous post. I don't have that data anymore but I can try repeating the process to see if I get a -180 to 180 surface dataset or a 0 to 360 dataset. Maybe there is something in that process that is causing the problem. In the meantime, I wonder if it's possible to manually fix the surface dataset you have.

 

[xgao304]

@oleson:

Thanks so much for your help. For manually fixing the surface dataset, could I take conservative interpolation using domain data as the base grid to interpolate? The only concern is that surface data set at 0.1x0.1 is a large file, interpolation may take a while to complete (I assume). If you have other
alternative idea, I would be happy to know. Thanks again. Xiang

 

[oleson]

It looks like the SCRIP file that we used to create the mapping files had longitude coordinates of -180 to 180 and that ended up causing the surface dataset to be -180 to 180. I found another SCRIP file that is 0 to 360 and created a new domain file, new mapping files, and then a new surface dataset, which is 0 to 360 in longitude. I've put that SCRIP file on our ftp site:

ftp://ftp.cgd.ucar.edu/pub/oleson/SCRIPgrid_3600x1800pt_Global_nomask_c231018.nc

So try creating the domain file and the mapping files again using this SCRIP file and then create a new surface dataset.
I've also put the domain and surface datasets I created on our ftp site as well.
Hopefully, that will work.

 

[xgao304]

@oleson: Thanks so much for your help. I will recreate the surface dataset. Best regards, Xiang

 

[xgao304]

@oleson:

The ftp site does not seem accessible. I tried several times and got the connection timed out. Also, are the domain and surface datasets you created located in the same ftp directory (ftp://ftp.cgd.ucar.edu/pub/oleson/) or in the regular cesm inputdata directory?

Thanks,

Xiang

 

[oleson]

They are in the ftp directory. Although I now see the surface dataset is still uploading.
Can you get to it through a web browser from this?

Index of /pub/oleson

 

[xgao304]

I can get to the site through a web browser, but not through linux/unix system. Downloading SCRIP file through web browse is feasible, but not for surface dataset you generated (it is too big) - although I don't have to download it (I can generate it on site, but want to compare with your version). Is it possible to download this data on ftp site using svn? Thanks.

 

[xgao304]

Just tried, after I clicked the data through the web browse, I got the following:

Forbidden​

You don't have permission to access /pub/oleson/SCRIPgrid_3600x1800pt_Global_nomask_c231018.nc on this server.

 

[oleson]

I've changed the permission on that file, see if that helps.

 

[xgao304]

@oleson:

I have some new issues here. In your previous post, you mentioned that "I found another SCRIP file that is 0 to 360 and created a new domain file, new mapping files, and then a new surface dataset, which is 0 to 360 in longitude".

If I followed the steps listed in this post (high resolution (0.1 degree) CLM5-BGC-crop regional simulation), it seems that only "create mapping files" (use regridbatch.sh) and "creating surface dataset" (use mksurfdata.pl) involve the use of SCRIP file. The first
two steps (creating mapping file for gen_domain and creating domain file) do not need the SCRIP file.

Here is the issue I have now. I recreated the mapping files and surface dataset using the new SCRIP file (0 ~ 360).

Case 1: if I use this new surface dataset with the domain file generated in our system, I still got "the surfdata/fatmgrid lon/lat mismatch error".
I tried twice and it was the same (see the attached log file).

Case 2: If I use this new surface dataset with the domain file downloaded from your pub directory, the run can proceed without any problem.

Certainly I could just use your domain file, but I would like to figure out why. The only problem should lie at the domain file generated (not the surface
dataset). I checked - the resulting domain file runs from 0 ~ 360. I also compared your domain file and my domain file, the differences in numbers are really small except along some coast lines (see the attached maps). In addition, creating domain file does not need the SCRIP file at all. Did you use different input files (gx1v7_151008.nc and 0.1x0.1_090917.nc) in the following step to create the domain file in the pub directory?

qcmd -- ./create_ESMF_map.sh --maptype aave --filesrc /glade/p/cesmdata/cseg/inputdata/share/scripgrids/gx1v7_151008.nc --filedst /glade/p/cesmdata/cseg/inputdata/share/scripgrids/0.1x0.1_090917.nc --namesrc gx1v7 --namedst 0.1x0.1

OUTPUT IS map_gx1v7_TO_0.1x0.1_aave.230512.nc

TO create domain file
---------------------
./gen_domain -m /glade/work/oleson/release-cesm2.1.3/cime/tools/mapping/gen_mapping_files/gen_ESMF_mapping_file/map_gx1v7_TO_0.1x0.1_aave.230512.nc -o gx1v7 -l 0.1x0.1

OUTPUT IS domain.lnd.0.1x0.1_gx1v7.230512.nc

Any thoughts about what is going on? The domain file is too big to send. But I share it with you in the google drive if that helps.

Thanks, Xiang