error in the creation of surface datasets for 3x3min

[dquaker]

Dear @oleson,
gitdescribe = 'ctsm5.1.dev105

In the past, I have been successful at creating the surface datasets and landuse time series for 0.23x0.23 at rcp,ssp or historical scenarios. So I have used similar approach for 3x3min but with errors.
I have been unable to create surface datasets at 3x3min resolution (global or regional domain) as it creates different forms of error that centers on mksurfdata_map: 34304.

These are the steps I followed using the global datasets.

1. ./mkmapdata.sh --res 3x3min ( All the maping files were created i.e
map_0.125x0.125_nomask_to_3x3min_nomask_aave_da_c221007.nc
map_0.25x0.25_nomask_to_3x3min_nomask_aave_da_c221007.nc
map_0.5x0.5_nomask_to_3x3min_nomask_aave_da_c221007.nc
map_0.9x1.25_nomask_to_3x3min_nomask_aave_da_c221007.nc
map_10x10min_nomask_to_3x3min_nomask_aave_da_c221007.nc
map_1km-merge-10min_HYDRO1K-merge-nomask_to_3x3min_nomask_aave_da_c221007.nc
map_3x3min_nomask_to_3x3min_nomask_aave_da_c221007.nc
map_5x5min_nomask_to_3x3min_nomask_aave_da_c221007.nc

2. ./mksurfdata.pl -res 3x3min
ERROR: could NOT find a mapping file for this resolution: 3x3min and type: lak at 3x3min and nomask

OR

3. . ./mksurfdata.pl -res usrspec -usr_gname 3x3min -usr_gdate 221007 -y 2000

fsurdat is 2d lat/lon grid
nlon= 7200 nlat= 3600
Output domain must be on a 0 to 360 longitude grid rather than a -180 to 180 g
rid as it is required for CESM
abort:
ERROR in mksurfdata_map: 34304

See attached error file for the complete information

I would really appreciate your helpful information on this error.

Is it possible to obtain via ftp the surface and land use time series at this resolution over my domain of interest ....f this is not too much to ask? I understand the global surface data at this resolution could be very large especially since I did for 0.23x0.23 and it was quite enormous.

 

[oleson]

@erik do you have any ideas here?
I tried this myself. The regridbatch.sh process completed successfully with no errors, but with a smaller number of mapping files created than I would normally expect:

map_0.125x0.125_nomask_to_3x3min_nomask_aave_da_c221009.nc
map_0.25x0.25_nomask_to_3x3min_nomask_aave_da_c221009.nc
map_0.5x0.5_nomask_to_3x3min_nomask_aave_da_c221009.nc
map_0.9x1.25_nomask_to_3x3min_nomask_aave_da_c221009.nc
map_10x10min_nomask_to_3x3min_nomask_aave_da_c221009.nc
map_1km-merge-10min_HYDRO1K-merge-nomask_to_3x3min_nomask_aave_da_c221009.nc
map_3x3min_nomask_to_3x3min_nomask_aave_da_c221009.nc
map_5x5min_nomask_to_3x3min_nomask_aave_da_c221009.nc

which I think might be why the user is getting the error in 2. when running mksurfdata.pl.

I'm not quite sure how the list of required mapping files gets generated.

 

[erik]

The problem seems to be that the 3x3min grid we use is on a -180 to 180 grid rather than a 0 to 360 grid. We know there are issues when you try to run the model on a -180 to 180 grid, so we disallow it in mksurfdata_map. Technically a -180 to 180 grid should work, but we know there are problems with it.

So what you would need to do is to create a 3x3min SCRIP grid file that's on a 0 to 360 grid and run mkmapdata.sh and then mksurfdata.pl using it.

 

[dquaker]

Thank you @oleson and @erik. I will attempt to create the SCRIP grid file based on your recommendation. I am not very familiar with the approach of creating the SCRIP grid file but will see if I can get some online resources to help with this task. In the meantime, I will really appreciate if you could point me to a script (if available to use). However, my first guess would be to use ncl or esmf but I am not so sure at the moment....

 

[oleson]

Thanks @erik
To generate a new SCRIP grid file I think this would work:

setenv PTNAME 3x3min
setenv S_LAT -90
setenv N_LAT 90
setenv W_LON 0
setenv E_LON 360
setenv NX 7200
setenv NY 3600

ncl mkscripgrid.ncl

The NCL script can be found in tools/mkmapgrids

 

[dquaker]

Dear @oleson and @erik

I followed the steps as specified to generate a new 3x3min SCRIP grid file and it was successful. Then I proceeded to run the mkmapdata.sh which generated the mapping files as well. The next step using mksurfdata.pl generated the errors below:

domain_checksame ERROR: input domain lon and gridmap lon not the same at ni =
1
domain lon= -179.975006103516
gridmap lon= 2.500000000000000E-002
abort:
ERROR in mksurfdata_map: 34304

Please find the attached log files.

Could you kindly assist me further in fixing this error?

 

[oleson]

It seems like your mapping files are still -180 to 180. Did you use the new SCRIP file you created to generate the mapping files?

I think the steps should be something like:

setenv RES 3x3min

setenv GRIDFILE $YOUR_NEW_SCRIP_FILE

./regridbatch.sh

OR

./mkmapdata.sh --gridfile $YOUR_NEW_SCRIP_FILE --res 3x3min

We would use regridbatch.sh on our machine here to submit the job to the compute nodes since this will require quite a bit of memory.

 

[oleson]

I think this mapping file:

/gpfs/scratch/pn25ki/di75jeq/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/map_3x3min_nomask_to_3x3min_nomask_aave_da_c221012.nc

should have:

xc_a = -179.975, -179.925, -179.875, -179.825, -179.775, -179.725, -179.675, etc.......

and

xc_b = 0.025, 0.075, 0.125, 0.175, 0.225, 0.275, 0.325, 0.375, 0.425, 0.475, etc.....

What do you see?

 

[dquaker]

I think this mapping file:

/gpfs/scratch/pn25ki/di75jeq/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/map_3x3min_nomask_to_3x3min_nomask_aave_da_c221012.nc

should have:

xc_a = -179.975, -179.925, -179.875, -179.825, -179.775, -179.725, -179.675, etc.......

and

xc_b = 0.025, 0.075, 0.125, 0.175, 0.225, 0.275, 0.325, 0.375, 0.425, 0.475, etc.....

What do you see?

I checked the mapping file and I have
xc_b= 0.025, 0.075, 0.125, 0.175, 0.225, 0.275, 0.325, 0.375, 0.425, 0.475, etc

xc_a= 0.025, 0.075, 0.125, 0.175, 0.225, 0.275, 0.325, 0.375, 0.425, 0.475, etc

I am not sure why they have the same values as indicated above because using the old SCRIP file gave the expected values for both xc_a and xc_b.

Below is the exact way tthe new SCRIP grid was generated which was used to generate the mapping files

export PTNAME=3x3min
export S_LAT=-90
export N_LAT=90
export E_LON=360
export W_LON=0
export NX=7200
export NY=3600
export PRINT=TRUE
export GRIDFILE=SCRIPgrid_3x3min_nomask_c200309.nc
ncl mkscripgrid.ncl

 

[oleson]

So, the new SCRIP file has a date of c200309 (same as the one in the repository)?

Can you check to see if the new SCRIP file you are using has this:

grid_center_lon = 0.025, 0.075, 0.125, 0.175, 0.225, 0.275, 0.325, 0.375,

 

[dquaker]

So, the new SCRIP file has a date of c200309 (same as the one in the repository)?

Can you check to see if the new SCRIP file you are using has this:

grid_center_lon = 0.025, 0.075, 0.125, 0.175, 0.225, 0.275, 0.325, 0.375,

Yes, I gave it the same name and placed in the same directory.Although I still have the old one in place with a new name( SCRIPgrid_3x3min_nomask_c200309.nc.orig)

The grid_center_lon has these values
grid_center_lon = 0.025, 0.075, 0.125, 0.175, 0.225, 0.275, 0.325, 0.375,
0.425, 0.475, 0.525, 0.575, 0.625, 0.675, 0.725, 0.775, 0.825, 0.87

 

[oleson]

That seems correct. I'm not sure why your mapping file doesn't end up with the right coordinates. I've put my SCRIP file and my version of that map file on our ftp site, so you can compare and perhaps try using them:

ftp://ftp.cgd.ucar.edu/pub/oleson/mapping

 

[dquaker]

That seems correct. I'm not sure why your mapping file doesn't end up with the right coordinates. I've put my SCRIP file and my version of that map file on our ftp site, so you can compare and perhaps try using them:

ftp://ftp.cgd.ucar.edu/pub/oleson/mapping

I was able to download and use the SCRIP file provided via the ftp. The SCRIP file generated the appropriate and expected values for the newly created mapping files as earlier explained. Thank you for the support provided.

New error came up with the mapping file for map_1km-merge-10min_HYDRO1K-merge-nomask_to_3x3min_nomask_aave_da_c221019.nc during the mksurfdata.pl process.

The error is

Attempting to make Topography statistics.....
domain_read_dims_1d read 1-d length from num_pixels
domain_read initialized domain
(gridmap_map_read) reading mapping matrix data...
(gridmap_map_read) * file name : /gpfs/scratch/pn25ki/di75jeq/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/forum/test/map_1km-merge-10min_HYDRO1K-merge-nomask_to_3x3min_nomask_aave_da_c221019.nc
* matrix dimensions rows x cols : 125599447 x 25920000
* number of non-zero elements: 197485665
(gridmap_map_read) ERROR: frac_dst out of bounds
max = 1.05284755675040 min = 0.000000000000000E+000
abort:
ERROR in mksurfdata_map: 34304


Kindly

 

[oleson]

I can try creating a surface dataset tomorrow if our HPC systems come back up. However, please see this post which seems to describe the same error:

An error in CLM5.0 mksurfdata process

When I run ./mksurfdata.pl -r usrspec -hirespft -usr_gname 0.05x0.05 -usr_gdate 210328 to make surface map at 0.05 degree, there exist an error below ------------------------------------- Attempting to make Topography statistics..... domain_read_dims_1d read 1-d length from num_pixels...

bb.cgd.ucar.edu

There, the suggestion was to check the SCRIP file for potential overlapping cells.

 

[dquaker]

Thank you for your feedback. I will attempt to see if I can find overlapping cells as suggested in the post.

 

[dquaker]

I can try creating a surface dataset tomorrow if our HPC systems come back up. However, please see this post which seems to describe the same error:

An error in CLM5.0 mksurfdata process

When I run ./mksurfdata.pl -r usrspec -hirespft -usr_gname 0.05x0.05 -usr_gdate 210328 to make surface map at 0.05 degree, there exist an error below ------------------------------------- Attempting to make Topography statistics..... domain_read_dims_1d read 1-d length from num_pixels...

bb.cgd.ucar.edu

There, the suggestion was to check the SCRIP file for potential overlapping cells.

Dear @oleson, I would like to follow-up to see if you are able to generate the surface datasets. All attempts to generate it on our HPC were unsucessful as I am not able to identify the overlapping gridcells. I would greatly appreciate your assistance on this issue. Thank you so much.

 

[oleson]

Yes, I did try this. Unfortunately, I got the same error when creating the surface dataset:

(gridmap_map_read) ERROR: frac_dst out of bounds
max = 1.05284755675040 min = 0.000000000000000E+000
abort:
ERROR in mksurfdata_map: 34304

I'm not sure where to go from here. Perhaps @erik has an idea?