memory requirement for mkmapdata

[zhenlinyoung@gmail_com]

Dear Bill Sacks,I am trying to run the CLM model for about 60k grids at 1/24th degree resolutions. I ran into trouble using mkmapdata.sh for SCRIPgrid_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc because of the limit in memories of our local cluster.could you please give me an estimation of the memory requirement for such a run? I need to ask for the order of extra memory.Do you think it help if I fist cut off my research region from the global grid file as in  SCRIPgrid_1km-merge-10min_HYDRO1K-mrge-nomask_c130402.nc????Thanks

 

[sacks]

Copying my email reply here: I'm sorry, I really can't answer how much memory you will need to do this, because I have not done much experimentation on it myself. My rough sense is that 200 GB per node is sufficient, but I'm not positive of that: 100 GB per node may be enough. Note that that is per-node, not for the whole cluster.  As for this:Do you think whether it help if I fist cut off my research region from the global grid file as in SCRIPgrid_1km-merge-10min_HYDRO1K-mrge-nomask_c130402.nc? Yes. But given the format of this file, this will have to be done with care. By the way: I'm curious how you were pointed to this particular file. I thought that the scripts pointed to UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc, which is the same information but in a different format. You may want to try the UGRID file, since it was created to alleviate many of the problems that arose from using a scrip-format file at this high resolution. But note that you'll need a recent version of the esmf tools to use that file.  Bill

 

[sacks]

Replying to Yang's latest inquiry: I only mentioned the 1km file as an example for my 1/8th degree run, actually when I tried to run the mkmapdata on 1/24 degree, the program are already stuck for other 3min resolution files. Your best bet would be to try to resolve the problems with the 3' files first; if you can't get those to work, you have little chance of getting the 1 km file to work. You likely need to use more processors, and maybe more memory. What you suggest for UGRID is worth to try, but I cannot go to https://svn-user-sacks.cgd.ucar.edu/cesm_misc/make_mapping_files/trunk/land1km_ugrid to download master_script_regional.sh and master_script_global.sh as in your attached readme file because it requires user name and password!!  You do not need the scripts there - they were simply needed to create the UGRID file initially. Moreover, the command mpirun.lsf /glade/u/home/peggyl/bin/ESMF_RegridWeightGen -s $source_gridfile -d $dest_gridfile -w $mapname -m conserve --src_type UGRID --src_meshname landmesh -i --netcdf4 as in the readme file does not look like using these two scripts. Maybe you want to clean up this readme file a bit?? When will we have the functionality in mkmapdata.sh for UGRID? As noted in the README file, that merely documented the procedure I used to create the files. The UGRID functionality is in place in mkmapdata as of the CESM1.2 release. However, as with other parts of that script, there are some assumptions hard-coded for NCAR's yellowstone machine that you will need to change - in particular the paths to the ESMF utility. Search for ugrid in that file to see what I mean. 

 

[sacks]

I just discussed this with some others at NCAR. We thought it might be valuable - to you and other users - if we create a few high resolution global datasets that you could then subset for your region of interest. That way you could avoid having to deal with mkmapdata at all. Would that help you? That would only work if you're using the out-of-the-box raw datasets to create a surface dataset - it wouldn't work if you were overriding the out-of-the-box ones with some of your own high-resolution datasets at different resolutions.If that would work for you, can you please tell me: (1) is my understanding right that you want a surface dataset at 1/8 deg resolution? (2) Is there something special about 1/8 deg, or would something similar (like 1/10 deg) work too?

 

[zhenlinyoung@gmail_com]

Dear Bills,thanks for your reply. I am working at both 1/8th and 1/24 degree resolutions. It will be great if you could create global datasets with such resolutions for thoese 3mins and 1km resolution files under mkmapdata.sh.On a seprate issue, could you please give me some suggestions on how to subset my research region on those global files bigger than 10GB?

 

[sacks]

On a seprate issue, could you please give me some suggestions on how to subset my research region on those global files bigger than 10GB?I don't understand the problem you're having. Please give more details.

 

[zhenlinyoung@gmail_com]

According to your reply, you are giving me netcdf files similar as SCRIPgrid_1km-merge-10min_HYDRO1K-mrge-nomask_c130402.nc but in a better resolution such as 1/8th or 24th degree for the whole world. However, such one single file is larger than 10 GB, how could I extract the data for my research region from this 10GB file?

 

[sacks]

Sorry for the confusion: We would not be providing those SCRIP data files (those already exist). Instead, we would provide global, high-resolution surface datasets - the output of CLM's mksurfdata_map - which could be subset for the region of interest.We would also provide a tool to do the subsetting.

 

[zhenlinyoung@gmail_com]

oh, yes. that is indeed very good solution. when do you expect these files be ready?so my question is, even we have such surface data files at 1/8th and 1/24 degree resolution, I did some calculation, the global file itself could be up to more than 10 GB. How would be extract the data of our research region within that giant global surface files? could you recommend some tools which can handle such giant surface files? Thanks for your efforts!

 

[sacks]

Based on discussion today, we will probably provide a surface dataset at 1/8 deg. Probably not 1/24 deg. (Note that that is higher resolution than most of CLM's raw datasets, which at best are at 3' resolution, so you do not gain much scientifically by running at such high resolution.) I'm not sure of the time frame at this point... probably in the next few weeks to month, but I can't say for certain. As I mentioned we will also provide a tool to subset this for a region of interest.

 

[yangk2010@lzu_edu_cn]

BillI just tried script mkmapdata to create mapping files, but when it turned to " map_1km-merge-10min_HYDRO1K-merge-nomask_to_1x1_maqu_nomask_aave_da_c141125.nc", its input grid is "UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc",but , error appeared : Creating mapping file: map_1km-merge-10min_HYDRO1K-merge-nomask_to_1x1_maqu_nomask_aave_da_c141125.nc
From input grid: /mnt/Disk16T/yangkai_file/data/inputdata/lnd/clm2/mappingdata/grids/UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc
For output grid: /home/yangkai/Single-point/cesm1_2_0/models/lnd/clm/tools/shared/mkmapgrids/SCRIPgrid_1x1_maqu_nomask_c141125.ncNo support for --netcdf4 or UGRID on machines other than yellowstone/caldera/geyser
/home/yangkai/Single-point/cesm1_2_0/scripts/ccsm_utils/Machines/config_compilers.xml intel linuxintelHow can I solve this problem on my cluster ?

 

[sacks]

I'm sorry: We are not able to support the creation of these mapping files on machines other than NCAR's own machines at this time. We still plan to create a 1/8 deg surface dataset that can be subset for regional runs, which will hopefully address people's needs and bypass the need to create these mapping files at all. This has taken much longer than expected, but we expect to have it ready soon. We can announce its availability via this forum once it is ready.

 

[yangk2010@lzu_edu_cn]

Bill, Does it mean that I can't  run single-point datasets with my own atmosphere forcing by CLM4.5 ?Is there the other way to  create of these mapping files or run CLM4.5 for single-point ?From the previous comments, there is the other mapping file "SCRIPgrid_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc ",which is the grid data in NETCDF form,but the script mkmapdata.sh points to the file "UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc".Can I use the first mentioned of two ?Thanks 

 

[sacks]

I am going to forward this to Erik Kluzek, since he has more expertise with different methods for running single-point. He is gone this week, but should be able to respond next week.But I think that using the SCRIPgrid file would make the problem worse, not better. (They are both in netcdf form, but the ugrid format is a more compact representation.)

 

[yangk2010@lzu_edu_cn]

 Bill, Hello ! I'm still troubled with the perplexing problem.And, I have tried many other ways, but it still did not work.Last time, you told me that  Erik Kluzek has more expertise with different methods for running single-point and should be able to respond this week.So, if  Erik Kluzek has come back, could you let him tell me hwo to run a single-point case with CLM4.5.Thank you very much !I'm looking forward to your reply.

 

[sacks]

I'm sorry, it turns out Erik is out all this week as well, and will be back next week.

 

[erik]

We have some new surface datasets that can be used with the latest development model version (clm4_5_1_r104) for subsetting high resolution areas. This dataset will NOT work with any of the public release versions of the model. Both datasets are available from the svn inputdata server.https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/surfdata_map/ The version for use with the prognostic crop model is...[*]surfdata_0.125x0.125_mp24_simyr2000_c150114.nc[/list]While the non crop version is...[*]surfdata_0.125x0.125_simyr2000_c150114.nc[/list]The script getregional_datasets.pl (part of the clm4_5_1_r104 tag) can be used to subset the above dataset for high resolution simulations over a particular region of interest.

 

[Xax]

Dear Erik,
thank you for the update. Will there also be tools provided to create the required domain files? Also if i may ask for some advice.
I tried running mkmapdata on a node with 64GB and still no luck. Will the memory requirements be split to several nodes if I run it in parallel mode?

I can not find a straightforward way to make a subset of my region from a global grid.