Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Creating mapfiles by using the ./mkmapdata.sh

qian yang

qian yang
Member
Dear all,I met some problems when creating mapfiles by using the ./mamapdata.sh.
see the relevant information
./mkmapdata.sh: line 104: 37967 Killed ${cmd}
Error status returned from mkmapdata script
I can not know more about it .could anyone help me? Thanks a lot.
some variable are setted
#mkmapdata
export ESMFBIN_PATH=/home/aircas/app/esmf-7-1-0/bin/binO/Linux.gfortran.64.mpiuni.default
export CSMDATA=/home/aircas/cesm/inputdata


#mkmapgrids
export PTNAME=0.1x0.1_YQ
export S_LAT=47.00
export N_LAT=53.40
export E_LON=126.50
export W_LON=115.50
export NX=111
export NY=65
export GRIDFILE=/home/aircas/cesm/inputdata/SCRIPgrid_hulunbr_$PTNAME.nc
 

Attachments

  • filename.log.txt
    6.2 KB · Views: 8

sacks

Bill Sacks
CSEG and Liaisons
Staff member
This tool requires a large amount of memory (much more than is available on most clusters), so we unfortunately are not able to support its use on machines other than NCAR's cheyenne supercomputer right now. We are working on an alternative that will be runnable on smaller systems.

That said, I am transferring this post to the land forums in case anyone there wants to add anything.
 

qian yang

qian yang
Member
I am so grateful for your help. @sacks. I want to know if I want to run a case with other resolution , is there other solution for it ? if I can just interpolation over existing data?
 

sacks

Bill Sacks
CSEG and Liaisons
Staff member
Can you please describe a little more what exactly you want to do?
 

qian yang

qian yang
Member
I want to run a case on with 0.1x0.1 resolution. But I can not use ./mamapdata.sh to create mapfiles , if I can just interpolation over existing data to realize my goal?
 

qian yang

qian yang
Member
I want to run a case on with 0.1x0.1 resolution. But I can not use ./mamapdata.sh to create mapfiles , because I do not have supercomputer.
if I can interpolate existing data to realize my goal? @sacks
 

sacks

Bill Sacks
CSEG and Liaisons
Staff member
Sorry, I don't understand what you are thinking of doing with interpolating existing data.
 

qian yang

qian yang
Member
Dear sacks
Thank for your reply. I want to know if I can interpolate surfdata by using ncl to replace original surfdata.
 

sacks

Bill Sacks
CSEG and Liaisons
Staff member
You could try that, but you might run into trouble, especially with the percent landcover fields that need to add up to exactly 100%. Another variable that would be problematic to interpolate is STD_ELEV, the standard deviation of topography within a grid cell, which is scale dependent; this affects the snow cover fraction parameterization.
 

oleson

Keith Oleson
CSEG and Liaisons
Staff member
I could try creating a surface dataset for you here. Can you provide or make available your GRIDFILE (/home/aircas/cesm/inputdata/SCRIPgrid_hulunbr_$PTNAME.nc), and the tag/version of CESM/CLM you are using?
 

qian yang

qian yang
Member
You could try that, but you might run into trouble, especially with the percent landcover fields that need to add up to exactly 100%. Another variable that would be problematic to interpolate is STD_ELEV, the standard deviation of topography within a grid cell, which is scale dependent; this affects the snow cover fraction parameterization.
thank you very much for your answer.
 

qian yang

qian yang
Member
I could try creating a surface dataset for you here. Can you provide or make available your GRIDFILE (/home/aircas/cesm/inputdata/SCRIPgrid_hulunbr_$PTNAME.nc), and the tag/version of CESM/CLM you are using?
thank your very much for your help. I now run clm5.0 with cesm2.0. here is my SCRIP file.
Can you get it ?
 

qian yang

qian yang
Member
dear oleson
I want to attach the file. But it failed. How could I give you my SCRIP file by other ways? and How could I get the surface dataset you created for me ?
Here are the settings for my SCRIP file
export PTNAME=0.1x0.1_YQ
export S_LAT=46.50
export N_LAT=54.00
export E_LON=126.1
export W_LON=115.0
export NX=111
export NY=75
export GRIDFILE=/home/aircas/cesm/inputdata/SCRIPgrid_hulunbr_$PTNAME.nc
 

oleson

Keith Oleson
CSEG and Liaisons
Staff member
Send me your SCRIP file by email (oleson at ucar dot edu).
If I'm successful in producing a surface dataset, I'll provide it via our ftp site.
 

LiwenWu

LiwenWu
New Member
Send me your SCRIP file by email (oleson at ucar dot edu).
If I'm successful in producing a surface dataset, I'll provide it via our ftp site.
Dear Oleson,
I met the same error (./mkmapdata.sh: line 106: 26775 Killed ${cmd}) when I run the ./mkmapdata.sh. Now I know it's due to insufficient memory. But my supercomputer has 512GB of RAM, and I don't have any better resources. Could you please help me to create a surface dataset? I want to generate a surface data for a 0.01deg SP regional case. My GRIDEFILE is SCRIPgrid_0.01x0.01_shanxi_nomask_c240315.nc and I have already sent you the SCRIP file (oleson@ucar.edu). I'm using CTSM5.1. As it's a SP case, the pft should be set to 16, and please specify "-no-crop" as part of the mksurfdata.pl command.
I also wonder if I want to spin up a case using atm forcing data I prepared from 1979 to 2000, which year's surface data should I use? 1979 or 2000?
I would really appreciate it if you can do me a favor. Thanks for your patience and help!
 

oleson

Keith Oleson
CSEG and Liaisons
Staff member
Unfortunately, the mksurfdata_map tools haven't and won't be ported to our new machine Derecho. So we don't have a way to generate mapping and surface dataset files using that tool.
However, we have a new tool which will be released soon, mksurfdata_esmf. There are instructions on how to create and use datasets using this tool here:


Basically, you would start by checking out the latest version of ctsm5.2.mksurdata.
Then, using your SCRIP file, create lnd_mesh.nc and mask_mesh.nc files.
Then create your surface dataset.
I'm not going to get into the habit of creating datasets for other people with this tool. However, I'm still learning about this tool so I went through the steps myself to create a 0.01x0.01 regional dataset:

git clone GitHub - ESCOMP/CTSM: Community Terrestrial Systems Model (includes the Community Land Model of CESM)
cd CTSM
git checkout ctsm5.2.mksurfdata
./manage_externals/checkout_externals
module unload python
module load conda
./py_env_create
conda activate ctsm_py
cd tools/mksurfdata_esmf
/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/oneapi/2023.2.1/7haa/bin/ESMF_Scrip2Unstruct SCRIPgrid_0.01x0.01_shanxi_nomask_c240315.nc lnd_mesh.nc 0

ncap2 -s 'elementMask(:)=0' lnd_mesh.nc mask_mesh.nc

ncdump -v grid_dims SCRIPgrid_0.01x0.01_shanxi_nomask_c240315.nc (to get nx and ny)

./gen_mksurfdata_namelist --start-year 2005 --end-year 2005 --model-mesh-nx 606 --model-mesh-ny 808 --nocrop --model-mesh lnd_mesh.nc --res 0.01x0.01_shanxi --hires_pft --hires_soitex

./gen_mksurfdata_jobscript_single --number-of-nodes 24 --tasks-per-node 12 --namelist-file surfdata_0.01x0.01_shanxi_hist_2005_16pfts_c240316.namelist

qsub ./mksurfdata_jobscript_single

I've put the resulting surface dataset and log file on our ftp site:

ftp://ftp.cgd.ucar.edu/pub/oleson/0.01x0.01

But you should try this yourself and see if you can get it to work on your machine.
Note that I selected --hires_pft --hires_soitex, which is the high resolution pft and soils data. These are only available for year 2005 which is why I set --start-year 2005 --end-year 2005. Regardless, your grid is extremely high resolution so a lot of the surface dataset fields are going to look blocky.
Note also that there are new fields on the dataset such that it will only be compatible with newer code. For example, you could create/run your cases using the ctsm5.2.mksurdata checkout itself.
 

LiwenWu

LiwenWu
New Member
Unfortunately, the mksurfdata_map tools haven't and won't be ported to our new machine Derecho. So we don't have a way to generate mapping and surface dataset files using that tool.
However, we have a new tool which will be released soon, mksurfdata_esmf. There are instructions on how to create and use datasets using this tool here:


Basically, you would start by checking out the latest version of ctsm5.2.mksurdata.
Then, using your SCRIP file, create lnd_mesh.nc and mask_mesh.nc files.
Then create your surface dataset.
I'm not going to get into the habit of creating datasets for other people with this tool. However, I'm still learning about this tool so I went through the steps myself to create a 0.01x0.01 regional dataset:

git clone GitHub - ESCOMP/CTSM: Community Terrestrial Systems Model (includes the Community Land Model of CESM)
cd CTSM
git checkout ctsm5.2.mksurfdata
./manage_externals/checkout_externals
module unload python
module load conda
./py_env_create
conda activate ctsm_py
cd tools/mksurfdata_esmf
/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/oneapi/2023.2.1/7haa/bin/ESMF_Scrip2Unstruct SCRIPgrid_0.01x0.01_shanxi_nomask_c240315.nc lnd_mesh.nc 0

ncap2 -s 'elementMask(:)=0' lnd_mesh.nc mask_mesh.nc

ncdump -v grid_dims SCRIPgrid_0.01x0.01_shanxi_nomask_c240315.nc (to get nx and ny)

./gen_mksurfdata_namelist --start-year 2005 --end-year 2005 --model-mesh-nx 606 --model-mesh-ny 808 --nocrop --model-mesh lnd_mesh.nc --res 0.01x0.01_shanxi --hires_pft --hires_soitex

./gen_mksurfdata_jobscript_single --number-of-nodes 24 --tasks-per-node 12 --namelist-file surfdata_0.01x0.01_shanxi_hist_2005_16pfts_c240316.namelist

qsub ./mksurfdata_jobscript_single

I've put the resulting surface dataset and log file on our ftp site:

ftp://ftp.cgd.ucar.edu/pub/oleson/0.01x0.01

But you should try this yourself and see if you can get it to work on your machine.
Note that I selected --hires_pft --hires_soitex, which is the high resolution pft and soils data. These are only available for year 2005 which is why I set --start-year 2005 --end-year 2005. Regardless, your grid is extremely high resolution so a lot of the surface dataset fields are going to look blocky.
Note also that there are new fields on the dataset such that it will only be compatible with newer code. For example, you could create/run your cases using the ctsm5.2.mksurdata checkout itself.
Dear Oleson,
Thanks for your kindly help, I downloaded the surface dataset you created and found that there are something wrong with the soil variables. As is shown in the picture below (ORGANIC and PCT_CLAY data for example), there are many horizontal pinstripes in some latitudes and this is abnormal. I think the raw data is definitely not like this, perhaps there was a problem reading the raw data and generating surface data? Do you have any suggestions regarding this?
I would like to download the raw data to verify if my hypothesis is correct but I just can't find the raw data in Index of /cesm/inputdata/lnd/clm2/rawdata. May I ask where I can download this data? Or maybe this data has not been publicly downloaded yet?
Looking forward to your timely reply. Thanks for your patience and help again!
1712632092786.png1712632142615.png
 

oleson

Keith Oleson
CSEG and Liaisons
Staff member
We've started seeing this problem when creating regional datasets at high resolution. An issue has been filed on this:


but no solution yet. The workaround in that case was to create a global dataset at that resolution and then subset it. But that dataset was at 0.05x0.05 resolution. I'm not sure it is feasible to create a global 0.01x0.01 dataset with this tool. You could try it on your machine.
 

erik

Erik Kluzek
CSEG and Liaisons
Staff member
@olyson the new mksurfdata_esmf tool allows you to bring in more processors and hence increase both memory and processing. We have gone down to 3.75km resolution which is something on the order of 0.03x0.03 (so 3X). We used 50 Derecho nodes for the 3.75km grid (6400 processors). So you could try that or increase it by 3X to 150 nodes if you have trouble.

We haven't got mksurfdata_esmf working on another machine besides Derecho, but it would be possible to do so with some work. We do plan on doing that and making it easier to port to different machines. But, I'm pretty confident that your 0.01x0.01 grid should work on Derecho. So doing that and then subsetting should be doable on Derecho.

The one thing, I can think of, that you might run into is limits on the NetCDF file formats. So it's possible you'll need to change the output file format if so...
 
Top