How select a special region to run the CLM from the global model in CLM?

[jteymoori]

Adding surface datasets to the namelist defaults isn't a required step. If you have created a surface dataset, e.g., surfdata_0.05x0.05_test_hist_78pfts_CMIP6_simyr2000_c200426.nc, then you just need to point to it in your user_nl_clm.

Dear @oleson ,
Thank you for your response
I have added a new resolution Name in the namelist_definition_ctsm.xml, so the 0.05x0.05_global was added to the end of the valid_values for the id="res" entry.
Now I want to create surface data for this new resolution and for that:

Firstly, for Create SCRIP Grid Files in the $CTSMROOT/tools/mkmapgrids/mkscripgrid.ncl I changed the below scripts:

name = getenv("0.05x0.05_global"); ; Get name of this point

latS = stringtodouble( getenv("-90") ); ; Get south latitude from env variable
latN = stringtodouble( getenv("90") ); ; Get north latitude from env variable
lonE = stringtodouble( getenv("180") ); ; Get east longitude from env variable
lonW = stringtodouble( getenv("-180") ); ; Get west longitude from env variable

nx = stringtointeger( getenv("7200" ) ); ; Get number of grids along longitude lines
ny = stringtointeger( getenv("3600" ) ); ; Get number of grids along latitude lines

Secondly for Generate Mapping Files in the $CTSMROOT/tools/mkmapdata/mkmapdata.s, Given that my new PTNAME is "0.05x0.05_global" and the script mkscripgrid.ncl is located in tools/mkmapgrids, here is how my would run the mkmapdata.sh script from the directory where it is located, I entered the below scripts:
./mkmapdata.sh -r 0.05x0.05 -f /glade/work/jteymoori/CTSM/tools/mkmapgrids/SCRIPgrid_0.05x0.05_global.nc -t global

But I faced this error:
CSMDATA path not known for host derecho7. Set manually before calling mkmapdata.sh. E.g., bash: export CSMDATA=/path/to/csmdata

How can I solve it?

 

[slevis]

But I faced this error:
CSMDATA path not known for host derecho7. Set manually before calling mkmapdata.sh. E.g., bash: export CSMDATA=/path/to/csmdata

How can I solve it?

Please read the error message, because it offers an example of how to solve it.

 

[jteymoori]

Please read the error message, because it offers an example of how to solve it.

Dear all
After run the below scripts for creating surface data:
./mkmapdata.sh -r 0.05x0.05 -f /glade/work/jteymoori/CTSM/tools/mkmapgrids/SCRIPgrid_0.05x0.05_global.nc -t global
I faced the below error:
CSMDATA path not known for host derecho7. Set manually before calling mkmapdata.sh. E.g., bash: export CSMDATA=/path/to/csmdata
I have tried very hard to find the answer by myself and refused to ask you to help me but unfortunately I couldn't find the good answer. I reviewed some of other persons question in the Discuss CESM frume and based on their answers I tested the below directories for define the CSMDATA directory . but I faced error in all of them.
export CSMDATA=/glade/p/cesmdata/cseg/inputdata/lnd/clm2/mappingdata/grids
export CSMDATA=/glade/p/cesmdata/cseg/inputdata/lnd/clm2/mappingdata/grids
export CSMDATA=/glade/p/cesmdata/inputdata
export CSMDATA=/glade/p/cesmdata/inputdata/lnd
export CSMDATA=/glade/p/cesmdata/inputdata/lnd/clm2
export CSMDATA=/glade/p/cesmdata/inputdata/lnd/clm2/srfdata
export CSMDATA=/glade/p/cesmdata/inputdata/lnd/clm2/surfdata
export CSMDATA=/glade/p/cesmdata/inputdata/lnd/clm2/surfdata_esmf
export CSMDATA=/glade/p/cesmdata/inputdata/lnd/clm2/surfdata_map
export CSMDATA=/glade/p/cesmdata/cseg
export CSMDATA=/glade/p/cesmdata/cseg/inputdata
export CSMDATA=/glade/p/cesmdata/cseg/inputdata/lnd/clm2
export CSMDATA=/glade/p/cesmdata/cseg/inputdata/lnd/clm2/surfdata
export CSMDATA=/glade/p/cesmdata/cseg/inputdata/lnd/clm2/surfdata_map
export CSMDATA=/glade/p/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf


Can you tell me the true direction for CSMDATA?

 

[oleson]

The new path should be /glade/campaign/cesm/cesmdata. /glade/p was the old file system.

 

[jteymoori]

The new path should be /glade/campaign/cesm/cesmdata. /glade/p was the old file system.

Thank you so much
But I faced the same error:

./mkmapdata.sh
Script to create mapping files required by mksurfdata_map
query command is ./../../bld/queryDefaultNamelist.pl -silent -namelist clmexp -justvalue -options sim_year=2000 -csmdata /glade/campaign/cesm/cesmdata

Using user specified scrip grid file: /glade/work/jteymoori/CTSM/tools/mkmapgrids/SCRIPgrid_0.05x0.05_global.nc
Output grid resolution is 0.05x0.05
Hostname = derecho7
Machine derecho7 NOT recognized
Path to ESMF binary directory does NOT exist:
Set the environment variable: ESMFBIN_PATH

 

[oleson]

The error:

Machine derecho7 NOT recognized

is related to the fact that that the surface dataset generation tools have not been officially ported to Derecho.

There may be a way to make the older tools work on Derecho, but that might end up being too complicated. It looks like maybe you gave up at some point on creating a 0.05x0.05 dataset based on the instructions below, that were also detailed in previous posts, that use the new mksurfdata_esmf tool that works on Derecho:

Setting up (high-res sparse) regional-grid CTSM simulations · ESCOMP CTSM · Discussion #1919

Context This started as issue #1773 from which I linked a google doc (now out of date and obsolete) with step-by-step instructions intended initially for a case that @XiulinGao needed to run. To th...

github.com

That tool has now been released in ctsm5.2.0. But to get you going with a surface dataset and mesh file, let's just skip walking you through that whole process again for now. I have a global 0.05x0.05 surface dataset that can be subsetted for your desired region. It would be compatible with the recently released tag ctsm5.2.0.

If that would work for you, we would need the south and north latitude and east and west longitude bounds of the region you want to simulate. It might take me a couple of days to generate the required files.

 

[jteymoori]

The error:

Machine derecho7 NOT recognized

is related to the fact that that the surface dataset generation tools have not been officially ported to Derecho.

There may be a way to make the older tools work on Derecho, but that might end up being too complicated. It looks like maybe you gave up at some point on creating a 0.05x0.05 dataset based on the instructions below, that were also detailed in previous posts, that use the new mksurfdata_esmf tool that works on Derecho:

Setting up (high-res sparse) regional-grid CTSM simulations · ESCOMP CTSM · Discussion #1919

Context This started as issue #1773 from which I linked a google doc (now out of date and obsolete) with step-by-step instructions intended initially for a case that @XiulinGao needed to run. To th...

github.com

That tool has now been released in ctsm5.2.0. But to get you going with a surface dataset and mesh file, let's just skip walking you through that whole process again for now. I have a global 0.05x0.05 surface dataset that can be subsetted for your desired region. It would be compatible with the recently released tag ctsm5.2.0.

If that would work for you, we would need the south and north latitude and east and west longitude bounds of the region you want to simulate. It might take me a couple of days to generate the required files.

Dear Dr. Oleson
Thank you for your response .
I wanted to do the section Fin the instruction (F) Make CTSM surface dataset (fsurdat file) using the mksurfdata_esmf tool) but this step must be run in the CTSM/tools/mksurfdata_esmf but my case root is not include the mksurfdata_esmf and it just includeing :
contrib , mkmapdata , mkmapgrids, mkprocdata_map , mksurfdata_map , modify_input_files , ncl_scripts , README , README.filecopies , README.testing , site_and_regional

I wanted to run the below scripts:
/gen_mksurfdata_build.sh
./gen_mksurfdata_namelist.py --start-year 2000 --end-year 2000 --model-mesh-nx 7200 --model-mesh-ny 3600 --nocrop --model-mesh lnd_mesh.nc --res 0.05x0.05_global
./gen_mksurfdata_jobscript_single.py --number-of-nodes 24 --tasks-per-node 12 --namelist-file SCRIPgrid_0.05x0.05_global.nc
qsub mksurfdata_jobscript_single

I will appreciate it if you share the global 0.05x0.05 surface datase with me.
The area that I want to run my model on it is the U.S and it have the below lat and lon:
South_lat: 24.2 N
North_ lat: 55 N
West_lon: 125 W
East_lon:lon: 65 W

 

[oleson]

The mksurfdata-esmf tool is now available in the release of ctsm5.2.0. It was not available in older checkouts of main ctsm, only on a separate branch. To get it just clone the CTSM repository and run manage_externals. It will be under tools/mksurfdata_esmf.
We are still working on subsetting a global 0.05x0.05 dataset for your region.

 

[oleson]

Sorry for the delay. At this point we are unable to subset a 0.05x0.05 global dataset for your specific region. We do have a subsetted dataset for a region similar to yours but smaller (CONUS). You could try working with that for now. You'd need to check out ctsm5.2.0. Then make the following changes to your case:

1. Add the following to your user_nl_clm in your case directory: fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/surfdata_0.05-conus-nomask_hist_2000_78pfts_c240323.nc'

2. Make the following xml changes in your case directory:

./xmlchange ATM_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange LND_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange MASK_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/mask_mesh_cdf5.nc

I ran a 1-month simulation with this, although it was very slow, you will likely have to throw more processors at it. Most of the time was spent in initialization and writing history and restart files. The create_newcase I used was:

./create_newcase --case clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000 --res CLM_USRDAT --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_SGLC_SWAV --run-unsupported

 

[jteymoori]

Sorry for the delay. At this point we are unable to subset a 0.05x0.05 global dataset for your specific region. We do have a subsetted dataset for a region similar to yours but smaller (CONUS). You could try working with that for now. You'd need to check out ctsm5.2.0. Then make the following changes to your case:

1. Add the following to your user_nl_clm in your case directory: fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/surfdata_0.05-conus-nomask_hist_2000_78pfts_c240323.nc'

2. Make the following xml changes in your case directory:

./xmlchange ATM_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange LND_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange MASK_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/mask_mesh_cdf5.nc

I ran a 1-month simulation with this, although it was very slow, you will likely have to throw more processors at it. Most of the time was spent in initialization and writing history and restart files. The create_newcase I used was:

./create_newcase --case clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000 --res CLM_USRDAT --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_SGLC_SWAV --run-unsupported

Dear Dr. Oleson
Thank you so much for sharing these scripts with me.
As I said before, I want to run the CTSM with 0.05x0.05 degree resolution. For this foal, based on your guidance in previous chats, I must firstly create a surface data set for 0.05x0.05 degree and secondly run my model with this resolution.
I ran the below scripts based on your guidance but I faced an error:
*The green lines are the scripts that you shared with me for 0.05x0.05 global surface data*
cd /glade/work/jteymoori
git clone GitHub - ESCOMP/CTSM: Community Terrestrial Systems Model (includes the Community Land Model of CESM)
cd CTSM
./manage_externals/checkout_externals
cd /CTSM/cime/scripts
module load conda
conda activate npl
./create_newcase --case clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000 --res CLM_USRDAT --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_SGLC_SWAV --run-unsupported --project UCNN0040

cd CTSM/tools/site_and_regional:
pip install numpy
pip install xarray
./subset_data region --lat1 24.2 --lat2 50.5 --lon1 235 --lon2 295 --reg united_states --create-surface
ctsm/tools/site_and_regional/subset_data_regional:
module load nco
cp /glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/surfdata_united_states_hist__2000_16pfts_c240503.nc landmask.nc
ncrename -v .PFTDATA_MASK,landmask landmask.nc
ncks -A -v .PFTDATA_MASK /glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/surfdata_united_states_hist_2000_16pfts_c240503.nc landmask.nc
ncrename -v .PFTDATA_MASK,mod_lnd_props landmask.nc
ncks --rgr infer --rgr scrip=surfdata_united_states_hist_2000_16pfts_c240503.nc landmask.nc foo.nc
/glade/u/apps/ch/opt/esmf-netcdf/8.0.0/intel/19.0.5/bin/bing/Linux.intel.64.mpiuni.default/ESMF_Scrip2Unstruct scrip.nc lnd_mesh.nc 0

Error:
-bash: /glade/u/apps/ch/opt/esmf-netcdf/8.0.0/intel/19.0.5/bin/bing/Linux.intel.64.mpiuni.default/ESMF_Scrip2Unstruct: Permission denied

cd case directory:
module load python
./xmlchange NTASKS=-4
./xmlchange NTASKS_PER_INST=144
./xmlchange JOB_WALLCLOCK_TIME=12:00:00
./xmlchange JOB_QUEUE=regular
./xmlchange PIO_TYPENAME=pnetcdf
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=1
./xmlchange LND_NX=7200
./xmlchange LND_NY=3600
./xmlchange ATM_NX=7200
./xmlchange ATM_NY=3600
./case.setup

vi user_nl_clm

fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/surfdata_0.05-conus-nomask_hist_2000_78pfts_c240323.nc'
hist_mfilt=12
hist_nhtfrq=0
Esc
:wq

Update lines in env_run.xml:
/xmlchange ATM_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange LND_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange MASK_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/mask_mesh_cdf5.nc

./case.build
./case.submit

 

[oleson]

We already provided the solution to your Permission denied error. Please look at your post from March 26 and my response on March 27.