How select a special region to run the CLM from the global model in CLM?

[jteymoori]

Adding surface datasets to the namelist defaults isn't a required step. If you have created a surface dataset, e.g., surfdata_0.05x0.05_test_hist_78pfts_CMIP6_simyr2000_c200426.nc, then you just need to point to it in your user_nl_clm.

Dear @oleson ,
Thank you for your response
I have added a new resolution Name in the namelist_definition_ctsm.xml, so the 0.05x0.05_global was added to the end of the valid_values for the id="res" entry.
Now I want to create surface data for this new resolution and for that:

Firstly, for Create SCRIP Grid Files in the $CTSMROOT/tools/mkmapgrids/mkscripgrid.ncl I changed the below scripts:

name = getenv("0.05x0.05_global"); ; Get name of this point

latS = stringtodouble( getenv("-90") ); ; Get south latitude from env variable
latN = stringtodouble( getenv("90") ); ; Get north latitude from env variable
lonE = stringtodouble( getenv("180") ); ; Get east longitude from env variable
lonW = stringtodouble( getenv("-180") ); ; Get west longitude from env variable

nx = stringtointeger( getenv("7200" ) ); ; Get number of grids along longitude lines
ny = stringtointeger( getenv("3600" ) ); ; Get number of grids along latitude lines

Secondly for Generate Mapping Files in the $CTSMROOT/tools/mkmapdata/mkmapdata.s, Given that my new PTNAME is "0.05x0.05_global" and the script mkscripgrid.ncl is located in tools/mkmapgrids, here is how my would run the mkmapdata.sh script from the directory where it is located, I entered the below scripts:
./mkmapdata.sh -r 0.05x0.05 -f /glade/work/jteymoori/CTSM/tools/mkmapgrids/SCRIPgrid_0.05x0.05_global.nc -t global

But I faced this error:
CSMDATA path not known for host derecho7. Set manually before calling mkmapdata.sh. E.g., bash: export CSMDATA=/path/to/csmdata

How can I solve it?

 

[slevis]

But I faced this error:
CSMDATA path not known for host derecho7. Set manually before calling mkmapdata.sh. E.g., bash: export CSMDATA=/path/to/csmdata

How can I solve it?

Please read the error message, because it offers an example of how to solve it.

 

[jteymoori]

Please read the error message, because it offers an example of how to solve it.

Dear all
After run the below scripts for creating surface data:
./mkmapdata.sh -r 0.05x0.05 -f /glade/work/jteymoori/CTSM/tools/mkmapgrids/SCRIPgrid_0.05x0.05_global.nc -t global
I faced the below error:
CSMDATA path not known for host derecho7. Set manually before calling mkmapdata.sh. E.g., bash: export CSMDATA=/path/to/csmdata
I have tried very hard to find the answer by myself and refused to ask you to help me but unfortunately I couldn't find the good answer. I reviewed some of other persons question in the Discuss CESM frume and based on their answers I tested the below directories for define the CSMDATA directory . but I faced error in all of them.
export CSMDATA=/glade/p/cesmdata/cseg/inputdata/lnd/clm2/mappingdata/grids
export CSMDATA=/glade/p/cesmdata/cseg/inputdata/lnd/clm2/mappingdata/grids
export CSMDATA=/glade/p/cesmdata/inputdata
export CSMDATA=/glade/p/cesmdata/inputdata/lnd
export CSMDATA=/glade/p/cesmdata/inputdata/lnd/clm2
export CSMDATA=/glade/p/cesmdata/inputdata/lnd/clm2/srfdata
export CSMDATA=/glade/p/cesmdata/inputdata/lnd/clm2/surfdata
export CSMDATA=/glade/p/cesmdata/inputdata/lnd/clm2/surfdata_esmf
export CSMDATA=/glade/p/cesmdata/inputdata/lnd/clm2/surfdata_map
export CSMDATA=/glade/p/cesmdata/cseg
export CSMDATA=/glade/p/cesmdata/cseg/inputdata
export CSMDATA=/glade/p/cesmdata/cseg/inputdata/lnd/clm2
export CSMDATA=/glade/p/cesmdata/cseg/inputdata/lnd/clm2/surfdata
export CSMDATA=/glade/p/cesmdata/cseg/inputdata/lnd/clm2/surfdata_map
export CSMDATA=/glade/p/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf


Can you tell me the true direction for CSMDATA?

 

[oleson]

The new path should be /glade/campaign/cesm/cesmdata. /glade/p was the old file system.

 

[jteymoori]

The new path should be /glade/campaign/cesm/cesmdata. /glade/p was the old file system.

Thank you so much
But I faced the same error:

./mkmapdata.sh
Script to create mapping files required by mksurfdata_map
query command is ./../../bld/queryDefaultNamelist.pl -silent -namelist clmexp -justvalue -options sim_year=2000 -csmdata /glade/campaign/cesm/cesmdata

Using user specified scrip grid file: /glade/work/jteymoori/CTSM/tools/mkmapgrids/SCRIPgrid_0.05x0.05_global.nc
Output grid resolution is 0.05x0.05
Hostname = derecho7
Machine derecho7 NOT recognized
Path to ESMF binary directory does NOT exist:
Set the environment variable: ESMFBIN_PATH

 

[oleson]

The error:

Machine derecho7 NOT recognized

is related to the fact that that the surface dataset generation tools have not been officially ported to Derecho.

There may be a way to make the older tools work on Derecho, but that might end up being too complicated. It looks like maybe you gave up at some point on creating a 0.05x0.05 dataset based on the instructions below, that were also detailed in previous posts, that use the new mksurfdata_esmf tool that works on Derecho:

Setting up (high-res sparse) regional-grid CTSM simulations · ESCOMP CTSM · Discussion #1919

Context This started as issue #1773 from which I linked a google doc (now out of date and obsolete) with step-by-step instructions intended initially for a case that @XiulinGao needed to run. To th...

github.com

That tool has now been released in ctsm5.2.0. But to get you going with a surface dataset and mesh file, let's just skip walking you through that whole process again for now. I have a global 0.05x0.05 surface dataset that can be subsetted for your desired region. It would be compatible with the recently released tag ctsm5.2.0.

If that would work for you, we would need the south and north latitude and east and west longitude bounds of the region you want to simulate. It might take me a couple of days to generate the required files.

 

[jteymoori]

The error:

Machine derecho7 NOT recognized

is related to the fact that that the surface dataset generation tools have not been officially ported to Derecho.

There may be a way to make the older tools work on Derecho, but that might end up being too complicated. It looks like maybe you gave up at some point on creating a 0.05x0.05 dataset based on the instructions below, that were also detailed in previous posts, that use the new mksurfdata_esmf tool that works on Derecho:

Setting up (high-res sparse) regional-grid CTSM simulations · ESCOMP CTSM · Discussion #1919

Context This started as issue #1773 from which I linked a google doc (now out of date and obsolete) with step-by-step instructions intended initially for a case that @XiulinGao needed to run. To th...

github.com

That tool has now been released in ctsm5.2.0. But to get you going with a surface dataset and mesh file, let's just skip walking you through that whole process again for now. I have a global 0.05x0.05 surface dataset that can be subsetted for your desired region. It would be compatible with the recently released tag ctsm5.2.0.

If that would work for you, we would need the south and north latitude and east and west longitude bounds of the region you want to simulate. It might take me a couple of days to generate the required files.

Dear Dr. Oleson
Thank you for your response .
I wanted to do the section Fin the instruction (F) Make CTSM surface dataset (fsurdat file) using the mksurfdata_esmf tool) but this step must be run in the CTSM/tools/mksurfdata_esmf but my case root is not include the mksurfdata_esmf and it just includeing :
contrib , mkmapdata , mkmapgrids, mkprocdata_map , mksurfdata_map , modify_input_files , ncl_scripts , README , README.filecopies , README.testing , site_and_regional

I wanted to run the below scripts:
/gen_mksurfdata_build.sh
./gen_mksurfdata_namelist.py --start-year 2000 --end-year 2000 --model-mesh-nx 7200 --model-mesh-ny 3600 --nocrop --model-mesh lnd_mesh.nc --res 0.05x0.05_global
./gen_mksurfdata_jobscript_single.py --number-of-nodes 24 --tasks-per-node 12 --namelist-file SCRIPgrid_0.05x0.05_global.nc
qsub mksurfdata_jobscript_single

I will appreciate it if you share the global 0.05x0.05 surface datase with me.
The area that I want to run my model on it is the U.S and it have the below lat and lon:
South_lat: 24.2 N
North_ lat: 55 N
West_lon: 125 W
East_lon:lon: 65 W

 

[oleson]

The mksurfdata-esmf tool is now available in the release of ctsm5.2.0. It was not available in older checkouts of main ctsm, only on a separate branch. To get it just clone the CTSM repository and run manage_externals. It will be under tools/mksurfdata_esmf.
We are still working on subsetting a global 0.05x0.05 dataset for your region.

 

[oleson]

Sorry for the delay. At this point we are unable to subset a 0.05x0.05 global dataset for your specific region. We do have a subsetted dataset for a region similar to yours but smaller (CONUS). You could try working with that for now. You'd need to check out ctsm5.2.0. Then make the following changes to your case:

1. Add the following to your user_nl_clm in your case directory: fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/surfdata_0.05-conus-nomask_hist_2000_78pfts_c240323.nc'

2. Make the following xml changes in your case directory:

./xmlchange ATM_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange LND_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange MASK_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/mask_mesh_cdf5.nc

I ran a 1-month simulation with this, although it was very slow, you will likely have to throw more processors at it. Most of the time was spent in initialization and writing history and restart files. The create_newcase I used was:

./create_newcase --case clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000 --res CLM_USRDAT --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_SGLC_SWAV --run-unsupported

 

[jteymoori]

Sorry for the delay. At this point we are unable to subset a 0.05x0.05 global dataset for your specific region. We do have a subsetted dataset for a region similar to yours but smaller (CONUS). You could try working with that for now. You'd need to check out ctsm5.2.0. Then make the following changes to your case:

1. Add the following to your user_nl_clm in your case directory: fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/surfdata_0.05-conus-nomask_hist_2000_78pfts_c240323.nc'

2. Make the following xml changes in your case directory:

./xmlchange ATM_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange LND_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange MASK_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/mask_mesh_cdf5.nc

I ran a 1-month simulation with this, although it was very slow, you will likely have to throw more processors at it. Most of the time was spent in initialization and writing history and restart files. The create_newcase I used was:

./create_newcase --case clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000 --res CLM_USRDAT --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_SGLC_SWAV --run-unsupported

Dear Dr. Oleson
Thank you so much for sharing these scripts with me.
As I said before, I want to run the CTSM with 0.05x0.05 degree resolution. For this foal, based on your guidance in previous chats, I must firstly create a surface data set for 0.05x0.05 degree and secondly run my model with this resolution.
I ran the below scripts based on your guidance but I faced an error:
*The green lines are the scripts that you shared with me for 0.05x0.05 global surface data*
cd /glade/work/jteymoori
git clone GitHub - ESCOMP/CTSM: Community Terrestrial Systems Model (includes the Community Land Model of CESM)
cd CTSM
./manage_externals/checkout_externals
cd /CTSM/cime/scripts
module load conda
conda activate npl
./create_newcase --case clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000 --res CLM_USRDAT --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_SGLC_SWAV --run-unsupported --project UCNN0040

cd CTSM/tools/site_and_regional:
pip install numpy
pip install xarray
./subset_data region --lat1 24.2 --lat2 50.5 --lon1 235 --lon2 295 --reg united_states --create-surface
ctsm/tools/site_and_regional/subset_data_regional:
module load nco
cp /glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/surfdata_united_states_hist__2000_16pfts_c240503.nc landmask.nc
ncrename -v .PFTDATA_MASK,landmask landmask.nc
ncks -A -v .PFTDATA_MASK /glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/surfdata_united_states_hist_2000_16pfts_c240503.nc landmask.nc
ncrename -v .PFTDATA_MASK,mod_lnd_props landmask.nc
ncks --rgr infer --rgr scrip=surfdata_united_states_hist_2000_16pfts_c240503.nc landmask.nc foo.nc
/glade/u/apps/ch/opt/esmf-netcdf/8.0.0/intel/19.0.5/bin/bing/Linux.intel.64.mpiuni.default/ESMF_Scrip2Unstruct scrip.nc lnd_mesh.nc 0

Error:
-bash: /glade/u/apps/ch/opt/esmf-netcdf/8.0.0/intel/19.0.5/bin/bing/Linux.intel.64.mpiuni.default/ESMF_Scrip2Unstruct: Permission denied

cd case directory:
module load python
./xmlchange NTASKS=-4
./xmlchange NTASKS_PER_INST=144
./xmlchange JOB_WALLCLOCK_TIME=12:00:00
./xmlchange JOB_QUEUE=regular
./xmlchange PIO_TYPENAME=pnetcdf
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=1
./xmlchange LND_NX=7200
./xmlchange LND_NY=3600
./xmlchange ATM_NX=7200
./xmlchange ATM_NY=3600
./case.setup

vi user_nl_clm

fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/surfdata_0.05-conus-nomask_hist_2000_78pfts_c240323.nc'
hist_mfilt=12
hist_nhtfrq=0
Esc
:wq

Update lines in env_run.xml:
/xmlchange ATM_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange LND_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange MASK_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/mask_mesh_cdf5.nc

./case.build
./case.submit

 

[oleson]

We already provided the solution to your Permission denied error. Please look at your post from March 26 and my response on March 27.

 

[jteymoori]

Dear all
I sincerely thank you for always answering my questions with patience and generosity.
As I said, I want to run the CTSM with 0.05x0.05 degree resolution. For this foal, based on your guidance in previous chats, I must first create a surface data set for the 0.05x0.05 degree global 0.05x0.05 dataset. I ran the below scripts for that and submitted my case, but I received the below email. Can you tell me what is wrong ?

Email:
PBS Job Id: 4400469.desched1
Job Name: st_archive.clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000
Aborted by PBS Server
Job deleted as result of dependency on job 4400468.desched

The scripts that I have ran:

cd /glade/work/jteymoori
git clone GitHub - ESCOMP/CTSM: Community Terrestrial Systems Model (includes the Community Land Model of CESM)
cd CTSM
./manage_externals/checkout_externals
cd /CTSM/cime/scripts
module load conda
conda activate npl
./create_newcase --case ~/0.05x0.05/clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000 --res CLM_USRDAT --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_SGLC_SWAV --run-unsupported --project UCNN0040

cd CTSM/tools/site_and_regional:
pip install numpy
pip install xarray
./subset_data region --lat1 24.2 --lat2 50.5 --lon1 235 --lon2 295 --reg united_states --create-surface
ctsm/tools/site_and_regional/subset_data_regional:
module load nco
cp /glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/surfdata_united_states_hist_2000_16pfts_c240508.nc landmask.nc

ncrename -v .PFTDATA_MASK,landmask landmask.nc
ncks -A -v .PFTDATA_MASK /glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/surfdata_united_states_hist_2000_16pfts_c240508.nc landmask.nc
ncrename -v .PFTDATA_MASK,mod_lnd_props landmask.nc
ncks --rgr infer --rgr scrip=scrip.nc landmask.nc foo.nc

/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/oneapi/2023.2.1/7haa/bin/ESMF_Scrip2Unstruct scrip.nc lnd_mesh.nc 0

cd case directory:
module load python
./xmlchange NTASKS=-4
./xmlchange NTASKS_PER_INST=144
./xmlchange JOB_WALLCLOCK_TIME=12:00:00
./xmlchange JOB_QUEUE=main
./xmlchange PIO_TYPENAME=pnetcdf
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=1
./xmlchange LND_NX=1200
./xmlchange LND_NY=562
./xmlchange ATM_NX=1200
./xmlchange ATM_NY=562
./case.setup

vi user_nl_clm

fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/surfdata_0.05-conus-nomask_hist_2000_78pfts_c240323.nc'
hist_mfilt=12
hist_nhtfrq=0
Esc
:wq

Update lines in env_run.xml:
./xmlchange ATM_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange LND_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc
./xmlchange MASK_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/mask_mesh_cdf5.nc

./case.build
./case.submit

 

[oleson]

The fact that you got an email saying that your short term archiver failed means that your main simulation failed to run:

PBS Job Id: 4400469.desched1
Job Name: st_archive.clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000
Aborted by PBS Server
Job deleted as result of dependency on job 4400468.desched

So we need to look at the log files that are in your run directory to see why the run failed.
So, in your run directory, which seems to be /glade/derecho/scratch/jteymoori/clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000/run, I see the following in your cesm log:

dec0077.hsn.de.hpc.ucar.edu 0: ERROR:
dec0077.hsn.de.hpc.ucar.edu 0: (dshr_mod:dshr_mesh_init) ERROR: model_meshfile /glade/campaign/cgd/tss/people/
dec0077.hsn.de.hpc.ucar.edu 0: oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc does not exist

That file does exist. But this might be a file permission problem. I've changed permissions. Can you try again please?

 

[jteymoori]

The fact that you got an email saying that your short term archiver failed means that your main simulation failed to run:

PBS Job Id: 4400469.desched1
Job Name: st_archive.clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000
Aborted by PBS Server
Job deleted as result of dependency on job 4400468.desched

So we need to look at the log files that are in your run directory to see why the run failed.
So, in your run directory, which seems to be /glade/derecho/scratch/jteymoori/clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000/run, I see the following in your cesm log:

dec0077.hsn.de.hpc.ucar.edu 0: ERROR:
dec0077.hsn.de.hpc.ucar.edu 0: (dshr_mod:dshr_mesh_init) ERROR: model_meshfile /glade/campaign/cgd/tss/people/
dec0077.hsn.de.hpc.ucar.edu 0: oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124/lnd_mesh_cdf5.nc does not exist

That file does exist. But this might be a file permission problem. I've changed permissions. Can you try again please?

Dear Dr. Oleson
Thank you for your response.
I tested it again and access to that directory and It is still not possible to access that directory, and as before, we are facing the Permission denied message.
Even I made another case, but after submitting it, I received an email similar to the previous email. In addition I tried to going to that directory (/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.mksurfdata_032124) and I faced Permission denied message again.

 

[oleson]

Sorry, the directories are write protected for people outside the group. I've fixed this hopefully, please try again.

 

[jteymoori]

Sorry, the directories are write protected for people outside the group. I've fixed this hopefully, please try again.

Dear Dr. Oleson

Thank you so much for your help. I ran the scripts and the permission problem has been solved.
Since I want to run the CTSM with 0.05x0.05 degree resolution, I want to know can I consider the mentioned scripts and the NETCDF out put of this model as 0.05x0.05 degree resolution or I need to add the new resolution for my model?

 

[oleson]

You should be all set for this particular configuration. You should try running the model for a few days and see if it works.

 

[jteymoori]

Dear all
I ran the model with 0.05x0.05 degree resolution with the surface data set that Dr. Oleson provided for me. Although I defined the border of the United States as you see in the attached picture, the output of the model just covers a part of the region that I defined.
There is the scripts that I used to define my region:
./subset_data region --lat1 24 --lat2 50.5 --lon1 235 --lon2 295 --reg united_states --create-surface

How can I solve this problem?

 

[oleson]

Right. That is what I indicated above:

"At this point we are unable to subset a 0.05x0.05 global dataset for your specific region. We do have a subsetted dataset for a region similar to yours but smaller (CONUS). You could try working with that for now."

We just wanted to see if you could run with the region we gave you (it does look like you were able to run the model, which is good). We are still unable to subset for your larger region. We'll let you know when we solve this issue.

 

[jteymoori]

Dear Dr.Oleson
Thank you so much for your prompt response.
I will wait for your new update.