How select a special region to run the CLM from the global model in CLM?

[oleson]

Ok, we have a dataset for your region. It should work for your configuration (BGC-Crop).

1. Add the following to your user_nl_clm in your case directory: fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/surfdata_0.05-conus5-nomask-casper-noSP_hist_2000_78pfts_c240516.nc'

2. Make the following xml changes in your case directory:

./xmlchange ATM_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/lnd_mesh.nc
./xmlchange LND_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/lnd_mesh.nc
./xmlchange MASK_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/mask_mesh.nc

 

[jteymoori]

Ok, we have a dataset for your region. It should work for your configuration (BGC-Crop).

1. Add the following to your user_nl_clm in your case directory: fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/surfdata_0.05-conus5-nomask-casper-noSP_hist_2000_78pfts_c240516.nc'

2. Make the following xml changes in your case directory:

./xmlchange ATM_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/lnd_mesh.nc
./xmlchange LND_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/lnd_mesh.nc
./xmlchange MASK_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/mask_mesh.nc

Dr. Oleson
Thank you so much for your new update.
I tried creating a model with this new dataset, but I faced an error after creating the model and changing the XML in the case directory, although this script was executed without any error in the past.

For example, when I ran the ./xmlchange NTASKS=-4 in my case directory I faced this error:
ERROR: Directory None does not appear to be a valid case directory

Here are all of the scripts that I ran:

cd /glade/work/jteymoori
git clone GitHub - ESCOMP/CTSM: Community Terrestrial Systems Model (includes the Community Land Model of CESM)
cd CTSM
./bin/git-fleximod update
cd /CTSM/cime/scripts
module load conda
conda activate npl
./create_newcase --case ~/0.05x0.05_June12/clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000 --res CLM_USRDAT --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_SGLC_SWAV --run-unsupported --project UCNN0040

cd CTSM/tools/site_and_regional:
pip install numpy
pip install xarray
./subset_data region --lat1 24.2 --lat2 50.5 --lon1 235 --lon2 295 --reg united_states --create-surface
cd CTSM/tools/site_and_regional/subset_data_regional:
module load nco
cp /glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/surfdata_united_states_hist_2000_16pfts_c240612.nc landmask.nc
ncrename -v .PFTDATA_MASK,landmask landmask.nc
ncks -A -v .PFTDATA_MASK /glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/surfdata_united_states_hist_2000_16pfts_c240612.nc landmask.nc
ncrename -v .PFTDATA_MASK,mod_lnd_props landmask.nc
ncks --rgr infer --rgr scrip=scrip.nc landmask.nc foo.nc

/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/oneapi/2023.2.1/7haa/bin/ESMF_Scrip2Unstruct scrip.nc lnd_mesh.nc 0

cd case directory:
cd /glade/u/home/jteymoori/0.05x0.05_June12/clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000>

module load conda
conda activate npl
./xmlchange NTASKS=-4
./xmlchange NTASKS_PER_INST=144
./xmlchange JOB_WALLCLOCK_TIME=12:00:00
./xmlchange JOB_QUEUE=main
./xmlchange PIO_TYPENAME=pnetcdf
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=1
./xmlchange LND_NX=1200
./xmlchange LND_NY=562
./xmlchange ATM_NX=1200
./xmlchange ATM_NY=562
./case.setup

vi user_nl_clm

fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/surfdata_0.05-conus5-nomask-casper-noSP_hist_2000_78pfts_c240516.nc'
hist_mfilt=12
hist_nhtfrq=0
Esc
:wq

Update lines in env_run.xml:
./xmlchange ATM_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/lnd_mesh.nc
./xmlchange LND_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/lnd_mesh.nc
./xmlchange MASK_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/mask_mesh.nc



./case.build
./case.submit

 

[oleson]

I see that you have some zero-length files in your case directory (which I assume is /glade/u/home/jteymoori/0.05x0.05_June12/clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000):

-rw-r--r-- 1 jteymoori ncar 0 Jun 12 08:59 env_case.xml
-rw-r--r-- 1 jteymoori ncar 0 Jun 12 08:59 env_run.xml
-rw-r--r-- 1 jteymoori ncar 0 Jun 12 08:59 env_build.xml

So it seems like the case creation didn't finish properly perhaps.
Can you post the output from your create_newcase command which was ./create_newcase --case /glade/u/home/jteymoori/0.05x0.05_June12/clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000 --res CLM_USRDAT --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_SGLC_SWAV --run-unsupported --project UCNN0040

Can you also post the output you get from this command in the top level of your CTSM clone:

./bin/git-fleximod status

 

[jteymoori]

jteymoori@derecho6:/glade/work/jteymoori/CTSM> ./bin/git-fleximod update
fates at tag sci.1.73.0_api.35.0.0
cism at tag cismwrap_2_2_001
fates up to date.
Recursively checking out submodules of cism

e source_cism not checked out, aligned at tag cism_main_2.01.013
root - ERROR - Failed to checkout source_cism False None /glade/work/jteymoori/CTSM/components/cism/source_cism source_cism
Traceback (most recent call last):
File "/glade/work/jteymoori/CTSM/./bin/git-fleximod", line 8, in <module>
sys.exit(main())
File "/glade/work/jteymoori/CTSM/.lib/git-fleximod/git_fleximod/git_fleximod.py", line 588, in main
submodules_update(gitmodules, root_dir, fxrequired, force)
File "/glade/work/jteymoori/CTSM/.lib/git-fleximod/git_fleximod/git_fleximod.py", line 394, in submodules_update
single_submodule_checkout(
File "/glade/work/jteymoori/CTSM/.lib/git-fleximod/git_fleximod/git_fleximod.py", line 255, in single_submodule_checkout
submodules_checkout(gitmodules, repodir, requiredlist, force=force)
File "/glade/work/jteymoori/CTSM/.lib/git-fleximod/git_fleximod/git_fleximod.py", line 500, in submodules_checkout
single_submodule_checkout(
File "/glade/work/jteymoori/CTSM/.lib/git-fleximod/git_fleximod/git_fleximod.py", line 257, in single_submodule_checkout
utils.fatal_error(
File "/glade/work/jteymoori/CTSM/.lib/git-fleximod/git_fleximod/utils.py", line 130, in fatal_error
raise RuntimeError("{0}ERROR: {1}".format(os.linesep, message))
RuntimeError:
ERROR: Failed to checkout source_cism False None /glade/work/jteymoori/CTSM/components/cism/source_cism source_cism



jteymoori@derecho6:/glade/work/jteymoori/CTSM> ./bin/git-fleximod status
fates at tag sci.1.73.0_api.35.0.0
cism at tag cismwrap_2_2_001
e rtm not checked out, aligned at tag rtm1_0_79
e mosart not checked out, aligned at tag mosart1_0_49
e mizuRoute not checked out, aligned at tag cesm-coupling.n02_v2.1.2
e ccs_config not checked out, aligned at tag ccs_config_cesm0.0.106
e cime not checked out, aligned at tag cime6.0.246
e cmeps not checked out, aligned at tag cmeps0.14.63
e cdeps not checked out, aligned at tag cdeps1.0.34
e share not checked out, aligned at tag share1.0.19
e mct not checked out, aligned at tag MCT_2.11.0
e parallelio not checked out, aligned at tag pio2_6_2
e doc-builder not checked out, aligned at tag v1.0.8
testfails = 0, local mods = 0, needs updates 11

 

[oleson]

I tried cloning the latest CTSM and running ./bin/git-fleximod update and it seemed to work fine on Derecho.
I guess I'd suggest deleting your /glade/work/jteymoori/CTSM directory (unless you have stuff there you want to save) and try cloning and running ./bin/git-fleximod again. There must be something wrong with your installation, not sure what.

 

[jteymoori]

I tried cloning the latest CTSM and running ./bin/git-fleximod update and it seemed to work fine on Derecho.
I guess I'd suggest deleting your /glade/work/jteymoori/CTSM directory (unless you have stuff there you want to save) and try cloning and running ./bin/git-fleximod again. There must be something wrong with your installation, not sure what.

Dr. Oleson
Thank you for your guidance.
I deleted the CTSM directory and cloned it again by ./bin/git-fleximod update, this script ran without an error. Next, to build the model when running the :
./xmlchange NTASKS=-4 I faced the same previous error :ERROR: Directory None does not appear to be a valid case directory.
I checked the case directory in the :
glade/u/home/jteymoori/0.05x0.05_June13/clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000
I saw that I have some zero-length files including :

-rw-r--r-- 1 jteymoori ncar 6121 Jun 13 15:02 env_archive.xml
-rw-r--r-- 1 jteymoori ncar 2483 Jun 13 15:02 env_batch.xml
-rw-r--r-- 1 jteymoori ncar 0 Jun 13 15:02 env_build.xml
-rw-r--r-- 1 jteymoori ncar 0 Jun 13 15:02 env_case.xml
-rw-r--r-- 1 jteymoori ncar 7217 Jun 13 15:02 env_mach_pes.xml
-rw-r--r-- 1 jteymoori ncar 4217 Jun 13 15:02 env_mach_specific.xml
-rw-r--r-- 1 jteymoori ncar 0 Jun 13 15:02 env_run.xml
-rw-r--r-- 1 jteymoori ncar 4107 Jun 13 15:02 env_workflow.xml
drwxr-xr-x 2 jteymoori ncar 4096 Jun 13 15:02 LockedFiles

I have attached screenshots of the errors for your review.

 

[oleson]

When I run ./bin/git-fleximod status in your CTSM clone I get:

s fates xxxx is out of sync with .gitmodules sci.1.73.0_api.35.0.0
M
s cism xxxx is out of sync with .gitmodules cismwrap_2_2_001
M
s rtm xxxx is out of sync with .gitmodules rtm1_0_79
M
s mosart xxxx is out of sync with .gitmodules mosart1_0_49
M
s mizuRoute xxxx is out of sync with .gitmodules cesm-coupling.n02_v2.1.2
M
s ccs_config xxxx is out of sync with .gitmodules ccs_config_cesm0.0.106
M
s cime xxxx is out of sync with .gitmodules cime6.0.246
M
s cmeps xxxx is out of sync with .gitmodules cmeps0.14.63
M
s cdeps xxxx is out of sync with .gitmodules cdeps1.0.34
M
s share xxxx is out of sync with .gitmodules share1.0.19
M
s mct xxxx is out of sync with .gitmodules MCT_2.11.0
M
s parallelio xxxx is out of sync with .gitmodules pio2_6_2
M
e doc-builder not checked out, aligned at tag v1.0.8
testfails = 12, local mods = 12, needs updates 13

The submodules labeled with 'M' above are not in a clean state.
The following are options for how to proceed:
(1) Go into each submodule which is not in a clean state and issue a 'git status'
Either revert or commit your changes so that the submodule is in a clean state.
(2) use the --force option to git-fleximod
(3) you can name the particular submodules to update using the git-fleximod command line
(4) As a last resort you can remove the submodule (via 'rm -fr [directory]')
then rerun git-fleximod update.

@erik , @slevis , do you have any thoughts on why git-fleximod isn't working for this user?

 

[slevis]

@jedwards may be the better person to help with this...

 

[jedwards]

I don't think that this is a git-fleximod issue - I suspect that the user is exceeding their quota on the glade filesystem - possibly work?

 

[oleson]

That's a good idea. @jteymoori , you can check your disc usage using "gladequota".

 

[jteymoori]

That's a good idea. @jteymoori , you can check your disc usage using "gladequota".

Dr.Oleson
Thank you so much for your efforts to solve my problem.
I deleted some old models I had made. Fortunately, the storage space reached enough space to allow me to set up the new model again.
I created my new case, but after submit I received an email with this theme:

" PBS Job Id: 4853262.desched1
Job Name: st_archive.clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000
Aborted by PBS Server
Job deleted as a result of dependency on job 4853261.desched1."


Based on received emails similar to this one for other models, I have made before, and based on your new data set provided to me for adding user_nl_clm in my case directory as below:

fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/surfdata_0.05-conus5-nomask-casper-noSP_hist_2000_78pfts_c240516.nc'

Although the /glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0 directory includes surfdata_0.05-conus5-nomask-casper-noSP_hist_2000_78pfts_c240516.nc, I received that email.

 

[jteymoori]

The reason for the previous similar errors was the lack of permission to enter the below directory:

/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0

I now have access and permission to the contents of this folder, but I am still facing the error.
Is there a good solution to fix this error?

 

[jteymoori]

Dr.Oleson
Thank you so much for your efforts to solve my problem.
I deleted some old models I had made. Fortunately, the storage space reached enough space to allow me to set up the new model again.
I created my new case, but after submit I received an email with this theme:

" PBS Job Id: 4853262.desched1
Job Name: st_archive.clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000
Aborted by PBS Server
Job deleted as a result of dependency on job 4853261.desched1."


Based on received emails similar to this one for other models, I have made before, and based on your new data set provided to me for adding user_nl_clm in my case directory as below:

fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/surfdata_0.05-conus5-nomask-casper-noSP_hist_2000_78pfts_c240516.nc'

Although the /glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0 directory includes surfdata_0.05-conus5-nomask-casper-noSP_hist_2000_78pfts_c240516.nc, I received that email.


View attachment 5498

The
-rw-r-----+ 1 oleson cgdtss 3406280392 May 16 13:09 surfdata_0.05-conus5-nomask-casper-noSP_hist_2000_78pfts_c240516.nc
needs permission to execute and read; please give me permission to read and execute it.

 

[oleson]

Done

 

[jteymoori]

Dear all
I have run the CTSM 5.2 for all of the U.S with the below surface data set:
fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/surfdata_0.05-conus5-nomask-casper-noSP_hist_2000_78pfts_c240516.nc'

and
./xmlchange ATM_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/lnd_mesh.nc
./xmlchange LND_DOMAIN_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/lnd_mesh.nc
./xmlchange MASK_MESH=/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/mask_mesh.nc

Now I want to run the model for a specific region with same resolution like pervious model, although by define a new domain for this new region
(./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface), the output of this new domain was the same for hole the U.S not that region.

I tried to solve it by using same surface data set the following :
fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/surfdata_0.05-conus5-nomask-casper-noSP_hist_2000_78pfts_c240516.nc'

but for update lines in env_run.xml, I used the lnd_mesh.nc in my case root as the following :

./xmlchange LND_DOMAIN_MESH=/glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/lnd_mesh.nc
./xmlchange ATM_DOMAIN_MESH=/glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/lnd_mesh.nc
./xmlchange MASK_MESH=/glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/lnd_mesh.nc

After submit the model I received the following email:
PBS Job Id: 4967065.desched1
Job Name: st_archive.clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000
Aborted by PBS Server
Job deleted as result of dependency on job 4967064.desched1

I think for solve this error, in the update lines in env_run.xml for LND_DOMAIN_MESH, ATM_DOMAIN_MESH, MASK_MESH, should use another data especially for ATM_DOMAIN_MESH.
How can I solve this error?

 

[oleson]

The mesh files and surface dataset files you are using don't match in terms of number of gridcells.
lnd_mesh.nc has:

elementCount = 1421

while the surface dataset you are using has:

lsmlon = 1200 ;
lsmlat = 616 ;

which means the elementCount in your mesh file should be 1200 X 616 = 739,200

 

[jteymoori]

Thank you, Dr. Oleson
Do you mean I should change lsmlon = 38 and lsmlat = 38?
Since my region is It is a square whose length and width are equal to 15 degrees so :
1412 = 37.69 X 37.69 ~= 38
How can I change the grid cells of the surface dataset?

 

[oleson]

You would need to create a new surface dataset.

I see this in your directory:

surfdata_US_hist_2000_16pfts_c240627.nc

which has:

lsmlat = 29 ;
lsmlon = 49 ;


So, 29 X 49= 1421 which matches the elementCount in your mesh files.

Is that the region and spatial resolution you want to simulate?
I also see you mentioned "I want to run the model for a specific region with same resolution like pervious model, although by define a new domain for this new region".
Does this mean you want to run 0.05x0.05 resolution over just the midwest? If so, you would need surface datasets and mesh files that are subsetted from the global 0.05x0.05 dataset. The ones you created were subsetted from the global 1deg dataset, not the 0.05x0.05 global dataset.

 

[jteymoori]

Yes, I want to run 0.05x0.05 resolution over just the Midwest for this I used the surface data set with 0.05x0.05 resolution for the whole U.S which is provided to me by you as following:

fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/surfdata_0.05-conus5-nomask-casper-noSP_hist_2000_78pfts_c240516.nc'

and then I defined the domain of this region as follwing :
./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface
and then I updated lines in env_run.xml as the following:
./xmlchange LND_DOMAIN_MESH=/glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/lnd_mesh.nc
./xmlchange ATM_DOMAIN_MESH=/glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/lnd_mesh.nc
./xmlchange MASK_MESH=/glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/lnd_mesh.nc

The output of the model is for the whole U.S not for the Midwest.
How can subset the Midwest from the global 0.05x0.05 dataset?

 

[oleson]

This command you've used:

./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface

creates a surface dataset from the global 1deg data by default.
If you want to subset from a different surface dataset, e.g, the global 0.05x0.05, then you need to specify it by modifying the configuration file (default_data_2000.cfg).
I've created a configuration file that will do that. It is here:

/glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg

So try this command:

./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface --crop --cfg-file /glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg --out-surface surfdata_midwest_hist_2000_78pfts_c240628.nc

You may need to run it on casper to get enough memory.
Then you'll need to create new mesh files using the method you used previously.