Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

How select a special region to run the CLM from the global model in CLM?

J

jteymoori

Javad Teymoori
Member
Dr. Oleson,
Thank you for your guidance.
I tried this scripts in both derecho and casper but I face this error:
./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface --crop --cfg-file

Error:
subset_data region: error: argument --cfg-file: expected one argument

1- Do I use the following script? Do you think the Red scripts are true?

cd /CTSM/tools/site_and_regional :
./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface --crop --cfg-file
cd /CTSM/tools/site_and_regional/subset_data_regional:
cp /glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg --out-surface surfdata_midwest_hist_2000_78pfts_c240628.nc landmask.nc

ncrename -v .PFTDATA_MASK,landmask landmask.nc

ncks -A -v .PFTDATA_MASK /glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg --out-surfac surfdata_midwest_hist_2000_78pfts_c240628.nc landmask.nc
ncrename -v .PFTDATA_MASK,mod_lnd_props landmask.nc
ncks --rgr infer --rgr scrip=scrip.nc landmask.nc foo.nc
/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/oneapi/2023.2.1/7haa/bin/ESMF_Scrip2Unstruct scrip.nc lnd_mesh.nc 0
in the case directory in home : cd midwest
./xmlchange NTASKS=-4
./xmlchange NTASKS_PER_INST=144
./xmlchange JOB_WALLCLOCK_TIME=12:00:00
./xmlchange JOB_QUEUE=main
./xmlchange PIO_TYPENAME=pnetcdf
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=1
./xmlchange LND_NX=?
./xmlchange LND_NY=?
./xmlchange ATM_NX=?
./xmlchange ATM_NY=?
./xmlchange RUN_STARTDATE=2012-01-01
./case.setup

fsurdat = '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/surfdata_0.05-conus5-nomask-casper-noSP_hist_2000_78pfts_c240516.nc'
Update lines in env_run.xml:

./xmlchange LND_DOMAIN_MESH=/glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/lnd_mesh.nc
./xmlchange ATM_DOMAIN_MESH=/glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/lnd_mesh.nc
./xmlchange MASK_MESH=/glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/lnd_mesh.nc

./case.build
./case.submit

2- What values should be entered for LND_NX, LND_NY, ATM_NX, ATM_NY ؟
 
slevis

slevis

Moderator
Please read @oleson's instructions again. From what I can tell, you did not type the cfg file path and name when running ./subset_data

...which is what that error message means.
 
J

jteymoori

Javad Teymoori
Member
oleson said:
This command you've used:

./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface

creates a surface dataset from the global 1deg data by default.
If you want to subset from a different surface dataset, e.g, the global 0.05x0.05, then you need to specify it by modifying the configuration file (default_data_2000.cfg).
I've created a configuration file that will do that. It is here:

/glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg

So try this command:

./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface --crop --cfg-file /glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg --out-surface surfdata_midwest_hist_2000_78pfts_c240628.nc

You may need to run it on casper to get enough memory.
Then you'll need to create new mesh files using the method you used previously.
Click to expand...


Dr. Oleson
Thank you for your guidance,
I used your configuration file (/glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg) and for subset my region "midwest" I used your scripts, but I faced the below error:
(npl) jteymoori@casper-login2:/glade/work/jteymoori/CTSM/tools/site_and_regional> ./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface --crop --cfg-file /glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg --out-surface surfdata_midwest_hist_2000_78pfts_c240628.nc
Traceback (most recent call last):
File "/glade/u/apps/opt/conda/envs/npl/lib/python3.10/configparser.py", line 791, in get
value = d[option]
File "/glade/u/apps/opt/conda/envs/npl/lib/python3.10/collections/__init__.py", line 986, in __getitem__
return self.__missing__(key) # support subclasses that define __missing__
File "/glade/u/apps/opt/conda/envs/npl/lib/python3.10/collections/__init__.py", line 978, in __missing__
raise KeyError(key)
KeyError: 'mesh_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/glade/work/jteymoori/CTSM/tools/site_and_regional/./subset_data", line 37, in <module>
main()
File "/glade/work/jteymoori/CTSM/tools/site_and_regional/../../python/ctsm/subset_data.py", line 794, in main
file_dict = setup_files(args, defaults, cesmroot)
File "/glade/work/jteymoori/CTSM/tools/site_and_regional/../../python/ctsm/subset_data.py", line 602, in setup_files
os.path.join(defaults.get("surfdat", "mesh_dir")),
File "/glade/u/apps/opt/conda/envs/npl/lib/python3.10/configparser.py", line 794, in get
raise NoOptionError(option, section)
configparser.NoOptionError: No option 'mesh_dir' in section: 'surfdat'

To solve this error I tried to add the " mesh_dir " option under the [surfdat] section: by the below scripts in your configuration file (/glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg):

[surfdat]
mesh_dir = /glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional

but I found this cfg is just read-only and I can not add or edit it, I faced the below error:

E45: 'readonly' option is set (add ! to override)
 
J

jteymoori

Javad Teymoori
Member
jteymoori said:
Dr. Oleson
Thank you for your guidance,
I used your configuration file (/glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg) and for subset my region "midwest" I used your scripts, but I faced the below error:
(npl) jteymoori@casper-login2:/glade/work/jteymoori/CTSM/tools/site_and_regional> ./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface --crop --cfg-file /glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg --out-surface surfdata_midwest_hist_2000_78pfts_c240628.nc
Traceback (most recent call last):
File "/glade/u/apps/opt/conda/envs/npl/lib/python3.10/configparser.py", line 791, in get
value = d[option]
File "/glade/u/apps/opt/conda/envs/npl/lib/python3.10/collections/__init__.py", line 986, in __getitem__
return self.__missing__(key) # support subclasses that define __missing__
File "/glade/u/apps/opt/conda/envs/npl/lib/python3.10/collections/__init__.py", line 978, in __missing__
raise KeyError(key)
KeyError: 'mesh_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/glade/work/jteymoori/CTSM/tools/site_and_regional/./subset_data", line 37, in <module>
main()
File "/glade/work/jteymoori/CTSM/tools/site_and_regional/../../python/ctsm/subset_data.py", line 794, in main
file_dict = setup_files(args, defaults, cesmroot)
File "/glade/work/jteymoori/CTSM/tools/site_and_regional/../../python/ctsm/subset_data.py", line 602, in setup_files
os.path.join(defaults.get("surfdat", "mesh_dir")),
File "/glade/u/apps/opt/conda/envs/npl/lib/python3.10/configparser.py", line 794, in get
raise NoOptionError(option, section)
configparser.NoOptionError: No option 'mesh_dir' in section: 'surfdat'

To solve this error I tried to add the " mesh_dir " option under the [surfdat] section: by the below scripts in your configuration file (/glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg):

[surfdat]
mesh_dir = /glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional

but I found this cfg is just read-only and I can not add or edit it, I faced the below error:

E45: 'readonly' option is set (add ! to override)
Click to expand...
Dear Dr. @oleson
Please accept my apologies for asking you for help with my problem again.
I have run the CTSM with 0.05x0.05 degree resolution for the whole U.S. with the default DATM; now, I want to run it for a small region (Midwest) and then change the DATM for this region.
To run the CTSM in the Midwest, I used your guidance, which provided the cfg file for the subset of my region (Midwest) from the global 0.05x0.05 degree, but I faced an error, which I mentioned in the previous post.
I read the jack, merignition, and majun questions for changing the DATM, but firstly, I need to run the CTSM in the Midwest. Secondly, try to change the DATM based on your guidance and the works of others such as jack, merignition, and majunwho had the same goal as me.
How can I solve this error?
 
slevis

slevis

Moderator
If you are still asking about the "readonly" error, and you are using the vi editor, then I think you should try the advice given to you in the error message. The suggestion is to add "!" after the "w" when you try to save the modified file.
 
J

jteymoori

Javad Teymoori
Member
oleson said:
This command you've used:

./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface

creates a surface dataset from the global 1deg data by default.
If you want to subset from a different surface dataset, e.g, the global 0.05x0.05, then you need to specify it by modifying the configuration file (default_data_2000.cfg).
I've created a configuration file that will do that. It is here:

/glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg

So try this command:

./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface --crop --cfg-file /glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg --out-surface surfdata_midwest_hist_2000_78pfts_c240628.nc

You may need to run it on casper to get enough memory.
Then you'll need to create new mesh files using the method you used previously.
Click to expand...

Thank you for your guidance.
I ran the scripts below:
./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface --crop --cfg-file /glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg --out-surface surfdata_midwest_hist_2000_78pfts_c240717.nc

But, I encountered two errors related to the absence of "mesh_dir" and "mesh_surf" in the .cfg file. I tried to add "mesh_dir" and "mesh_surf" in 0.05-conus6_data.cfg, but this file was read-only, and I could not add or edit it. So I copied the 0.05-conus6_data.cfg from the "/glade/work/oleson/ctsm5.2.0/tools/site_and_regional " into my directory in the "/glade/work/jteymoori/CTSM/tools/site_and_regional " , then I added "mesh_dir" and "mesh_surf" to it as a below:
[surfdat]
mesh_dir = /glade/work/jteymoori/CTSM/tools/site_and_redional
mesh_surf = /glade/work/jteymoori/CTSM/tools/site_and_redional

Then The script for subsetting from the global 0.05x0.05 data changed to:
./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface --crop --cfg-file /glade/work/jteymoori/CTSM/tools/site_and_regional/0.05-conus6_data.cfg --out-surface surfdata_midwest_hist_2000_78pfts_c240717.nc
By running the last scripts, I faced an error below:

PermissionError: [Errno 13] Permission denied: '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/GLOBAL/surfdata_0.05x0.05-hires_noSP_hist_2005_78pfts_c240516.nc'

I tried to solve this error by copying the surfdata_0.05x0.05-hires_noSP_hist_2005_78pfts_c240516.nc'
from the /glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/GLOBAL
to my case directory as : /glade/work/jteymoori/CTSM/tools/site_and_regional
by using this scripts:
cp /glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/GLOBAL/surfdata_0.05x0.05-hires_hist_2005_78pfts_c240425.nc /glade/work/jteymoori/CTSM/tools/site_and_regional

Then I ran the below scripts again to subset the regional surface data set from global 0.05x0.05 as below:
./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface --crop --cfg-file /glade/work/jteymoori/CTSM/tools/site_and_regional/0.05-conus6_data.cfg --out-surface surfdata_midwest_hist_2000_78pfts_c240717.nc

But I faced the same error for permission :
PermissionError: [Errno 13] Permission denied: '/glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/GLOBAL/surfdata_0.05x0.05-hires_noSP_hist_2005_78pfts_c240516.nc'

How can I solve this error?
1721235184493.png
 
J

jteymoori

Javad Teymoori
Member
oleson said:
I've changed the permissions on that file, try again please.
Click to expand...
Dear Dr. Oleson,
Thank you for your great helps.
As you said, I did all this work in Casper, but when I wanted do ./case.setup for my case, I faced an error which says it could not create a directory in /glade/scratch by Casper machine:

Setting resource.RLIMIT_STACK to -1 from (8388608, -1)
Machine/Decomp/Pes configuration has already been done ...skipping
ERROR: Could not make directory '/glade/scratch/jteymoori/midwest_clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000/bld/cpl/obj', error: [Errno 13] Permission denied: '/glade/scratch'

To solve this error, Can I do all the previous steps in Derecho instead of Casper?
 
O

oleson

Keith Oleson
CSEG and Liaisons
Staff member
I would do your subset_data steps on casper, but you'll want to do all of your case setup for your simulation on derecho.
 
J

jteymoori

Javad Teymoori
Member
According to your recommendation, I have also done all the steps, including create_newcas, subset_data, Running the CTSM, and .. on the Casper environment, and now I have encountered this error during ./case. setup to run the model.
I am now looking to fix this error.
 
O

oleson

Keith Oleson
CSEG and Liaisons
Staff member
Don't do create_newcase, ./case.setup, etc. on Casper, do them on Derecho. subset_data can be run on casper. It is independent of the case creation and case setup process and can be run anytime.
 
J

jteymoori

Javad Teymoori
Member
oleson said:
Don't do create_newcase, ./case.setup, etc. on Casper, do them on Derecho. subset_data can be run on casper. It is independent of the case creation and case setup process and can be run anytime.
Click to expand...
Dear all
Based on Oleson's guidance, I could subset a small region from the global 0.05x0.05 surface dataset for running the CTSM with 0.05-degree resolution.
After ./case submit, I received an email as below:

PBS Job Id: 5371545.desched1
Job Name: st_archive.midwest_clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000
Aborted by PBS Server
Job deleted as result of dependency on job 5371544.desched1.

Since I had received the same email before, and based on Oleson's guidance, I checked the log file in my run directory, which seems to be /glade/derecho/scratch/jteymoori/midwest_clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000/run, I see some ERROR and Invalid log file in my cesm log directory.
Do you think the job deleting my model is related to these log files, or is it related to other things?
How can I solve this error?


These are the scripts that I used for this model:
#in derecho
cd /glade/work/jteymoori
git clone GitHub - ESCOMP/CTSM: Community Terrestrial Systems Model (includes the Community Land Model of CESM)
cd CTSM
./bin/git-fleximod update
cp /glade/work/oleson/ctsm5.2.0/tools/site_and_regional/0.05-conus6_data.cfg /glade/work/jteymoori/CTSM/tools/site_and_regional/
cd/CTSM/tools/site_and_reginal
adding "mesh_dir" and "mesh_surf" to the 0.05-conus6_data.cfg
vi 0.05-conus6_data.cfg
[surfdat]
mesh_dir = /glade/work/jteymoori/CTSM/tools/site_and_redional
mesh_surf = /glade/work/jteymoori/CTSM/tools/site_and_redional


cp /glade/campaign/cgd/tss/people/oleson/CLM5_datasets/ctsm5.2.0/GLOBAL/surfdata_0.05x0.05-hires_noSP_hist_2005_78pfts_c240516.nc /glade/work/jteymoori/CTSM/tools/site_and_regional


cd /CTSM/cime/scripts
module load conda
conda activate npl
./create_newcase --case ~/midwest/midwest_clm50_ctsm520_0.05x0.05conus_GSWP3V1_2000 --res CLM_USRDAT --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_SGLC_SWAV --run-unsupported --project UCNN0040

#subset from global 0.05x0.05 surface dataset in Casper

cd CTSM/tools/site_and_regional:
pip install numpy
pip install xarray
./subset_data region --lat1 30 --lat2 45 --lon1 255 --lon2 270 --reg midwest --create-surface --crop --cfg-file /glade/work/jteymoori/CTSM/tools/site_and_regional/0.05-conus6_data.cfg --out-surface surfdata_midwest_hist_2000_78pfts_c240723.nc
ctsm/tools/site_and_regional/subset_data_regional:
module load nco
cp /glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/surfdata_midwest_hist_2000_78pfts_c240723.nc landmask.nc
ncrename -v .PFTDATA_MASK,landmask landmask.nc
ncks -A -v .PFTDATA_MASK /glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/surfdata_midwest_hist_2000_78pfts_c240723.nc landmask.nc
ncrename -v .PFTDATA_MASK,mod_lnd_props landmask.nc
ncks --rgr infer --rgr scrip=scrip.nc landmask.nc foo.nc

in derecho
cd /tools/site_and_regional/subset_data_regional
/glade/u/apps/derecho/23.09/spack/opt/spack/esmf/8.6.0/cray-mpich/8.1.27/oneapi/2023.2.1/7haa/bin/ESMF_Scrip2Unstruct scrip.nc lnd_mesh.nc 0

in case directory:
module load conda
conda activate npl
./xmlchange NTASKS=-4
./xmlchange NTASKS_PER_INST=144
./xmlchange JOB_WALLCLOCK_TIME=12:00:00
./xmlchange JOB_QUEUE=main
./xmlchange PIO_TYPENAME=pnetcdf
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=1
./xmlchange LND_NX=13
./xmlchange LND_NY=17
./xmlchange ATM_NX=13
./xmlchange ATM_NY=17
./xmlchange RUN_STARTDATE=2012-01-01
./case.setup

vi user_nl_clm
fsurdat = '/glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/surfdata_midwest_hist_2000_78pfts_c240723.nc'
hist_mfilt=12
hist_nhtfrq=0

./xmlchange LND_DOMAIN_MESH=/glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/lnd_mesh.nc
./xmlchange ATM_DOMAIN_MESH=/glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/lnd_mesh.nc
./xmlchange MASK_MESH=/glade/work/jteymoori/CTSM/tools/site_and_regional/subset_data_regional/lnd_mesh.nc
./case.build
./case.submit
 
O

oleson

Keith Oleson
CSEG and Liaisons
Staff member
The system is deleting your short-term archiver job because the main job failed.
I think the problem is that your fsurdat is missing a dimension that is required, lsmpft.

Try executing the following to add that dimension to your surface dataset and then run the case again:

ncap2 -h -A -s 'defdim("lsmpft",79)' surfdata_midwest_hist_2000_78pfts_c240723.nc
 
You must log in or register to reply here.
Top