Question about setting up a global simulation (the same as the case in the CTSM tutorial)?

[mengqi]

Hi,

I am a green hand in the global simulation using CLM5. There is an official CTSM tutorial for global simulation (CTSM-Tutorial/notebooks/Day1a_GlobalCase.ipynb at main · NCAR/CTSM-Tutorial). It really helps!

I tried to set up the same case but I came across some issues with the global simulation. Here are the scripts/steps below:

1. Create a new case
./create_newcase --case ~/A_corn_simulation_region/I2000_CTSM_singlept_region_test_3 --res f45_g37 --compset I2000Clm51FatesSpRsGs --project UIUC0025 --run-unsupported

2. Set up a case and build the executable
./case.setup
./xmlchange MPILIB=mpi-serial
qcmd -- ./case.build

3. Submit the run
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=1
./xmlchange DATM_YR_ALIGN=2000
./xmlchange DATM_YR_START=2000
./xmlchange DATM_YR_END=2001

4. Submit the case
./case.submit

Unlike the successful case in the tutorial, however, I noticed that my simulation failed. It shows: "
2023-08-27 14:50:30: model execution error
ERROR: Command: ' /glade/scratch/mengqij/I2000_CTSM_singlept_region_test_3/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed with error '' from dir '/glade/scratch/mengqij/I2000_CTSM_singlept_region_test_3/run'
---------------------------------------------------
2023-08-27 14:50:30: case.run error
ERROR: RUN FAIL: Command ' /glade/scratch/mengqij/I2000_CTSM_singlept_region_test_3/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed
See log file for details: /glade/scratch/mengqij/I2000_CTSM_singlept_region_test_3/run/cesm.log.2906708.chadmin1.ib0.cheyenne.ucar.edu.230827-145024
---------------------------------------------------"

Thus, I checked the log files but I am still confused. Could anyone give some input on that? Thanks! For your review, I also attached some key files including env_build.xml, env_run.xml, and cesm.bldlog.230827-142548.

 

[oleson]

In your PET0.ESMF_LogFile file in your run directory, I see:

20230827 145030.851 ERROR PET0 esm.F90:965 Not valid - Invalid NTASKS value specified for component: cpl ntasks: 144

I think this is because you've done:

./xmlchange MPILIB=mpi-serial

I think mpi-serial is only used for single processor jobs, e.g., for a single-point simulation. Try using the default, which should be "mpt". You'll have to reset and re-build your case, or set it up from scratch again:

./case.setup --reset
./case.build --clean-all
qcmd -- ./case.build

 

[mengqi]

In your PET0.ESMF_LogFile file in your run directory, I see:

20230827 145030.851 ERROR PET0 esm.F90:965 Not valid - Invalid NTASKS value specified for component: cpl ntasks: 144

I think this is because you've done:

./xmlchange MPILIB=mpi-serial

I think mpi-serial is only used for single processor jobs, e.g., for a single-point simulation. Try using the default, which should be "mpt". You'll have to reset and re-build your case, or set it up from scratch again:

./case.setup --reset
./case.build --clean-all
qcmd -- ./case.build

Hi Oleson,

Much appreciated! It does work now for the case in the tutorial!

There is another issue below: For my case, I need to change the compset to I2000Clm50BgcCrop and fine resolution. Thus, this is the script for a new case:

1. Create a new case
./create_newcase --case ~/A_corn_simulation_region/I2000_CTSM_singlept_region_test_6 --res f02_g17 --compset I2000Clm50BgcCrop --project UIUC0025 --run-unsupported

2. Set up a case and build the executable
./case.setup
qcmd -- ./case.build

3. Submit the run
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=1
./xmlchange DATM_YR_ALIGN=2000
./xmlchange DATM_YR_START=2000
./xmlchange DATM_YR_END=2001

4. Submit the case
./case.submit

However, it failed due to "ERROR: Command /glade/u/home/mengqij/CTSM/bld/build-namelist failed rc=255 out= err=ERROR : CLM build-namelist::CLMBuildNamelist::add_default() : No default value found for fsurdat. Are defaults provided for this resolution and land mask?" My understanding is that there was no surface data. Thus, I subset the surface dataset (a region-scale instead of global-scale) and added it to the user_nl_clm:
fsurdat = '/glade/u/home/mengqij/surface_dataset_corn/surfdata_0.9x1.25_hist_78pfts_CMIP6_simyr2000_CO_region_c230825.nc'

After that, I rerun the case but it failed again. The error is from "ERROR: model_meshfile UNSET does not exist; ERROR: (dshr_mod:dshr_mesh_init) ERROR: model_meshfile UNSET does not exist"

Taken together, I ran the previous case in the CTSM tutorial and I noticed I did not consider the surface dataset and model_meshfile (they were set automatically??). However, when I changed the compset, these files could not be assigned. My understanding is right?

If so, I think I can solve how to set the surface dataset. However, I cannot select the right model_meshfile file. I know the model_meshfile file should be in the /glade/p/cesmdata/cseg/inputdata/share/meshes/. But I cannot find the right file when I use "res f02_g17". And where should I add in the user_nl_clm?

Could you give some insights about the issue? I also attached a few key files for your review.

Thank you so much!

 

[slevis]

Hi @mengqi,
I responded about mesh files here:

Question about subsetingDATM files for a regional case?

Hi, I am working on a regional simulation using CLM5. Concretely, I tried to subset DATM files but I failed. Here is my script below: :~/my_ctsm_sandbox/tools/site_and_regional> qcmd -- ./subset_data region --lat1 -40.0 --lat2 15.0 --lon1 275.0 --lon2 330.0 --reg CO_region --create-surface...

bb.cgd.ucar.edu

 

[mengqi]

Hi @mengqi,
I responded about mesh files here:

Question about subsetingDATM files for a regional case?

Hi, I am working on a regional simulation using CLM5. Concretely, I tried to subset DATM files but I failed. Here is my script below: :~/my_ctsm_sandbox/tools/site_and_regional> qcmd -- ./subset_data region --lat1 -40.0 --lat2 15.0 --lon1 275.0 --lon2 330.0 --reg CO_region --create-surface...

bb.cgd.ucar.edu

Thank you, Slevis!! Will check it!

 

[mengqi]

Hi @mengqi,
I responded about mesh files here:

Question about subsetingDATM files for a regional case?

Hi, I am working on a regional simulation using CLM5. Concretely, I tried to subset DATM files but I failed. Here is my script below: :~/my_ctsm_sandbox/tools/site_and_regional> qcmd -- ./subset_data region --lat1 -40.0 --lat2 15.0 --lon1 275.0 --lon2 330.0 --reg CO_region --create-surface...

bb.cgd.ucar.edu

Hi @slevis,

Thanks for your document! It helps!

I strictly followed your guide; however, when I submitted my case, it reported an error. In brief, it said
ERROR: Could not find all inputdata on any server

I attached detailed info in a file for your reference. Could you please have a look at it? Thanks!

 

[oleson]

It looks like it is trying to find:
/glade/p/cesmdata/cseg/inputdata/mask_mesh.nc
/glade/p/cesmdata/cseg/inputdata/lnd_mesh.nc

I don't see this in the instructions, but maybe you have to specify the full path for these files that I assume you created, in env_run.xml:
"ATM_DOMAIN_MESH"
"LND_DOMAIN_MESH"
"MASK_MESH"

@slevis ?

 

[slevis]

@oleson seems reasonable for @mengqi to try the full path, since the model says that it's looking for these files in /glade/p/cesmdata/cseg/inputdata

@mengqi let us know if it works, and I will update the github discussion with the new information. Thanks.

 

[mengqi]

To @slevis and @oleson: I rerun the case with full paths:

./xmlchange LND_DOMAIN_MESH="/glade/u/home/mengqij/CTSM/tools/site_and_regional/lnd_mesh.nc"
./xmlchange ATM_DOMAIN_MESH="/glade/u/home/mengqij/CTSM/tools/site_and_regional/lnd_mesh.nc"
./xmlchange MASK_MESH="/glade/u/home/mengqij/CTSM/tools/site_and_regional/mask_mesh.nc"

I am sure the case does work now. Thanks for your document!

To @oleson

I want to run a high-resolution (0.1 degree) case in the mid-west US (2000-2014 period). And I noticed the useful post below:

high resolution (0.1 degree) CLM5-BGC-crop regional simulation

Dear Sir: I am able to run the CLM5-BGC-crop at half degree resolution over the user-defined region (eg., USA) using the following command: create_newcase --case BgcCrop2000_USA --compset 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_SROF_SGLC_SWAV --res hcru_hcru --user-mods-dir USAdata...

bb.cgd.ucar.edu

In general, I should create a global 0.1x0.1 surface dataset and domain files and then subset them using get_regional.pl. Thus, there should be some steps. I just want to double-check if this method does work now. @oleson Thank you!

(a) TO create mapping file for gen_domain
qcmd -- ./create_ESMF_map.sh --maptype aave --filesrc /glade/p/cesmdata/cseg/inputdata/share/scripgrids/gx1v7_151008.nc --filedst /glade/p/cesmdata/cseg/inputdata/share/scripgrids/0.1x0.1_090917.nc --namesrc gx1v7 --namedst 0.1x0.1

(b) TO create domain file
./gen_domain -m /glade/work/oleson/release-cesm2.1.3/cime/tools/mapping/gen_mapping_files/gen_ESMF_mapping_file/map_gx1v7_TO_0.1x0.1_aave.230512.nc -o gx1v7 -l 0.1x0.1

(c) TO create mapping files
setenv RES 0.1x0.1
setenv GRIDFILE /glade/p/cesmdata/cseg/inputdata/lnd/clm2/mappingdata/grids/SCRIPgrid_0.1x0.1_nomask_c110712.nc
qsub ./regridbatch.sh

(d) TO create surface dataset
setenv CDATE `date +%y%m%d`
setenv GRIDNAME 0.1x0.1
qcmd -- ./mksurfdata.pl -r usrspec -usr_gname $GRIDNAME -usr_gdate $CDATE -hirespft -years 2005

 

[oleson]

Those steps worked for me at some point with a release version of the model (release-cesm2.1.3).
However, from the previous conversations, I think you are working with a version of the model that requires mesh, not domain, files.
In that case, I'm not sure what to suggest other than to do (c) and (d), subset the global surface dataset, and then generate a mesh file. Perhaps @slevis can suggest a method for generating a regional mesh file.

 

[slevis]

We offer a couple of examples for generating regional mesh files in Setting up (high-res sparse) regional-grid CTSM simulations · ESCOMP/CTSM · Discussion #1919

 

[mengqi]

We offer a couple of examples for generating regional mesh files in Setting up (high-res sparse) regional-grid CTSM simulations · ESCOMP/CTSM · Discussion #1919

Got it! Thank you @slevis!