Question about running a regional case with generating a mesh file?

[mengqi]

Hi @slevis and @oleson,

According to the guidebook (Setting up (high-res sparse) regional-grid CTSM simulations · ESCOMP/CTSM · Discussion #1919), I am able to run the CLM5-BGC-crop at regular 1-degree resolution in the mid-west US. However, I want to run a high-resolution (0.125 degree) case in the mid-west US. I got stuck here for several days. Could you please give me some input here? Thanks!

I used the following command:

1. Subset the surface data
~/CTSM/tools/site_and_regional> qcmd -- ./subset_data region --lat1 37.00 --lat2 49.00 --lon1 -105.00 --lon2 -80.00 --reg CO_region --create-surface --create-user-mods --datm-syr 2000 --datm-eyr 2005 --crop --outdir /glade/scratch/mengqij/my_subset_data_Bo1_region_1

Note: This is surface datafor mid-west US.

2. Create a new case
./create_newcase --case ~/A_corn_simulation_region/I2000_CTSM_singlept_region_test_a --res ne240_ne240_mg17 --compset I2000Clm51BgcCrop --project UIUC0025 --run-unsupported

Note: I changed the --res from "CLM_USRDAT" to "ne240_ne240_mg17". ne240_ne240_mg17 is one of the supported resolutions.

3. Generate mesh files (note the Scrip2Unstruct command has been updated in Discussion 1919)
In ctsm/tools/site_and_regional/
module load nco
ncks --rgr infer --rgr scrip=scrip.nc ~/surface_dataset_corn/surfdata_CO_region_hist_78pfts_CMIP6_simyr2000_c230828.nc foo.nc
/glade/u/apps/ch/opt/esmf-netcdf/8.0.0/intel/19.0.5/bin/bing/Linux.intel.64.mpiuni.default/ESMF_Scrip2Unstruct scrip.nc lnd_mesh.nc 0
ncap2 -s 'elementMask(:)=0' lnd_mesh.nc mask_mesh.nc

Note: I did not change these commands. Right?

4. Set up a case and build the executable
./case.setup
qcmd -- ./case.build

5. Submit the run
./xmlchange STOP_OPTION=nyears.
./xmlchange STOP_N=1
./xmlchange DATM_YR_ALIGN=2000./xmlchange DATM_YR_START=2000
./xmlchange DATM_YR_END=2001
./xmlchange LND_DOMAIN_MESH= "$DIN_LOC_ROOT/share/meshes/ne240np4_ESMFmesh_cdf5_c20211018.nc"
./xmlchange ATM_DOMAIN_MESH= "$DIN_LOC_ROOT/share/meshes/ne240np4_ESMFmesh_cdf5_c20211018.nc"
./xmlchange MASK_MESH= "$DIN_LOC_ROOT/share/meshes/gx1v7_151008_ESMFmesh.nc"

Note: For LND_DOMAIN_MESH, ATM_DOMAIN_MESH, and MASK_MESH, I noticed that these variables were automatically set.

However, in the guidebook, they seem to be:
./xmlchange LND_DOMAIN_MESH='/path/to/your/lnd_mesh.nc'
./xmlchange ATM_DOMAIN_MESH='/path/to/your/lnd_mesh.nc'
./xmlchange MASK_MESH='/path/to/your/mask_mesh.nc'

Thus, I am not sure if I should revise the default values.

6. Submit the case
./case.submit

The run does not work. I attached a log file for your reference. In brief, it said:

Input land mesh file /glade/p/cesmdata/cseg/inputdata/share/meshes/ne240np4_ESMFmesh_cdf5_c20211018.nc
Input mask mesh file /glade/p/cesmdata/cseg/inputdata/share/meshes/gx1v7_151008_ESMFmesh.nc
Obtaining land mask and fraction from mask file /glade/p/cesmdata/cseg/inputdata/share/meshes/gx1v7_151008_ESMFmesh.nc

Attempting to read global dimensions from surface dataset
(GETFIL): attempting to find local file
surfdata_CO_region_hist_78pfts_CMIP6_simyr2000_c230828.nc
(GETFIL): using /glade/u/home/mengqij/surface_dataset_corn/surfdata_CO_region_hist_78pfts_CMIP6_simyr2000_c230828.nc
global ni,nj = 20 13
model grid is 2-dimensional

Computing land fraction and land mask by mapping mask from mesh_mask file
decompInit_lnd(): Number of processes exceeds number of land grid cells
1152 0
ENDRUN:
ERROR: ERROR in decompInitMod.F90 at line 170

 

[slevis]

In step 3 you created new mesh files. In step 5 you need to change the pre-filled mesh files to point to the new mesh files. This may be the only problem.

However, I also wonder if you are sure that you subset a 0.125 file when you used subset_data to create a new surface dataset? If not, then this would be another problem.

 

[mengqi]

Thank you @slevis I changed the pre-filled mesh files but it still failed. However, your second point makes sense but it leads me to my outstanding question.

In general, there are two methods to subset surface data. One is using ./subset_data command, the other is using getregional_datasets.pl command. For the former, it does work when I carry out both a single-point run and the second case (1-degree) in your instruction. For the latter, I never use it.

According to your advice, I tried to look at the surface data in detail using ./subset_data command. You can see information about the surface data file's Attributes below. It said this is a 0.25-degree surface data. Thus, It is a bit weird because I did not change anything. It suggests that 0.25 is a pre-filled value?

In this context, should I use getregional_datasets.pl command to get a 0.125-degree surface data? If so, could you please give me a relevant case? I would like to look at how I use getregional_datasets.pl to obtain a 0.125-degree surface data file.

Thank you!



Attributes:
Conventions :NCAR-CSMSource :Community Land Model: CLM5no_inlandwet :TRUEnglcec :10Input_grid_dataset :map_0.25x0.25_MODIS_to_0.9x1.25_nomask_aave_da_c170321.ncInput_gridtype :globalVOC_EF_raw_data_file_name :mksrf_vocef_0.5x0.5_simyr2000.c110531.ncInland_lake_raw_data_file_name :mksrf_LakePnDepth_3x3min_simyr2004_csplk_c151015.ncInland_wetland_raw_data_file_name :mksrf_lanwat.050425.ncGlacier_raw_data_file_name :mksrf_glacier_3x3min_simyr2000.c120926.ncGlacier_region_raw_data_file_name :mksrf_GlacierRegion_10x10min_nomask_c170616.ncUrban_Topography_raw_data_file_name :mksrf_topo.10min.c080912.ncUrban_raw_data_file_name :mksrf_urban_0.05x0.05_simyr2000.c120621.ncagfirepkmon_raw_data_file_name :mksrf_abm_0.5x0.5_AVHRR_simyr2000.c130201.ncgdp_raw_data_file_name :mksrf_gdp_0.5x0.5_AVHRR_simyr2000.c130228.ncpeatland_raw_data_file_name :mksrf_peatf_0.5x0.5_AVHRR_simyr2000.c130228.ncsoildepth_raw_data_file_name :mksf_soilthk_5x5min_ORNL-Soil_simyr1900-2015_c170630.nctopography_stats_raw_data_file_name :mksrf_topostats_1km-merge-10min_HYDRO1K-merge-nomask_simyr2000.c130402.ncmap_pft_file_name :map_0.25x0.25_MODIS_to_0.9x1.25_nomask_aave_da_c170321.ncmap_lakwat_file :map_3x3min_MODIS-wCsp_to_0.9x1.25_nomask_aave_da_c160425.ncmap_wetlnd_file :map_0.5x0.5_lanwat_to_0.9x1.25_aave_da_110307.ncmap_glacier_file :map_3x3min_GLOBE-Gardner_to_0.9x1.25_nomask_aave_da_c120923.ncmap_glacier_region_file :map_10minx10min_topo_to_0.9x1.25_aave_da_110630.ncmap_soil_texture_file :map_5minx5min_soitex_to_0.9x1.25_aave_da_110722.ncmap_soil_color_file :map_0.25x0.25_MODIS_to_0.9x1.25_nomask_aave_da_c170321.ncmap_soil_organic_file :map_5x5min_ISRIC-WISE_to_0.9x1.25_nomask_aave_da_c120525.ncmap_urban_file :map_3x3min_LandScan2004_to_0.9x1.25_nomask_aave_da_c120522.ncmap_fmax_file :map_3x3min_USGS_to_0.9x1.25_nomask_aave_da_c120926.ncmap_VOC_EF_file :map_0.5x0.5_lanwat_to_0.9x1.25_aave_da_110307.ncmap_harvest_file :map_0.25x0.25_MODIS_to_0.9x1.25_nomask_aave_da_c170321.ncmap_urban_topography_file :map_10minx10min_topo_to_0.9x1.25_aave_da_110630.ncmap_agfirepkmon_file :map_0.5x0.5_lanwat_to_0.9x1.25_aave_da_110307.ncmap_gdp_file :map_0.5x0.5_lanwat_to_0.9x1.25_aave_da_110307.ncmap_peatland_file :map_0.5x0.5_lanwat_to_0.9x1.25_aave_da_110307.ncmap_soildepth_file :map_5x5min_ORNL-Soil_to_0.9x1.25_nomask_aave_da_c170706.ncmap_topography_stats_file :map_1km-merge-10min_HYDRO1K-merge-nomask_to_0.9x1.25_nomask_aave_da_c130405.ncSoil_texture_raw_data_file_name :mksrf_soitex.10level.c010119.ncSoil_color_raw_data_file_name :mksrf_soilcolor_CMIP6_simyr2005.c170623.ncFmax_raw_data_file_name :mksrf_fmax_3x3min_USGS_c120911.ncOrganic_matter_raw_data_file_name :mksrf_organic_10level_5x5min_ISRIC-WISE-NCSCD_nlev7_c120830.ncLai_raw_data_file_name :mksrf_lai_78pfts_simyr2005.c170413.ncVegetation_type_raw_data_filename :mksrf_landuse_histclm50_LUH2_2000.c170629.ncCreated_on :2023-08-28Created_by :mengqijCreated_with :./subset_data -- 311ed6db8Created_from :/glade/p/cesmdata/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/surfdata_0.9x1.25_hist_78pfts_CMIP6_simyr2000_c190214.nc

 

[slevis]

I may not be looking where you are looking; I see at the bottom of all these attributes:
"Created_from :/glade/p/cesmdata/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/surfdata_0.9x1.25_hist_78pfts_CMIP6_simyr2000_c190214.nc"
which tells me that you were subsetting a 0.9x1.25 file, which also means that you ended up with that same resolution. I recommend that you try "./subset_data --help" because, from what I remember, this surface dataset is the default input file used by ./subset_data. There is a way to choose your own input. If you want to end up with a 0.125 file, you need to use as input a surface dataset with that resolution. You can find all available surface datasets on cheyenne in /glade/p/cesmdata/cseg/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18

I'm not familiar with the get_regional tool that you asked about.

 

[mengqi]

Thank you! @slevis Will try it.

 

[mengqi]

Hi @slevis,

I would like to follow up with the question. I used "./subset_data --help" and found that there is no resolution option. Thus, the surface dataset via "./subset_data" should be the default input file since I cannot modify the resolution.

According to your advice, I did find a supported surface dataset with a 0.125 resolution on Cheyenne. I noticed that the surface dataset should be a global dataset; however, I was wondering if I could get a US midwest surface dataset by subsetting from that global dataset. Otherwise, it is a time-consuming process for my study.

Could you give me some input on how to deliver it? I am not sure if using "getregional_datasets.pl" tool might be a way.

Thanks!!

 

[slevis]

The version of subset_data that I use points to this configure file
/tools/site_and_regional/default_data.cfg
I was thinking you might change the contents of this configure file to accommodate your needs.

 

[mengqi]

The version of subset_data that I use points to this configure file
/tools/site_and_regional/default_data.cfg
I was thinking you might change the contents of this configure file to accommodate your needs.

Thank you, @slevis! It does work now!

 

[mengqi]

Hi @slevis

I am working on running regional at regular 1-degree resolution, and I referred to the material (Setting up (high-res sparse) regional-grid CTSM simulations · ESCOMP/CTSM · Discussion #1919).

However, I found that there are a few differences between the attached document and the context online. For example, for C) Generate mesh files, there is a difference, and I guess this is due to Derecho and Cheyenne? for D) Generate mask_mesh.nc, I did see this step in the document only.

I was wondering if you could clarify these differences. Thanks!

 

[slevis]

@mengqi thank you for pointing out these discrepancies. Could you clarify which "attached document" you compared to Discussion #1919, so that I may look at it and remove the discrepancies? Thanks again.

 

[slevis]

Sorry... If you mean the google doc referenced in Discussion #1919, then you do not need to use that. The google doc served as a starting point for gathering the instructions. If it's confusing, I should just remove it from Discussion #1919.

 

[mengqi]

Sorry... If you mean the google doc referenced in Discussion #1919, then you do not need to use that. The google doc served as a starting point for gathering the instructions. If it's confusing, I should just remove it from Discussion #1919.

Thank you, @slevis Yes, the Google doc seems to be different from Discussion #1919. Ok, I will refer to Discussion #1919.

One more question: according to Discussion #1919, I do not have to consider the "mask_mesh" command? Currently, there are just three steps: (a) module load nco; (b) Generate scrip.nc; and (c) Generate lnd_mesh.nc. However, I remember that there was a fourth step, which is Generate mask_mesh.nc. For example,

ncap2 -s 'elementMask(:)=0' lnd_mesh.nc mask_mesh.nc

Is my understanding right?

Thanks!

 

[slevis]

At some point I think we realized that the fourth step was unnecessary for the specific case, so we removed step 4. If we made a mistake, please let us know. Also, it's possible that your case differs and has different requirements than the one described.

 

[mengqi]

At some point I think we realized that the fourth step was unnecessary for the specific case, so we removed step 4. If we made a mistake, please let us know. Also, it's possible that your case differs and has different requirements than the one described.

Got it! Thank you, @slevis