Adding a new grid for cesm2

[thakur.abubakar]

I have been using CESM 2.1.1 in the aquaplanet setting for certain idealized studies. In this regard, I would like to perform certain experiments with a very coarse resolution in the longitudinal direction, say, for example, maybe a dlon of 15 ~ 20 degrees.
In this regard, I have been following the instructions provided in section 3.2 of 3. Model grids. I have been able to complete step 4 in these instructions. Related to step 5, I found this thread on the discussion forum. However, I'm having difficulty building the executable following the instructions given in $CESMROOT/tools/mapping/gen_mapping_files/runoff_to_ocn/INSTALL.
The sequence of commands while in this directory, as in INSTALL, is as follows:

1. cd src/
2. ../../../../configure --macros-format Makefile --machine sahasrat
3. (. ./.env_mach_specific.sh; gmake)

This generates an error message, attached err_out.txt
I have attached the .env_mach_specific.sh file generated in Step 2 above and the modules_file listing the modules I've using on the Cray XC40 (machine name sahasrat) system to run CESM. I've added the ESMF environment variables declarations in the modules as well. I have also attached the config_machines, config_batch and config_compilers.xml. However, I'm not sure if that's relevant here.

Kindly help me fix this issue. Any insight/ guidance in this regard will be greatly appreciated.

 

[sacks]

It looks like there may have been a problem generating the dependency rules in the build, so the Makefile process is building the source files in the runoff_to_ocn tool in the wrong order. But before we try to dig into that problem, I have a few suggestions that could make this unnecessary:

First, you may want to consider if the existing 10 deg x 15 deg finite volume grid, f10_f10_mg37, would work for you. We use that grid heavily for software testing of a variety of configurations, including some aquaplanet configurations. So if that grid works for you, then you can avoid needing to define your own grid and mapping files.

Second, even if you do want to create your own grid, I believe you do not need runoff mapping files for an aquaplanet configuration: I don't think there is a runoff model in these configurations.

Let us know if you need additional support with this.

 

[thakur.abubakar]

Thank you so much for your response, Bill. Actually, I would prefer to create my own grid. In order to check if the machinery is working correctly, I tried out the steps in the instructions (except steps 5 & 6) with the SCRIP grid file for the 10degx15deg grid.
On step 7, I have edited the config_grids.xml (attached) file from the directory $CIMEROOT/config/cesm.
However, on step 8 when I'm creating a new case with the command
./create_newcase --case /mnt/lustre/cas2/casabu/Abu/cesm21/cases/test_10x15 --compset ADAQP3 --res test_10x15 --run-unsupported --machine sahasrat

I'm getting the following error.
Compset longname is 2000_DATM%NYF_SLND_SICE_DOCN%AQP3_SROF_SGLC_SWAV
Compset specification file is /mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/src/drivers/mct/cime_config/config_compsets.xml
Traceback (most recent call last):
File "./create_newcase", line 218, in <module>
_main_func(__doc__)
File "./create_newcase", line 213, in _main_func
input_dir=input_dir, driver=driver, workflowid=workflow)
File "/mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/scripts/Tools/../../scripts/lib/CIME/case/case.py", line 1448, in create
input_dir=input_dir, driver=driver, workflowid=workflowid)
File "/mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/scripts/Tools/../../scripts/lib/CIME/case/case.py", line 783, in configure
grids = Grids(gridfile)
File "/mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/scripts/Tools/../../scripts/lib/CIME/XML/grids.py", line 22, in __init__
GenericXML.__init__(self, infile, schema)
File "/mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/scripts/Tools/../../scripts/lib/CIME/XML/generic_xml.py", line 57, in __init__
self.read(infile, schema)
File "/mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/scripts/Tools/../../scripts/lib/CIME/XML/generic_xml.py", line 87, in read
self.read_fd(fd)
File "/mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/scripts/Tools/../../scripts/lib/CIME/XML/generic_xml.py", line 112, in read_fd
self.tree = ET.parse(fd)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1182, in parse
tree.parse(source, parser)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 656, in parse
parser.feed(data)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1659, in feed
self._raiseerror(v)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1523, in _raiseerror
raise err
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1167, column 27

./query_config --grids yields a similar error.

Traceback (most recent call last):
File "./query_config", line 350, in <module>
_main_func(__doc__)
File "./query_config", line 334, in _main_func
query_grids(long_output=args.long)
File "./query_config", line 31, in query_grids
grids = Grids(config_file)
File "/mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/scripts/Tools/../../scripts/lib/CIME/XML/grids.py", line 22, in __init__
GenericXML.__init__(self, infile, schema)
File "/mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/scripts/Tools/../../scripts/lib/CIME/XML/generic_xml.py", line 57, in __init__
self.read(infile, schema)
File "/mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/scripts/Tools/../../scripts/lib/CIME/XML/generic_xml.py", line 87, in read
self.read_fd(fd)
File "/mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/scripts/Tools/../../scripts/lib/CIME/XML/generic_xml.py", line 112, in read_fd
self.tree = ET.parse(fd)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1182, in parse
tree.parse(source, parser)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 656, in parse
parser.feed(data)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1659, in feed
self._raiseerror(v)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1523, in _raiseerror
raise err
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 689, column 27

I think there is an issue related to the updation of the config_grids.xml. Please have a look and let me know what you think.

 

[thakur.abubakar]

Please find attached the updated config_grids.xml file. I had made an error in commenting the code.

./query_config --grids yields

ERROR: Command: '/usr/bin/xmllint --noout --schema /mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/config/xml_schemas/config_grids_v2.xsd /mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/config/cesm/config_grids.xml' failed with error '/mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/config/cesm/config_grids.xml:1687: element gridmap: Schemas validity error : Element 'gridmap', attribute 'ocn_grid': '10x15' is not a valid value of the atomic type 'xs:NCName'.
/mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/config/cesm/config_grids.xml fails to validate' from dir '/mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/scripts'

 

[sacks]

It looks like the problem is in the last block you added, starting with <gridmap atm_grid="10x15" ocn_grid="10x15">. From playing around with this, my best guess is that the ocn_grid attribute in a gridmap is expected to start with a letter, not a number. That may be too restrictive, and you can work around this issue by introducing the following diff in the file cime/config/xml_schemas/config_grids_v2.xsd:

diff --git a/config/xml_schemas/config_grids_v2.xsd b/config/xml_schemas/config_grids_v2.xsd
index b4680b93f..f427ee7c0 100644
--- a/config/xml_schemas/config_grids_v2.xsd
+++ b/config/xml_schemas/config_grids_v2.xsd
@@ -15,7 +15,7 @@
   <xs:attribute name="atm_grid" type="xs:NMTOKEN"/>
   <xs:attribute name="glc_grid" type="xs:NCName"/>
   <xs:attribute name="lnd_grid" type="xs:NMTOKEN"/>
-  <xs:attribute name="ocn_grid" type="xs:NCName"/>
+  <xs:attribute name="ocn_grid" type="xs:NMTOKEN"/>
   <xs:attribute name="rof_grid" type="xs:NCName"/>
   <xs:attribute name="wav_grid" type="xs:NCName"/>

However, in this case, where you have the same atmosphere & ocean grids, you shouldn't need to define any mapping files between atm and ocean, so you should be able to leave out this gridmap block entirely.

 

[thakur.abubakar]

Thanks, Bill. That has worked very well. The ./create_case command executes successfully. This is the output generated.

Compset longname is 2000_DATM%NYF_SLND_SICE_DOCN%AQP3_SROF_SGLC_SWAV
Compset specification file is /mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/src/drivers/mct/cime_config/config_compsets.xml
Compset forcing is 1972-2004
ATM component is Data driven ATM COREv2 normal year forcing
LND component is Stub land component
ICE component is Stub ice component
OCN component is DOCN analytic aquaplanet sst - option 3
ROF component is Stub river component
GLC component is Stub glacier (land ice) component
WAV component is Stub wave component
ESP component is
Pes specification file is /mnt/lustre/cas2/casabu/Abu/cesm21/model/cime/src/drivers/mct/cime_config/config_pes.xml
Could not find machine match for 'nid00006' or 'login1'
Machine is sahasrat
Pes setting: grid is a%10x15_l%null_oi%10x15_r%null_g%null_w%null_m%null
Pes setting: compset is 2000_DATM%NYF_SLND_SICE_DOCN%AQP3_SROF_SGLC_SWAV
Pes setting: tasks is {'NTASKS_ATM': -1, 'NTASKS_ICE': -1, 'NTASKS_CPL': -1, 'NTASKS_LND': -1, 'NTASKS_WAV': -1, 'NTASKS_ROF': -1, 'NTASKS_OCN': -1, 'NTASKS_GLC': -1, 'NTASKS_ESP': -1}
Pes setting: threads is {'NTHRDS_ICE': 1, 'NTHRDS_ATM': 1, 'NTHRDS_ROF': 1, 'NTHRDS_LND': 1, 'NTHRDS_WAV': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1, 'NTHRDS_GLC': 1, 'NTHRDS_ESP': 1}
Pes setting: rootpe is {'ROOTPE_ESP': 0, 'ROOTPE_OCN': 0, 'ROOTPE_LND': 0, 'ROOTPE_ATM': 0, 'ROOTPE_ICE': 0, 'ROOTPE_WAV': 0, 'ROOTPE_CPL': 0, 'ROOTPE_ROF': 0, 'ROOTPE_GLC': 0}
Pes setting: pstrid is {}
Pes other settings: {}
Pes comments: none
Compset is: 2000_DATM%NYF_SLND_SICE_DOCN%AQP3_SROF_SGLC_SWAV
Grid is: a%10x15_l%null_oi%10x15_r%null_g%null_w%null_m%null
Components in compset are: ['datm', 'slnd', 'sice', 'docn', 'srof', 'sglc', 'swav', 'sesp', 'drv', 'dart']
No project info available
No charge_account info available, using value from PROJECT
No project info available
cesm model version found: cesm2.1.3-rc.01
Batch_system_type is pbs
job is case.run USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S
job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S
Creating Case directory /mnt/lustre/cas2/casabu/Abu/cesm21/cases/test_10x15

I think the PES settings can be changed in the case directory via xmlchange. Hope this is nothing to worry about.

 

[sacks]

Great, I'm glad to hear that this is moving forward well. Yes, the PE settings can be changed in your case directory. The message you're getting in the create_newcase command is just informational, indicating that by default this will run on one node (-1 means a full node, however many processors that means for the given machine).

 

[thakur.abubakar]

Thanks again, Bill. Although, I think we may have hit a roadblock now.
After being able to create the above case, I was able to successfully build it as well. When I tried running the case, the job exited after a few seconds. Please find attached the log files for the case.

Also, I tried out creating another case using an active atmosphere and data ocean via the command
./create_newcase --case /mnt/lustre/cas2/casabu/Abu/cesm21/cases/test1_10x15 --compset 2000_CAM60_SLND_SICE_DOCN%AQP3_SROF_SGLC_SWAV --res test_10x15 --run-unsupported --machine sahasrat

The case was created successfully (attached create_case output) but the build fails (attached build_case output).

 

[thakur.abubakar]

Hi Bill,
I've tried npr_yz=6,8,8,6 in the user_nl_cam namelist of test1_10x15.
I could build it successfully, and it ran for around 2 mins before exiting.

 

[thakur.abubakar]

Moving towards the resolution I intend to use, I have created a new grid 'my_2x20' (attached config_grids). I am able to create a new case using

./create_newcase --case /mnt/lustre/cas2/casabu/Abu/cesm21/cases/test2_2x20 --compset 2000_CAM60_SLND_SICE_DOCN%AQP3_SROF_SGLC_SWAV --res my_2x20 --run-unsupported --machine sahasrat

However, the build fails with the following error:
Building case in directory /mnt/lustre/cas2/casabu/Abu/cesm21/cases/test2_2x20
sharedlib_only is False
model_only is False
Generating component namelists as part of build
Creating component namelists
Calling /mnt/lustre/cas2/casabu/Abu/cesm21/model/components/cam//cime_config/buildnml
...calling cam buildcpp to set build time options
ERROR: Command: '/mnt/lustre/cas2/casabu/Abu/cesm21/model/components/cam/bld/configure -s -fc_type intel -ccsm_seq -dyn fv -hgrid 2x20 -usr_src /mnt/lustre/cas2/casabu/Abu/cesm21/cases/test2_2x20/SourceMods/src.cam -spmd -nosmp -ocn docn -phys cam6 -aquaplanet' failed with error 'set_horiz_grid: no match for dycore fv and hgrid 2x20' from dir '/mnt/lustre/cas2/casabu/Abu/cesm21/cases/test2_2x20/Buildconf/camconf'

This seems like the model is unable to identify the grid I'm specifying, even though I have defined it in config_grids.xml

 

[brianpm]

Hi Bill,
I've tried npr_yz=6,8,8,6 in the user_nl_cam namelist of test1_10x15.
I could build it successfully, and it ran for around 2 mins before exiting.

It looks like the run did not succeed, is that correct? The log file indicates that there could be a problem with "incompatible domain grid coordinates". It might be good to get that case running correctly before introducing the new grid.

 

[thakur.abubakar]

Hi Brian,
That is a case that I have used extensively previously and it has worked fine for f19_f19_mg17 (attached log files and README.case file from one such case).

 

[hannay]

I am not an expert on this but looking at your file: config_grids.xml, I see you haven't defined the lnd:

Your file contains:
<model_grid alias="my_2x20">
<grid name="atm">2x20</grid>
<grid name="ocnice">2x20</grid>
<mask>null</mask>
</model_grid>


From looking at other resolutions, should you have:

Your file contains:
<model_grid alias="my_2x20">
<grid name="atm">2x20</grid>
<grid name="lnd">2x20</grid>
<grid name="ocnice">2x20</grid>
<mask>null</mask>
</model_grid>

 

[thakur.abubakar]

Hi Cecile,
Thanks for your response. I made the changes you suggested and I'm still getting the same error on ./case.build

 

[jet]

You also need to add a line defining your grid and the associated dycore to cam/bld/config_files/horiz_grid.xml.

Something like
<horiz_grid dyn="fv" hgrid="2x20" nlat="96" nlon="18" />

Just FYI besides the grid definition you may need to check the namelist defaults for your new grid to make sure the model parameterizations are using appropriate defaults. I would compare the atm_in files from an f19 and f10 to see which parameters changed between the 2 degree and 10x15 degree models to get an idea of which parameters you might want to play with or change if the run dies due to a parameterization instability. I'm guessing that the defaults you will have for the new grid will be the values for the 1 or 2 degree model which should work but may not be optimal for your 2x20 grid.

John

 

[thakur.abubakar]

Hello John,

Thank you for your response. After incorporating the grid definition, I'm getting the following error on build.

casabu@login2:/mnt/lustre/cas2/casabu/Abu/cesm21/cases/test2x20> ./case.build
Building case in directory /mnt/lustre/cas2/casabu/Abu/cesm21/cases/test2x20
sharedlib_only is False
model_only is False
Generating component namelists as part of build
Creating component namelists
Calling /mnt/lustre/cas2/casabu/Abu/cesm21/model/components/cam//cime_config/buildnml
...calling cam buildcpp to set build time options
ERROR: Command /mnt/lustre/cas2/casabu/Abu/cesm21/model/components/cam/bld/build-namelist -ntasks 240 -csmdata /mnt/lustre/cas2/casabu/Abu/cesm21/cesm_input -infile /mnt/lustre/cas2/casabu/Abu/cesm21/cases/test2x20/Buildconf/camconf/namelist -ignore_ic_year -use_case aquaplanet_cam6 -inputdata /mnt/lustre/cas2/casabu/Abu/cesm21/cases/test2x20/Buildconf/cam.input_data_list -namelist " &atmexp /" failed rc=255
out=CAM build-namelist - ERROR: No default value found for ncdata
user defined attributes:
key=ic_md val=00010101
err=Died at /mnt/lustre/cas2/casabu/Abu/cesm21/model/components/cam/bld/build-namelist line 4054

Following your advice, here is the diff between the atm_in files for the two resolutions.

7c7
< seasalt_emis_scale = 1.00D0
---
> seasalt_emis_scale = 1.10D0
19c19
< ncdata = '/mnt/lustre/cas2/casabu/Abu/cesm21/cesm_input/atm/cam/inic/fv/aqua_0000-01-01_10x15_L32_c170103.nc'
---
> ncdata = '/mnt/lustre/cas2/casabu/Abu/cesm21/cesm_input/atm/cam/inic/fv/aqua_0006-01-01_1.9x2.5_L32_c161020.nc'
96c96
< clubb_gamma_coef = 0.308
---
> clubb_gamma_coef = 0.280
148c148
< micro_mg_dcs = 500.D-6
---
> micro_mg_dcs = 200.D-6
255c255
< npr_yz = 6,8,8,6
---
> npr_yz = 30,8,8,30

I think I can specify default values specific to the new grid in cam/bld/namelist_files/namelist_defaults_cam.xml. However, is there any functionality to generate the ncdata (highlighted above) file?

 

[thakur.abubakar]

I used the existing input data file for 1.9x2.5 to generate one for the new resolution. I am able to build the case successfully. However, the runs are exiting abruptly after a few seconds. Please find attached the log files and user_nl_cam file for the case.

 

[thakur.abubakar]

The last few lines of cesm.log are as follows:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
cesm.exe 0000000001CDA963 Unknown Unknown Unknown
libpthread-2.26.s 00002AAAB4B502D0 Unknown Unknown Unknown
cesm.exe 0000000000654BEE parutilitiesmodul 1757 parutilitiesmodule.F90
cesm.exe 00000000005CF574 dynamics_vars_mp_ 564 dynamics_vars.F90
cesm.exe 0000000000EF2F7C spmd_dyn_mp_spmdi 971 spmd_dyn.F90
cesm.exe 00000000005C3877 dyn_grid_mp_dyn_g 128 dyn_grid.F90
cesm.exe 00000000004FE76B cam_comp_mp_cam_i 169 cam_comp.F90
cesm.exe 00000000004F50DC atm_comp_mct_mp_a 209 atm_comp_mct.F90
cesm.exe 00000000004373F4 component_mod_mp_ 267 component_mod.F90
cesm.exe 0000000000425B1B cime_comp_mod_mp_ 1231 cime_comp_mod.F90
cesm.exe 000000000043442E MAIN__ 114 cime_driver.F90
cesm.exe 00000000004167D2 Unknown Unknown Unknown
libc-2.26.so 00002AAAB50B834A __libc_start_main Unknown Unknown
cesm.exe 00000000004166EA Unknown Unknown Unknown
[NID 01249] 2021-09-25 17:34:51 Apid 1088128: initiated application termination
Application 1088128 exit codes: 174
Application 1088128 exit signals: Killed

Doing a bit of searching on the forum led me to two solutions to this.

1. Setting ulimit -s unlimited
2. Changing the optimization from -O2 to -O1 or -O3.

However, neither seem to be working in my case. Any help would be greatly appreciated.

 

[thakur.abubakar]

Following John's advice, I have specified certain namelist variables in user_nl_cam. This has led to the job stopping due to a different error (exit codes: 134).
Please find attached the log files and user_nl_cam files for this run. I have also attached the initial condition file that I have used. The simulation is attempted with DEBUG=FALSE. It crashes even with DEBUG=TRUE