Running iCESM on Cheyenne

[michael_p_erb@gmail_com]

I would like to run some iCESM simulations and I have a small allocation of time on Cheyenne to do some initial tests. I ran some CESM1.2 simulations years ago on Yellowstone, but I'm a little unclear about how to get started with iCESM on Cheyenne. I see that the CESM1.2 code is located at "/glade/p/cesm/releases/cesm1_2_2/". I also see that some documentation is provided at NCAR/iCESM1.2. Is there other material that describes how to get started? For example, my very first issue is that, when I try to use the "create_newcase" command, "cheyenne" isn't a valid choice for the "mach" option, so I feel like I'm missing something. Any tutorials or advice that you all could offer would be most appreciated. Thanks!

 

[nusbaume]

Hi Michael,

I would recommend cloning the iCESM1.2 source code from Github to Cheyenne, as that source code has a "cheyenne" machine option along with the isotope/tracer code itself. This can be done by typing the following command on Cheyenne:

git clone https://github.com/NCAR/iCESM1.2.git

This will download a directory labeled "iCESM1.2", which should behave like a normal CESM source code directory. From there just follow the instructions you found and it should work on Cheyenne without any issues. Of course if you do run into any problems please let me know.

Thanks, and have a great day!

Jesse

 

[michael_p_erb@gmail_com]

Thanks, Jesse! A related question: it sounds like Cheyenne will only be active until mid-2022. Do you know if iCESM1.2 will be set up to run on Cheyenne's successor (e.g. will there be a "machine option" for the new system)?

 

[nusbaume]

Hi Michael,

Yep, iCESM1.2 should be supported on the new machine. I sadly can't give an exact timeline on when it will be officially ported over, but it shouldn't be too long after CESM2 itself makes the transition.

Hope that helps!

Jesse

 

[michael_p_erb@gmail_com]

I'm trying to run iCESM1.2 on Cheyenne. I've cloned the "NCAR/iCESM1.2" repository and gone through the instructions at NCAR/iCESM1.2, but I'm running into problems. I suspect that my problems stem from step 5 at the link above.

First, I tried step 5v1. I downloaded the restart files from https://doi.org/10.5281/zenodo.3553753, put them in the right directory, and went through the rest of the steps. However, after I run the *.run script (step 10), the job seems to fail. The cesm.log.* file gives me errors like this.

1: NetCDF: Variable not found
1: ERROR: reading restart file for tracer-CLM initialization, but some/all tracers
1: missing on file. Ending model now.
1: ENDRUN: called without a message string

It sounds like something is missing from the restart files. Is that correct?

At any rate, I went back and tried step 5v2 instead. I used the “iclm4_IC_create.ncl” script to create a new input file and specified it in user_nl_clm with "finidat=". When I ran the build script, I got the following error:

CLM build-namelist ERROR: Can NOT set both -clm_startfile option AND finidat on namelist
ERROR: clm.buildnml.csh failed
ERROR from preview namelist – EXITING

At that point, I used “./xmlchange RUN_TYPE=startup” to make this a startup run, then used the build script and run script. When using the run script, the output ends with:

Input Data List Files Found:
/glade/u/home/merb/cesm/iso_EXAMPLE_CASE/Buildconf/rtm.input_data_list
/glade/u/home/merb/cesm/iso_EXAMPLE_CASE/Buildconf/clm.input_data_list
/glade/u/home/merb/cesm/iso_EXAMPLE_CASE/Buildconf/pop2.input_data_list
/glade/u/home/merb/cesm/iso_EXAMPLE_CASE/Buildconf/cpl.input_data_list
/glade/u/home/merb/cesm/iso_EXAMPLE_CASE/Buildconf/cice.input_data_list
/glade/u/home/merb/cesm/iso_EXAMPLE_CASE/Buildconf/cam.input_data_list
File status unknown: same_as_TS

CESM PRESTAGE SCRIPT HAS FINISHED SUCCESSFULLY
-------------------------------------------------------------------------
Mon May 3 11:48:42 MDT 2021 -- CSM EXECUTION BEGINS HERE
Mon May 3 11:48:42 MDT 2021 -- CSM EXECUTION HAS FINISHED
grep: cpl.log.210503-114809: No such file or directory
Model did not complete - see /glade/scratch/merb/iso_EXAMPLE_CASE/run/cesm.log.210503-114809

If I check the cesm.log.* file, all it says is:

MPT ERROR: bad process count
(HPE MPT 2.19 02/23/19 05:31:12)

So I'm not really sure how to proceed. Any ideas? I wanted to do a hybrid or branch simulation starting with restart files, but doing a startup simulation is fine too.

 

[nusbaume]

Hi Michael,

It looks like you are using CAM4 in your model setup, which is not isotope-enabled (and thus will cause all sorts of weird errors in iCESM). You should be able to fix this by modifying the variable CAM_CONFIG_OPTS in env_build.xml to say -phys cam5 instead of -phys cam4. You will need to then re-build the model before re-submitting it.

I am not sure if that will solve all of your problems, but it will definitely be a first step, so let me know how it goes.

Thanks, and have a great day!

Jesse

 

[michael_p_erb@gmail_com]

Thanks, Jesse. Switching to CAM5 fixed some of my problems, but another one has popped up. I am trying to run a hybrid simulation using the restart files linked in step 5 of NCAR/iCESM1.2. However, when submitting the run, I get the following error:

29: check_dim ERROR: mismatch of input dimension 8012 with expected value
29: 8010 for variable landunit

I am using f19_g16 resolution, which is the same as the restart files, so I am surprised to see an error about dimension size. "landunit" is in the *clm2.r.* file. If I compare the lats and lons associated with "landunit" in this file vs a different restart file, the lats and lons are identical, except that two locations show up an extra time:

42.6315789473683 N, 60.0 E
76.736842105263 N, 102.49999999999999E

Any idea why the restart files from iCESM1.2 restart file from "The Connected Isotopic Water Cycle in the Community Earth System Model Version 1" appear to have two more locations than the code expects?

 

[nusbaume]

Hi Michael,

I am not sure why there are two extra grid points in that file, but you may be able to resolve the issue by using the "interpinic" program, which is located in the iCESM1.2 source code here:

iCESM1.2/models/lnd/clm/tools/clm4_0/interpinic

There is a README file there that explains how to build and run the program, which you can use to interpolate the problematic restart file data to another CLM restart file that you know has the appropriate dimensions. Then you should be able to use that new restart file instead, which ideally will no longer have the dimension issue.

Hope that helps, and have a great day!

Jesse

 

[Redlichia]

Hi Michael,

I am not sure why there are two extra grid points in that file, but you may be able to resolve the issue by using the "interpinic" program, which is located in the iCESM1.2 source code here:

iCESM1.2/models/lnd/clm/tools/clm4_0/interpinic

There is a README file there that explains how to build and run the program, which you can use to interpolate the problematic restart file data to another CLM restart file that you know has the appropriate dimensions. Then you should be able to use that new restart file instead, which ideally will no longer have the dimension issue.

Hope that helps, and have a great day!

Jesse

Hi, Dear Jesse，

I am a rookie about iCESM, but I am interested in using iCESM in my research. I am running fully coupled paleoclimate simulation (use B1850) with CESM1_2_2. I only found oxygen isotope modules in the above question. Does this version CESM1.2.2 support the inclusion of carbon isotopes, and how do you run experiments with carbon isotope modules. Is there a relevant manual for iCESM?
Thanks a lot. I am sorry to bother you.

Kind regards
Chen.