Differences between identical simulations on Derecho and Cheyenne

[bbucho]

I’m trying to recreate a control simulation on Derecho that originally was run on Cheyenne, using CESM2.2.0 with the DOCN-SOM. I was able to successfully build and run CESM2.2.0 without any errors, but the monthly output is quite different between the simulation on Derecho versus the simulation on Cheyenne. For example, the discrepancy in TREFHT exceeds 5 K in some regions (see below for a plot of RMSE over the first 20 years).



I’ve checked my setup multiple times to make sure the CESM components, compset, restart files and reference date for the branch, namelists, parameter files, and slab ocean forcing are identical between the two simulations. I've included my run script and output from ./describe_version in the following files. Is it possible that the this difference could be related to changing from Cheyenne to Derecho? Any help or insight would be appreciated!

 

[erik]

@bbucho thanks for the heads up about this. This is an interesting result that we are all interested in.

It could very well just be differences in the machine. We do expect different machines to be different and to essentially be a different "ensemble" member of a set of simulations. However, when porting to new machines we want to check that although the details are different that they are giving the same climate. As such we have a test that helps to see if that's the case. See this cime page about using that test...

6. Porting and validating CIME on a new platform — CIME master documentation

Since, this is a question about porting, I'm moving this to the porting forum.

 

[aswann2]

Thanks @erik ! We will look into running those tests and report back. We are still curious if anyone else who has ported CESM2 to Derecho could report back about how similar their results were?

 

[wwieder]

I know that getting CESM2.1 and CESM2.2 tags that run on Derecho are a priority for CSEG in the next few weeks. The warmer temperatures you're showing here are surprising!

 

[wwieder]

have you checked this isn't internal variability?

 

[aswann2]

Thanks @wwieder. We are now thinking that this is within the range of expected variability.

This figure from Zarakas et al. 2020 in JClim (in the supplement) shows the distribution of 20yr average temperature anomaly in the PI control for several models. CESM2 used in C4MIP is panel f.