Question about branching from CESM2-LE

[hsjung]

What version of the code are you using?
CESM2.1.5

Have you made any changes to files in the source tree?
No

Describe every step you took leading up to the problem:
I am trying to branch from a restart file from the CESM2-LE.
The restart file that I'm using to branch from is : /glade/campaign/cesm/collections/CESM2-LE/restarts/b.e21.BHISTsmbb.f09_g17.LE2-1011.001/rest/2010-01-01-00000
I have built the case using the compset BHISTsmbb.
Modifications are only made to change the frequency of some history files and the pe layout.
The runtype that I use is branch.
The case runs successfully for a year.

Describe your problem or question:
When I compare the CICE history files archived on glade versus the outputs from my simulation, the ice fields look different.
I have attached a figure showing the differences.
There is nearly no difference after 1 day of model integration but the differences amplify with time.
My understanding is that if I branch from a restart file and use the same compset, the results should be bit-by-bit reproduceable.
So I am wondering where this difference is coming from?

Some guesses would be that the CESM2-LE doesn't use the BHISTsmbb compset? Or has some modifications made to the namelists/source code that make it not possible to run out-of-the-box? If so I would be very much interested in where I could find details of such modifications.
Another guess would be that the CESM versions used to generate the LE and my runs are different? So maybe the forcing files are different?
Any advice on why this happens would be greatly appreciated!

Best,
Hee-Sung

 

[xnnzka]

Hi, Hee-Sung
I also used CESM2LENS restart files to restart my cases. I want to know if your machine is Aleph (CESM2LENS's machine)?
I think the biases might be because the environment. I think I need to explain more:
I have ever used one member from CESM2LENS restart files and run it for 15 years without any modifications. But I found the RESTOM and T2m are very different. I asked my supervisor and he kindly sent my result plot to Sun-Seon Lee (the IBS Center for Climate Physics, the computing resources for CESM2LENS). She said that she thought this result is reasonable beacuse the system that we are using is different from Aleph, and PE layout is also different from theirs. She also pointed out a very intersting perspective: "Even we do the same experiment on Aleph with different PE layout, the results may be different from original one."
I also do 2 tests because in our department we have 2 clusters. And I find the same restart file, the same experiment, the same set but very different results.
Above is my experience and thoughts. I hope this could help you!

 

[xnnzka]

And ps: I see your id is very familiar. We might have interacted under other posts as well.

 

[hsjung]

Thank you for sharing your experience!
I am working on a different machine with a different PE layout.
There also does seem to be some differences in the namelist variables in the CESM2 LENS runs versus what gets set automatically when I build the case with compset BHIST or BHISTsmbb. So perhaps these two were the cause of the differences...

Yes I think you may have helped me with some configurations on getting the historical simulations to cycle over a single year, thanks for helping out multiple times!

 

[katec]

I agree, a different machine, a different PE layout and, likely, a different compiler and compiled libraries will all result in slightly different results. We only expect B4B restarts when run in the same environments (compiler, libraries, machine, etc) each time.

 

[xnnzka]

Thank you for sharing your experience!
I am working on a different machine with a different PE layout.
There also does seem to be some differences in the namelist variables in the CESM2 LENS runs versus what gets set automatically when I build the case with compset BHIST or BHISTsmbb. So perhaps these two were the cause of the differences...

Yes I think you may have helped me with some configurations on getting the historical simulations to cycle over a single year, thanks for helping out multiple times!

yes, I think they modify the ouputs in source codes when running CESM2LENS. Don't be worry and anxious about the differences (because I have even been so and wasted much time to try to reproduce TAT). I am happy that my thoughts and experiences could help you because I have ever received many help from this forum.

 

[xnnzka]

yes, I think they modify the ouputs in source codes when running CESM2LENS. Don't be worry and anxious about the differences (because I have even been so and wasted much time to try to reproduce TAT). I am happy that my thoughts and experiences could help you because I have ever received many help from this forum.

aaah I want to say ''have ever been so'' not ''even''