Hi Team,I noticed recently that when running CAM6 with the QPC6 compset and the standard resolution (f09_f09_mg17) on Cheyenne I am not getting bit reproducibility if I change the number of cpus (pecount). I was under the impression that bit reproducibility should be expected even if the number of processors change, is this not the case? Has anyone else experienced this? I am using the released version (CESM2.1).Thanks,Kevin
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
Bit Reproducibility in CAM6
- Thread starter kareed
- Start date
Hi Brian,First, I referenced the wroung compset, I actually noticed this issue with FHIST.In particular, I notice the differences when I am using --pecount 360 or 576 in the case below:./create_newcase --case $CASEROOT --compset FHIST --res f09_f09_mg17 --mach cheyenne --project ######## --pecount 576Unforunately I don't have the cases currently. But I could replicate this issue if need be.Kevin
I'm able to reproduce your result. I thought this was only an issue for B compsets, but apparently not. I see roundoff level differences in the very first timestep in the fields coming from the coupler. There are flags to set to force the coupler to give task count independent results, but these flags are not set by default because they incur a performance penalty. I think if you issue the command "./xmlchange BFBFLAG=TRUE" from your run script that will address this issue. Let me know if that doesn't work.
Hi,nick! I ran FHIST with different numbers of nodes and got different results. Is this normal?Answered part of the above. Can confirm the BFBFLAG has to be set to TRUE on all runs, which is a shame since I've run a bunch already without this flag.
All I can say is that I didn't get different results, while setting BFBFLAG=TRUE on both runs.Hi,nick! I ran FHIST with different numbers of nodes and got different results. Is this normal?
Thank you for your reply! Indeed, when I enable this option, the results are independent of the nodes, but when it's not enabled, different nodes produce different results, which I find very strange.All I can say is that I didn't get different results, while setting BFBFLAG=TRUE on both runs.