pelayout introduces huge errors

[Mikasa]

Hello, I made an intresting test that the pelayout of POP will significantly affect the result. I run two similar run of compset B1PCTcmip6 for 1 year. The only difference between them is the pelayout.
case1 is:

Comp NTASKS NTHRDS ROOTPE
CPL : 768/ 1; 0
ATM : 768/ 1; 0
LND : 320/ 1; 0
ICE : 448/ 1; 320
OCN : 640/ 1; 768
ROF : 320/ 1; 0
GLC : 32/ 1; 0
WAV : 32/ 1; 1408
ESP : 1/ 1; 0

case 2 is:

Comp NTASKS NTHRDS ROOTPE
CPL : 768/ 1; 0
ATM : 768/ 1; 0
LND : 320/ 1; 0
ICE : 448/ 1; 320
OCN : 1280/ 1; 768
ROF : 320/ 1; 0
GLC : 32/ 1; 0
WAV : 32/ 1; 2048
ESP : 1/ 1; 0

The CAM output "TS" difference between them is (case 2 minus case 1) shown in the picture:
Why the TS difference is so high?

 

[jedwards]

By changing the pelayout in pop you introduce small (round off level) differences in each timestep as the mpi reduction operations
are not bit for bit exact. These small differences lead to a divergence of results. To assure that this change doesn't affect the models climate the procedure is to create an ensemble using perturbations of initial conditions at one pelayout then do a set of runs at the other pelayout, the
change in pelayout should be indistinguishable from the initial perturbation ensemble.