CESM2 pelayout optimized for higher POP dt_count

[jacob_stu_allen]

Hello all!

I'm running an abrupt 4xCO2 case and have encountered what seems to be a fairly common numerical instability flagged by MARBL about 50 years into my run. Doubling POP's dt_count from 24 to 48 has resolved this problem, but my SYPD has dipped down to 8 years compared to the ~15 I was getting before. Just curious if anyone has an optimized pelayout for this so I can get some better performance.

Thanks in advance and I'll post back soon if my own testing improves the SYPD.
Jacob

 

[michelle_dvorak]

Hi Jacob,

I just encountered the same problem when branching from an existing 2xCO2 simulation (B1850 f09_g17).

After doubling the dt_count as you did, I made the following changes to match a PE layout by Cecile Hannay that I found on the original run:

./xmlchange NTASKS_ATM = 1728
./xmlchange NTASKS_CPL = 1728
./xmlchange NTASKS_OCN = 768
./xmlchange NTASKS_WAV = 36
./xmlchange NTASKS_GLC = 1728
./xmlchange NTASKS_ICE = 252
./xmlchange NTASKS_ROF = 1440
./xmlchange NTASKS_LND = 1440
./xmlchange NTASKS_ESP = 1

./xmlchange ROOTPE_ATM = 0
./xmlchange ROOTPE_CPL = 0
./xmlchange ROOTPE_OCN = 1728
./xmlchange ROOTPE_WAV = 1692
./xmlchange ROOTPE_GLC = 0
./xmlchange ROOTPE_ICE = 1440
./xmlchange ROOTPE_ROF = 0
./xmlchange ROOTPE_LND = 0
./xmlchange ROOTPE_ESP = 0

I kept "MAX_TASKS_PER_NODE" and "MAX_MPITASKS_PER_NODE" at 128.

Now I'm getting a throughput of 23.3 SYPD at a cost of 2635 pe-hours/sim year, which is actually better than it was before the timestep change. Curious to know if this is expensive compared to what others are getting on Derecho.

Michelle

 

[jacob_stu_allen]

Hello Michelle,

Thank you for this! For a throughput of ~15.5 years per day I was using 1975 pe-hours/sim year on Derecho. So it's a bit more expensive - but will be really useful for some quicker runs.

Cheers,
Jacob