Finding the best phys_loadbalance option in CAM

jedwards · Oct 16, 2014

Optimal configuration of CESM may depend on a number of factors including the performance characteristics of the target hardware. Here is an easy way to find the optimal phys_loadbalance option for cam.
First build a PFS test with a compset and grid which match your intended experiment. I am interested in optimizing CAM5 performance on yellowstone on an ne30 grid. Even if your intended target experiment is a fully coupled (B) case you will want to use an F case for this experiment.
From the CESM source scripts directory:
./create_test -testname PFS.ne30_ne30.F1850C5.yellowstone_intel
Assure that the pe layout for the ATM matches the layout that you intend to use in your target experiment. If your target is a B case match the LND and ICE layouts as well but set the OCN layout to be the same as ATM. Once you have the pe layout set and have run cesm_setup to generate the new job scripts, edit the $CASE.test script and add loop around the call to the $CASE.run command.

Code:

foreach phys_lbo ( -1 0 1 2 3 4 5)
 cp user_nl_cam.tmp user_nl_cam
 echo "phys_loadbalance=$phys_lbo" >> user_nl_cam
 # cat user_nl_cam
 ./$CASE.run
end

This will, in one job submission, do a 20 day run with each of the possible phys_loadbalance options. Copy the user_nl_cam file in your case directory to user_nl_cam.tmp and put any other options you want to use into this file, make sure there is a newline at the end of the file. You are now ready to build and submit the test.
Once the test has completed, check the overall performance:
zgrep years logs/cpl*
logs/cpl.log.141015-150648.gz:(seq_mct_drv): =============== # simulated years / cmp-day = 13.180 ===============
logs/cpl.log.141015-151443.gz:(seq_mct_drv): =============== # simulated years / cmp-day = 13.894 ===============
logs/cpl.log.141015-152212.gz:(seq_mct_drv): =============== # simulated years / cmp-day = 15.080 ===============
logs/cpl.log.141015-152917.gz:(seq_mct_drv): =============== # simulated years / cmp-day = 16.216 ===============
logs/cpl.log.141015-153558.gz:(seq_mct_drv): =============== # simulated years / cmp-day = 14.853 ===============
logs/cpl.log.141015-154304.gz:(seq_mct_drv): =============== # simulated years / cmp-day = 14.266 ===============
logs/cpl.log.141015-155025.gz:(seq_mct_drv): =============== # simulated years / cmp-day = 14.017 ===============

So we see that phys_loadbalance=2 gives the best performance for this case. It's always a good idea when changing model options to conduct a
restart test (ERS) with the new options before using them in your experiment.
Read more about phys_loadbalance options -1-3 in
http://www.csm.ornl.gov/~bbd/pubs/IJHPCAWorley2005.pdf

mfwehner@lbl_gov · Aug 22, 2016