Question about FHIST PE layouts for CESM2.1.5?

[mengqi]

Hi,

I am working on a global simulation using CESM2.1.5. This is my case info below:
(npl) mengqij@derecho6:~/cesm_215/cime/scripts> ./create_newcase --case ~/D_simulation_globe/FHIST_BGC_b --res f19_g17 --compset FHIST_BGC --project UIUC0025 --run-unsupported

I got about 10 years per day, which seems reasonable. I want to go faster by adjusting fully coupled PE layouts, although I have no experience. Here is relevant info in the env_mach_pes.xml. Could anyone please offer some insights? I really appreciate any help anyone can provide.

<entry id="NTASKS">
<type>integer</type>
<values>
<value compclass="ATM">-2</value>
<value compclass="CPL">-2</value>
<value compclass="OCN">-2</value>
<value compclass="WAV">-2</value>
<value compclass="GLC">-2</value>
<value compclass="ICE">-2</value>
<value compclass="ROF">-2</value>
<value compclass="LND">-2</value>
<value compclass="ESP">1</value>
</values>
<desc>number of tasks for each component</desc>
</entry>
<entry id="NTASKS_PER_INST">
<type>integer</type>
<values>
<value compclass="ATM">256</value>
<value compclass="OCN">256</value>
<value compclass="WAV">256</value>
<value compclass="GLC">256</value>
<value compclass="ICE">256</value>
<value compclass="ROF">256</value>
<value compclass="LND">256</value>
<value compclass="ESP">1</value>
</values>
<desc>Number of tasks per instance for each component. DO NOT EDIT: Set automatically by case.setup based on NTASKS, NINST and MULTI_DRIVER</desc>
</entry>
<entry id="NTHRDS">
<type>integer</type>
<values>
<value compclass="ATM">1</value>
<value compclass="CPL">1</value>
<value compclass="OCN">1</value>
<value compclass="WAV">1</value>
<value compclass="GLC">1</value>
<value compclass="ICE">1</value>
<value compclass="ROF">1</value>
<value compclass="LND">1</value>
<value compclass="ESP">1</value>
</values>
<desc>number of threads for each task in each component</desc>
</entry>
<entry id="ROOTPE">
<type>integer</type>
<values>
<value compclass="ATM">0</value>
<value compclass="CPL">0</value>
<value compclass="OCN">0</value>
<value compclass="WAV">0</value>
<value compclass="GLC">0</value>
<value compclass="ICE">0</value>
<value compclass="ROF">0</value>
<value compclass="LND">0</value>
<value compclass="ESP">0</value>
</values>
<desc>ROOTPE (mpi task in MPI_COMM_WORLD) for each component</desc>
</entry>

 

[katec]

HI there,

So the FHIST compset isn't exactly fully coupled, which makes adjusting the PE layouts easier! For this compset you can just do (for example):

./xmlchange NTASKS=-3
./xmlchange NTASKS_ESP=1

The PE layout you have above is 2 nodes (with 128 tasks per node so 256 tasks total) for all components except ESP which is not being used so only needs 1 task. The commands above increase that to 3 nodes (384 total tasks) for everything except ESP. For a 2-degree (f19_g17) simulation you should be able to get faster runs up to about 1000 tasks, but if you go much more than that, you may start having issues. Remember that when you increase your PEs, you increase the cost of your simulation in your allocation and you might increase your wait time in the queue as well.

 

[slevis]

I edited the title of this thread to make it more closely relevant to the question and response.