Problem determining the number of nodes to be used

[Juan]

Dear all,
We are trying to run the B1850 case in our supercomputer. We are able of modifying in the env_mach_pes.xml the max_tasks_per_node and max_mpi_tasks_per_node; we also can change NTASKS with xmlchange. However, with independence of the modifications to such variables, after a case.setup a preview_run always gives as a result that the the number of nodes is 9.
We want to change the number of nodes to be used. Obviously we are missing something, maybe in the definition of our machine in the config_batch.xml.
How should we do to modify the number of nodes too?

CASE INFO:
nodes: 9
total tasks: 266
tasks per node: 32
thread count: 1

Thanks,
Juan

 

[jedwards]

what is the output of the ./pelayout script in your case directory?

 

[Juan]

After a simple create_case something like

Comp NTASKS NTHRDS ROOTPE
CPL : 512/ 1; 0
ATM : 512/ 1; 0
LND : 256/ 1; 0
ICE : 256/ 1; 256
OCN : 64/ 1; 512
ROF : 256/ 1; 0
GLC : 512/ 1; 0
WAV : 512/ 1; 0
ESP : 1/ 1; 0


If for example I do .xmlchange NTASKS=10

Comp NTASKS NTHRDS ROOTPE
CPL : 10/ 1; 0
ATM : 10/ 1; 0
LND : 10/ 1; 0
ICE : 10/ 1; 128
OCN : 10/ 1; 256
ROF : 10/ 1; 0
GLC : 10/ 1; 0
WAV : 10/ 1; 0
ESP : 10/ 1; 0

 

[Juan]

I going to try to express it better:
If for example I change max_tasks_per_node=32 and max_mpitasks_per_node=32
and then I do
.xmlchange NTASKS=10

Comp NTASKS NTHRDS ROOTPE
CPL : 10/ 1; 0
ATM : 10/ 1; 0
LND : 10/ 1; 0
ICE : 10/ 1; 128
OCN : 10/ 1; 256
ROF : 10/ 1; 0
GLC : 10/ 1; 0
WAV : 10/ 1; 0
ESP : 10/ 1; 0

After a simple create_case something like, by default:
Comp NTASKS NTHRDS ROOTPE
CPL : 512/ 1; 0
ATM : 512/ 1; 0
LND : 256/ 1; 0
ICE : 256/ 1; 256
OCN : 64/ 1; 512
ROF : 256/ 1; 0
GLC : 512/ 1; 0
WAV : 512/ 1; 0
ESP : 1/ 1; 0

with
CASE INFO:
nodes: 9
total tasks: 576
tasks per node: 64
thread count: 1

 

[jedwards]

You need to be aware of the ROOTPE variable which is currently set to 128 for ICE and 256 for OCN. You should change it so that
ROOTPE_ICE=NTASKS_LND and ROOTPE_OCN=NTASKS_ATM, you should also set NTASKS_LND+NTASKS_ICE=NTASKS_ATM.

 

[Juan]

And how this is going to change the number of nodes that I am requesting? how the number of nodes is determined?
Many thanks.

 

[Juan]

I am going to be more specific. I need to run this case using 4 nodes (in exclusive) with 64 tasks per node. Therefore I need that my CASE INFO reads:
CASE INFO:
nodes: 4
total tasks: 256
tasks per node: 64
thread count: 1

After doing a case.build, what files and variables I need to modify to make it possible?
Thanks.

 

[jedwards]

Comp NTASKS NTHRDS ROOTPE
CPL : 512/ 1; 0
ATM : 512/ 1; 0
LND : 256/ 1; 0
ICE : 256/ 1; 256
OCN : 64/ 1; 512
ROF : 256/ 1; 0
GLC : 512/ 1; 0
WAV : 512/ 1; 0
ESP : 1/ 1; 0

Is what you get out of the box - correct?
Here is a suggested layout using 256 total tasks:

Comp NTASKS NTHRDS ROOTPE
CPL : 224/ 1; 0
ATM : 224/ 1; 0
LND : 128/ 1; 0
ICE : 96/ 1; 128
OCN : 32/ 1; 224
ROF : 128/ 1; 0
GLC : 32/ 1; 0
WAV : 32/ 1; 0
ESP : 1/ 1; 0


The total number of tasks is determined by MAX(NTASKS_XXX + ROOTPE_XXX) so in this case NTASKS_OCN+ROOTPE_OCN=256.

 

[Juan]

Dear Jim,
Many thanks for your suggestion. However, this does not address my question. How is it determined the number of "nodes" to be used?.

Thanks,
Juan

 

[jedwards]

What exactly do you not understand? The total number of tasks is determined by MAX(NTASKS_XXX + ROOTPE_XXX) so in this case NTASKS_OCN+ROOTPE_OCN=256. The total number of nodes is the total number of tasks/ MAX_MPITASKS_PER_NODE.

 

[Juan]

Hi Jim,

Now I have got it. What was not obvious to me is that the number of nodes was dynamically modified according to the setup of MAX(NTASKS_XXX + ROOTPE_XXX)/MAX_MPITASKS_PER_NODE, and that I should create my own pelayout "template" step by step using an ./xmlchange ROOTPE_XXX, and similar for NTASKS and NTHRDS.
Our machine is composed of nodes with 64 cores/node, and we were trying to modify the number of nodes to be used to do some performance tests.
That said, is there a rule or suggestion to distribute the NTASK and ROOTPE used to run the ATM, ICE, CPL, etc. components. I see that for example OCN gets 8 times less task (computing) than the ROOTPE value, and the NTASKS for the coupler and atmosphere.

Thanks again for all this help.
Juan

 

[jedwards]

I'm afraid that there is no hard rule, I've given you all the guidance I can - now it's up to you to optimize for your particular combination of cpu and network.

 

[Juan]

Really appreciated anyway, the question on the nodes is now clear.
Many thanks for the help.
Juan

 

[yongsheng zheng]

Comp NTASKS NTHRDS ROOTPE
CPL : 512/ 1; 0
ATM : 512/ 1; 0
LND : 256/ 1; 0
ICE : 256/ 1; 256
OCN : 64/ 1; 512
ROF : 256/ 1; 0
GLC : 512/ 1; 0
WAV : 512/ 1; 0
ESP : 1/ 1; 0

Is what you get out of the box - correct?
Here is a suggested layout using 256 total tasks:

Comp NTASKS NTHRDS ROOTPE
CPL : 224/ 1; 0
ATM : 224/ 1; 0
LND : 128/ 1; 0
ICE : 96/ 1; 128
OCN : 32/ 1; 224
ROF : 128/ 1; 0
GLC : 32/ 1; 0
WAV : 32/ 1; 0
ESP : 1/ 1; 0


The total number of tasks is determined by MAX(NTASKS_XXX + ROOTPE_XXX) so in this case NTASKS_OCN+ROOTPE_OCN=256.

Hi jedwards,
I ran the B1850C5CN case in the PE layout set by you on CESM1.2.2, but the operation failed, and the run log is as follows. I can't understand the reason. Could you give me an explanation？