Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Large restart file output error in a high resolution configuration for CAM5-EUL dynamic core

E

eaton

CSEG and Liaisons
How much memory and how many cores on a shared memory node?How many tasks are you using per node?  What is the total number of nodes?The output log will contain information about the pio_stride and pio_numtasks.  What are those numbers?  
 
E

eaton

CSEG and Liaisons
How much memory and how many cores on a shared memory node?How many tasks are you using per node?  What is the total number of nodes?The output log will contain information about the pio_stride and pio_numtasks.  What are those numbers?  
 
E

eaton

CSEG and Liaisons
How much memory and how many cores on a shared memory node?How many tasks are you using per node?  What is the total number of nodes?The output log will contain information about the pio_stride and pio_numtasks.  What are those numbers?  
 
E

eaton

CSEG and Liaisons
How much memory and how many cores on a shared memory node?How many tasks are you using per node?  What is the total number of nodes?The output log will contain information about the pio_stride and pio_numtasks.  What are those numbers?  
 
E

eaton

CSEG and Liaisons
How much memory and how many cores on a shared memory node?How many tasks are you using per node?  What is the total number of nodes?The output log will contain information about the pio_stride and pio_numtasks.  What are those numbers?  
 
Z

zhangyi@lasg_iap_ac_cn

Member
Hi eaton, the machine is 12 cpus per node, but I don't know the memory.Normally, I use 400 cpus for this resolution, because this is the maximum cores that EUL-core can use.34 nodes are used, so tasks per node is 12 or less. In this setting, pio_stride=4 pio_numtasks= 100To double the nodes, actually I use 1000 cpus on 84 nodes, of course, only 400 cores for dynamics.pio_stride=4 pio_numtasks= 250.
 
Z

zhangyi@lasg_iap_ac_cn

Member
Hi eaton, the machine is 12 cpus per node, but I don't know the memory.Normally, I use 400 cpus for this resolution, because this is the maximum cores that EUL-core can use.34 nodes are used, so tasks per node is 12 or less. In this setting, pio_stride=4 pio_numtasks= 100To double the nodes, actually I use 1000 cpus on 84 nodes, of course, only 400 cores for dynamics.pio_stride=4 pio_numtasks= 250.
 
Z

zhangyi@lasg_iap_ac_cn

Member
Hi eaton, the machine is 12 cpus per node, but I don't know the memory.Normally, I use 400 cpus for this resolution, because this is the maximum cores that EUL-core can use.34 nodes are used, so tasks per node is 12 or less. In this setting, pio_stride=4 pio_numtasks= 100To double the nodes, actually I use 1000 cpus on 84 nodes, of course, only 400 cores for dynamics.pio_stride=4 pio_numtasks= 250.
 
Z

zhangyi@lasg_iap_ac_cn

Member
Hi eaton, the machine is 12 cpus per node, but I don't know the memory.Normally, I use 400 cpus for this resolution, because this is the maximum cores that EUL-core can use.34 nodes are used, so tasks per node is 12 or less. In this setting, pio_stride=4 pio_numtasks= 100To double the nodes, actually I use 1000 cpus on 84 nodes, of course, only 400 cores for dynamics.pio_stride=4 pio_numtasks= 250.
 
Z

zhangyi@lasg_iap_ac_cn

Member
Hi eaton, the machine is 12 cpus per node, but I don't know the memory.Normally, I use 400 cpus for this resolution, because this is the maximum cores that EUL-core can use.34 nodes are used, so tasks per node is 12 or less. In this setting, pio_stride=4 pio_numtasks= 100To double the nodes, actually I use 1000 cpus on 84 nodes, of course, only 400 cores for dynamics.pio_stride=4 pio_numtasks= 250.
 
Z

zhangyi@lasg_iap_ac_cn

Member
Hi eaton, the machine is 12 cpus per node, but I don't know the memory.Normally, I use 400 cpus for this resolution, because this is the maximum cores that EUL-core can use.34 nodes are used, so tasks per node is 12 or less. In this setting, pio_stride=4 pio_numtasks= 100To double the nodes, actually I use 1000 cpus on 84 nodes, of course, only 400 cores for dynamics.pio_stride=4 pio_numtasks= 250.
 
Z

zhangyi@lasg_iap_ac_cn

Member
Hi eaton, the machine is 12 cpus per node, but I don't know the memory.Normally, I use 400 cpus for this resolution, because this is the maximum cores that EUL-core can use.34 nodes are used, so tasks per node is 12 or less. In this setting, pio_stride=4 pio_numtasks= 100To double the nodes, actually I use 1000 cpus on 84 nodes, of course, only 400 cores for dynamics.pio_stride=4 pio_numtasks= 250.
 
Z

zhangyi@lasg_iap_ac_cn

Member
Hi eaton, the machine is 12 cpus per node, but I don't know the memory.Normally, I use 400 cpus for this resolution, because this is the maximum cores that EUL-core can use.34 nodes are used, so tasks per node is 12 or less. In this setting, pio_stride=4 pio_numtasks= 100To double the nodes, actually I use 1000 cpus on 84 nodes, of course, only 400 cores for dynamics.pio_stride=4 pio_numtasks= 250.
 
E

eaton

CSEG and Liaisons
I would try to double the nodes by requesting 68 nodes and only assign 6 tasks per node.  This gives each task twice as much memory as in the successful configuration used for the 30 level grid.  I would also try setting the namelist variable atm_pio_stride=6.  This will put just 1 pio task on each node which should minimize the overhead incurred when writing the restart file.
 
E

eaton

CSEG and Liaisons
I would try to double the nodes by requesting 68 nodes and only assign 6 tasks per node.  This gives each task twice as much memory as in the successful configuration used for the 30 level grid.  I would also try setting the namelist variable atm_pio_stride=6.  This will put just 1 pio task on each node which should minimize the overhead incurred when writing the restart file.
 
E

eaton

CSEG and Liaisons
I would try to double the nodes by requesting 68 nodes and only assign 6 tasks per node.  This gives each task twice as much memory as in the successful configuration used for the 30 level grid.  I would also try setting the namelist variable atm_pio_stride=6.  This will put just 1 pio task on each node which should minimize the overhead incurred when writing the restart file.
 
E

eaton

CSEG and Liaisons
I would try to double the nodes by requesting 68 nodes and only assign 6 tasks per node.  This gives each task twice as much memory as in the successful configuration used for the 30 level grid.  I would also try setting the namelist variable atm_pio_stride=6.  This will put just 1 pio task on each node which should minimize the overhead incurred when writing the restart file.
 
E

eaton

CSEG and Liaisons
I would try to double the nodes by requesting 68 nodes and only assign 6 tasks per node.  This gives each task twice as much memory as in the successful configuration used for the 30 level grid.  I would also try setting the namelist variable atm_pio_stride=6.  This will put just 1 pio task on each node which should minimize the overhead incurred when writing the restart file.
 
E

eaton

CSEG and Liaisons
I would try to double the nodes by requesting 68 nodes and only assign 6 tasks per node.  This gives each task twice as much memory as in the successful configuration used for the 30 level grid.  I would also try setting the namelist variable atm_pio_stride=6.  This will put just 1 pio task on each node which should minimize the overhead incurred when writing the restart file.
 
E

eaton

CSEG and Liaisons
I would try to double the nodes by requesting 68 nodes and only assign 6 tasks per node.  This gives each task twice as much memory as in the successful configuration used for the 30 level grid.  I would also try setting the namelist variable atm_pio_stride=6.  This will put just 1 pio task on each node which should minimize the overhead incurred when writing the restart file.
 
You must log in or register to reply here.
Top