Minimal geometry for running CCSM 3?

[strombrg@dcs_nac_uci_edu]

The University of California at Irvine has an IBM SP2 compute cluster that was purchased primarily for the purpose of running CCSM.

We're running CCSM 2 OK, but CCSM 3 is not running (it seems to run one I set all tasks and threads to 1, but I'm told that should not work).

I'm also told that folks at NCAR have some smallish compute clusters there, which if we were to copy from might give UCI a CCSM 3 geometry that would work better on our hardware.

What we have, are 7 nodes with 8 CPU's (one of which is almost entirely for interactive use, so it's really more like 6), and 1 node that has 32 CPU's.

Is there anyone who could provide us with such a minimal geometry for CCSM 3?

Thanks!

 

[murphys]

hi strom,

just to clarify are you looking for the task and threads specification that goes into the scripts or something else?

sylvia

 

[strombrg@dcs_nac_uci_edu]

hi strom,

just to clarify are you looking for the task and threads specification that goes into the scripts or something else?

sylvia

Hi.

Yes, we're looking for suitable tasks and threads settings for 6 nodes with 8 CPU's and 1 node with 32 CPU's, which I believe would appear in our "env.ibm.esmf" file.

That "esmf" name might be worth detailing a bit: Apparently you folks have some software named "esmf". Coincidentally, the UCI IBM SP2 computer I'm describing is also called ESMF, for "Earth System Modeling Facility".

Thanks!

 

[gcarr@ucar_edu]

One of the configurations that we have run on NCAR's IBM Power4 machine is as follows for a T42x1 resolution:

ocn 32 cpus
atm 16 cpus configured as 4 MPI tasks, 4 threads each
ice 8 cpus
lnd 4 cpus
cpl 4 cpus

Total 64 cpus
Only the atm component is configured for hybrid (MPI and OMP).

This configuration was run with a prerelease version of CCSM3, beta18 but should be a good place to start for the CCSM3 release version. On our system we were able to get 5.03 simulation years per wall clock day at the T42x1 resolution.

As a starting point for a smaller configuration, I would reduce each processor count in half for each component.

 

[strombrg@dcs_nac_uci_edu]

One of the configurations that we have run on NCAR's IBM Power4 machine is as follows for a T42x1 resolution:

ocn 32 cpus
atm 16 cpus configured as 4 MPI tasks, 4 threads each
ice 8 cpus
lnd 4 cpus
cpl 4 cpus

Total 64 cpus
Only the atm component is configured for hybrid (MPI and OMP).

This configuration was run with a prerelease version of CCSM3, beta18 but should be a good place to start for the CCSM3 release version. On our system we were able to get 5.03 simulation years per wall clock day at the T42x1 resolution.

As a starting point for a smaller configuration, I would reduce each processor count in half for each component.

OK, I've created a small webpage describing the situation I'm encountering now at http://dcs.nac.uci.edu/~strombrg/CCSM3-geometry/. In short, I cut the geometry above in half, but when I submit, the job just sits there in the queue, and it seems to be requiring 10 node types - does that mean we would need 10 nodes, each with a sufficient number of CPU's, to run with the geometryl I'm trying to use now?

Thanks!

 

[strombrg@dcs_nac_uci_edu]

One of the configurations that we have run on NCAR's IBM Power4 machine is as follows for a T42x1 resolution:

ocn 32 cpus
atm 16 cpus configured as 4 MPI tasks, 4 threads each
ice 8 cpus
lnd 4 cpus
cpl 4 cpus

Total 64 cpus
Only the atm component is configured for hybrid (MPI and OMP).

This configuration was run with a prerelease version of CCSM3, beta18 but should be a good place to start for the CCSM3 release version. On our system we were able to get 5.03 simulation years per wall clock day at the T42x1 resolution.

As a starting point for a smaller configuration, I would reduce each processor count in half for each component.

OK, I've created a small webpage describing the situation I'm encountering now at http://dcs.nac.uci.edu/~strombrg/CCSM3-geometry/. In short, I cut the geometry above in half, but when I submit, the job just sits there in the queue, and it seems to be requiring 10 node types - does that mean we would need 10 nodes, each with a sufficient number of CPU's, to run with the geometryl I'm trying to use now?

Thanks!

I got in touch with IBM about the way my job was getting stuck in a queue, and they confirmed my suspicion: The geometry we're trying is requiring 10 nodes, while we have 7 (or 8 if we push it) nodes available for our jobs.

The Task_geometry we're getting, according to llq -s, is:

{(0,1,2,3)(4,5,6,7)(8,9,10,11)(12,13,14,15)(16,17,18,19)(20,21,22,23)(24)(25)(26)(27)}

Is there a good way of reducing the number of nodes required?

Thanks!

 

[jedwards]

Is your 32 way node in the loadleveler class as your 8 ways? Is the interactive node defined in the class or not - it probably should not be.

Part of the problem in my opinion is that you are starting from the largest model configuration and trying to get it to run on your small cluster. I suggest that you take a different approach. Start from a configuration with all data models and just run the coupler. When that works try adding the atmosphere, then the ocean and finally when you can do that add the other components.

Another approach that might lend some insight to the problem is to wirte a 'hello world' type excerise which you submit in various mpmd configuration to test your load leveler setup.

When you can subimit a simple job with the following task geometry and have it executed you will have the problem 95% solved.

{(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15)(16,17,18,19)(20,21,22,23)(24)(25)(26)(27)}

thats 7 nodes including the 32 way.

It looks like your original configuration was based on the definition for the 4proc/node blackforest machine at NCAR - perhaps basing it on the bluesky 8proc/node would have been a better starting point.

 

[strombrg@dcs_nac_uci_edu]

Is your 32 way node in the loadleveler class as your 8 ways? Is the interactive node defined in the class or not - it probably should not be.

Part of the problem in my opinion is that you are starting from the largest model configuration and trying to get it to run on your small cluster. I suggest that you take a different approach. Start from a configuration with all data models and just run the coupler. When that works try adding the atmosphere, then the ocean and finally when you can do that add the other components.

Another approach that might lend some insight to the problem is to wirte a 'hello world' type excerise which you submit in various mpmd configuration to test your load leveler setup.

When you can subimit a simple job with the following task geometry and have it executed you will have the problem 95% solved.

{(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15)(16,17,18,19)(20,21,22,23)(24)(25)(26)(27)}

thats 7 nodes including the 32 way.

It looks like your original configuration was based on the definition for the 4proc/node blackforest machine at NCAR - perhaps basing it on the bluesky 8proc/node would have been a better starting point.

OK, I modified my -very- skeletal loadleveler job to use the above geometry, and a queue (class) that should have the physical resources to run a job with this geometry.

We're using the "backfill" scheduler.

The Earth System Modeling Facility at UCI is somewhat busy - is it possible that this test job, because it requires so many nodes, could starve from attention as long as there are a handful of other jobs running on the system, and we aren't using a preemptive scheduler?

To make it easier to help UCI if you're so inclined, I'll re-include the URL documenting our efforts: http://dcs.nac.uci.edu/~strombrg/CCSM3-geometry/.

BTW, if I ever ask a question that you folks feel would be better posed to IBM, please feel free to point that out.

Thanks!

 

[jedwards]

==================== EVALUATIONS FOR JOB STEP esmf04m.3350.0 ====================

SUMMARY

This LoadLeveler cluster does not have sufficient resources at the present time to run this job step.

I don't think its going to run if you see this message. Start smaller and work your way up.

 

[strombrg@dcs_nac_uci_edu]

==================== EVALUATIONS FOR JOB STEP esmf04m.3350.0 ====================

SUMMARY

This LoadLeveler cluster does not have sufficient resources at the present time to run this job step.

I don't think its going to run if you see this message. Start smaller and work your way up.

OK, I've come up with a largest job that will get executed given the present load on the system. I've described what I did and found again at http://dcs.nac.uci.edu/~strombrg/CCSM3-geometry/.

In short, phost_mpi (similar to LOADL_PROCESSOR_LIST) gives strange errors, but $LOADL_PROCESSOR_LIST works OK.

The largest geometry that worked was:

# @ task_geometry = {(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15)(16,17,18,19)(20,21,22,23)(24)}

Is there a way we can get CCSM3 to run with such a small geometry?

Thanks!

 

[jedwards]

If you can run with that you should be able to use all 32 procs of the 32 way + 8*3 = 56 processors.

ocn 28
cpl 4
atm 16
lnd 4
cpl 4

Put the ocn + one of the 4 processor components on the 32 way , give the atmos 2- 8 ways and the other tow components the third. I believe that you can tell the ccsm3 script the total number of processors that you have to work with, then edit by hand the task_geometry before you submit the job to finish the setup.

 

[strombrg@dcs_nac_uci_edu]

If you can run with that you should be able to use all 32 procs of the 32 way + 8*3 = 56 processors.

ocn 28
cpl 4
atm 16
lnd 4
cpl 4

Put the ocn + one of the 4 processor components on the 32 way , give the atmos 2- 8 ways and the other tow components the third. I believe that you can tell the ccsm3 script the total number of processors that you have to work with, then edit by hand the task_geometry before you submit the job to finish the setup.

Apologies in advance for what I'm sure must be a really basic question to you, but how might I tell which elements of the task_geometry correspond to which modles, and how can I make two models run on the same node?

If this is more of an IBM question, please feel free to point that out...

Thanks!

 

[jedwards]

It's a question of understanding the
scripts that you are using.
They create a poe.cmdfile with the names of the executables, one per line, these correspond to the executable numbers starting from zero. So if you have a poe.cmdfile that looks like

one
one
two
three

and a task_geometry = {(0,3)(1,2)}
you will get executables one and two on the first node and one and three on the second node. You can put as many different executables on a node as you have processors there is no issue of conflict.

Please don't take offense but it is not IBM you need to ask and its not CGD either. You need to read the manuals and get a basic understanding of poe, mpi and ccsm3. Read through the scripts that you are submitting and try to understand what they are intended to do.

 

[strombrg@dcs_nac_uci_edu]

It's a question of understanding the
scripts that you are using.
They create a poe.cmdfile with the names of the executables, one per line, these correspond to the executable numbers starting from zero. So if you have a poe.cmdfile that looks like

one
one
two
three

and a task_geometry = {(0,3)(1,2)}
you will get executables one and two on the first node and one and three on the second node. You can put as many different executables on a node as you have processors there is no issue of conflict.

Please don't take offense but it is not IBM you need to ask and its not CGD either. You need to read the manuals and get a basic understanding of poe, mpi and ccsm3. Read through the scripts that you are submitting and try to understand what they are intended to do.

I'm not offended at all!

I'm expert in enough areas that I know sometimes a little advice from an expert can save a newbie a lot of flailing around in the manuals. :)

But point taken, I'll try to rely on your ability to contextualize and conjoin the vast stores of information in the IBM doc less often.

I googled up a web page about task_geometry's meaning. Between that, and your very helpful hints above, I think I may be getting the meaning of that better now.

However, is it possible the task_geometry example you've given above is slightly off (If so, I'm sure it's a silly little mistake or a test for a pupil, and not a conceptual problem).

IOW, I am under the impression that there's a mapping like so:

0 means one
1 means one
2 means two
3 means three

...by virtue of the order of the values in teh poe.cmdfile (or other poe file), and the first pair of parens is the first node, and the second pair of parens is the second node of two. So shouldn't "one" and "three" be running on the 1st node, and "one" and "two" be running on the 2nd?

Thanks!

 

[jedwards]

...by virtue of the order of the values in teh poe.cmdfile (or other poe file), and the first pair of parens is the first node, and the second pair of parens is the second node of two. So shouldn't "one" and "three" be running on the 1st node, and "one" and "two" be running on the 2nd?

Yup - you're right. Good Luck.