OpenMP

[mengpai@gmail_com]

Hi there,

Does anyone know how to set up the CCSM3 running under OpenMP?
Beside "-mp" option, do I have to set the variables like "setenv OPM_NUM_THREADS #" somewhere or something else? If so, how should I do it?
Thank you very much.

Odden

 

[suvarchal_kumar@gmail_com]

By default if you are using build scripts(for a machine) given in ccsm3 it sets env variable OMP_NUM_THREADS by itself according to nthrds_*** in file env_mach.***** for each component executable, . Again whole process might depend on what your machine, mpi run(or any) expect.

Hi there,

Does anyone know how to set up the CCSM3 running under OpenMP?
Beside "-mp" option, do I have to set the variables like "setenv OPM_NUM_THREADS #" somewhere or something else? If so, how should I do it?
Thank you very much.

Odden

 

[mengpai@gmail_com]

By default if you are using build scripts(for a machine) given in ccsm3 it sets env variable OMP_NUM_THREADS by itself according to nthrds_*** in file env_mach.***** for each component executable, . Again whole process might depend on what your machine, mpi run(or any) expect.

Thanks for reply.
I am struggling with cpu assgined. I can build a case, but the case runs always fail due to the incompatible cpu numbers. What is the rule that I must follow up for the cpu assigned?
Here is my hardware information:

Machine: IBM AMD Opteron 1350 cluster with Linux, 4 cpus per node
mpi: mvapich 0.9.9, using mpiexec
pgi: 7.0-4
Batch: PBS
netCDF: netcdf 3.6.2
Network: Infiniband

Any suggestions will be appreciated. Thank you very much.

Odden

 

[suvarchal_kumar@gmail_com]

can you tell show me error in log?

Thanks for reply.
I am struggling with cpu assgined. I can build a case, but the case runs always fail due to the incompatible cpu numbers. What is the rule that I must follow up for the cpu assigned?
Here is my hardware information:

Machine: IBM AMD Opteron 1350 cluster with Linux, 4 cpus per node
mpi: mvapich 0.9.9, using mpiexec
pgi: 7.0-4
Batch: PBS
netCDF: netcdf 3.6.2
Network: Infiniband

Any suggestions will be appreciated. Thank you very much.

Odden

 

[mengpai@gmail_com]

can you tell show me error in log?

Here is my task and thread setting:

setenv NTASKS_ATM 64
setenv NTHRDS_ATM 1
setenv NTASKS_LND 24
setenv NTHRDS_LND 1
setenv NTASKS_ICE 32
setenv NTHRDS_ICE 1
setenv NTASKS_OCN 96
setenv NTHRDS_OCN 1
setenv NTASKS_CPL 8
setenv NTHRDS_CPL 1

total tasks = 224

PBS header:
## BATCH INFO
#PBS -N THY.01a
# Maximum number of processes
#PBS -l nodes=56:ppn=4
#PBS -l walltime=04:00:00
# Put standar error and standar out in same file
#PBS -j oe
#PBS -m ae
# Export all Environment vaiables
#PBS -V
# End of options

The error message:
-------------------------------------------------------------------------
Preparing T31_gx3v5 component models for execution
-------------------------------------------------------------------------
- Create execution directories for atm,cpl,lnd,ice,ocn
- If a restart run then copy restart files into executable directory
ccsm_getrestart: get /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref restarts from /d/odden/caserun/archive/THY.01a.T31_gx3v5.B.glenn.110152.ref/restart
- Check validity of configuration
- Determine if build must happen (env variable BLDTYPE)
- Build flag (BLDTYPE) is TRUE
- Build Libraries: esmf, mph, mct
Sun Apr 6 11:04:19 EDT 2008 esmf.buildlib.080406-110417
Sun Apr 6 11:04:52 EDT 2008 mph.buildlib.080406-110417
Sun Apr 6 11:04:55 EDT 2008 mct.buildlib.080406-110417
- Create model directories for each platform
- Determine if models must be rebuilt
- Build model executables, create namelist files, prestage input data
Sun Apr 6 11:06:31 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/cpl/cpl.log.080406-110417
Sun Apr 6 11:06:31 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/cpl/cpl.buildexe.080406-110417
Sun Apr 6 11:08:02 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/ice/ice.log.080406-110417
Sun Apr 6 11:08:02 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/ice/ice.buildexe.080406-110417
ERROR: csim.buildexe.csh failed, see ice.buildexe.080406-110417
ERROR: cat /home/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/ice/ice.buildexe.080406-110417


"ice.buildexe.080406-110417" shows:

ERROR: NX must divide evenly into grid, 100,16


So far I haven't apply OpenMP setting yet.
Does it have any rule to assign cpu number for each component?

Thank you for your time.

-Odden

 

[mengpai@gmail_com]

Here is my task and thread setting:

setenv NTASKS_ATM 64
setenv NTHRDS_ATM 1
setenv NTASKS_LND 24
setenv NTHRDS_LND 1
setenv NTASKS_ICE 32
setenv NTHRDS_ICE 1
setenv NTASKS_OCN 96
setenv NTHRDS_OCN 1
setenv NTASKS_CPL 8
setenv NTHRDS_CPL 1

total tasks = 224

PBS header:
## BATCH INFO
#PBS -N THY.01a
# Maximum number of processes
#PBS -l nodes=56:ppn=4
#PBS -l walltime=04:00:00
# Put standar error and standar out in same file
#PBS -j oe
#PBS -m ae
# Export all Environment vaiables
#PBS -V
# End of options

The error message:
-------------------------------------------------------------------------
Preparing T31_gx3v5 component models for execution
-------------------------------------------------------------------------
- Create execution directories for atm,cpl,lnd,ice,ocn
- If a restart run then copy restart files into executable directory
ccsm_getrestart: get /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref restarts from /d/odden/caserun/archive/THY.01a.T31_gx3v5.B.glenn.110152.ref/restart
- Check validity of configuration
- Determine if build must happen (env variable BLDTYPE)
- Build flag (BLDTYPE) is TRUE
- Build Libraries: esmf, mph, mct
Sun Apr 6 11:04:19 EDT 2008 esmf.buildlib.080406-110417
Sun Apr 6 11:04:52 EDT 2008 mph.buildlib.080406-110417
Sun Apr 6 11:04:55 EDT 2008 mct.buildlib.080406-110417
- Create model directories for each platform
- Determine if models must be rebuilt
- Build model executables, create namelist files, prestage input data
Sun Apr 6 11:06:31 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/cpl/cpl.log.080406-110417
Sun Apr 6 11:06:31 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/cpl/cpl.buildexe.080406-110417
Sun Apr 6 11:08:02 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/ice/ice.log.080406-110417
Sun Apr 6 11:08:02 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/ice/ice.buildexe.080406-110417
ERROR: csim.buildexe.csh failed, see ice.buildexe.080406-110417
ERROR: cat /home/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/ice/ice.buildexe.080406-110417


"ice.buildexe.080406-110417" shows:

ERROR: NX must divide evenly into grid, 100,16


So far I haven't apply OpenMP setting yet.
Does it have any rule to assign cpu number for each component?

Thank you for your time.

-Odden

Just fund out the answer for this question. The script shows the csim cpu number limitation in
/$CASE/Buildexe/csim.buildexe.csh

According to this script, the cpu assignment for gx3v5 to csim should be:
1, 2, 4, 5, 8, 10, 16, 20, 25, 40, 50, 100, and 200.

For gx1v3, the csim cpu assignment should be:
1, 2, 4, 5, 8, 10, 16, 20, 32, 40, 60, 80, 128, 160, 320, and 640.

For the ocean, the rule is showed at the chapter 2.2.2 Processor configuration in POP user's guide:
http://www.ccsm.ucar.edu/models/ccsm3.0/pop/doc/POPusers_chap2.html

Notice that it shows the POP simulation with a 384x288 horizontal grid, which is different from that showed at page 7 in CCSM3 user's guide.

The csim.buildexe.csh script shows that the one in CCSM3 user's guide is correct.

In CCSM3 user's guide, the ocean and ice resolution are:

gx1v3 - 320 lon, 384 lat (384x320).
gx3v5 - 100 lon, 116 lat (116x100)


-odden

 

[mengpai@gmail_com]

Just fund out the answer for this question. The script shows the csim cpu number limitation in
/$CASE/Buildexe/csim.buildexe.csh

According to this script, the cpu assignment for gx3v5 to csim should be:
1, 2, 4, 5, 8, 10, 16, 20, 25, 40, 50, 100, and 200.

For gx1v3, the csim cpu assignment should be:
1, 2, 4, 5, 8, 10, 16, 20, 32, 40, 60, 80, 128, 160, 320, and 640.

For the ocean, the rule is showed at the chapter 2.2.2 Processor configuration in POP user's guide:
http://www.ccsm.ucar.edu/models/ccsm3.0/pop/doc/POPusers_chap2.html

Notice that it shows the POP simulation with a 384x288 horizontal grid, which is different from that showed at page 7 in CCSM3 user's guide.

The csim.buildexe.csh script shows that the one in CCSM3 user's guide is correct.

In CCSM3 user's guide, the ocean and ice resolution are:

gx1v3 - 320 lon, 384 lat (384x320).
gx3v5 - 100 lon, 116 lat (116x100)


-odden

The pop building script shows its processor assignment in:
/$CASE/Buildexe/pop.buildexe.csh

The processor for pop could be:
1, 2, 4, 8, 10, 12, 16, 20, 24, 32, 40, 48, 64, 80, and 96.

I don't know if it can be added even more.
==========================================
Back to the openMP question, my config file for mpiexec looks like:
-n $ntask : /d/odden/caserun/1870ctr/all/cpl
-n $ntask : /d/odden/caserun/1870ctr/all/csim
-n $ntask : /d/odden/caserun/1870ctr/all/clm
-n $ntask : /d/odden/caserun/1870ctr/all/pop
-n $ntask : /d/odden/caserun/1870ctr/all/cam


If I would like to run with openMP, I should put this file like:
-n $ntask_cpl : env OMP_NUM_THREADS $nthrds_cpl /d/odden/caserun/1870ctr/all/cpl
-n $ntask_csim : env OMP_NUM_THREADS $nthrds_csim /d/odden/caserun/1870ctr/all/csim
-n $ntask_clm : env OMP_NUM_THREADS $nthrds_clm /d/odden/caserun/1870ctr/all/clm
-n $ntask_pop : env OMP_NUM_THREADS $nthrds_pop /d/odden/caserun/1870ctr/all/pop
-n $ntask_cam :env OMP_NUM_THREADS $nthrds_cam /d/odden/caserun/1870ctr/all/cam

Am I right? Is there environment variable for openMP that I have to set it up?
Thanks for the answer.

-Odden

 

[suvarchal_kumar@gmail_com]

i think you cannot add more ntasks with out changing way pop is decomposed.
i dont know how mpiexec expects arguments but looking at your file it looks correct. I guess OMP_NUM_THREADS is enough.

The pop building script shows its processor assignment in:
/$CASE/Buildexe/pop.buildexe.csh

The processor for pop could be:
1, 2, 4, 8, 10, 12, 16, 20, 24, 32, 40, 48, 64, 80, and 96.

I don't know if it can be added even more.
==========================================
Back to the openMP question, my config file for mpiexec looks like:
-n $ntask : /d/odden/caserun/1870ctr/all/cpl
-n $ntask : /d/odden/caserun/1870ctr/all/csim
-n $ntask : /d/odden/caserun/1870ctr/all/clm
-n $ntask : /d/odden/caserun/1870ctr/all/pop
-n $ntask : /d/odden/caserun/1870ctr/all/cam


If I would like to run with openMP, I should put this file like:
-n $ntask_cpl : env OMP_NUM_THREADS $nthrds_cpl /d/odden/caserun/1870ctr/all/cpl
-n $ntask_csim : env OMP_NUM_THREADS $nthrds_csim /d/odden/caserun/1870ctr/all/csim
-n $ntask_clm : env OMP_NUM_THREADS $nthrds_clm /d/odden/caserun/1870ctr/all/clm
-n $ntask_pop : env OMP_NUM_THREADS $nthrds_pop /d/odden/caserun/1870ctr/all/pop
-n $ntask_cam :env OMP_NUM_THREADS $nthrds_cam /d/odden/caserun/1870ctr/all/cam

Am I right? Is there environment variable for openMP that I have to set it up?
Thanks for the answer.

-Odden

 

[mengpai@gmail_com]

i think you cannot add more ntasks with out changing way pop is decomposed.
i dont know how mpiexec expects arguments but looking at your file it looks correct. I guess OMP_NUM_THREADS is enough.

Thanks for your reply.

-Odden