Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

OpenMP

M

mengpai@gmail_com

New Member
Hi there,

Does anyone know how to set up the CCSM3 running under OpenMP?
Beside "-mp" option, do I have to set the variables like "setenv OPM_NUM_THREADS #" somewhere or something else? If so, how should I do it?
Thank you very much.

Odden
 
S

suvarchal_kumar@gmail_com

Member
By default if you are using build scripts(for a machine) given in ccsm3 it sets env variable OMP_NUM_THREADS by itself according to nthrds_*** in file env_mach.***** for each component executable, . Again whole process might depend on what your machine, mpi run(or any) expect.


odden said:
Hi there,

Does anyone know how to set up the CCSM3 running under OpenMP?
Beside "-mp" option, do I have to set the variables like "setenv OPM_NUM_THREADS #" somewhere or something else? If so, how should I do it?
Thank you very much.

Odden
Click to expand...
 
M

mengpai@gmail_com

New Member
suvarchal said:
By default if you are using build scripts(for a machine) given in ccsm3 it sets env variable OMP_NUM_THREADS by itself according to nthrds_*** in file env_mach.***** for each component executable, . Again whole process might depend on what your machine, mpi run(or any) expect.
Click to expand...

Thanks for reply.
I am struggling with cpu assgined. I can build a case, but the case runs always fail due to the incompatible cpu numbers. What is the rule that I must follow up for the cpu assigned?
Here is my hardware information:

Machine: IBM AMD Opteron 1350 cluster with Linux, 4 cpus per node
mpi: mvapich 0.9.9, using mpiexec
pgi: 7.0-4
Batch: PBS
netCDF: netcdf 3.6.2
Network: Infiniband

Any suggestions will be appreciated. Thank you very much.

Odden
 
S

suvarchal_kumar@gmail_com

Member
can you tell show me error in log?
odden said:
Thanks for reply.
I am struggling with cpu assgined. I can build a case, but the case runs always fail due to the incompatible cpu numbers. What is the rule that I must follow up for the cpu assigned?
Here is my hardware information:

Machine: IBM AMD Opteron 1350 cluster with Linux, 4 cpus per node
mpi: mvapich 0.9.9, using mpiexec
pgi: 7.0-4
Batch: PBS
netCDF: netcdf 3.6.2
Network: Infiniband

Any suggestions will be appreciated. Thank you very much.

Odden
Click to expand...
 
M

mengpai@gmail_com

New Member
suvarchal said:
can you tell show me error in log?
Click to expand...

Here is my task and thread setting:

setenv NTASKS_ATM 64
setenv NTHRDS_ATM 1
setenv NTASKS_LND 24
setenv NTHRDS_LND 1
setenv NTASKS_ICE 32
setenv NTHRDS_ICE 1
setenv NTASKS_OCN 96
setenv NTHRDS_OCN 1
setenv NTASKS_CPL 8
setenv NTHRDS_CPL 1

total tasks = 224

PBS header:
## BATCH INFO
#PBS -N THY.01a
# Maximum number of processes
#PBS -l nodes=56:ppn=4
#PBS -l walltime=04:00:00
# Put standar error and standar out in same file
#PBS -j oe
#PBS -m ae
# Export all Environment vaiables
#PBS -V
# End of options

The error message:
-------------------------------------------------------------------------
Preparing T31_gx3v5 component models for execution
-------------------------------------------------------------------------
- Create execution directories for atm,cpl,lnd,ice,ocn
- If a restart run then copy restart files into executable directory
ccsm_getrestart: get /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref restarts from /d/odden/caserun/archive/THY.01a.T31_gx3v5.B.glenn.110152.ref/restart
- Check validity of configuration
- Determine if build must happen (env variable BLDTYPE)
- Build flag (BLDTYPE) is TRUE
- Build Libraries: esmf, mph, mct
Sun Apr 6 11:04:19 EDT 2008 esmf.buildlib.080406-110417
Sun Apr 6 11:04:52 EDT 2008 mph.buildlib.080406-110417
Sun Apr 6 11:04:55 EDT 2008 mct.buildlib.080406-110417
- Create model directories for each platform
- Determine if models must be rebuilt
- Build model executables, create namelist files, prestage input data
Sun Apr 6 11:06:31 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/cpl/cpl.log.080406-110417
Sun Apr 6 11:06:31 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/cpl/cpl.buildexe.080406-110417
Sun Apr 6 11:08:02 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/ice/ice.log.080406-110417
Sun Apr 6 11:08:02 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/ice/ice.buildexe.080406-110417
ERROR: csim.buildexe.csh failed, see ice.buildexe.080406-110417
ERROR: cat /home/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/ice/ice.buildexe.080406-110417


"ice.buildexe.080406-110417" shows:

ERROR: NX must divide evenly into grid, 100,16


So far I haven't apply OpenMP setting yet.
Does it have any rule to assign cpu number for each component?

Thank you for your time.

-Odden
 
M

mengpai@gmail_com

New Member
odden said:
Here is my task and thread setting:

setenv NTASKS_ATM 64
setenv NTHRDS_ATM 1
setenv NTASKS_LND 24
setenv NTHRDS_LND 1
setenv NTASKS_ICE 32
setenv NTHRDS_ICE 1
setenv NTASKS_OCN 96
setenv NTHRDS_OCN 1
setenv NTASKS_CPL 8
setenv NTHRDS_CPL 1

total tasks = 224

PBS header:
## BATCH INFO
#PBS -N THY.01a
# Maximum number of processes
#PBS -l nodes=56:ppn=4
#PBS -l walltime=04:00:00
# Put standar error and standar out in same file
#PBS -j oe
#PBS -m ae
# Export all Environment vaiables
#PBS -V
# End of options

The error message:
-------------------------------------------------------------------------
Preparing T31_gx3v5 component models for execution
-------------------------------------------------------------------------
- Create execution directories for atm,cpl,lnd,ice,ocn
- If a restart run then copy restart files into executable directory
ccsm_getrestart: get /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref restarts from /d/odden/caserun/archive/THY.01a.T31_gx3v5.B.glenn.110152.ref/restart
- Check validity of configuration
- Determine if build must happen (env variable BLDTYPE)
- Build flag (BLDTYPE) is TRUE
- Build Libraries: esmf, mph, mct
Sun Apr 6 11:04:19 EDT 2008 esmf.buildlib.080406-110417
Sun Apr 6 11:04:52 EDT 2008 mph.buildlib.080406-110417
Sun Apr 6 11:04:55 EDT 2008 mct.buildlib.080406-110417
- Create model directories for each platform
- Determine if models must be rebuilt
- Build model executables, create namelist files, prestage input data
Sun Apr 6 11:06:31 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/cpl/cpl.log.080406-110417
Sun Apr 6 11:06:31 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/cpl/cpl.buildexe.080406-110417
Sun Apr 6 11:08:02 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/ice/ice.log.080406-110417
Sun Apr 6 11:08:02 EDT 2008 /d/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/ice/ice.buildexe.080406-110417
ERROR: csim.buildexe.csh failed, see ice.buildexe.080406-110417
ERROR: cat /home/odden/caserun/THY.01a.T31_gx3v5.B.glenn.110152.ref/ice/ice.buildexe.080406-110417


"ice.buildexe.080406-110417" shows:

ERROR: NX must divide evenly into grid, 100,16


So far I haven't apply OpenMP setting yet.
Does it have any rule to assign cpu number for each component?

Thank you for your time.

-Odden
Click to expand...

Just fund out the answer for this question. The script shows the csim cpu number limitation in
/$CASE/Buildexe/csim.buildexe.csh

According to this script, the cpu assignment for gx3v5 to csim should be:
1, 2, 4, 5, 8, 10, 16, 20, 25, 40, 50, 100, and 200.

For gx1v3, the csim cpu assignment should be:
1, 2, 4, 5, 8, 10, 16, 20, 32, 40, 60, 80, 128, 160, 320, and 640.

For the ocean, the rule is showed at the chapter 2.2.2 Processor configuration in POP user's guide:
http://www.ccsm.ucar.edu/models/ccsm3.0/pop/doc/POPusers_chap2.html

Notice that it shows the POP simulation with a 384x288 horizontal grid, which is different from that showed at page 7 in CCSM3 user's guide.

The csim.buildexe.csh script shows that the one in CCSM3 user's guide is correct.

In CCSM3 user's guide, the ocean and ice resolution are:

gx1v3 - 320 lon, 384 lat (384x320).
gx3v5 - 100 lon, 116 lat (116x100)


-odden
 
M

mengpai@gmail_com

New Member
odden said:
Just fund out the answer for this question. The script shows the csim cpu number limitation in
/$CASE/Buildexe/csim.buildexe.csh

According to this script, the cpu assignment for gx3v5 to csim should be:
1, 2, 4, 5, 8, 10, 16, 20, 25, 40, 50, 100, and 200.

For gx1v3, the csim cpu assignment should be:
1, 2, 4, 5, 8, 10, 16, 20, 32, 40, 60, 80, 128, 160, 320, and 640.

For the ocean, the rule is showed at the chapter 2.2.2 Processor configuration in POP user's guide:
http://www.ccsm.ucar.edu/models/ccsm3.0/pop/doc/POPusers_chap2.html

Notice that it shows the POP simulation with a 384x288 horizontal grid, which is different from that showed at page 7 in CCSM3 user's guide.

The csim.buildexe.csh script shows that the one in CCSM3 user's guide is correct.

In CCSM3 user's guide, the ocean and ice resolution are:

gx1v3 - 320 lon, 384 lat (384x320).
gx3v5 - 100 lon, 116 lat (116x100)


-odden
Click to expand...

The pop building script shows its processor assignment in:
/$CASE/Buildexe/pop.buildexe.csh

The processor for pop could be:
1, 2, 4, 8, 10, 12, 16, 20, 24, 32, 40, 48, 64, 80, and 96.

I don't know if it can be added even more.
==========================================
Back to the openMP question, my config file for mpiexec looks like:
-n $ntask : /d/odden/caserun/1870ctr/all/cpl
-n $ntask : /d/odden/caserun/1870ctr/all/csim
-n $ntask : /d/odden/caserun/1870ctr/all/clm
-n $ntask : /d/odden/caserun/1870ctr/all/pop
-n $ntask : /d/odden/caserun/1870ctr/all/cam


If I would like to run with openMP, I should put this file like:
-n $ntask_cpl : env OMP_NUM_THREADS $nthrds_cpl /d/odden/caserun/1870ctr/all/cpl
-n $ntask_csim : env OMP_NUM_THREADS $nthrds_csim /d/odden/caserun/1870ctr/all/csim
-n $ntask_clm : env OMP_NUM_THREADS $nthrds_clm /d/odden/caserun/1870ctr/all/clm
-n $ntask_pop : env OMP_NUM_THREADS $nthrds_pop /d/odden/caserun/1870ctr/all/pop
-n $ntask_cam :env OMP_NUM_THREADS $nthrds_cam /d/odden/caserun/1870ctr/all/cam

Am I right? Is there environment variable for openMP that I have to set it up?
Thanks for the answer.

-Odden
 
S

suvarchal_kumar@gmail_com

Member
i think you cannot add more ntasks with out changing way pop is decomposed.
i dont know how mpiexec expects arguments but looking at your file it looks correct. I guess OMP_NUM_THREADS is enough.

odden said:
The pop building script shows its processor assignment in:
/$CASE/Buildexe/pop.buildexe.csh

The processor for pop could be:
1, 2, 4, 8, 10, 12, 16, 20, 24, 32, 40, 48, 64, 80, and 96.

I don't know if it can be added even more.
==========================================
Back to the openMP question, my config file for mpiexec looks like:
-n $ntask : /d/odden/caserun/1870ctr/all/cpl
-n $ntask : /d/odden/caserun/1870ctr/all/csim
-n $ntask : /d/odden/caserun/1870ctr/all/clm
-n $ntask : /d/odden/caserun/1870ctr/all/pop
-n $ntask : /d/odden/caserun/1870ctr/all/cam


If I would like to run with openMP, I should put this file like:
-n $ntask_cpl : env OMP_NUM_THREADS $nthrds_cpl /d/odden/caserun/1870ctr/all/cpl
-n $ntask_csim : env OMP_NUM_THREADS $nthrds_csim /d/odden/caserun/1870ctr/all/csim
-n $ntask_clm : env OMP_NUM_THREADS $nthrds_clm /d/odden/caserun/1870ctr/all/clm
-n $ntask_pop : env OMP_NUM_THREADS $nthrds_pop /d/odden/caserun/1870ctr/all/pop
-n $ntask_cam :env OMP_NUM_THREADS $nthrds_cam /d/odden/caserun/1870ctr/all/cam

Am I right? Is there environment variable for openMP that I have to set it up?
Thanks for the answer.

-Odden
Click to expand...
 
M

mengpai@gmail_com

New Member
suvarchal said:
i think you cannot add more ntasks with out changing way pop is decomposed.
i dont know how mpiexec expects arguments but looking at your file it looks correct. I guess OMP_NUM_THREADS is enough.
Click to expand...

Thanks for your reply.

-Odden
 
You must log in or register to reply here.
Top