calculation of no. of processor in cam5 run

[sbshewale@gmail_com]

Hello,I have been runnning cam5.0 (in standalone mode) on our HPC (linux based) with hgrid 1.9x2.5 using ifort.my 1st expt was to look, runtime by cam5.0 in 32 no .of processor(at the time of configuration I mentioned  -ntasks 32 -nosmp)It has run successfully.Now I have given same run for 64 no. of processor, but after some point it shows lines as,Mod_comm - r4 windows disabled
Mod_comm t1_win window size = 53568
Mod_comm r8_win window size = 1365984
Mod_comm r4_win window size = 1
Mod_comm i4_win window size = 1244160
ENDRUN:SPMDINIT_DYN: less than 3 latitudes per subdomain
mpiexec_compute-0-14.local: mpd_uncaught_except_tb handling:
exceptions.IOError: [Errno 28] No space left on device
/opt/mpich2_intel/bin/mpirun 1073 handle_cli_stderr_input
sys.stderr.write(msg)
/opt/mpich2_intel/bin/mpdlib.py 762 handle_active_streams
handler(stream,*args)
/opt/mpich2_intel/bin/mpirun 530 mpiexec
rv = streamHandler.handle_active_streams(timeout=1.0)
/opt/mpich2_intel/bin/mpirun 1446 ?
mpiexec()If anybody knows about this, Any kind of help will be appreciated.Thanks in advance. 

 

[rneale]

By default CAM does 1d domain decompostion in latitude only. There is a restriction that there has to be at least 3 latitudes per processor so with 32PEs it's ok but with 64PEs there are not enough latitudes at 2 deg resolution. I think you should be able to switch to 2d decomposition to use 64PEs but this may not necessarily see much of a speed-up due to the increased communication overhead.

 

[hannay]

It also look like you might have a space problem:See error message: "No space left on device"

 

[eaton]

To follow up on Rich's suggestion, we routinely use the 2D decomposition with FV to scale beyond the 1D limit.  For the 1.9x2.5 grid w/ 30 levels you should see reasonable scaling out to 480 tasks using the namelist setting npr_zy=32,15,15,32.  It's possible that the scaling will continue out to 960 tasks using npr_yz=32,30,30,32 although the parallel efficiency will be low pushing things to this limit.

 

[prathamandwal@gmail_com]

Can you  tell me how do run that way ? or what set of changes you make?So that I can also run with more than f19g16 resolution with more than 32 mpi processAlso I have read that we cannot change the number of tasks between configure and build and run ,and when I confiure with more than  32 mpi process,the build fails ,  RegardsPratha