Specifying Number of MPI Tasks

[sbasu@iarc_uaf_edu]

Hi,
I am doing a standalone CAM 5.3 run (without CICE) with climatological dataset. I configured the model with -nosmp -nospmd.  ../cesm1_2_2/models/atm/cam/bld/configure -dyn fv -hgrid 0.9x1.25 -nc_inc /opt/cray/netcdf/4.2.0/pgi/119/include -nc_lib /opt/cray/netcdf/4.2.0/pgi/119/lib -nc_mod /opt/cray/netcdf/4.2.0/pgi/119/include -mpi_inc /opt/cray/mpt/7.0.0/gni/mpich2-pgi/141/include -mpi_lib /opt/cray/mpt/7.0.0/gni/mpich2-pgi/141/lib -fc ftn -fc_type pgi -nospmd -nosmp -test -vBut, I am wondering what exactly the option -ntasks under build-namelist does. I was trying to set -ntasks via the build-namelist command but it could not generate the namelist. It did not give me any error either. It was just stuck there so I had to kill the command. fish1 %   ../cesm1_2_2/models/atm/cam/bld/build-namelist -test -config /center/w/basu/cam_5/cam_bld_test/config_cache.xml -case test_3 -namelist "&atm nhtfrq(1) = -6 mfilt(1) = 1460 stop_option = 'nyears' stop_n = 1 -ntasks 12 bndtvs = '/center/w/basu/cam_5/inputdata/atm/cam/sst/sst_HadOIBl_bc_0.9x1.25_clim_c040926a.nc'/"^C
fish1 % lsfish1 %I am submitting a batch script for the model execution where I am specifying OMP_NUM_THREADS#!/bin/bash

#PBS -N Test_3

#PBS -q standard

#PBS -l walltime=24:00:00

#PBS -l nodes=12:ppn=12

#PBS -j oe

cd $PBS_O_WORKDIR

export LID="`date +%y%m%d-%H%M%S`"

echo "`date` -- CAM 5.3 EXECUTION BEGINS HERE"

export OMP_NUM_THREADS=12

aprun -n 12 -d 12 /center/w/basu/cam_5/cam_bld_test/cam >& cam.log

echo "`date` -- CAM 5.3 RUN ENDS HERE"

Currently the model is taking ~20 hours to complete 1 year model simulation so I am trying to make the model run faster.Also, for doing a restart run should I simply change the runtype to continue via the build-namelist command after the startup run? Thanks,Soumik 

 

[jedwards]

-nospmd turns off MPI parallel -nosmp turns off OPENMP parallel what you have left can only run on one task and so setting -ntasks 12 doesn't make sense.

 

[sbasu@iarc_uaf_edu]

Thank you for your reply.I configured the model with -nosmp and -nospmd in order to not to use CICE. In the user guide it says under CAM Parallelization to build the model with smp and spmd switched off if it doesn't contain CICE. There it also says it's task count can be set at run time.So, how do I run standalone CAM without CICE because I will use prescribed sea ice and prescribed SST in the future for my experiments.  Thanks,Soumik

 

[eaton]

As Jim pointed out the configure options "-nospmd -nosmp" are for
a serial build of the model.  This has nothing to do with the
scientific configuration.

The CICE component requires that the number of tasks and threads
be specified at build time.  This is the reason the -ntasks and
-nthreads arguments were added to configure.  If you build CAM
without the CICE component (e.g. when doing adiabatic or ideal
physics runs, or when doing an aquaplanet run) then the task and
thread counts can be specified at runtime.  This is the instance
where it is sufficient to turn on MPI and/or OMP simply by using
the -spmd and/or -smp arguments to configure.

The -ntasks argument to build-namelist is a convenience feature
that is used only to set the 2D decomposition specification for
the FV dycore (the variable npr_yz).  It has nothing to do with
turning MPI on or off.

The reason your build-namelist command hung is that you gave it
an illegal namelist syntax and the namelist parser was not robust
enough to catch the error gracefully.  "-ntasks 12" is an
argument to build-namelist.  It is not part of the namelist
string which is the value of the -namelist argument.

The CICE component is used in CAM standalone the same way it's
used in a CESM "F" compset.  It is the component that provides
the functionality for prescribed sea ice, and it does the
thermodynamic calculations that provide the necessary fluxes to
the atmosphere. 

 

[sbasu@iarc_uaf_edu]

Thank you so much for your detailed reply.I have one more question. When I am running standalone CAM along with CAM history files the model is producing CICE history files too. In the CICE component history files there is "grid cell mean ice thickness" variable. In the previous version (for example : CAM 3) the thickness used to be constant. In this standalone version of CAM 5.3 how can the model calculate sea ice thickness unless it is fully coupled with CICE model?Does this model calculated thickenss affects the monthly prescribed sea ice forcings at each grid point? Thanks,Soumik  

 

[eaton]

CAM3 used a different sea ice package (csim4).  I don't know how cice in prescribed mode treats ice thickness.  Perhaps the cice documentation has info.  Or a question to the CICE section of the BB.

 

[sbasu@iarc_uaf_edu]

Thank you so much.Soumik