Successful compiler options ifort, intel linux cluster, openmpi

[kshell]

Here are the compiler flags that worked for me (CCSM passed DB, ER, HR, and YR tests, plus 20 year run looks reasonable when compared with NCAR control run using AMWG diagnostic package). Some of the flags are redundant, but it works, so I don't want to waste time if it doesn't change the output. If you see a problem with anything, please let me know though.

I'm using ifort (/opt/intel/fce/10.1.015/bin/ifort) and OpenMPI. OS is CentOS (latest Rocks release). Intel E5420 chips. If you want additional info, let me know.

Macros.Linux:

INCLDIR := -I. -I$(INC_NETCDF) -I$(INCROOT) -I$(INC_MPI) -I$(NETCDF_MOD)
SLIBS := -L$(LIB_NETCDF) -lnetcdf -llapack -lblas -L$(LIB_MPI) -lmpi_f77 -lmpi

ULIBS := -L$(LIBROOT) -lesmf -lmct -lmpeu -lmph
CPP := NONE
CPPFLAGS := -DLINUX -DPGF90 -DNO_SHR_VMATH
CPPDEFS := -DLINUX
CC := icc
CFLAGS := -c
CFLAGS += -DUSE_GCC
FIXEDFLAGS :=

FC := ifort
FFLAGS := -cpp -c -r8 -i4 -132 -autodouble -convert big_endian -fp-model precise -prec-div -prec-sqrt
# c: prevents linking
# cpp: use c pre-processor
# r8 8byte reals (autodouble does this as well)
# i4 4 byte ints
# 132 statements end at column 132
CPPFLAGS := -DLINUX -DPGF90 -DNO_SHR_VMATH -DINTEL_COMPILER -DpgiFortran -Df2cFortran
# some of these probably can be eliminated. I didn't want to spend too much time refining this, since I have something that works


MOD_SUFFIX := mod
LD := $(FC)

AR := ar

ifeq ($(MODEL),pop)
CPPDEFS := $(CPPDEFS) -DPOSIX -Dimpvmix -Dcoupled
-DNPROC_X=$(NX) -DNPROC_Y=$(NY)
FIXEDFLAGS := -convert big_endian -assume byterecl
endif

ifeq ($(MODEL),csim)
CPPDEFS := $(CPPDEFS) -Dcoupled -DNPROC_X=$(NX) -DNPROC_Y=$(NY) -D_MPI
FIXEDFLAGS := -convert big_endian -assume byterecl
endif

ifeq ($(THREAD),TRUE)
CPPDEFS := $(CPPDEFS) -D_OPENMP -DTHREADED_PTHREADS
FREEFLAGS := $(FREEFLAGS) -openmp
LDFLAGS := $(LDFLAGS) -openmp
endif
# haven't checked OMP flags yet.

ifeq ($(DEBUG),TRUE)
FFLAGS := $(FFLAGS) -debug
endif

I haven't tried hybrid MPI/openmp yet. I submit the jobs with sun grid engine, and my mpirun command (in the run script) is

mpirun -v -x KMP_STACKSIZE -np $NTASKS[1] $COMPONENTS[1] : -np $NTASKS[2] $COMPONENTS[2] : -np $NTASKS[3] $COMPONENTS[3] : -np $NTASKS[4] $COMPONENTS[4] : -np $NTASKS[5] $COMPONENTS[5]

My env.linux.machine file includes:

set MPICH = /usr/mpi/intel/openmpi-1.2.5
setenv INC_MPI $MPICH/include
setenv LIB_MPI $MPICH/lib64
setenv PATH ${MPICH}/bin:${PATH}
setenv MPI_LIB_NAME mpi_f77
setenv INC_NETCDF /usr/local/include
setenv LIB_NETCDF /usr/local/lib64

Enjoy!
-Karen

 

[avenger@avenger_ws]

I'm not sure if your host has SSE4 (is it only newer xeons I guess), but you might try some intel optimizations flags to make things go faster (and check whether they are breakin results or not).

If you like the idea, the flags I am using (on older CCM3, I am migrating to CCSM3 by now) are:

On models/bld/Macros.Linux:

CPPFLAGS   := -DLINUX -DINTEL -DNO_SHR_VMATH -O3 -c -axST -ip -ipo -xST -mtune=core2 -extend-source -r8 -i4
CFLAGS     := -O3 -c -march=nocona
(...)
FFLAGS     := -fc=ifort -O3 -c -axST -ip -ipo -xST -mtune=core2 -extend-source -r8 -i4

It is worth trying. I could make the old CCM3 up to three times faster on a single processor and about 15x faster with the -openmp flag (making it use the various processors at once).

Hope it helps in anything! :]

 

[jfarran@uci_edu]

mpirun -v -x KMP_STACKSIZE -np $NTASKS[1] $COMPONENTS[1] : -np $NTASKS[2] $COMPONENTS[2] : -np $NTASKS[3] $COMPONENTS[3] : -np $NTASKS[4] $COMPONENTS[4] : -np $NTASKS[5] $COMPONENTS[5]

-Karen

Hi Karen.

Have you been able to run over multiple nodes and if so, how are you using mpirun to spread the programs over different nodes?

Joseph

 

[aerler@atmosp_physics_utoronto_ca]

Hi,
I'm trying to compile the single column model of the CCM3 on my own laptop; I have kubuntu 8.10, 64bit, with gcc and intel compilers. It does not seem to work with gcc/gfortan, but it supposedly works with intel.

I tried the compiler options Karen posted, but the linking fails. But of course I'm not using MPI, so there may be some adjustments necessary...

Compiling the test-files works, but icc gives me the following warning:

icc -c -c -DUSE_GCC -g -o /home/me/Research/NCAR/cam1/models/atm/cam/tools/scam/obj/ctest.o -DLINUX -DPGF90 -DNO_SHR_VMATH -DINTEL_COMPILER -DpgiFortran -Df2cFortran -I/home/me/Research/NCAR/cam1/models/atm/cam/tools/scam/scm_init/home/me/Research/NCAR/cam1/models/atm/cam/tools/scam/obj/ctest.c
/home/me/Research/NCAR/cam1/models/atm/cam/tools/scam/obj/ctest.c(16): warning #181: argument is incompatible with corresponding format string conversion
fprintf( stderr, "ERROR - C compiler uses %d byte integersn", sizeof(i[0]) );
^

/home/me/Research/NCAR/cam1/models/atm/cam/tools/scam/obj/ctest.c(20): warning #181: argument is incompatible with corresponding format string conversion
fprintf( stderr, "ERROR - C compiler uses %d byte doublesn", sizeof(d[0]) );

Does anyone know what that means? ifort does not complain at all.
But linking fails completely with the following error message:

icc -o /home/me/Research/NCAR/cam1/models/atm/cam/tools/scam/obj/testprg/home/me/Research/NCAR/cam1/models/atm/cam/tools/scam/obj/ctest.o /home/me/Research/NCAR/cam1/models/atm/cam/tools/scam/obj/f90test.o /home/me/Research/NCAR/cam1/models/atm/cam/tools/scam/obj/f77test.o -lifport -lifcore -limf -Wabi -Wcast-align
icc: command line warning #10156: ignoring option '-W'; no argument required
/usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'

You can see the compiler flags in the output... I have no idea what would be the correct flags would be, since I'm fairly new to all this business... (I also tried linking with ifort but the result is similar)
Any help is highly appreciated!
Thanks,
Andre

 

[kshell]

Thanks, avenger, for the suggestions. I did try some optimizations a while ago, but the resulting code didn't pass the restart test. You've got some new ones listed, so I might give them a try. I know that using openmpi will speed things up; I just haven't set up the ntasks and nthrds to use it.

Joseph, I am running over multiple nodes. I use the mpi command I listed. Are you asking about the number of processors per model component? RIght now, I'm just using the default.

Andre, it looks like you need to find the right flag for your c compiler to indicate the size of the integers and doubles. Note the "-r8 -i4" in my FFLAGS line. You might need to add something to your C flags.

Good luck,
-Karen

 

[avenger@avenger_ws]

Andre,

This problems looks like 'linking' to modules both compiled in c and in fortran. I had such a problem with old CCM3, and what I have to do was to give an option to the compiler so it generated both modules with the same underscoring.
For example, if the c code is looking for symbols like 'main' and fortran is looking for symbols like '_main', in the linking trouble will happen.

I don't know the default, but the -Dpgifortran is likely to give you trouble as you are not using the portland group set of compilers (pgi, pgf90 and so on).
Also, the -Df2cfortran could be igniting some default on the defines that could be breaking the code.
I don't recognise -Wabi nor -Wcast-align on ifort. You should understand what are these flags and then input the correct ones on fortran. Aren't there pgi also?
A lillte research on manual pages made me think these flags are gfortran only, and the -W are just to suppress-show some kinds of warnings. Have you revised the compiler flags while trying to compile with intel fortran? Note that the flags for each compiler differs from intel to pgi and to gnu's.
How have you switched between compilers?.. Give the flags I shown here a try and they might do it (drop at least the -Dpgifortran parameter and get rid of pgi).

Hope this helped you a bit.