Extremely slow compilation on a generic_linux_machine

[gchiodo@fis_ucm_es]

Dear all, I am trying to port CESM1.0.1 to a "generic Linux machine", which is a super-computer with HP-rx7640 servers, with Intel(R) Xeon(TM) CPUs.

I am encountering extremely slow compilation-time; the compiler used is IFORT/11.1.

The model builds (apparently correctly, but then execution fails with a segfault) in 15 to 20 hours. Is such a compilation-time normal?

The flags used for the compilation are as follows:

CFLAGS := $(CPPDEFS) -w -O2 -ftz -tpp2 -fno-alias -fno-fnalias -ip -g
FREEFLAGS := -FR
FFLAGS := $(CPPDEFS) -w -cm -cpp -WB -fpp -ftz -fpconstant -mtune=itanium2 -autodouble -tpp2 -fno-alias -fno-fnalias -stack_temps -ip -assume byterecl -convert big_endian -g
FFLAGS_OPT := -O2
FFLAGS_NOOPT := $(FFLAGS)
LDFLAGS := -Wl,--noinhibit-exec -Vaxlib -posixlib
AR := ar
MOD_SUFFIX := mod

Could the compilation-speed be related to some vectorization, which is automatically done by the ifort compiler?

Kind regards

 

[eaton]

see http://forum.cgd.ucar.edu/showthread.php?1818-Workaround-for-slow-build-with-intel-ifort

The workaround described in this post is part of the cesm1_0_2 release.

 

[gchiodo@fis_ucm_es]

Thanks for your prompt reply.

I tried to compile the cesm1_0_2 version, but the problem persists. The compilation seems to hang hours and hours in the LAND module (the other components are compiled in 10-15 minutes...). I'm building cesm for the "B_1850-2000_WACCM_CN" compset.

One example of the ifort invocations reads as follows:

/opt/hpmpi/bin/mpif90 -mpif90 ifort -c -I. -I/opt/cesga/netcdf-3.6.2/include -I/opt/cesga/netcdf-3.6.2/include -I/opt/cesga/hp-mpi-2.3/MPICH2.0//include -I. -I/sfs/home/uvi/fa/waccm/test-waccm4/B20TRWCN/SourceMods/src.clm -I/home/uvi/fa/waccm/waccm4/cesm1_0_2/models/lnd/clm/src/main -I/home/uvi/fa/waccm/waccm4/cesm1_0_2/models/lnd/clm/src/biogeophys -I/home/uvi/fa/waccm/waccm4/cesm1_0_2/models/lnd/clm/src/biogeochem -I/home/uvi/fa/waccm/waccm4/cesm1_0_2/models/lnd/clm/src/riverroute -I/home/uvi/fa/waccm/waccm4/cesm1_0_2/models/lnd/clm/src/main/cpl_mct -I/sfs/home/uvi/fa/waccm/test-waccm4//B20TRWCN/lib/include -DMAXPATCH_PFT=numpft+1 -DPROGSSLT -DRTM -DDUST -DCN -DCLAMP -D_USEBOX -D_NETCDF -DMCT_INTERFACE -DHAVE_MPI -DCO2A -DLINUX -DSEQ_ -DFORTRANUNDERSCORE -DNO_R16 -DNO_SHR_VMATH -w -cm -cpp -WB -fpp -ftz -fpconstant -mtune=itanium2 -autodouble -tpp2 -fno-alias -fno-fnalias -stack_temps -ip -assume byterecl -convert big_endian -g -O2 -FR /home/uvi/fa/waccm/waccm4/cesm1_0_2/models/lnd/clm/src/main/initSurfAlbMod.F90


What could be the cause for the extremely slow compilation?

I also tried to compile this version on another machine with ifort, and experienced the same problem, so it does not seem to be related to this particular machine.

Regards

 

[dfeddema@ucar_edu]

I was not able to reproduce this slow compilation problem at NCAR. I suspect it is because we are using slightly different versions of the compiler. Hence, I think your best bet is to use a different version of ifort – ifort 11.0 and ifort 11.1.059 work for me.

I tested ifort 11.0 on Edinburgh (unix cluster) compiling CESM1.0.1 with all of the same compiler options you used and the same compset (B_1850-2000_WACCM_CN). I set compiler options “-m” and “-x” to generate code for the Intel® Xeon™ processor. I tested ifort 11.1.059 on Lynx (Cray XT5) again cross compiling for intel Xeon processor and using all of the options you used. My compile times were in the range of 20-23 minutes for clm on both of these machines (Edniburgh and Lynx). So the main difference between what I’m running and what you’re running is the compiler version – of course we are running on different hardware and OS but that’s probably not the problem. Hence, I suggest trying a different version of ifort.

If it is difficult for you to change your compiler version you could try removing the “ip” option, which turns on interprocedural optimizations. If it is still slow I would turn off optimization completely (-O0) and see what happens. If this fixes the problem I would bump the optimization level up to –O1 to find an optimization level that is working.

 

[gchiodo@fis_ucm_es]

The removal of the -ip flag solved the problem.
CESM does now compile correctly with ifort/11.1 in about 28 minutes.

Thanks a lot!