POP.F terminated by signal 9

[mengpai@gmail_com]

Hi there,
I am building the ccsm3 on IBM AMD cluster. The compliation passes through lnd, cpl, ice, but crased at pop with a message: TERMINATED by signal 9. I was wondering if anybody have solved the same problem before, or any suggestions to the solution. You will be really appreciated.
Thank you.
-Odden

Due to the text limitation, I cut the lines for the part of successful compiling files.

My environment is:
Machine: IBM AMD Opteron 1350 cluster with Linux
mpi: mvapich 0.9.9
pgi: 7.0-4
Batch: PBS
netCDF: netcdf 3.6.2
Network: Infiniband
Rrunning message:
## Warning: SCRATCH not defined in system environment. Set SCRATCH to be /home/usr4918
-------------------------------------------------------------------------
Preparing T31_gx3v5 component models for execution
-------------------------------------------------------------------------
- Create execution directories for atm,cpl,lnd,ice,ocn
- If a restart run then copy restart files into executable directory
ccsm_getrestart: get /home/usr4918 restarts from /home/usr4918/archive/TER.01a.T31_gx3v5.B.glenn.131725/restart
- Check validity of configuration
- Determine if build must happen (env variable BLDTYPE)
- Build flag (BLDTYPE) is TRUE
- Build Libraries: esmf, mph, mct
Fri Feb 15 16:54:20 EST 2008 esmf.buildlib.080215-165420
Fri Feb 15 16:54:21 EST 2008 mph.buildlib.080215-165420
Fri Feb 15 16:54:21 EST 2008 mct.buildlib.080215-165420
- Create model directories for each platform
- Determine if models must be rebuilt
- Build model executables, create namelist files, prestage input data
Fri Feb 15 16:54:23 EST 2008 /home/usr4918/cpl/cpl.log.080215-165420
Fri Feb 15 16:54:23 EST 2008 /home/usr4918/cpl/cpl.buildexe.080215-165420
Fri Feb 15 16:54:24 EST 2008 /home/usr4918/ice/ice.log.080215-165420
Fri Feb 15 16:54:24 EST 2008 /home/usr4918/ice/ice.buildexe.080215-165420
Fri Feb 15 16:54:25 EST 2008 /home/usr4918/lnd/lnd.log.080215-165420
Fri Feb 15 16:54:25 EST 2008 /home/usr4918/lnd/lnd.buildexe.080215-165420
Fri Feb 15 16:54:26 EST 2008 /home/usr4918/ocn/ocn.log.080215-165420
Fri Feb 15 16:54:26 EST 2008 /home/usr4918/ocn/ocn.buildexe.080215-165420
ERROR: pop.buildexe.csh failed, see ocn.buildexe.080215-165420
ERROR: cat /home/usr4918/ocn/ocn.buildexe.080215-165420

ocn.buildexe.080215-165420 shows:
Tue Feb 19 13:22:10 EST 2008 /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/ocn.buildexe.080219-132022
## Warning: SCRATCH not defined in system environment. Set SCRATCH to be /home/usr4918
-------------------------------------------------------------------------
a. Build an executable in /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj
-------------------------------------------------------------------------
a.1 Calculate processor tiling based on 24
-----------------------------------------------------------------
a2. creating the internal directory structure
-----------------------------------------------------------------
a3. create model_size.F in /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source, first computing NT
-----------------------------------------------------------------
ocn.iage.setup.csh : setting nt
-----------------------------------------------------------------
copy the necessary files into $source_dir
-----------------------------------------------------------------
cp: No match.
cp: No match.
diff: /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/ocnres.old: No such file or directory
-----------------------------------------------------------------
a4. compile and copy the executable into /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn directory
-----------------------------------------------------------------
Building dependency for xdisplay.d
Building dependency for vmix_rich.d
...(successfully passes through)
Building dependency for baroclinic.d
Building dependency for advection.d
Building dependency for POP.d
mpif90 -I. -I/usr/local/netcdf-3.6.2/include -I/usr/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/lib/include -I/usr/local/mpi/mvapich-0.9.8-pgi/include -I/usr/local/pgi/linux86-64/7.0-4/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source -c -r8 -i4 -Kieee -Mdalign -Mextend -Msave -tp amd64 -O3 -byteswapio -DLINUX -DPGF90 -DNO_SHR_VMATH -DLINUX -DPOSIX -Dimpvmix -Dcoupled -DNPROC_X=4 -DNPROC_Y=6 /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source/kinds_mod.F
mpif90 -I. -I/usr/local/netcdf-3.6.2/include -I/usr/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/lib/include -I/usr/local/mpi/mvapich-0.9.8-pgi/include -I/usr/local/pgi/linux86-64/7.0-4/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source -c -r8 -i4 -Kieee -Mdalign -Mextend -Msave -tp amd64 -O3 -byteswapio -DLINUX -DPGF90 -DNO_SHR_VMATH -DLINUX -DPOSIX -Dimpvmix -Dcoupled -DNPROC_X=4 -DNPROC_Y=6 /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source/model_size.F
...(successfully passes through)
mpif90 -I. -I/usr/local/netcdf-3.6.2/include -I/usr/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/lib/include -I/usr/local/mpi/mvapich-0.9.8-pgi/include -I/usr/local/pgi/linux86-64/7.0-4/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source -c -r8 -i4 -Kieee -Mdalign -Mextend -Msave -tp amd64 -O3 -byteswapio -DLINUX -DPGF90 -DNO_SHR_VMATH -DLINUX -DPOSIX -Dimpvmix -Dcoupled -DNPROC_X=4 -DNPROC_Y=6 /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source/forcing.F
mpif90 -I. -I/usr/local/netcdf-3.6.2/include -I/usr/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/lib/include -I/usr/local/mpi/mvapich-0.9.8-pgi/include -I/usr/local/pgi/linux86-64/7.0-4/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source -c -r8 -i4 -Kieee -Mdalign -Mextend -Msave -tp amd64 -O3 -byteswapio -DLINUX -DPGF90 -DNO_SHR_VMATH -DLINUX -DPOSIX -Dimpvmix -Dcoupled -DNPROC_X=4 -DNPROC_Y=6 /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source/diagnostics.F
mpif90 -I. -I/usr/local/netcdf-3.6.2/include -I/usr/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/lib/include -I/usr/local/mpi/mvapich-0.9.8-pgi/include -I/usr/local/pgi/linux86-64/7.0-4/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source -c -r8 -i4 -Kieee -Mdalign -Mextend -Msave -tp amd64 -O3 -Mfree -DLINUX -DPGF90 -DNO_SHR_VMATH -DLINUX -DPOSIX -Dimpvmix -Dcoupled -DNPROC_X=4 -DNPROC_Y=6 /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source/shr_vmath_mod.F90
...(successfully passes through)
mpif90 -I. -I/usr/local/netcdf-3.6.2/include -I/usr/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/lib/include -I/usr/local/mpi/mvapich-0.9.8-pgi/include -I/usr/local/pgi/linux86-64/7.0-4/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source -c -r8 -i4 -Kieee -Mdalign -Mextend -Msave -tp amd64 -O3 -byteswapio -DLINUX -DPGF90 -DNO_SHR_VMATH -DLINUX -DPOSIX -Dimpvmix -Dcoupled -DNPROC_X=4 -DNPROC_Y=6 /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source/initial.F
mpif90 -I. -I/usr/local/netcdf-3.6.2/include -I/usr/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/lib/include -I/usr/local/mpi/mvapich-0.9.8-pgi/include -I/usr/local/pgi/linux86-64/7.0-4/include -I/tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source -c -r8 -i4 -Kieee -Mdalign -Mextend -Msave -tp amd64 -O3 -byteswapio -DLINUX -DPGF90 -DNO_SHR_VMATH -DLINUX -DPOSIX -Dimpvmix -Dcoupled -DNPROC_X=4 -DNPROC_Y=6 /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source/POP.F
pgf90-Fatal-/usr/local/pgi/linux86-64/7.0-4/bin/pgf901 TERMINATED by signal 9
Arguments to /usr/local/pgi/linux86-64/7.0-4/bin/pgf901
/usr/local/pgi/linux86-64/7.0-4/bin/pgf901 /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source/POP.F -opt 3 -terse 1 -inform warn -nohpf -nostatic -x 19 0x400000 -quad -x 59 4 -x 59 4 -x 15 2 -x 49 0x400004 -x 51 0x20 -x 57 0x4c -x 58 0x10000 -x 124 0x1000 -x 57 0xfb0000 -x 58 0x78031040 -x 70 0x6c00 -x 47 0x400000 -x 48 4608 -x 49 0x100 -x 120 0x200 -stdinc /usr/local/pgi/linux86-64/7.0-4/include-gcc41:/usr/local/pgi/linux86-64/7.0-4/include:/usr/local/include:/usr/lib/gcc/x86_64-redhat-linux/4.1.1/include:/usr/lib/gcc/x86_64-redhat-linux/4.1.1/include:/usr/include -def unix -def __unix -def __unix__ -def linux -def __linux -def __linux__ -def __NO_MATH_INLINES -def __x86_64__ -def __LONG_MAX__=9223372036854775807L -def '__SIZE_TYPE__=unsigned long int' -def '__PTRDIFF_TYPE__=long int' -def __THROW= -def __extension__= -def __amd64__ -def __SSE__ -def __MMX__ -def __SSE2__ -idir . -idir /usr/local/netcdf-3.6.2/include -idir /usr/include -idir /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/lib/include -idir /usr/local/mpi/mvapich-0.9.8-pgi/include -idir /usr/local/pgi/linux86-64/7.0-4/include -idir /tmp/usr4918/TER.01a.T31_gx3v5.B.glenn.131725/ocn/obj/source -idir /usr/local/mpi/mvapich-0.9.9-pgi-fixed-shmem/include/f90base -def LINUX -def PGF90 -def NO_SHR_VMATH -def LINUX -def POSIX -def impvmix -def coupled -def NPROC_X=4 -def NPROC_Y=6 -preprocess -nofreeform -i4 -extend -x 125 2 -vect 48 -x 124 0x8 -save -output /tmp/pgf90H8WdJeuM2hA6.ilm
gmake: *** [POP.o] Error 127

 

[mengpai@gmail_com]

Hi there,

The problem is due to the user limitation at our machine. I solved it by turning my resources to unlimited.

Thanks, anyway.

-Odden