error in building CESM

[devarajun@gmail_com]

Hi

I am compiling newer released version of CESM code on Regatta. Below is the content of
log file generated while compiling. To me it seems compiler problem.



mpxlf90_r -c -I. -I/usr/include -I/home/caos2/casgbala/ranjith/files_from_kart/netcdf_from_kart/inetcdf64/include -I/home/caos2/casgbala/ranjith/files_from_kart/netcdf_from_kart/inetcdf64/include -I. -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/scripts/I2000CN/SourceMods/src.share -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/drv/shr_mct -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/drv/shr -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/dead_share -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/csm_share/shr -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/utils/timing -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/utils/esmf_wrf_timemgr -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/scripts/scratch/I2000CN/lib/include -WF,-DMCT_INTERFACE -WF,-DHAVE_MPI -WF,-DAIX -WF,-DSEQ_ -WF,-DFORTRAN_SAME -q64 -g -qfullpath -qhot -qsuffix=f=f90:cpp=F90 -qmaxmem=-1 -qarch=auto -qsigtrap=xl__trcedump -O2 -qstrict -Q -qsuffix=f=f90:cpp=F90 /home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/csm_share/shr/shr_strdata_mod.F90
** shr_strdata_mod === End of Compilation 1 ===
1501-510 Compilation successful for file shr_strdata_mod.F90.
mpxlf90_r -c -I. -I/usr/include -I/home/caos2/casgbala/ranjith/files_from_kart/netcdf_from_kart/inetcdf64/include -I/home/caos2/casgbala/ranjith/files_from_kart/netcdf_from_kart/inetcdf64/include -I. -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/scripts/I2000CN/SourceMods/src.share -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/drv/shr_mct -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/drv/shr -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/dead_share -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/csm_share/shr -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/utils/timing -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/utils/esmf_wrf_timemgr -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/scripts/scratch/I2000CN/lib/include -WF,-DMCT_INTERFACE -WF,-DHAVE_MPI -WF,-DAIX -WF,-DSEQ_ -WF,-DFORTRAN_SAME -q64 -g -qfullpath -qhot -qsuffix=f=f90:cpp=F90 -qmaxmem=-1 -qarch=auto -qsigtrap=xl__trcedump -O2 -qstrict -Q -qsuffix=f=f90:cpp=F90 /home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/csm_share/shr/shr_scam_mod.F90
"/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/csm_share/shr/shr_scam_mod.F90", 1512-132 (U) An error occured while attempting to write to a temporary file in the directory /tmp/F8860182x4ibyc.
1501-511 Compilation failed for file shr_scam_mod.F90.
gmake: *** [shr_scam_mod.o] Error 1


I am grateful if any one help me out.

-Dev

 

[eaton]

I'm wondering whether this is possibly a system problem rather than a compiler problem. The message:

/shr/shr_scam_mod.F90", 1512-132 (U) An error occured while attempting to write to a temporary file in the directory /tmp/F8860182x4ibyc.

is the kind of thing you expect to see when the /tmp filesystem partition is full.

Is this error reproducible?

 

[devarajun@gmail_com]

Hi Eaton,

Thanks for the reply,Yes that's a system problem. Our system admin cleared the error and Now I am getting the following error while running. System admin says its a code problem, But I am not able to find out.
Below is the content of ccsm.log.110325-133624

:
15: proc= 15 beg atmcell = 5311 end atmcell = 5663 total atmcells per proc = 353
15: proc= 15 beg gridcell= 5311 end gridcell= 5663 total gridcells per proc= 353
15: proc= 15 beg landunit= 7510 end landunit= 8010 total landunits per proc= 501
15: proc= 15 beg column = 12778 end column = 13693 total columns per proc = 916
15: proc= 15 beg pft = 72378 end pft = 77245 total pfts per proc = 4868
15: proc= 15 atm ngseg = 639 atm nlseg = 39
15: proc= 15 lnd ngseg = 639 lnd nlseg = 39
15: proc= 15 gce ngseg = 320 gce nlseg = 20
15: proc= 15 lun ngseg = 320 lun nlseg = 20
15: proc= 15 col ngseg = 320 col nlseg = 20
15: proc= 15 pft ngseg = 320 pft nlseg = 20
15: proc= 15 nclumps = 1
15: proc= 15 clump no = 1 clump id= 16 beg gridcell= 5311 end gridcell= 5663 total gridcells per clump= 353
15: proc= 15 clump no = 1 clump id= 16 beg landunit= 7510 end landunit= 8010 total landunits per clump = 501
14: proc= 14 clump no = 1 clump id= 15 beg column = 11887 end column = 12777 total columns per clump = 891
14: proc= 14 clump no = 1 clump id= 15 beg pft = 67247 end pft = 72377 total pfts per clump = 5131
15: proc= 15 clump no = 1 clump id= 16 beg column = 12778 end column = 13693 total columns per clump = 916
15: proc= 15 clump no = 1 clump id= 16 beg pft = 72378 end pft = 77245 total pfts per clump = 4868
1: rtm decomp info proc = 1 begr = 7107 endr = 13955 numr = 6849
1: proc = 1 begrl= 6726 endrl= 13003 numrl= 6278
1: proc = 1 begro= 382 endro= 952 numro= 571
7: rtm decomp info proc = 7 begr = 48412 endr = 55247 numr = 6836
7: proc = 7 begrl= 41000 endrl= 45709 numrl= 4710
7: proc = 7 begro= 7413 endro= 9538 numro= 2126
14: rtm decomp info proc = 14 begr = 96788 endr = 103660 numr = 6873
14: proc = 14 begrl= 75038 endrl= 80055 numrl= 5018
14: proc = 14 begro= 21751 endro= 23605 numro= 1855
15: rtm decomp info proc = 15 begr = 103661 endr = 109217 numr = 5557
15: proc = 15 begrl= 80056 endrl= 84511 numrl= 4456
15: proc = 15 begro= 23606 endro= 24706 numro= 1101
INFO: 0031-656 I/O file STDOUT closed by task 9
INFO: 0031-656 I/O file STDERR closed by task 9
ERROR: 0031-250 task 1: Segmentation fault
ERROR: 0031-250 task 2: Segmentation fault
ERROR: 0031-250 task 4: Segmentation fault
ERROR: 0031-250 task 8: Segmentation fault
ERROR: 0031-250 task 9: Segmentation fault
ERROR: 0031-250 task 12: Segmentation fault
INFO: 0031-656 I/O file STDOUT closed by task 3
INFO: 0031-656 I/O file STDERR closed by task 3
INFO: 0031-656 I/O file STDOUT closed by task 6
INFO: 0031-656 I/O file STDERR closed by task 6
INFO: 0031-656 I/O file STDOUT closed by task 7
INFO: 0031-656 I/O file STDERR closed by task 7
INFO: 0031-656 I/O file STDOUT closed by task 11
INFO: 0031-656 I/O file STDERR closed by task 11
INFO: 0031-656 I/O file STDOUT closed by task 13
INFO: 0031-656 I/O file STDERR closed by task 13
INFO: 0031-656 I/O file STDOUT closed by task 14
INFO: 0031-656 I/O file STDERR closed by task 14
ERROR: 0031-250 task 7: Segmentation fault
ERROR: 0031-250 task 13: Segmentation fault
ERROR: 0031-250 task 6: Segmentation fault
ERROR: 0031-250 task 11: Segmentation fault
ERROR: 0031-250 task 3: Segmentation fault
ERROR: 0031-250 task 14: Segmentation fault
INFO: 0031-656 I/O file STDOUT closed by task 5
INFO: 0031-656 I/O file STDERR closed by task 5
ERROR: 0031-250 task 5: Segmentation fault
INFO: 0031-656 I/O file STDOUT closed by task 15
INFO: 0031-656 I/O file STDERR closed by task 15
ERROR: 0031-250 task 15: Segmentation fault
INFO: 0031-656 I/O file STDOUT closed by task 10
INFO: 0031-656 I/O file STDERR closed by task 10
ERROR: 0031-250 task 10: Segmentation fault
INFO: 0031-656 I/O file STDOUT closed by task 0
INFO: 0031-656 I/O file STDERR closed by task 0
ERROR: 0031-250 task 0: Segmentation fault
INFO: 0031-639 Exit status from pm_respond = 0



Looking for your help.

Thanks
-Dev

 

[eaton]

I don't see any indication of what the problem might be from the output you've provided.

In your original post I notice that the compiler optimization flag '-qhot' has been added. I would recommend against using this option. In our testing at NCAR we have run into many problems over the years trying to use this aggressive optimization, so we no longer try to use it.

Another place to look for potential problems is in the scripts/ccsm_utils/Machines/env_machopts.generic_ibm file (I assume you're using the generic_ibm machine though you didn't specify that). This file contains alot of environment variables that are used on our ibm power6 at NCAR, but may not be appropriate at other ibm installations. I'd recommend reviewing these settings with your system admin to be sure they are appropriate.

 

[devarajun@gmail_com]

hi Eaton,
As you suggested we have edited according to our IBM regatta: scripts/ccsm_utils/Machines/env_machopts.generic_ibm file and we removed qhot at Macro's file. I am getting the error below. I will be grateful if you can help me out this problem.

Error:

ct -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/drv/shr -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/dead_share -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/csm_share/shr -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/utils/timing -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/utils/esmf_wrf_timemgr -I/home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/scripts/scratch/CN/lib/include -WF,-DMCT_INTERFACE -WF,-DHAVE_MPI -WF,-DAIX -WF,-DSEQ_ -WF,-DFORTRAN_SAME -q64 -g -qfullpath -qsuffix=f=f90:cpp=F90 -qmaxmem=-1 -qarch=auto -qsigtrap=xl__trcedump -O2 -qstrict -Q -qsuffix=f=f90:cpp=F90 /home/caos2/casgbala/dev/CESM102/CLM4_spinup_14march11/cesm1_0_2/models/csm_share/shr/mct_mod.F90
xlf90_r: 1501-230 Internal compiler error; please contact your Service Representative
1501-511 Compilation failed for file mct_mod.F90.
gmake: *** [mct_mod.o] Error 40

 

[eaton]

This appears to be another system related problem. It's possible that your system is still having problems with /tmp filling up. I'd check that first. Usually there is a way to specify the directory that the compiler writes its temporary files to. You may be able to do that to get around a problem with /tmp being too small.