forrtl: error (78): process killed (SIGTERM)

[hannay]

Yes, you can also changed the initial condition in a start up run. For a F case, you would only need to change land and atm initial condition.finidat is the land initial condition
ncdata is the atmosphere initial conditionWhat is important is to change the land initai condition as it takes decades to spin up. The atmosphere spinups so quickly that it doesn't really matter if you change the initial condition. So you NEED to change the land initial condition with finidat. finidat = '/glade/p/cesm/amwg/hannay/inputdata/FAMIPC5_ne120_79to05_03_omp2/rest/2000-01-01-00000/FAMIPC5_ne120_79to05_03_omp2.clm2.r.2000-01-01-00000.nc'If you want you COULD also change the atmosphere initial condition use with ncdata (but it is not necessary)

ncdata = '/glade/p/cesm/amwg/hannay/inputdata/FAMIPC5_ne120_79to05_03_omp2/rest/2000-01-01-00000/FAMIPC5_ne120_79to05_03_omp2.cam.i.2000-01-01-00000.nc'

 

[yingli@atmos_colostate_edu]

Thanks for your clear explanation.  I plan to run a 25 ensembles with each run from 1 Nov through 31 Mar. So it would be great to have the land initial file at date Oct-31. If there is no such initial files from the existing run. I guess I need to do my own run to get initial conditions. Below is my understanding, please correct me if I miss anything. 1. ran from Jan-01 to Oct-31, and save the restart files for that day. 2. change the output frequency of the initial files in user_nl_cam to get the various atmosphere initial conditions 

 

[hannay]

Yes, this is the way to process. Indeed you can save the land initial condition for Nov 1. To get the initial condition for the atmosphere for Nov 1, set the namelist variable   

 

[amostai@cuhk_edu_hk]

Dear Software Engineer,I had a similar problem with "forrtl: error (78): process killed (SIGTERM)", but it seems like the causes are different, and I'd like to see if you may kindly help. I was running CESM1.2.2 with FMOZ (CAM4 + CLM4.0-CN-nofire). There were issues with Intel 14, so I followed advices in the forum to change "spmd_dyn.F90" and "ice_itd.F90", which worked (thanks!). I also set my "coredumpsize" and "stacksize" to unlimited just in case. Then the model crashed on the second simulation day, without any obvious error messages. The end of "cesm.log" looks something like this:... QNEG3 from vertical diffusion/SO2:m= 86 lat/lchnk=    237 Min. mixing ratio violated at    3 points.  Reset to  1.0E-36 Worst =-4.7E-12 at i,k=   5 26 QNEG3 from vertical diffusion/NH4NO3:m= 91 lat/lchnk=    328 Min. mixing ratio violated at  144 points.  Reset to  1.0E-36 Worst =-1.9E-12 at i,k=  14 26 QNEG3 from vertical diffusion/SO2:m= 86 lat/lchnk=    504 Min. mixing ratio violated at    2 points.  Reset to  1.0E-36 Worst =-3.0E-12 at i,k=   5 26forrtl: error (78): process killed (SIGTERM)Image              PC                Routine            Line        Source           Stack trace terminated abnormally.forrtl: error (78): process killed (SIGTERM)Image              PC                Routine            Line        SourceStack trace terminated abnormally.          ... (there are 10+ more sets of lines like the last three lines) I've attached the other logs, user_nl_* and some env_*.xml files. There aren't any error messages in other logs either. I was using one node with 20 cores. I have modified the CAM and CLM namelists quite a bit and didn't expect them to cause problem, but please check. Your insights will be highly appreciated! 

 

[santos]

Have you tried running a case with debugging options enabled? I.e. do this before building:xmlchange DEBUG=TRUEAre there any core files in your run directory?

 

[amostai@cuhk_edu_hk]

The DEBUG has always been turned on, and there is a core file of 254M in the run directory. Anyhow, I tried to run again and finally there were some error messages coming out at the end of the "cesm.log", which I'm attaching again. The halt seems to arise from multiple places in the model... I also looked at the standard error log (#PBS -e cesm.err), which says "PBS: job killed: walltime 3566 exceeded limit 3540"... I'm not sure what it means or whether it's related to the errors as indicated by "cesm.log". Your help is much appreciated!

 

[jedwards]

It's difficult to tell if the model is stopping because you've run out of time or the model is hanging until you run out of time in the queue.    What are the timestamps on the cam.log file and the cesm.log file?   

 

[amostai@cuhk_edu_hk]

Where do I look for the timestamps? I've attached the cesm.log and atm.log again FYI. Do you think it has anything to do with my PES setting? I've pretty much set all NTASKS* = TOTALPES = MAX_TASKS_PER_NODE = 20, coz each of my nodes only have 20 cores (nodes=1:ppn=20).

 

[jedwards]

timestamps of the files:  the date and time returned when you run the unix command 'ls -l' I suspect that you are also running out of memory - try using more tasks

 

[amostai@cuhk_edu_hk]

Thanks! It's now fixed. It turns out to be a problem with the walltime limit setting. I've set it to 59 min, so the run just halted after that. I relaxed the time limit and the run went fine. Thanks!

 

[chinuiitkgp@gmail_com]

Hi,I am trying to install CESM_1_2_2 in Stampede and got error massage after invoking ./cesm_setup. 1) ./create_newcase -case /scratch/03778/tg830775/cesm_Releases/CJ_CESM_TEST -compset F_2000_STRATMAM7_CN -res f09_g16 -mach stampede2)./xmlchange -file env_build.xml -id CAM_CONFIG_OPTS -val '-phys cam5 -chem trop_strat_mam7 -usr_mech_infile strat_cb05ge_newmam7.in'3)./xmlchange -file env_run.xml -id RUN_STARTDATE -val '2007-01-01'4)./xmlchange -file env_run.xml -id CLM_BLDNML_OPTS -val '-megan'5)./xmlchange -file env_build.xml -id CALENDAR -val 'GREGORIAN' I got these error messages after invoking ./cesm_setup Macros script already created ...skipping Creating batch script CJ_CESM_TEST.run**These are the modules loaded before compiler and mpi are selected**Currently Loaded Modules:  1) intel/15.0.2   2) mvapich2/2.1   3) xalt/0.6   4) TACC   5) perl/5.16.2   6) cmake/3.1.0Buidling for Xeon HostInactive Modules:  1) pnetcdf/1.6.0Activating Modules:  1) pnetcdf/1.6.0Currently Loaded Modules:  1) xalt/0.6   3) perl/5.16.2   5) intel/15.0.2   7) pnetcdf/1.6.0       9) netcdf/4.3.3.1 (m)  2) TACC       4) cmake/3.1.0   6) impi/5.0.2     8) hdf5/1.8.16   (m)  Where:   m:  built for host and native MICLocking file env_mach_pes.xmlRunning preview_namelist script**These are the modules loaded before compiler and mpi are selected**Currently Loaded Modules:  1) intel/15.0.2   2) mvapich2/2.1   3) xalt/0.6   4) TACC   5) perl/5.16.2   6) cmake/3.1.0Buidling for Xeon HostInactive Modules:  1) pnetcdf/1.6.0Activating Modules:  1) pnetcdf/1.6.0Currently Loaded Modules:  1) xalt/0.6   3) perl/5.16.2   5) intel/15.0.2   7) pnetcdf/1.6.0       9) netcdf/4.3.3.1 (m)  2) TACC       4) cmake/3.1.0   6) impi/5.0.2     8) hdf5/1.8.16   (m)  Where:   m:  built for host and native MIC**These are the modules loaded before compiler and mpi are selected**Currently Loaded Modules:  1) intel/15.0.2   2) mvapich2/2.1   3) xalt/0.6   4) TACC   5) perl/5.16.2   6) cmake/3.1.0Buidling for Xeon HostInactive Modules:  1) pnetcdf/1.6.0Activating Modules:  1) pnetcdf/1.6.0Currently Loaded Modules:  1) xalt/0.6   3) perl/5.16.2   5) intel/15.0.2   7) pnetcdf/1.6.0       9) netcdf/4.3.3.1 (m)  2) TACC       4) cmake/3.1.0   6) impi/5.0.2     8) hdf5/1.8.16   (m)  Where:   m:  built for host and native MICUse of qw(...) as parentheses is deprecated at /scratch/03778/tg830775/cesm_Releases/cesm1_2_2_new/models/drv/bld/build-namelist line 784. infile is /scratch/03778/tg830775/cesm_Releases/CJ_CESM_TEST/Buildconf/cplconf/cesm_namelist *** Chem preprocessor FAILED. See: /scratch/03778/tg830775/cesm_Releases/CJ_CESM_TEST/Buildconf/camconf/chem_proc/chem_mech.docERROR: cam.buildnml.csh failedERROR: /scratch/03778/tg830775/cesm_Releases/CJ_CESM_TEST/preview_namelists failed: 25344  Is there any advice on it?  Thanks,Chinmay 

 

[chinuiitkgp@gmail_com]

Dear CESM Software engineer,I am trying to install CESM_1_2_2 in Stampede and got error messages. ERROR:/scratch/03778/tg830775/cesm_Releases/cesm1_2_2_new/models/utils/pio/calcdisplace_mod.F90(56): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [ALLOC_MOD]/scratch/03778/tg830775/CJ_CESM_TEST1/bld/intel/impi/nodebug/nothreads/pio/rearrange.F90(18): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [BOX_REARRANGE]/scratch/03778/tg830775/cesm_Releases/cesm1_2_2_new/models/utils/pio/ionf_mod.F90(6): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [ALLOC_MOD]  use box_rearrange    use alloc_mod, only : alloc_check,dealloc_check------^--------^  use alloc_mod------^compilation aborted for /scratch/03778/tg830775/CJ_CESM_TEST1/bld/intel/impi/nodebug/nothreads/pio/rearrange.F90 (code 1)gmake[3]: *** [CMakeFiles/pio.dir/rearrange.F90.o] Error 1gmake[3]: Leaving directory `/scratch/03778/tg830775/CJ_CESM_TEST1/bld/intel/impi/nodebug/nothreads/pio'gmake[2]: *** [CMakeFiles/pio.dir/rearrange.F90.o.provides] Error 2gmake[2]: *** Waiting for unfinished jobs..../scratch/03778/tg830775/cesm_Releases/cesm1_2_2_new/models/utils/pio/calcdisplace_mod.F90(164): error #6406: Conflicting attributes or multiple declaration of name.   [DEALLOC_CHECK]          call dealloc_check(displace)---------------^/scratch/03778/tg830775/cesm_Releases/cesm1_2_2_new/models/utils/pio/calcdisplace_mod.F90(165): error #6406: Conflicting attributes or multiple declaration of name.   [ALLOC_CHECK]          call alloc_check(displace,0)---------------^/scratch/03778/tg830775/cesm_Releases/cesm1_2_2_new/models/utils/pio/calcdisplace_mod.F90(56): error #6580: Name in only-list does not exist.   [ALLOC_CHECK]    use alloc_mod, only : alloc_check,dealloc_check--------------------------^  use alloc_mod, only : alloc_check,dealloc_check/scratch/03778/tg830775/cesm_Releases/cesm1_2_2_new/models/utils/pio/nf_mod.F90(15): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [ALLOC_MOD]--------------------------------------^  use alloc_mod------^compilation aborted for /scratch/03778/tg830775/cesm_Releases/cesm1_2_2_new/models/utils/pio/calcdisplace_mod.F90 (code 1)gmake[3]: *** [CMakeFiles/pio.dir/calcdisplace_mod.F90.o] Error 1gmake[3]: Leaving directory `/scratch/03778/tg830775/CJ_CESM_TEST1/bld/intel/impi/nodebug/nothreads/pio'gmake[2]: *** [CMakeFiles/pio.dir/calcdisplace_mod.F90.o.provides] Error 2/scratch/03778/tg830775/CJ_CESM_TEST1/bld/intel/impi/nodebug/nothreads/pio/iompi_mod.F90(22): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [ALLOC_MOD]  use alloc_mod, only : alloc_check------^/scratch/03778/tg830775/CJ_CESM_TEST1/bld/intel/impi/nodebug/nothreads/pio/iompi_mod.F90(22): error #6580: Name in only-list does not exist.   [ALLOC_CHECK]  use alloc_mod, only : alloc_check------------------------^compilation aborted for /scratch/03778/tg830775/cesm_Releases/cesm1_2_2_new/models/utils/pio/ionf_mod.F90 (code 1)compilation aborted for /scratch/03778/tg830775/cesm_Releases/cesm1_2_2_new/models/utils/pio/nf_mod.F90 (code 1)gmake[3]: *** [CMakeFiles/pio.dir/ionf_mod.F90.o] Error 1gmake[3]: Leaving directory `/scratch/03778/tg830775/CJ_CESM_TEST1/bld/intel/impi/nodebug/nothreads/pio'gmake[2]: *** [CMakeFiles/pio.dir/ionf_mod.F90.o.provides] Error 2gmake[3]: *** [CMakeFiles/pio.dir/nf_mod.F90.o] Error 1gmake[3]: Leaving directory `/scratch/03778/tg830775/CJ_CESM_TEST1/bld/intel/impi/nodebug/nothreads/pio'compilation aborted for /scratch/03778/tg830775/CJ_CESM_TEST1/bld/intel/impi/nodebug/nothreads/pio/iompi_mod.F90 (code 1)gmake[2]: *** [CMakeFiles/pio.dir/nf_mod.F90.o.provides] Error 2gmake[3]: *** [CMakeFiles/pio.dir/iompi_mod.F90.o] Error 1gmake[3]: Leaving directory `/scratch/03778/tg830775/CJ_CESM_TEST1/bld/intel/impi/nodebug/nothreads/pio'gmake[2]: *** [CMakeFiles/pio.dir/iompi_mod.F90.o.provides] Error 2gmake[2]: Leaving directory `/scratch/03778/tg830775/CJ_CESM_TEST1/bld/intel/impi/nodebug/nothreads/pio'gmake[1]: *** [CMakeFiles/pio.dir/all] Error 2gmake[1]: Leaving directory `/scratch/03778/tg830775/CJ_CESM_TEST1/bld/intel/impi/nodebug/nothreads/pio'gmake: *** [all] Error 2========================================================================================================================== Could you please give me some suggestions?