CESM 1.2.0 termination EXIT CODE:139

I update netcdf version from 4.3.3.1 to 4.4.1.1, netcdf-fortran from 4.4.2 to 4.4.4, build step is success but when run the model, it crashes. I have no idea why the process is terminated. The error shows as follows: (seq_comm_setcomm)  initialize ID (  1 GLOBAL          ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)(seq_comm_setcomm)  initialize ID (  2 CPL             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)(seq_comm_setcomm)  initialize ID ( 17 ATM             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 18 CPLATM          ) join IDs =     2    17       ( npes =     8) ( nthreads =  1)(seq_comm_jcommarr) initialize ID (  3 ALLATMID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 10 CPLALLATMID     ) join IDs =     2     3       ( npes =     8) ( nthreads =  1)(seq_comm_setcomm)  initialize ID ( 19 LND             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 20 CPLLND          ) join IDs =     2    19       ( npes =     8) ( nthreads =  1)(seq_comm_jcommarr) initialize ID (  4 ALLLNDID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 11 CPLALLLNDID     ) join IDs =     2     4       ( npes =     8) ( nthreads =  1)(seq_comm_setcomm)  initialize ID ( 21 OCN             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 22 CPLOCN          ) join IDs =     2    21       ( npes =     8) ( nthreads =  1)(seq_comm_jcommarr) initialize ID (  5 ALLOCNID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 12 CPLALLOCNID     ) join IDs =     2     5       ( npes =     8) ( nthreads =  1)(seq_comm_setcomm)  initialize ID ( 23 ICE             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 24 CPLICE          ) join IDs =     2    23       ( npes =     8) ( nthreads =  1)(seq_comm_jcommarr) initialize ID (  6 ALLICEID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 13 CPLALLICEID     ) join IDs =     2     6       ( npes =     8) ( nthreads =  1)(seq_comm_setcomm)  initialize ID ( 25 GLC             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 26 CPLGLC          ) join IDs =     2    25       ( npes =     8) ( nthreads =  1)(seq_comm_jcommarr) initialize ID (  7 ALLGLCID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 14 CPLALLGLCID     ) join IDs =     2     7       ( npes =     8) ( nthreads =  1)(seq_comm_setcomm)  initialize ID ( 27 ROF             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 28 CPLROF          ) join IDs =     2    27       ( npes =     8) ( nthreads =  1)(seq_comm_jcommarr) initialize ID (  8 ALLROFID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 15 CPLALLROFID     ) join IDs =     2     8       ( npes =     8) ( nthreads =  1)(seq_comm_setcomm)  initialize ID ( 29 WAV             ) pelist   =     0     7     1 ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 30 CPLWAV          ) join IDs =     2    29       ( npes =     8) ( nthreads =  1)(seq_comm_jcommarr) initialize ID (  9 ALLWAVID        ) join multiple comp IDs       ( npes =     8) ( nthreads =  1)(seq_comm_joincomm) initialize ID ( 16 CPLALLWAVID     ) join IDs =     2     9       ( npes =     8) ( nthreads =  1)(seq_comm_printcomms)     1     0     8     1  GLOBAL:(seq_comm_printcomms)     2     0     8     1  CPL:(seq_comm_printcomms)     3     0     8     1  ALLATMID:(seq_comm_printcomms)     4     0     8     1  ALLLNDID:(seq_comm_printcomms)     5     0     8     1  ALLOCNID:(seq_comm_printcomms)     6     0     8     1  ALLICEID:(seq_comm_printcomms)     7     0     8     1  ALLGLCID:(seq_comm_printcomms)     8     0     8     1  ALLROFID:(seq_comm_printcomms)     9     0     8     1  ALLWAVID:(seq_comm_printcomms)    10     0     8     1  CPLALLATMID:(seq_comm_printcomms)    11     0     8     1  CPLALLLNDID:(seq_comm_printcomms)    12     0     8     1  CPLALLOCNID:(seq_comm_printcomms)    13     0     8     1  CPLALLICEID:(seq_comm_printcomms)    14     0     8     1  CPLALLGLCID:(seq_comm_printcomms)    15     0     8     1  CPLALLROFID:(seq_comm_printcomms)    16     0     8     1  CPLALLWAVID:(seq_comm_printcomms)    17     0     8     1  ATM:(seq_comm_printcomms)    18     0     8     1  CPLATM:(seq_comm_printcomms)    19     0     8     1  LND:(seq_comm_printcomms)    20     0     8     1  CPLLND:(seq_comm_printcomms)    21     0     8     1  OCN:(seq_comm_printcomms)    22     0     8     1  CPLOCN:(seq_comm_printcomms)    23     0     8     1  ICE:(seq_comm_printcomms)    24     0     8     1  CPLICE:(seq_comm_printcomms)    25     0     8     1  GLC:(seq_comm_printcomms)    26     0     8     1  CPLGLC:(seq_comm_printcomms)    27     0     8     1  ROF:(seq_comm_printcomms)    28     0     8     1  CPLROF:(seq_comm_printcomms)    29     0     8     1  WAV:(seq_comm_printcomms)    30     0     8     1  CPLWAV: (t_initf) Read in prof_inparm namelist from: drv_in8 MB memory   alloc in MB is             8.008 MB memory dealloc in MB is             0.00Memory block size conversion in bytes is          1022.00seq_flds_mod: read seq_cplflds_inparm namelist from: drv_in drv_in8 MB memory   alloc in MB is             8.008 MB memory dealloc in MB is             0.00Memory block size conversion in bytes is          1022.008 MB memory   alloc in MB is             8.008 MB memory   alloc in MB is             8.00seq_flds_mod: read seq_cplflds_userspec namelist from: drv_in8 MB memory dealloc in MB is             0.00Memory block size conversion in bytes is          1019.52 8 MB memory dealloc in MB is             0.008 MB memory   alloc in MB is             8.00Memory block size conversion in bytes is          1021.518 MB memory   alloc in MB is             8.008 MB memory dealloc in MB is             0.00 8 MB memory dealloc in MB is             0.00Memory block size conversion in bytes is          1020.51Memory block size conversion in bytes is          1022.008 MB memory   alloc in MB is             8.008 MB memory dealloc in MB is             0.00Memory block size conversion in bytes is          1019.52seq_flds_mod: seq_flds_a2x_states= Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Sa_dens:Sa_pslv:Sa_co2prog:Sa_co2diagseq_flds_mod: seq_flds_a2x_fluxes= Faxa_rainc:Faxa_rainl:Faxa_snowc:Faxa_snowl:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_swnet:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4seq_flds_mod: seq_flds_x2a_states= Sf_lfrac:Sf_ifrac:Sf_ofrac:Sx_avsdr:Sx_anidr:Sx_avsdf:Sx_anidf:Sx_tref:Sx_qref:So_t:Sx_t:Sl_fv:Sl_ram1:Sl_snowh:Si_snowh:So_ssq:So_re:Sx_u10:So_ustarseq_flds_mod: seq_flds_x2a_fluxes= Faxx_taux:Faxx_tauy:Faxx_lat:Faxx_sen:Faxx_lwup:Faxx_evap:Fall_flxdst1:Fall_flxdst2:Fall_flxdst3:Fall_flxdst4seq_flds_mod: seq_flds_l2x_states= Sl_avsdr:Sl_anidr:Sl_avsdf:Sl_anidf:Sl_tref:Sl_qref:Sl_t:Sl_fv:Sl_ram1:Sl_snowh:Sl_u10seq_flds_mod: seq_flds_l2x_fluxes= Fall_swnet:Fall_taux:Fall_tauy:Fall_lat:Fall_sen:Fall_lwup:Fall_evap:Fall_flxdst1:Fall_flxdst2:Fall_flxdst3:Fall_flxdst4:Flrl_rofliq:Flrl_roficeseq_flds_mod: seq_flds_x2l_states= Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Slrr_volr:Sa_co2prog:Sa_co2diagseq_flds_mod: seq_flds_x2l_fluxes= Faxa_rainc:Faxa_rainl:Faxa_snowc:Faxa_snowl:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Flrr_floodseq_flds_mod: seq_flds_i2x_states= Si_avsdr:Si_anidr:Si_avsdf:Si_anidf:Si_tref:Si_qref:Si_t:Si_snowh:Si_u10:Si_ifracseq_flds_mod: seq_flds_i2x_fluxes= Faii_swnet:Fioi_swpen:Faii_taux:Fioi_taux:Faii_tauy:Fioi_tauy:Faii_lat:Faii_sen:Faii_lwup:Faii_evap:Fioi_melth:Fioi_meltw:Fioi_saltseq_flds_mod: seq_flds_x2i_states= Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_pbot:Sa_dens:So_t:So_s:So_u:So_v:So_dhdx:So_dhdyseq_flds_mod: seq_flds_x2i_fluxes= Faxa_rain:Faxa_snow:Faxa_lwdn:Faxa_swndr:Faxa_swvdr:Faxa_swndf:Faxa_swvdf:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Fioo_qseq_flds_mod: seq_flds_o2x_states= So_t:So_s:So_u:So_v:So_dhdx:So_dhdy:So_bldepthseq_flds_mod: seq_flds_o2x_fluxes= Fioo_qseq_flds_mod: seq_flds_x2o_states= Sa_pslv:So_duu10n:Si_ifrac:Sw_lamult:Sw_ustokes:Sw_vstokes:Sw_hstokesseq_flds_mod: seq_flds_x2o_fluxes= Faxa_rain:Faxa_snow:Faxa_prec:Faxa_lwdn:Foxx_swnet:Faxa_bcphidry:Faxa_bcphodry:Faxa_bcphiwet:Faxa_ocphidry:Faxa_ocphodry:Faxa_ocphiwet:Faxa_dstwet1:Faxa_dstwet2:Faxa_dstwet3:Faxa_dstwet4:Faxa_dstdry1:Faxa_dstdry2:Faxa_dstdry3:Faxa_dstdry4:Foxx_taux:Foxx_tauy:Foxx_lat:Foxx_sen:Foxx_lwup:Foxx_evap:Fioi_melth:Fioi_meltw:Fioi_salt:Forr_roff:Forr_ioffseq_flds_mod: seq_flds_s2x_states=  seq_flds_mod: seq_flds_s2x_fluxes=  seq_flds_mod: seq_flds_x2s_states=  seq_flds_mod: seq_flds_x2s_fluxes=  seq_flds_mod: seq_flds_g2x_states=  seq_flds_mod: seq_flds_g2x_fluxes=  seq_flds_mod: seq_flds_x2g_states=  seq_flds_mod: seq_flds_x2g_fluxes=  seq_flds_mod: seq_flds_xao_states= So_tref:So_qref:So_ssq:So_re:So_u10:So_duu10n:So_ustarseq_flds_mod: seq_flds_xao_albedo= So_avsdr:So_anidr:So_avsdf:So_anidfseq_flds_mod: seq_flds_r2x_states= Slrr_volrseq_flds_mod: seq_flds_r2x_fluxes= Forr_roff:Forr_ioff:Flrr_floodseq_flds_mod: seq_flds_x2r_states=  seq_flds_mod: seq_flds_x2r_fluxes= Flrl_rofliq:Flrl_roficeseq_flds_mod: seq_flds_w2x_states= Sw_lamult:Sw_ustokes:Sw_vstokes:Sw_hstokesseq_flds_mod: seq_flds_w2x_fluxes=  seq_flds_mod: seq_flds_x2w_states= Sa_u:Sa_v:Sa_tbot:Si_ifrac:So_t:So_u:So_v:So_bldepthseq_flds_mod: seq_flds_x2w_fluxes=              8 pes participating in computation ----------------------------------- TASK#  NAME  0  node4  1  node4  2  node4  3  node4  4  node4  5  node4  6  node4  7  node4 ====================================================================================   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES=   PID 9237 RUNNING AT node4=   EXIT CODE: 139=   CLEANING UP REMAINING PROCESSES=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES===================================================================================YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)This typically refers to a problem with your application.Please see the FAQ page for debugging suggestions Layna 
 

jedwards

CSEG and Liaisons
Staff member
There is no error apparent in this log - check the component logs and the output from your batch system for the error.    Try compiling with DEBUG=TRUE and running again.
 
Back
Top