forrtl: severe (174): SIGSEGV, segmentation fault occurred

[Amit K Sharma]

Hello everyone,
I am trying to run some simulations by changing the optical properties (refractive indices) of anthropogenic aerosols with AMIP compset using CAM 5.3 (cesm 1.2.2), with intel compiler and impi.
When I am implementing the changes in all anthropogenic aerosols, the model crashes with following error:

”””
ANALYSES_INT: Read analyses for date (yyyymmdd) 20000103 sec 0
QNEG3 from vertical diffusion/SO2:m= 8 lat/lchnk= 1212 Min. mixing ratio violated at 1 points. Reset to 1.0E-36 Worst =-1.5E-12 at i,k= 6 30
QNEG3 from vertical diffusion/SO2:m= 8 lat/lchnk= 1833 Min. mixing ratio violated at 2 points. Reset to 1.0E-36 Worst =-1.1E-12 at i,k= 4 30
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
cesm.exe 0000000002458C81 Unknown Unknown Unknown
cesm.exe 00000000024573D7 Unknown Unknown Unknown
cesm.exe 00000000023F1F84 Unknown Unknown Unknown
cesm.exe 00000000023F1D96 Unknown Unknown Unknown
cesm.exe 000000000238186F Unknown Unknown Unknown
cesm.exe 000000000238B02D Unknown Unknown Unknown
libpthread.so.0 00002B70C18615D0 Unknown Unknown Unknown
cesm.exe 000000000175011E tp_core_mp_xtpv_ 469 tp_core.F90
cesm.exe 000000000174B8DE tp_core_mp_tp2d_ 231 tp_core.F90
cesm.exe 000000000174A03D tp_core_mp_tp2c_ 119 tp_core.F90
cesm.exe 0000000001731681 sw_core_mp_c_sw_ 309 sw_core.F90
cesm.exe 0000000001632714 cd_core_ 732 cd_core.F90
cesm.exe 00000000013BB0C8 dyn_comp_mp_dyn_r 1822 dyn_comp.F90
cesm.exe 000000000115F5D2 stepon_mp_stepon_ 422 stepon.F90
cesm.exe 0000000000507B6A cam_comp_mp_cam_r 262 cam_comp.F90
cesm.exe 00000000004FD1F5 atm_comp_mct_mp_a 561 atm_comp_mct.F90
cesm.exe 000000000045569F ccsm_comp_mod_mp_ 4079 ccsm_comp_mod.F90
cesm.exe 000000000045E75E MAIN__ 91 ccsm_driver.F90
cesm.exe 0000000000414DCE Unknown Unknown Unknown
libc.so.6 00002B70C1D92495 Unknown Unknown Unknown
cesm.exe 0000000000414CD9 Unknown Unknown Unknown
”””

Whereas if the change in refractive index is implemented for single anthropogenic aerosol, the model runs for 4 months but crashes after that with the error stated below:

”’’’
ANALYSES_INT: Read analyses for date (yyyymmdd) 20000607 sec 64800
memory_write: model date = 20000607 0 memory = 106.24 MB (highwater) -0.00 MB (usage) (pe= 520 comps= OCN ICE GLC ROF WAV)
memory_write: model date = 20000607 0 memory = 109.03 MB (highwater) -0.00 MB (usage) (pe= 400 comps= LND)
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
cesm.exe 0000000002458C81 Unknown Unknown Unknown
cesm.exe 00000000024573D7 Unknown Unknown Unknown
cesm.exe 00000000023F1F84 Unknown Unknown Unknown
cesm.exe 00000000023F1D96 Unknown Unknown Unknown
cesm.exe 000000000238186F Unknown Unknown Unknown
cesm.exe 000000000238AFFB Unknown Unknown Unknown
libpthread.so.0 00002B3BFD4055D0 Unknown Unknown Unknown
cesm.exe 000000000110AD4C rrtmg_sw_spcvmc_m 609 rrtmg_sw_spcvmc.f90
cesm.exe 00000000010F63CB rrtmg_sw_rad_mp_r 696 rrtmg_sw_rad.f90
cesm.exe 00000000009212DD radsw_mp_rad_rrtm 531 radsw.F90
cesm.exe 00000000008FEB2E radiation_mp_radi 1013 radiation.F90
cesm.exe 00000000008A3AEC physpkg_mp_tphysb 2178 physpkg.F90
cesm.exe 00000000008945E2 physpkg_mp_phys_r 954 physpkg.F90
cesm.exe 000000000050816A cam_comp_mp_cam_r 272 cam_comp.F90
cesm.exe 00000000004FD1F5 atm_comp_mct_mp_a 561 atm_comp_mct.F90
cesm.exe 000000000045569F ccsm_comp_mod_mp_ 4079 ccsm_comp_mod.F90
cesm.exe 000000000045E75E MAIN__ 91 ccsm_driver.F90
cesm.exe 0000000000414DCE Unknown Unknown Unknown
libc.so.6 00002B3BFD936495 Unknown Unknown Unknown
cesm.exe 0000000000414CD9 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
”””

I am attaching the log file from simulation.
Any hints would be helpful!
Thanks in advance!


-Amit Kumar Sharma

 

[sacks]

At this time, CESM1.2 and iCESM1.2 are only supported for deep time paleoclimate and water isotope applications, respectively. Are you using the model for one of those purposes? If so, I will direct this to the appropriate forum. If not, I apologize that we are no longer able to provide support for CESM1.2; we encourage you to update to the CESM2 series. See CESM Support Policy for support details.

 

[Amit K Sharma]

At this time, CESM1.2 and iCESM1.2 are only supported for deep time paleoclimate and water isotope applications, respectively. Are you using the model for one of those purposes? If so, I will direct this to the appropriate forum. If not, I apologize that we are no longer able to provide support for CESM1.2; we encourage you to update to the CESM2 series. See CESM Support Policy for support details.

Thank You sir for the reply.
No, I am not using these simulations for deep time paleoclimate and water isotope applications. So I will try these simulations with CESM2 series.

Thanks
-Amit