Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)

[jizh]

Hi all,
I am running CESM 2.1.5 with the F2000climo compset. The compiler is Intel 2021. After running for a few days, the model stops, and the following message appears in the cesm.log file.

[b3203r3n6:4519 :0:4519] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n7:119552:0:119552] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n8:48120:0:48120] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r4n1:10843:0:10843] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n4:128749:0:128749] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n7:119536:0:119536] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b2102r4n4:38143:0:38143] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b2102r4n4:38127:0:38127] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n7:119544:0:119544] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b2102r4n5:98690:0:98690] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b2102r4n4:38135:0:38135] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n8:48111:0:48111] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n6:4526 :0:4526] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n6:4534 :0:4534] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n6:4518 :0:4518] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b2102r4n5:98674:0:98674] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r4n1:10835:0:10835] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n4:128756:0:128756] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b2102r4n5:98698:0:98698] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n6:4549 :0:4549] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n6:4557 :0:4557] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n4:128748:0:128748] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n5:25038:0:25038] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)
[b3203r3n4:128764:0:128764] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xfffffffc09c4f360)

/data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/physics/rrtmg/aer_src/rrtmg_lw_rtrnmc.f90: [ rrtmg_lw_rtrnmc_mp_rtrnmc_() ]
...

372 else
373 tblind = odepth/(bpade+odepth)
374 itr = tblint*tblind+0.5_r8
==> 375 transc = exp_tbl(itr)
376 atrans(lev) = 1._r8-transc
377 tausfac = tfn_tbl(itr)
378 bbd = plfrac*(blay+tausfac*dplankdn)
373 tblind = odepth/(bpade+odepth)
378 bbd = plfrac*(blay+tausfac*dplankdn)
==== backtrace (tid: 10843) ====
0 0x0000000000b14eda rrtmg_lw_rtrnmc_mp_rtrnmc_() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/physics/rrtmg/aer_src/rrtmg_lw_rtrnmc.f90:375
1 0x0000000000b10afc rrtmg_lw_rad_mp_rrtmg_lw_() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/physics/rrtmg/aer_src/rrtmg_lw_rad.f90:386
2 0x000000000072d368 radlw_mp_rad_rrtmg_lw_() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/physics/rrtmg/radlw.F90:191
3 0x0000000000714902 radiation_mp_radiation_tend_() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/physics/rrtmg/radiation.F90:1153
4 0x00000000006d04a9 physpkg_mp_tphysbc_() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/physics/cam/physpkg.F90:2272
5 0x00000000006c96f8 physpkg_mp_phys_run1_() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/physics/cam/physpkg.F90:1057
6 0x0000000000506cac cam_comp_mp_cam_run1_() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/control/cam_comp.F90:258
7 0x00000000004f8762 atm_comp_mct_mp_atm_run_mct_() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/cpl/atm_comp_mct.F90:454
8 0x000000000043d0a3 component_mod_mp_component_run_() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/cime/src/drivers/mct/main/component_mod.F90:728
9 0x0000000000422e9e cime_comp_mod_mp_cime_run_() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/cime/src/drivers/mct/main/cime_comp_mod.F90:3465
10 0x000000000043ccf7 MAIN__() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/cime/src/drivers/mct/main/cime_driver.F90:125
11 0x000000000041f222 main() ???:0
12 0x00000000000223d5 __libc_start_main() ???:0
13 0x000000000041f129 _start() ???:0
=================================

In another run, the error messages are from a different F90 file.

/data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/dynamics/fv/tp_core.F90: [ tp_core_mp_xtpv_() ]
...
454 do i=1,im
455 iu = real(i,r8) - cv(i,j)
456 fxv(i,j) = mfxv(i,j)*qtmpv(iu,j)
==> 457 enddo
458 else
459
460 qtmpv(-1,j) = qv(im-1,j)

Does anyone have suggestions on how to diagnose or resolve this issue? Thank you!
Jian

 

[peverley]

Hi Jian,

A couple of thoughts/questions.

1. Can you rerun your case with DEBUG on? That could potentially point to the "real" culprit here (as you mentioned, it's dying in different places which often indicates that there's a bounds issue or memory issue elsewhere in the code.

"./xmlchange DEBUG=TRUE"

2. Do you have any code modifications (either directly modifying source code or via SourceMods)?

Courtney

 

[jizh]

Hi Jian,

A couple of thoughts/questions.

1. Can you rerun your case with DEBUG on? That could potentially point to the "real" culprit here (as you mentioned, it's dying in different places which often indicates that there's a bounds issue or memory issue elsewhere in the code.


2. Do you have any code modifications (either directly modifying source code or via SourceMods)?

Courtney

Hi Courtney,
Thanks for your reply.

I have not modified any source code.
After setting debug = true, I rebuilt and reran the model. When it crashed again, I found the following message in the cesm.log file.

[b3207r4n5:53793:0:53793] Caught signal 8 (Floating point exception: floating-point invalid operation)
/data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/chemistry/mozart/mo_jlong.F90: [ mo_jlong_mp_interpolate_rsf_() ]
...
859 v3ratu = colo3_in(k) / colo3(pind-1)
860 do iv = 1,numcolo3
861 if( o3rat(iv) > v3ratu ) then
==> 862 exit
863 end if
864 end do
865 ratindu = max( min( iv,numcolo3 ) - 1,1 )
==== backtrace (tid: 53793) ====
0 0x0000000003d73774 mo_jlong_mp_interpolate_rsf_() /data1/elpt_2023_000223/jzheng/cesm-.1.5/components/cam/src/chemistry/mozart/mo_jlong.F90:862
1 0x0000000003d6b41c mo_jlong_mp_jlong_photo_() /data1/elpt_2023_000223/jzheng/cesm-.1.5/components/cam/src/chemistry/mozart/mo_jlong.F90:709
2 0x0000000002e76bb5 mo_photo_mp_table_photo_() /data1/elpt_2023_000223/jzheng/cesm-.1.5/components/cam/src/chemistry/mozart/mo_photo.F90:877
3 0x0000000002d9778b mo_gas_phase_chemdr_mp_gas_phase_chemdr_() /data1/elpt_2023_000223/jzheng/cesm-.1.5/components/cam/src/chemistry/mozart/mo_gas_phase_chemdr.F90:844
4 0x00000000025299c4 chemistry_mp_chem_timestep_tend_() /data1/elpt_2023_000223/jzheng/cesm-.1.5/components/cam/src/chemistry/mozart/chemistry.F90:1293
5 0x0000000001053e5e physpkg_mp_tphysac_() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/physics/cam/physpkg.F90:1418
6 0x0000000001050cff physpkg_mp_phys_run2_() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/physics/cam/physpkg.F90:1163
7 0x0000000000831af3 cam_comp_mp_cam_run2_() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/components/cam/src/control/cam_comp.F90:295
8 0x00000000008013ae atm_comp_mct_mp_atm_run_mct_() /data1/elpt_2023_000223/jzheng/cesm-.1.5/components/cam/src/cpl/atm_comp_mct.F90:433
9 0x000000000047bf70 component_mod_mp_component_run_() /data1/elpt_2023_000223/jzheng/cesm-.1.5/cime/src/drivers/mct/main/component_mod.F90:728
10 0x00000000004534d0 cime_comp_mod_mp_cime_run_() /data1/elpt_2023_000223/jzheng/cesm-.1.5/cime/src/drivers/mct/main/cime_comp_mod.F90:3465
11 0x0000000000463940 MAIN__() /data1/elpt_2023_000223/jzheng/cesm-2.1.5/cime/src/drivers/mct/main/cime_driver.F90:125
12 0x000000000041eda2 main() ???:0
13 0x00000000000223d5 __libc_start_main() ???:0
14 0x000000000041eca9 _start() ???:0
=================================

What puzzles me is that the model usually crashes on the 16th day of one month (as indicated in the cpl.log). However, if I restart the model from the most recent restart files, it can usually get past the previous crash point.

 

[peverley]

Hi again,

Can you provide the following? I'd like to see if I can replicate your issue.

1. The tag of CAM that you are running. You can find this by navigating into $CESM/components/cam and running "git describe --tags"
2. The ./create_newcase command you used to set up your case.
3. Any namelist (user_nl_xxx) or XML changes you made in your case.

Also, could you confirm that you are using ifort rather than ifx as the intel compiler? Just want to make sure. CESM2 will likely not work with ifx.

Thanks!

 

[jizh]

@peverley
Thank you for following up on this issue.

1. The tag of CAM is cam_cesm2_1_rel_60.
2. create_newcase --case /data1/elpt_2023_000223/jzheng/cesm-2.1.5/case/testF2000climo --compset F2000climo --res f09_f09_mg17 --mach huan --compiler intel --mpilib impi
3. I did not make any changes other than adjusting STOP_N and STOP_OPTION. The model was initialized from restart files generated by a previous F2000climo run on another server, rather than from the official initial condition files.

The compiler used is ifort 2021.3.1 and intelmpi 2021.3.1. The following are from Macros.make:
SCC := icc
SCXX := icpc
SFC := ifort
NETCDF_PATH := $(NETCDF)
ifeq ($(MPILIB),impi)
MPICC := mpiicc
MPICXX := mpicpc
MPIFC := mpiifort
endif

Additionally, I came across a message that I am not sure whether it is relevant. For context, I initially downloaded the entire code to my local computer and then uploaded it to the server, since the server cannot directly access GitHub.

 

[peverley]

Hi Jian,

Have you tried running the same configuration with the official initial condition files? If it runs past this point with those, it might signal an issue with the restart files.

Courtney