Segmentation fault when reading tracer_nml

[AKAleksinski]

Hello,

I am attempting to run an iCESM1.2 case, which is a G case (POP2 and CICE) that I am attempting to drive from the output of a fully-coupled B case that I have previously run. I am encountering an error upon submission. The CESM log reads:

0: Reading setup_nml
0: Reading grid_nml
0: Reading ice_nml
0: Reading tracer_nml
0:CalcWorkPerBlock: Total blocks: 1920 Ice blocks: 496 IceFree blocks: 890 Land blocks: 534
1:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x0000000011124bc0 ***
5:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x000000001988d0a0 ***
49:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x000000001132cfd0 ***
7:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x0000000019608fd0 ***
50:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000112a25d0 ***
18:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000176b43e0 ***
52:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x000000001132bdd0 ***
19:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000103180f0 ***
54:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x000000001a616790 ***
20:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000104ce580 ***
55:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x000000001a617c70 ***
29:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000198adf40 ***
57:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x000000001a618580 ***
32:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000102c45f0 ***
59:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x000000001129cb80 ***
34:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000102bc960 ***
61:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x000000001a8a6b70 ***
6:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x0000000010317c90 ***
38:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000186b5a50 ***
8:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000196104d0 ***
40:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000114dff50 ***
9:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000176a7c40 ***
43:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x000000001a6176c0 ***
11:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000104cc040 ***
44:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x000000001a8b9530 ***
14:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000176a7e60 ***
46:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x000000001132dc20 ***
30:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000104cdec0 ***
48:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x000000001a8b8ef0 ***
31:*** Error in `../bld/cesm.exe': corrupted double-linked list: 0x00000000104ce600 ***
38:======= Backtrace: =========
38:/glade/u/apps/ch/os/lib64/libc.so.6(+0x774ef)[0x2b62d1f734ef]
3:MPT ERROR: Rank 3(g:3) received signal SIGSEGV(11).
3: Process ID: 50354, Host: r3i5n12, Program: /glade/scratch/aaleksinski/G_test/bld/cesm.exe
3: MPT Version: HPE MPT 2.19 02/23/19 05:30:09

The rest of the log consists of more backtraces and memory logs. It seems to me that reading tracer_nml triggered this, but nothing looks incorrect in my ice_in. Does anybody have any thoughts as to what happened here?

Many thanks,

Adam Aleksinski

 

[dbailey]

Hi Adam,

Did you change versions of the code? It looks like the namelist is read fine because you are getting the WorkPerBlock line. Can you try setting DEBUG to TRUE and re-running to see if you get more of a traceback?

 

[AKAleksinski]

Sure thing. Turning on DEBUG shows this as the traceback:

0: Reading setup_nml 0: Reading grid_nml 0: Reading ice_nml 0: Reading tracer_nml 0:CalcWorkPerBlock: Total blocks: 1920 Ice blocks: 496 IceFree blocks: 890 Land blocks: 534 56:forrtl: severe (408): fort: (2): Subscript #2 of the array BLOCKINDEX has value 7 which is greater than the upper bound of 6 56: 56:Image PC Routine Line Source 56:cesm.exe 00000000033AE256 Unknown Unknown Unknown 56:cesm.exe 0000000001101779 ice_distribution_ 1215 ice_distribution.F90 56:cesm.exe 00000000010F1799 ice_distribution_ 164 ice_distribution.F90 56:cesm.exe 000000000082C05A ice_domain_mp_ini 508 ice_domain.F90 56:cesm.exe 00000000008FC537 ice_grid_mp_init_ 254 ice_grid.F90 56:cesm.exe 0000000000F038EF cice_initmod_mp_c 107 CICE_InitMod.F90 56:cesm.exe 000000000077F90A ice_comp_mct_mp_i 256 ice_comp_mct.F90 56:cesm.exe 000000000042573C ccsm_comp_mod_mp_ 1179 ccsm_comp_mod.F90 56:cesm.exe 00000000004CD975 MAIN__ 90 ccsm_driver.F90 56:cesm.exe 0000000000407ADE Unknown Unknown Unknown 56:libc-2.22.so 00002AEF07B1DA35 __libc_start_main Unknown Unknown 56:cesm.exe 00000000004079E9 Unknown Unknown Unknown