CESM1-1-2-LENS-n21 error: MPT ERROR: Rank 526(g:526) received signal SIGSEGV(11).

[im2527@columbia_edu]

Hi,

I am running cesm1_1_2_LENS_n21 with abrupt 2xCO2 runs. I am able to run the model for 14 years and 9 months, and then it fails and I get the following error in the cesm.log:

318:MPT: --------stack traceback-------
88:OMP: Warning #190: Forking a process while a parallel region is active is potentially unsafe.
526:MPT ERROR: Rank 526(g:526) received signal SIGSEGV(11).
526: Process ID: 51425, Host: r8i0n3, Program: /glade/scratch/im2527/2xCO2.B1850LENS.n21.f09_g16/bld/cesm.exe
526: MPT Version: HPE MPT 2.19 02/23/19 05:30:09
526:
526:MPT: --------stack traceback-------
526:OMP: Warning #190: Forking a process while a parallel region is active is potentially unsafe.
318:OMP: Warning #190: Forking a process while a parallel region is active is potentially unsafe.
208:MPT ERROR: Rank 208(g:208) received signal SIGSEGV(11).
208: Process ID: 15894, Host: r5i2n28, Program: /glade/scratch/im2527/2xCO2.B1850LENS.n21.f09_g16/bld/cesm.exe
208: MPT Version: HPE MPT 2.19 02/23/19 05:30:09
208:
208:MPT: --------stack traceback-------
208:OMP: Warning #190: Forking a process while a parallel region is active is potentially unsafe.
-1:MPT ERROR: MPI_COMM_WORLD rank 466 has terminated without calling MPI_Finalize()
-1: aborting job
MPT: Received signal 11

Any suggestions?

Thanks!

 

[wanzi0411@gmail_com]

Hi,

I am running cesm1_1_2_LENS_n21 with abrupt 2xCO2 runs. I am able to run the model for 14 years and 9 months, and then it fails and I get the following error in the cesm.log:

318:MPT: --------stack traceback-------
88:OMP: Warning #190: Forking a process while a parallel region is active is potentially unsafe.
526:MPT ERROR: Rank 526(g:526) received signal SIGSEGV(11).
526: Process ID: 51425, Host: r8i0n3, Program: /glade/scratch/im2527/2xCO2.B1850LENS.n21.f09_g16/bld/cesm.exe
526: MPT Version: HPE MPT 2.19 02/23/19 05:30:09
526:
526:MPT: --------stack traceback-------
526:OMP: Warning #190: Forking a process while a parallel region is active is potentially unsafe.
318:OMP: Warning #190: Forking a process while a parallel region is active is potentially unsafe.
208:MPT ERROR: Rank 208(g:208) received signal SIGSEGV(11).
208: Process ID: 15894, Host: r5i2n28, Program: /glade/scratch/im2527/2xCO2.B1850LENS.n21.f09_g16/bld/cesm.exe
208: MPT Version: HPE MPT 2.19 02/23/19 05:30:09
208:
208:MPT: --------stack traceback-------
208:OMP: Warning #190: Forking a process while a parallel region is active is potentially unsafe.
-1:MPT ERROR: MPI_COMM_WORLD rank 466 has terminated without calling MPI_Finalize()
-1: aborting job
MPT: Received signal 11

Any suggestions?

Thanks!

Hi there,

I am having similar issues with my simulation. My run was aborted right after the submission, and the error message from the cesm log file looks quite similar to yours. If you already solved the problem, might you share some insights? Thanks a lot!

Best,
Wenwen

 

[im2527@columbia_edu]

Hi,

In env_mach_pes.xml, I changed NTHRDS_ATM, NTHRDS_LND, NTHRDS_ICE, NTHRDS_OCN, NTHRDS_CPL, NTHRDS_GLC, and NTHRDS_ROF to 1 and it worked.

Best,
Ivan Mitevski