Error when executing ./case.submit in CESM2.1.3, ERROR: Segmentation fault (core dumped)

[Lei]

Dear all,
I am currently running the case with --res f19_g17 --compset BSSP585cmip6 on my machine. After successfully created and built the case, when execute command ./case.submit, it will die immediately. It shows the following error information in the cesm.log has only one line: srun: error: c8n1: task 2: Segmentation fault (core dumped)

I don't know why the case with --res f09_g17 --compset BSSP126cmip6 can run successfully on my machine but this one fails, and I really appreciate it if anyone can help me with this. Thank a lot!!

 

[dbailey]

I'm going to move this to the porting forum.

 

[jedwards]

I don't know either - but it's dying before the model starts. Can you compare
./preview_run for the two cases and see if anything is apparent there?
Also is it possible that the f19_g17 run just landed on a bad node? (c8n1 might be suspect)

 

[Lei]

I don't know either - but it's dying before the model starts. Can you compare
./preview_run for the two cases and see if anything is apparent there?
Also is it possible that the f19_g17 run just landed on a bad node? (c8n1 might be suspect)

Thanks for your reply! The results in both cases are the same, and when I avoid the C8N1 node to submit the job, a new error in cesm.log has occurred: /share/home/intel/compilers_and_libraries_2020.0.139/linux/mpi/intel64/bin/mpirun: line 103: 155770 Segmentation fault (core dumped) mpiexec.hydra "$@" 0<&0. What's even worse is that the same error occurred when I resubmitted the case that I could have successfully run.
How can this be resolved? Thank you in advance!

 

[jedwards]

I'm sorry but this is a system error and not a cesm error, you will need to discuss with your local system administrator.

 

[Lei]

I'm sorry but this is a system error and not a cesm error, you will need to discuss with your local system administrator.

Thank you for your suggestion. As you said, this is not a model error. I found a solution on the intel community: export I_MPI_HYDRA_TOPOLIB=ipl.