Program Exception - illegal instruction happening when running mpirun -np 1 ../bld/cesm.exe

[Shane]

Hi, I am running CLM on my own server. I successfully create, set up, and build the case. When I run mpirun -np 1 ../bld/cesm.exe, there is an error below:

forrtl: severe (168): Program Exception - illegal instruction
Image PC Routine Line Source
cesm.exe 000000000112EE2B Unknown Unknown Unknown
libpthread-2.17.s 00007F6CBC23E630 Unknown Unknown Unknown
libmpi.so.12.0.0 00007F6CBCC4BC6C Unknown Unknown Unknown
libmpi.so.12.0.0 00007F6CBCC4D2E6 Unknown Unknown Unknown
libmpi.so.12.0.0 00007F6CBCC4CE5C Unknown Unknown Unknown
libmpi.so.12.0.0 00007F6CBCC4A72A impi_malloc Unknown Unknown
libmpi.so.12.0.0 00007F6CBCE2B914 Unknown Unknown Unknown
libmpi.so.12.0.0 00007F6CBCCECAFB Unknown Unknown Unknown
libmpi.so.12.0.0 00007F6CBCCEC51B MPI_Init Unknown Unknown
libmpifort.so.12. 00007F6CBE18B85B PMPI_INIT Unknown Unknown
cesm.exe 0000000000436883 cime_comp_mod_mp_ 678 cime_comp_mod.F90
cesm.exe 00000000004383E5 MAIN__ 61 cime_driver.F90
cesm.exe 0000000000418562 Unknown Unknown Unknown
libc-2.17.so 00007F6CBBB81555 __libc_start_main Unknown Unknown
cesm.exe 0000000000418469 Unknown Unknown Unknown

Could I know how to solve this issue? Thank you very much in advance!

 

[erik]

Unfortunately, the above doesn't help much as to what is going on. I recommend doing some of the advice in the CLM User's Guide in troubleshooting. Since, you are running with one processor, I'd try building with MPILIB=mpi-serial, as that's simpler and it might help you see problems in MPI. I'd also clean the build and rebuild with DEBUG=TRUE to see if it gives you a more helpful traceback. To make the simplest possible case, I'd also suggest doing a simulation for a single point grid rather than a global grid (if that's what you are doing, you don't give the specifics of the resolution here).