Hi all,
I'm trying to use CARMA/PMC with WACCM4.
I created a case using F1850W compset and enabled CARMA/PMC using:
./xmlchange -file env_build.xml -id CAM_CONFIG_OPTS -val '-phys cam4 -chem waccm_mozart -carma pmc'
(following: http://lasp.colorado.edu/media/meet...rkshop/Papers/Session4/BardeenCAWSES2012.pdf; I couldn't find documentation with more details)
The case builds successfully. But when I submit it, I get this error in cesm.log file:
"MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1001.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
cesm.exe 00000000015FDA0B Unknown Unknown Unknown
libpthread-2.28.s 000014EC70C1FB20 Unknown Unknown Unknown
libuct_ib.so.0.0. 000014EC56B89138 Unknown Unknown Unknown
libucp.so.0.0.0 000014EC577A10EA ucp_worker_progre Unknown Unknown
mca_pml_ucx.so 000014EC5C0BE317 mca_pml_ucx_progr Unknown Unknown
libopen-pal.so.40 000014EC6BF02E8B opal_progress Unknown Unknown
libmpi.so.40.20.2 000014EC7190F4A5 ompi_request_defa Unknown Unknown
libmpi.so.40.20.2 000014EC7194A0B5 ompi_coll_base_bc Unknown Unknown
libmpi.so.40.20.2 000014EC7194A551 ompi_coll_base_bc Unknown Unknown
mca_coll_tuned.so 000014EC544BC5FB ompi_coll_tuned_b Unknown Unknown
libmpi.so.40.20.2 000014EC71925F7A MPI_Bcast Unknown Unknown
libmpi_mpifh.so 000014EC711F88F5 Unknown Unknown Unknown
cesm.exe 00000000008FBBCF carma_intr_mp_car 753 carma_intr.F90
cesm.exe 0000000000705385 physpkg_mp_phys_i 738 physpkg.F90
cesm.exe 00000000004B4F23 cam_comp_mp_cam_i 181 cam_comp.F90
cesm.exe 00000000004B220C atm_comp_mct_mp_a 276 atm_comp_mct.F90
cesm.exe 0000000000440E11 ccsm_comp_mod_mp_ 1058 ccsm_comp_mod.F90"
I checked the atm.log file as well. It seems the model fails when it executes carma_intr() and tries to create the carma_reft.nc file (reference T for CARMA). I get “NetCDF: Unknown file format” error when I tried to open carma_reft.nc.
Deleting the carma_reft.nc file and resubmitting the case does not work.
I do not understand what is causing this issue.
Do you have any suggestions how to resolve this?
Thanks a lot!
Deepashree
I'm trying to use CARMA/PMC with WACCM4.
I created a case using F1850W compset and enabled CARMA/PMC using:
./xmlchange -file env_build.xml -id CAM_CONFIG_OPTS -val '-phys cam4 -chem waccm_mozart -carma pmc'
(following: http://lasp.colorado.edu/media/meet...rkshop/Papers/Session4/BardeenCAWSES2012.pdf; I couldn't find documentation with more details)
The case builds successfully. But when I submit it, I get this error in cesm.log file:
"MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1001.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
cesm.exe 00000000015FDA0B Unknown Unknown Unknown
libpthread-2.28.s 000014EC70C1FB20 Unknown Unknown Unknown
libuct_ib.so.0.0. 000014EC56B89138 Unknown Unknown Unknown
libucp.so.0.0.0 000014EC577A10EA ucp_worker_progre Unknown Unknown
mca_pml_ucx.so 000014EC5C0BE317 mca_pml_ucx_progr Unknown Unknown
libopen-pal.so.40 000014EC6BF02E8B opal_progress Unknown Unknown
libmpi.so.40.20.2 000014EC7190F4A5 ompi_request_defa Unknown Unknown
libmpi.so.40.20.2 000014EC7194A0B5 ompi_coll_base_bc Unknown Unknown
libmpi.so.40.20.2 000014EC7194A551 ompi_coll_base_bc Unknown Unknown
mca_coll_tuned.so 000014EC544BC5FB ompi_coll_tuned_b Unknown Unknown
libmpi.so.40.20.2 000014EC71925F7A MPI_Bcast Unknown Unknown
libmpi_mpifh.so 000014EC711F88F5 Unknown Unknown Unknown
cesm.exe 00000000008FBBCF carma_intr_mp_car 753 carma_intr.F90
cesm.exe 0000000000705385 physpkg_mp_phys_i 738 physpkg.F90
cesm.exe 00000000004B4F23 cam_comp_mp_cam_i 181 cam_comp.F90
cesm.exe 00000000004B220C atm_comp_mct_mp_a 276 atm_comp_mct.F90
cesm.exe 0000000000440E11 ccsm_comp_mod_mp_ 1058 ccsm_comp_mod.F90"
I checked the atm.log file as well. It seems the model fails when it executes carma_intr() and tries to create the carma_reft.nc file (reference T for CARMA). I get “NetCDF: Unknown file format” error when I tried to open carma_reft.nc.
Deleting the carma_reft.nc file and resubmitting the case does not work.
I do not understand what is causing this issue.
Do you have any suggestions how to resolve this?
Thanks a lot!
Deepashree