Problem with carma_reft.nc file

[deepashree_dutta@student_unsw_edu_au]

Hi all,

I'm trying to use CARMA/PMC with WACCM4. (sorry for cross posting here!)
I created a case using F1850W compset and enabled CARMA/PMC using:
./xmlchange -file env_build.xml -id CAM_CONFIG_OPTS -val '-phys cam4 -chem waccm_mozart -carma pmc'
(following: http://lasp.colorado.edu/media/meet...rkshop/Papers/Session4/BardeenCAWSES2012.pdf; I couldn't find documentation with more details)
The case builds successfully. But when I submit it, I get this error in cesm.log file:

"MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1001.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
cesm.exe 00000000015FDA0B Unknown Unknown Unknown
libpthread-2.28.s 000014EC70C1FB20 Unknown Unknown Unknown
libuct_ib.so.0.0. 000014EC56B89138 Unknown Unknown Unknown
libucp.so.0.0.0 000014EC577A10EA ucp_worker_progre Unknown Unknown
mca_pml_ucx.so 000014EC5C0BE317 mca_pml_ucx_progr Unknown Unknown
libopen-pal.so.40 000014EC6BF02E8B opal_progress Unknown Unknown
libmpi.so.40.20.2 000014EC7190F4A5 ompi_request_defa Unknown Unknown
libmpi.so.40.20.2 000014EC7194A0B5 ompi_coll_base_bc Unknown Unknown
libmpi.so.40.20.2 000014EC7194A551 ompi_coll_base_bc Unknown Unknown
mca_coll_tuned.so 000014EC544BC5FB ompi_coll_tuned_b Unknown Unknown
libmpi.so.40.20.2 000014EC71925F7A MPI_Bcast Unknown Unknown
libmpi_mpifh.so 000014EC711F88F5 Unknown Unknown Unknown
cesm.exe 00000000008FBBCF carma_intr_mp_car 753 carma_intr.F90
cesm.exe 0000000000705385 physpkg_mp_phys_i 738 physpkg.F90
cesm.exe 00000000004B4F23 cam_comp_mp_cam_i 181 cam_comp.F90
cesm.exe 00000000004B220C atm_comp_mct_mp_a 276 atm_comp_mct.F90
cesm.exe 0000000000440E11 ccsm_comp_mod_mp_ 1058 ccsm_comp_mod.F90"

I checked the atm.log file as well. It seems the model fails when it executes carma_intr() and tries to create the carma_reft.nc file (reference T for CARMA). I get “NetCDF: Unknown file format” error when I tried to open carma_reft.nc.
Deleting the carma_reft.nc file and resubmitting the case does not work.

Is there any chance it might be related to different versions of different libraries?
Do you have any suggestions how to resolve this?
Thanks a lot!
Deepashree

 

[cacraig]

CARMA is handled by the CAM-Chem group, so I will be moving this to their forum