12wang_xin@tongji_edu_cn
New Member
Hi, I try to run CESM on a cluster which has 3 nodes using MPI and Hydra. I build the case with this : ./create_newcase -case /cesm/cesm1_2_0/test4 -mach newmach -compset F_1850_CAM5 -res f19_g16. And it failed. Logs are as follows. Any suggestions? Thanks a lot! -----------------------------------TASK# NAME0 node11 node12 node13 node14 node25 node26 node27 node28 node39 node310 node311 node312 node113 node114 node115 node116 node217 node218 node219 node220 node321 node322 node323 node324 node125 node126 node127 node128 node229 node230 node231 node232 node333 node334 node335 node336 node137 node138 node139 node140 node241 node242 node243 node244 node345 node346 node347 node348 node149 node150 node151 node152 node253 node254 node255 node256 node357 node358 node359 node360 node161 node162 node163 node1Opened existing file/cesm/cesm1_2_0/inputdata/atm/cam/inic/fv/cami-mam3_0000-01-01_1.9x2.5_L30_c090306.nc65536Opened existing file/cesm/cesm1_2_0/inputdata/atm/cam/topo/USGS-gtopo30_1.9x2.5_remap_c050602.nc131072 ==================================================================================== BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES= PID 7389 RUNNING AT node2= EXIT CODE: 11= CLEANING UP REMAINING PROCESSES= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES===================================================================================[proxy:0:0@node1] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:885): assert (!closed) failed[proxy:0:0@node1] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status[proxy:0:0@node1] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event[mpiexec@node1] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting[mpiexec@node1] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion[mpiexec@node1] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion[mpiexec@node1] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion