praveenkumar_jrf@tropmet_res_in
New Member
Hello, I am trying to run a futureclimate simulation for atmospheric chemistry using the BRCP85C5CN compset of CESM 1.2.2. The ocn/ice grid is g16 and the atm/lnd grid is f19. I am using runtype as startup run.When trying to running my model with a short 31-day run, the run crashes within the first 1 minutes with Run-time issues forum: "ERROR: remap transport: bad departure points." I assume that this is a timestep issue between the cice, cam, or pop models and i am also use CFL voilance xndtv_=2.0 in ice_in but again showing same error after submitting job.. CalcWorkPerBlock: Total blocks: 1920 Ice blocks: 522 IceFree blocks: 1066 Land blocks: 332MCT::m_Router::initp_: GSMap indices not increasing...Will correctMCT::m_Router::initp_: RGSMap indices not increasing...Will correctMCT::m_Router::initp_: RGSMap indices not increasing...Will correctMCT::m_Router::initp_: GSMap indices not increasing...Will correctMCT::m_Router::initp_: GSMap indices not increasing...Will correctMCT::m_Router::initp_: RGSMap indices not increasing...Will correctMCT::m_Router::initp_: RGSMap indices not increasing...Will correctMCT::m_Router::initp_: GSMap indices not increasing...Will correctMCT::m_Router::initp_: GSMap indices not increasing...Will correctMCT::m_Router::initp_: RGSMap indices not increasing...Will correctMCT::m_Router::initp_: RGSMap indices not increasing...Will correctMCT::m_Router::initp_: GSMap indices not increasing...Will correctMCT::m_Router::initp_: GSMap indices not increasing...Will correctMCT::m_Router::initp_: RGSMap indices not increasing...Will correctMCT::m_Router::initp_: RGSMap indices not increasing...Will correctMCT::m_Router::initp_: GSMap indices not increasing...Will correctcalcsize j,iq,jac, lsfrm,lstoo 1 1 1 21 17calcsize j,iq,jac, lsfrm,lstoo 1 1 2 21 17calcsize j,iq,jac, lsfrm,lstoo 1 2 1 18 11calcsize j,iq,jac, lsfrm,lstoo 1 2 2 18 11calcsize j,iq,jac, lsfrm,lstoo 1 3 1 19 13calcsize j,iq,jac, lsfrm,lstoo 1 3 2 19 13calcsize j,iq,jac, lsfrm,lstoo 1 4 1 20 16calcsize j,iq,jac, lsfrm,lstoo 1 4 2 20 16calcsize j,iq,jac, lsfrm,lstoo 2 1 1 17 21calcsize j,iq,jac, lsfrm,lstoo 2 1 2 17 21calcsize j,iq,jac, lsfrm,lstoo 2 2 1 11 18calcsize j,iq,jac, lsfrm,lstoo 2 2 2 11 18calcsize j,iq,jac, lsfrm,lstoo 2 3 1 13 19calcsize j,iq,jac, lsfrm,lstoo 2 3 2 13 19calcsize j,iq,jac, lsfrm,lstoo 2 4 1 16 20Warning: Departure points out of bounds in remap my_task, i, j = 85 2 7 dpx, dpy = -0.289688487888129 -1961308.83137652 HTN(i,j), HTN(i+1,j) = 12847.1778616464 12973.2192227869 HTE(i,j), HTE(i,j+1) = 54723.6576367913 55050.5682880806 istep1, my_task, iblk = 1095001 85 5 Global block: 1915 Global i and j: 273 382(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stoppingRank 85 [Tue Sep 4 22:43:09 2018] [c0-0c0s5n1] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 85 Warning: Departure points out of bounds in remap my_task, i, j = 17 2 6 dpx, dpy = -3.446450635914853E-002 -295396.823662202 HTN(i,j), HTN(i+1,j) = 76197.9261699294 76272.4941905301 HTE(i,j), HTE(i,j+1) = 45493.8229863717 45402.0256307558 istep1, my_task, iblk = 1095001 17 5 Global block: 1638 Global i and j: 297 325(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping Warning: Departure points out of bounds in remap my_task, i, j = 19 9 9 dpx, dpy = -347095.341664454 4.391475628600266E-002 HTN(i,j), HTN(i+1,j) = 64778.2833053467 64923.3737248582 HTE(i,j), HTE(i,j+1) = 48174.8206549177 48146.6460732683 istep1, my_task, iblk = 1095001 19 5 Global block: 1675 Global i and j: 280 336(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping Warning: Departure points out of bounds in remap Warning: Departure points out of bounds in remap my_task, i, j = 73 9 9 Warning: Departure points out of bounds in remap my_task, i, j = 73 9 9 my_task, i, j = 75 9 9 dpx, dpy = -0.783965726279263 -5382049.09389143 dpx, dpy = -0.423631288757127 -2820417.26344656 HTN(i,j), HTN(i+1,j) = 21863.8132470141 21697.2637106703 HTN(i,j), HTN(i+1,j) = 14813.9228427050 14707.0773524118 HTE(i,j), HTE(i,j+1) = 51605.7175137492 51884.7615278120 HTE(i,j), HTE(i,j+1) = 60569.1697359262 60515.8397111647 istep1, my_task, iblk = 1095001 73 5 Global block: 1845 istep1, my_task, iblk = 1095001 75 5 Global i and j: 40 376 Global block: 1851 Global i and j: 88 376(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping Warning: Departure points out of bounds in remap my_task, i, j = 67 9 9 dpx, dpy = -4289101.81886372 0.650229822653047 Warning: Departure points out of bounds in remap HTN(i,j), HTN(i+1,j) = 20340.6071829624 20251.3914950048 my_task, i, j = 65 9 9 HTE(i,j), HTE(i,j+1) = 64505.3057057442 64357.2473596999 istep1, my_task, iblk = 1095001 67 5 dpx, dpy = -1.13211361179695 -7390920.79386104 Global block: 1814 HTN(i,j), HTN(i+1,j) = 18217.8734748957 18217.8734748957 Global i and j: 112 368 HTE(i,j), HTE(i,j+1) = 67206.5053842787 66965.1293719670(shr_sys_abort) ERROR: remap transport: bad departure points istep1, my_task, iblk = 1095001 65 5(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping Global block: 1820 Global i and j: 160 368Rank 17 [Tue Sep 4 22:43:09 2018] [c0-0c0s1n1] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 17Rank 19 [Tue Sep 4 22:43:09 2018] [c0-0c0s1n1] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 19 Warning: Departure points out of bounds in remap my_task, i, j = 37 5 6 dpx, dpy = -5959080.72596010 0.885840443624786 HTN(i,j), HTN(i+1,j) = 38547.9596439050 38636.0048307147 HTE(i,j), HTE(i,j+1) = 68360.6894337423 68264.5378630737 istep1, my_task, iblk = 1095001 37 5 Global block: 1745 Global i and j: 196 349(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stoppingRank 75 [Tue Sep 4 22:43:09 2018] [c0-0c0s5n0] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 75Rank 73 [Tue Sep 4 22:43:09 2018] [c0-0c0s5n0] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 73Rank 65 [Tue Sep 4 22:43:09 2018] [c0-0c0s2n3] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 65Rank 67 [Tue Sep 4 22:43:09 2018] [c0-0c0s2n3] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 67Rank 37 [Tue Sep 4 22:43:09 2018] [c0-0c0s1n3] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 37 Warning: Departure points out of bounds in remap my_task, i, j = 13 9 9 dpx, dpy = -1969807.83543119 -0.276936587254058 HTN(i,j), HTN(i+1,j) = 49542.1320208640 49542.1320208640 HTE(i,j), HTE(i,j+1) = 59395.4550164216 59395.4550164216 istep1, my_task, iblk = 1095001 13 2 Global block: 88 Global i and j: 64 24 Warning: Departure points out of bounds in remap my_task, i, j = 61 8 3 dpx, dpy = -7008947.51157620 1.00621322534992 HTN(i,j), HTN(i+1,j) = 35007.3766590201 35210.7855901806 HTE(i,j), HTE(i,j+1) = 53552.8029337019 53635.0251756751 HTN(i,j), HTN(i+1,j) = 35007.3766590201 35210.7855901806 HTE(i,j), HTE(i,j+1) = 53552.8029337019 53635.0251756751 istep1, my_task, iblk = 1095001 61 6 Global block: 1833 Global i and j: 263 362(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping (shr_sys_abort) ERROR: remap transport: bad departure points Warning: Departure points out of bounds in remap(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping my_task, i, j = 11 9 9 dpx, dpy = -3046780.65777675 -0.428540816081658 HTN(i,j), HTN(i+1,j) = 49542.1320208640 49542.1320208640 HTE(i,j), HTE(i,j+1) = 59395.4550164216 59395.4550164216 istep1, my_task, iblk = 1095001 11 2 Global block: 82 Global i and j: 16 24(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping Warning: Departure points out of bounds in remap Warning: Departure points out of bounds in remap my_task, i, j = 85 9 7 dpx, dpy = -2024058.92984983 0.299101805707265 HTN(i,j), HTN(i+1,j) = 12721.7662440709 12847.1778616464 HTE(i,j), HTE(i,j+1) = 54889.3713126878 55205.9371082698 istep1, my_task, iblk = 1095001 85 4 Global block: 1914 Global i and j: 272 382(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stoppingRank 61 [Tue Sep 4 22:43:09 2018] [c0-0c0s2n2] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 61 Warning: Departure points out of bounds in remap my_task, i, j = 19 7 8 dpx, dpy = -6813950.37692591 -0.962210391481831 my_task, i, j = 19 7 8 dpx, dpy = -6813950.37692591 -0.962210391481831 HTN(i,j), HTN(i+1,j) = 48469.1558581211 48469.1558581211 HTE(i,j), HTE(i,j+1) = 59395.4550164216 59395.4550164216 istep1, my_task, iblk = 1095001 19 2 Warning: Departure points out of bounds in remap Global block: 112 my_task, i, j = 17 9 9 Global i and j: 254 23(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping dpx, dpy = 0.454576952674012 -3231958.59755057 HTN(i,j), HTN(i+1,j) = 49542.1320208640 49542.1320208640Rank 13 [Tue Sep 4 22:43:09 2018] [c0-0c0s1n0] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 13Rank 11 [Tue Sep 4 22:43:09 2018] [c0-0c0s1n0] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 11 HTE(i,j), HTE(i,j+1) = 59395.4550164216 59395.4550164216 istep1, my_task, iblk = 1095001 17 2 Global block: 106 Global i and j: 208 24(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping my_task, i, j = 61 9 9 dpx, dpy = 0.857176166447552 -6930312.97904853 HTN(i,j), HTN(i+1,j) = 73792.9526489570 73792.9526489570 HTE(i,j), HTE(i,j+1) = 59395.4550164223 59395.4550164223 istep1, my_task, iblk = 1095001 61 2 Global block: 203 Global i and j: 24 48(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stoppingRank 85 [Tue Sep 4 22:43:09 2018] [c0-0c0s5n1] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 85Rank 61 [Tue Sep 4 22:43:09 2018] [c0-0c0s2n2] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 61Rank 19 [Tue Sep 4 22:43:09 2018] [c0-0c0s1n1] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 19Rank 17 [Tue Sep 4 22:43:09 2018] [c0-0c0s1n1] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 17 Warning: Departure points out of bounds in remapWarning: Departure points out of bounds in remap Warning: Departure points out of bounds in remap my_task, i, j = 75 9 9 my_task, i, j = 73 9 9 dpx, dpy = -5413340.39939717 0.815943009735375 dpx, dpy = -3481478.34479738 0.510142005316014 HTN(i,j), HTN(i+1,j) = 14011.1938688021 13919.0994084930 HTN(i,j), HTN(i+1,j) = 20521.4190992681 20353.6401819884 HTE(i,j), HTE(i,j+1) = 61590.3013943326 61488.1966604617 HTE(i,j), HTE(i,j+1) = 53313.7300885600 53542.8237222411 istep1, my_task, iblk = 1095001 75 4 istep1, my_task, iblk = 1095001 73 4 Global block: 1852 Global block: 1846 Global i and j: 96 376 Global i and j: 48 376(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping Warning: Departure points out of bounds in remap my_task, i, j = 65 9 9 dpx, dpy = -7024089.90493764 1.07562939000972 HTN(i,j), HTN(i+1,j) = 18280.7495327568 18266.7647951898 HTE(i,j), HTE(i,j+1) = 67133.7964013498 66895.0677952360 istep1, my_task, iblk = 1095001 65 4 Global block: 1819 Global i and j: 152 368(shr_sys_abort) ERROR: remap transport: bad departure points(shr_sys_abort) WARNING: calling shr_mpi_abort() and stoppingRank 73 [Tue Sep 4 22:43:09 2018] [c0-0c0s5n0] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 73Rank 75 [Tue Sep 4 22:43:09 2018] [c0-0c0s5n0] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 75Rank 65 [Tue Sep 4 22:43:09 2018] [c0-0c0s2n3] application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 65forrtl: error (76): Abort trap signalImage PC Routine Line Sourcecesm.exe 0000000002D90E4A Unknown Unknown UnknownImage PC Routine Line Sourcecesm.exe 0000000002D90E4A Unknown Unknown Unknowncesm.exe 00000000027EA560 Unknown Unknown Unknowncesm.exe 0000000002D6C19B Unknown Unknown Unknowncesm.exe 0000000002E9919A Unknown Unknown Unknowncesm.exe 0000000002656E52 Unknown Unknown Unknowncesm.exe 00000000026FB74B Unknown Unknown Unknowncesm.exe 000000000262B0F5 Unknown Unknown Unknowncesm.exe 000000000220BD19 shr_mpi_mod_mp_sh 2064 shr_mpi_mod.F90cesm.exe 000000000225E8E6 shr_sys_mod_mp_sh 295 shr_sys_mod.F90cesm.exe 00000000014E1F05 ice_transport_rem 698 ice_transport_remap.F90cesm.exe 0000000002C7F0B3 Unknown Unknown Unknowncesm.exe 0000000002C37830 Unknown Unknown Unknowncesm.exe 0000000002C36AB5 Unknown Unknown Unknowncesm.exe 0000000002C7F479 Unknown Unknown Unknowncesm.exe 00000000027E7124 Unknown Unknown Unknowncesm.exe 0000000002F00FC9 Unknown Unknown Unknownforrtl: error (76): Abort trap signalImage PC Routine Line Sourcecesm.exe 0000000002D90E4A Unknown Unknown Unknowncesm.exe 00000000027EA560 Unknown Unknown Unknowncesm.exe 0000000002D6C19B Unknown Unknown Unknowncesm.exe 0000000002E9919A Unknown Unknown Unknowncesm.exe 0000000002656E52 Unknown Unknown Unknowncesm.exe 00000000026FB74B Unknown Unknown Unknowncesm.exe 000000000262B0F5 Unknown Unknown Unknowncesm.exe 000000000220BD19 shr_mpi_mod_mp_sh 2064 shr_mpi_mod.F90cesm.exe 000000000225E8E6 shr_sys_mod_mp_sh 295 shr_sys_mod.F90cesm.exe 00000000014E1F05 ice_transport_rem 698 ice_transport_remap.F90cesm.exe 0000000002C7F0B3 Unknown Unknown Unknowncesm.exe 0000000002C37830 Unknown Unknown Unknowncesm.exe 0000000002C36AB5 Unknown Unknown Unknowncesm.exe 0000000002C7F479 Unknown Unknown Unknowncesm.exe 00000000027E7124 Unknown Unknown Unknowncesm.exe 0000000002F00FC9 Unknown Unknown Unknownforrtl: error (76): Abort trap signalImage PC Routine Line Sourcecesm.exe 0000000002D90E4A Unknown Unknown UnknownImage PC Routine Line Sourcecesm.exe 0000000002D90E4A Unknown Unknown Unknowncesm.exe 00000000027EA560 Unknown Unknown Unknowncesm.exe 0000000002D6C19B Unknown Unknown Unknowncesm.exe 0000000002E9919A Unknown Unknown Unknowncesm.exe 0000000002656E52 Unknown Unknown Unknowncesm.exe 00000000026FB74B Unknown Unknown Unknowncesm.exe 000000000262B0F5 Unknown Unknown Unknowncesm.exe 000000000220BD19 shr_mpi_mod_mp_sh 2064 shr_mpi_mod.F90cesm.exe 000000000225E8E6 shr_sys_mod_mp_sh 295 shr_sys_mod.F90cesm.exe 00000000014E1F05 ice_transport_rem 698 ice_transport_remap.F90cesm.exe 0000000002C7F0B3 Unknown Unknown Unknowncesm.exe 0000000002C37830 Unknown Unknown Unknowncesm.exe 0000000002C38A53 Unknown Unknown Unknowncesm.exe 0000000002C18E46 Unknown Unknown Unknowncesm.exe 00000000014DC29E ice_transport_rem 588 ice_transport_remap.F90cesm.exe 00000000014CC0D3 ice_transport_dri 549 ice_transport_driver.F90cesm.exe 00000000014AEFF5 ice_step_mod_mp_s 679 ice_step_mod.F90cesm.exe 00000000013D1DF6 ice_comp_mct_mp_i 631 ice_comp_mct.F90cesm.exe 000000000040DD54 ccsm_comp_mod_mp_ 3248 ccsm_comp_mod.F90cesm.exe 000000000042C90B MAIN__ 91 ccsm_driver.F90cesm.exe 000000000040A80E Unknown Unknown Unknowncesm.exe 0000000002E92699 Unknown Unknown Unknowncesm.exe 000000000040A6F9 Unknown Unknown Unknownforrtl: error (76): Abort trap signalImage PC Routine Line Sourcecesm.exe 0000000002D90E4A Unknown Unknown Unknowncesm.exe 00000000027EA560 Unknown Unknown Unknowncesm.exe 0000000002D6C19B Unknown Unknown Unknowncesm.exe 0000000002E9919A Unknown Unknown Unknowncesm.exe 0000000002656E52 Unknown Unknown Unknowncesm.exe 00000000026FB74B Unknown Unknown Unknowncesm.exe 000000000262B0F5 Unknown Unknown Unknowncesm.exe 000000000220BD19 shr_mpi_mod_mp_sh 2064 shr_mpi_mod.F90cesm.exe 000000000225E8E6 shr_sys_mod_mp_sh 295 shr_sys_mod.F90cesm.exe 00000000014E1F05 ice_transport_rem 698 ice_transport_remap.F90cesm.exe 0000000002C7F0B3 Unknown Unknown Unknownforrtl: error (76): Abort trap signalImage PC Routine Line Sourcecesm.exe 0000000002D90E4A Unknown Unknown Unknowncesm.exe 00000000027EA560 Unknown Unknown Unknowncesm.exe 0000000002D6C19B Unknown Unknown Unknowncesm.exe 0000000002E9919A Unknown Unknown Unknowncesm.exe 0000000002656E52 Unknown Unknown Unknowncesm.exe 00000000026FB74B Unknown Unknown Unknowncesm.exe 000000000262B0F5 Unknown Unknown Unknowncesm.exe 000000000220BD19 shr_mpi_mod_mp_sh 2064 shr_mpi_mod.F90cesm.exe 000000000225E8E6 shr_sys_mod_mp_sh 295 shr_sys_mod.F90cesm.exe 00000000014E1F05 ice_transport_rem 698 ice_transport_remap.F90cesm.exe 0000000002C7F0B3 Unknown Unknown Unknowncesm.exe 0000000002C37830 Unknown Unknown Unknowncesm.exe 0000000002C38A53 Unknown Unknown Unknowncesm.exe 0000000002C18E46 Unknown Unknown Unknowncesm.exe 00000000014DC29E ice_transport_rem 588 ice_transport_remap.F90cesm.exe 00000000014CC0D3 ice_transport_dri 549 ice_transport_driver.F90cesm.exe 00000000014AEFF5 ice_step_mod_mp_s 679 ice_step_mod.F90cesm.exe 00000000013D1DF6 ice_comp_mct_mp_i 631 ice_comp_mct.F90cesm.exe 000000000040DD54 ccsm_comp_mod_mp_ 3248 ccsm_comp_mod.F90cesm.exe 000000000042C90B MAIN__ 91 ccsm_driver.F90cesm.exe 000000000040A80E Unknown Unknown Unknowncesm.exe 0000000002E92699 Unknown Unknown Unknowncesm.exe 000000000040A6F9 Unknown Unknown Unknown_pmiu_daemon(SIGCHLD): [NID 00020] [c0-0c0s5n0] [Tue Sep 4 22:43:09 2018] PE RANK 73 exit signal Aborted[NID 00020] 2018-09-04 22:43:09 Apid 12709820: initiated application terminationApplication 12709820 exit codes: 134Application 12709820 exit signals: KilledApplication 12709820 resources: utime ~67s, stime ~14s, Rss ~395088, inblocks ~0, outblocks ~0 Thanks and regardsPraveen KuamrJRF IITM Pune, Indiapraveenkumar.jrf@tropmet.res.in
