gmodica@aer_com
Member
I have encountered the subject message in a hybrid restart from the output of a 6-hr cold start beginning at 2008-11-14-00000. Using the F_2000_WACCM compset. Additional output below.
Previous posts of similar problems suggest the rpointer files could be at issue, but my rpointer files all appear to have the correct restart timestamp of . A different restart run was successful when started from the output of a 6-hr cold start from 2001-01-01-00000.
Could there be another explanation for this sync problem?
George
(seq_domain_areafactinit) : min/max mdl2drv 1.00000000000000 1.00000000000000 areafact_a_ATM0002
(seq_domain_areafactinit) : min/max drv2mdl 1.00000000000000 1.00000000000000 areafact_a_ATM0002
(seq_domain_areafactinit) : min/max mdl2drv 1.00000000000000 1.00000000000000 areafact_l_LND0002
(seq_domain_areafactinit) : min/max drv2mdl 1.00000000000000 1.00000000000000 areafact_l_LND0002
(seq_domain_areafactinit) : min/max mdl2drv 1.00000000000000 1.00000000000000 areafact_i_ICE0002
(seq_domain_areafactinit) : min/max drv2mdl 1.00000000000000 1.00000000000000 areafact_i_ICE0002
(seq_mct_drv) : Initialize atm component phase 2 ATM0002
Opened existing file CAMDART_F_2000_WACCM_VAL.cam_0002.rs.2008-11-13-21600.nc
917504
Opened existing file CAMDART_F_2000_WACCM_VAL.cam_0001.rs.2008-11-13-21600.nc
917504
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
MPI: MPI_COMM_WORLD rank 4 has terminated without calling MPI_Finalize()
MPI: aborting job
/opt/sgi/mpt/mpt-2.02/bin/mpiexec_mpt: line 53: 18958 Killed $mpicmdline_prefix -f $paramfile
ccsm.log.140916-154842._0002 lines 1658-1718/1718 (END)
Previous posts of similar problems suggest the rpointer files could be at issue, but my rpointer files all appear to have the correct restart timestamp of . A different restart run was successful when started from the output of a 6-hr cold start from 2001-01-01-00000.
Could there be another explanation for this sync problem?
George
(seq_domain_areafactinit) : min/max mdl2drv 1.00000000000000 1.00000000000000 areafact_a_ATM0002
(seq_domain_areafactinit) : min/max drv2mdl 1.00000000000000 1.00000000000000 areafact_a_ATM0002
(seq_domain_areafactinit) : min/max mdl2drv 1.00000000000000 1.00000000000000 areafact_l_LND0002
(seq_domain_areafactinit) : min/max drv2mdl 1.00000000000000 1.00000000000000 areafact_l_LND0002
(seq_domain_areafactinit) : min/max mdl2drv 1.00000000000000 1.00000000000000 areafact_i_ICE0002
(seq_domain_areafactinit) : min/max drv2mdl 1.00000000000000 1.00000000000000 areafact_i_ICE0002
(seq_mct_drv) : Initialize atm component phase 2 ATM0002
Opened existing file CAMDART_F_2000_WACCM_VAL.cam_0002.rs.2008-11-13-21600.nc
917504
Opened existing file CAMDART_F_2000_WACCM_VAL.cam_0001.rs.2008-11-13-21600.nc
917504
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
cice ymd= 20080625 cice tod= 23400
sync ymd= 20081113 sync tod= 23400
(shr_sys_abort) ERROR: ice_run_mct:: Internal sea-ice clock not in sync with Sync Clock
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
MPI: MPI_COMM_WORLD rank 4 has terminated without calling MPI_Finalize()
MPI: aborting job
/opt/sgi/mpt/mpt-2.02/bin/mpiexec_mpt: line 53: 18958 Killed $mpicmdline_prefix -f $paramfile
ccsm.log.140916-154842._0002 lines 1658-1718/1718 (END)