Hello, everyone！
Thanks for opening this thread. I am running fully coupled climate simulation (use B1850, ) with CESM1.2. The ocean tracer module is set to "iage ecosys" for this simulation. The model has been running successfully for 3000 years.When I change to 'hybrid' continue to run the next (I also change the dtime by modifying ATM_NCPL in env_run.xml file, because i wanna to reduce the timestep to avoid numerical instability) , but it failed.
I set "ATM_NCPL"=72 (dtime=1200 seconds), the simulation is also paused, cesm.log file prompt that “ERROR: ocn_run_mct: Internal pop2 clock not in sync with Sync Clock”. The error message I got is mainly from POP2 module.
Here is the last few lines from cesm.log file:
2393 --------------------------------------------------------------------------
2394 pop2 ymd= 30010102 pop2 tod= 0
2395 sync ymd= 10103 sync tod= 0
2396 Internal pop2 clock not in sync with Sync Clock
2397 pop2 ymd= 30010102 pop2 tod= 0
2398 sync ymd= 10103 sync tod= 0
2399 Internal pop2 clock not in sync with Sync Clock
2400 --------------------------------------------------------------------------
2401 MPI_ABORT was invoked on rank 40 in communicator MPI_COMM_WORLD
2402 with errorcode 1001.
2403
2404 NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
2405 You may or may not see output from other processes, depending on
2406 exactly when Open MPI kills them.
2407 --------------------------------------------------------------------------
2408 (shr_sys_abort) ERROR: ocn_run_mct:: Internal pop2 clock not in sync with Sync Clock
2409 (shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
2410 --------------------------------------------------------------------------
I also checked the ocn.log file:
5315 (io_pio_init) create file ./AI_500X.pop.h.ecosys.nday1.3001-01-01.nc
5316
5317 data appended to tavg file: ./AI_500X.pop.h.ecosys.nday1.3001-01-01.nc
5318 pop2 ymd= 30010102 pop2 tod= 0
5319 sync ymd= 10103 sync tod= 0
5320 Internal pop2 clock not in sync with Sync Clock
5321 (shr_sys_abort) ERROR: ocn_run_mct:: Internal pop2 clock not in sync with Sync Clock
5322 (shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
--------------------------------------------------------------------------
Attached is my cesm.log and ocn.log file.
--------------------------------------------------------------------------
I'm a little confused because I didn't change the dtime of the pop module. I don't know what went wrong, do you have any solution to solve this error. Maybe it has something to do with cpl.
I hope to get your help. I would really appreciate your help!
Thank you!
Best,Chen
Thanks for opening this thread. I am running fully coupled climate simulation (use B1850, ) with CESM1.2. The ocean tracer module is set to "iage ecosys" for this simulation. The model has been running successfully for 3000 years.When I change to 'hybrid' continue to run the next (I also change the dtime by modifying ATM_NCPL in env_run.xml file, because i wanna to reduce the timestep to avoid numerical instability) , but it failed.
I set "ATM_NCPL"=72 (dtime=1200 seconds), the simulation is also paused, cesm.log file prompt that “ERROR: ocn_run_mct: Internal pop2 clock not in sync with Sync Clock”. The error message I got is mainly from POP2 module.
Here is the last few lines from cesm.log file:
2393 --------------------------------------------------------------------------
2394 pop2 ymd= 30010102 pop2 tod= 0
2395 sync ymd= 10103 sync tod= 0
2396 Internal pop2 clock not in sync with Sync Clock
2397 pop2 ymd= 30010102 pop2 tod= 0
2398 sync ymd= 10103 sync tod= 0
2399 Internal pop2 clock not in sync with Sync Clock
2400 --------------------------------------------------------------------------
2401 MPI_ABORT was invoked on rank 40 in communicator MPI_COMM_WORLD
2402 with errorcode 1001.
2403
2404 NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
2405 You may or may not see output from other processes, depending on
2406 exactly when Open MPI kills them.
2407 --------------------------------------------------------------------------
2408 (shr_sys_abort) ERROR: ocn_run_mct:: Internal pop2 clock not in sync with Sync Clock
2409 (shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
2410 --------------------------------------------------------------------------
I also checked the ocn.log file:
5315 (io_pio_init) create file ./AI_500X.pop.h.ecosys.nday1.3001-01-01.nc
5316
5317 data appended to tavg file: ./AI_500X.pop.h.ecosys.nday1.3001-01-01.nc
5318 pop2 ymd= 30010102 pop2 tod= 0
5319 sync ymd= 10103 sync tod= 0
5320 Internal pop2 clock not in sync with Sync Clock
5321 (shr_sys_abort) ERROR: ocn_run_mct:: Internal pop2 clock not in sync with Sync Clock
5322 (shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping
--------------------------------------------------------------------------
Attached is my cesm.log and ocn.log file.
--------------------------------------------------------------------------
I'm a little confused because I didn't change the dtime of the pop module. I don't know what went wrong, do you have any solution to solve this error. Maybe it has something to do with cpl.
I hope to get your help. I would really appreciate your help!
Thank you!
Best,Chen