MOSART clock is not in sync with Master Sync clock

[KeerZ]

Hello all,

I set up a branch run using I2000Clm50SpGs compset. I changed the forcing data to be CRUJRA dataset and used individual soil column by modifying the scripts.

The first-year simulation was finished successfully. Then I specified ./xmlchange STOP_OPTION=nmonths,STOP_N=12,RESUBMIT=2,JOB_WALLCLOCK_TIME=6:00:00 and submitted another two-year simulations. This time I encountered an error and in the cesm log fils it says

467: rof_run_mct :: MOSART clock is not in sync with Master Sync
467: clock
106: mosart ymd= 940101 mosart tod= 10800
106: sync ymd= 950101 sync tod= 10800
106: ERROR:
106: rof_run_mct :: MOSART clock is not in sync with Master Sync
106: clock
322: mosart ymd= 940101 mosart tod= 10800
322: sync ymd= 950101 sync tod= 10800
322: ERROR:
322: rof_run_mct :: MOSART clock is not in sync with Master Sync
322: clock

I also attached the cesm log file. It seems that my simulation year is 0095-01-01 but MOSART year is still 0094-01-01. But I did not modify anything related to MOSART. I am not sure why this error happened and how I can resolve it. Can you give me some suggestions? Thank you!

- My case can be found at /glade/u/home/keerzhang/cases/i.e20.I2000.f02_g16.URBAN_CRU09_15

 

[erik]

OK, so the thing to track down is why MOSART didn't write it's restart files out when the other components did for Jan/1 of year 95. Look at the log files for the simulation that ran to see what happened. Especially look at the rof.log file to see what restarts where written out by MOSART. What you do then depends on how many restarts you saved. If you have restarts for all components for Jan/1 of year 94, you can likely restart from that point, but you'll have to make sure all the restart pointer files point to the right files and those restarts are in your $RUNDIR.

Hopefully, that makes sense and helps.

 

[KeerZ]

OK, so the thing to track down is why MOSART didn't write it's restart files out when the other components did for Jan/1 of year 95. Look at the log files for the simulation that ran to see what happened. Especially look at the rof.log file to see what restarts where written out by MOSART. What you do then depends on how many restarts you saved. If you have restarts for all components for Jan/1 of year 94, you can likely restart from that point, but you'll have to make sure all the restart pointer files point to the right files and those restarts are in your $RUNDIR.

Hopefully, that makes sense and helps.

I found that the MOSART restart file for 0095-01-01 was written successfully in the $RUNDIR. But for some reason, the MOSART restart file for 0094-01-01 was not removed. Not sure whether this is the reason for the MOSART clock error.

Yes, as you said, I branched the case using the latest restart file I have and the error did not happen again. Thank you, Erik!

 

[CGL]

I found that the MOSART restart file for 0095-01-01 was written successfully in the $RUNDIR. But for some reason, the MOSART restart file for 0094-01-01 was not removed. Not sure whether this is the reason for the MOSART clock error.

Yes, as you said, I branched the case using the latest restart file I have and the error did not happen again. Thank you, Erik!

I got the same error. I moved all of MOSART restart file to the new case. Still, i got the same error:
rof_run_mct :: MOSART clock is not in sync with Master Sync
clock

 

[CGL]

I got the same error. I moved all of MOSART restart file to the new case. Still, i got the same error:
rof_run_mct :: MOSART clock is not in sync with Master Sync
clock
View attachment 3834

Now, I change the REF_DATE to the newest date, the model seems like work.