Error related to the CORE2.10.TArcFactor data stream

[sur23beeb]

Hi,
I am running cesm with active ocean/ice components and forced by atmospheric reanalysis (JRA). The ocean grid is tx0.1v3.

resolution: TL319_t13
compset: 2000_DATM%JRA_SLND_CICE_POP2_DROF%JRA_SGLC_SWAV
cesmtag: cesm2.1.4-rc.08

I have finished 34 years without any issues but for some reason my latest restart run stopped midway into the run. I do my restart runs in two-year chunks and the latest run stopped exactly at the end of year 1. The error appears to be related to the dtlimit value corresponding to the TArcFactor data stream. Here is the output from the last part of the atm.log file:

(datm_comp_run) atm: model date  2831231   81600s
(datm_comp_run) atm: model date  2831231   82200s
(datm_comp_run) atm: model date  2831231   82800s
(datm_comp_run) atm: model date  2831231   83400s
(datm_comp_run) atm: model date  2831231   84000s
(datm_comp_run) atm: model date  2831231   84600s
(datm_comp_run) atm: model date  2831231   85200s
(datm_comp_run) atm: model date  2831231   85800s
(datm_comp_run) atm: model date  2840101       0s
(shr_dmodel_readstrm) file ub: /ihesp/obs_root/inputdata/ocn/jra55/v1.3_noleap/JRA.v1.3.swdn.TL319.1997.171019.nc       2
(shr_dmodel_readstrm) file ub: /ihesp/obs_root/inputdata/ocn/jra55/v1.3_noleap/JRA.v1.3.q_10.TL319.1997.171019.nc       2
(shr_dmodel_readstrm) file ub: /ihesp/obs_root/inputdata/ocn/jra55/v1.3_noleap/JRA.v1.3.slp.TL319.1997.171019.nc       2
(shr_dmodel_readstrm) file ub: /ihesp/obs_root/inputdata/ocn/jra55/v1.3_noleap/JRA.v1.3.t_10.TL319.1997.171019.nc       2
(shr_dmodel_readstrm) file ub: /ihesp/obs_root/inputdata/ocn/jra55/v1.3_noleap/JRA.v1.3.u_10.TL319.1997.171019.nc       2
(shr_dmodel_readstrm) file ub: /ihesp/obs_root/inputdata/ocn/jra55/v1.3_noleap/JRA.v1.3.v_10.TL319.1997.171019.nc       2
(shr_dmodel_readstrm) file ub: /ihesp/obs_root/inputdata/atm/datm7/CORE2/CORE2.t_10.ArcFactor.T62.1997-2004.nc       2
(shr_strdata_advance) ERROR: for stream            9
(shr_strdata_advance) ERROR: dt limit1    19345.0000000000    
   14.0000000000000        1.50000000000000    
(shr_strdata_advance) ERROR: dt limit2      2840101           0     2840115
           0
ERROR: (shr_strdata_advance) ERROR dt limit for stream

Here are the dtlimit values in the datm_in file in the run/ folder:

dtlimit = 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5

I ran ncdump on the CORE2.t_10.ArcFactor.T62.1997-2004.nc file. It has 98 timestamps which span a period of 8 years. What I don't understand is why the run is failing now after running for 34 years.

Have other users seen this kind of error in their simulations?

 

[sur23beeb]

Hi this got sorted out and I wanted to provide an update in case somebody has a similar issue. I also finally understood what purpose dtlimit serves.

1. My run stopped at the beginning of year 40, not 35 as I had implied previously.
2. The ArcFactor data stream refers to a correction for the Arctic air temperature which kicks in only for years 1997-2004. My simulation began in 1958 which would make 1997 year 40.
3. This means until the beginning of year 40, that data stream was not even being used, so the dtlimit value for that data stream did not matter.
4. The dtlimit value constrains the ratio of maximum to minimum temporal spacing (in days) between two successive values in a data stream. It is probably there to catch potentially erroneous gaps in the data stream. A data stream with values available daily would have dtlimit = 1. For a monthly stream, the dtlimit value would be 31/28 and so on.
5. The ArcFactor data stream has only 98 values spanning the entire duration 1997-2004, so it is clearly not a daily stream and should have a dtlimit greater than 1. Because it was set to 1, however, the code threw an error as the real dtlimit exceeds 1 for this data stream.
6. This can be fixed by computing the actual maximum and minimum time gaps (in days) for the ArcFactor data stream and setting dtlimit equal to the ratio of max/min. Or it can be fixed by just setting dtlimit for that particular data stream to a really large value (e.g., 1e30), which ensures this value will not be exceeded even in irregularly spaced data streams with large gaps (like the ArcFactor stream).
7. I changed dtlimit for the data stream to 1e30 and the code now works properly.