ERROR: (shr_stream_verifyTCoord) ERROR: elapsed seconds must be increasing

[xiaxx666]

I was running CTSM, and got a error in cesm.log:

.........
(seq_comm_printcomms) 32 0 64 1 CPLALLESPID:
(seq_comm_printcomms) 33 0 64 1 ESP:
(seq_comm_printcomms) 34 0 64 1 CPLESP:
(seq_comm_printcomms) 35 0 64 1 ALLIACID:
(seq_comm_printcomms) 36 0 64 1 CPLALLIACID:
(seq_comm_printcomms) 37 0 64 1 IAC:
(seq_comm_printcomms) 38 0 64 1 CPLIAC:
(t_initf) Read in prof_inparm namelist from: drv_in
(t_initf) Using profile_disable= F
(t_initf) profile_timer= 4
(t_initf) profile_depth_limit= 4
(t_initf) profile_detail_limit= 2
(t_initf) profile_barrier= F
(t_initf) profile_outpe_num= 1
(t_initf) profile_outpe_stride= 0
(t_initf) profile_single_file= F
(t_initf) profile_global_stats= T
(t_initf) profile_ovhd_measurement= F
(t_initf) profile_add_detail= F
(t_initf) profile_papi_enable= F
ERROR: (shr_stream_verifyTCoord) ERROR: elapsed seconds must be increasing
Image PC Routine Line Source
cesm.exe 000000000122B10D Unknown Unknown Unknown
cesm.exe 0000000000E47C82 shr_abort_mod_mp_ 114 shr_abort_mod.F90
cesm.exe 0000000000F7CF7C shr_stream_mod_mp 2973 shr_stream_mod.F90
cesm.exe 0000000000F799A0 shr_stream_mod_mp 1057 shr_stream_mod.F90
cesm.exe 0000000000EE1CCD shr_dmodel_mod_mp 662 shr_dmodel_mod.F90
cesm.exe 0000000000F59D53 shr_strdata_mod_m 885 shr_strdata_mod.F90
cesm.exe 00000000004F85D6 datm_comp_mod_mp_ 664 datm_comp_mod.F90
cesm.exe 00000000004FE616 datm_comp_mod_mp_ 551 datm_comp_mod.F90
cesm.exe 00000000004F6E9B atm_comp_mct_mp_a 167 atm_comp_mct.F90
cesm.exe 000000000043A098 component_mod_mp_ 257 component_mod.F90
cesm.exe 0000000000428D81 cime_comp_mod_mp_ 1415 cime_comp_mod.F90
cesm.exe 00000000004370EB MAIN__ 122 cime_driver.F90
.........

This error happened when reading file drv_in
I find my atm.log have some error. It seems that this inputdata files cannot be read by time.



my CLM inputdata have those :



attached my drv_in、atm.log and cesm.log，looking forward to your replay.

 

[oleson]

I would check your time dimension on your forcing file (2012-01) to make sure it is increasing.

 

[xiaxx666]

I would check your time dimension on your forcing file (2012-01) to make sure it is increasing.

ncdump -v time ~/nhl/inputdata/atm/datm7/CLM1PT_data/1x1_maqu/2012-01.nc，then got:

time = 0, 0.02083334, 0.04166667, 0.06250001, 0.08333334, 0.1041667, 0.125,
0.1458333, 0.1666667, 0.1875, 0.2083334, 0.2291667, 0.25, 0.2708334,
0.2916667, 0.3125, 0.3333334, 0.3541667, 0.375, 0.3958334, 0.4166667,
0.4375001, 0.4583334, 0.4791667, 0.5000001, 0.5208334, 0.5416667,
0.5625001, 0.5833334, 0.6041667, 0.6250001, 0.6458334, 0.6666667,
0.6875001, 0.7083334, 0.7291667, 0.7500001, 0.7708334, 0.7916667,
0.8125001, 0.8333334, 0.8541667, 0.8750001, 0.8958334, 0.9166667,
0.9375001, 0.9583334, 0.9791667, 1, 1.020833, 1.041667, 1.0625, 1.083333,
1.104167, 1.125, 1.145833, 1.166667, 1.1875, 1.208333, 1.229167, 1.25,
1.270833, 1.291667, 1.3125, 1.333333, 1.354167, 1.375, 1.395833,
1.416667, 1.4375, 1.458333, 1.479167, 1.5, 1.520833, 1.541667, 1.5625,
1.583333, 1.604167, 1.625, 1.645833, 1.666667, 1.6875, 1.708333,
1.729167, 1.75, 1.770833, 1.791667, 1.8125, 1.833333, 1.854167, 1.875,
1.895833, 1.916667, 1.9375, 1.958333, 1.979167, 2, 2.020833, 2.041667,
2.0625, 2.083333, 2.104167, 2.125, 2.145833, 2.166667, 2.1875, 2.208333,
2.229167, 2.25, 2.270833, 2.291667, 2.3125, 2.333333, 2.354167, 2.375,
2.395833, 2.416667, 2.4375, 2.458333, 2.479167, 2.5, 2.520833, 2.541667,
2.5625, 2.583333, 2.604167, 2.625, 2.645833, 2.666667, 2.6875, 2.708333,
2.729167, 2.75, 2.770834, 2.791667, 2.8125, 2.833334, 2.854167, 2.875,
2.895834, 2.916667, 2.9375, 2.958334, 2.979167, 3, 3.020834, 3.041667,
3.0625, 3.083334, 3.104167, 3.125, 3.145834, 3.166667, 3.1875, 3.208334,
3.229167, 3.25, 3.270834, 3.291667, 3.3125, 3.333334, 3.354167, 3.375,
3.395834, 3.416667, 3.4375, 3.458334, 3.479167, 3.5, 3.520834, 3.541667,
3.5625, 3.583334, 3.604167, 3.625, 3.645834, 3.666667, 3.6875, 3.708334,
3.729167, 3.75, 3.770834, 3.791667, 3.8125, 3.833334, 3.854167, 3.875,
3.895834, 3.916667, 3.9375, 3.958334, 3.979167, 4, 4.020834, 4.041667,
4.0625, 4.083334, 4.104167, 4.125, 4.145834, 4.166667, 4.1875, 4.208334,
4.229167, 4.25, 4.270834, 4.291667, 4.3125, 4.333334, 4.354167, 4.375,
4.395834, 4.416667, 4.4375, 4.458334, 4.479167, 4.5, 4.520834, 4.541667,
4.5625, 4.583334, 4.604167, 4.625, 4.645834, 4.666667, 4.6875, 4.708334,
4.729167, 4.75, 4.770834, 4.791667, 4.8125, 4.833334, 4.854167, 4.875,
4.895834, 4.916667, 4.9375, 4.958334, 4.979167, 5, 5.020834, 5.041667,
5.0625, 5.083334, 5.104167, 5.125, 5.145834, 5.166667, 5.1875, 5.208334,
5.229167, 5.25, 5.270834, 5.291667, 5.3125, 5.333334, 5.354167, 5.375,
5.395834, 5.416667, 5.4375, 5.458334, 5.479167, 5.5, 5.520834, 5.541667,
5.5625, 5.583334, 5.604167, 5.625, 5.645834, 5.666667, 5.6875, 5.708334,
5.729167, 5.75, 5.770834, 5.791667, 5.8125, 5.833334, 5.854167, 5.875,
5.895834, 5.916667, 5.9375, 5.958334, 5.979167, 6, 6.020834, 6.041667,
6.0625, 6.083334, 6.104167, 6.125, 6.145834, 6.166667, 6.1875, 6.208334,
6.229167, 6.25, 6.270834, 6.291667, 6.3125, 6.333334, 6.354167, 6.375,
6.395834, 6.416667, 6.4375, 6.458334, 6.479167, 6.5, 6.520834, 6.541667,
6.5625, 6.583334, 6.604167, 6.625, 6.645834, 6.666667, 6.6875, 6.708334,
6.729167, 6.75, 6.770834, 6.791667, 6.8125, 6.833334, 6.854167, 6.875,
6.895834, 6.916667, 6.9375, 6.958334, 6.979167, 7.000001, 7.020834,
7.041667, 7.062501, 7.083334, 7.104167, 7.125001, 7.145834, 7.166667,
7.187501, 7.208334, 7.229167, 7.250001, 7.270834, 7.291667, 7.312501,
7.333334, 7.354167, 7.375001, 7.395834, 7.416667, 7.437501, 7.458334,
7.479167, 7.500001, 7.520834, 7.541667, 7.562501, 7.583334, 7.604167,
7.625001, 7.645834, 7.666667, 7.687501, 7.708334, 7.729167, 7.750001,
.............

Because of the word limit, I put the output into a file ' time.log' attached at the end

I'm sure the time is increasing.

 

[oleson]

The file you checked with ncdump (~/nhl/inputdata/atm/datm7/CLM1PT_data/1x1_maqu/2012-01.nc) seems to have a different path than the one the model is reading in (/work/home/zky_niuhl/nhl/software/CTSM/inputdata/atm/datm7/CLM1PT_data/1x1_maqu/2012-01.nc). Maybe they are indeed the same file or are linked somehow, but I'd make sure you are checking the right file.
Otherwise, I agree that the time dimension is increasing in that file, yet the model seems to think that the seconds are the same in the first and second time slices (secs(n), secs(n+1) = 00000000 00000000). You might have to trace the calculation of secs in the code to see why (cime/src/share/streams/shr_stream_mod.F90).

 

[xiaxx666]

The file you checked with ncdump (~/nhl/inputdata/atm/datm7/CLM1PT_data/1x1_maqu/2012-01.nc) seems to have a different path than the one the model is reading in (/work/home/zky_niuhl/nhl/software/CTSM/inputdata/atm/datm7/CLM1PT_data/1x1_maqu/2012-01.nc). Maybe they are indeed the same file or are linked somehow, but I'd make sure you are checking the right file.
Otherwise, I agree that the time dimension is increasing in that file, yet the model seems to think that the seconds are the same in the first and second time slices (secs(n), secs(n+1) = 00000000 00000000). You might have to trace the calculation of secs in the code to see why (cime/src/share/streams/shr_stream_mod.F90).

Thank you for your reminder.
I found the related variable secs in the shr_ stream_ mod.F90 file. I'm not familiar with the fortran language. Can I understand the definition of 'secs' as one second? In other words, are these time data normal in 2012-01.nc?
I see other times of error posts，and there is no such small data as mine.

 

[oleson]

secs is calculated from the date that is read in (in units of seconds). The difference between the first two time slices of your data is 0.0208334 and the units are determined by the "units" attribute of the time dimension. In your case, this seems to be "days since 2012-01-01 00:00:00". So I would expect the second time slice to be 0.0208334 X seconds in a day = 0.0208334 X 86400 = 1800 seconds. That all seems normal. I'm not sure why your secs(n+1) = secs2 = 0.

 

[oleson]

You could also check the aerosol deposition and topo files to make sure they have data and aren't corrupted.

/work/home/zky_niuhl/nhl/inputdata/atm/cam/chem/trop_mozart_aero/aero/aerosoldep_WACCM.ensmean_monthly_hist_1849-2015_0.9x1.25_CMIP6_c180926.nc

/work/home/zky_niuhl/nhl/inputdata/atm/datm7/topo_forcing/topodata_0.9x1.25_USGS_070110_stream_c151201.nc

It's possible the error is referring to one of those files.

 

[xiaxx666]

You could also check the aerosol deposition and topo files to make sure they have data and aren't corrupted.

/work/home/zky_niuhl/nhl/inputdata/atm/cam/chem/trop_mozart_aero/aero/aerosoldep_WACCM.ensmean_monthly_hist_1849-2015_0.9x1.25_CMIP6_c180926.nc

/work/home/zky_niuhl/nhl/inputdata/atm/datm7/topo_forcing/topodata_0.9x1.25_USGS_070110_stream_c151201.nc

It's possible the error is referring to one of those files.

Thank you Thank you.I found my aerosoldep_WACCM.ensmean_monthly_hist_1849-2015_0.9x1.25_CMIP6_c180926.nc is only 961KB.
ncdump -v time aerosoldep_WACCM.ensmean_monthly_hist_1849-2015_0.9x1.25_CMIP6_c180926.nc , then got：

I'll redownload and try again.

 

[xiaxx666]

You could also check the aerosol deposition and topo files to make sure they have data and aren't corrupted.

/work/home/zky_niuhl/nhl/inputdata/atm/cam/chem/trop_mozart_aero/aero/aerosoldep_WACCM.ensmean_monthly_hist_1849-2015_0.9x1.25_CMIP6_c180926.nc

/work/home/zky_niuhl/nhl/inputdata/atm/datm7/topo_forcing/topodata_0.9x1.25_USGS_070110_stream_c151201.nc

It's possible the error is referring to one of those files.

Yes！Yes！ That's why，thank you so much !

 

[jinping F]

You could also check the aerosol deposition and topo files to make sure they have data and aren't corrupted.

/work/home/zky_niuhl/nhl/inputdata/atm/cam/chem/trop_mozart_aero/aero/aerosoldep_WACCM.ensmean_monthly_hist_1849-2015_0.9x1.25_CMIP6_c180926.nc

/work/home/zky_niuhl/nhl/inputdata/atm/datm7/topo_forcing/topodata_0.9x1.25_USGS_070110_stream_c151201.nc

It's possible the error is referring to one of those files.

Hi! Can you re-upload this file (/inputdata/atm/datm7/topo_forcing/topodata_0.9x1.25_USGS_070110_stream_c151201.nc ) ?
Because this has an error that shows the data to 0, I don 't know why this happens.

[fengjinping@swarm01 topo_forcing]$ ncdump -v time old_topodata_0.9x1.25_USGS_070110_stream_c151201.nc
netcdf old_topodata_0.9x1.25_USGS_070110_stream_c151201 {
dimensions:
lat = 192 ;
lon = 288 ;
time = 1 ;
variables:
double LATIXY(lat, lon) ;
LATIXY:long_name = "latitude" ;
LATIXY:units = "degrees north" ;
double LONGXY(lat, lon) ;
LONGXY:long_name = "longitude" ;
LONGXY:units = "degrees east" ;
int NUMLON(lat) ;
NUMLON:long_name = "number of grid cells at each latitude" ;
NUMLON:units = "unitless" ;
double TOPO(time, lat, lon) ;
TOPO:long_name = "topography height" ;
TOPO:units = "m" ;
double area(lat, lon) ;
area:long_name = "grid cell area" ;
area:units = "radian2" ;
int mask(lat, lon) ;
mask:long_name = "mask of valid data (0 = invalid, 1 = valid)" ;
mask:units = "unitless" ;
double time(time) ;
time:long_name = "time" ;
time:units = "days since 0001-01-01 00:00:00" ;
time:calendar = "noleap" ;

// global attributes:
:Conventions = "NCAR-CSM" ;
:History_Log = "created on: 01-05-07 14:25:29" ;
:Logname = "nanr" ;
:Host = "bs1201en" ;
:fcamfile = "/fs/cgd/csm/inputdata/atm/cam/topo/USGS-gtopo30_0.9x1.25_remap_c051027.nc" ;
:fccsmdom = "" ;
:fcamtopo = "" ;
:Other_Info = "" ;
:Topo_Input_Filename = "" ;
:Source = "Community Land Model: CLM3" ;
:history = "Tue Dec 1 13:55:06 2015: ncks -x -v TOPO_old topodata_0.9x1.25_USGS_070110_stream_c151201.nc topodata_new.nc" ;
:NCO = "4.4.2" ;
data:

time = 0 ;
}

 

[slevis]

I suspect that the important data in this file are in the variable TOPO. Are you sure you care about the time variable? As far as I can tell, the file is fine.

 

[jinping F]

Thank you for your timely reply.The reason why I care about this file is because I read this post to see if I can solve the problem in this way.
I encountered an error running '. / case.submit '.
(shr_stream_verifyTCoord) ERROR: elapsed seconds on a date must be strickly increasing
(shr_stream_verifyTCoord) secs(n), secs(n+1) = 00075600 00000000
ERROR: (shr_stream_verifyTCoord) ERROR: elapsed seconds must be increasing
I try my best to solve the problem by using 'shr_stream_verifyTCoord ' as the keyword to search in the forum. The error ' 00075600 ' was not found.
Attach the running files. Hope to get your reply, let me have some direction.

 

[oleson]

I think time = 0 is normal for the topo file. You could check your aerosol deposition file to make sure it is not corrupted.

 

[jinping F]

I directly re-downloaded the following three files and re-. / case.submit. But the error still exists. Same as the four attachment errors above me.

aerosoldep_WACCM.ensmean_monthly_hist_1849-2015_0.9x1.25_CMIP6_c180926.nc
fco2_datm_global_simyr_1750-2014_CMIP6_c180929.nc
topodata_0.9x1.25_USGS_070110_stream_c151201.nc

The content of this attachment is what is displayed in “/home/fengjinping/scratch/cesmlab/atm/cam/chem/trop_mozart_aero/aero/aerosoldep_WACCM.ensmean_monthly_hist_1849-2015_0.9x1.25_CMIP6_c180926.nc".

Can you share these three files to me separately ? And put forward some guiding suggestions on my mistakes, thank you !

 

[oleson]

The time variable on the aerosol file looks ok to me. I would do an ncdump -v time on the fco2 file as well. The error could also be associated with the atm forcing files, so you could check those as well. Also, you have the years in datm set to:

1901 1979 2015

There isn't any 2015 GSWP3 data available. And it seems you are aligning model year 1901 with 1979 forcing data?

 

[jinping F]

The contents of "ncdump -v time /home/fengjinping/scratch/cesmlab/atm/datm7/CO2/fco2_datm_global_simyr_1750-2014_CMIP6_c180929.nc" can be found in the attachment.

I use my own forced data to drive CLM5, this post(Run CLM with my own atmosphere forcing data) gives me a lot of inspiration.

I set up the process and related documents attached as an annex, I hope you can give me some suggestions, may datm_in Settings have some problems.The forcing data is 0.1 × 0.1 °, fsurdat and domain and flanduse _ timeserie are 0.125 × 0.125 °. Are these two non-uniform effects ?

 

[oleson]

The co2 file looks ok. Now that I know you are using your own atmospheric forcing, I would look at the time variable in your 1979-01 files, and in your 2018-12.nc files. Since you are starting in 1979, the datm is probably trying to get a time slice from the end of the 2018-12 file to do interpolation.

The flanduse_timeseries file you've specified is:

/home/user/cesmlab/lnd/clm2/surfdata_map/release-clm5.0.18/landuse.timeseries_0.9x1.25_SSP1-2.6_78pfts_CMIP6_simyr1850-2100_c181220.nc

Is that actually at 0.125 resolution, or is it 0.9x1.25 as indicated by the filename?