Wrong time stamps running CESM2.1.3

[dliu]

Dear community,

My colleagues and I are running CESM2.1.3 on Lonestar 6 at TACC UT Austin using compset 'BSSP370cmip6' and resolution 'f09_g17'. We modified the user name lists for more variables and frequencies (attached). We encountered this problem that the CAM daily output (every 5 days per file) show wrong time stamps as shown in the following screenshot.


The wrong time stamp issue also happens to 6-hourly output. Monthly, 3 hourly and interestingly, daily output every 365 days per file don't have the issue. The results of 'ncdump -v time filename' for other frequencies are also attached (results-ncdump-v-time.txt).

Regardless of the wrong time stamps, results like time series of TREFHT look fine.

After some communications with CESM experts, the issue seems to happen to Frontera at TACC once. We don't have an answer for this yet. If anyone would have any idea on what's going on, we very much appreciate your help and comment!

Many thanks,
Dunyu

 

[erik]

@peverley @nusbaume this looks like a problem in CAM that may be an important bug to fix. Can you look into this?

 

[erik]

Hi Dunyu

Thanks for pointing out this problem. This looks like an important bug to fix to me.

I'd like you to try to simplify your case as much as possible to see where it happens. This looks like it's a problem in CAM, so run a simpler F compset and see if it shows up in lower resolutions for the same output frequency. Since, this is cesm2.1.3 which is now fairly old, it's possible this has been fixed. So go to the CAM github page and see if you can find an issue for it.

I looked and didn't, but good to have more people do the same..

Issues · ESCOMP/CAM

Community Atmosphere Model. Contribute to ESCOMP/CAM development by creating an account on GitHub.

github.com

I'm pointing CAM people to this issue so we'll see what they find.

 

[dliu]

Hi Dunyu

Thanks for pointing out this problem. This looks like an important bug to fix to me.

I'd like you to try to simplify your case as much as possible to see where it happens. This looks like it's a problem in CAM, so run a simpler F compset and see if it shows up in lower resolutions for the same output frequency. Since, this is cesm2.1.3 which is now fairly old, it's possible this has been fixed. So go to the CAM github page and see if you can find an issue for it.

I looked and didn't, but good to have more people do the same..

Issues · ESCOMP/CAM

Community Atmosphere Model. Contribute to ESCOMP/CAM development by creating an account on GitHub.

github.com

I'm pointing CAM people to this issue so we'll see what they find.

Thanks much, Erik

It happens that I am running this f.e21.FWHISTBgcCrop.f09_f09_mg17.CMIP6-AMIP-WACCM.001 model. I see the same errors in day_5 and day_1 outputs.



I informed our colleagues at TACC and we currently suspect the cause may come from pnetcdf version. I will let you know if we have any progress. I will take a look at CAM github page as well.

Thanks,
Dunyu

 

[erik]

It might be good to turn pnetcdf off then in your case and see what happens. There's variables to do that for each component in
PIO_TYPENAME. You should set it to netcdf and see what it does.