postprocessing timeseries has probems with monthly history files

[TriD]

Dear CESM list,

I’m having some trouble with the creation of timeseries based on the postprocessing code and I’m hoping you can help. It appears to be working just fine for the 3-hourly and daily history files, but with very similar configurations for monthly history files, the logs suggest that it’s chugging away with creating a timeseries, but nothing is actually output to the folders.

Is there something in the xml that I’m missing here such that it’s pointing it to another directory?

Thanks in advance,

R. Tri Datta


Xml entry for 3 hourly timeseries (working):
<file_extension suffix=".h2.[0-9]">

<subdir>hist</subdir>

<tseries_create>TRUE</tseries_create>

<tseries_output_format>netcdf4c</tseries_output_format>

<tseries_tper>hour_3</tseries_tper>

<tseries_filecat_tper>years</tseries_filecat_tper>

<tseries_filecat_n>5</tseries_filecat_n>

</file_extension>


Xml entry for monthly timeseries (NOT working):

<file_extension suffix=".h0.[0-9]">

<subdir>hist</subdir>

<tseries_create> TRUE </tseries_create>

<tseries_output_format>netcdf4c</tseries_output_format>

<tseries_tper>month_1</tseries_tper>

<tseries_filecat_tper>years</tseries_filecat_tper>

<tseries_filecat_n>50</tseries_filecat_n>

</file_extension>

 

[strandwg]

Dear CESM list,

I’m having some trouble with the creation of timeseries based on the postprocessing code and I’m hoping you can help. It appears to be working just fine for the 3-hourly and daily history files, but with very similar configurations for monthly history files, the logs suggest that it’s chugging away with creating a timeseries, but nothing is actually output to the folders.

Is there something in the xml that I’m missing here such that it’s pointing it to another directory?

Thanks in advance,

R. Tri Datta


Xml entry for 3 hourly timeseries (working):
<file_extension suffix=".h2.[0-9]">

<subdir>hist</subdir>

<tseries_create>TRUE</tseries_create>

<tseries_output_format>netcdf4c</tseries_output_format>

<tseries_tper>hour_3</tseries_tper>

<tseries_filecat_tper>years</tseries_filecat_tper>

<tseries_filecat_n>5</tseries_filecat_n>

</file_extension>


Xml entry for monthly timeseries (NOT working):

<file_extension suffix=".h0.[0-9]">

<subdir>hist</subdir>

<tseries_create> TRUE </tseries_create>

<tseries_output_format>netcdf4c</tseries_output_format>

<tseries_tper>month_1</tseries_tper>

<tseries_filecat_tper>years</tseries_filecat_tper>

<tseries_filecat_n>50</tseries_filecat_n>

</file_extension>

Are the spaces in around TRUE in "<tseries_create> TRUE </tseries_create>" a potential problem? I wouldn't think so, but that's the only difference I see between the two sets.

 

[TriD]

Hi Gary,

Apologies for the late response. I'm afraid that changing the spaces made no difference. I think that the timeseries postprocessing has some known issues, but the thing is that it IS working for 3_hourly data under the condition that the variable is 1-D (variables at multiple levels have problems). I don't know if this helps isolate the particular bug.

 

[jedwards]

Please try again using the code in /glade/u/home/jedwards/sandboxes/CESM_postprocessing
or checkout branch cheyenne_update_py3

and let me know if you still have the same issue.

 

[TriD]

Hi Jim, I tried both codebases (just to be sure) and I'm getting the following error. Is there a newer codebase?

Traceback (most recent call last):


File "./cesm_tseries_generator.py", line 406, in <module>
debugMsg('Running on {0} cores'.format(size), header=True)
File "/glade/work/jedwards/sandboxes/CESM_postprocessing/cesm-env2/lib/python3.7/site-packages/asaptools/vprinter.py", line 109, in __call__
if verbosity < self.verbosity:
TypeError: '<' not supported between instances of 'int' and 'list'

 

[jedwards]

I think that it's fixed, please try again.

 

[TriD]

Hi Jim, I tried the new codebase and it looks like it's creating the folders, but not actually depositing any files there. Is there maybe something wrong with my setup? As an example snippet (full file in :

<file_extension suffix=".h1.[0-9]">
<subdir>hist</subdir>
<tseries_create>TRUE</tseries_create>
<tseries_output_format>netcdf4c</tseries_output_format>
<tseries_tper>month_1</tseries_tper>
<tseries_filecat_tper>years</tseries_filecat_tper>
<tseries_filecat_n>10</tseries_filecat_n>
</file_extension>
<file_extension suffix=".h2.[0-9]">
<subdir>hist</subdir>
<tseries_create>TRUE</tseries_create>
<tseries_output_format>netcdf4c</tseries_output_format>
<tseries_tper>hour_3</tseries_tper>
<tseries_filecat_tper>years</tseries_filecat_tper>
<tseries_filecat_n>1</tseries_filecat_n>
</file_extension>

From the logs:
42/144 opening /glade/scratch/rtdatta/archive/cesm2.2.0_FHIST_ne0np4.ANTSI.ne30x4_mt12_3600pes_210420_Nx4yR/lnd/hist/cesm2.2.0_FHIST_ne0np4.ANTSI.ne30x4_mt12_3600pes_210420_Nx4yR.clm2.h0.2005-08.nc
142/144 opening /glade/scratch/rtdatta/archive/cesm2.2.0_FHIST_ne0np4.ANTSI.ne30x4_mt12_3600pes_210420_Nx4yR/lnd/hist/cesm2.2.0_FHIST_ne0np4.ANTSI.ne30x4_mt12_3600pes_210420_Nx4yR.clm2.h2.1985-12-27-00000.nc
time_period_freq = hour_3

 

[oleson]

Jim asked me to look at this since I frequently run the timeseries postprocessing on land files.
I tried it using my own sandbox (/glade/work/oleson/postprocessing) and it worked fine.
My sandbox is fairly old but it should be fine for creating timeseries files, so you could point to it if you want.

I also tried JIm's sandbox (/glade/work/jedwards/sandboxes/CESM_postprocessing) and, as you described, it did find the history files and it created the subdirectories and is running, but didn't create any files.
The only possible error I see in the log file is:

Verbosity level must be an integer.