How to use my own forcing data to drive CLM single site simulation.

[jinmuluo]

I do not have much experience looking at datm code, so you will need to investigate yourself or ask someone else.

Alternatively, it may be easier to initially compare the structure of your file in detail against one of the equivalent files provided with the CTSM. Try to look for anything that looks different that might explain a failure.

Hi Sam,

Thank you for your suggestions. I guess one difference between my dataset with the GSWP3 is the time format, it has to be cftime.DatetimeNoLeap?

array([cftime.DatetimeNoLeap(2014, 12, 1, 0, 0, 0, 0, has_year_zero=True),
cftime.DatetimeNoLeap(2014, 12, 1, 3, 0, 0, 0, has_year_zero=True),
cftime.DatetimeNoLeap(2014, 12, 1, 6, 0, 0, 0, has_year_zero=True), ...,
cftime.DatetimeNoLeap(2014, 12, 31, 15, 0, 0, 0, has_year_zero=True),
cftime.DatetimeNoLeap(2014, 12, 31, 18, 0, 0, 0, has_year_zero=True),
cftime.DatetimeNoLeap(2014, 12, 31, 21, 0, 0, 0, has_year_zero=True)],
dtype=object)

another question is the dimension of the dataset, what's the meaning of this scalar?

 

[jinmuluo]

Hi Sam,

Thank you for your suggestions. I guess one difference between my dataset with the GSWP3 is the time format, it has to be cftime.DatetimeNoLeap?

array([cftime.DatetimeNoLeap(2014, 12, 1, 0, 0, 0, 0, has_year_zero=True),
cftime.DatetimeNoLeap(2014, 12, 1, 3, 0, 0, 0, has_year_zero=True),
cftime.DatetimeNoLeap(2014, 12, 1, 6, 0, 0, 0, has_year_zero=True), ...,
cftime.DatetimeNoLeap(2014, 12, 31, 15, 0, 0, 0, has_year_zero=True),
cftime.DatetimeNoLeap(2014, 12, 31, 18, 0, 0, 0, has_year_zero=True),
cftime.DatetimeNoLeap(2014, 12, 31, 21, 0, 0, 0, has_year_zero=True)],
dtype=object)

another question is the dimension of the dataset, what's the meaning of this scalar?
View attachment 3997

and my dataset looks like that

 

[slevis]

I don't know the answers to your questions, but I would suggest that you make yours look more like GSWP3, since we know that GSWP3 works.

 

[jinmuluo]

@oleson​

Hi Oleson,

Do you know what time format of the NetCDF file CLM needs? I tried to use the same format merra2 has but it looks like unsuccessfully.

My file's time variable
------------------------------------
array(['2022-12-01T00:00:00.000000000', '2022-12-01T01:00:00.000000000',
'2022-12-01T02:00:00.000000000', ..., '2022-12-31T21:00:00.000000000',
'2022-12-31T22:00:00.000000000', '2022-12-31T23:00:00.000000000'],
dtype='datetime64[ns]')

GSWP3
------------------------------------
array([cftime.DatetimeNoLeap(2014, 12, 1, 0, 0, 0, 0, has_year_zero=True),
cftime.DatetimeNoLeap(2014, 12, 1, 3, 0, 0, 0, has_year_zero=True),
cftime.DatetimeNoLeap(2014, 12, 1, 6, 0, 0, 0, has_year_zero=True), ...,
cftime.DatetimeNoLeap(2014, 12, 31, 15, 0, 0, 0, has_year_zero=True),
cftime.DatetimeNoLeap(2014, 12, 31, 18, 0, 0, 0, has_year_zero=True),
cftime.DatetimeNoLeap(2014, 12, 31, 21, 0, 0, 0, has_year_zero=True)],
dtype=object)

and I don't know how to convert the "numpy.datetime" to "cftime.DatatimeNoLeap". Do you happen to know any study resources to learn about the NetCDF CLM needs?

best,

Jinmu

 

[oleson]

There is some documentation here, e.g., for release-cesm2.1.3:

/glade/work/oleson/release-cesm2.1.3/cime/src/components/data_comps/datm/doc/streams.xml

It says "The input data must be in a netcdf file and the time axis in that file must be CF-1.0 compliant."

You can search for what that means, e.g.,

The CF Metadata Convention - Centre for Environmental Analysis (CEDA) Help Docs Site

Introduction In addition to the basic data in a netCDF file, the creator of the dataset needs to include information about the data themselves (e.g. type, units

help.ceda.ac.uk

 

[oleson]

Maybe leave off the decimal places in your time variable. I.e., instead of:

time:units = "hours since 2014-01-01 00:00:00.000000

do:

time:units = "hours since 2014-01-01 00:00:00"

to follow more closely the GSWP3 convention.

 

[jinmuluo]

Maybe leave off the decimal places in your time variable. I.e., instead of:

time:units = "hours since 2014-01-01 00:00:00.000000

do:

time:units = "hours since 2014-01-01 00:00:00"

to follow more closely the GSWP3 convention.

Thank you for your suggestion! My meteorological dataset is in one hour step which is finer than CLM default (3 hours), anything I should do with the namelist input?

Best,

 

[oleson]

For this type of situation we usually run the model at a time step that is equal to the forcing time step. So in your case, you'd run the model at 1 hour time steps. You can set this in env_run.xml using ATM_NCPL. For a one hour forcing and model time step, set ATM_NCPL=24.
Furthermore, you would then not need to do any kind of time interpolation on the forcing data, e.g., coszen for solar. So you would set "tintalgo" in your streams file to "nearest" for all of the streams. It's not necessary but you could also put all of the forcing data into one file since all of the streams can now be treated similarly.

If you are still having trouble with your run, you could try compiling and running in DEBUG mode which might provide more useful error output. DEBUG=TRUE in env_build.xml

 

[jinmuluo]

For this type of situation we usually run the model at a time step that is equal to the forcing time step. So in your case, you'd run the model at 1 hour time steps. You can set this in env_run.xml using ATM_NCPL. For a one hour forcing and model time step, set ATM_NCPL=24.
Furthermore, you would then not need to do any kind of time interpolation on the forcing data, e.g., coszen for solar. So you would set "tintalgo" in your streams file to "nearest" for all of the streams. It's not necessary but you could also put all of the forcing data into one file since all of the streams can now be treated similarly.

If you are still having trouble with your run, you could try compiling and running in DEBUG mode which might provide more useful error output. DEBUG=TRUE in env_build.xml

Hi Oleson,

Thank you so much for your help! I have another question about the time format of the NetCDF files. I saw the time variable in GSWP3 is in the unit of "Days since yyyy-mm-dd", can CLM read the unit and time index of "hours since yyyy-mm-dd"?

Best,

Jinmu

 

[oleson]

The datm should be fine reading "hours since".

 

[jinmuluo]

For this type of situation we usually run the model at a time step that is equal to the forcing time step. So in your case, you'd run the model at 1 hour time steps. You can set this in env_run.xml using ATM_NCPL. For a one hour forcing and model time step, set ATM_NCPL=24.
Furthermore, you would then not need to do any kind of time interpolation on the forcing data, e.g., coszen for solar. So you would set "tintalgo" in your streams file to "nearest" for all of the streams. It's not necessary but you could also put all of the forcing data into one file since all of the streams can now be treated similarly.

If you are still having trouble with your run, you could try compiling and running in DEBUG mode which might provide more useful error output. DEBUG=TRUE in env_build.xml

Hi Olsen,

Thank you so much for your help!
I've done what you said and I believe my meteorological dataset has the same time format as GSWP3, but CLM still gives me the same error. I found the possible error from the atm.log, and it says the dimensions 1 and 2 of the variable FSDS are 1, but I'm pretty sure this variable should have the dimension of (time, lat, lon).. I use Python to create the nc files, I'm not sure if the data input into the nc files should be in the dimension of (lon, lat, time)?


atm.log
-------------------------

(shr_strdata_set_stream_iodesc) setting iodesc for : FSDS with dimlens(1), dimlens(2) = 1 1 variable as time dimension time



ncdump -h /glade/scratch/jinmuluo/MESOCOM_fluxtower_BOOI4/datmdata/clmforc.BOOI4.Solr.isu_point.2014-01.nc
-------------------------

dimensions:


time = 744 ;


lat = 1 ;


lon = 1 ;


variables:


float time(time) ;


time:_FillValue = NaNf ;


time:long_name = "observation time" ;


time:units = "days since 2014-01-01" ;


float lat(lat) ;


lat:_FillValue = NaNf ;


float lon(lon) ;


lon:_FillValue = NaNf ;


float FSDS(time, lat, lon) ;


FSDS:_FillValue = NaNf ;


FSDS:units = "W/m**2" ;


FSDS:long_name = "surface_incoming_shortwave_flux" ;

 

[oleson]

(time,lat,lon) is the correct order for FSDS in the file.
I looked at atm log of a GSWP3 run I did and it reported:

(shr_strdata_set_stream_iodesc) setting iodesc for : FSDS with dimlens(1), dimlens(2) = 720 360 variable as time dimension time.

The dimensions of lon/lat in the file are 720/360, so that is correct.

 

[oleson]

I don't know if this is related to your error, but you have set <entry id="CALENDAR" value="GREGORIAN"> in your env_build.xml and you don't currently have a calendar attribute for time in this file:
/glade/scratch/jinmuluo/MESOCOM_fluxtower_BOOI4/datmdata/clmforc.BOOI4.Solr.isu_point.2014-01.nc

You'd need to set that in the file, i.e., time:calendar = "gregorian"

 

[jinmuluo]

I don't know if this is related to your error, but you have set <entry id="CALENDAR" value="GREGORIAN"> in your env_build.xml and you don't currently have a calendar attribute for time in this file:
/glade/scratch/jinmuluo/MESOCOM_fluxtower_BOOI4/datmdata/clmforc.BOOI4.Solr.isu_point.2014-01.nc

You'd need to set that in the file, i.e., time:calendar = "gregorian"

If the calendar of my meteorological data is in "no leap" as GSWP3, can I set <entry id="CALENDAR" value="NOLEAP"> in y env_build.xml?

 

[oleson]

Yes. <entry id="CALENDAR" value="NO_LEAP">. You need the underscore.

 

[oleson]

I also see this error at the very beginning of the atm log file:
Error in `/glade/scratch/jinmuluo/MESOCOM/bld/cesm.exe': free(): corrupted unsorted chunks:

Searching for this error indicates possibly due to "This error occurs because you have corrupted a pointer the system uses to keep track of space that has been malloc()ed. The most common causes for this type of corruption are (1) using an uninitialized pointer and (2) writing more data into memory than was allocated for the buffer into which the data is being written."

Have you run this code for some other case successfully or is this the first time you've used this version?

 

[jinmuluo]

I also see this error at the very beginning of the atm log file:
Error in `/glade/scratch/jinmuluo/MESOCOM/bld/cesm.exe': free(): corrupted unsorted chunks:

Searching for this error indicates possibly due to "This error occurs because you have corrupted a pointer the system uses to keep track of space that has been malloc()ed. The most common causes for this type of corruption are (1) using an uninitialized pointer and (2) writing more data into memory than was allocated for the buffer into which the data is being written."

Have you run this code for some other case successfully or is this the first time you've used this version?

Hi Olsen,

I've been running this code many times for global simulation but this is the first time I have used this code to run a single site simulation with my own forcing dataset. I did run a successful sp simulation with GSWP3 forcing, so I guess the code wouldn't have any problems.

 

[jinmuluo]

I also see this error at the very beginning of the atm log file:
Error in `/glade/scratch/jinmuluo/MESOCOM/bld/cesm.exe': free(): corrupted unsorted chunks:

Searching for this error indicates possibly due to "This error occurs because you have corrupted a pointer the system uses to keep track of space that has been malloc()ed. The most common causes for this type of corruption are (1) using an uninitialized pointer and (2) writing more data into memory than was allocated for the buffer into which the data is being written."

Have you run this code for some other case successfully or is this the first time you've used this version?

I went back to check the different cases using this code, global simulation is still working. But sp simulation with GSWP3 has been running into some issues.... I have no idea about the error, can you help to check this? I've attached the scripts used to create the case for your convenience.

 

[jinmuluo]

I went back to check the different cases using this code, global simulation is still working. But sp simulation with GSWP3 has been running into some issues.... I have no idea about the error, can you help to check this? I've attached the scripts used to create the case for your convenience.

Hi,

Please ignore this, I've solved this question by adding "DATM_YR_ALIGN", "ATM_YR_START" and "DATM_YR_END" into the "env_run.xml".

Best,

 

[jinmuluo]

I also see this error at the very beginning of the atm log file:
Error in `/glade/scratch/jinmuluo/MESOCOM/bld/cesm.exe': free(): corrupted unsorted chunks:

Searching for this error indicates possibly due to "This error occurs because you have corrupted a pointer the system uses to keep track of space that has been malloc()ed. The most common causes for this type of corruption are (1) using an uninitialized pointer and (2) writing more data into memory than was allocated for the buffer into which the data is being written."

Have you run this code for some other case successfully or is this the first time you've used this version?

Hi Oleson,

Since I can use this code to conduct a global simulation and sp simulation with GSWP3 forcing, so, the errors were definitely caused by the NetCDF files. I'm wondering do you know is there any way to compare my files with GSWP3 thoroughly? Or if you could provide a simple script of how to use Python to produce a netcdf, that would be very helpful.

Best,