Guidance on Running with LME datasets with CESM2.1.3 (CLM5)

[wvsi3w]

I doubt there is a problem with the finidat file since that file would be interpolated to your new grid/domain at the beginning of the simulation and your simulation runs fine for a number of years.
One question I have is why do you have the anomaly forcing repeated in the file above, i.e.,

"datm.streams.txt.Anomaly.Forcing.Precip 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Temperature 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Pressure 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Humidity 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Uwind 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Vwind 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Shortwave 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Longwave 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Precip 2006 2006 2300",
"datm.streams.txt.Anomaly.Forcing.Temperature 2006 2006 2300",
"datm.streams.txt.Anomaly.Forcing.Pressure 2006 2006 2300",
"datm.streams.txt.Anomaly.Forcing.Humidity 2006 2006 2300",
"datm.streams.txt.Anomaly.Forcing.Uwind 2006 2006 2300",
"datm.streams.txt.Anomaly.Forcing.Vwind 2006 2006 2300",
"datm.streams.txt.Anomaly.Forcing.Shortwave 2006 2006 2300",
"datm.streams.txt.Anomaly.Forcing.Longwave 2006 2006 2300"

Also, I'd check the precipitation anomaly forcing file to make sure it has valid data throughout time, could it be corrupted from a bad download?

Yes that was odd for me. At first, I thought this could be normal.

Also, this is my user_nl_datm file:

anomaly_forcing = "Anomaly.Forcing.Precip","Anomaly.Forcing.Temperature","Anomaly.Forcing.Pressure","Anomaly.Forcing.Humidity","Anomaly.Forcing.Uwind","Anomal
y.Forcing.Vwind","Anomaly.Forcing.Shortwave","Anomaly.Forcing.Longwave"

fillalgo = "nn", "nn", "nn", "nn", "nn", "nn", "nn", "nn", "nn", "nn", "nn", "nn", "nn"

streams = "datm.streams.txt.CLMGSWP3v1.Solar 2000 2000 2014",
"datm.streams.txt.CLMGSWP3v1.Precip 2000 2000 2014",
"datm.streams.txt.CLMGSWP3v1.TPQW 2000 2000 2014",
"datm.streams.txt.presaero.SSP2-4.5 2015 2015 2101",
"datm.streams.txt.topo.observed 1 1 1",
"datm.streams.txt.co2tseries.SSP2-4.5 2015 2015 2500",
"datm.streams.txt.Anomaly.Forcing.Precip 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Temperature 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Pressure 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Humidity 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Uwind 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Vwind 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Shortwave 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Longwave 2015 2015 2100"

As you see, the anomaly list is not repeated in my namelist. I don't know why datm_in is showing a repeated list, also the repeated part is showing a different range (2006-2300) while my namelist was from 2015-2100!!!

Do you suggest removing these repeated lines manually and starting again from the beginning?

About the "precipitation anomaly forcing file" I checked the "pr" variable in both files (af.allvars.CESM.SSP1-2.6.2015-2100_c20220628.nc & af.allvars.CESM.SSP2-4.5.2015-2100_c20220628.nc), and they are valid.

 

[oleson]

I don't know why the repeated lines are being generated in your resolved datm_in. You are still usihg release-cesm2.1.3?
What domain file are you using for the anomaly forcing, e.g., in your datm.streams.txt.Anomaly.Forcing.Precip? It should be domain.b.e21.BHIST.f09_g17.CMIP6-historical.010.c200203.nc

 

[wvsi3w]

I don't know why the repeated lines are being generated in your resolved datm_in. You are still usihg release-cesm2.1.3?
What domain file are you using for the anomaly forcing, e.g., in your datm.streams.txt.Anomaly.Forcing.Precip? It should be domain.b.e21.BHIST.f09_g17.CMIP6-historical.010.c200203.nc

Yes I am still using 2.1.3 because my second objective hasn't yet finished.

That is odd because the anomaly forcing files initially had this domain (domain.permafrostRCN_P2.c2013.nc) written in them so I thought I shouldnt change it:

e.g. datm.streams.txt.Anomaly.Forcing.Precip :

<?xml version="1.0"?>
<file id="stream" version="1.0">
<dataSource>
   GENERIC
</dataSource>
<domainInfo>
  <variableNames>
     time    time
        xc      lon
        yc      lat
        area    area
        mask    mask
  </variableNames>
  <filePath>
 /scratch/USER/cesm/inputdata/atm/datm7/anomaly
  </filePath>
  <fileNames>
     domain.permafrostRCN_P2.c2013.nc
  </fileNames>
</domainInfo>
<fieldInfo>
   <variableNames>
     pr  prec_af
   </variableNames>
   <filePath>
 /scratch/USER/cesm/inputdata/atm/datm7/anomaly
   </filePath>
   <fileNames>
    af.allvars.CESM.SSP1-2.6.2015-2100_c20220628.nc
   </fileNames>
   <offset>
      0
   </offset>
</fieldInfo>
</file>

I will change it to what you mentioned and let you know how it goes.

 

[wvsi3w]

Dear Keith,
I have tried downloading the domain (domain.b.e21.BHIST.f09_g17.CMIP6-historical.010.c200203.nc), and it fails to find it:

svn export failed with output:  and errput svn: E170000: URL 'https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/atm/datm7/anomaly_forcing/domain.b.e21.BHIST.f09_g17.CMIP6-historical.010.c200203.nc' doesn't exist

Checking server None with protocol None
Setting resource.RLIMIT_STACK to 300000000 from (300000000, -1)
Client protocol None not enabled
ERROR: Could not find all inputdata on any server

To get it manually with wget, I also couldn't find the domain file online (is not in this link either: - Revision 70792: /trunk/inputdata/share/domains )

 

[oleson]

Well, the domain file you are using should work as well, since it has a mask of 1 everywhere on the grid and there is precip anomaly forcing everywhere.
I guess I would check your atm log file and see what atmospheric (not anomaly) forcing is being read in and then check the precip field on that forcing file. Since the model is failing at the same point no matter what the SSP is, it seems like that might point to a problem with the forcing.

 

[wvsi3w]

Well, the domain file you are using should work as well, since it has a mask of 1 everywhere on the grid and there is precip anomaly forcing everywhere.
I guess I would check your atm log file and see what atmospheric (not anomaly) forcing is being read in and then check the precip field on that forcing file. Since the model is failing at the same point no matter what the SSP is, it seems like that might point to a problem with the forcing.

Thank you
It was indeed one of the files from Precip. I checked the atm log file, and it ended in these:

/scratch/USER/cesm/inputdata/atm/datm7/atm_forcing.datm7.GSWP3.0.5d.v1.c170516/Precip/clmforc.GSWP3.c2011.0.5x0.5.Prec.2003-09.nc     169
(shr_dmodel_readstrm) file ub: /scratch/USER/cesm/inputdata/atm/datm7/atm_forcing.datm7.GSWP3.0.5d.v1.c170516/TPHWL/clmforc.GSWP3.c2011.0.5x0.5.TPQWL.2003-09.nc     169
(datm_comp_run) atm: model date 20330921   82800s
(datm_comp_run) atm: model date 20330921   84600s
(datm_comp_run) atm: model date 20330922       0s
(shr_dmodel_readstrm) file ub: /scratch/USER/cesm/inputdata/atm/datm7/atm_forcing.datm7.GSWP3.0.5d.v1.c170516/Solar/clmforc.GSWP3.c2011.0.5x0.5.Solr.2003-09.nc     170
(datm_comp_run) atm: model date 20330922    1800s

So I checked the 2003-09 Precip file with ncview, and it was indeed damaged in the middle. removed it and downloaded it again and continued the run, but again another file was broken (2004-01), so I did the same thing, and it is running now.

Do you know what we should do if this happens a lot? I mean, is there a way to check the forcing files when we download them with some command or some script to see if they are all ok and not damaged?

 

[oleson]

I'm not sure, maybe something like: ./check_input_data --download --debug, or ./check_input_data --download --chksum?

 

[wvsi3w]

I'm not sure, maybe something like: ./check_input_data --download --debug, or ./check_input_data --download --chksum?

Thanks for your help.
I have found another issue which is weird:

 (shr_strdata_advance) ERROR: for stream           15
 (shr_strdata_advance) ERROR: dt limit1    76316.0000000000
   28.0000000000000        1.50000000000000
 (shr_strdata_advance) ERROR: dt limit2     20150116           0    20150213
           0
 ERROR: (shr_strdata_advance) ERROR dt limit for stream

If you remember last time, I had an error regarding the dtlimit (Guidance on Running with LME datasets with CESM2.1.3 (CLM5)), and you suggested using 1.e36 to get around it.

This time, instead of starting from 2020, I was trying to start my SSP run from 2015, and I faced the same error! This didn't happen when I started with 2020 restart file.

I am using the restart file from that simulation where dtlimit in its user_nl_datm was set to 1.e36 (from 2000-2015).

Prior to this, I was starting from 2020-01-01, and the restart file from 2020-01-01 is from a simulation in which the dtlimit was also set to 1.e36 in the datm user namelist file, but my SSP run (from 2020 onwards) didn't have any problem and is running now.

I only see this error when I am starting from earlier (like 2015).

Do you think this occurs because the 2015-2020 simulation was forced with a SSP and the 2000-2015 simulation was not? (In both, I did set dtlimit to 1.e36, I don't know what is wrong here)

I think I should set dtlimit to 1.e36 if I want to run from 2015-2100 using the restart file from 2015 (?)

 

[wvsi3w]

Thanks for your help.
I have found another issue which is weird:

 (shr_strdata_advance) ERROR: for stream           15
 (shr_strdata_advance) ERROR: dt limit1    76316.0000000000
   28.0000000000000        1.50000000000000
 (shr_strdata_advance) ERROR: dt limit2     20150116           0    20150213
           0
 ERROR: (shr_strdata_advance) ERROR dt limit for stream

If you remember last time, I had an error regarding the dtlimit (Guidance on Running with LME datasets with CESM2.1.3 (CLM5)), and you suggested using 1.e36 to get around it.

This time, instead of starting from 2020, I was trying to start my SSP run from 2015, and I faced the same error! This didn't happen when I started with 2020 restart file.

I am using the restart file from that simulation where dtlimit in its user_nl_datm was set to 1.e36 (from 2000-2015).

Prior to this, I was starting from 2020-01-01, and the restart file from 2020-01-01 is from a simulation in which the dtlimit was also set to 1.e36 in the datm user namelist file, but my SSP run (from 2020 onwards) didn't have any problem and is running now.

I only see this error when I am starting from earlier (like 2015).

Do you think this occurs because the 2015-2020 simulation was forced with a SSP and the 2000-2015 simulation was not? (In both, I did set dtlimit to 1.e36, I don't know what is wrong here)

I think I should set dtlimit to 1.e36 if I want to run from 2015-2100 using the restart file from 2015 (?)

I did set the dtlimit in user_nl_datm to 1.e36 for a case that starts from 2015 (for all of the 13 streams), and it failed with the same error!

I guess I have to start from 2020 :(

 

[wvsi3w]

Hello all,
I have done some simulations with SSP scenarios in the past few months, and all of them ran successfully; however, there is a problem:
When you start the SSPs (2015-2100), between 2015 to 2020, a sudden increase in temperature appears, which makes the time-series jump a lot in that initial five years of future simulations. This appears to be related to several issues, one of which is that SSPs read DATM files (in my case 1995-2014 from GSWP3 files) while my simulations from 500 CE to 2015 was using different boundary conditions (as mentioned in this thread these DATM files were monthly resolution 1deg, which required me to do some modifications to the domain and other files in order to run the model with them successfully). So there is a big difference between the default DATM that model reads (GSWP3 in here) and the DATM that I had up until 2015, and this could explain the big jump in temperature time-series.

To resolve this issue, I started the following case, in which I tried to change the DATM so that the anomaly forcing would be applied to my boundary conditions (and not the GSWP3 files):

cime/scripts/create_newcase --case /MYCASES/FUTURE/DATMchanged_SSP245_Mem1_Deep --compset ISSP245Clm50BgcCrop --res f09_g17 --run-unsupported

cd /MYCASES/FUTURE/DATMchanged_SSP245_Mem1_Deep
./case.setup

./xmlchange STOP_OPTION=nyears,STOP_N=5,JOB_WALLCLOCK_TIME=03:59:00,RUN_STARTDATE=2015-01-01,STOP_DATE=21000101,RESUBMIT=16,DATM_CLMNCEP_YR_END=2014,DATM_CLMNCEP_YR_START=1995,DATM_CLMNCEP_YR_ALIGN=1995

# cp clm2.r file and rpointer to case directory

emacs user_nl_clm
use_init_interp = .true.
soil_layerstruct= '23SL_3.5m_D500'
finidat ='/MYCASES/FUTURE/DATAMchanged_SSP245_Mem1_Deep/DeepMem1_2000to2015_IHist.clm2.r.2015-01-01-00000.nc'

emacs user_nl_datm

anomaly_forcing = "Anomaly.Forcing.Precip","Anomaly.Forcing.Temperature","Anomaly.Forcing.Pressure","Anomaly.Forcing.Humidity","Anomaly.Forcing.Uwind","Anomaly.Forcing.Vwind","Anomaly.Forcing.Shortwave","Anomaly.Forcing.Longwave"

&datm_nml
  decomp = "1d"
  factorfn = "null"
  force_prognostic_true = .false.
  iradsw = 1
  presaero = .true.
  restfilm = "undefined"
  restfils = "undefined"
  wiso_datm = .false.
/
&shr_strdata_nml
  datamode = "CLMNCEP"
  domainfile = "/share/domains/domain.lnd.fv0.9x1.25_gx1v7.151020.nc"
  dtlimit = 1.e36, 1.e36, 1.e36, 1.e36, 1.e36, 1.e36, 1.e36, 1.e36, 1.e36, 1.e36, 1.e36, 1.e36, 1.e36, 1.e36
  fillalgo = "nn", "nn", "nn", "nn", "nn", "nn", "nn", "nn", "nn", "nn", "nn", "nn", "nn", "nn"
  fillmask = "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask"
  fillread = "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET"
  fillwrite = "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET"
  mapalgo = "bilinear", "bilinear", "bilinear", "bilinear", "bilinear", "nn", "bilinear", "bilinear", "bilinear", "bilinear", "bilinear", "bilinear", "bilinear", "bilinear"
  mapmask = "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask", "nomask"
  mapread = "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET"
  mapwrite = "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET", "NOT_SET"
  readmode = "single", "single", "single", "single", "single", "single", "single", "single", "single", "single", "single", "single", "single", "single"
streams = "datm.streams.txt.CLMGSWP3v1.Solar 1995 1995 2014",
"datm.streams.txt.CLMGSWP3v1.Precip 1995 1995 2014",
"datm.streams.txt.CLMGSWP3v1.TPQW 1995 1995 2014",
"datm.streams.txt.presaero.SSP2-4.5 2015 2015 2101",
"datm.streams.txt.topo.observed 1 1 1",
"datm.streams.txt.co2tseries.SSP2-4.5 2015 2015 2500",
"datm.streams.txt.Anomaly.Forcing.Precip 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Temperature 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Pressure 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Humidity 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Uwind 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Vwind 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Shortwave 2015 2015 2100",
"datm.streams.txt.Anomaly.Forcing.Longwave 2015 2015 2100"
  taxmode = "cycle", "cycle", "cycle", "cycle", "cycle", "extend",  "cycle", "cycle", "cycle", "cycle", "cycle", "cycle", "cycle", "cycle"
  tintalgo = "coszen", "nearest", "linear", "linear", "lower", "linear", "nearest",  "linear", "linear",  "linear", "linear",  "linear", "coszen", "coszen"
  vectors = "null"
/

# COPY THE "USER_DATM.STREAMS.TXT.ANOMALY.FORCING" FILES FROM PREVIOUS CASE DIRECTORY AND PASTE IT INTO YOUR NEW CASE DIRECTORY

# emacs existing user_datm.stream*
# change the domain path and name to the domain file that is adjusted to my DATAM files:

  <filePath>
     /SHARED_FOLDER/mem1/1deg
  </filePath>
  <fileNames>
     domain.lnd.fv0.9x1.25_gx1v7.151020.nc
  </fileNames>

# cd my DATM path for Precip, Solar, TPWHL and rename the years 1995-2014 to "clmforc.GSWP3.c2011.0.5x0.5." format. and copy them into the GSWP3 directory

./case.build
./case.submit

But this job failed with the following error:

(shr_dmodel_readstrm) ERROR in data sizes      288     192       1  259200
 ERROR: (shr_dmodel_readstrm) ERROR in data sizes
.
.
Stack trace terminated abnormally.
[rc13020:75919:0:75919] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil))
Image              PC                Routine            Line        Source
.
.
Stack trace terminated abnormally.
[rc13020:75921:0:77991] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil))
Image              PC                Routine            Line        Source
.
.
Stack trace terminated abnormally.
[rc13020:75920:0:75920] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace (tid:  75920) ====
 0 0x0000000000038790 __sigaction()  ???:0
.
.
[rc13020:76028:0:76028] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x1)
[rc13020:76028:1:78135] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x10b)

I assume this is because the SSP anomaly files are not compatible with my datm files. Do you know how to resolve this? I genuinely need to use my own datm files from 1995-2014 for the SSP anomaly to be applied on them. How do I make the anomaly file (or maybe even its domain file) compatible with my datm files?

Thank you for your help and support in advance.

 

[wvsi3w]

the key part of the error is "288 192 1 259200" which is saying my 1 deg datm files (192 x 288 = 55296) is not equal to the 0.5 deg resolution GSWP3 files (259200 = 720 × 360). This created that incompatibility I guess.

here I don't know if I need to modify my datm files to become 0.5 deg (also my domain needs to be modified too), or I need to modify the SSP anomaly and its domain to be 1 deg.

or there is another option you know?

 

[wvsi3w]

An update on this issue:
I re-grid the datm files I have to be 0.5 deg resolution and it is running now (not sure if it finishes without issue or not).

Also in the user_nl_datm file I made a mistake that tintalgo for Longwave should be "linear" and not "coszen".

lets see if it works.

 

[oleson]

The anomaly forcing is at 1deg and typically applied to 0.5 deg GSWP3 data. So they don't have to be at the same resolution. As long as you use domain files appropriate for each set of forcing. E.g., the typical domain file setting for the GSWP3 data is (in, for example, datm.streams.txt.CLMGSWP3v1.Precip):

<domainInfo>
<variableNames>
time time
xc lon
yc lat
area area
mask mask
</variableNames>
<filePath>
/glade/p/cgd/tss/CTSM_datm_forcing_data/atm_forcing.datm7.GSWP3.0.5d.v1.c170516
</filePath>
<fileNames>
domain.lnd.360x720_gswp3.0v1.c170606.nc
</fileNames>
</domainInfo>

and the typical (depends on your model version) domain file setting for anomaly forcing is (in, for example, datm.streams.txt.Anomaly.Forcing.Humidity):

<domainInfo>
<variableNames>
time time
xc lon
yc lat
area area
mask mask
</variableNames>
<filePath>
/glade/p/cgd/tss/people/oleson/anomaly_forcing
</filePath>
<fileNames>
domain.b.e21.BHIST.f09_g17.CMIP6-historical.010.c200203.nc
</fileNames>
</domainInfo>

One other thing is that all of anomaly forcing should have tintalgo = nearest. Maybe it doesn't matter, but I'm not sure what the datm is doing with your setting of coszen for the solar anomaly. coszen should only be used for your solar forcing data.

 

[wvsi3w]

The anomaly forcing is at 1deg and typically applied to 0.5 deg GSWP3 data. So they don't have to be at the same resolution. As long as you use domain files appropriate for each set of forcing. E.g., the typical domain file setting for the GSWP3 data is (in, for example, datm.streams.txt.CLMGSWP3v1.Precip):

<domainInfo>
<variableNames>
time time
xc lon
yc lat
area area
mask mask
</variableNames>
<filePath>
/glade/p/cgd/tss/CTSM_datm_forcing_data/atm_forcing.datm7.GSWP3.0.5d.v1.c170516
</filePath>
<fileNames>
domain.lnd.360x720_gswp3.0v1.c170606.nc
</fileNames>
</domainInfo>

and the typical (depends on your model version) domain file setting for anomaly forcing is (in, for example, datm.streams.txt.Anomaly.Forcing.Humidity):

<domainInfo>
<variableNames>
time time
xc lon
yc lat
area area
mask mask
</variableNames>
<filePath>
/glade/p/cgd/tss/people/oleson/anomaly_forcing
</filePath>
<fileNames>
domain.b.e21.BHIST.f09_g17.CMIP6-historical.010.c200203.nc
</fileNames>
</domainInfo>

One other thing is that all of anomaly forcing should have tintalgo = nearest. Maybe it doesn't matter, but I'm not sure what the datm is doing with your setting of coszen for the solar anomaly. coszen should only be used for your solar forcing data.

Dear Keith,
Thank you so much for the response.

About the "tintalgo = nearest", do you think that would significantly impact my results if I use "linear" instead?
I couldn't find relevant explanation in the user guide link for this question.

 

[oleson]

The anomaly forcing is a monthly value and I believe you would want to apply the same monthly value to all of the forcing in that month, which is what nearest would do. If you used linear then you would be creating anomalies whose monthly mean would not be the same as the provided monthly value. At least that's how I interpret the choice of nearest.