CTSM CLM standalone runs error

[dquaker]

git describe
ctsm5.1.dev105

./manage_externals/checkout_externals --status --verbose
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_CISM.cfg
Processing externals description file : .gitmodules
Processing submodules description file : .gitmodules
Processing externals description file : Externals_CDEPS.cfg
Checking status of externals: clm, fates, cism, source_cism, rtm, mosart, mizuroute, ccs_config, cime, cmeps, cdeps, fox, genf90, cpl7, share, mct, parallelio, doc-builder,
./ccs_config
clean sandbox, on ccs_config_cesm0.0.38
M ./cime
modified sandbox, on cime6.0.40
./components/cdeps
clean sandbox, on cdeps0.12.41
./components/cdeps/fox
clean sandbox, on 4.1.2.1
./components/cdeps/share/genf90
clean sandbox, on genf90_200608
./components/cism
clean sandbox, on cismwrap_2_1_95
./components/cism/source_cism
clean sandbox, on cism_main_2.01.011
./components/cmeps
clean sandbox, on cmeps0.13.68
./components/cpl7
clean sandbox, on cpl7.0.13
./components/mizuRoute
clean sandbox, on 34723c2e4df7caa16812770f8d53ebc83fa22360
./components/mosart
clean sandbox, on mosart1_0_45
./components/rtm
clean sandbox, on rtm1_0_78
e-o ./doc/doc-builder
-, not checked out --> v1.0.8
./libraries/mct
clean sandbox, on MCT_2.11.0
./libraries/parallelio
clean sandbox, on pio2_5_7
./share
clean sandbox, on share1.0.12
./src/fates
clean sandbox, on sci.1.58.1_api.24.1.0



My attempt to make a regional simulations are as follows:

[A] cd ~/CTSM/tools/site_and_regional

./subset_data region --reg tes5e --lat1 0.0 --lat2 24.0 --lon1 0 --lon2 40 --create-mesh --create-domain --create-surface --create-landuse --create-datm --datm-syr 1980 --datm-eyr 2014 --create-user-mods --user-mods-dir /gpfs/scratch/pn25ki/di75jeq/inputdata/tes5e --outdir /gpfs/scratch/pn25ki/di75jeq/inputdata --overwrite

{C] ./create_newcase --case ~/ctsm/tes5e --compset IHistClm50Sp --res f09_g17 --machine cm2 --run-unsupported --user-mods-dirs /gpfs/scratch/pn25ki/di75jeq/inputdata/tes5e --input-dir /gpfs/scratch/pn25ki/di75jeq/inputdata --output-root /gpfs/scratch/pn25ki/di75jeq

[D] cd ~/ctsm/tes5e
./xmlchange RUN_STARTDATE=2001-01-01
./xmlchange DATM_YR_START="2001"
./xmlchange DATM_YR_END="2001"
./xmlchange STOP_OPTION="nyears"
./xmlchange JOB_WALLCLOCK_TIME=02:20:00
./xmlchange RESUBMIT=5
./xmlchange STOP_N=5
./xmlchange MOSART_MODE=NULL
./xmlchange CLM_USRDAT_DIR=/gpfs/scratch/pn25ki/di75jeq/inputdata
./xmlchange CLM_USRDAT_NAME=tes5e

./case.setup
./case.build
./case.submit

Kindly help me with the error encountered as attached in my log files. I

 

[oleson]

It looks like your message was cut off, I don't see any log files.

 

[dquaker]

Thank you for your attention.

I hereby includ the attachment in the acceptable format. I tried to do a similar case as described earlier using a different case name. It is the same error as I am not sure if it could be an ESMF installation problem. Kindle note that the global and point simulation was successful using the same compset and resolution but the regional simulation has never been successfully done.

 

[oleson]

I was able to replicate this ESMF error here. I'm not familiar with the subset_data approach used here. May need some help from @erik @negins @wwieder @afoster

 

[afoster]

Hi, I am fairly certain we have not finished updating the subset mesh capabilities of subset_data (is this correct @negins?). Because of this, you will need to use the old mct driver in the create_newcase call:

./create_newcase --case ~/ctsm/tes5e --compset IHistClm50Sp --res f09_g17 --machine cm2 --driver mct --run-unsupported --user-mods-dirs /gpfs/scratch/pn25ki/di75jeq/inputdata/tes5e --input-dir /gpfs/scratch/pn25ki/di75jeq/inputdata --output-root /gpfs/scratch/pn25ki/di75jeq

The files should be set up correctly in your user mods but let us know if this does not work

 

[oleson]

Thanks @afoster
I tried this and got a different error. It seems that the global domain file is being used instead of the regional one that was generated from subset_data.
Is the user mods supposed to set this up?
I still have this in my env_run.xml:

<entry id="LND_DOMAIN_FILE" value="domain.lnd.fv0.9x1.25_gx1v7.151020.nc">
<entry id="LND_DOMAIN_PATH" value="$DIN_LOC_ROOT/share/domains">

 

[afoster]

Aha okay, so it is not currently set up by subset data, we should fix that.

That LND_DOMAIN_FILE and LND_DOMAIN_PATH should be set to the subset domain set up with subset data I think you can check the user_nl_clm to check where it was placed.

 

[afoster]

Sorry - the user_nl_clm should indicate where the surface data was placed, which hopefully is also where the subset domain files were placed.

 

[oleson]

This gets me past that error:

./xmlchange LND_DOMAIN_FILE=domain.lnd.fv0.9x1.25_gx1v7_tes5e_c220825.nc
./xmlchange LND_DOMAIN_PATH=/glade/scratch/oleson/inputdata

Next error is that the surface dataset created is for simyr2000 and the landuse time series file created is for simyr1850-2015.
The surface dataset should be for simyr1850.

 

[afoster]

Aha okay, for that you will need to update the default_data.cfg file that tells the subset_data script which files to subset (https://github.com/ESCOMP/CTSM/blob/master/tools/site_and_regional/default_data.cfg). You'll need to make sure the directory(s) and filename are set.

[main]
clmforcingindir = /glade/p/cesmdata/inputdata - I think this should be the same?

[landuse]
	dir = lnd/clm2/surfdata_map/release-clm5.0.18
	landuse_16pft = landuse.timeseries_0.9x1.25_hist_16pfts_Irrig_CMIP6_simyr1850-2015_c190214.nc
	landuse_78pft = landuse.timeseries_0.9x1.25_hist_78pfts_CMIP6_simyr1850-2015_c190214.nc

I'm guessing those landuse are the ones you would want to change

 

[oleson]

There was an amended post by @afoster that did not seem to show up here.
You would want to set the following in default_data.cfg instead of changing the landuse files:

[surfdat]
dir = lnd/clm2/surfdata_map/release-clm5.0.18
surfdat_16pft = surfdata_0.9x1.25_hist_16pfts_Irrig_CMIP6_simyr1850_c190214.nc
surfdat_78pft = surfdata_0.9x1.25_hist_78pfts_CMIP6_simyr1850_c190214.nc

Another change I had to make to the case:

./xmlchange ATM_DOMAIN_FILE=domain.lnd.fv0.9x1.25_gx1v7_tes5e_c220825.nc
./xmlchange ATM_DOMAIN_PATH=/glade/scratch/oleson/inputdata

With all of that, my case is now running.

 

[afoster]

whoops sorry! I got mixed up! Yes it should be the surface data that gets updated!

 

[dquaker]

There was an amended post by @afoster that did not seem to show up here.
You would want to set the following in default_data.cfg instead of changing the landuse files:

[surfdat]
dir = lnd/clm2/surfdata_map/release-clm5.0.18
surfdat_16pft = surfdata_0.9x1.25_hist_16pfts_Irrig_CMIP6_simyr1850_c190214.nc
surfdat_78pft = surfdata_0.9x1.25_hist_78pfts_CMIP6_simyr1850_c190214.nc

Another change I had to make to the case:

./xmlchange ATM_DOMAIN_FILE=domain.lnd.fv0.9x1.25_gx1v7_tes5e_c220825.nc
./xmlchange ATM_DOMAIN_PATH=/glade/scratch/oleson/inputdata

With all of that, my case is now running.

Thank you very much for the guided instructions toward resolving the errors.

My case was on the queue earlier which prevented me from giving feedback.

I followed the specific instructions with respect to creating a new case. However, it stopped with errors related to "MCT::m_SparseMatrixPlus:: FATAL--length of vector y different from row count of sMat.Length of y = 825 Number of rows in sMat = 55296".

Kindly find attached log files and other relevant files that could point to the source of the error. I look forward to your assistant in resolving this.

 

[dquaker]

@oleson ,@afoster
Kindly find attached additional log files that migbt be useful for identification of the error source.

 

[oleson]

The error indicates that there is a likely still a mismatch between the domain file you are using and the grid you are running on.
I can't tell from the log files what the specific problem is.
I would check your domain file, surface dataset, and landuse timeseries to make sure the latxlon is 25x33=825 in all cases.
Also, can you attach your lnd_in, datm_in, and streams files (e.g., datm.streams.txt.CLMGSWP3v1.Precip)?

A couple of minor things:
1. For mct, these variables don't exist and so you should be getting an error:
./xmlchange DATM_YR_START=2001
./xmlchange DATM_YR_END=2001

2. At least in my case, the pe-layout was setup by default with too many processors for the grid, so I had to do this:

./xmlchange NTASKS_LND=-10
./xmlchange NTASKS_ROF=-10
./xmlchange NTASKS_ICE=-10
./xmlchange NTASKS_OCN=-10
./xmlchange NTASKS_CPL=-10
./xmlchange NTASKS_GLC=-10
./xmlchange NTASKS_WAV=-10

 

[oleson]

It seems that MOSART is not actually off in your run (evident because you have rof log file). I thought MOSART_MODE=NULL would shut it off but you might need to specify the stub runoff model in your compset longname when using ./create_newcase, e.g.,

./create_newcase --case tes5e --compset HIST_DATM%GSWP3v1_CLM50%SP_SICE_SOCN_SROF_SGLC_SWAV_SIAC_SESP --res f09_g17 --driver mct --run-unsupported --user-mods-dirs /glade/scratch/oleson/inputdata/tes5e --input-dir /glade/scratch/oleson/inputdata/ --output-root /glade/scratch/oleson/

 

[dquaker]

The error indicates that there is a likely still a mismatch between the domain file you are using and the grid you are running on.
I can't tell from the log files what the specific problem is.
I would check your domain file, surface dataset, and landuse timeseries to make sure the latxlon is 25x33=825 in all cases.
Also, can you attach your lnd_in, datm_in, and streams files (e.g., datm.streams.txt.CLMGSWP3v1.Precip)?

A couple of minor things:
1. For mct, these variables don't exist and so you should be getting an error:
./xmlchange DATM_YR_START=2001
./xmlchange DATM_YR_END=2001

2. At least in my case, the pe-layout was setup by default with too many processors for the grid, so I had to do this:

./xmlchange NTASKS_LND=-10
./xmlchange NTASKS_ROF=-10
./xmlchange NTASKS_ICE=-10
./xmlchange NTASKS_OCN=-10
./xmlchange NTASKS_CPL=-10
./xmlchange NTASKS_GLC=-10
./xmlchange NTASKS_WAV=-10

Thank you for the recommendations. I will apply apply this suggestion to create a new case and see how it proceeds. Please find below my ressponses to the points or observations raised.

I can confirm that the surface dataset, and landuse timeseries and domain file are all having the same dimension of 25x33=825. Please find attached the requested files.

1. For mct case in this compset I used the below as it can be found in the replay.sh file
./xmlchange DATM_CLMNCEP_YR_ALIGN=2001

./xmlchange DATM_CLMNCEP_YR_START=2001

./xmlchange DATM_CLMNCEP_YR_END=2001

2. I will modify the pe-layout as recommended. Thank you

 

[dquaker]

It seems that MOSART is not actually off in your run (evident because you have rof log file). I thought MOSART_MODE=NULL would shut it off but you might need to specify the stub runoff model in your compset longname when using ./create_newcase, e.g.,

./create_newcase --case tes5e --compset HIST_DATM%GSWP3v1_CLM50%SP_SICE_SOCN_SROF_SGLC_SWAV_SIAC_SESP --res f09_g17 --driver mct --run-unsupported --user-mods-dirs /glade/scratch/oleson/inputdata/tes5e --input-dir /glade/scratch/oleson/inputdata/ --output-root /glade/scratch/oleson/

I created a new but similar case with this approach and the job is currently on a queue. I will revert back here on the status as soon as it gets going. Thank you for your advice, it is highly appreciated.

 

[dquaker]

Dear Oleson,
This is to inform you that the case ran sucessfully now. Thank you for your assistance.