Issues downloading input data in clm5 single point mode

[cdevaneprugh]

I'm having some issues downloading input data.

I am following this page to do a single point clm 5 run on cesm 2.1.5. I know support for specifying PTS_LAT/PTS_LON in create_newcase was dropped from cime at some point, but I can just change it after the case is created.

# create the test
./create_newcase --case testPTS --res f19_g17 --compset I1850Clm50SpCru
cd testPTS

# change variables to run on a single point
./xmlchange PTS_MODE=TRUE, PTS_LAT=29.7, PTS_LON=-82.0
./xmlchange CLM_FORCE_COLDSTART=on, RUN_TYPE=startup

# change variables to use a single core and adjust wall time
./xmlchange NTASKS=1

# setup, build, and submit the case as usual
./case.setup
./case.build
./case.submit

If I do all this, the case is created, setup, and built successfully. However, when submitting the case I get an error and the following output.

Loading input file list: 'Buildconf/clm.input_data_list'
Model clm no file specified for finidat
Checking server None with protocol None
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Client protocol None not enabled
ERROR: Could not find all inputdata on any server

If I go to the case directory and run

./check_input_data --download

I'll get errors like this

Model datm missing file domain1 = '/blue/gerber/earth_models/inputdata/clmforce/atm_forcing.datm7.cruncep_qianFill.0.5d.v7.c160715/domain.lnd.360x720_cruncep.130305.nc'
Trying to download file: 'clmforce/atm_forcing.datm7.cruncep_qianFill.0.5d.v7.c160715/domain.lnd.360x720_cruncep.130305.nc' to path '/blue/gerber/earth_models/inputdata/clmforce/atm_forcing.datm7.cruncep_qianFill.0.5d.v7.c160715/domain.lnd.360x720_cruncep.130305.nc' using WGET protocol.
wget failed with output: b'' and errput b'--2024-05-03 13:18:05--  ftp://ftp.cgd.ucar.edu/cesm/inputdata/clmforce/atm_forcing.datm7.cruncep_qianFill.0.5d.v7.c160715/domain.lnd.360x720_cruncep.130305.nc\n           => \xe2\x80\x98/blue/gerber/earth_models/inputdata/clmforce/atm_forcing.datm7.cruncep_qianFill.0.5d.v7.c160715/domain.lnd.360x720_cruncep.130305.nc\xe2\x80\x99\nResolving ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)... 128.117.13.41\nConnecting to ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)|128.117.13.41|:21... connected.\nLogging in as anonymous ... Logged in!\n==> SYST ... done.    ==> PWD ... done.\n==> TYPE I ... done.  ==> CWD (1) /cesm/inputdata/clmforce/atm_forcing.datm7.cruncep_qianFill.0.5d.v7.c160715 ... \nNo such directory \xe2\x80\x98cesm/inputdata/clmforce/atm_forcing.datm7.cruncep_qianFill.0.5d.v7.c160715\xe2\x80\x99.'

or this

Model datm missing file file231 = '/blue/gerber/earth_models/inputdata/clmforce/atm_forcing.datm7.cruncep_qianFill.0.5d.v7.c160715/Precip6Hrly/clmforc.cruncep.V7.c2016.0.5d.Prec.1920-03.nc'
Trying to download file: 'clmforce/atm_forcing.datm7.cruncep_qianFill.0.5d.v7.c160715/Precip6Hrly/clmforc.cruncep.V7.c2016.0.5d.Prec.1920-03.nc' to path '/blue/gerber/earth_models/inputdata/clmforce/atm_forcing.datm7.cruncep_qianFill.0.5d.v7.c160715/Precip6Hrly/clmforc.cruncep.V7.c2016.0.5d.Prec.1920-03.nc' using SVN protocol.
svn export failed with output:  and errput svn: E170000: URL 'https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/clmforce/atm_forcing.datm7.cruncep_qianFill.0.5d.v7.c160715/Precip6Hrly/clmforc.cruncep.V7.c2016.0.5d.Prec.1920-03.nc' doesn't exist

for all of the files its trying to download.

Any ideas? Am I just connecting to the wrong server?

 

[slevis]

In your errors I see
No such directory \xe2\x80\x98cesm/inputdata/clmforce/atm_forcing.datm7.cruncep_qianFill.0.5d.v7.c160715\xe2\x80\x99.'
and
https://svn-ccsm-inputdata.cgd.ucar...clmforc.cruncep.V7.c2016.0.5d.Prec.1920-03.nc' doesn't exist

The server may be right, but the paths seem wrong to me. On derecho the path to the data looks like this:
/glade/campaign/cesm/cesmdata/inputdata/atm/datm7/atm_forcing.datm7.cruncep_qianFill.0.5d.v7.c160715
which makes me suspicious of the "clmforce" and "trunk" that I see in your paths. I don't know off the top of my head how you can adjust the path, but this would be my first troubleshooting suggestion.

 

[slevis]

From a similar issue in a different thread, I now gather that "/trunk" is correct, but I still think that "/clmforce" is wrong.

 

[cdevaneprugh]

I had the path set wrong. I set the clm force directory as "clmforce" not realizing it had to mirror the path structure on the server. Changed it to "$DIN_LOC_ROOT/atm/datm7" and every thing looks like it's downloading. However, I am still having some issues trying to run a single point case.

I created two identical cases with the same compset and resolution. Both are using only one processor. The only difference is that one is set to run on a single point. I did this with the following.

./create_newcase --case osbsPTS --res f19_g17 --compset I1850Clm50SpCru
./create_newcase --case osbs --res f19_g17 --compset I1850Clm50SpCru

# change variables in single point case
cd osbsPTS
./xmlchange NTASKS=1,JOB_WALLCLOCK_TIME=1:00:00
./xmlchange PTS_MODE=TRUE,PTS_LAT=29.7,PTS_LON=-82.0
./xmlchange CLM_FORCE_COLDSTART=on,RUN_TYPE=startup

# change variables in standard case
cd osbs
./xmlchange NTASKS=1,JOB_WALLCLOCK_TIME=1:00:00

I then setup, built the cases, and downloaded all the input data with ./check_input_data --download. It appears everything gets downloaded for both cases. The only output I don't understand is coming from the single point case. When running ./check_input_data after downloading, one of the lines it outputs is:

Model clm no file specified for finidat

Not sure if this is a problem or can be ignored.

When I submit the cases, the normal case runs and finished successfully. The single point case gives an MPI error.

MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 2.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on exactly when Open MPI kills them.

Both cases are using openmpi to run. Should I switch to mpi-serial maybe? I've attached the CaseStatus and cesm log for the single point case (named
osbs_I1850Clm50SpCru).

Thanks!

 

[oleson]

When you set CLM_FORCE_COLDSTART=on, you are requesting a cold start and there won't be an finidat specified, so the message "Model clm no file specified for finidat" is normal.
The error message in the cesm log is:

MCT::m_SparseMatrixPlus:: FATAL--length of vector y different from row count of sMat.Length of y = 1 Number of rows in sMat = 13824
000.MCT(MPEU)::die.: from MCT::m_SparseMatrixPlus::initDistributed_()

It seems like there must be mismatch between two of the datasets you are using. One seems to be single point (y=1) and the other is 1.9x2.5 (144x96=13824). I think it might be because the compset you are using has CISM and MOSART in it. The long name for that compset is:

1850_DATM%CRUv7_CLM50%SP_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV

Try specifying stubs for those, e.g.,

1850_DATM%CRUv7_CLM50%SP_SICE_SOCN_SROF_SGLC_SWAV

 

[cdevaneprugh]

Thanks for the suggestion. The case ran successfully with your compset modification. I played around with some of the different default compsets and narrowed it down to just an issue with CISM. The IHistClm50BgcQianGs compset (long name HIST_DATM%QIA_CLM50%BGC_SICE_SOCN_MOSART_SGLC_SWAV) works just fine in single point mode.