Porting to a local machine, can't automatically download the input data.

[felfelan@egr_msu_edu]

I am trying to run CLM5.0 on my local machine. I was able to successfully build a case and run the model in the past (using the same machine), but I am getting an error this time (with case.submit while trying to download the input data), which seems to be related to connection to the repository using "wget" command (please see error message below). I checked some of the discussions here and found that there are other folks having similar issue as a result of their domain being blocked or blacklisted, which is not the case for me as checked with help@cgd.ucar.edu. I also prioritized the svn and gftp protocols (i.e., over wget) in the config_inputdata.xml file, however, those were also disabled. I would greatly appreciate if you could check the error message and let me know of any potential solution.

git describe:
release-clm5.0.34-3-gcdc544df

./manage_externals/checkout_externals --status --verbose:
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_CISM.cfg
Checking status of externals: clm, fates, ptclm, cism, source_cism, rtm, mosart, cime,
./cime
clean sandbox, on cime5.6.33
./components/cism
clean sandbox, on cism-release-cesm2.1.2_02
./components/cism/source_cism
clean sandbox, on release-cism2.1.03
./components/mosart
clean sandbox, on release-cesm2.0.04
./components/rtm
clean sandbox, on release-cesm2.0.04
./src/fates
clean sandbox, on sci.1.30.0_api.8.0.0
./tools/PTCLM
clean sandbox, on PTCLM2_20200121



=======================================================
Client protocol gftp not enabled
Using protocol wget with user anonymous and passwd user@example.edu (mailto:user@example.edu)
Could not connect to repo 'ftp://ftp.cgd.ucar.edu/cesm/inputdata'
This is most likely either a proxy, or network issue .
Trying to download file: '../inputdata_checksum.dat' to path
'/media/data/ENE828/2020Fall/CESM2/out/test_I1PtClm50SpGs_mexicocityMEX/run/inputdata_checksum.dat.raw'
using NoneType protocol.
Traceback (most recent call last):
File "./case.submit", line 107, in <module>
_main_func(__doc__)
File "./case.submit", line 104, in _main_func
mail_user=mail_user, mail_type=mail_type, batch_args=batch_args)
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 157, in submit
custom_success_msg_functor=verbatim_success_msg)
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 1683, in run_and_log_case_status
rv = func()
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 155, in <lambda>
batch_args=batch_args)
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 85, in _submit
case.check_case()
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 171, in check_case
self.check_all_input_data()
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py",line 163, in check_all_input_data
_download_checksum_file(self.get_value("RUNDIR"))
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 54, in _download_checksum_file
success = server.getfile(rel_path, new_file)
AttributeError: 'NoneType' object has no attribute 'getfile'

 

[jedwards]

This line "using NoneType protocol." indicates that none of the protocols in your config_inputdata.xml file were used
are you sure that the format is valid xml after your changes? Can you connect with a brower to repo 'ftp://ftp.cgd.ucar.edu/cesm/inputdata'?
Can you connect to this site from the node you are running on?

 

[felfelan@egr_msu_edu]

Thanks Jim for your quick reply. I just tried to connect to the ftp link from the node and it worked. I also checked the xml file, also replaced it with the original one. Still got the same error.

Thanks

 

[jedwards]

From your case directory please run ./check_input_data --download --debug and post the output and the file check_input_data.log

 

[felfelan@egr_msu_edu]

./check_input_data --download --debug
RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_entry_id.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_case.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_case.xml validates

RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_entry_id.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_run.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_run.xml validates

RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_entry_id.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_build.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_build.xml validates

RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_mach_pes.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_mach_pes.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_mach_pes.xml validates

RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_batch.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_batch.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_batch.xml validates

RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_mach_specific.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_mach_specific.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_mach_specific.xml validates

RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_archive.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_archive.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_archive.xml validates

Loading input file list: 'Buildconf/clm.input_data_list'
Model clm missing file fsnowoptics = '/media/data/ENE828/2020Fall/CESM2/cesm_input/lnd/clm2/snicardata/snicar_optics_5bnd_c090915.nc'
Model clm missing file fatmlndfrc = '/media/data/ENE828/2020Fall/CESM2/cesm_input/share/domains/domain.clm/domain.lnd.1x1pt-mexicocityMEX_navy.090715.nc'
Model clm missing file fsurdat = '/media/data/ENE828/2020Fall/CESM2/cesm_input/lnd/clm2/surfdata_map/release-clm5.0.18/surfdata_1x1_mexicocityMEX_hist_16pfts_Irrig_CMIP6_simyr2000_c190214.nc'
Model clm missing file finidat = '/media/data/ENE828/2020Fall/CESM2/cesm_input/lnd/clm2/initdata_map/clmi.I2000Clm50BgcCrop.2011-01-01.1.9x2.5_gx1v7_gl4_simyr2000_c190312.nc'
Model clm missing file paramfile = '/media/data/ENE828/2020Fall/CESM2/cesm_input/lnd/clm2/paramdata/clm5_params.c171117.nc'
Model clm missing file fsnowaging = '/media/data/ENE828/2020Fall/CESM2/cesm_input/lnd/clm2/snicardata/snicar_drdt_bst_fit_60_c070416.nc'
Model clm missing file megan_factors_file = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/cam/chem/trop_mozart/emis/megan21_emis_factors_78pft_c20161108.nc'
Model clm missing file stream_fldfilename_urbantv = '/media/data/ENE828/2020Fall/CESM2/cesm_input/lnd/clm2/urbandata/CLM50_tbuildmax_Oleson_2016_0.9x1.25_simyr1849-2106_c160923.nc'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/datm.input_data_list'
Model datm missing file domain1 = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/datm7/domain.clm/domain.lnd.1x1pt-mexicocityMEX_navy.090715.nc'
Model datm missing file file1 = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/datm7/CLM1PT_data/mexicocityMEX.c080124/clm1pt-1993-12.nc'
Model datm missing file domain1 = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/cam/chem/trop_mozart_aero/aero/aerosoldep_WACCM.ensmean_monthly_hist_1849-2015_0.9x1.25_CMIP6_c180926.nc'
Model datm missing file file1 = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/cam/chem/trop_mozart_aero/aero/aerosoldep_WACCM.ensmean_monthly_hist_1849-2015_0.9x1.25_CMIP6_c180926.nc'
Model datm missing file domain1 = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/datm7/topo_forcing/topodata_0.9x1.25_USGS_070110_stream_c151201.nc'
Model datm missing file file1 = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/datm7/topo_forcing/topodata_0.9x1.25_USGS_070110_stream_c151201.nc'
Model datm missing file domainfile = '/media/data/ENE828/2020Fall/CESM2/cesm_input/share/domains/domain.clm/domain.lnd.1x1pt-mexicocityMEX_navy.090715.nc'
Loading input file list: 'Buildconf/mosart.input_data_list'
Client protocol gftp not enabled
Using protocol wget with user anonymous and passwd user@example.edu
RUN: wget --no-check-certificate --user anonymous --password user@example.edu --spider ftp://ftp.cgd.ucar.edu/cesm/inputdata
stat: 8

errput: --2020-10-01 11:18:58-- ftp://ftp.cgd.ucar.edu/cesm/inputdata
=> ‘inputdata’
Resolving ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)... 128.117.23.220
Connecting to ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)|128.117.23.220|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /cesm ... done.
==> SIZE inputdata ... done.

==> PASV ... done. --2020-10-01 11:18:58-- ftp://ftp.cgd.ucar.edu/cesm/inputdata
=> ‘.listing’
==> CWD (1) /cesm ... done.
==> PASV ... done. ==> LIST ... done.

0K 131K=0.006s

Removed ‘.listing’.

File ‘inputdata’ exists.

Could not connect to repo 'ftp://ftp.cgd.ucar.edu/cesm/inputdata'
This is most likely either a proxy, or network issue .
Trying to download file: '../inputdata_checksum.dat' to path '/media/data/ENE828/2020Fall/CESM2/out/test_I1PtClm50SpGs_mexicocityMEX/run/inputdata_checksum.dat.raw' using NoneType protocol.
Traceback (most recent call last):
File "./check_input_data", line 76, in <module>
_main_func(__doc__)
File "./check_input_data", line 71, in _main_func
chksum=chksum) else 1)
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 163, in check_all_input_data
_download_checksum_file(self.get_value("RUNDIR"))
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 54, in _download_checksum_file
success = server.getfile(rel_path, new_file)
AttributeError: 'NoneType' object has no attribute 'getfile'

 

[jedwards]

I can't see the problem. This line
> File ‘inputdata’ exists.
makes me wonder if there is an existing file where cime is expecting a directory.
What is in:
/media/data/ENE828/2020Fall/CESM2/cesm_input

 

[felfelan@egr_msu_edu]

I made the subdirectories (atm, forcing, lnd, share) in /media/data/ENE828/2020Fall/CESM2/cesm_input, which are all empty.

 

[felfelan@egr_msu_edu]

I think I solved the issue of wget connecting to the repository by commenting out these two lines in clm5.0/cime/scripts/lib/CIME/Servers/wget.py. Seems that in both cases in "try except" it returned "None" which doesn't seem correct!

 

[pma2107]

I had the same problem but discovered that the problem was with the ftp command:

wget --no-check-certificate --user anonymous --password user@example.edu --spider ftp://ftp.cgd.ucar.edu/cesm/inputdata

Since there is no "/" at the end of "ftp://ftp.cgd.ucar.edu/cesm/inputdata", wget seems to interpret it as a file rather than a directory, and the wget command throws an error.

The solution was to edit the config_inputdata.xml file in cime/config/cesm, adding a slash at the end of the ftp address, changing ftp://ftp.cgd.ucar.edu/cesm/inputdata to ftp://ftp.cgd.ucar.edu/cesm/inputdata/

<server>
<protocol>wget</protocol>
<address>ftp://ftp.cgd.ucar.edu/cesm/inputdata/</address>
<user>anonymous</user>
<password>user@example.edu</password>
<checksum>../inputdata_checksum.dat</checksum>
</server>

This appears to solve the problem. Editing wget.py as in the previous post should also solve the problem, but only by ignoring the error produced by the wget command.