Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Porting to a local machine, can't automatically download the input data.

I am trying to run CLM5.0 on my local machine. I was able to successfully build a case and run the model in the past (using the same machine), but I am getting an error this time (with case.submit while trying to download the input data), which seems to be related to connection to the repository using "wget" command (please see error message below). I checked some of the discussions here and found that there are other folks having similar issue as a result of their domain being blocked or blacklisted, which is not the case for me as checked with help@cgd.ucar.edu. I also prioritized the svn and gftp protocols (i.e., over wget) in the config_inputdata.xml file, however, those were also disabled. I would greatly appreciate if you could check the error message and let me know of any potential solution.

git describe:
release-clm5.0.34-3-gcdc544df

./manage_externals/checkout_externals --status --verbose:
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_CISM.cfg
Checking status of externals: clm, fates, ptclm, cism, source_cism, rtm, mosart, cime,
./cime
clean sandbox, on cime5.6.33
./components/cism
clean sandbox, on cism-release-cesm2.1.2_02
./components/cism/source_cism
clean sandbox, on release-cism2.1.03
./components/mosart
clean sandbox, on release-cesm2.0.04
./components/rtm
clean sandbox, on release-cesm2.0.04
./src/fates
clean sandbox, on sci.1.30.0_api.8.0.0
./tools/PTCLM
clean sandbox, on PTCLM2_20200121



=======================================================
Client protocol gftp not enabled
Using protocol wget with user anonymous and passwd user@example.edu (mailto:user@example.edu)
Could not connect to repo 'ftp://ftp.cgd.ucar.edu/cesm/inputdata'
This is most likely either a proxy, or network issue .
Trying to download file: '../inputdata_checksum.dat' to path
'/media/data/ENE828/2020Fall/CESM2/out/test_I1PtClm50SpGs_mexicocityMEX/run/inputdata_checksum.dat.raw'
using NoneType protocol.
Traceback (most recent call last):
File "./case.submit", line 107, in <module>
_main_func(__doc__)
File "./case.submit", line 104, in _main_func
mail_user=mail_user, mail_type=mail_type, batch_args=batch_args)
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 157, in submit
custom_success_msg_functor=verbatim_success_msg)
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 1683, in run_and_log_case_status
rv = func()
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 155, in <lambda>
batch_args=batch_args)
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 85, in _submit
case.check_case()
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 171, in check_case
self.check_all_input_data()
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py",line 163, in check_all_input_data
_download_checksum_file(self.get_value("RUNDIR"))
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 54, in _download_checksum_file
success = server.getfile(rel_path, new_file)
AttributeError: 'NoneType' object has no attribute 'getfile'
 

Attachments

  • config_machines.zip
    717 bytes · Views: 8

jedwards

CSEG and Liaisons
Staff member
This line "using NoneType protocol." indicates that none of the protocols in your config_inputdata.xml file were used
are you sure that the format is valid xml after your changes? Can you connect with a brower to repo 'ftp://ftp.cgd.ucar.edu/cesm/inputdata'?
Can you connect to this site from the node you are running on?
 
Thanks Jim for your quick reply. I just tried to connect to the ftp link from the node and it worked. I also checked the xml file, also replaced it with the original one. Still got the same error.

Thanks
 

jedwards

CSEG and Liaisons
Staff member
From your case directory please run ./check_input_data --download --debug and post the output and the file check_input_data.log
 
./check_input_data --download --debug
RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_entry_id.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_case.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_case.xml validates

RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_entry_id.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_run.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_run.xml validates

RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_entry_id.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_build.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_build.xml validates

RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_mach_pes.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_mach_pes.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_mach_pes.xml validates

RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_batch.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_batch.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_batch.xml validates

RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_mach_specific.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_mach_specific.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_mach_specific.xml validates

RUN: /media/data/PACKAGES/anaconda/DIR/bin/xmllint --noout --schema /media/data/ENE828/2020Fall/CESM2/clm5.0/cime/config/xml_schemas/env_archive.xsd /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_archive.xml
errput: /media/data/ENE828/2020Fall/CESM2/cases/test_I1PtClm50SpGs_mexicocityMEX/env_archive.xml validates

Loading input file list: 'Buildconf/clm.input_data_list'
Model clm missing file fsnowoptics = '/media/data/ENE828/2020Fall/CESM2/cesm_input/lnd/clm2/snicardata/snicar_optics_5bnd_c090915.nc'
Model clm missing file fatmlndfrc = '/media/data/ENE828/2020Fall/CESM2/cesm_input/share/domains/domain.clm/domain.lnd.1x1pt-mexicocityMEX_navy.090715.nc'
Model clm missing file fsurdat = '/media/data/ENE828/2020Fall/CESM2/cesm_input/lnd/clm2/surfdata_map/release-clm5.0.18/surfdata_1x1_mexicocityMEX_hist_16pfts_Irrig_CMIP6_simyr2000_c190214.nc'
Model clm missing file finidat = '/media/data/ENE828/2020Fall/CESM2/cesm_input/lnd/clm2/initdata_map/clmi.I2000Clm50BgcCrop.2011-01-01.1.9x2.5_gx1v7_gl4_simyr2000_c190312.nc'
Model clm missing file paramfile = '/media/data/ENE828/2020Fall/CESM2/cesm_input/lnd/clm2/paramdata/clm5_params.c171117.nc'
Model clm missing file fsnowaging = '/media/data/ENE828/2020Fall/CESM2/cesm_input/lnd/clm2/snicardata/snicar_drdt_bst_fit_60_c070416.nc'
Model clm missing file megan_factors_file = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/cam/chem/trop_mozart/emis/megan21_emis_factors_78pft_c20161108.nc'
Model clm missing file stream_fldfilename_urbantv = '/media/data/ENE828/2020Fall/CESM2/cesm_input/lnd/clm2/urbandata/CLM50_tbuildmax_Oleson_2016_0.9x1.25_simyr1849-2106_c160923.nc'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/datm.input_data_list'
Model datm missing file domain1 = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/datm7/domain.clm/domain.lnd.1x1pt-mexicocityMEX_navy.090715.nc'
Model datm missing file file1 = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/datm7/CLM1PT_data/mexicocityMEX.c080124/clm1pt-1993-12.nc'
Model datm missing file domain1 = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/cam/chem/trop_mozart_aero/aero/aerosoldep_WACCM.ensmean_monthly_hist_1849-2015_0.9x1.25_CMIP6_c180926.nc'
Model datm missing file file1 = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/cam/chem/trop_mozart_aero/aero/aerosoldep_WACCM.ensmean_monthly_hist_1849-2015_0.9x1.25_CMIP6_c180926.nc'
Model datm missing file domain1 = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/datm7/topo_forcing/topodata_0.9x1.25_USGS_070110_stream_c151201.nc'
Model datm missing file file1 = '/media/data/ENE828/2020Fall/CESM2/cesm_input/atm/datm7/topo_forcing/topodata_0.9x1.25_USGS_070110_stream_c151201.nc'
Model datm missing file domainfile = '/media/data/ENE828/2020Fall/CESM2/cesm_input/share/domains/domain.clm/domain.lnd.1x1pt-mexicocityMEX_navy.090715.nc'
Loading input file list: 'Buildconf/mosart.input_data_list'
Client protocol gftp not enabled
Using protocol wget with user anonymous and passwd user@example.edu
RUN: wget --no-check-certificate --user anonymous --password user@example.edu --spider ftp://ftp.cgd.ucar.edu/cesm/inputdata
stat: 8

errput: --2020-10-01 11:18:58-- ftp://ftp.cgd.ucar.edu/cesm/inputdata
=> ‘inputdata’
Resolving ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)... 128.117.23.220
Connecting to ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)|128.117.23.220|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /cesm ... done.
==> SIZE inputdata ... done.

==> PASV ... done. --2020-10-01 11:18:58-- ftp://ftp.cgd.ucar.edu/cesm/inputdata
=> ‘.listing’
==> CWD (1) /cesm ... done.
==> PASV ... done. ==> LIST ... done.

0K 131K=0.006s

Removed ‘.listing’.

File ‘inputdata’ exists.

Could not connect to repo 'ftp://ftp.cgd.ucar.edu/cesm/inputdata'
This is most likely either a proxy, or network issue .
Trying to download file: '../inputdata_checksum.dat' to path '/media/data/ENE828/2020Fall/CESM2/out/test_I1PtClm50SpGs_mexicocityMEX/run/inputdata_checksum.dat.raw' using NoneType protocol.
Traceback (most recent call last):
File "./check_input_data", line 76, in <module>
_main_func(__doc__)
File "./check_input_data", line 71, in _main_func
chksum=chksum) else 1)
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 163, in check_all_input_data
_download_checksum_file(self.get_value("RUNDIR"))
File "/media/data/ENE828/2020Fall/CESM2/clm5.0/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 54, in _download_checksum_file
success = server.getfile(rel_path, new_file)
AttributeError: 'NoneType' object has no attribute 'getfile'
 

Attachments

  • check_input_data.zip
    7.5 KB · Views: 3

jedwards

CSEG and Liaisons
Staff member
I can't see the problem. This line
> File ‘inputdata’ exists.
makes me wonder if there is an existing file where cime is expecting a directory.
What is in:
/media/data/ENE828/2020Fall/CESM2/cesm_input
 

pma2107

Patrick Alexander
New Member
I had the same problem but discovered that the problem was with the ftp command:

wget --no-check-certificate --user anonymous --password user@example.edu --spider ftp://ftp.cgd.ucar.edu/cesm/inputdata

Since there is no "/" at the end of "ftp://ftp.cgd.ucar.edu/cesm/inputdata", wget seems to interpret it as a file rather than a directory, and the wget command throws an error.

The solution was to edit the config_inputdata.xml file in cime/config/cesm, adding a slash at the end of the ftp address, changing ftp://ftp.cgd.ucar.edu/cesm/inputdata to ftp://ftp.cgd.ucar.edu/cesm/inputdata/

<server>
<protocol>wget</protocol>
<address>ftp://ftp.cgd.ucar.edu/cesm/inputdata/</address>
<user>anonymous</user>
<password>user@example.edu</password>
<checksum>../inputdata_checksum.dat</checksum>
</server>

This appears to solve the problem. Editing wget.py as in the previous post should also solve the problem, but only by ignoring the error produced by the wget command.
 
Top