Problem in downloading input data sets (ERROR: Could not connect to repo '{0}') -- clm5.0

Z

zorkgtzorkgt

Xie Zhijie
New Member
divanova@ucsd_edu said:
I have similar problem and I don't think it is related to the CLM5.0. I am working with the latest release cesm2.1.3 on pleaides (NASA)
After compiling successfully my case I tried several ways to get the missing inputdata files:

1) check_input_data with wget- NO SUCCESS
pfe23.divanova 226> ./check_input_data -i /nobackup/divanova/cesm2/inputdata/ --download --p wget --server ftp://ftp.cgd.ucar.edu/cesm/inputdata
Using protocol wget with user None and passwd None
Could not connect to repo 'ftp://ftp.cgd.ucar.edu/cesm/inputdata'
This is most likely either a proxy, or network issue .
ERROR: Could not find all inputdata on any server

2) wget manually (one file) - SUCCESS

pfe23.divanova 227> wget ftp://ftp.cgd.ucar.edu/cesm/inputdata/ocn/pop/gx1v7/grid/region_mask_20151008.ieeei4
--2020-03-31 22:52:15-- ftp://ftp.cgd.ucar.edu/cesm/inputdata/ocn/pop/gx1v7/grid/region_mask_20151008.ieeei4
=> 'region_mask_20151008.ieeei4.1'
Resolving ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)... 128.117.23.220
Connecting to ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)|128.117.23.220|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /cesm/inputdata/ocn/pop/gx1v7/grid ... done.
==> SIZE region_mask_20151008.ieeei4 ... 491520
==> PASV ... done. ==> RETR region_mask_20151008.ieeei4 ... done.
Length: 491520 (480K) (unauthoritative)
100%[================================================================================================================================>] 491,520 2.20MB/s in 0.2s
2020-03-31 22:52:16 (2.20 MB/s) - 'region_mask_20151008.ieeei4.1' saved [491520]

3) check_input_data with svn - NO SUCCESS
pfe23.divanova 229> ./check_input_data -i /nobackup/divanova/cesm2/inputdata/ --download --p svn --server https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata

Using protocol svn with user None and passwd None
Loading input file list: 'Buildconf/cam.input_data_list'
Model cam missing file ncdata = '/nobackup/divanova/cesm2/inputdata/atm/cam/inic/fv/cami-mam3_0000-01-01_0.9x1.25_L30_c100618.nc'
Trying to download file: 'atm/cam/inic/fv/cami-mam3_0000-01-01_0.9x1.25_L30_c100618.nc' to path '/nobackup/divanova/cesm2/inputdata/atm/cam/inic/fv/cami-mam3_0000-01-01_0.9x1.25_L30_c100618.nc' using SVN protocol.
SUCCESS

ERROR: Failed to find or download file /nobackup/divanova/b.BC5L45BGC.f09_g16/run/inputdata_checksum.dat

4) svn manually (one directory) - SUCCESS
pfe23.divanova 233> svn co https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/ocn/pop/gx1v7/grid/
A grid/topography_20161215.ieeei4
A grid/region_mask_20151008.ieeei4
A grid/horiz_grid_20010402.ieeer8
Checked out revision 33431.

I suspect the issues are different in the different protocols but any suggestions would be appreciated.
Click to expand...
divanova@ucsd_edu said:
Yes I am in my case directory.
here is the end of what I got running the command you suggested:
===========================================================

Using protocol wget with user anonymous and passwd user@example.edu


RUN: wget --no-check-certificate --user anonymous --password user@example.edu --spider ftp://ftp.cgd.ucar.edu/cesm/inputdata


stat: 8





errput: --2020-04-01 09:02:17-- ftp://ftp.cgd.ucar.edu/cesm/inputdata


=> 'inputdata'


Resolving ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)... 128.117.23.220


Connecting to ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)|128.117.23.220|:21... connected.


Logging in as anonymous ... Logged in!


==> SYST ... done. ==> PWD ... done.


==> TYPE I ... done. ==> CWD (1) /cesm ... done.


==> SIZE inputdata ... done.





==> PASV ... done. --2020-04-01 09:02:17-- ftp://ftp.cgd.ucar.edu/cesm/inputdata


=> '.listing'


==> CWD (1) /cesm ... done.


==> PASV ... done. ==> LIST ... done.





0K 3.72K=0.2s





Removed '.listing'.





File 'inputdata' exists.





Could not connect to repo 'ftp://ftp.cgd.ucar.edu/cesm/inputdata'


This is most likely either a proxy, or network issue .


Trying to download file: '../inputdata_checksum.dat' to path '/nobackup/divanova/b.BC5L45BGC.f09_g16/run/inputdata_checksum.dat.raw' using NoneType protocol.


Traceback (most recent call last):


File "./check_input_data", line 76, in <module>


_main_func(__doc__)


File "./check_input_data", line 71, in _main_func


chksum=chksum) else 1)


File "/home4/divanova/cesm2/cesm2.1.3.repo/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 163, in check_all_input_data


_download_checksum_file(self.get_value("RUNDIR"))


File "/home4/divanova/cesm2/cesm2.1.3.repo/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 54, in _download_checksum_file


success = server.getfile(rel_path, new_file)


AttributeError: 'NoneType' object has no attribute 'getfile'


pf
Click to expand...
Hello，have you solve your problem 1 now？
I have similar problem to build a case
 
J

jbuzan@purdue_edu

Member
Hi CESM staff,
I, likewise, am having a similar issue. But when I connect using one of the suggested methods above, I get an error saying that there's no such directory as 'cesm'.

Cheers,
-Jonathan


I executed:
> wget --no-check-certificate --user anonymous --password user@example.edu --spider ftp://ftp.cgd.ucar.edu/cesm/inputdata
--2022-01-05 16:10:45-- ftp://ftp.cgd.ucar.edu/cesm/inputdata
=> ‘inputdata’
Resolving ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)... 128.117.23.220
Connecting to ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)|128.117.23.220|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /cesm ...
No such directory ‘cesm’.
 
J

jbuzan@purdue_edu

Member
Addendum:
I contacted my IT staff at my cluster.

../cime/config/cesm/
modify
config_inputdata.xml

Put the SVN command first. The staff said it downloads the first file, then the checksum cannot be found.

Cheers,
-Jonathan
 
A

aschloegl

Alois Schlögl
New Member
jedwards said:
Seems like a problem on our end, I'm looking into it now.
Click to expand...
I git the same problem with cesm-2.1.3. A slash "/" is missing at the end of the address. This patch fixes this:

diff --git a/config/cesm/config_inputdata.xml b/config/cesm/config_inputdata.xml index 985f5ee02..9274bb896 100644 --- a/config/cesm/config_inputdata.xml +++ b/config/cesm/config_inputdata.xml @@ -16,7 +16,7 @@ <server> <protocol>wget</protocol> - <address>ftp://ftp.cgd.ucar.edu/cesm/inputdata</address> + <address>ftp://ftp.cgd.ucar.edu/cesm/inputdata/</address> <user>anonymous</user> <password>user@example.edu</password> <checksum>../inputdata_checksum.dat</checksum> @@ -25,7 +25,7 @@ <server> <comment> ftp requires the python package ftplib </comment> <protocol>ftp</protocol> - <address>ftp.cgd.ucar.edu/cesm/inputdata</address> + <address>ftp.cgd.ucar.edu/cesm/inputdata/</address> <user>anonymous</user> <password>user@example.edu</password> <checksum>../inputdata_checksum.dat</checksum>

To give credit where credit is due, the solution was posted first here:
bb.cgd.ucar.edu

Porting to a local machine, can't automatically download the input data.

I am trying to run CLM5.0 on my local machine. I was able to successfully build a case and run the model in the past (using the same machine), but I am getting an error this time (with case.submit while trying to download the input data), which seems to be related to connection to the repository...
bb.cgd.ucar.edu bb.cgd.ucar.edu
 
You must log in or register to reply here.
Back
Top