Problem with downloading input data

[minkwan]

I have replaced my config_inputdata.xml with the latest one from the link above. When I run check_input_data --download --debug --protocol wget, I got an error message as :

==========================
....

2025-09-10 07:54:05 (647 KB/s) - ‘/media/data/mkk1u16/projects/cesm/scratch/testrun_3/run/inputdata_checksum.dat.raw’ saved [6854903/6854903]

SUCCESS

Using protocol wget with user and passwd
RUN: wget --no-check-certificate --spider - Revision 70790: /trunk/inputdata
errput: Spider mode enabled. Check if remote file exists.
--2025-09-10 07:54:05-- - Revision 70790: /trunk/inputdata
Resolving svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)... 128.117.13.55
Connecting to svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)|128.117.13.55|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

Trying to download file: 'None' to path '/media/data/mkk1u16/projects/cesm/scratch/testrun_3/run/inputdata_checksum.dat.raw' using WGET protocol.
Traceback (most recent call last):
File "./check_input_data", line 76, in <module>
_main_func(__doc__)
File "./check_input_data", line 71, in _main_func
chksum=chksum) else 1)
File "/media/data/mkk1u16/cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 164, in check_all_input_data
_download_checksum_file(self.get_value("RUNDIR"))
File "/media/data/mkk1u16/cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 55, in _download_checksum_file
success = server.getfile(rel_path, new_file)
File "/media/data/mkk1u16/cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/Servers/wget.py", line 49, in getfile
full_url = os.path.join(self._server_loc, rel_path)
File "/home/mkk1u16/.pyenv/versions/3.7.17/lib/python3.7/posixpath.py", line 94, in join
genericpath._check_arg_types('join', a, *p)
File "/home/mkk1u16/.pyenv/versions/3.7.17/lib/python3.7/genericpath.py", line 153, in _check_arg_types
(funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'NoneType'
==========================
I also attached check_input_data.log file.

 

[jedwards]

Try adding a -r to your wget command. Try adding it here cime/CIME/Servers/wget.py at master · ESMCI/cime

The error that I do not understand is Trying to download file: 'None'. Is there a filename listed as None in any of your input_data_list files in the Buildconf subdirectory of your case?

 

[minkwan]

I have check it, but there is no input_data_list files in the Buildconf containing 'None'

 

[minkwan]

BTW, I have checked '- Revision 70791: /trunk/inputdata' but there is no 'inputdata_checksum.dat'. Will it be a problem?

 

[jedwards]

The checksum is not on the svn server, it is on https://ftp.cgd.ucar.edu/cesm/inputdata_checksum.dat
I don't know where your python error is coming from - do you have another machine you can try from?

 

[minkwan]

Unfortunately, my other linux machine gives a same error.

It seems some reason it try to download inputdat_checksum.dat from - Revision 70791: /trunk/inputdata after it downloaded the file successfully from Index of /cesm/inputdata

 

[jedwards]

Is that because you still have a checksum file associated with your svn server?

 

[minkwan]

Honestly, I am not sure why it is trying to download it even if it already downloaded it from ftp server. As I thought it was the main reason for the error message, I have forced not to access svn server by deleting it from .xml file. Then actually it works. But having ftp server raise other issue downloading the entire input file. Therefore, I tried suppressing to use inputdat_check by altering its python script and removing ftp server (but adding svn). Then it works ok and download the input file only needed. I am not sure the reason for it, and how I can resolve it appropriately. Any thought?

 

[inos@bas_ac_uk]

I have accidently added '<checksum>../inputdata_checksum.dat</checksum>' on the end of every server in the config_inputdata.xml. It seems this resolved the issue. For testing whether the issue was resolve by accidently updating config_inputdata file, I am trying to run a test case in CESM 2.1 (WACCM6) and CEAM 2.1 (CAM6-chem) tutorial (excise 1) as:

• ./create_newcase --case $HOMEDATA/projects/cesm/scratch/testrun_3 --compset FW2000climo --res f09_f09_mg17

And I run check_input_data --download --protocol wget. But it seems it keep downloading inputdata for several hours. Now its downloaded input data is over 400GB. Is it normal?

Thanks for this hint! This is getting me past the issue with apparently trying to download a file called "None" and so far it has only been downloading files I actually need. I'll keep monitoring to make sure I'm not accidentally downloading unneeded input files, but so far so good. It's a strange solution to a strange problem though and I don't get at all why this is making the difference.

 

[Andy_Wai]

The checksum is not on the svn server, it is on https://ftp.cgd.ucar.edu/cesm/inputdata_checksum.dat
I don't know where your python error is coming from - do you have another machine you can try from?

the link https://ftp.cgd.ucar.edu/cesm/inputdata_checksum.dat , it does not work!
this following links do not work as well:

https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/

https://ftp.cgd.ucar.edu/cesm/inputdata/

PS: I am using CESM 2.1.5;