Problem with downloading input data

[johanna_teresa]

Hi all,

I am trying to download required input data for an FWHIST compset run but when trying to download input data I keep getting error messages. I have tried moving the svn protocol to before the wget protocol in case there was an issue with wget, but it says 'Client protocol svn not enabled' and directly attempts to download via the wget protocol. It then successfully downloads the 'inputdata_checksum.dat' file but fails directly after.

Trying to download file: '../inputdata_checksum.dat' to path '/capstor/scratch/cscs/jmalle/waccm_test_1/waccm_test_1/run/inputdata_checksum.dat.raw' using WGET protocol.
SUCCESS

It fails with the following message when running ./check_input_data --download --debug:

Using protocol wget with user  and passwd
RUN: wget --no-check-certificate  --spider https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/
  errput: Spider mode enabled. Check if remote file exists.
--2025-04-10 16:14:38--  https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/
Resolving svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)... 128.117.13.55
Connecting to svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)|128.117.13.55|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

I have copied my entries to config_input_data.xml below.

When I completely delete the e.g.

.../inputdata/cesm2_init/f.e21.FWHIST.f09_f09_mg17.cesm2.1-exp011.001_v2/1979-01-01/

folder, it is attempting to download things already during the ./preview_namelist call, but somehow it adds many more files than what I see on - Revision 70327: /trunk/inputdata/cesm2_init/f.e21.FWHIST.f09_f09_mg17.cesm2.1-exp011.001_v2/1979-01-01


Any hints would be greatly appreciated.

Thanks a lot,
Johanna

What version of the code are you using? CESM 2.1.3

Have you made any changes to files in the source tree? I have played around with my config_inputdata.xml script. Currently it looks like this:

<inputdata>
  <!-- server precidence is order in this file.  Highest preference at top -->
  <!-- If the client doesn't have the protocol it will be skipped -->
  <!-- chksum verification of inputfiles is possible.  If a file with name -->
  <!-- inputdata_chksum.dat is found on the server in the directory above inputdata -->
  <!-- it will be searched for filename and chksum of each downloaded file.  -->
  <!-- see the file ftp://ftp.cgd.ucar.edu/cesm/inputdata_chksum.dat for proper format. -->

  <server>
    <protocol>svn</protocol>
    <address>https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata</address>
  </server>

 <server>
    <protocol>wget</protocol>
    <address>https://ftp.cgd.ucar.edu/cesm/inputdata/</address>
    <checksum>../inputdata_checksum.dat</checksum>
  </server>

  <server>
    <comment> wget access to subversion </comment>
    <protocol>wget</protocol>
    <address>https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/</address>
  </server>

  <server>
    <protocol>svn</protocol>
    <address>https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata</address>
  </server>

  <server>
    <protocol>wget</protocol>
    <address>https://redoak.cs.toronto.edu/twitcher/ows/proxy/thredds/fileServer/datasets/CESM/inputdata/</address>
  </server>

</inputdata>

 

[jedwards]

I think that the problem may be that you are using an old version of the code. Please update to the latest cesm2.1.x (at this time 2.1.5) and
try again.

 

[inos@bas_ac_uk]

Hi,

I'm having a very similar problem, but using CESM2.1.5. When I do check_input_data --download --debug it seems to successfully complete the first step:

2025-07-31 11:10:06 (633 KB/s) - ‘/work/n02/n02/inos/cesm/CESM2.1.5/runs/f.e215.FXHIST.f19_f19.ssp5-8.5t/run/inputdata_checksum.dat.raw’ saved [6854903/6854903]

SUCCESS

Although, when I check the directory above for the file 'inputdata_checksum.dat.raw' it actually only contains 'inputdata_checksum.dat.tmp'. I'm not sure if that points to the problem, but it fails on the next step with the following:

Using protocol wget with user and passwd
RUN: wget --no-check-certificate --spider - Revision 70721: /trunk/inputdata
errput: Spider mode enabled. Check if remote file exists.
--2025-07-31 11:10:06-- - Revision 70721: /trunk/inputdata
Resolving svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)... 128.117.13.55
Connecting to svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)|128.117.13.55|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: - Revision 70721: /trunk/inputdata [following]
Spider mode enabled. Check if remote file exists.
--2025-07-31 11:10:07-- - Revision 70721: /trunk/inputdata
Connecting to svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)|128.117.13.55|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

Trying to download file: 'None' to path '/work/n02/n02/inos/cesm/CESM2.1.5/runs/f.e215.FXHIST.f19_f19.ssp5-8.5t/run/inputdata_checksum.dat.raw' using WGET protocol.
Traceback (most recent call last):
File "/mnt/lustre/a2fs-work2/work/n02/n02/inos/cesm/CESM2.1.5/case/f.e215.FXHIST.f19_f19.ssp5-8.5t/./check_input_data", line 76, in <module>
_main_func(__doc__)
File "/mnt/lustre/a2fs-work2/work/n02/n02/inos/cesm/CESM2.1.5/case/f.e215.FXHIST.f19_f19.ssp5-8.5t/./check_input_data", line 66, in _main_func
sys.exit(0 if case.check_all_input_data(protocol=protocol,
File "/mnt/lustre/a2fs-work2/work/n02/n02/inos/cesm/CESM2.1.5/my_cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 164, in check_all_input_data
_download_checksum_file(self.get_value("RUNDIR"))
File "/mnt/lustre/a2fs-work2/work/n02/n02/inos/cesm/CESM2.1.5/my_cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 55, in _download_checksum_file
success = server.getfile(rel_path, new_file)
File "/mnt/lustre/a2fs-work2/work/n02/n02/inos/cesm/CESM2.1.5/my_cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/Servers/wget.py", line 49, in getfile
full_url = os.path.join(self._server_loc, rel_path)
File "/opt/cray/pe/python/3.9.13.1/lib/python3.9/posixpath.py", line 90, in join
genericpath._check_arg_types('join', a, *p)
File "/opt/cray/pe/python/3.9.13.1/lib/python3.9/genericpath.py", line 152, in _check_arg_types
raise TypeError(f'{funcname}() argument must be str, bytes, or '
TypeError: join() argument must be str, bytes, or os.PathLike object, not 'NoneType'

Any help with this would be much appreciated!

Thanks,
Ingrid

 

[jedwards]

What happens when you run from your case directory:

./check_input_data

?
How about

./check_input_data --download

 

[inos@bas_ac_uk]

Hi, I'm atttaching two files with the output for the above two commands. Does that give you any further clues? I'm off on holiday now, but will check back when I'm back at work (from Aug 11th). Thanks for your help!

Ingrid

 

[inos@bas_ac_uk]

Hello again,

I was just checking for a response and obviously nothing yet, but I found that there is also a check_input_data log file which I had missed previously. I'm attaching it here in case it's useful. It looks to me like the script is somehow not getting the correct filename so that it ends up trying to download a file called "None" which then fails. But I don't understand the script well enough to figure out why this happens or how to fix it. I hope someone can help!

Thanks very much,
Ingrid