Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

Problem with downloading input data

johanna_teresa

Johanna Malle
New Member
Hi all,

I am trying to download required input data for an FWHIST compset run but when trying to download input data I keep getting error messages. I have tried moving the svn protocol to before the wget protocol in case there was an issue with wget, but it says 'Client protocol svn not enabled' and directly attempts to download via the wget protocol. It then successfully downloads the 'inputdata_checksum.dat' file but fails directly after.

Code:
Trying to download file: '../inputdata_checksum.dat' to path '/capstor/scratch/cscs/jmalle/waccm_test_1/waccm_test_1/run/inputdata_checksum.dat.raw' using WGET protocol.
SUCCESS

It fails with the following message when running ./check_input_data --download --debug:

Code:
Using protocol wget with user  and passwd
RUN: wget --no-check-certificate  --spider https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/
  errput: Spider mode enabled. Check if remote file exists.
--2025-04-10 16:14:38--  https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/
Resolving svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)... 128.117.13.55
Connecting to svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)|128.117.13.55|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

I have copied my entries to config_input_data.xml below.

When I completely delete the e.g.
Code:
.../inputdata/cesm2_init/f.e21.FWHIST.f09_f09_mg17.cesm2.1-exp011.001_v2/1979-01-01/
folder, it is attempting to download things already during the ./preview_namelist call, but somehow it adds many more files than what I see on - Revision 70327: /trunk/inputdata/cesm2_init/f.e21.FWHIST.f09_f09_mg17.cesm2.1-exp011.001_v2/1979-01-01
1744294856311.png

Any hints would be greatly appreciated.

Thanks a lot,
Johanna

What version of the code are you using? CESM 2.1.3

Have you made any changes to files in the source tree?
I have played around with my config_inputdata.xml script. Currently it looks like this:

Code:
<inputdata>
  <!-- server precidence is order in this file.  Highest preference at top -->
  <!-- If the client doesn't have the protocol it will be skipped -->
  <!-- chksum verification of inputfiles is possible.  If a file with name -->
  <!-- inputdata_chksum.dat is found on the server in the directory above inputdata -->
  <!-- it will be searched for filename and chksum of each downloaded file.  -->
  <!-- see the file ftp://ftp.cgd.ucar.edu/cesm/inputdata_chksum.dat for proper format. -->

  <server>
    <protocol>svn</protocol>
    <address>https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata</address>
  </server>

 <server>
    <protocol>wget</protocol>
    <address>https://ftp.cgd.ucar.edu/cesm/inputdata/</address>
    <checksum>../inputdata_checksum.dat</checksum>
  </server>

  <server>
    <comment> wget access to subversion </comment>
    <protocol>wget</protocol>
    <address>https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/</address>
  </server>

  <server>
    <protocol>svn</protocol>
    <address>https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata</address>
  </server>

  <server>
    <protocol>wget</protocol>
    <address>https://redoak.cs.toronto.edu/twitcher/ows/proxy/thredds/fileServer/datasets/CESM/inputdata/</address>
  </server>

</inputdata>
 

jedwards

CSEG and Liaisons
Staff member
I think that the problem may be that you are using an old version of the code. Please update to the latest cesm2.1.x (at this time 2.1.5) and
try again.
 

inos@bas_ac_uk

Ingrid Cnossen
Member
Hi,

I'm having a very similar problem, but using CESM2.1.5. When I do check_input_data --download --debug it seems to successfully complete the first step:

2025-07-31 11:10:06 (633 KB/s) - ‘/work/n02/n02/inos/cesm/CESM2.1.5/runs/f.e215.FXHIST.f19_f19.ssp5-8.5t/run/inputdata_checksum.dat.raw’ saved [6854903/6854903]

SUCCESS

Although, when I check the directory above for the file 'inputdata_checksum.dat.raw' it actually only contains 'inputdata_checksum.dat.tmp'. I'm not sure if that points to the problem, but it fails on the next step with the following:

Using protocol wget with user and passwd
RUN: wget --no-check-certificate --spider - Revision 70721: /trunk/inputdata
errput: Spider mode enabled. Check if remote file exists.
--2025-07-31 11:10:06-- - Revision 70721: /trunk/inputdata
Resolving svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)... 128.117.13.55
Connecting to svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)|128.117.13.55|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: - Revision 70721: /trunk/inputdata [following]
Spider mode enabled. Check if remote file exists.
--2025-07-31 11:10:07-- - Revision 70721: /trunk/inputdata
Connecting to svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)|128.117.13.55|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

Trying to download file: 'None' to path '/work/n02/n02/inos/cesm/CESM2.1.5/runs/f.e215.FXHIST.f19_f19.ssp5-8.5t/run/inputdata_checksum.dat.raw' using WGET protocol.
Traceback (most recent call last):
File "/mnt/lustre/a2fs-work2/work/n02/n02/inos/cesm/CESM2.1.5/case/f.e215.FXHIST.f19_f19.ssp5-8.5t/./check_input_data", line 76, in <module>
_main_func(__doc__)
File "/mnt/lustre/a2fs-work2/work/n02/n02/inos/cesm/CESM2.1.5/case/f.e215.FXHIST.f19_f19.ssp5-8.5t/./check_input_data", line 66, in _main_func
sys.exit(0 if case.check_all_input_data(protocol=protocol,
File "/mnt/lustre/a2fs-work2/work/n02/n02/inos/cesm/CESM2.1.5/my_cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 164, in check_all_input_data
_download_checksum_file(self.get_value("RUNDIR"))
File "/mnt/lustre/a2fs-work2/work/n02/n02/inos/cesm/CESM2.1.5/my_cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 55, in _download_checksum_file
success = server.getfile(rel_path, new_file)
File "/mnt/lustre/a2fs-work2/work/n02/n02/inos/cesm/CESM2.1.5/my_cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/Servers/wget.py", line 49, in getfile
full_url = os.path.join(self._server_loc, rel_path)
File "/opt/cray/pe/python/3.9.13.1/lib/python3.9/posixpath.py", line 90, in join
genericpath._check_arg_types('join', a, *p)
File "/opt/cray/pe/python/3.9.13.1/lib/python3.9/genericpath.py", line 152, in _check_arg_types
raise TypeError(f'{funcname}() argument must be str, bytes, or '
TypeError: join() argument must be str, bytes, or os.PathLike object, not 'NoneType'

Any help with this would be much appreciated!

Thanks,
Ingrid
 

jedwards

CSEG and Liaisons
Staff member
What happens when you run from your case directory:
Code:
./check_input_data
?
How about
Code:
./check_input_data --download
 

inos@bas_ac_uk

Ingrid Cnossen
Member
Hi, I'm atttaching two files with the output for the above two commands. Does that give you any further clues? I'm off on holiday now, but will check back when I'm back at work (from Aug 11th). Thanks for your help!

Ingrid
 

Attachments

  • check_input_data_output1.txt
    11.2 KB · Views: 0
  • check_input_data_output2.txt
    13 KB · Views: 0
Top