Issues with downloading input data on CESM2.1.3

[zzheng]

Hi,

I am testing CESM2.1.3 on my local Linux (Ubuntu). I have built the case successfully but there is an error (AttributeError: 'NoneType' object has no attribute 'getfile') when I run "./case.submit".

I did follow the suggestions by changing ftp://ftp.cgd.ucar.edu/cesm/inputdata to ftp://ftp.cgd.ucar.edu/cesm/inputdata/ and moving "svn" to the top. It still doesn't work.

# Move to the script directory
cd $HOME/my_cesm_sandbox/cime/scripts

# Step1: Create a case
./create_newcase --case $HOME/projects/cesm/scratch/testrun --compset QPC4 --res f45_f45_mg37 --run-unsupported

# Move to the build and run directory
cd $HOME/projects/cesm/scratch/testrun

# Generate a 3-day simulation for test purposes only
./xmlchange STOP_OPTION=ndays,STOP_N=3

# Step2: Setting up the case
./case.setup

# Step3: Building the model
./case.build

# Step4: Running the model
./case.submit

Below is the error:

Time spent not building: 0.553834 sec
Time spent building: 92.886199 sec
MODEL BUILD HAS FINISHED SUCCESSFULLY

~/projects/cesm/scratch/testrun$ ./case.submit

Creating component namelists
   Calling /home/zhonghua/my_cesm_sandbox/components/cam//cime_config/buildnml
CAM namelist copy: file1 /home/zhonghua/projects/cesm/scratch/testrun/Buildconf/camconf/atm_in file2 /home/zhonghua/projects/cesm/scratch/testrun/run/atm_in
   Calling /home/zhonghua/my_cesm_sandbox/cime/src/components/stub_comps/slnd/cime_config/buildnml
   Calling /home/zhonghua/my_cesm_sandbox/cime/src/components/stub_comps/sice/cime_config/buildnml
   Calling /home/zhonghua/my_cesm_sandbox/cime/src/components/data_comps/docn/cime_config/buildnml
File not found: domainfile = "/home/zhonghua/projects/cesm/inputdata/share/domains/domain.ocn.4x5_gx3v7_100120.nc", will attempt to download in check_input_data phase
   Calling /home/zhonghua/my_cesm_sandbox/cime/src/components/stub_comps/srof/cime_config/buildnml
   Calling /home/zhonghua/my_cesm_sandbox/cime/src/components/stub_comps/sglc/cime_config/buildnml
   Calling /home/zhonghua/my_cesm_sandbox/cime/src/components/stub_comps/swav/cime_config/buildnml
   Calling /home/zhonghua/my_cesm_sandbox/cime/src/components/stub_comps/sesp/cime_config/buildnml
   Calling /home/zhonghua/my_cesm_sandbox/cime/src/drivers/mct/cime_config/buildnml
Finished creating component namelists
Checking that inputdata is available as part of case submission
Loading input file list: 'Buildconf/cam.input_data_list'
  Model cam missing file prescribed_ozone_file = '/home/zhonghua/projects/cesm/inputdata/atm/cam/ozone/apeozone_cam3_5_54.nc'
  Model cam missing file tropopause_climo_file = '/home/zhonghua/projects/cesm/inputdata/atm/cam/chem/trop_mozart/ub/clim_p_trop.nc'
  Model cam missing file ncdata = '/home/zhonghua/projects/cesm/inputdata/atm/cam/inic/fv/cami_0001-01-01_4x5_L26_c060608.nc'
  Model cam missing file absems_data = '/home/zhonghua/projects/cesm/inputdata/atm/cam/rad/abs_ems_factors_fastvx.c030508.nc'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/docn.input_data_list'
  Model docn missing file domainfile = '/home/zhonghua/projects/cesm/inputdata/share/domains/domain.ocn.4x5_gx3v7_100120.nc'
Using protocol svn with user  and passwd
Trying to download file: 'None' to path '/home/zhonghua/projects/cesm/scratch/testrun/run/inputdata_checksum.dat.raw' using SVN protocol.
Could not automatically download file /home/zhonghua/projects/cesm/scratch/testrun/run/inputdata_checksum.dat
Client protocol gftp not enabled
Using protocol wget with user anonymous and passwd user@example.edu
Could not connect to repo 'ftp://ftp.cgd.ucar.edu/cesm/inputdata/'
This is most likely either a proxy, or network issue .
Trying to download file: '../inputdata_checksum.dat' to path '/home/zhonghua/projects/cesm/scratch/testrun/run/inputdata_checksum.dat.raw' using NoneType protocol.
Traceback (most recent call last):
  File "/home/zhonghua/projects/cesm/scratch/testrun/./case.submit", line 107, in <module>
    _main_func(__doc__)
  File "/home/zhonghua/projects/cesm/scratch/testrun/./case.submit", line 102, in _main_func
    case.submit(job=job, no_batch=no_batch, prereq=prereq, allow_fail=allow_fail,
  File "/home/zhonghua/my_cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 156, in submit
    run_and_log_case_status(functor, "case.submit", caseroot=caseroot,
  File "/home/zhonghua/my_cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 1683, in run_and_log_case_status
    rv = func()
  File "/home/zhonghua/my_cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 151, in <lambda>
    functor = lambda: _submit(self, job=job, no_batch=no_batch, prereq=prereq,
  File "/home/zhonghua/my_cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 85, in _submit
    case.check_case()
  File "/home/zhonghua/my_cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 171, in check_case
    self.check_all_input_data()
  File "/home/zhonghua/my_cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 163, in check_all_input_data
    _download_checksum_file(self.get_value("RUNDIR"))
  File "/home/zhonghua/my_cesm_sandbox/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 54, in _download_checksum_file
    success = server.getfile(rel_path, new_file)
AttributeError: 'NoneType' object has no attribute 'getfile'

Below is my config_inputdata.xml file:

<?xml version="1.0"?>

<inputdata>
  <!-- server precidence is order in this file.  Highest preference at top -->
  <!-- If the client doesn't have the protocol it will be skipped -->
  <!-- chksum verification of inputfiles is possible.  If a file with name -->
  <!-- inputdata_chksum.dat is found on the server in the directory above inputdata -->
  <!-- it will be searched for filename and chksum of each downloaded file.  -->
  <!-- see the file ftp://ftp.cgd.ucar.edu/cesm/inputdata_chksum.dat for proper format. -->
 
  <server>
    <protocol>svn</protocol>
    <address>https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/</address>
  </server>

  <server>
    <comment>grid ftp requires the globus-url-copy tool on the client side </comment>
    <protocol>gftp</protocol>
    <address>ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/</address>
    <checksum>../inputdata_checksum.dat</checksum>
  </server>

  <server>
    <protocol>wget</protocol>
    <address>ftp://ftp.cgd.ucar.edu/cesm/inputdata/</address>
    <user>anonymous</user>
    <password>user@example.edu</password>
    <checksum>../inputdata_checksum.dat</checksum>
  </server>

  <server>
    <comment> ftp requires the python package ftplib </comment>
    <protocol>ftp</protocol>
    <address>ftp.cgd.ucar.edu/cesm/inputdata/</address>
    <user>anonymous</user>
    <password>user@example.edu</password>
    <checksum>../inputdata_checksum.dat</checksum>
  </server>

</inputdata>

Looking forward to hearing from you. Thanks!

 

[jedwards]

Try running ./check_input_data --download --debug from the case directory and send the output along with the file
check_input_data.log.

 

[zzheng]

Try running ./check_input_data --download --debug from the case directory and send the output along with the file
check_input_data.log.

Thanks. Attached is the file.

 

[zzheng]

Try running ./check_input_data --download --debug from the case directory and send the output along with the file
check_input_data.log.

Hi Jim, any suggestions regarding how to resolve this issue? I've uploaded the "check_input_data.log."

Thanks!

 

[jedwards]

You need to figure out why this doesn't work - is there a system administrator you can ask?
07-11 15:49 CIME.utils INFO RUN: wget --no-check-certificate --user anonymous --password user@example.edu --spider ftp://ftp.cgd.ucar.edu/cesm/inputdata/
07-11 15:50 CIME.Servers.wget WARNING Could not connect to repo 'ftp://ftp.cgd.ucar.edu/cesm/inputdata/'

I've confirmed that the server is up and functional on our end, the issue appears to be on your end.

 

[zzheng]

You need to figure out why this doesn't work - is there a system administrator you can ask?
07-11 15:49 CIME.utils INFO RUN: wget --no-check-certificate --user anonymous --password user@example.edu --spider ftp://ftp.cgd.ucar.edu/cesm/inputdata/
07-11 15:50 CIME.Servers.wget WARNING Could not connect to repo 'ftp://ftp.cgd.ucar.edu/cesm/inputdata/'

I've confirmed that the server is up and functional on our end, the issue appears to be on your end.

Hi Jim, I run the command using bash, seems it works. But I don't know why it doesn't work when I use "case.submit" and "./check_input_data --download --debug"

~/projects/cesm/scratch/testrun$ wget --no-check-certificate --user anonymous --password user@example.edu --spider ftp://ftp.cgd.ucar.edu/cesm/inputdata/
--2023-07-12 17:40:30--  ftp://ftp.cgd.ucar.edu/cesm/inputdata/
           => ‘.listing’
Resolving ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)... 128.117.13.41
Connecting to ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)|128.117.13.41|:21... connected.

 

[zzheng]

You need to figure out why this doesn't work - is there a system administrator you can ask?
07-11 15:49 CIME.utils INFO RUN: wget --no-check-certificate --user anonymous --password user@example.edu --spider ftp://ftp.cgd.ucar.edu/cesm/inputdata/
07-11 15:50 CIME.Servers.wget WARNING Could not connect to repo 'ftp://ftp.cgd.ucar.edu/cesm/inputdata/'

I've confirmed that the server is up and functional on our end, the issue appears to be on your end.

Hi Jim, it will be great to know your suggestions on the next steps for debugging. Thanks!

 

[jedwards]

I don't know - is it something to do with your python version? Did you run from bash on the same system that you are running case.submit on?

 

[zzheng]

I don't know - is it something to do with your python version? Did you run from bash on the same system that you are running case.submit on?

Good point, Jim. I used the same environment to run "case.submit" and wget. But I wonder if I should make any modifications to Shebang for any scripts?
What version of Python is recommended? Thanks!

 

[Johnny]

Good point, Jim. I used the same environment to run "case.submit" and wget. But I wonder if I should make any modifications to Shebang for any scripts?
What version of Python is recommended? Thanks!

Porting to a local machine, can't automatically download the input data.

I am trying to run CLM5.0 on my local machine. I was able to successfully build a case and run the model in the past (using the same machine), but I am getting an error this time (with case.submit while trying to download the input data), which seems to be related to connection to the repository...

bb.cgd.ucar.edu

"
The solution was to edit the config_inputdata.xml file in cime/config/cesm, adding a slash at the end of the ftp address, changing ftp://ftp.cgd.ucar.edu/cesm/inputdata to ftp://ftp.cgd.ucar.edu/cesm/inputdata/

<server>
<protocol>wget</protocol>
<address>ftp://ftp.cgd.ucar.edu/cesm/inputdata/</address>
<user>anonymous</user>
<password>user@example.edu</password>
<checksum>../inputdata_checksum.dat</checksum>
</server>

"
I just shrugged 3 hours to find the relevant post and answer. We should patch 2.1.3 release

 

[jedwards]

I believe that you will find the patch in cesm2.1.4 - cesm2.1.5 should be out this coming week. cesm2.1.3 should no longer be used.