CESM 2.1.1 Build Error - Lack of rpointer Files

[rmokos@ncsa_illinois_edu]

Hi,

I'm running into errors when trying to build CESM 2.1.1 on Blue Waters. From a fresh login shell, I run this script:

#!/bin/bash

# load Python 2.7.14 and svn 1.9.7
module load bwpy/0.3.2 Subversion

# set env variables
export CASENAME=___FHIST_09
export CASEDIR=$HOME/___FHIST_09
export CESMROOT=$HOME/___cesm2_1_1
export CESMDATAROOT=/scratch/sciteam/$USER/___cesm_data

# make cesm data directory on scratch
mkdir -p $CESMDATAROOT

# download cesm 2.1.1
cd $HOME
git clone -b release-cesm2.1.1 https://github.com/ESCOMP/cesm.git ___cesm2_1_1

# download externals
cd ___cesm2_1_1
./manage_externals/checkout_externals

# create new case
cd $HOME
$CESMROOT/cime/scripts/create_newcase --case $CASEDIR --compset FHIST --res f09_f09_mg17 --project fyy --input-dir $CESMDATAROOT

# set up case
cd $CASENAME
./case.setup

# add to user_nl_cam:
cat > user_nl_cam <<EOF
! Users should add all user specific namelist changes below in the form of 
! namelist_var = new_namelist_value 
  
! assume "h1" file with instantaneous value every 3 hour
nhtfrq=0,-3
! assume "h1" files with 8 timesteps each
mfilt=1,8
fincl2='T:I','RELHUM:I','SZA:I','SOAG_SRF:I','DMS_SRF:I','H2SO4_SRF:I','O3:I','H2O2_SRF:I','SO2_SRF:I','bc_a1_SRF:I','bc_a4_SRF:I','dst_a1_SRF:I','dst_a2_SRF:I','dst_a3_SRF:I','ncl_a1_SRF:I','ncl_a2_SRF:I','ncl_a3_SRF:I','pom_a1_SRF:I','pom_a4_SRF:I','so4_a1_SRF:I','so4_a2_SRF:I','so4_a3_SRF:I','soa_a1_SRF:I','soa_a2_SRF:I'
EOF

# change run lenth (assume one-month)
./xmlchange STOP_N=1,STOP_OPTION=nmonths

# build the case
./case.build

# submit the case
./case.submit

Things seem to go well until here:

Building case in directory /mnt/a/u/staff/rmokos/___FHIST_09
sharedlib_only is False
model_only is False
Generating component namelists as part of build
Refcase not found in /scratch/sciteam/rmokos/___cesm_data/cesm2_init/f.e20.FHIST.f09_f09.cesm2_1.001_v2/1979-01-01, will attempt to download from inputdata
Checking server ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/ with protocol gftp
Using protocol gftp with user  and passwd
Loading input file list: 'Buildconf/refcase.input_data_list'
  Model refcase missing file refdir = '/scratch/sciteam/rmokos/___cesm_data/cesm2_init/f.e20.FHIST.f09_f09.cesm2_1.001_v2/1979-01-01/'
Trying to download file: 'cesm2_init/f.e20.FHIST.f09_f09.cesm2_1.001_v2/1979-01-01' to path '/scratch/sciteam/rmokos/___cesm_data/cesm2_init/f.e20.FHIST.f09_f09.cesm2_1.001_v2/1979-01-01' using GridFTP protocol.
Creating directory /scratch/sciteam/rmokos/___cesm_data/cesm2_init/f.e20.FHIST.f09_f09.cesm2_1.001_v2/1979-01-01
FAIL: GridFTP repo 'ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/' does not have directory 'cesm2_init/f.e20.FHIST.f09_f09.cesm2_1.001_v2/1979-01-01' error=error: Unable to list url ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/cesm2_init/f.e20.FHIST.f09_f09.cesm2_1.001_v2/1979-01-01/:
globus_ftp_client: the server responded with an error
550 550-GlobusError: v=1 c=PATH_NOT_FOUND
550-GridFTP-Errno: 2
550-GridFTP-Reason: System error in stat
550-GridFTP-Error-String: No such file or directory
550 End.

Checking server ftp://ftp.cgd.ucar.edu/cesm/inputdata with protocol wget
Using protocol wget with user anonymous and passwd user@example.edu
Loading input file list: 'Buildconf/refcase.input_data_list'
 - Prestaging REFCASE (/scratch/sciteam/rmokos/___cesm_data/cesm2_init/f.e20.FHIST.f09_f09.cesm2_1.001_v2/1979-01-01) to /scratch/sciteam/rmokos/___FHIST_09/run
ERROR: Reference case directory /scratch/sciteam/rmokos/___cesm_data/cesm2_init/f.e20.FHIST.f09_f09.cesm2_1.001_v2/1979-01-01 does not contain any rpointer files
 - Prestaging REFCASE (/scratch/sciteam/rmokos/___cesm_data/cesm2_init/f.e20.FHIST.f09_f09.cesm2_1.001_v2/1979-01-01) to /scratch/sciteam/rmokos/___FHIST_09/run
ERROR: Reference case directory /scratch/sciteam/rmokos/___cesm_data/cesm2_init/f.e20.FHIST.f09_f09.cesm2_1.001_v2/1979-01-01 does not contain any rpointer files

I have no issues if I run the same script with CESM 2.1.0 (i.e., git clone -b release-cesm2.1.0 https://github.com/ESCOMP/cesm.git ___cesm2_1_0).

Can someone please help me understand what the issue is?

Thanks in advance.

 

[emir toker]

****
edit : I think the problem was about the server of ucar (ftp://ftp.cgd.ucar.edu/cesm/) I mailed them and they respond to me.
"This is fixed. The partition did not mount properly after a reboot."
you can go and find your files even manually; ftp://ftp.cgd.ucar.edu/cesm/inputdata/cesm2_init/
****

hi,

I had a similar problem and I think the problem is about connected to the server ( with ftp, svn or wget) and it is related to "config_inputdata.xml".

config_inputdata.xml is different in release-cesm2.1.0 and release-cesm2.1.1

release-cesm2.1.1 includes extra lines consist of username and password for file transfer protocols, so it maybe cannot reach the server to download input data (.nc) and some files (like rpointer files) because of this.

My solution: Change your config_inputdata.xml ($CESM/cime/config/cesm/.), using config_inputdata.xml which belonged to release-cesm2.1.0, before download input data.

I give you these two different config_inputdata.xml files.

config_inputdata in release-cesm2.1.0

<?xml version="1.0"?>

<inputdata>
  <!-- server precidence is order in this file.  Highest preference at top -->
  <!-- If the client doesn't have the protocol it will be skipped -->
  <server>
    <comment>grid ftp requires the globus-url-copy tool on the client side </comment>
    <protocol>gftp</protocol>
    <address>ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/</address>
  </server>

  <server>
    <protocol>svn</protocol>
    <address>https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata</address>
  </server>

  <server>
    <comment> ftp requires the python package ftplib </comment>
    <protocol>ftp</protocol>
    <address>ftp.cgd.ucar.edu/cesm/inputdata</address>
  </server>

  <server>
    <protocol>wget</protocol>
    <address>ftp://ftp.cgd.ucar.edu/cesm/inputdata</address>
  </server>

</inputdata>

and

config_inputdata in release-cesm2.1.1

<?xml version="1.0"?>

<inputdata>
  <!-- server precidence is order in this file.  Highest preference at top -->
  <!-- If the client doesn't have the protocol it will be skipped -->
  <!-- chksum verification of inputfiles is possible.  If a file with name -->
  <!-- inputdata_chksum.dat is found on the server in the directory above inputdata -->
  <!-- it will be searched for filename and chksum of each downloaded file.  -->
  <!-- see the file ftp://ftp.cgd.ucar.edu/cesm/inputdata_chksum.dat for proper format. -->
  <server>
    <comment>grid ftp requires the globus-url-copy tool on the client side </comment>
    <protocol>gftp</protocol>
    <address>ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/</address>
    <checksum>../inputdata_checksum.dat</checksum>
  </server>

  <server>
    <protocol>wget</protocol>
    <address>ftp://ftp.cgd.ucar.edu/cesm/inputdata</address>
    <user>anonymous</user>
    <password>user@example.edu</password>
    <checksum>../inputdata_checksum.dat</checksum>
  </server>

  <server>
    <comment> ftp requires the python package ftplib </comment>
    <protocol>ftp</protocol>
    <address>ftp.cgd.ucar.edu/cesm/inputdata</address>
    <user>anonymous</user>
    <password>user@example.edu</password>
    <checksum>../inputdata_checksum.dat</checksum>
  </server>

  <server>
    <protocol>svn</protocol>
    <address>https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata</address>
  </server>

</inputdata>

 

[rmokos@ncsa_illinois_edu]

Thanks for the info.

UCAR's server issues would explain why there was a time when I was having trouble building 2.1.0. But unfortunately, fixing their server hasn't fixed my 2.1.1 build.

I had tried using the 2.1.0 config_inputdata file, and I tried it again after your suggestion, but I still had issues.

I downloaded the rpointer files manually via ftp (thanks for the idea!), and that got me through the build phase. But when I ran the submit script, I got this python error:

Loading input file list: 'Buildconf/cpl.input_data_list'
  Model cpl missing file glc2lnd_smapname = '/scratch/sciteam/rmokos/___cesm_data/cpl/gridmaps/gland4km/map_gland4km_TO_fv0.9x1.25_aave.170429.nc'
  Model cpl missing file lnd2rof_fmapname = '/scratch/sciteam/rmokos/___cesm_data/lnd/clm2/mappingdata/maps/0.9x1.25/map_0.9x1.25_nomask_to_0.5x0.5_nomask_aave_da_c120522.nc'
  Model cpl missing file glc2lnd_fmapname = '/scratch/sciteam/rmokos/___cesm_data/cpl/gridmaps/gland4km/map_gland4km_TO_fv0.9x1.25_aave.170429.nc'
  Model cpl missing file lnd2glc_fmapname = '/scratch/sciteam/rmokos/___cesm_data/cpl/gridmaps/fv0.9x1.25/map_fv0.9x1.25_TO_gland4km_aave.170429.nc'
  Model cpl missing file rof2lnd_fmapname = '/scratch/sciteam/rmokos/___cesm_data/lnd/clm2/mappingdata/maps/0.9x1.25/map_0.5x0.5_nomask_to_0.9x1.25_nomask_aave_da_c121019.nc'
  Model cpl missing file lnd2glc_smapname = '/scratch/sciteam/rmokos/___cesm_data/cpl/gridmaps/fv0.9x1.25/map_fv0.9x1.25_TO_gland4km_blin.170429.nc'
Using protocol gftp with user  and passwd
Trying to download file: 'None' to path '/scratch/sciteam/rmokos/___FHIST_09/run/inputdata_checksum.dat.raw' using GridFTP protocol.
Traceback (most recent call last):
  File "./case.submit", line 107, in <module>
    _main_func(__doc__)
  File "./case.submit", line 104, in _main_func
    mail_user=mail_user, mail_type=mail_type, batch_args=batch_args)
  File "/mnt/a/u/staff/rmokos/___cesm2_1_1/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 156, in submit
    custom_success_msg_functor=verbatim_success_msg)
  File "/mnt/a/u/staff/rmokos/___cesm2_1_1/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 1667, in run_and_log_case_status
    rv = func()
  File "/mnt/a/u/staff/rmokos/___cesm2_1_1/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 154, in <lambda>
    batch_args=batch_args)
  File "/mnt/a/u/staff/rmokos/___cesm2_1_1/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 84, in _submit
    case.check_case()
  File "/mnt/a/u/staff/rmokos/___cesm2_1_1/cime/scripts/Tools/../../scripts/lib/CIME/case/case_submit.py", line 170, in check_case
    self.check_all_input_data()
  File "/mnt/a/u/staff/rmokos/___cesm2_1_1/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 160, in check_all_input_data
    _download_checksum_file(self.get_value("RUNDIR"))
  File "/mnt/a/u/staff/rmokos/___cesm2_1_1/cime/scripts/Tools/../../scripts/lib/CIME/case/check_input_data.py", line 54, in _download_checksum_file
    success = server.getfile(rel_path, new_file)
  File "/mnt/a/u/staff/rmokos/___cesm2_1_1/cime/scripts/Tools/../../scripts/lib/CIME/Servers/gftp.py", line 23, in getfile
    stat, _,err = run_cmd("globus-url-copy -v {} file://{}".format(os.path.join(self._root_address, rel_path), full_path))
  File "/mnt/bwpy/single/usr/lib/python-exec/python2.7/../../../lib/python2.7/posixpath.py", line 68, in join
    if b.startswith('/'):
AttributeError: 'NoneType' object has no attribute 'startswith'

Any ideas about this?

 

[emir toker]

I had got a very similar error in the same step. First I thought this is a python error, but now I think it was not. I think it is related to missing data or files which CESM needs to build the case. I tried the same ways that you did. But maybe downloading files manually is not working for you.

My idea is that the problem is the protocol. I tried this one at the last option. Try this one.

Just change wget protocol with svn. I added to down. Or try some other combination about protocol.

<?xml version="1.0"?>

<inputdata>
  <!-- server precidence is order in this file.  Highest preference at top -->
  <!-- If the client doesn't have the protocol it will be skipped -->
  <!-- chksum verification of inputfiles is possible.  If a file with name -->
  <!-- inputdata_chksum.dat is found on the server in the directory above inputdata -->
  <!-- it will be searched for filename and chksum of each downloaded file.  -->
  <!-- see the file ftp://ftp.cgd.ucar.edu/cesm/inputdata_chksum.dat for proper format. -->
  <server>
    <comment>grid ftp requires the globus-url-copy tool on the client side </comment>
    <protocol>gftp</protocol>
    <address>ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/</address>
    <checksum>../inputdata_checksum.dat</checksum>
  </server>

  <server>
    <protocol>svn</protocol>
    <address>ftp://ftp.cgd.ucar.edu/cesm/inputdata</address>
    <user>anonymous</user>
    <password>user@example.edu</password>
    <checksum>../inputdata_checksum.dat</checksum>
  </server>

  <server>
    <comment> ftp requires the python package ftplib </comment>
    <protocol>ftp</protocol>
    <address>ftp.cgd.ucar.edu/cesm/inputdata</address>
    <user>anonymous</user>
    <password>user@example.edu</password>
    <checksum>../inputdata_checksum.dat</checksum>
  </server>

  <server>
    <protocol>svn</protocol>
    <address>https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata</address>
  </server>

</inputdata>

 

[QINKONG]

I used to have the same problem, and solve it by prioritizing wget (by placing it ahead). I guess this should be a server problem and you can try to adjust the order of servers.