Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

CESM2 out of box, fail download input files

J

jbuzan@purdue_edu

Member
Hello CESM Help,

I am attempting to run out-of-box simulation using the tutorial from 2021 YouTube.

I encounter an error while downloading the input files. CESM1 automatically connects to servers to download files, but CESM2 refuses to connect.

Cheers,
-Jonathan

> ./case.build


Building case in directory /scratch/snx3000/jbuzan/cases/CESMTWO/case01


sharedlib_only is False


model_only is False


Generating component namelists as part of build


Refcase not found in /scratch/snx3000/jbuzan/inputdata/cesm2_init/b.e20.B1850.f19_g17.release_cesm2_1_0.020/0301-01-01, will attempt to download from inputdata


Checking server ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/ with protocol gftp


Client protocol gftp not enabled


Checking server ftp://ftp.cgd.ucar.edu/cesm/inputdata with protocol wget


Using protocol wget with user anonymous and passwd user@example.edu


ERROR: Could not connect to repo '{0}'


This is most likely either a proxy, or network issue .
 
nusbaume

nusbaume

Jesse Nusbaumer
CSEG and Liaisons
Staff member
Hi Jonathan,

I am moving this thread over to the Infrastructure forum, as the forum watchers there are the ones who manage the input data server, and will most likely have the solution to your problem.

Hope that helps, and have a great day!

Jesse
 
J

jedwards

CSEG and Liaisons
Staff member
Currently there are hardware issues on the inputdata server, we hope to have them resolved in the next hour or so.
 
J

jedwards

CSEG and Liaisons
Staff member
The problem should now be resolved. Before you try again you should make sure that you don't have any 0 length files in your
local inputdata path. If you do delete them then run ./check_input_data --download from your case directory.
 
J

jbuzan@purdue_edu

Member
Thanks for all of your help. Unfortunately, I still have the same error. :(

Client protocol gftp not enabled
Using protocol wget with user anonymous and passwd user@example.edu
ERROR: Could not connect to repo '{0}'
This is most likely either a proxy, or network issue .

I executed:
> ./check_input_data --download
Loading input file list: 'Buildconf/pop.input_data_list'
Model pop missing file restore_data_filenames_derived = '/scratch/snx3000/jbuzan/inputdata/ocn/pop/gx1v6/forcing/ecosys_restore_POP_gx1v6_20170113.nc'
...
...
Model cice missing file grid_file = '/scratch/snx3000/jbuzan/inputdata/ocn/pop/gx1v7/grid/horiz_grid_20010402.ieeer8'
Model cice missing file kmt_file = '/scratch/snx3000/jbuzan/inputdata/ocn/pop/gx1v7/grid/topography_20161215.ieeei4'
Client protocol gftp not enabled
Using protocol wget with user anonymous and passwd user@example.edu
ERROR: Could not connect to repo '{0}'
This is most likely either a proxy, or network issue .
 
J

jbuzan@purdue_edu

Member
> cat config_inputdata.xml
<?xml version="1.0"?>
<inputdata>
<!-- server precidence is order in this file. Highest preference at top -->
<!-- If the client doesn't have the protocol it will be skipped -->
<!-- chksum verification of inputfiles is possible. If a file with name -->
<!-- inputdata_chksum.dat is found on the server in the directory above inputdata -->
<!-- it will be searched for filename and chksum of each downloaded file. -->
<!-- see the file ftp://ftp.cgd.ucar.edu/cesm/inputdata_chksum.dat for proper format. -->
<server>
<comment>grid ftp requires the globus-url-copy tool on the client side </comment>
<protocol>gftp</protocol>
<address>ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/</address>
<checksum>../inputdata_checksum.dat</checksum>
</server>
<server>
<protocol>wget</protocol>
<address>ftp://ftp.cgd.ucar.edu/cesm/inputdata</address>
<user>anonymous</user>
<password>user@example.edu</password>
<checksum>../inputdata_checksum.dat</checksum>
</server>
<server>
<comment> ftp requires the python package ftplib </comment>
<protocol>ftp</protocol>
<address>ftp.cgd.ucar.edu/cesm/inputdata</address>
<user>anonymous</user>
<password>user@example.edu</password>
<checksum>../inputdata_checksum.dat</checksum>
</server>
<server>
<protocol>svn</protocol>
<address>https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata</address>
</server>
</inputdata>
 
J

jedwards

CSEG and Liaisons
Staff member
That all looks fine - just a shot in the dark now - what python version are you using?
 
J

jedwards

CSEG and Liaisons
Staff member
Just tested again and it's working for me, here is the output I see from command
./check_input_data --download, do you see something similar?

./check_input_data --download
Setting resource.RLIMIT_STACK to -1 from (307200000, -1)
Loading input file list: 'Buildconf/cpl.input_data_list'
Model cpl missing file mesh_mask = '/glade/scratch/jedwards/id1/share/meshes/gx1v6_090205_ESMFmesh.nc'
Model cpl missing file rof2ocn_liq_rmapname = '/glade/scratch/jedwards/id1/cpl/gridmaps/rx1/map_rx1_to_gx1v6_nn_open_ocean_nnsm_e1000r300_marginal_sea_170503.nc'
Model cpl missing file rof2ocn_ice_rmapname = '/glade/scratch/jedwards/id1/cpl/gridmaps/rx1/map_rx1_to_gx1v6_nnsm_e1000r300_170503.nc'
Model cpl missing file mesh_atm = '/glade/scratch/jedwards/id1/share/meshes/fv1.9x2.5_141008_ESMFmesh.nc'
Model cpl missing file mesh_ice = '/glade/scratch/jedwards/id1/share/meshes/gx1v6_090205_ESMFmesh.nc'
Model cpl missing file mesh_ocn = '/glade/scratch/jedwards/id1/share/meshes/gx1v6_090205_ESMFmesh.nc'
Model cpl missing file mesh_rof = '/glade/scratch/jedwards/id1/share/meshes/rx1_nomask_181022_ESMFmesh.nc'
Loading input file list: 'Buildconf/datm.input_data_list'
Model datm missing file model_meshfile = '/glade/scratch/jedwards/id1/share/meshes/fv1.9x2.5_141008_ESMFmesh.nc'
Model datm missing file model_maskfile = '/glade/scratch/jedwards/id1/share/meshes/fv1.9x2.5_141008_ESMFmesh.nc'
Model datm missing file factorfn_data = '/glade/scratch/jedwards/id1/atm/datm7/CORE2/COREv2.correction_factors.T62.121007.nc'
Model datm missing file factorfn_mesh = '/glade/scratch/jedwards/id1/share/meshes/T62_040121_ESMFmesh_c20190714.nc'
Model datm missing file mesh = '/glade/scratch/jedwards/id1/share/meshes/T62_040121_ESMFmesh_c20190714.nc'
Model datm missing file file1 = '/glade/scratch/jedwards/id1/atm/datm7/NYF/nyf.giss.T62.051007.nc'
Model datm missing file file1 = '/glade/scratch/jedwards/id1/atm/datm7/NYF/nyf.gxgxs.T62.051007.nc'
Model datm missing file file1 = '/glade/scratch/jedwards/id1/atm/datm7/NYF/nyf.ncep.T62.050923.nc'
Loading input file list: 'Buildconf/drof.input_data_list'
Model drof missing file model_meshfile = '/glade/scratch/jedwards/id1/share/meshes/rx1_nomask_181022_ESMFmesh.nc'
Model drof missing file model_maskfile = '/glade/scratch/jedwards/id1/share/meshes/rx1_nomask_181022_ESMFmesh.nc'
Model drof missing file mesh = '/glade/scratch/jedwards/id1/lnd/dlnd7/RX1/runoff.daitren.annual_ESMFmesh_120520.nc'
Model drof missing file file1 = '/glade/scratch/jedwards/id1/lnd/dlnd7/RX1/runoff.daitren.annual.20190226.nc'
Loading input file list: 'Buildconf/dice.input_data_list'
Model dice missing file model_meshfile = '/glade/scratch/jedwards/id1/share/meshes/gx1v6_090205_ESMFmesh.nc'
Model dice missing file model_maskfile = '/glade/scratch/jedwards/id1/share/meshes/gx1v6_090205_ESMFmesh.nc'
Model dice missing file mesh = '/glade/scratch/jedwards/id1/ice/dice7/SSMI/ssmi.ifrac.0.5x0.5_ESMFmesh_120520.nc'
Model dice missing file file1 = '/glade/scratch/jedwards/id1/ice/dice7/SSMI/ssmi_ifrac.clim.x0.5.090319.nc'
Loading input file list: 'Buildconf/docn.input_data_list'
Model docn missing file model_meshfile = '/glade/scratch/jedwards/id1/share/meshes/gx1v6_090205_ESMFmesh.nc'
Model docn missing file model_maskfile = '/glade/scratch/jedwards/id1/share/meshes/gx1v6_090205_ESMFmesh.nc'
Model docn missing file mesh = '/glade/scratch/jedwards/id1/atm/cam/sst/sst_HadOIBl_bc_1x1_clim_c101029_ESMFmesh_120520.nc'
Model docn missing file file1 = '/glade/scratch/jedwards/id1/atm/cam/sst/sst_HadOIBl_bc_1x1_clim_c101029.nc'
Client protocol gftp not enabled
Using protocol wget with user anonymous and passwd user@example.edu
Trying to download file: '../inputdata_checksum.dat' to path '/glade/scratch/jedwards/SMS_Ln9_P1.f19_g16_rx1.A.cheyenne_intel.20220104_162313_xqi0a1/run/inputdata_checksum.dat.raw' using WGET protocol.
SUCCESS

Using protocol ftp with user anonymous and passwd user@example.edu
server address ftp.cgd.ucar.edu root path cesm/inputdata
Trying to download file: '../inputdata_checksum.dat' to path '/glade/scratch/jedwards/SMS_Ln9_P1.f19_g16_rx1.A.cheyenne_intel.20220104_162313_xqi0a1/run/inputdata_checksum.dat.raw' using FTP protocol.
Using protocol svn with user and passwd
Client protocol None not enabled
Checking server ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/ with protocol gftp
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Client protocol gftp not enabled
Checking server ftp://ftp.cgd.ucar.edu/cesm/inputdata/ with protocol wget
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Using protocol wget with user anonymous and passwd user@example.edu
Loading input file list: 'Buildconf/cpl.input_data_list'
Model cpl missing file mesh_mask = '/glade/scratch/jedwards/id1/share/meshes/gx1v6_090205_ESMFmesh.nc'
Trying to download file: 'share/meshes/gx1v6_090205_ESMFmesh.nc' to path '/glade/scratch/jedwards/id1/share/meshes/gx1v6_090205_ESMFmesh.nc' using WGET protocol.
SUCCESS
 
J

jbuzan@purdue_edu

Member
I get similar results up to:
Loading input file list: 'Buildconf/refcase.input_data_list'
Loading input file list: 'Buildconf/cice.input_data_list'
Model cice missing file grid_file = '/scratch/snx3000/jbuzan/inputdata/ocn/pop/gx1v7/grid/horiz_grid_20010402.ieeer8'
Model cice missing file kmt_file = '/scratch/snx3000/jbuzan/inputdata/ocn/pop/gx1v7/grid/topography_20161215.ieeei4'
Client protocol gftp not enabled
Using protocol wget with user anonymous and passwd user@example.edu
ERROR: Could not connect to repo '{0}'
This is most likely either a proxy, or network issue .

Something about the step when protocol wget with user anonymous etc. then the system fails.
 
G

gunnar_jansen@climate_unibe_ch

New Member
I was able to reproduce the error and I have dug a little deeper into this. This could potentially have been a bug in version 2.1.1. With version 2.1.3 I was able to successfully download some files (at least using svn) automatically.

However, while investigating this further on version 2.1.3, I noticed that the wget protocol using ./check_input_data --download currently doesn't seem to work properly. During the file checks in check_input_data, the presence of the requested file is tested using wget --spider option. Testing this command manually on the command line for any file on ftp://ftp.cgd.ucar.edu/cesm/inputdata/ this returns code 8. This stands for "Server issued an error response" according to the wget man page. This triggers the check_input_data to abort with the message that the repository is not available.

Looking at the debug output of wget --no-check-certificate --user anonymous --password user@example.edu --spider ftp://ftp.cgd.ucar.edu/cesm/inputdata/../inputdata_checksum.dat -d as an example doesn't show any obvious reason for the error response. I have attached the output in wget.txt.
Removing the --spider option from the wget command and downloading the file manually works correctly.

@jedwards, Could this be connected to the issues on the inputdata server you reported on yesterday?

Manually switching to the ftp protocol for check_input_data by deleting the wget option from config_inputdata.xml (since the --protocol command line switch doesn't seem to work for me), downloads the requested input data.
 

Attachments

  • wget.txt
    3.4 KB · Views: 2
J

jbuzan@purdue_edu

Member
Hello Everyone,

Both Gunnar Jansen at the University of Bern and Theofilos from CSCS help desk have come up with 2 different solutions.

1) Gunnar Jansen's solution
Remove the lines from the config_inputdata.xml

<server>
<protocol>wget</protocol>
<address>ftp://ftp.cgd.ucar.edu/cesm/inputdata</address>
<user>anonymous</user>
<password>user@example.edu</password>
<checksum>../inputdata_checksum.dat</checksum>
</server>

2) Theofilos' solution
change:

<server>
<protocol>wget</protocol>
<address>ftp://ftp.cgd.ucar.edu/cesm/inputdata</address>
<user>anonymous</user>
<password>user@example.edu</password>
<checksum>../inputdata_checksum.dat</checksum>
</server>

to:
<server>
<protocol>wget</protocol>
<address>ftp://ftp.cgd.ucar.edu/cesm/inputdata/</address>
<user>anonymous</user>
<password>user@example.edu</password>
<checksum>../inputdata_checksum.dat</checksum>
</server>

It seems to be that this is just a missing '/' in the wget protocol.
Both methods work.



Cheers,
-Jonathan
 
You must log in or register to reply here.
Top