File missing

[matthias_demuzere@ees_kuleuven_be]

Hi all, I currently trying out the CVPD package, but unfortunately, I do not get far because of a "File missing error".My namelist is as follows: CESM1-BGC | /media/Matthias_2/data/cmip5/ESM/mon/CESM1-BGC/ | 1850 | 2005, with the path containing all CESM1-BGC files from the esmHistorical run.I am testing CVPD with the psl.trends, since I suppose this module only needs psl data. This file is available under the path above as: psl_Amon_CESM1-BGC_esmHistorical_r1i1p1_185001-200512.nc. If I am not mistaken, the file name follows the conventions?Yet, when running the script, I get the following:13:07:08-matthiasd@rcs-Precision-T5600:cvdp.v4.1.0.12222016$ ncl driver.ncl  Copyright (C) 1995-2015 - All Rights Reserved University Corporation for Atmospheric Research NCAR Command Language Version 6.3.0 The use of this software is governed by a License Agreement. See http://www.ncl.ucar.edu/ for more details.(0)Starting: Climate Variability Diagnostics Package (Die Feb  7 13:07:20 CET 2017) Copyright (C) 1995-2015 - All Rights Reserved University Corporation for Atmospheric Research NCAR Command Language Version 6.3.0 The use of this software is governed by a License Agreement. See http://www.ncl.ucar.edu/ for more details.Starting: namelist.nclFinished: namelist.nclStarting: psl.trends.nclFile missing, creating blank array of data. View PSL namelist for details.Finished: psl.trends.nclStarting: ncfiles.append.nclFinished: ncfiles.append.nclls: cannot access '/home/matthiasd/plots/CVDP/*.png': No such file or directoryconvert: unable to open image `missing': No such file or directory @ error/blob.c/OpenBlob/2712.convert: no decode delegate for this image format `' @ error/constitute.c/ReadImage/501.convert: no images defined `missing' @ error/convert.c/ConvertImageCommand/3210.Starting: webpage.nclFinished: webpage.ncl(0)Finished: Climate Variability Diagnostics Package (Die Feb  7 13:07:22 CET 2017) The output of namelist_psl is as follows:CESM1-BGC | missing | 1850 | 2005 Any idea of what is wrong?Thanks

 

[asphilli]

Hi Matthias,Thanks for your interest in running the CVDP. Unfortunately files cannot be read into the package whose names start with a variable followed by a period. This is noted on the Known Issues page as well as the bottom of the first page within the README file.  I suggest that you create an alternate directory and use soft links to rename the files.This file naming oddity is something that I would like to attend to in a future CVDP release, but unfortunately there is not an easy solution. If you have any other issues running the CVDP let me know.Adam  

 

[matthias_demuzere@ees_kuleuven_be]

Hi Adam, Thanks for your swift reply. Apparently I missed the naming conventions. Sorry about that.So I changed the settings according to your suggestions: - new folder: /media/Matthias_2/data/cmip5/ESM/mon/CESM1-BGC/cvdp/- file name changed. First I tried with a symbolic link, then I just copied the file. I tried various file names (eg. modelA.PSL.185001-200512.nc, 185001-200512.nc, test_185001-200512.nc, etc ...) but none of these combinations work. When I check the namelist_byvar/namelist_psl file, it contiues to state "missing". Would you have any other suggestions on how to fix this?Thanks,Matthias

 

[asphilli]

Hi Matthias,If you have the PSL data sitting here:/media/Matthias_2/data/cmip5/ESM/mon/CESM1-BGC/cvdp[/b]/modelA.PSL.185001-200512.ncthen within the namelist you should refer to this data like this:CESM1-BGC #1 | /media/Matthias_2/data/cmip5/ESM/mon/CESM1-BGC/cvdp[/b]/ | 1850 | 2005(The first setting of "CESM1-BGC #1" is arbitrary, the third ("1850") and fourth ("2005") just have to fall within the date range of the input data files.) Note that you do not specify a file directly here, you specify a directory where the file is present. You can use wildcards to narrow down the list of files within a directory as is shown in the first row of namelist example 2 within the README file, but you do not have to.Typing "ls  /media/Matthias_2/data/cmip5/ESM/mon/CESM1-BGC/cvdp[/b]/" should show that your PSL file is present.There are numerous examples of namelists available in the example_namelists directory found in the CVDP release code. If you are still having trouble please respond with the following attached: Your namelist and a screenshot of the results of typing:"ls  /media/Matthias_2/data/cmip5/ESM/mon/CESM1-BGC/cvdp[/b]/"Adam

 

[matthias_demuzere@ees_kuleuven_be]

mmm... unfortunately this is what I have? So I am not sure what is wrong.Please find in attach a screenshot listing the namelist file, ls statement on folder + info of file and ouput of ncl driver.nl.Also in attach is the actual namelist file (I added .txt extension, otherwise the file does not want to upload).To be sure I also added the adjusted driver.ncl file (also with a changed extension).I hope this helps?M.

 

[asphilli]

Your issue has me quite puzzled. I set up a temporary directory here with a single file in it named modelA.PSL.000101-100012.nc. The CVDP sees the file and the namelist_psl file is correctly set. It is quite odd that whatever you change the filename to, the results are the same with a missing return. My (only) suggestion: Within the ncl_scripts/namelist.ncl script add the following statement between lines 193 and 194:print("ls "+paths(gg)+"*{"+tstring0+"}*.nc")Please post back the results of that statement, along with what happens when you paste the results into your terminal window. Finally, I would also change the driver.ncl option to namelists_only = "True" as there's no reason to run the package fully without the variable namelists being correctly set up. Adam 

 

[matthias_demuzere@ees_kuleuven_be]

I now added the statement in the namelist.ncl script. The output of that command is correct (note I am testing this on another machine now, hence different dir):ls /home/maffie/Documents/cmip5/*{PSL_,PSL.,psl_,psl.,slp.,slp_}*.nc 2> /dev/nullIf I use this in my terminal, the file is listed properly.Now, if I do print(fsst) in the namelist.ncl script, I get the following:  Variable: fsstType: stringTotal Size: 8 bytes            1 valuesNumber of Dimensions: 1Dimensions and sizes:[1]Coordinates: Number Of Attributes: 1  _FillValue :missingmissing When repeating the same directly in ncl, I get the same error. If I remove the curly brackets, having tstring0 reduced to one entry (PSL_), it does work, both in ncl directly as well as in the script.This results in the following entry in namelist_psl: CESM1-BGC #1 | /home/maffie/Documents/cmip5/modelA.PSL_185001-200512.nc | 1850 | 2005 And now I can continue running the script. So if I keep my naming convections fixed and adjust the script on the tstring definitions, at least this should work. Would be nice, since the software seems very powerful. Yet, would be nice to find a solution of course.I thought that maybe it was an issue with the ncl version, but I have the same behavior for 6.2.1 and 6.3.0.I also had a look an ncl's systemfunc page, but I can't see any hints there ...So, not sure what is causing this?M.  

 

[matthias_demuzere@ees_kuleuven_be]

Hi Adam,When bumping into the systemfunc behavior, I also contact the ncl developers via ncl-talk. And they found the reason why your routines behave as they do under my ubuntu system. Here is the message that Rick Brownrigg has sent me: The so-called "brace-expansion" appears to be a bash thing. I speculate your shell in your terminal window is indeed bash.  However, by default, NCL under the covers invokes "/bin/sh" during the systemfunc() call. On many Linux systems, /bin/sh is simply a symlink to bash (that's the case on my system and why things work for me).  However, apparently on Ubuntu systems, /bin/sh is a symlink to a shell called "dash" (I'd never heard of it), and dash is the one that does not understand brace-expansion -- you can try it yourself by invoking "dash" on the command-line and then trying a command that involves brace-expansion.  More on this here:    http://askubuntu.com/questions/87853/what-is-default-shell-for-terminal

So a solution in your case would be something like this:

    files = systemfunc("bash -c 'ls /DIRECTORY/*{.nc,.hdf,.pdf}'") In particular note the "-c" switch to bash, and that the command you want bash to execute is surrounded by single-quotes.  As such, I adjusted all the systemfunc commands that use curly brackets to include bash -c surrounded by double quotes. This seems to do the trick. Please find my updated namelist.ncl (renamed to .txt) in attach. Perhaps this information could be included in the "known issues"? Or an if statement could be added when invoking namelist.ncl, to use the "ubuntu" version of it? Something along the lines: if [[ $(cat /etc/*-release | grep DISTRIB_ID) == *"Ubuntu"* ]];then   namelist=namelist.ubuntu.nclelse   namelist=namelist.nclfi 

 

[asphilli]

Hi Matthias,Thank you for the update. I was wondering if there was something different about your shell environment as I could not think of any other explanation of the behavior that you were seeing. I will definitely make note of this on the CVDP Known Issues page and test Rick's/your modifications to namelist.ncl so that I can include them in the next CVDP release. Thanks again,Adam