[ncl-talk] Skipping Missing Files [SEC=UNCLASSIFIED]

[Barry Lynn]

Hi Griffith:

Thank you.  I agree that knowledge is power.  I have been using a curl
script written by someone else, but without checking whether the file was
received correctly.

This curl script downloads multiple files at once (and occasionally one is
incomplete).

I will see if I can adapt the script I am using, using your suggestions.

Thank you.

On Mon, Dec 19, 2016 at 6:54 AM, Griffith Young <griffith.young at bom.gov.au>
wrote:

> Hello Barry,
>
>                 "I assume that Guido's approach will be better for this."
> - I agree
>
>
>
>                 "Of course, ... sometimes my download fails ... .  I am
> looking for a way to download grib files from NCEP without missing part of
> the file (sometimes)."
>
>
>
>                 Sometimes I find it is better to use "curl" or "wget" as
> they have settings to cope with retries and errors:
>
>                                 $ man curl
>
>                                 $ man wget
>
>
>
>                 Then you can do something like:
>
>                                 for f in a b c ; do
>
>                                 wget http://www.example.org/
> directory/sub/${f}
>
>                                 if [ $? -ne 0 ] ; then echo "wget failed
> with non-zero return code" ; continue ; fi
>
>                                 if [ ! -f ${f} ] ; then echo "file ${f}
> does not exist" ; continue ; fi
>
>                                 ncl f=\"${f}\" << EOF
>
>                                 print("f: " + f)
>
>                                 EOF
>
>                                 done
>
>
>
>                 Knowledge is Power.  Your call of course.
>
>
>
> Regards,
>
>                 Griff.
>
>
>
>
>
> *From:* Barry Lynn [mailto:barry.h.lynn at gmail.com]
> *Sent:* Monday, 19 December 2016 2:13 PM
> *To:* Griffith Young
> *Cc:* ncl-talk
> *Subject:* Re: [ncl-talk] Skipping Missing Files [SEC=UNCLASSIFIED]
>
>
>
> Griff:
>
>
>
> Thank you for your suggestion.
>
>
>
> I will need that particular NCL script to continue to plot data beyond the
> time of the missing file.  I assume that Guido's approach will be better
> for this.
>
>
>
> Of course, the elephant in the room issue here is that sometimes my
> download fails even though the file is not missing or incomplete.  I am
> looking for a way to download grib files from NCEP without missing part of
> the file (sometimes).
>
>
>
> Barry
>
>
>
> On Mon, Dec 19, 2016 at 1:56 AM, Griffith Young <griffith.young at bom.gov.au>
> wrote:
>
> Hello Barry,
>
>                 An alternative, but possibly less cool method, is to call
> an NCL script from bash (or your preferred shell) for each file.
>
>
>
>                 The effect is then such that NCL can succeed or fail, but
> your script continues.
>
>                                 for f in a b c ; do
>
>                                 ncl f=\"$f\" << EOF
>
>                                 print("f: " + f)
>
>                                 EOF
>
>                                 done
>
>
>
>                 I prefer Guido's suggestion, but when it comes to quick
> and dirty, this is also an option.
>
>
>
> Regards,
>
>                 Griff.
>
>
>
> *From:* ncl-talk-bounces at ucar.edu [mailto:ncl-talk-bounces at ucar.edu] *On
> Behalf Of *Guido Cioni
> *Sent:* Sunday, 18 December 2016 4:05 AM
> *To:* Barry Lynn
> *Cc:* ncl-talk
> *Subject:* Re: [ncl-talk] Skipping Missing Files
>
>
>
> Hey Barry,
>
> An easy way of doing it would be to encapsulate all the file operations
> into an IF statement which uses either the isfilevar (
> http://www.ncl.ucar.edu/Document/Functions/Built-in/isfilevar.shtml)
> function or the isfilepresent (http://www.ncl.ucar.edu/
> Document/Functions/Built-in/isfilepresent.shtml) function to check the
> existence of either variable in a file or the file itself. You just have to
> put an OR so that the operations are skipped, but the script is not
> interrupted, if either the variable or the file are not found.
>
> Cheers
>
>
>
> Il 17 dic 2016 5:29 PM, "Barry Lynn" <barry.h.lynn at gmail.com> ha scritto:
>
> Hi:
>
>
>
> I am downloading a lot of data and processing it, file by file.
>
>
>
> It happens that sometimes a file is missing, or a even a variable within
> the file is missing.
>
>
>
> How can I tell NCL to skip the read, instead of a returning an error and
> ending the program.
>
>
>
> Basically, I want NCL to note errors, and bypass them, instead of
> terminating.
>
>
>
> Barry
>
>
>
> --
>
> Barry H. Lynn, Ph.D
>
> Senior Lecturer,
>
> The Institute of the Earth Science,
> The Hebrew University of Jerusalem,
> Givat Ram, Jerusalem 91904, Israel
>
> Tel: 972 547 231 170
> Fax: (972)-25662581
>
>
>
> C.E.O, Weather It Is, LTD
> Weather and Climate Focus
> http://weather-it-is.com
> Jerusalem, Israel
> Local: 02 930 9525
> Cell: 054 7 231 170
> Int-IS: x972 2 930 9525
> US 914 432 3108 <(914)%20432-3108>
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
>
>
> --
>
> Barry H. Lynn, Ph.D
>
> Senior Lecturer,
>
> The Institute of the Earth Science,
> The Hebrew University of Jerusalem,
> Givat Ram, Jerusalem 91904, Israel
>
> Tel: 972 547 231 170
> Fax: (972)-25662581
>
>
>
> C.E.O, Weather It Is, LTD
> Weather and Climate Focus
> http://weather-it-is.com
> Jerusalem, Israel
> Local: 02 930 9525
> Cell: 054 7 231 170
> Int-IS: x972 2 930 9525
> US 914 432 3108 <(914)%20432-3108>
>



-- 
Barry H. Lynn, Ph.D
Senior Lecturer,
The Institute of the Earth Science,
The Hebrew University of Jerusalem,
Givat Ram, Jerusalem 91904, Israel
Tel: 972 547 231 170
Fax: (972)-25662581

C.E.O, Weather It Is, LTD
Weather and Climate Focus
http://weather-it-is.com
Jerusalem, Israel
Local: 02 930 9525
Cell: 054 7 231 170
Int-IS: x972 2 930 9525
US 914 432 3108
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20161219/f4577adb/attachment.html