[Wrf-users] GFS file size problem

[David Smith]

I ran into this problem even with the 0.5 data.  I was able to get around it using mget rather than wget or curl.  I have a bash script that generates filenames for all need and then I make a simple call


mget $gribfile1 $gribfile2 $gribfile3 $gribfile4 $gribfile5 $gribfile6 $gribfile

7 $gribfile8 $gribfile9 $gribfile10 $gribfile11 $gribfile12 $gribfile13 $gribfil

e14 $gribfile15 $gribfile16 $gribfile17


  I've never been able to find out why wget and curl don't reliably work, but mget seems to always work - error recovery perhaps.  Good luck.


Dave


Dr. David J Smith
College of Science and Mathematics
University of the Virgin Islands
St. Thomas,  VI  00802
340 693-1257
dsmith at uvi.edu

From: Micha Silver <micha at arava.co.il<mailto:micha at arava.co.il>>
Date: Sat, 24 Oct 2015 14:53:58 +0300
To: Bart Brashers <bbrashers at ramboll.com<mailto:bbrashers at ramboll.com>>
Cc: <Wrf-users at ucar.edu<mailto:Wrf-users at ucar.edu>>, erick fredj <erick.fredj at gmail.com<mailto:erick.fredj at gmail.com>>
Subject: Re: [Wrf-users] GFS file size problem


Hello Bart:
Thanks
It seems with "curl -l" I am getting only the file names, not sizes.
??
Best,
Micha

On 23/10/15 20:51, Bart Brashers wrote:
Does "curl -l http://..." return the same filesizes as wget? Curl might be more honest...

Bart
________________________________________
Bart Brashers, Ph.D.
Senior Manager, Air Quality Meteorology

D +1 425 4121812
M +1 206 5502606
bbrashers at ramboll.com<mailto:bbrashers at ramboll.com>  -- Please note my new email address

Ramboll Environ
19020 33rd Avenue West
Suite 310
Lynnwood, WA 98036
USA
<http://www.ramboll-environ.com/>www.ramboll-environ.com<http://www.ramboll-environ.com>

[Ramboll_Environ_Logo]

From: wrf-users-bounces at ucar.edu<mailto:wrf-users-bounces at ucar.edu> [mailto:wrf-users-bounces at ucar.edu] On Behalf Of Micha Silver
Sent: Friday, October 23, 2015 10:29 AM
To: Wrf-users at ucar.edu<mailto:Wrf-users at ucar.edu>
Subject: [Wrf-users] GFS file size problem

Hello all:

We are running WRF-Hydro in an operational flood forecasting system. We download GFS data twice daily from http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/ using wget. In order to be sure that we are getting the full data files, we first get the file sizes with the wget options:  --spider --server-response. We save the file sizes before download, then compare the sizes after download. If there's any differences, we restart the download.

This mostly works well. When downloading the 0.5 degree data it works perfectly. However when getting the 0.25 data we have noticed that occasionally the sizes reported by the NCEP server appear to not be the actual file sizes. This mismatch does not occur consistently, but rather only about once in two weeks or so. When this happens, then even waiting a few hours, and restarting the simulation run does not help. Our wget --spider and comparison of downloaded file sizes fails again. Then at the next GFS cycle everything goes back to normal.

Has anyone seen this behavior of wget receiving incorrect file sizes? Is there some other recommended way to ensure that the downloaded GFS data are the final, complete files.

many thanks,
Micha

________________________________
This message contains information that may be confidential, privileged or otherwise protected by law from disclosure. It is intended for the exclusive use of the Addressee(s). Unless you are the addressee or authorized agent of the addressee, you may not review, copy, distribute or disclose to anyone the message or any information contained within. If you have received this message in error, please contact the sender by electronic reply to email at ramboll.com<mailto:email at ramboll.com> and immediately delete all copies of the message.

This mail was received via Mail-SeCure System.

_______________________________________________ Wrf-users mailing list Wrf-users at ucar.edu<mailto:Wrf-users at ucar.edu> http://mailman.ucar.edu/mailman/listinfo/wrf-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/wrf-users/attachments/20151026/0f6c8c00/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Preview.app Document
Type: image/png
Size: 1494 bytes
Desc: Preview.app Document
Url : http://mailman.ucar.edu/pipermail/wrf-users/attachments/20151026/0f6c8c00/attachment-0001.png