<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Jennifer,<br>
<br>
please be aware that this is not only taking up your bandwidth but
also that from everyone. That's why I'm not entirely comfortable
with automatic systems that can't detect a problem (in this case
downloading a 1G file 21 times which might have run over the weekend
or even longer). But that's just to give some information on the
implications of such procedures.<br>
<br>
Now regarding the file corruption... I can't tell you for sure
what's happening at your end. The wget script is recognizing the
file is corrupt and downloading it again, which is the expected (and
desired) behavior.<br>
Now why you are getting a corrupt file in such extremely high
proportions is beyond my knowledge, I've never seen a transport
failing in such a high rate.<br>
<br>
The md5 hash you've shown is the correct one:<br>
$ md5sum
/gpfs_750/projects/CMIP5/data/cmip5/output2/MPI-M/MPI-ESM-P/piControl/mon/ocean/Omon/r1i1p1/v20111028/rhopoto/rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc
036aabfc10caa76a8943f967bc10ad4d
/gpfs_750/projects/CMIP5/data/cmip5/output2/MPI-M/MPI-ESM-P/piControl/mon/ocean/Omon/r1i1p1/v20111028/rhopoto/rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc<br>
<br>
So this has at least a couple of possible causes:<br>
1) There's a network issue (might be since you are seeing corruption
from multiple files and datasets and the behavior is erratically)<br>
2) wget is not working as expected (I think you are on MacOS and
that system has not been properly tested, at least not by me).
Perhaps it's behaving like curl and writing to the file all errors
found, this is not the case with the wget version I've been testing
it width (and makes no sense in such a non-interrupted download)<br>
3) You have a disk failure (data gets corrupted when written to
disk)<br>
4) you have other memory/network buffer errors (unlikely as you
should have seen this happening with other connections in Internet)<br>
<br>
My advice: <br>
- alter the wget script and remove all file but one of the
problematic files.<br>
- Use "-f" to leave the file even if its md5 doesn't match, and be
sure this is the case (there will be an output to the console)<br>
- rename the file and get it again without the "-f" until is finally
there.<br>
- compare the files to see the difference (you'll see if there's
text in there, like an html page, or just bytes or a block of
bytes). In bash: <br>
diff <(hexdump -C file1) <(hexdump -C file2) | less<br>
- try getting the same file (3 times perhaps?) from a different
machine (even better if the OS is different). If it always succeeds
the problem is definitely related to your machine. If the behavior
is the same, then the issue is definitely in the network.<br>
<br>
Hope this helps,<br>
Estani<br>
<br>
Am 11.04.2012 20:51, schrieb Jennifer Adams:
<blockquote
cite="mid:AF9B9FF3-5FDF-4E3D-8C30-FBBC5191F1A0@cola.iges.org"
type="cite">Hi, Everyone --
<div>I'm trying to download some fairly large files (~1Gb) from
the piControl run (monthly ocean variables) and find that the
checksum fails to match several times and then will be ok. In
some cases, it can take 10 or more re-tries before the checksum
succeeds. </div>
<div><br>
</div>
<div>The problem is not with a specific data node. Here are some
of the dataset IDs for the troublesome downloads: </div>
<div><span class="Apple-style-span" style="font-family: Courier;
font-size: 13px; ">cmip5.output1.CCCma.CanESM2.piControl.mon.ocean.Omon.r1i1p1.v20111028</span></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">cmip5.output1.INM.inmcm4.piControl.mon.ocean.Omon.r1i1p1.v20110323</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">cmip5.output1.MIROC.MIROC-ESM.piControl.mon.ocean.Omon.r1i1p1.v20110929</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">cmip5.output1.MRI.MRI-CGCM3.piControl.mon.ocean.Omon.r1i1p1.v20110831</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">cmip5.output1.NCAR.CCSM4.piControl.mon.ocean.Omon.r1i1p1.v20120220</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">cmip5.output1.NCC.NorESM1-M.piControl.mon.ocean.Omon.r1i1p1.v20110901</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">cmip5.output2.MRI.MRI-CGCM3.piControl.mon.ocean.Omon.r1i1p1.v20110831</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">cmip5.output2.NCC.NorESM1-M.piControl.mon.ocean.Omon.r1i1p1.v20110901</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">cmip5.output1.MPI-M.MPI-ESM-LR.piControl.mon.ocean.Omon.r1i1p1.v20120315</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">cmip5.output1.MPI-M.MPI-ESM-P.piControl.mon.ocean.Omon.r1i1p1.v20120315</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">cmip5.output2.MPI-M.MPI-ESM-P.piControl.mon.ocean.Omon.r1i1p1.v20111028</span></font></div>
<div><br>
</div>
<div>
<div>For example, from the final two datasets in the list, here
is an entry from the wget script:</div>
<div><span class="Apple-style-span" style="font-family: Courier;
font-size: 13px; ">'rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
'<a moz-do-not-send="true"
href="http://bmbf-ipcc-ar5.dkrz.de/thredds/fileServer/cmip5/output2/MPI-M/MPI-ESM-P/piControl/mon/ocean/Omon/r1i1p1/v20111028/rhopoto/rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc%27">http://bmbf-ipcc-ar5.dkrz.de/thredds/fileServer/cmip5/output2/MPI-M/MPI-ESM-P/piControl/mon/ocean/Omon/r1i1p1/v20111028/rhopoto/rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'</a>
'MD5' '036aabfc10caa76a8943f967bc10ad4d'</span></div>
</div>
<div><br>
</div>
<div>Here are the 21 download tries so far today, taking 5 hours,
the "md5 failed!" message appears in the log file after each
one: </div>
<div><span class="Apple-style-span" style="font-family: Courier;
font-size: 13px; ">2012-04-11 09:19:18 (2.19 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></div>
<div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
09:35:05 (1.13 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
09:53:26 (1009 KB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
10:05:52 (1.49 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
10:17:03 (1.61 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
10:31:14 (1.30 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
10:48:50 (1.04 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
11:01:09 (1.46 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
11:14:01 (1.40 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
11:29:46 (1.15 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
11:42:39 (1.40 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
12:01:05 (1011 KB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
12:18:25 (1.03 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
12:35:30 (1.04 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
12:49:44 (1.35 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
13:08:38 ( 960 KB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
13:26:11 (1.01 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
13:36:21 (1.78 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
13:50:53 (1.25 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
14:06:26 (1.15 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">2012-04-11
14:19:43 (1.39 MB/s) -
`rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'
saved [1083611268/1083611268]</span></font></div>
</div>
<div><br>
</div>
<div>This one failed 14 times before finally getting the "md5 ok"
message -- it took 3 hrs 45 minutes to get this file:</div>
<div>
<div><font class="Apple-style-span" face="Courier"><span
class="Apple-style-span" style="font-size: 13px;">'so_Omon_MPI-ESM-P_piControl_r1i1p1_189001-189912.nc'
'<a moz-do-not-send="true"
href="http://bmbf-ipcc-ar5.dkrz.de/thredds/fileServer/cmip5/output1/MPI-M/MPI-ESM-P/piControl/mon/ocean/Omon/r1i1p1/v20120315/so/so_Omon_MPI-ESM-P_piControl_r1i1p1_189001-189912.nc%27">http://bmbf-ipcc-ar5.dkrz.de/thredds/fileServer/cmip5/output1/MPI-M/MPI-ESM-P/piControl/mon/ocean/Omon/r1i1p1/v20120315/so/so_Omon_MPI-ESM-P_piControl_r1i1p1_189001-189912.nc'</a>
'MD5' '175d6c9dd3ffea30186e6bc9c7e3dee1'</span></font></div>
</div>
<div><br>
</div>
<div>This problem is sucking up my bandwidth and my time, which
are not unlimited. Is there any remedy? </div>
<div>--Jennifer</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:gonzalez@dkrz.de">gonzalez@dkrz.de</a> </pre>
</body>
</html>