<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:2 7 4 9 2 2 5 2 4 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap: break-word;-webkit-nbsp-mode: space;-webkit-line-break: after-white-space">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Jennifer,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="line-height:200%"><span style="font-size:11.0pt;line-height:200%;font-family:"Calibri","sans-serif";color:#1F497D">I found a similar problem last year, transferring data from CSIRO and Japan to BADC – though I stopped after 4 attempts
to get one file, which all resulted in different checksums. In each case I was using wget and it was doing multiple automatic restarts. I suppressed the problem by running with “--limit-rate=500k” in the wget command line. This resulted in transfers running
with no interruptions and restarts, and consistent checksums. It is also, of course, slower – but I haven’t found a way around that. I found some discussion on various web fora which may be related, suggesting that a delay between automated restarts is useful
to prevent corruption – but all the discussions I found were ambiguous, so I didn’t keep a record. For data I’m fetching from European and North American nodes I think this problem is too rare to merit running with a limited transfer rate – the few transfers
that fail simply have to be repeated,<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:200%"><span style="font-size:11.0pt;line-height:200%;font-family:"Calibri","sans-serif";color:#1F497D">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:200%"><span style="font-size:11.0pt;line-height:200%;font-family:"Calibri","sans-serif";color:#1F497D">Martin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> go-essp-tech-bounces@ucar.edu [mailto:go-essp-tech-bounces@ucar.edu]
<b>On Behalf Of </b>Jennifer Adams<br>
<b>Sent:</b> 11 April 2012 19:52<br>
<b>To:</b> go-essp-tech@ucar.edu<br>
<b>Subject:</b> [Go-essp-tech] repeated md5 chksum failures<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi, Everyone -- <o:p></o:p></p>
<div>
<p class="MsoNormal">I'm trying to download some fairly large files (~1Gb) from the piControl run (monthly ocean variables) and find that the checksum fails to match several times and then will be ok. In some cases, it can take 10 or more re-tries before the
checksum succeeds. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The problem is not with a specific data node. Here are some of the dataset IDs for the troublesome downloads: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">cmip5.output1.CCCma.CanESM2.piControl.mon.ocean.Omon.r1i1p1.v20111028</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">cmip5.output1.INM.inmcm4.piControl.mon.ocean.Omon.r1i1p1.v20110323</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">cmip5.output1.MIROC.MIROC-ESM.piControl.mon.ocean.Omon.r1i1p1.v20110929</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">cmip5.output1.MRI.MRI-CGCM3.piControl.mon.ocean.Omon.r1i1p1.v20110831</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">cmip5.output1.NCAR.CCSM4.piControl.mon.ocean.Omon.r1i1p1.v20120220</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">cmip5.output1.NCC.NorESM1-M.piControl.mon.ocean.Omon.r1i1p1.v20110901</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">cmip5.output2.MRI.MRI-CGCM3.piControl.mon.ocean.Omon.r1i1p1.v20110831</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">cmip5.output2.NCC.NorESM1-M.piControl.mon.ocean.Omon.r1i1p1.v20110901</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">cmip5.output1.MPI-M.MPI-ESM-LR.piControl.mon.ocean.Omon.r1i1p1.v20120315</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">cmip5.output1.MPI-M.MPI-ESM-P.piControl.mon.ocean.Omon.r1i1p1.v20120315</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">cmip5.output2.MPI-M.MPI-ESM-P.piControl.mon.ocean.Omon.r1i1p1.v20111028</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">For example, from the final two datasets in the list, here is an entry from the wget script:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">'rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' '<a href="http://bmbf-ipcc-ar5.dkrz.de/thredds/fileServer/cmip5/output2/MPI-M/MPI-ESM-P/piControl/mon/ocean/Omon/r1i1p1/v20111028/rhopoto/rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'">http://bmbf-ipcc-ar5.dkrz.de/thredds/fileServer/cmip5/output2/MPI-M/MPI-ESM-P/piControl/mon/ocean/Omon/r1i1p1/v20111028/rhopoto/rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc'</a>
'MD5' '036aabfc10caa76a8943f967bc10ad4d'</span></span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Here are the 21 download tries so far today, taking 5 hours, the "md5 failed!" message appears in the log file after each one: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 09:19:18 (2.19 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 09:35:05 (1.13 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 09:53:26 (1009 KB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 10:05:52 (1.49 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 10:17:03 (1.61 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 10:31:14 (1.30 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 10:48:50 (1.04 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 11:01:09 (1.46 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 11:14:01 (1.40 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 11:29:46 (1.15 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 11:42:39 (1.40 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 12:01:05 (1011 KB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 12:18:25 (1.03 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 12:35:30 (1.04 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 12:49:44 (1.35 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 13:08:38 ( 960 KB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 13:26:11 (1.01 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 13:36:21 (1.78 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 13:50:53 (1.25 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 14:06:26 (1.15 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">2012-04-11 14:19:43 (1.39 MB/s) - `rhopoto_Omon_MPI-ESM-P_piControl_r1i1p1_185001-185912.nc' saved [1083611268/1083611268]</span></span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This one failed 14 times before finally getting the "md5 ok" message -- it took 3 hrs 45 minutes to get this file:<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:10.0pt;font-family:Courier">'so_Omon_MPI-ESM-P_piControl_r1i1p1_189001-189912.nc' '<a href="http://bmbf-ipcc-ar5.dkrz.de/thredds/fileServer/cmip5/output1/MPI-M/MPI-ESM-P/piControl/mon/ocean/Omon/r1i1p1/v20120315/so/so_Omon_MPI-ESM-P_piControl_r1i1p1_189001-189912.nc'">http://bmbf-ipcc-ar5.dkrz.de/thredds/fileServer/cmip5/output1/MPI-M/MPI-ESM-P/piControl/mon/ocean/Omon/r1i1p1/v20120315/so/so_Omon_MPI-ESM-P_piControl_r1i1p1_189001-189912.nc'</a>
'MD5' '175d6c9dd3ffea30186e6bc9c7e3dee1'</span></span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This problem is sucking up my bandwidth and my time, which are not unlimited. Is there any remedy? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">--Jennifer<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<br><p>--
<BR>Scanned by iCritical.
</p>
<br></body>
</html>