<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap: break-word;-webkit-nbsp-mode: space;-webkit-line-break: after-white-space">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hello Jennifer,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I have also found that files become corrupted if wget is run too often with the “-c” option, resulting in a file which is as big as or larger than the source
file, but with the wrong checksum. Apparently this is not meant to happen, but it does. My solution is to remove incoming file and start again if it has not been successfully downloaded after 3 or 4 attempts with “wget -c” (I don’t know what the optimal number
is here – it obviously varies with the network speed, reliability of the thredds server at the other end, and size of the data files, and probably many other things as well --- too many variables to do a systematic survey). In some cases I have also found
it necessary to impose a rate limit (e.g. “--limit-rate=400k”), to stop corruption happening during automatic retries – this clearly slows down the data rate on individual threads, but increases reliability;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Sincerely,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Martin Juckes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> go-essp-tech-bounces@ucar.edu [mailto:go-essp-tech-bounces@ucar.edu]
<b>On Behalf Of </b>Jennifer Adams<br>
<b>Sent:</b> 20 December 2011 01:08<br>
<b>To:</b> go-essp-tech@ucar.edu<br>
<b>Subject:</b> [Go-essp-tech] zombie processes<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Dear Colleagues, <o:p></o:p></p>
<div>
<p class="MsoNormal">I am doing all my wgetting on a suite of 64-bit CentOS boxes with a common gluster filesystem. My downloads are leaving a trail of hung processes on the servers that can't be killed (we refer to them as zombies). Too many zombies and the
only remedy is a reboot, which is inconvenient for all the other users doing stuff on the servers. Some file-locking errors have shown up in the logs, which makes sense because the zombies are usually 'mv' or 'ls' commands, or the wget command itself, or similar.
One pattern I've seen is a file gets stuck as a partial download, the checksums don't match, so it starts the download again, but the file can't be overwritten, and the wget gets caught in an infinite loop. And you can't kill the process or delete the file
in question without spawning more zombies. It also seems to happen when I poke at the 0-byte files that occur when a download fails. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">My sysadm is considering a gluster upgrade, but that would be big job and highly disruptive and before I send him down that path I'd like to be sure that we've ruled out other possible causes. Nobody at COLA but us CMIP5 downloaders are
reporting any problems. So, has anyone else experienced these kinds of issues? Is it possible that the interaction of wget with a busy data node is the cause?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">> wget --version<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">GNU Wget 1.11.4 Red Hat modified<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">> uname -a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Linux <a href="http://cola2.gmu.edu">cola2.gmu.edu</a> 2.6.18-238.12.1.el5 #1 SMP Tue May 31 13:22:04 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">--Jennifer<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:9.0pt">--</span></span><o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">Jennifer M. Adams<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">IGES/COLA<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">4041 Powder Mill Road, Suite 302<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">Calverton, MD 20705<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black"><a href="mailto:jma@cola.iges.org">jma@cola.iges.org</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black"><br>
<br>
</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<br><p>--
<BR>Scanned by iCritical.
</p>
<br></body>
</html>