<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap: break-word;-webkit-nbsp-mode: space;-webkit-line-break: after-white-space">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hello Jennifer,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Just to add to what Sebastien says: I have also been able to fetch data using PKI enabled wget from tds.ucar.edu,
</span><a href="http://dap.cccma.uvic.ca">dap.cccma.uvic.ca</a>, <a href="http://bcccsm.cma.gov.cn">
bcccsm.cma.gov.cn</a>, <a href="http://pcmdi9.llnl.gov">pcmdi9.llnl.gov</a>. The Chinese node was down for much of last week, but appeared to be stable this week.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">On most nodes I find a few files with persistent 400, 403, 500 errors or checksums differing from those in the catalogue. The data node managers I’ve contacted about these generally know about the problems and are doing their best to fix
them. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">A few months ago I was running scripts with 60 to 100 parallel wget threads without problems. In December and January this became less successful – I think some data nodes imposed limits, but more generally I think the problem is that 60
requests from one user is OK, but 60 requests from 10 users is perhaps to much. So now I limit my scripts to 6 to 10 threads per data node.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I also found I was getting frequent file corruptions on data transfers from Australia – and have got far better success rate since adding a “--limit-rate=800k” flag to transfers from China, Japan and Australia (but you may have better network
links to these nodes than us). <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal">Martin<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> go-essp-tech-bounces@ucar.edu [mailto:go-essp-tech-bounces@ucar.edu]
<b>On Behalf Of </b>Jennifer Adams<br>
<b>Sent:</b> 02 February 2012 00:54<br>
<b>To:</b> go-essp-tech@ucar.edu<br>
<b>Subject:</b> Re: [Go-essp-tech] Progress and Problems with P2P<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Feb 1, 2012, at 4:20 PM, Sébastien Denvil wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal">Hi Jennifer, all<br>
<br>
see below:<br>
<br>
Le 01/02/2012 17:08, Jennifer Adams a écrit :<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">Hi, Everyone --<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">I have been working with Luca and Gavin and Estani on testing the P2P system. I'm happy to report that the many significant parts of my workflow for downloading data can now be fully automated:<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">1. Search for available data sets that meet my desired requirements (e.g. decadal1980/atmos/mon/Amon, all models, all members, selected variables)<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">2. Compare search results to a list of what I've already got<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">3. Build script to download and run wget scripts for datasets I still need<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">All this using shell scripts and without touching a browser!<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">There are still a few wrinkles, however. The worst of these is that not all data nodes are authenticating certificates properly.<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">These data nodes are working:<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="http://bmbf-ipcc-ar5.dkrz.de">bmbf-ipcc-ar5.dkrz.de</a><o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">cmip-dn.badc.rl.ac.uk<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">dias-esg-nd.tkl.iis.u-tokyo.ac.jp<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="http://esgdata.gfdl.noaa.gov">esgdata.gfdl.noaa.gov</a><o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="http://esg.cnrm-game-meteo.fr">esg.cnrm-game-meteo.fr</a><o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="http://norstore-trd-bio1.hpc.ntnu.no">norstore-trd-bio1.hpc.ntnu.no</a><o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="http://pcmdi11.llnl.gov">pcmdi11.llnl.gov</a><o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="http://vesg.ipsl.fr">vesg.ipsl.fr</a><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
To help you diagnose the source of your PKI issues :<br>
<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">These data nodes are not:<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="http://dap.cccma.uvic.ca">dap.cccma.uvic.ca</a><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
It works for us as I write. PCMDI openid if it matters.<o:p></o:p></p>
</div>
<p class="MsoNormal">I believe it does matter which OpenID you use, and PCMDI is the one that works. Gavin told me that he didn't have many problems with downloads from these nodes either. I think it makes a difference whether you are wgetting from a data node
or from someplace else. Gavin seems to have the keys to all the castles.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><a href="http://esg.nccs.nasa.gov">esg.nccs.nasa.gov</a><o:p></o:p></p>
<p class="MsoNormal">We have never been able to download from there using PKI.<o:p></o:p></p>
</div>
<p class="MsoNormal">I got some email directly from Ellen Salmon at GSFC who explained that they have been working for months to get all the upgraded ESGF software packages to work properly, but there is hope that it will be working soon (~weeks). I know this
has been a chronic problem for data node administrators.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><a href="http://bcccsm.cma.gov.cn">bcccsm.cma.gov.cn</a><o:p></o:p></p>
<p class="MsoNormal">We have been able to download from there using PKI recently.<o:p></o:p></p>
</div>
<p class="MsoNormal">This one fails 95% of the time for me. Estani says BCC is a special case.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><a href="http://tds.ucar.edu">tds.ucar.edu</a><o:p></o:p></p>
<p class="MsoNormal">We have been able to download from there using PKI recently.<o:p></o:p></p>
</div>
<p class="MsoNormal">This may another case of it working for requests from other data nodes. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><a href="http://pcmdi9.llnl.gov">pcmdi9.llnl.gov</a><o:p></o:p></p>
<p class="MsoNormal">We used pcmdi3 up to know (it worked), will give a try to pcmdi9<o:p></o:p></p>
</div>
<p class="MsoNormal">Gavin has been tweaking this one lately. I haven't tested it in the past few days. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><a href="http://esg-datanode.jpl.nasa.gov">esg-datanode.jpl.nasa.gov</a><o:p></o:p></p>
<p class="MsoNormal">Did not tried yet.<br>
<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">P2P wget scripts to download data from the second set of data nodes always fail completely (for me, a "pure client" user). I know that Luca and Gavin and Estani are working to fix these problems, but here's some encouragement to the data
node administrators to help them resolve this issue. Wgets that rely on the quickly-expiring authorization tokens ought to be deprecated as soon as possible -- they are the biggest, most irritating source of errors in the whole system (my opinion only).<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">The second problem is that the wget scripts don't often succeed in getting all the files they are configured to grab. Of the 64 scripts that I ran this morning, 55 were incomplete (that's 86%). The errors in log files are all "ERROR 403:
Forbidden." I was running the 64 scripts at one time, but each one was grabbing its list of files in order -- I am not parallelizing the wgets in the way that some other users have described. When I rerun the wget scripts from the incomplete runs, some of
them finish the job on the 2nd try, others will take as many as 8 tries before all the files are in.<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">These are the data nodes that are not allowing me to complete the download (for the decadal1980 example mentioned above).<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="http://esg.cnrm-game-meteo.fr">esg.cnrm-game-meteo.fr</a><o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="http://vesg.ipsl.fr">vesg.ipsl.fr</a><o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">dias-esg-nd.tkl.iis.u-tokyo.ac.jp<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">cmip-dn.badc.rl.ac.uk<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="http://bmbf-ipcc-ar5.dkrz.de">bmbf-ipcc-ar5.dkrz.de</a><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
I notice that those node seems to be far away from your home location.<br>
May I suggest you to give those options to wget ; from our experiences with synchro-data they appear to improve substantially the success rate :<br>
wget --timeout=20 --tries=10<o:p></o:p></p>
</div>
<p class="MsoNormal">I will definitely give those options a try. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
A lot has been said about this issue.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I'm sorry I didn't go searching through the forum archives before writing about wget failures. I didn't join this list until fairly recently, when I started to actively work on downloading CMIP5 data. I was subscribed to plain go-essp (not
tech) and there was no traffic at all. Here is where the the players are!! I'm glad I'm here … lately I've been following the threads in this forum with more interest than in my own GrADS forum. <o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal">Difficult to attribute properly the responsibility of those failure to such or such components/parties. Give a try to the parameter above.<br>
<br>
If you send me the IP adress you used we could investigate the <a href="http://vesg.ipsl.fr">
vesg.ipsl.fr</a> case.<o:p></o:p></p>
</div>
<p class="MsoNormal">The boxes I have been using are inside a firewall and do not have public IP addresses. I did a test by pinging my public GDS from there and in the log file it came up as
<a href="http://cola.gmu.edu">cola.gmu.edu</a>. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">I suspect there are some throttling settings in place to limit the number of wgets that a data node will allow at any particular time. I think my use of the data nodes is reasonable, and these throttles are set too high.<o:p></o:p></p>
<p class="MsoNormal"><br>
It's quiet hard to set this up properly using a tomcat free version. Some commercial tomcat version offers easiest way to achieve that. Up to know (on our node :
<a href="http://vesg.ipsl.fr">vesg.ipsl.fr</a>) we haven't done anything specific regarding that.<o:p></o:p></p>
</div>
<p class="MsoNormal">Huh! Well, if you think it would be worthwhile, I would be willing to try to slam your data node with a barrage of wgets from one of our servers that has a resolvable IP and we could see what it takes to receive the Forbidden error on a
default tomcat implementation. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">--Jennifer<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
regards.<br>
Sébastien<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">The dreaded "Forbidden" errors may be related to another problem, but are still a nuisance no matter what the reason. YAO (Yet Another Obstacle). Please put this on the high priority list of things to fix, right behind the certificate authentication
issue.<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Respectfully submitted,<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Jennifer<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">_______________________________________________<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">GO-ESSP-TECH mailing list<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
<br>
-- <br>
Sébastien Denvil<br>
IPSL, Pôle de modélisation du climat<br>
UPMC, Case 101, 4 place Jussieu,<br>
75252 Paris Cedex 5<br>
<br>
Tour 45-55 2ème étage Bureau 209<br>
Tel: 33 1 44 27 21 10<br>
Fax: 33 1 44 27 39 02<br>
<br>
<br>
_______________________________________________<br>
GO-ESSP-TECH mailing list<br>
<a href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><br>
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">Jennifer M. Adams<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">IGES/COLA<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">4041 Powder Mill Road, Suite 302<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">Calverton, MD 20705<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black"><a href="mailto:jma@cola.iges.org">jma@cola.iges.org</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black"><br>
<br>
</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<br><p>--
<BR>Scanned by iCritical.
</p>
<br></body>
</html>