<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
<br>
If the performance Karl is seeing is caused by a trans-Atlantic
bottleneck, then this may be one of the cases we discussed in the past
where the data need to be shipped rather than transferred over the
network. <br>
<br>
Ann<br>
<br>
<br>
On 3/3/11 12:04 AM, Estanislao Gonzalez wrote:
<blockquote cite="mid:4D6F4B8A.4080509@zmaw.de" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
Hi Karl,<br>
<br>
I'm currently testing the replication via wget and got an average of
350 Mbps (around 40MB/s) with 7 parallel streams in the last 14 hours.
I've already downloaded 2TB.<br>
<br>
So I guess the problem is across the Atlantic. I'd suggest to start
many more parallel streams.<br>
<br>
In any case I think this might be caused by some problems/congestion in
intermediate networks. The network throughput to ECMWF *might* not be
comparable because of the network architecture, e.g. intermediate
proxies, firewalls, etc. But a 10x speed up might be of no use anyway
as downloading 1TB in 2 Months seems hardly to be an option...<br>
<br>
When we all have the data replicated worldwide though, it will be
possible to download from all 3 servers at the same time, which will
more than double the speed from this side of the ocean, and will be a
considerable speed up from yours, although probably not as much (don't
nail me on this, I'll deny it vehemently <- this is not my writing
:-)<br>
<br>
Anyway, this being said, the rest should be answered by BADC directly.<br>
<br>
Thanks,<br>
Estani<br>
<br>
Am 03.03.2011 02:26, schrieb Karl Taylor:<br>
<span style="white-space: pre;">> this should have been sent to<br>
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a><br>
<br>
> <br>
<br>
> sorry about that.<br>
<br>
> <br>
<br>
> Karl<br>
<br>
> <br>
<br>
> -------- Original Message -------- Subject: download rates<br>
Date:<br>
<br>
> Wed, 2 Mar 2011 16:43:42 -0800 From: Karl Taylor<br>
<br>
> <a moz-do-not-send="true" class="moz-txt-link-rfc2396E"
href="mailto:taylor13@llnl.gov"><taylor13@llnl.gov></a> To: <a
moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:esg-support@earthsystemgrid.org">esg-support@earthsystemgrid.org</a><br>
Grid<br>
<br>
> <a moz-do-not-send="true" class="moz-txt-link-rfc2396E"
href="mailto:esg-support@earthsystemgrid.org"><esg-support@earthsystemgrid.org></a><br>
<br>
> <br>
<br>
> <br>
<br>
> <br>
<br>
> Hi all,<br>
<br>
> <br>
<br>
> When I download a 25 MB CMIP5 file from BADC using the wget<br>
method: *<br>
<br>
> it took 35 sec to get started * then it downloaded at an<br>
average rate<br>
<br>
> of 1.1 MB/min<br>
<br>
> <br>
<br>
> When I download the same file using the point and click<br>
method: * it<br>
<br>
> took 20 sec to get started * then it downloaded at an average<br>
rate of<br>
<br>
> 1.4 MB/min<br>
<br>
> <br>
<br>
> When I download 4 files simultaneously (in parallel) using<br>
the click<br>
<br>
> method, I get somewhat slower download rate per file, so<br>
somewhat<br>
<br>
> less than 4 times the data transfer rate I would have gotten<br>
<br>
> downloading them in series.<br>
<br>
> <br>
<br>
> Note that at 1 MB/min, I could download 1 GB in about 17<br>
hours or 1<br>
<br>
> TB (i.e., about 1/1000 of the entire archive) in about 2<br>
years.<br>
<br>
> <br>
<br>
> Any way to speed this up some? (A colleague here downloads<br>
files at<br>
<br>
> about 10 times this rate from ECMWF.)<br>
<br>
> <br>
<br>
> Note that I haven't checked how fast download rates are from<br>
nodes<br>
<br>
> published to our (PCMDI's) gateway (since they are not yet<br>
publicly<br>
<br>
> available).<br>
<br>
> <br>
<br>
> Best regards, Karl<br>
<br>
> <br>
<br>
> <br>
<br>
> <br>
<br>
> <br>
<br>
> _______________________________________________ GO-ESSP-TECH<br>
mailing<br>
<br>
> list <a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a> <br>
<br>
> <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a></span><br>
<br>
<br>
-- <br>
Estanislao Gonzalez<br>
<br>
Max-Planck-Institut für Meteorologie (MPI-M)<br>
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre<br>
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany<br>
<br>
Phone: +49 (40) 46 00 94-126<br>
E-Mail: <a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:estanislao.gonzalez@zmaw.de">estanislao.gonzalez@zmaw.de</a>
<br>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
</body>
</html>