<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<font face="Times New Roman">Hi,<br>
<br>
I've added my download rates to Estani's wiki. I've also included
the number of "streams" that were active. <br>
<br>
Estani, please correct anything I've done wrong.<br>
<br>
thanks,<br>
Karl<br>
</font><br>
On 3/3/11 3:13 AM, Estanislao Gonzalez wrote:
<blockquote cite="mid:4D6F77D6.3010209@zmaw.de" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
Hi,<br>
<br>
by Michael suggestion I started a page in the esgf.org wiki were
we could write our measurement regarding CMIP5 federation.<br>
It is not intended for the technical staff, so don't complain
about it being too simplified :-)<br>
On the contrary, it is intended to be a very first and rough
approximation to what's happening in the cloud by allowing
everyone in 5 sec to write a measurement down.<br>
<br>
Everyone has write access to this and if we see the necessity we
could evolve it into something more detailed, but which will
certainly leave the lay users outside.<br>
<br>
And the link:<br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://esgf.org/wiki/Cmip5Status/Datarates">http://esgf.org/wiki/Cmip5Status/Datarates</a><br>
<br>
Does someone else find this useful? Any comments?<br>
<br>
Thanks,<br>
Estani<br>
<br>
Am 03.03.2011 09:04, schrieb Estanislao Gonzalez:
<blockquote cite="mid:4D6F4B8A.4080509@zmaw.de" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
Hi Karl,<br>
<br>
I'm currently testing the replication via wget and got an
average of 350 Mbps (around 40MB/s) with 7 parallel streams in
the last 14 hours. I've already downloaded 2TB.<br>
<br>
So I guess the problem is across the Atlantic. I'd suggest to
start many more parallel streams.<br>
<br>
In any case I think this might be caused by some
problems/congestion in intermediate networks. The network
throughput to ECMWF *might* not be comparable because of the
network architecture, e.g. intermediate proxies, firewalls, etc.
But a 10x speed up might be of no use anyway as downloading 1TB
in 2 Months seems hardly to be an option...<br>
<br>
When we all have the data replicated worldwide though, it will
be possible to download from all 3 servers at the same time,
which will more than double the speed from this side of the
ocean, and will be a considerable speed up from yours, although
probably not as much (don't nail me on this, I'll deny it
vehemently <- this is not my writing :-)<br>
<br>
Anyway, this being said, the rest should be answered by BADC
directly.<br>
<br>
Thanks,<br>
Estani<br>
<br>
Am 03.03.2011 02:26, schrieb Karl Taylor:<br>
<span style="white-space: pre;">> this should have been sent
to<br>
<br>
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a><br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> sorry about that.<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> Karl<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> -------- Original Message -------- Subject: download
rates<br>
<br>
Date:<br>
<br>
<br>
<br>
> Wed, 2 Mar 2011 16:43:42 -0800 From: Karl Taylor<br>
<br>
<br>
<br>
> <a moz-do-not-send="true" class="moz-txt-link-rfc2396E"
href="mailto:taylor13@llnl.gov"><taylor13@llnl.gov></a><br>
To: <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:esg-support@earthsystemgrid.org">esg-support@earthsystemgrid.org</a><br>
<br>
Grid<br>
<br>
<br>
<br>
> <a moz-do-not-send="true" class="moz-txt-link-rfc2396E"
href="mailto:esg-support@earthsystemgrid.org"><esg-support@earthsystemgrid.org></a><br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> Hi all,<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> When I download a 25 MB CMIP5 file from BADC using the
wget<br>
<br>
method: *<br>
<br>
<br>
<br>
> it took 35 sec to get started * then it downloaded at an<br>
<br>
average rate<br>
<br>
<br>
<br>
> of 1.1 MB/min<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> When I download the same file using the point and click<br>
<br>
method: * it<br>
<br>
<br>
<br>
> took 20 sec to get started * then it downloaded at an<br>
average<br>
<br>
rate of<br>
<br>
<br>
<br>
> 1.4 MB/min<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> When I download 4 files simultaneously (in parallel)
using<br>
<br>
the click<br>
<br>
<br>
<br>
> method, I get somewhat slower download rate per file, so<br>
<br>
somewhat<br>
<br>
<br>
<br>
> less than 4 times the data transfer rate I would have<br>
gotten<br>
<br>
<br>
<br>
> downloading them in series.<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> Note that at 1 MB/min, I could download 1 GB in about 17<br>
<br>
hours or 1<br>
<br>
<br>
<br>
> TB (i.e., about 1/1000 of the entire archive) in about 2<br>
<br>
years.<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> Any way to speed this up some? (A colleague here
downloads<br>
<br>
files at<br>
<br>
<br>
<br>
> about 10 times this rate from ECMWF.)<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> Note that I haven't checked how fast download rates are<br>
from<br>
<br>
nodes<br>
<br>
<br>
<br>
> published to our (PCMDI's) gateway (since they are not
yet<br>
<br>
publicly<br>
<br>
<br>
<br>
> available).<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> Best regards, Karl<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
> _______________________________________________<br>
GO-ESSP-TECH<br>
<br>
mailing<br>
<br>
<br>
<br>
> list <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><br>
<br>
<br>
<br>
<br>
> <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a></span><br>
<br>
<br>
-- <br>
Estanislao Gonzalez<br>
<br>
Max-Planck-Institut für Meteorologie (MPI-M)<br>
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing
Centre<br>
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany<br>
<br>
Phone: +49 (40) 46 00 94-126<br>
E-Mail: <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:estanislao.gonzalez@zmaw.de">estanislao.gonzalez@zmaw.de</a>
<br>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:estanislao.gonzalez@zmaw.de">estanislao.gonzalez@zmaw.de</a> </pre>
</blockquote>
</body>
</html>