<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Estani,<div><br></div><div>Are all these numbers using a HTTP based download?</div><div><br></div><div>Rachana</div><div><br></div><div><div><div>On Mar 3, 2011, at 5:13 AM, Estanislao Gonzalez wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<div bgcolor="#ffffff" text="#000000">
Hi,<br>
<br>
by Michael suggestion I started a page in the <a href="http://esgf.org">esgf.org</a> wiki were we
could write our measurement regarding CMIP5 federation.<br>
It is not intended for the technical staff, so don't complain about
it being too simplified :-)<br>
On the contrary, it is intended to be a very first and rough
approximation to what's happening in the cloud by allowing everyone
in 5 sec to write a measurement down.<br>
<br>
Everyone has write access to this and if we see the necessity we
could evolve it into something more detailed, but which will
certainly leave the lay users outside.<br>
<br>
And the link:<br>
<a class="moz-txt-link-freetext" href="http://esgf.org/wiki/Cmip5Status/Datarates">http://esgf.org/wiki/Cmip5Status/Datarates</a><br>
<br>
Does someone else find this useful? Any comments?<br>
<br>
Thanks,<br>
Estani<br>
<br>
Am 03.03.2011 09:04, schrieb Estanislao Gonzalez:
<blockquote cite="mid:4D6F4B8A.4080509@zmaw.de" type="cite">
Hi Karl,<br>
<br>
I'm currently testing the replication via wget and got an average
of 350 Mbps (around 40MB/s) with 7 parallel streams in the last 14
hours. I've already downloaded 2TB.<br>
<br>
So I guess the problem is across the Atlantic. I'd suggest to
start many more parallel streams.<br>
<br>
In any case I think this might be caused by some
problems/congestion in intermediate networks. The network
throughput to ECMWF *might* not be comparable because of the
network architecture, e.g. intermediate proxies, firewalls, etc.
But a 10x speed up might be of no use anyway as downloading 1TB in
2 Months seems hardly to be an option...<br>
<br>
When we all have the data replicated worldwide though, it will be
possible to download from all 3 servers at the same time, which
will more than double the speed from this side of the ocean, and
will be a considerable speed up from yours, although probably not
as much (don't nail me on this, I'll deny it vehemently <- this
is not my writing :-)<br>
<br>
Anyway, this being said, the rest should be answered by BADC
directly.<br>
<br>
Thanks,<br>
Estani<br>
<br>
Am 03.03.2011 02:26, schrieb Karl Taylor:<br>
<span style="white-space: pre;">> this should have been sent to<br>
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a><br>
<br>
> <br>
<br>
> sorry about that.<br>
<br>
> <br>
<br>
> Karl<br>
<br>
> <br>
<br>
> -------- Original Message -------- Subject: download rates<br>
Date:<br>
<br>
> Wed, 2 Mar 2011 16:43:42 -0800 From: Karl Taylor<br>
<br>
> <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:taylor13@llnl.gov"><taylor13@llnl.gov></a>
To: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:esg-support@earthsystemgrid.org">esg-support@earthsystemgrid.org</a><br>
Grid<br>
<br>
> <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:esg-support@earthsystemgrid.org"><esg-support@earthsystemgrid.org></a><br>
<br>
> <br>
<br>
> <br>
<br>
> <br>
<br>
> Hi all,<br>
<br>
> <br>
<br>
> When I download a 25 MB CMIP5 file from BADC using the wget<br>
method: *<br>
<br>
> it took 35 sec to get started * then it downloaded at an<br>
average rate<br>
<br>
> of 1.1 MB/min<br>
<br>
> <br>
<br>
> When I download the same file using the point and click<br>
method: * it<br>
<br>
> took 20 sec to get started * then it downloaded at an
average<br>
rate of<br>
<br>
> 1.4 MB/min<br>
<br>
> <br>
<br>
> When I download 4 files simultaneously (in parallel) using<br>
the click<br>
<br>
> method, I get somewhat slower download rate per file, so<br>
somewhat<br>
<br>
> less than 4 times the data transfer rate I would have
gotten<br>
<br>
> downloading them in series.<br>
<br>
> <br>
<br>
> Note that at 1 MB/min, I could download 1 GB in about 17<br>
hours or 1<br>
<br>
> TB (i.e., about 1/1000 of the entire archive) in about 2<br>
years.<br>
<br>
> <br>
<br>
> Any way to speed this up some? (A colleague here downloads<br>
files at<br>
<br>
> about 10 times this rate from ECMWF.)<br>
<br>
> <br>
<br>
> Note that I haven't checked how fast download rates are
from<br>
nodes<br>
<br>
> published to our (PCMDI's) gateway (since they are not yet<br>
publicly<br>
<br>
> available).<br>
<br>
> <br>
<br>
> Best regards, Karl<br>
<br>
> <br>
<br>
> <br>
<br>
> <br>
<br>
> <br>
<br>
> _______________________________________________
GO-ESSP-TECH<br>
mailing<br>
<br>
> list <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<br>
<br>
> <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a></span><br>
<br>
<br>
-- <br>
Estanislao Gonzalez<br>
<br>
Max-Planck-Institut für Meteorologie (MPI-M)<br>
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing
Centre<br>
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany<br>
<br>
Phone: +49 (40) 46 00 94-126<br>
E-Mail: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:estanislao.gonzalez@zmaw.de">estanislao.gonzalez@zmaw.de</a>
<br>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:estanislao.gonzalez@zmaw.de">estanislao.gonzalez@zmaw.de</a> </pre>
</div>
_______________________________________________<br>GO-ESSP-TECH mailing list<br><a href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><br>http://mailman.ucar.edu/mailman/listinfo/go-essp-tech<br></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Rachana Ananthakrishnan</div><div>Argonne National Lab | University of Chicago</div></div></span></div></span></div></span></span>
</div>
<br></div></body></html>