<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Indeed it will Rachana.<br>
<br>
I'm just performing a test (or was, as it's not working anymore
after 3TB... because of some technical difficulties :-) to see how
good/bad this is. And it was not bad at all (while it worked).<br>
<br>
I had quite some problems with guc too, so I'm just trying different
possibilities out.<br>
<br>
And I see the SSL double redirection now, nice :-)<br>
<br>
I think Stephen is already at that anyway, there were some minimal
problems with the publication.<br>
<br>
And a single wget connection is now running at 12MB/s.<br>
<br>
Thanks,<br>
Estani<br>
<br>
Am 03.03.2011 18:26, schrieb Rachana Ananthakrishnan:
<blockquote
cite="mid:4D504901-4980-4472-87E9-1C09FCCBDA48@mcs.anl.gov"
type="cite"><br>
<div>
<div>On Mar 3, 2011, at 11:23 AM, Estanislao Gonzalez wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div bgcolor="#ffffff" text="#000000"> Hi Rachana,<br>
<br>
yes the the only data that can be downloaded at the moment
is the one published at BADC and is http only... <br>
well actually that I think it again, if the user sends a
certificate it means it got redirected to an SSL end point,
right? Is he then redirected back to the original or is the
connection being made via SSL? I'm not sure now...<br>
<br>
</div>
</blockquote>
<div><br>
</div>
Data is download via HTTP, the redirection to SSL is to verify
identity of the user with certificates. </div>
<div><br>
</div>
<div>I am reaching out to the various site administrators to
understand barriers for providing a GridFTP servers for this. It
appears to me that your experiment of replicating TBs of data
would benefit from a system that is built with such use cases as
focus. </div>
<div><br>
</div>
<div>Thanks,</div>
<div>Rachana</div>
<div><br>
</div>
<div>
<blockquote type="cite">
<div bgcolor="#ffffff" text="#000000"> Thanks,<br>
Estani<br>
<br>
Am 03.03.2011 17:52, schrieb Rachana Ananthakrishnan:
<blockquote
cite="mid:0153FEE9-2276-4047-A134-9FFCF7C6723F@mcs.anl.gov"
type="cite">Estani,
<div><br>
</div>
<div>Are all these numbers using a HTTP based download?</div>
<div><br>
</div>
<div>Rachana</div>
<div><br>
</div>
<div>
<div>
<div>On Mar 3, 2011, at 5:13 AM, Estanislao Gonzalez
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div bgcolor="#ffffff" text="#000000"> Hi,<br>
<br>
by Michael suggestion I started a page in the <a
moz-do-not-send="true" href="http://esgf.org/">esgf.org</a>
wiki were we could write our measurement regarding
CMIP5 federation.<br>
It is not intended for the technical staff, so
don't complain about it being too simplified :-)<br>
On the contrary, it is intended to be a very first
and rough approximation to what's happening in the
cloud by allowing everyone in 5 sec to write a
measurement down.<br>
<br>
Everyone has write access to this and if we see
the necessity we could evolve it into something
more detailed, but which will certainly leave the
lay users outside.<br>
<br>
And the link:<br>
<a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://esgf.org/wiki/Cmip5Status/Datarates">http://esgf.org/wiki/Cmip5Status/Datarates</a><br>
<br>
Does someone else find this useful? Any comments?<br>
<br>
Thanks,<br>
Estani<br>
<br>
Am 03.03.2011 09:04, schrieb Estanislao Gonzalez:
<blockquote cite="mid:4D6F4B8A.4080509@zmaw.de"
type="cite"> Hi Karl,<br>
<br>
I'm currently testing the replication via wget
and got an average of 350 Mbps (around 40MB/s)
with 7 parallel streams in the last 14 hours.
I've already downloaded 2TB.<br>
<br>
So I guess the problem is across the Atlantic.
I'd suggest to start many more parallel streams.<br>
<br>
In any case I think this might be caused by some
problems/congestion in intermediate networks.
The network throughput to ECMWF *might* not be
comparable because of the network architecture,
e.g. intermediate proxies, firewalls, etc. But a
10x speed up might be of no use anyway as
downloading 1TB in 2 Months seems hardly to be
an option...<br>
<br>
When we all have the data replicated worldwide
though, it will be possible to download from all
3 servers at the same time, which will more than
double the speed from this side of the ocean,
and will be a considerable speed up from yours,
although probably not as much (don't nail me on
this, I'll deny it vehemently <- this is not
my writing :-)<br>
<br>
Anyway, this being said, the rest should be
answered by BADC directly.<br>
<br>
Thanks,<br>
Estani<br>
<br>
Am 03.03.2011 02:26, schrieb Karl Taylor:<br>
<span style="white-space: pre;">> this should
have<br>
been sent to<br>
<br>
<br>
<br>
<a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> sorry about that.<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> Karl<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> -------- Original Message --------
Subject:<br>
download rates<br>
<br>
<br>
<br>
Date:<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> Wed, 2 Mar 2011 16:43:42 -0800 From: Karl
Taylor<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:taylor13@llnl.gov"><taylor13@llnl.gov></a><br>
<br>
To: <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:esg-support@earthsystemgrid.org">esg-support@earthsystemgrid.org</a><br>
<br>
<br>
<br>
Grid<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:esg-support@earthsystemgrid.org"><esg-support@earthsystemgrid.org></a><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> Hi all,<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> When I download a 25 MB CMIP5 file from
BADC<br>
using the wget<br>
<br>
<br>
<br>
method: *<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> it took 35 sec to get started * then it<br>
downloaded at an<br>
<br>
<br>
<br>
average rate<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> of 1.1 MB/min<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> When I download the same file using the
point and<br>
click<br>
<br>
<br>
<br>
method: * it<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> took 20 sec to get started * then it
downloaded<br>
at an<br>
<br>
average<br>
<br>
<br>
<br>
rate of<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> 1.4 MB/min<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> When I download 4 files simultaneously
(in<br>
parallel) using<br>
<br>
<br>
<br>
the click<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> method, I get somewhat slower download
rate per<br>
file, so<br>
<br>
<br>
<br>
somewhat<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> less than 4 times the data transfer rate
I would<br>
have<br>
<br>
gotten<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> downloading them in series.<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> Note that at 1 MB/min, I could download 1
GB in<br>
about 17<br>
<br>
<br>
<br>
hours or 1<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> TB (i.e., about 1/1000 of the entire
archive) in<br>
about 2<br>
<br>
<br>
<br>
years.<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> Any way to speed this up some? (A
colleague here<br>
downloads<br>
<br>
<br>
<br>
files at<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> about 10 times this rate from ECMWF.)<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> Note that I haven't checked how fast
download<br>
rates are<br>
<br>
from<br>
<br>
<br>
<br>
nodes<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> published to our (PCMDI's) gateway (since
they<br>
are not yet<br>
<br>
<br>
<br>
publicly<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> available).<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> Best regards, Karl<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
>
_______________________________________________<br>
<br>
GO-ESSP-TECH<br>
<br>
<br>
<br>
mailing<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> list <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
> <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a></span><br>
<br>
<br>
-- <br>
Estanislao Gonzalez<br>
<br>
Max-Planck-Institut für Meteorologie (MPI-M)<br>
Deutsches Klimarechenzentrum (DKRZ) - German
Climate Computing Centre<br>
Room 108 - Bundesstrasse 45a, D-20146 Hamburg,
Germany<br>
<br>
Phone: +49 (40) 46 00 94-126<br>
E-Mail: <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:estanislao.gonzalez@zmaw.de">estanislao.gonzalez@zmaw.de</a>
<br>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:estanislao.gonzalez@zmaw.de">estanislao.gonzalez@zmaw.de</a> </pre>
</div>
_______________________________________________<br>
GO-ESSP-TECH mailing list<br>
<a moz-do-not-send="true"
href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><br>
<a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a><br>
</blockquote>
</div>
<br>
<div> <span class="Apple-style-span"
style="border-collapse: separate; font-family:
Helvetica; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal;
line-height: normal; orphans: 2; text-indent: 0px;
text-transform: none; white-space: normal; widows:
2; word-spacing: 0px; font-size: medium;"><span
class="Apple-style-span" style="border-collapse:
separate; font-family: Helvetica; font-style:
normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal;
orphans: 2; text-indent: 0px; text-transform:
none; white-space: normal; widows: 2;
word-spacing: 0px; font-size: medium;">
<div style="word-wrap: break-word;"><span
class="Apple-style-span"
style="border-collapse: separate; font-family:
Helvetica; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing:
normal; line-height: normal; orphans: 2;
text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing:
0px; font-size: medium;">
<div style="word-wrap: break-word;"><span
class="Apple-style-span"
style="border-collapse: separate;
font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing:
normal; line-height: normal; orphans: 2;
text-indent: 0px; text-transform: none;
white-space: normal; widows: 2;
word-spacing: 0px;">
<div style="word-wrap: break-word;">
<div>Rachana Ananthakrishnan</div>
<div>Argonne National Lab | University
of Chicago</div>
</div>
</span></div>
</span></div>
</span></span> </div>
<br>
</div>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:estanislao.gonzalez@zmaw.de">estanislao.gonzalez@zmaw.de</a> </pre>
</div>
</blockquote>
</div>
<br>
<div>
<span class="Apple-style-span" style="border-collapse: separate;
color: rgb(0, 0, 0); font-family: Helvetica; font-style:
normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: 2;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px; font-size: medium;"><span
class="Apple-style-span" style="border-collapse: separate;
color: rgb(0, 0, 0); font-family: Helvetica; font-style:
normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: 2;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px; font-size: medium;">
<div style="word-wrap: break-word;"><span
class="Apple-style-span" style="border-collapse:
separate; color: rgb(0, 0, 0); font-family: Helvetica;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
font-size: medium;">
<div style="word-wrap: break-word;"><span
class="Apple-style-span" style="border-collapse:
separate; color: rgb(0, 0, 0); font-family:
Helvetica; font-size: medium; font-style: normal;
font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal;
orphans: 2; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;">
<div style="word-wrap: break-word;">
<div>Rachana Ananthakrishnan</div>
<div>Argonne National Lab | University of Chicago</div>
</div>
</span></div>
</span></div>
</span></span>
</div>
<br>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:estanislao.gonzalez@zmaw.de">estanislao.gonzalez@zmaw.de</a> </pre>
</body>
</html>