[Go-essp-tech] download rates
Rachana Ananthakrishnan
ranantha at mcs.anl.gov
Thu Mar 3 09:52:11 MST 2011
Estani,
Are all these numbers using a HTTP based download?
Rachana
On Mar 3, 2011, at 5:13 AM, Estanislao Gonzalez wrote:
> Hi,
>
> by Michael suggestion I started a page in the esgf.org wiki were we could write our measurement regarding CMIP5 federation.
> It is not intended for the technical staff, so don't complain about it being too simplified :-)
> On the contrary, it is intended to be a very first and rough approximation to what's happening in the cloud by allowing everyone in 5 sec to write a measurement down.
>
> Everyone has write access to this and if we see the necessity we could evolve it into something more detailed, but which will certainly leave the lay users outside.
>
> And the link:
> http://esgf.org/wiki/Cmip5Status/Datarates
>
> Does someone else find this useful? Any comments?
>
> Thanks,
> Estani
>
> Am 03.03.2011 09:04, schrieb Estanislao Gonzalez:
>>
>> Hi Karl,
>>
>> I'm currently testing the replication via wget and got an average of 350 Mbps (around 40MB/s) with 7 parallel streams in the last 14 hours. I've already downloaded 2TB.
>>
>> So I guess the problem is across the Atlantic. I'd suggest to start many more parallel streams.
>>
>> In any case I think this might be caused by some problems/congestion in intermediate networks. The network throughput to ECMWF *might* not be comparable because of the network architecture, e.g. intermediate proxies, firewalls, etc. But a 10x speed up might be of no use anyway as downloading 1TB in 2 Months seems hardly to be an option...
>>
>> When we all have the data replicated worldwide though, it will be possible to download from all 3 servers at the same time, which will more than double the speed from this side of the ocean, and will be a considerable speed up from yours, although probably not as much (don't nail me on this, I'll deny it vehemently <- this is not my writing :-)
>>
>> Anyway, this being said, the rest should be answered by BADC directly.
>>
>> Thanks,
>> Estani
>>
>> Am 03.03.2011 02:26, schrieb Karl Taylor:
>> > this should have been sent to
>>
>> go-essp-tech at ucar.edu
>>
>>
>>
>> >
>>
>>
>>
>> > sorry about that.
>>
>>
>>
>> >
>>
>>
>>
>> > Karl
>>
>>
>>
>> >
>>
>>
>>
>> > -------- Original Message -------- Subject: download rates
>>
>> Date:
>>
>>
>>
>> > Wed, 2 Mar 2011 16:43:42 -0800 From: Karl Taylor
>>
>>
>>
>> > <taylor13 at llnl.gov>
>> To: esg-support at earthsystemgrid.org
>>
>> Grid
>>
>>
>>
>> > <esg-support at earthsystemgrid.org>
>>
>>
>>
>> >
>>
>>
>>
>> >
>>
>>
>>
>> >
>>
>>
>>
>> > Hi all,
>>
>>
>>
>> >
>>
>>
>>
>> > When I download a 25 MB CMIP5 file from BADC using the wget
>>
>> method: *
>>
>>
>>
>> > it took 35 sec to get started * then it downloaded at an
>>
>> average rate
>>
>>
>>
>> > of 1.1 MB/min
>>
>>
>>
>> >
>>
>>
>>
>> > When I download the same file using the point and click
>>
>> method: * it
>>
>>
>>
>> > took 20 sec to get started * then it downloaded at an
>> average
>>
>> rate of
>>
>>
>>
>> > 1.4 MB/min
>>
>>
>>
>> >
>>
>>
>>
>> > When I download 4 files simultaneously (in parallel) using
>>
>> the click
>>
>>
>>
>> > method, I get somewhat slower download rate per file, so
>>
>> somewhat
>>
>>
>>
>> > less than 4 times the data transfer rate I would have
>> gotten
>>
>>
>>
>> > downloading them in series.
>>
>>
>>
>> >
>>
>>
>>
>> > Note that at 1 MB/min, I could download 1 GB in about 17
>>
>> hours or 1
>>
>>
>>
>> > TB (i.e., about 1/1000 of the entire archive) in about 2
>>
>> years.
>>
>>
>>
>> >
>>
>>
>>
>> > Any way to speed this up some? (A colleague here downloads
>>
>> files at
>>
>>
>>
>> > about 10 times this rate from ECMWF.)
>>
>>
>>
>> >
>>
>>
>>
>> > Note that I haven't checked how fast download rates are
>> from
>>
>> nodes
>>
>>
>>
>> > published to our (PCMDI's) gateway (since they are not yet
>>
>> publicly
>>
>>
>>
>> > available).
>>
>>
>>
>> >
>>
>>
>>
>> > Best regards, Karl
>>
>>
>>
>> >
>>
>>
>>
>> >
>>
>>
>>
>> >
>>
>>
>>
>> >
>>
>>
>>
>> > _______________________________________________
>> GO-ESSP-TECH
>>
>> mailing
>>
>>
>>
>> > list GO-ESSP-TECH at ucar.edu
>>
>>
>>
>>
>> > http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>>
>> --
>> Estanislao Gonzalez
>>
>> Max-Planck-Institut für Meteorologie (MPI-M)
>> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
>> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>>
>> Phone: +49 (40) 46 00 94-126
>> E-Mail: estanislao.gonzalez at zmaw.de
>
>
> --
> Estanislao Gonzalez
>
> Max-Planck-Institut für Meteorologie (MPI-M)
> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>
> Phone: +49 (40) 46 00 94-126
> E-Mail: estanislao.gonzalez at zmaw.de
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
Rachana Ananthakrishnan
Argonne National Lab | University of Chicago
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110303/fab7872c/attachment.html
More information about the GO-ESSP-TECH
mailing list