[Go-essp-tech] download rates

[Karl Taylor]

Hi,

I've added my download rates to Estani's wiki.  I've also included the 
number of "streams" that were active.

Estani, please correct anything I've done wrong.

thanks,
Karl

On 3/3/11 3:13 AM, Estanislao Gonzalez wrote:
> Hi,
>
> by Michael suggestion I started a page in the esgf.org wiki were we 
> could write our measurement regarding CMIP5 federation.
> It is not intended for the technical staff, so don't complain about it 
> being too simplified :-)
> On the contrary, it is intended to be a very first and rough 
> approximation to what's happening in the cloud by allowing everyone in 
> 5 sec to write a measurement down.
>
> Everyone has write access to this and if we see the necessity we could 
> evolve it into something more detailed, but which will certainly leave 
> the lay users outside.
>
> And the link:
> http://esgf.org/wiki/Cmip5Status/Datarates
>
> Does someone else find this useful? Any comments?
>
> Thanks,
> Estani
>
> Am 03.03.2011 09:04, schrieb Estanislao Gonzalez:
>> Hi Karl,
>>
>> I'm currently testing the replication via wget and got an average of 
>> 350 Mbps (around 40MB/s) with 7 parallel streams in the last 14 
>> hours. I've already downloaded 2TB.
>>
>> So I guess the problem is across the Atlantic. I'd suggest to start 
>> many more parallel streams.
>>
>> In any case I think this might be caused by some problems/congestion 
>> in intermediate networks. The network throughput to ECMWF *might* not 
>> be comparable because of the network architecture, e.g. intermediate 
>> proxies, firewalls, etc. But a 10x speed up might be of no use anyway 
>> as downloading 1TB in 2 Months seems hardly to be an option...
>>
>> When we all have the data replicated worldwide though, it will be 
>> possible to download from all 3 servers at the same time, which will 
>> more than double the speed from this side of the ocean, and will be a 
>> considerable speed up from yours, although probably not as much 
>> (don't nail me on this, I'll deny it vehemently <- this is not my 
>> writing :-)
>>
>> Anyway, this being said, the rest should be answered by BADC directly.
>>
>> Thanks,
>> Estani
>>
>> Am 03.03.2011 02:26, schrieb Karl Taylor:
>> > this should have been sent to
>>
>> go-essp-tech at ucar.edu
>>
>>
>>
>> >
>>
>>
>>
>> > sorry about that.
>>
>>
>>
>> >
>>
>>
>>
>> > Karl
>>
>>
>>
>> >
>>
>>
>>
>> > -------- Original Message -------- Subject: download rates
>>
>> Date:
>>
>>
>>
>> > Wed, 2 Mar 2011 16:43:42 -0800 From: Karl Taylor
>>
>>
>>
>> > <taylor13 at llnl.gov>
>> To: esg-support at earthsystemgrid.org
>>
>> Grid
>>
>>
>>
>> > <esg-support at earthsystemgrid.org>
>>
>>
>>
>> >
>>
>>
>>
>> >
>>
>>
>>
>> >
>>
>>
>>
>> > Hi all,
>>
>>
>>
>> >
>>
>>
>>
>> > When I download a 25 MB CMIP5 file from BADC using the wget
>>
>> method: *
>>
>>
>>
>> > it took 35 sec to get started * then it downloaded at an
>>
>> average rate
>>
>>
>>
>> > of 1.1 MB/min
>>
>>
>>
>> >
>>
>>
>>
>> > When I download the same file using the point and click
>>
>> method: * it
>>
>>
>>
>> > took 20 sec to get started * then it downloaded at an
>> average
>>
>> rate of
>>
>>
>>
>> > 1.4 MB/min
>>
>>
>>
>> >
>>
>>
>>
>> > When I download 4 files simultaneously (in parallel) using
>>
>> the click
>>
>>
>>
>> > method, I get somewhat slower download rate per file, so
>>
>> somewhat
>>
>>
>>
>> > less than 4 times the data transfer rate I would have
>> gotten
>>
>>
>>
>> > downloading them in series.
>>
>>
>>
>> >
>>
>>
>>
>> > Note that at 1 MB/min, I could download 1 GB in about 17
>>
>> hours or 1
>>
>>
>>
>> > TB (i.e., about 1/1000 of the entire archive) in about 2
>>
>> years.
>>
>>
>>
>> >
>>
>>
>>
>> > Any way to speed this up some? (A colleague here downloads
>>
>> files at
>>
>>
>>
>> > about 10 times this rate from ECMWF.)
>>
>>
>>
>> >
>>
>>
>>
>> > Note that I haven't checked how fast download rates are
>> from
>>
>> nodes
>>
>>
>>
>> > published to our (PCMDI's) gateway (since they are not yet
>>
>> publicly
>>
>>
>>
>> > available).
>>
>>
>>
>> >
>>
>>
>>
>> > Best regards, Karl
>>
>>
>>
>> >
>>
>>
>>
>> >
>>
>>
>>
>> >
>>
>>
>>
>> >
>>
>>
>>
>> > _______________________________________________
>> GO-ESSP-TECH
>>
>> mailing
>>
>>
>>
>> > list GO-ESSP-TECH at ucar.edu
>>
>>
>>
>>
>> > http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>>
>> -- 
>> Estanislao Gonzalez
>>
>> Max-Planck-Institut für Meteorologie (MPI-M)
>> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
>> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>>
>> Phone:   +49 (40) 46 00 94-126
>> E-Mail: estanislao.gonzalez at zmaw.de
>
>
> -- 
> Estanislao Gonzalez
>
> Max-Planck-Institut für Meteorologie (MPI-M)
> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>
> Phone:   +49 (40) 46 00 94-126
> E-Mail:estanislao.gonzalez at zmaw.de  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110303/5c55977f/attachment-0001.html