[Go-essp-tech] download rates

[Rachana Ananthakrishnan]

Estani,

Are all these numbers using a HTTP based download?

Rachana

On Mar 3, 2011, at 5:13 AM, Estanislao Gonzalez wrote:

> Hi,
> 
> by Michael suggestion I started a page in the esgf.org wiki were we could write our measurement regarding CMIP5 federation.
> It is not intended for the technical staff, so don't complain about it being too simplified :-)
> On the contrary, it is intended to be a very first and rough approximation to what's happening in the cloud by allowing everyone in 5 sec to write a measurement down.
> 
> Everyone has write access to this and if we see the necessity we could evolve it into something more detailed, but which will certainly leave the lay users outside.
> 
> And the link:
> http://esgf.org/wiki/Cmip5Status/Datarates
> 
> Does someone else find this useful? Any comments?
> 
> Thanks,
> Estani
> 
> Am 03.03.2011 09:04, schrieb Estanislao Gonzalez:
>> 
>> Hi Karl,
>> 
>> I'm currently testing the replication via wget and got an average of 350 Mbps (around 40MB/s) with 7 parallel streams in the last 14 hours. I've already downloaded 2TB.
>> 
>> So I guess the problem is across the Atlantic. I'd suggest to start many more parallel streams.
>> 
>> In any case I think this might be caused by some problems/congestion in intermediate networks. The network throughput to ECMWF *might* not be comparable because of the network architecture, e.g. intermediate proxies, firewalls, etc. But a 10x speed up might be of no use anyway as downloading 1TB in 2 Months seems hardly to be an option...
>> 
>> When we all have the data replicated worldwide though, it will be possible to download from all 3 servers at the same time, which will more than double the speed from this side of the ocean, and will be a considerable speed up from yours, although probably not as much (don't nail me on this, I'll deny it vehemently <- this is not my writing :-)
>> 
>> Anyway, this being said, the rest should be answered by BADC directly.
>> 
>> Thanks,
>> Estani
>> 
>> Am 03.03.2011 02:26, schrieb Karl Taylor:
>> > this should have been sent to
>> 
>>         go-essp-tech at ucar.edu
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > sorry about that.
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > Karl
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > -------- Original Message -------- Subject: download rates
>> 
>>         Date:
>> 
>>         
>> 
>>         > Wed, 2 Mar 2011 16:43:42 -0800 From: Karl Taylor
>> 
>>         
>> 
>>         > <taylor13 at llnl.gov>
>>         To: esg-support at earthsystemgrid.org
>> 
>>         Grid
>> 
>>         
>> 
>>         > <esg-support at earthsystemgrid.org>
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > Hi all,
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > When I download a 25 MB CMIP5 file from BADC using the wget
>> 
>>         method: *
>> 
>>         
>> 
>>         > it took 35 sec to get started * then it downloaded at an
>> 
>>         average rate
>> 
>>         
>> 
>>         > of 1.1 MB/min
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > When I download the same file using the point and click
>> 
>>         method: * it
>> 
>>         
>> 
>>         > took 20 sec to get started * then it downloaded at an
>>         average
>> 
>>         rate of
>> 
>>         
>> 
>>         > 1.4 MB/min
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > When I download 4 files simultaneously (in parallel) using
>> 
>>         the click
>> 
>>         
>> 
>>         > method, I get somewhat slower download rate per file, so
>> 
>>         somewhat
>> 
>>         
>> 
>>         > less than 4 times the data transfer rate I would have
>>         gotten
>> 
>>         
>> 
>>         > downloading them in series.
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > Note that at 1 MB/min, I could download 1 GB in about 17
>> 
>>         hours or 1
>> 
>>         
>> 
>>         > TB (i.e., about 1/1000 of the entire archive) in about 2
>> 
>>         years.
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > Any way to speed this up some? (A colleague here downloads
>> 
>>         files at
>> 
>>         
>> 
>>         > about 10 times this rate from ECMWF.)
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > Note that I haven't checked how fast download rates are
>>         from
>> 
>>         nodes
>> 
>>         
>> 
>>         > published to our (PCMDI's) gateway (since they are not yet
>> 
>>         publicly
>> 
>>         
>> 
>>         > available).
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > Best regards, Karl
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > 
>> 
>>         
>> 
>>         > _______________________________________________
>>         GO-ESSP-TECH
>> 
>>         mailing
>> 
>>         
>> 
>>         > list GO-ESSP-TECH at ucar.edu
>>         
>> 
>>         
>> 
>>         > http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>> 
>> 
>> -- 
>> Estanislao Gonzalez
>> 
>> Max-Planck-Institut für Meteorologie (MPI-M)
>> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
>> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>> 
>> Phone:   +49 (40) 46 00 94-126
>> E-Mail:  estanislao.gonzalez at zmaw.de 
> 
> 
> -- 
> Estanislao Gonzalez
> 
> Max-Planck-Institut für Meteorologie (MPI-M)
> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
> 
> Phone:   +49 (40) 46 00 94-126
> E-Mail:  estanislao.gonzalez at zmaw.de 
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

Rachana Ananthakrishnan
Argonne National Lab | University of Chicago

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110303/fab7872c/attachment.html