[Go-essp-tech] download rates

[Ann Chervenak]

If the performance Karl is seeing is caused by a trans-Atlantic 
bottleneck, then this may be one of the cases we discussed in the past 
where the data need to be shipped rather than transferred over the network.

Ann


On 3/3/11 12:04 AM, Estanislao Gonzalez wrote:
> Hi Karl,
>
> I'm currently testing the replication via wget and got an average of 
> 350 Mbps (around 40MB/s) with 7 parallel streams in the last 14 hours. 
> I've already downloaded 2TB.
>
> So I guess the problem is across the Atlantic. I'd suggest to start 
> many more parallel streams.
>
> In any case I think this might be caused by some problems/congestion 
> in intermediate networks. The network throughput to ECMWF *might* not 
> be comparable because of the network architecture, e.g. intermediate 
> proxies, firewalls, etc. But a 10x speed up might be of no use anyway 
> as downloading 1TB in 2 Months seems hardly to be an option...
>
> When we all have the data replicated worldwide though, it will be 
> possible to download from all 3 servers at the same time, which will 
> more than double the speed from this side of the ocean, and will be a 
> considerable speed up from yours, although probably not as much (don't 
> nail me on this, I'll deny it vehemently <- this is not my writing :-)
>
> Anyway, this being said, the rest should be answered by BADC directly.
>
> Thanks,
> Estani
>
> Am 03.03.2011 02:26, schrieb Karl Taylor:
> > this should have been sent to
> go-essp-tech at ucar.edu
>
> >
>
> > sorry about that.
>
> >
>
> > Karl
>
> >
>
> > -------- Original Message -------- Subject: download rates
> Date:
>
> > Wed, 2 Mar 2011 16:43:42 -0800 From: Karl Taylor
>
> > <taylor13 at llnl.gov> To: esg-support at earthsystemgrid.org
> Grid
>
> > <esg-support at earthsystemgrid.org>
>
> >
>
> >
>
> >
>
> > Hi all,
>
> >
>
> > When I download a 25 MB CMIP5 file from BADC using the wget
> method: *
>
> > it took 35 sec to get started * then it downloaded at an
> average rate
>
> > of 1.1 MB/min
>
> >
>
> > When I download the same file using the point and click
> method: * it
>
> > took 20 sec to get started * then it downloaded at an average
> rate of
>
> > 1.4 MB/min
>
> >
>
> > When I download 4 files simultaneously (in parallel) using
> the click
>
> > method, I get somewhat slower download rate per file, so
> somewhat
>
> > less than 4 times the data transfer rate I would have gotten
>
> > downloading them in series.
>
> >
>
> > Note that at 1 MB/min, I could download 1 GB in about 17
> hours or 1
>
> > TB (i.e., about 1/1000 of the entire archive) in about 2
> years.
>
> >
>
> > Any way to speed this up some? (A colleague here downloads
> files at
>
> > about 10 times this rate from ECMWF.)
>
> >
>
> > Note that I haven't checked how fast download rates are from
> nodes
>
> > published to our (PCMDI's) gateway (since they are not yet
> publicly
>
> > available).
>
> >
>
> > Best regards, Karl
>
> >
>
> >
>
> >
>
> >
>
> > _______________________________________________ GO-ESSP-TECH
> mailing
>
> > list GO-ESSP-TECH at ucar.edu
>
> > http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
>
> -- 
> Estanislao Gonzalez
>
> Max-Planck-Institut für Meteorologie (MPI-M)
> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>
> Phone:   +49 (40) 46 00 94-126
> E-Mail: estanislao.gonzalez at zmaw.de
>
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>    
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110303/430df838/attachment.html