[Go-essp-tech] download rates

[Estanislao Gonzalez]

Hi Karl,

I'm currently testing the replication via wget and got an average of 350 
Mbps (around 40MB/s) with 7 parallel streams in the last 14 hours. I've 
already downloaded 2TB.

So I guess the problem is across the Atlantic. I'd suggest to start many 
more parallel streams.

In any case I think this might be caused by some problems/congestion in 
intermediate networks. The network throughput to ECMWF *might* not be 
comparable because of the network architecture, e.g. intermediate 
proxies, firewalls, etc. But a 10x speed up might be of no use anyway as 
downloading 1TB in 2 Months seems hardly to be an option...

When we all have the data replicated worldwide though, it will be 
possible to download from all 3 servers at the same time, which will 
more than double the speed from this side of the ocean, and will be a 
considerable speed up from yours, although probably not as much (don't 
nail me on this, I'll deny it vehemently <- this is not my writing :-)

Anyway, this being said, the rest should be answered by BADC directly.

Thanks,
Estani

Am 03.03.2011 02:26, schrieb Karl Taylor:
>  this should have been sent to go-essp-tech at ucar.edu
>
>  sorry about that.
>
>  Karl
>
>  -------- Original Message -------- Subject: download rates Date:
>  Wed, 2 Mar 2011 16:43:42 -0800 From: Karl Taylor
>  <taylor13 at llnl.gov> To: esg-support at earthsystemgrid.org Grid
>  <esg-support at earthsystemgrid.org>
>
>
>
>  Hi all,
>
>  When I download a 25 MB CMIP5 file from BADC using the wget method: *
>  it took 35 sec to get started * then it downloaded at an average rate
>  of 1.1 MB/min
>
>  When I download the same file using the point and click method: * it
>  took 20 sec to get started * then it downloaded at an average rate of
>  1.4 MB/min
>
>  When I download 4 files simultaneously (in parallel) using the click
>  method, I get somewhat slower download rate per file, so somewhat
>  less than 4 times the data transfer rate I would have gotten
>  downloading them in series.
>
>  Note that at 1 MB/min, I could download 1 GB in about 17 hours or 1
>  TB (i.e., about 1/1000 of the entire archive) in about 2 years.
>
>  Any way to speed this up some? (A colleague here downloads files at
>  about 10 times this rate from ECMWF.)
>
>  Note that I haven't checked how fast download rates are from nodes
>  published to our (PCMDI's) gateway (since they are not yet publicly
>  available).
>
>  Best regards, Karl
>
>
>
>
>  _______________________________________________ GO-ESSP-TECH mailing
>  list GO-ESSP-TECH at ucar.edu
>  http://mailman.ucar.edu/mailman/listinfo/go-essp-tech


-- 
Estanislao Gonzalez

Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany

Phone:   +49 (40) 46 00 94-126
E-Mail:  estanislao.gonzalez at zmaw.de
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110303/e8bf757b/attachment.html