[Go-essp-tech] download rates
Estanislao Gonzalez
estanislao.gonzalez at zmaw.de
Thu Mar 3 01:04:26 MST 2011
Hi Karl,
I'm currently testing the replication via wget and got an average of 350
Mbps (around 40MB/s) with 7 parallel streams in the last 14 hours. I've
already downloaded 2TB.
So I guess the problem is across the Atlantic. I'd suggest to start many
more parallel streams.
In any case I think this might be caused by some problems/congestion in
intermediate networks. The network throughput to ECMWF *might* not be
comparable because of the network architecture, e.g. intermediate
proxies, firewalls, etc. But a 10x speed up might be of no use anyway as
downloading 1TB in 2 Months seems hardly to be an option...
When we all have the data replicated worldwide though, it will be
possible to download from all 3 servers at the same time, which will
more than double the speed from this side of the ocean, and will be a
considerable speed up from yours, although probably not as much (don't
nail me on this, I'll deny it vehemently <- this is not my writing :-)
Anyway, this being said, the rest should be answered by BADC directly.
Thanks,
Estani
Am 03.03.2011 02:26, schrieb Karl Taylor:
> this should have been sent to go-essp-tech at ucar.edu
>
> sorry about that.
>
> Karl
>
> -------- Original Message -------- Subject: download rates Date:
> Wed, 2 Mar 2011 16:43:42 -0800 From: Karl Taylor
> <taylor13 at llnl.gov> To: esg-support at earthsystemgrid.org Grid
> <esg-support at earthsystemgrid.org>
>
>
>
> Hi all,
>
> When I download a 25 MB CMIP5 file from BADC using the wget method: *
> it took 35 sec to get started * then it downloaded at an average rate
> of 1.1 MB/min
>
> When I download the same file using the point and click method: * it
> took 20 sec to get started * then it downloaded at an average rate of
> 1.4 MB/min
>
> When I download 4 files simultaneously (in parallel) using the click
> method, I get somewhat slower download rate per file, so somewhat
> less than 4 times the data transfer rate I would have gotten
> downloading them in series.
>
> Note that at 1 MB/min, I could download 1 GB in about 17 hours or 1
> TB (i.e., about 1/1000 of the entire archive) in about 2 years.
>
> Any way to speed this up some? (A colleague here downloads files at
> about 10 times this rate from ECMWF.)
>
> Note that I haven't checked how fast download rates are from nodes
> published to our (PCMDI's) gateway (since they are not yet publicly
> available).
>
> Best regards, Karl
>
>
>
>
> _______________________________________________ GO-ESSP-TECH mailing
> list GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: estanislao.gonzalez at zmaw.de
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110303/e8bf757b/attachment.html
More information about the GO-ESSP-TECH
mailing list