[Go-essp-tech] download rates

Fri Mar 4 02:19:40 MST 2011

Hello,

Just a comment on Estani's comment about the problem being somewhere across the atlantic: we have done a lot of tests on transfer rates between BADC and PCMDI in the last year, and it appears to be something within our firewall which is not a problem with regional area transfers but becomes a problem on transatlantic transfers. Investigations are ongoing,

cheers,
Martin

________________________________
From: go-essp-tech-bounces at ucar.edu [go-essp-tech-bounces at ucar.edu] on behalf of Karl Taylor [taylor13 at llnl.gov]
Sent: 03 March 2011 16:50
To: Estanislao Gonzalez
Cc: go-essp-tech at ucar.edu
Subject: Re: [Go-essp-tech] download rates

Hi,

I've added my download rates to Estani's wiki.  I've also included the number of "streams" that were active.

Estani, please correct anything I've done wrong.

thanks,
Karl

On 3/3/11 3:13 AM, Estanislao Gonzalez wrote:
Hi,

by Michael suggestion I started a page in the esgf.org wiki were we could write our measurement regarding CMIP5 federation.
It is not intended for the technical staff, so don't complain about it being too simplified :-)
On the contrary, it is intended to be a very first and rough approximation to what's happening in the cloud by allowing everyone in 5 sec to write a measurement down.

Everyone has write access to this and if we see the necessity we could evolve it into something more detailed, but which will certainly leave the lay users outside.

And the link:
http://esgf.org/wiki/Cmip5Status/Datarates

Does someone else find this useful? Any comments?

Thanks,
Estani

Am 03.03.2011 09:04, schrieb Estanislao Gonzalez:
Hi Karl,

I'm currently testing the replication via wget and got an average of 350 Mbps (around 40MB/s) with 7 parallel streams in the last 14 hours. I've already downloaded 2TB.

So I guess the problem is across the Atlantic. I'd suggest to start many more parallel streams.

In any case I think this might be caused by some problems/congestion in intermediate networks. The network throughput to ECMWF *might* not be comparable because of the network architecture, e.g. intermediate proxies, firewalls, etc. But a 10x speed up might be of no use anyway as downloading 1TB in 2 Months seems hardly to be an option...

When we all have the data replicated worldwide though, it will be possible to download from all 3 servers at the same time, which will more than double the speed from this side of the ocean, and will be a considerable speed up from yours, although probably not as much (don't nail me on this, I'll deny it vehemently <- this is not my writing :-)

Anyway, this being said, the rest should be answered by BADC directly.

Thanks,
Estani

Am 03.03.2011 02:26, schrieb Karl Taylor:
> this should have been sent to

go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>

>

> sorry about that.

>

> Karl

>

> -------- Original Message -------- Subject: download rates

Date:

> Wed, 2 Mar 2011 16:43:42 -0800 From: Karl Taylor

> <taylor13 at llnl.gov><mailto:taylor13 at llnl.gov>
To: esg-support at earthsystemgrid.org<mailto:esg-support at earthsystemgrid.org>

Grid

> <esg-support at earthsystemgrid.org><mailto:esg-support at earthsystemgrid.org>

>

>

>

> Hi all,

>

> When I download a 25 MB CMIP5 file from BADC using the wget

method: *

> it took 35 sec to get started * then it downloaded at an

average rate

> of 1.1 MB/min

>

> When I download the same file using the point and click

method: * it

> took 20 sec to get started * then it downloaded at an
average

rate of

> 1.4 MB/min

>

> When I download 4 files simultaneously (in parallel) using

the click

> method, I get somewhat slower download rate per file, so

somewhat

> less than 4 times the data transfer rate I would have
gotten

> downloading them in series.

>

> Note that at 1 MB/min, I could download 1 GB in about 17

hours or 1

> TB (i.e., about 1/1000 of the entire archive) in about 2

years.

>

> Any way to speed this up some? (A colleague here downloads

files at

> about 10 times this rate from ECMWF.)

>

> Note that I haven't checked how fast download rates are
from

nodes

> published to our (PCMDI's) gateway (since they are not yet

publicly

> available).

>

> Best regards, Karl

>

>

>

>

> _______________________________________________
GO-ESSP-TECH

mailing

> list GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>

> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

--
Estanislao Gonzalez

Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany

Phone:   +49 (40) 46 00 94-126
E-Mail:  estanislao.gonzalez at zmaw.de<mailto:estanislao.gonzalez at zmaw.de>

--
Estanislao Gonzalez

Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany

Phone:   +49 (40) 46 00 94-126
E-Mail:  estanislao.gonzalez at zmaw.de<mailto:estanislao.gonzalez at zmaw.de>
-- 
Scanned by iCritical.