[Go-essp-tech] Wget data transfer rates

Mon Jul 4 23:32:19 MDT 2011

Hi Sébastien,

thanks for those numbers, it looks as though we do have a network issue at BADC.

In terms of per thread transfer rates, we get around 1MB/s from IPSL and CNRM -- which is close to what you get from us. For transfer rates from CCCMA, however, we only get around 0.1MB/s whereas you are actually get faster rates from Canada than from the UK.

I also have no clue how to deal with data nodes that don't support PKI -- I'll contact Bob about that and see if we can get any progress,

cheers,
Martin
________________________________________
From: Sébastien Denvil [sebastien.denvil at ipsl.jussieu.fr]
Sent: 04 July 2011 14:56
To: Juckes, Martin (STFC,RAL,RALSP)
Cc: Luca.Cinquini at jpl.nasa.gov; go-essp-tech at ucar.edu
Subject: Re: [Go-essp-tech] Data node authorization

  Hi Martin

On 01/07/2011 11:13, martin.juckes at stfc.ac.uk wrote:
> Hi Sebastien,
>
> How fast are your fast rates from CCCMA? I'm getting a few tens of GB/hour using up to 60 wget threads -- perhaps I should be using more (I got cautious about the numbers because at one point I had 40 threads finish in a short time and the machine was then frozen by 40 md5sum threads trying to run in parallel).

Indeed it seems that we don't have the same standards to characterize a
fast rate :-)
we used 8 threads over a 1 Gbps network. CCCMA : 32 MB/s ; HADGEM : 7
MB/s ; CNRM : 19 MB/s. The subsets was small at this stage (200 GB/per
model) so we need a larger case to derive robust numbers.

> Are you planning to do transfers from BCC using the tokenised scripts? I would prefer to use certificates, but they appear to support this at present,

We will use the PKI approach for every downloads. Because I have no clue
how I could generate a token in a programmatic way.

Will post other numbers when the larger case will be done.

Cheers.
Sébastien

> Cheers,
> Martin
>
>>> -----Original Message-----
>>> From: Sébastien Denvil [mailto:sebastien.denvil at ipsl.jussieu.fr]
>>> Sent: 01 July 2011 09:33
>>> To: Juckes, Martin (STFC,RAL,RALSP)
>>> Cc: Luca.Cinquini at jpl.nasa.gov; jamie.kettleborough at metoffice.gov.uk;
>>> go-essp-tech at ucar.edu
>>> Subject: Re: [Go-essp-tech] Data node authorization
>>>
>>>   Hi all,
>>>
>>> we are in the process to download a subset of the already published
>>> data
>>> to sustain analysis activity (using multiple wget threads).
>>>
>>> We did that already for BADC, CNRM and CCCMA. Unlike Martin, we had
>>> fast
>>> transfert rate from CCCMA (network mysteries).
>>>
>>> I will let you know any interesting findings.
>>>
>>> The list we plan to download a subset from:
>>>
>>> bcc-csm1-1
>>> CanCM4
>>> CanESM2
>>> CNRM-CM5
>>> GISS-E2-H
>>> GISS-E2-R
>>> HadGEM2-A
>>> HadGEM2-ES
>>> inmcm4
>>> bcc-csm1-1
>>> NorESM1-M
>>>
>>> Regards.
>>> Sébastien
>>>
>>> On 01/07/2011 09:55, martin.juckes at stfc.ac.uk wrote:
>>>> Hi Jamie,
>>>>
>>>> As Luca says, the plan is to move to the myproxy system. Like the
>>> wheels of justice, the wheels of ESGF grind exceedingly slow, but you
>>> can't take the analogy much further. Like you, I've started to look at
>>> data from other nodes. I've found that the myproxy system works for
>>> BADC, IPSL, CNRM and CCCMA nodes. For the last two, you will get a
>>> tokenised wget script if you go through the gateway (because they are
>>> published through the PCMDI gateway) but it you build your own wget
>>> scripts, you can use myproxy certificates. If you want data from
>>> CCCMA, I find that the transatlantic transfer is very slow and you may
>>> want to copy what I already have at BADC -- let me know if you do as
>>> this copy won't be available through the BADC gateway until QC L2 has
>>> been completed, which may be some way off.
>>>> Another issue is the publication of checksums, because there is a
>>> significant risk of data corruption when moving large volumes. BADC
>>> and IPSL have the checksums in the THREDDS catalogues, CNRM is in the
>>> process of re-publishing to achieve this. I'm going to get in touch
>>> with CCCMA today to encourage them to do the same,
>>>> Regards,
>>>> Martin
>>>>
>>>>>> -----Original Message-----
>>>>>> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
>>>>>> bounces at ucar.edu] On Behalf Of Cinquini, Luca (3880)
>>>>>> Sent: 30 June 2011 20:10
>>>>>> To: Kettleborough, Jamie
>>>>>> Cc: go-essp-tech at ucar.edu
>>>>>> Subject: Re: [Go-essp-tech] Data node authorization
>>>>>>
>>>>>> Hi Jamie,
>>>>>>  the established plan is to move all sites to the MyProxy (SAML-
>>>>>> based) authentication and authorization system, and to gradually
>>>>>> phased out the token
>>>>>> based system. I know JPL and BADC have already moved, and that
>>> PCMDI
>>>>>> is still on the token based system. AS for the timeline at each
>>> site,
>>>>>> the corresponding
>>>>>> administrators will have to chime in.
>>>>>> thanks, Luca
>>>>>>
>>>>>> On Jun 30, 2011, at 7:22 AM, Kettleborough, Jamie wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Earlier this week I was trying to get data from different data
>>>>>> nodes.
>>>>>>> There seemed to be two authorization methods in place - one based
>>> on
>>>>>>> MyProxy, the other based on a token in the HTTP query string.
>>>>>>>
>>>>>>> Is this the long term plan?
>>>>>>> If not then how soon will just one method be supported across all
>>>>>> nodes?
>>>>>>> If it is then I guess there will be follow up questions about how
>>> to
>>>>>>> handle both...
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Jamie
>>>>>>> _______________________________________________
>>>>>>> GO-ESSP-TECH mailing list
>>>>>>> GO-ESSP-TECH at ucar.edu
>>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>> _______________________________________________
>>>>>> GO-ESSP-TECH mailing list
>>>>>> GO-ESSP-TECH at ucar.edu
>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>
>>> --
>>> Sébastien Denvil
>>> IPSL, Pôle de modélisation du climat
>>> UPMC, Case 101, 4 place Jussieu,
>>> 75252 Paris Cedex 5
>>>
>>> Tour 45-55 2ème étage Bureau 209
>>> Tel: 33 1 44 27 21 10
>>> Fax: 33 1 44 27 39 02
>>>
--
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5

Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02

-- 
Scanned by iCritical.