[Go-essp-tech] Data node authorization

[martin.juckes at stfc.ac.uk]

Hi Sebastien, 

How fast are your fast rates from CCCMA? I'm getting a few tens of GB/hour using up to 60 wget threads -- perhaps I should be using more (I got cautious about the numbers because at one point I had 40 threads finish in a short time and the machine was then frozen by 40 md5sum threads trying to run in parallel). 

Are you planning to do transfers from BCC using the tokenised scripts? I would prefer to use certificates, but they appear to support this at present,

Cheers,
Martin

> >-----Original Message-----
> >From: Sébastien Denvil [mailto:sebastien.denvil at ipsl.jussieu.fr]
> >Sent: 01 July 2011 09:33
> >To: Juckes, Martin (STFC,RAL,RALSP)
> >Cc: Luca.Cinquini at jpl.nasa.gov; jamie.kettleborough at metoffice.gov.uk;
> >go-essp-tech at ucar.edu
> >Subject: Re: [Go-essp-tech] Data node authorization
> >
> >  Hi all,
> >
> >we are in the process to download a subset of the already published
> >data
> >to sustain analysis activity (using multiple wget threads).
> >
> >We did that already for BADC, CNRM and CCCMA. Unlike Martin, we had
> >fast
> >transfert rate from CCCMA (network mysteries).
> >
> >I will let you know any interesting findings.
> >
> >The list we plan to download a subset from:
> >
> >bcc-csm1-1
> >CanCM4
> >CanESM2
> >CNRM-CM5
> >GISS-E2-H
> >GISS-E2-R
> >HadGEM2-A
> >HadGEM2-ES
> >inmcm4
> >bcc-csm1-1
> >NorESM1-M
> >
> >Regards.
> >Sébastien
> >
> >On 01/07/2011 09:55, martin.juckes at stfc.ac.uk wrote:
> >> Hi Jamie,
> >>
> >> As Luca says, the plan is to move to the myproxy system. Like the
> >wheels of justice, the wheels of ESGF grind exceedingly slow, but you
> >can't take the analogy much further. Like you, I've started to look at
> >data from other nodes. I've found that the myproxy system works for
> >BADC, IPSL, CNRM and CCCMA nodes. For the last two, you will get a
> >tokenised wget script if you go through the gateway (because they are
> >published through the PCMDI gateway) but it you build your own wget
> >scripts, you can use myproxy certificates. If you want data from
> >CCCMA, I find that the transatlantic transfer is very slow and you may
> >want to copy what I already have at BADC -- let me know if you do as
> >this copy won't be available through the BADC gateway until QC L2 has
> >been completed, which may be some way off.
> >>
> >> Another issue is the publication of checksums, because there is a
> >significant risk of data corruption when moving large volumes. BADC
> >and IPSL have the checksums in the THREDDS catalogues, CNRM is in the
> >process of re-publishing to achieve this. I'm going to get in touch
> >with CCCMA today to encourage them to do the same,
> >>
> >> Regards,
> >> Martin
> >>
> >>>> -----Original Message-----
> >>>> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
> >>>> bounces at ucar.edu] On Behalf Of Cinquini, Luca (3880)
> >>>> Sent: 30 June 2011 20:10
> >>>> To: Kettleborough, Jamie
> >>>> Cc: go-essp-tech at ucar.edu
> >>>> Subject: Re: [Go-essp-tech] Data node authorization
> >>>>
> >>>> Hi Jamie,
> >>>> 	the established plan is to move all sites to the MyProxy (SAML-
> >>>> based) authentication and authorization system, and to gradually
> >>>> phased out the token
> >>>> based system. I know JPL and BADC have already moved, and that
> >PCMDI
> >>>> is still on the token based system. AS for the timeline at each
> >site,
> >>>> the corresponding
> >>>> administrators will have to chime in.
> >>>> thanks, Luca
> >>>>
> >>>> On Jun 30, 2011, at 7:22 AM, Kettleborough, Jamie wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> Earlier this week I was trying to get data from different data
> >>>> nodes.
> >>>>> There seemed to be two authorization methods in place - one based
> >on
> >>>>> MyProxy, the other based on a token in the HTTP query string.
> >>>>>
> >>>>> Is this the long term plan?
> >>>>> If not then how soon will just one method be supported across all
> >>>> nodes?
> >>>>> If it is then I guess there will be follow up questions about how
> >to
> >>>>> handle both...
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Jamie
> >>>>> _______________________________________________
> >>>>> GO-ESSP-TECH mailing list
> >>>>> GO-ESSP-TECH at ucar.edu
> >>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >>>> _______________________________________________
> >>>> GO-ESSP-TECH mailing list
> >>>> GO-ESSP-TECH at ucar.edu
> >>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >
> >
> >--
> >Sébastien Denvil
> >IPSL, Pôle de modélisation du climat
> >UPMC, Case 101, 4 place Jussieu,
> >75252 Paris Cedex 5
> >
> >Tour 45-55 2ème étage Bureau 209
> >Tel: 33 1 44 27 21 10
> >Fax: 33 1 44 27 39 02
> >

-- 
Scanned by iCritical.