[Go-essp-tech] Update Nodes & Gateways Web Page

Estanislao Gonzalez gonzalez at dkrz.de
Thu Jul 14 03:27:28 MDT 2011 

Hi Eric,

I've sent the script again to gateway-dev. We now have a contrib repo in 
esgf for "miscellaneous" (And the applause goes to Gavin :-) Thank you 
very much for this) where I added that file too (under estani/gateway). 
The repo is in esgf.org/esgf-contrib.git

I think we should move towards a script that provides both 
functionalities. Or, better yet, let the gateway store the configured 
access for every data node that published data to it and use this to set 
up the script (i.e. add the token to the urls from datanodes that have 
no PKI in place). The gateway can collect this data "on demand", i.e. 
when the data node redirect to one of the attribute service. This could 
also be in the registry, but it will take longer to get there and it 
currently makes no sense, since those data nodes won't be upgraded to a 
node-manager version that provides such functionality. Of course the 
"fall back" solution should be token based, as it is now.

It does sound "simple" enough to me, but I might have missed something...

Thanks,
Estani


Am 13.07.2011 19:10, schrieb Eric Nienhouse:
> Hi Estani, All,
>
> Estanislao Gonzalez wrote:
>> Hi Sebastien,
>>
>> I still can't understand why there are data nodes not accepting PKI 
>> access. I've asked around but never got any answer.
> Certain data nodes do not have the "tokenless" (ie. PKI certificate) 
> access configured yet.  It may simply be a matter of time for these 
> data nodes to update.  Long ago we all planned to remove token support 
> once we had consensus amongst the node data managers to do so.
>
> Now is a good time to revive this discussion IMO.
>>
>> What we can do is mark the data node we know has no PKI in place in 
>> the list as such. Non marked one are supposed to be working.
> Sounds like a good idea to me.
>>
>> Furthermore I'm not sure if it's really the data node not accepting 
>> PKI access or the gateways not configured to provide them while 
>> generating the wget script.
> Gateways may be configured to generate WGet scripts with tokens per 
> data node service via the Admin UI.  This capability is in place to 
> allow migration of nodes off the token based auth-z.
>
> As you know these are short lived tokens and only work for the 
> originating gateway's nodes.  Though they are simple to use, they are 
> limited in capability and will eventually be phased out.
>
> Some gateways are still configured to generate tokens for downloads 
> (wget and direct gateway UI download link access).  NCAR is currently 
> configured in this way even though the NCAR data node also accepts PKI 
> certs.  We chose to do this to support our current users who are 
> familiar with token based access and are primarily accessing CMIP3 
> datasets.  We're expecting CMIP5 data publication soon, so we're 
> reviewing when it would be best to convert fully to PKI based access.  
> We'll update the CMIP5 wiki page when we do :-)
>>
>> By the way, I'm still not sure which version is around... The latest 
>> I've built used the checksums to assure the files were downloaded 
>> correctly, are all Gateways using that one?
> I don't think all gateways are using this latest script.  Could you 
> provide it?  (Especially in the FreeMarker template format it you have 
> it?)  Thank you!
>>
>> Now regarding http://esgf.org/wiki/Cmip5Status/ArchiveView I've 
>> created that page just recently (glad you've found it out ;-). It's a 
>> prototype so I might have to tune it a little bit.
>> The main idea is to harvest known Gateways/top level collections for 
>> new datasets and building that summary you see (plus the DB we have).
>> Sadly the Gateway API prior to 1.3.1 together with the non 
>> conformance to DRS structures make it "unstable" so there are some 
>> data nodes that I just can't get information from.
>>
>> I'll try to keep evolving it as time permits.
>>
>> Thanks,
>> Estani
>>
>>
>> Am 13.07.2011 16:32, schrieb Sébastien Denvil:
>>> Dear all,
>>>
>>> from this page http://esgf.org/wiki/Cmip5Status the 
>>> http://esgf.org/wiki/Cmip5Status/ArchiveView page is very useful. 
>>> Thanks.
>>>
>>> I'm sure it's in the pipeline and I mention it in case it's not:
>>> - it would be great if we can have for each datanode a PKI access 
>>> (yes/no) flag we could refer to.
>>>
>>> All the best.
>>> Sébastien
>>>
>>> On 12/07/2011 21:35, Williams, Dean N. wrote:
>>>> Thanks Bob, Eric, and everyone else for updating the information on 
>>>> the website. We will have network connectivity listed on this site 
>>>> soon.
>>>>
>>>> Best regards,
>>>>     Dean
>>>>
>>>>
>>>> Bob Drach On 7/12/11 11:55 AM, "Bob Drach" <drach1 at llnl.gov> wrote:
>>>>
>>>>     OK, done for the PCMDI nodes. --Bob
>>>>
>>>>
>>>>     On 7/12/11 6:45 AM, "Williams, Dean N." <williams13 at llnl.gov> 
>>>> wrote:
>>>>
>>>>         Dear Administrators,
>>>>
>>>>             Please update the Nodes & Gateways web page found at:
>>>>         _http://esgf.org/wiki/Cmip5Status_ . For example, if you
>>>>         have updated your test gateway to 1.3.0, please indicate it.
>>>>
>>>>
>>>>         Thanks,
>>>>             Dean
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> GO-ESSP-TECH mailing list
>>>> GO-ESSP-TECH at ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>
>>>
>>> -- 
>>> Sébastien Denvil
>>> IPSL, Pôle de modélisation du climat
>>> UPMC, Case 101, 4 place Jussieu,
>>> 75252 Paris Cedex 5
>>>
>>> Tour 45-55 2ème étage Bureau 209
>>> Tel: 33 1 44 27 21 10
>>> Fax: 33 1 44 27 39 02
>>>
>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>>
>> -- 
>> Estanislao Gonzalez
>>
>> Max-Planck-Institut für Meteorologie (MPI-M)
>> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
>> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>>
>> Phone:   +49 (40) 46 00 94-126
>> E-Mail:  gonzalez at dkrz.de 
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>


-- 
Estanislao Gonzalez

Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany

Phone:   +49 (40) 46 00 94-126
E-Mail:  gonzalez at dkrz.de

More information about the GO-ESSP-TECH mailing list