[Go-essp-tech] wget data download script

Thu Jul 29 07:19:20 MDT 2010

Hi Phil,

Thank you - this is good timing as I am working on the wget script 
generation and MyProxyLogon WebStart integration over the next week.

Your script has a lot of good default settings that make for easier use 
which is important for our end users.  I'm hoping that the workflow 
including MyProxyLogon WebStart and script generation can be made as 
user-friendly as possible.  Much will rely on the location and naming of 
certificates.

One thing I'm looking at is asserting/testing the wget version.  Older 
wget versions have slightly different security related switches (eg. 
--sslcertfile vs. --certificate, etc.)  We've also had a number of 
support requests historically related to file size limits (<2Gb) in pre 
1.10 wget versions.  (This may not affect the AR5 dataset downloads, 
however.)

A typical download use case involves scripted download of a large set of 
files in a batch script.  Your script should be useful in support of this.

I'm also thinking about curl as a script client.  Have you done any 
testing with curl?

I'm focussed first on wget in the near-term as I believe it is more 
broadly used and is more familiar to our end users.  However, this is 
based more on anecdotal knowledge rather than hard facts.

Thanks again,

-Eric

philip.kershaw at stfc.ac.uk wrote:
> Hi Eric,
>
> Just picking up from this action from the last security telco:
>
>   
>> Action Items:
>> - Eric to work with BADC on standardized wget script to link off
>> Gateway
>>     
>
> I've created a data download wget script which wraps the security settings:
>
> http://proj.badc.rl.ac.uk/ndg/browser/TI12-security/trunk/esg_wget_script/esg-download.sh
>
> The basic invocation is:
>
> $ ./esg-download.sh <download URI> --certificate=./user.pem --private-key=./user.pem
>
> The certificate and key would usually be obtained from a MyProxy call via the Java WebStart application.  Many of the settings take defaults or are configurable via environment variables.
>
> I hope this helps.  It should be possible to adapt it for your needs for the Gateway scripts.
>
> Cheers,
> Phil
>
>   
>> -----Original Message-----
>> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
>> bounces at ucar.edu] On Behalf Of Rachana Ananthakrishnan
>> Sent: 19 July 2010 18:52
>> To: GO-ESSP
>> Subject: Re: [Go-essp-tech] Gateway Security Telecon
>>
>> Notes and action items from today's call.
>>
>> Rachana
>>
>> 1.  OpenID whitelisting
>>
>> - configurable in Gateway s/w 1.1.
>> - PKI trust root and Open ID trust root (endpoints and CAs)
>> - Trust root provisioning: http download from central server, MyProxy
>> get trust roots operations
>> - Format: PEM certificate files, Java keystore
>> - All sites should have this version installed by next week.
>>
>> Action Items:
>> - Rachana to document trust roots accepted across the federation
>> - Rachana to initiate discussion on automating trust root provisioning
>> from central store
>>
>> 2. wget scripts with SSL
>>
>> - On target for 1.2 release in August.
>> - NCAR working on the script
>> - BADC has sorted out versions and parameters questions
>> - Mutual authentication is required
>> - Token-less security will be released as part of 1.2, but tokens will
>> also be supported for some time. Currently discontinue by 1.4.
>> - curl might be an interesting option to support, in addition to wget.
>>
>> Action Items:
>> - Eric to work with BADC on standardized wget script to link off
>> Gateway
>> - Eric to develop a deprecation plan for token based security
>> - Eric to determine user requirements to see if curl should be
>> supported.
>>
>> 3. MyProxy Java Webstart
>>
>> - Required for 1.2 release
>> - Code and s/w available
>> - Need to sort of certificate for signing the executable. Should be a
>> widely trusted CA
>> - Gateway integration is outstanding
>> - Need to agree on provisioning directory and advertise across ESG
>>
>> Action Items:
>> - Rachana to ensure the webstart is signed
>> - Rachana to document provisioning directory location, and advertise
>> for other tools to leverage.
>> - Eric to integrate with Gateway
>>
>> 4. PCMDI Gateway to be updated to recent s/w version (Resolved)
>>
>> 5. ORP Whitelisting
>> - Has been implemented, and needs to be configured manually
>> - Code is in PCMDI repository, and OpenID has been added to Request
>> Scope to aid in metrics gathering
>> - With 1.2 release, update ORP release version to 1.2.
>>
>> Action Items:
>> - Gavin to move code to git repository
>> - Luca to create distribution, and provide latest release for use
>> - ? to create compatibility matrix of Gateway s/w and ORP s/w.
>>
>> 6. Attribute and Authorization Services:
>> - These services have been implemented and are part of the distribution
>> - Need to add whitelist to prevent open access to these services
>> - Policy should allow other Gateways, Attribute and AuthZ services,
>> Data Node Manager to query attribute service
>> - Attribute registration system in place, needs further testing
>>
>> Action Items
>> - Eric to augment these services with ability to authorize requests to
>> determine if user is in whitelist
>> - Stephen to test the attribute registration system as a end user from
>> BADC
>> - Rachana and Stephen to document use case scenarios for these
>> services, to feed into acceptance tests
>>
>> 7. PCMDI Gateway certificate
>> Action Items
>> - Bob will have a valid certificate installed for the Gateway
>>
>> 8. Publisher client
>> - Does not validate server's certificate
>>
>> Action Items:
>> - Bob and Gavin to fix the issue
>>
>> On Jul 16, 2010, at 10:58 AM, Rachana Ananthakrishnan wrote:
>>
>>     
>>> Hi,
>>>
>>> We plan to discuss open issues on the Gateway Security aspects on
>>> Monday July 19th at 9am CDT. Please see coordinates and agenda below:
>>>
>>> Dial-in: 1 (925) 424-8105
>>> Access code 305757#
>>>
>>> Agenda:
>>>
>>> 1. OpenID Whitelisting at Gateways
>>> 2. wget scripts
>>> 3. Consistent CA trust roots across the federation
>>> 4. MyProxy Java Web Start from Gateways
>>> 5. Authorization and attribute service security
>>> 6. Attribute service at PCMDI
>>>
>>> We'll have a separate call to discuss the data node security aspects.
>>>
>>> Given we have postponed this many times, it was decided we would hold
>>> a separate meeting on this. I'll send notes and summary to this list,
>>> and we can follow-up as needed.
>>>
>>> Rachana
>>>
>>> Rachana Ananthakrishnan
>>> Argonne National Lab | University of Chicago
>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>       
>> Rachana Ananthakrishnan
>> Argonne National Lab | University of Chicago
>>
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>     