[Go-essp-tech] wget data download script

Fri Jul 30 01:27:11 MDT 2010

Hi Eric,

> Your script has a lot of good default settings that make for easier use
> which is important for our end users.  I'm hoping that the workflow
> including MyProxyLogon WebStart and script generation can be made as
> user-friendly as possible.  Much will rely on the location and naming
> of
> certificates.

[Kershaw, Philip (STFC,RAL,SSTD)] That's the critical issue :)

It would be ideal if the WebStart app set a default location for the user certificate and trust roots which the wget scripts likewise pick up as their default.  Rachana suggested a separate ESG specific location for trust roots to differentiate from certificates a user might user with other projects.  I've used: $HOME/.globus/certificates/esg

> One thing I'm looking at is asserting/testing the wget version.  Older
> wget versions have slightly different security related switches (eg.
> --sslcertfile vs. --certificate, etc.)  
[Kershaw, Philip (STFC,RAL,SSTD)] Yes, it sounds reasonable to restrict.

> A typical download use case involves scripted download of a large set
> of
> files in a batch script.  Your script should be useful in support of
> this.
[Kershaw, Philip (STFC,RAL,SSTD)] There's also an altered version in the same directory which supports recursive download but I'm not sure TDS supports this or if it fits with your use case.

http://proj.badc.rl.ac.uk/ndg/browser/TI12-security/trunk/esg_wget_script/esg-recursive-download.sh

> 
> I'm also thinking about curl as a script client.  Have you done any
> testing with curl?

[Kershaw, Philip (STFC,RAL,SSTD)] Yes. It works with curl too.  I used curl for another project where we needed support for partial downloads using HTTP content ranges in requests.  Wget can't do this.  On the other hand I don't think curl has a recursive download option.  The same concerns as with wget apply over varying syntax for different versions.

> I'm focussed first on wget in the near-term as I believe it is more
> broadly used and is more familiar to our end users.  However, this is
> based more on anecdotal knowledge rather than hard facts.

[Kershaw, Philip (STFC,RAL,SSTD)] It's my impression too.

Cheers,
Phil
> 
> philip.kershaw at stfc.ac.uk wrote:
> > Hi Eric,
> >
> > Just picking up from this action from the last security telco:
> >
> >
> >> Action Items:
> >> - Eric to work with BADC on standardized wget script to link off
> >> Gateway
> >>
> >
> > I've created a data download wget script which wraps the security
> settings:
> >
> > http://proj.badc.rl.ac.uk/ndg/browser/TI12-
> security/trunk/esg_wget_script/esg-download.sh
> >
> > The basic invocation is:
> >
> > $ ./esg-download.sh <download URI> --certificate=./user.pem --
> private-key=./user.pem
> >
> > The certificate and key would usually be obtained from a MyProxy call
> via the Java WebStart application.  Many of the settings take defaults
> or are configurable via environment variables.
> >
> > I hope this helps.  It should be possible to adapt it for your needs
> for the Gateway scripts.
> >
> > Cheers,
> > Phil
> >
> >
> >> -----Original Message-----
> >> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
> >> bounces at ucar.edu] On Behalf Of Rachana Ananthakrishnan
> >> Sent: 19 July 2010 18:52
> >> To: GO-ESSP
> >> Subject: Re: [Go-essp-tech] Gateway Security Telecon
> >>
> >> Notes and action items from today's call.
> >>
> >> Rachana
> >>
> >> 1.  OpenID whitelisting
> >>
> >> - configurable in Gateway s/w 1.1.
> >> - PKI trust root and Open ID trust root (endpoints and CAs)
> >> - Trust root provisioning: http download from central server,
> MyProxy
> >> get trust roots operations
> >> - Format: PEM certificate files, Java keystore
> >> - All sites should have this version installed by next week.
> >>
> >> Action Items:
> >> - Rachana to document trust roots accepted across the federation
> >> - Rachana to initiate discussion on automating trust root
> provisioning
> >> from central store
> >>
> >> 2. wget scripts with SSL
> >>
> >> - On target for 1.2 release in August.
> >> - NCAR working on the script
> >> - BADC has sorted out versions and parameters questions
> >> - Mutual authentication is required
> >> - Token-less security will be released as part of 1.2, but tokens
> will
> >> also be supported for some time. Currently discontinue by 1.4.
> >> - curl might be an interesting option to support, in addition to
> wget.
> >>
> >> Action Items:
> >> - Eric to work with BADC on standardized wget script to link off
> >> Gateway
> >> - Eric to develop a deprecation plan for token based security
> >> - Eric to determine user requirements to see if curl should be
> >> supported.
> >>
> >> 3. MyProxy Java Webstart
> >>
> >> - Required for 1.2 release
> >> - Code and s/w available
> >> - Need to sort of certificate for signing the executable. Should be
> a
> >> widely trusted CA
> >> - Gateway integration is outstanding
> >> - Need to agree on provisioning directory and advertise across ESG
> >>
> >> Action Items:
> >> - Rachana to ensure the webstart is signed
> >> - Rachana to document provisioning directory location, and advertise
> >> for other tools to leverage.
> >> - Eric to integrate with Gateway
> >>
> >> 4. PCMDI Gateway to be updated to recent s/w version (Resolved)
> >>
> >> 5. ORP Whitelisting
> >> - Has been implemented, and needs to be configured manually
> >> - Code is in PCMDI repository, and OpenID has been added to Request
> >> Scope to aid in metrics gathering
> >> - With 1.2 release, update ORP release version to 1.2.
> >>
> >> Action Items:
> >> - Gavin to move code to git repository
> >> - Luca to create distribution, and provide latest release for use
> >> - ? to create compatibility matrix of Gateway s/w and ORP s/w.
> >>
> >> 6. Attribute and Authorization Services:
> >> - These services have been implemented and are part of the
> distribution
> >> - Need to add whitelist to prevent open access to these services
> >> - Policy should allow other Gateways, Attribute and AuthZ services,
> >> Data Node Manager to query attribute service
> >> - Attribute registration system in place, needs further testing
> >>
> >> Action Items
> >> - Eric to augment these services with ability to authorize requests
> to
> >> determine if user is in whitelist
> >> - Stephen to test the attribute registration system as a end user
> from
> >> BADC
> >> - Rachana and Stephen to document use case scenarios for these
> >> services, to feed into acceptance tests
> >>
> >> 7. PCMDI Gateway certificate
> >> Action Items
> >> - Bob will have a valid certificate installed for the Gateway
> >>
> >> 8. Publisher client
> >> - Does not validate server's certificate
> >>
> >> Action Items:
> >> - Bob and Gavin to fix the issue
> >>
> >> On Jul 16, 2010, at 10:58 AM, Rachana Ananthakrishnan wrote:
> >>
> >>
> >>> Hi,
> >>>
> >>> We plan to discuss open issues on the Gateway Security aspects on
> >>> Monday July 19th at 9am CDT. Please see coordinates and agenda
> below:
> >>>
> >>> Dial-in: 1 (925) 424-8105
> >>> Access code 305757#
> >>>
> >>> Agenda:
> >>>
> >>> 1. OpenID Whitelisting at Gateways
> >>> 2. wget scripts
> >>> 3. Consistent CA trust roots across the federation
> >>> 4. MyProxy Java Web Start from Gateways
> >>> 5. Authorization and attribute service security
> >>> 6. Attribute service at PCMDI
> >>>
> >>> We'll have a separate call to discuss the data node security
> aspects.
> >>>
> >>> Given we have postponed this many times, it was decided we would
> hold
> >>> a separate meeting on this. I'll send notes and summary to this
> list,
> >>> and we can follow-up as needed.
> >>>
> >>> Rachana
> >>>
> >>> Rachana Ananthakrishnan
> >>> Argonne National Lab | University of Chicago
> >>>
> >>> _______________________________________________
> >>> GO-ESSP-TECH mailing list
> >>> GO-ESSP-TECH at ucar.edu
> >>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >>>
> >> Rachana Ananthakrishnan
> >> Argonne National Lab | University of Chicago
> >>
> >> _______________________________________________
> >> GO-ESSP-TECH mailing list
> >> GO-ESSP-TECH at ucar.edu
> >> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >>

-- 
Scanned by iCritical.