[Go-essp-tech] DRS hierarchy on disk

Tue Nov 24 01:26:04 MST 2009

Hi Dean + thanks Neill for looking this up.

It seems to me that if you could map the DRS hierarchy onto the GridFTP service it would mean you could apply the same security policy as other HTTP data access services.  That could make things a lot simpler.

Cheers,
Phil

> -----Original Message-----
> From: Dean N. Williams [mailto:williams13 at llnl.gov] 
> Sent: 23 November 2009 20:53
> To: Neill Miller
> Cc: go-essp-tech at ucar.edu; Kershaw, Philip (STFC,RAL,SSTD)
> Subject: Re: [Go-essp-tech] DRS hierarchy on disk
> 
> 
> Hi Neill,
> 
> 	Thanks for hunting down this information. It is good to 
> know, in case  
> we need it for GridFTP and DRS. Phil does this information 
> answer your  
> question?
> 
> Best regards,
> 	Dean
> 
> On Nov 23, 2009, at 12:32 PM, Neill Miller wrote:
> 
> > Hello,
> >
> > We've contacted a GridFTP developer asking "Is there a way of
> > altering the GridFTP server side interface so that it could see a  
> > virtual file system structure based on the DRS?".  The 
> following was  
> > the response that I got:
> >
> > --------------
> > "I haven't heard if anything exists, but it would be a 
> fairly simple 
> > thing to do with a custom DSI.  Everything with a path goes through
> > the
> > DSI before any system calls are made with that path.
> >
> > get request with drs-path
> > query something to find out the true path
> > run the normal operation using the true path
> > if the action requires returning data to the server about the path
> > (i.e.
> > stat for dir listings, make sure any returned info uses the drs-path
> >
> > It would probably also be something I'd be willing to make 
> a serperate 
> > path-transform plugin interface for.  without it, the whole DSI
> > would be
> > the same as the existing file dsi except for a few calls to 
> transform
> > the path.  and this would give us an easy way to handle our own  
> > virtual
> > dir support.  (of course I have no idea where something like this  
> > would
> > fit into planning)"
> > --------------
> >
> > What are your thoughts on this?
> >
> > -Neill.
> >
> > ----- Original Message -----
> > From: "Dean N. Williams" <williams13 at llnl.gov>
> > To: "philip kershaw" <philip.kershaw at stfc.ac.uk>
> > Cc: go-essp-tech at ucar.edu
> > Sent: Friday, November 20, 2009 9:43:26 AM GMT -06:00 US/Canada
> > Central
> > Subject: Re: [Go-essp-tech] DRS hierarchy on disk
> >
> >
> > Hi Phil,
> >
> >
> > This is a good question. Perhaps Rachana or Frank can look 
> into this  
> > and return a proper answer. My first thought is no, but I 
> am not an  
> > expert on GridFTP.
> >
> >
> > Best regards,
> > Dean
> >
> >
> >
> > On Nov 20, 2009, at 2:28 AM, < philip.kershaw at stfc.ac.uk > wrote:
> >
> >
> >
> >
> > This is a problem for something like GridFTP where you 'see' the  
> > file system on the server not the DRS hierarchy. It's a 
> headache for  
> > applying security policy as presumably this is based on the DRS  
> > structure. Some kind of mapping would need to be maintained.
> >
> > Is there a way of altering the GridFTP server side 
> interface so that  
> > it could see a virtual file system structure based on the DRS?
> >
> > Cheers,
> > Phil
> >
> >
> >
> > -----Original Message-----
> > From: go-essp-tech-bounces at ucar.edu [ 
> mailto:go-essp-tech-bounces at ucar.edu 
> >  ] On Behalf Of Dean N. Williams
> > Sent: 19 November 2009 12:14
> > To: Lawrence, Bryan (STFC,RAL,SSTD)
> > Cc: go-essp-tech at ucar.edu
> > Subject: Re: [Go-essp-tech] DRS hierarchy on disk
> >
> > This is a correct statement.
> >
> >
> > -Dean
> >
> >
> >
> > On Nov 19, 2009, at 4:04 AM, < bryan.lawrence at stfc.ac.uk > wrote:
> >
> >
> >
> >
> >
> > Hi Stephen
> >
> > Bob or someone else can confirm, but I think the issue is that ESG  
> > in general is aiming at supporting more than just the DRS, 
> so while  
> > conforming to DRS is a given for CMIP5, it’s not a given 
> for all the  
> > datasets that ESG may work with.
> >
> > Bryan
> >
> >
> >
> >
> > From: go-essp-tech-bounces at ucar.edu [ 
> mailto:go-essp-tech-bounces at ucar.edu 
> >  ] On Behalf Of stephen.pascoe at stfc.ac.uk
> > Sent: 19 November 2009 11:49
> > To: drach at llnl.gov
> > Cc: go-essp-tech at ucar.edu
> > Subject: [Go-essp-tech] DRS hierarchy on disk
> >
> >
> > At the telco this Tuesday I was slightly surprised when someone (I  
> > think Bob) asserted data is not likely to follow the DRS hierarchy  
> > on disk. Although we know the disk layout can be mapped to 
> DRS URLs  
> > in the web server I thought our default position was that we would  
> > lay out the data in the DRS hierarchy because it is the simplest  
> > solution.
> >
> >
> >
> > This issue came up in a meeting with UKMO yesterday where they  
> > thought there was a strong reason for using the DRS on disk 
> so that  
> > fallback positions like ftp follow a consistent structure. I also  
> > learnt that CMOR2 will output to the DRS hierarchy automatically.
> >
> >
> >
> > Given this, why wouldn't we want to insist all datasets are stored  
> > on disk in the DRS hierarchy?
> >
> >
> >
> > Thanks,
> >
> > Stephen.
> >
> >
> > ---
> > Stephen Pascoe +44 (0)1235 445980
> > British Atmospheric Data Centre
> > Rutherford Appleton Laboratory
> >
> >
> >
> >
> > -- 
> > Scanned by iCritical.
> >
> >
> >
> > -- 
> > Scanned by iCritical.
> > _______________________________________________
> > GO-ESSP-TECH mailing list
> > GO-ESSP-TECH at ucar.edu
> > http://***mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >
> >
> >
> >
> > -- 
> > Scanned by iCritical.
> > _______________________________________________
> > GO-ESSP-TECH mailing list
> > GO-ESSP-TECH at ucar.edu
> > http://**mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >
> >
> > _______________________________________________
> > GO-ESSP-TECH mailing list
> > GO-ESSP-TECH at ucar.edu
> > http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >
> 
> 
-- 
Scanned by iCritical.