[Go-essp-tech] DRS hierarchy on disk

[Neill Miller]

Hello,

We've contacted a GridFTP developer asking "Is there a way of altering the GridFTP server side interface so that it could see a virtual file system structure based on the DRS?".  The following was the response that I got:

--------------
"I haven't heard if anything exists, but it would be a fairly simple
thing to do with a custom DSI.  Everything with a path goes through the
DSI before any system calls are made with that path.

get request with drs-path
query something to find out the true path
run the normal operation using the true path
if the action requires returning data to the server about the path (i.e.
stat for dir listings, make sure any returned info uses the drs-path

It would probably also be something I'd be willing to make a serperate
path-transform plugin interface for.  without it, the whole DSI would be
the same as the existing file dsi except for a few calls to transform
the path.  and this would give us an easy way to handle our own virtual
dir support.  (of course I have no idea where something like this would
fit into planning)"
--------------

What are your thoughts on this?

-Neill.

----- Original Message -----
From: "Dean N. Williams" <williams13 at llnl.gov>
To: "philip kershaw" <philip.kershaw at stfc.ac.uk>
Cc: go-essp-tech at ucar.edu
Sent: Friday, November 20, 2009 9:43:26 AM GMT -06:00 US/Canada Central
Subject: Re: [Go-essp-tech] DRS hierarchy on disk


Hi Phil, 


This is a good question. Perhaps Rachana or Frank can look into this and return a proper answer. My first thought is no, but I am not an expert on GridFTP. 


Best regards, 
Dean 



On Nov 20, 2009, at 2:28 AM, < philip.kershaw at stfc.ac.uk > wrote: 




This is a problem for something like GridFTP where you 'see' the file system on the server not the DRS hierarchy. It's a headache for applying security policy as presumably this is based on the DRS structure. Some kind of mapping would need to be maintained. 

Is there a way of altering the GridFTP server side interface so that it could see a virtual file system structure based on the DRS? 

Cheers, 
Phil 



-----Original Message----- 
From: go-essp-tech-bounces at ucar.edu [ mailto:go-essp-tech-bounces at ucar.edu ] On Behalf Of Dean N. Williams 
Sent: 19 November 2009 12:14 
To: Lawrence, Bryan (STFC,RAL,SSTD) 
Cc: go-essp-tech at ucar.edu 
Subject: Re: [Go-essp-tech] DRS hierarchy on disk 

This is a correct statement. 


-Dean 



On Nov 19, 2009, at 4:04 AM, < bryan.lawrence at stfc.ac.uk > wrote: 





Hi Stephen 

Bob or someone else can confirm, but I think the issue is that ESG in general is aiming at supporting more than just the DRS, so while conforming to DRS is a given for CMIP5, it’s not a given for all the datasets that ESG may work with. 

Bryan 




From: go-essp-tech-bounces at ucar.edu [ mailto:go-essp-tech-bounces at ucar.edu ] On Behalf Of stephen.pascoe at stfc.ac.uk 
Sent: 19 November 2009 11:49 
To: drach at llnl.gov 
Cc: go-essp-tech at ucar.edu 
Subject: [Go-essp-tech] DRS hierarchy on disk 


At the telco this Tuesday I was slightly surprised when someone (I think Bob) asserted data is not likely to follow the DRS hierarchy on disk. Although we know the disk layout can be mapped to DRS URLs in the web server I thought our default position was that we would lay out the data in the DRS hierarchy because it is the simplest solution. 



This issue came up in a meeting with UKMO yesterday where they thought there was a strong reason for using the DRS on disk so that fallback positions like ftp follow a consistent structure. I also learnt that CMOR2 will output to the DRS hierarchy automatically. 



Given this, why wouldn't we want to insist all datasets are stored on disk in the DRS hierarchy? 



Thanks, 

Stephen. 


--- 
Stephen Pascoe +44 (0)1235 445980 
British Atmospheric Data Centre 
Rutherford Appleton Laboratory 




-- 
Scanned by iCritical. 



-- 
Scanned by iCritical. 
_______________________________________________ 
GO-ESSP-TECH mailing list 
GO-ESSP-TECH at ucar.edu 
http://**mailman.ucar.edu/mailman/listinfo/go-essp-tech 




-- 
Scanned by iCritical. 
_______________________________________________ 
GO-ESSP-TECH mailing list 
GO-ESSP-TECH at ucar.edu 
http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech 


_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech