[Go-essp-tech] TDS access control for OpenDAP requests

Fri May 20 08:18:09 MDT 2011

Hi John,
	sorry it took me a couple of days to get back on this... I think I now understand more of what the problem is, and we have a solution.

First of all, I may be wrong but I think that an HTTP request to a TDS aggregation does not result in a subsequent HTTP request to the individual netcdf files - perhaps the TDS forwards this request to itself internally (i.e. it does NOT redirect the browser client), or perhaps the files are simply opened locally ? This is what I see in the logs before I get stopped by the TDS standard authorization mechanism:

2011-05-20T07:08:58.032 -0700 [     79325][      14] INFO  - thredds.server.catalogservice.LocalCatalogServiceController - handleRequestInternal(): Remote host: 128.149.250.137 - Request: "GET /thredds/esgcet/1/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.v1.html?dataset=obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1 HTTP/1.1"
2011-05-20T07:08:58.036 -0700 [     79329][      14] INFO  - thredds.server.catalogservice.LocalCatalogServiceController - handleRequestInternal(): Request Completed - 200 - 3001 - 4
2011-05-20T07:08:58.208 -0700 [     79501][      15] INFO  - thredds.server.root.RootController - handleRequestInternal(): Remote host: 128.149.250.137 - Request: "GET /thredds/threddsIcon.gif HTTP/1.1"
2011-05-20T07:08:58.208 -0700 [     79501][      15] INFO  - thredds.util.RequestForwardUtils - forwardRequest() Forwarding request to "threddsIcon.gif": Request Completed - 1001 - -1 - 0
2011-05-20T07:08:58.221 -0700 [     79514][      16] INFO  - thredds.server.root.RootController - handleRequestInternal(): Remote host: 128.149.250.137 - Request: "GET /thredds/tds.css HTTP/1.1"
2011-05-20T07:08:58.221 -0700 [     79514][      16] INFO  - thredds.util.RequestForwardUtils - forwardRequest() Forwarding request to "tds.css": Request Completed - 1001 - -1 - 0
2011-05-20T07:09:03.971 -0700 [     85264][      17] INFO  - thredds.server.opendap.OpendapServlet - doGet(): Remote host: 128.149.250.137 - Request: "GET /thredds/dodsC/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.html HTTP/1.1"
2011-05-20T07:09:03.972 -0700 [     85265][      17] DEBUG - thredds.server.opendap.OpendapServlet - doGet path=/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.html
2011-05-20T07:09:03.973 -0700 [     85266][      17] DEBUG - thredds.server.opendap.OpendapServlet - doGet(): baseURI was set = http://test-datanode.jpl.nasa.gov/thredds/dodsC/
2011-05-20T07:09:03.974 -0700 [     85267][      17] DEBUG - thredds.servlet.DatasetHandler - DatasetHandler wants obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1
2011-05-20T07:09:03.976 -0700 [     85269][      17] INFO  - thredds.server.opendap.OpendapServlet - Request Completed - 200 - -1 - 6

So it seems to me like the standard TDS authorization is based on securing the URI "obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1" which is the id of the aggregation. It all makes sense, since this is the ID that is secured with restrictAccess="....." in the TDS catalog.

Our problems originated from the fact that we were checking the full uri: "/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods" for access restrictions. Now instead we can reduce that uri to the ID of the aggregation: "obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1", which can be fed to the DatasetHandler to obtain the correct access control. Here's what happens in the logs when I use ESGF security with the latest modifications (before I am redirected to the ORP):

2011-05-20T07:13:48.354 -0700 [     54281][      13] DEBUG - esg.orp.app.AuthenticationFilter - Establishing access control for URL=http://test-datanode.jpl.nasa.gov/thredds/dodsC/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods?plev[0:1:16] session id=5A10A09C78DC8AD6C98885433C54398B current authorization attribute=null
2011-05-20T07:13:48.354 -0700 [     54281][      13] DEBUG - esg.orp.app.tds.TDSPolicyService - URI=/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods resource control=null
2011-05-20T07:13:48.354 -0700 [     54281][      13] DEBUG - esg.orp.app.tds.TDSPolicyService - Uri changed.
2011-05-20T07:13:48.354 -0700 [     54281][      13] DEBUG - esg.orp.app.tds.TDSPolicyService - URI=obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1 resource control=esg-user is secure=true
2011-05-20T07:13:48.354 -0700 [     54281][      13] DEBUG - esg.orp.app.AuthenticationFilter - URL=http://test-datanode.jpl.nasa.gov/thredds/dodsC/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods?plev[0:1:16] is secure
2011-05-20T07:13:48.356 -0700 [     54283][      13] DEBUG - esg.orp.app.AuthenticationFilter - Authentication COOKIE NOT FOUND IN REQUEST, redirecting to: https://test-datanode.jpl.nasa.gov/OpenidRelyingParty/home.htm?redirect=http%3A%2F%2Ftest-datanode.jpl.nasa.gov%2Fthredds%2FdodsC%2Fobs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods%3Fplev%5B0%3A1%3A16%5D
2011-05-20T07:13:48.356 -0700 [     54283][      13] DEBUG - esg.orp.app.AuthorizationFilter - Establishing access control for URL=http://test-datanode.jpl.nasa.gov/thredds/dodsC/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods?plev[0:1:16] session id=5A10A09C78DC8AD6C98885433C54398B current authorization attribute=null
2011-05-20T07:13:48.356 -0700 [     54283][      13] DEBUG - esg.node.filters.AccessLoggingFilter - No match against: http://test-datanode.jpl.nasa.gov/thredds/dodsC/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods
2011-05-20T07:13:48.356 -0700 [     54283][      14] INFO  - thredds.server.opendap.OpendapServlet - doGet(): Remote host: 128.149.250.137 - Request: "GET /thredds/dodsC/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods?plev[0:1:16] HTTP/1.1"
2011-05-20T07:13:48.357 -0700 [     54284][      14] DEBUG - thredds.server.opendap.OpendapServlet - doGet path=/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods
2011-05-20T07:13:48.357 -0700 [     54284][      14] DEBUG - thredds.servlet.DatasetHandler - DatasetHandler wants obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1
2011-05-20T07:13:48.357 -0700 [     54284][      14] DEBUG - esg.orp.app.tds.TDSAuthorizer - Authorization Request Attribute:null
2011-05-20T07:13:48.357 -0700 [     54284][      14] INFO  - thredds.server.opendap.OpendapServlet - Request Completed - 200 - -1 - 1

thanks for your help, I think we have a solution, at least on the data node side... (more to follow).

Luca

On May 17, 2011, at 9:53 AM, John Caron wrote:

> On 5/17/2011 8:05 AM, Cinquini, Luca (3880) wrote:
>> Hi John:
>> 
>> On May 16, 2011, at 7:25 PM, John Caron wrote:
>> 
>>> On 5/16/2011 11:01 AM, Cinquini, Luca (3880) wrote:
>>>> Hi John,
>>>> 	how's going ? I have a follow-up question to our brief conversation at GO-ESSP last week. Could you confirm that theoretically the TDS access control model should be able to secure access to http requests ending in .dods, besides those ending in .nc ?
>>>> 
>>>> The reason I am asking is because looking at the log files it seems otherwise. The ESG filters establish whether or not a URL is secure by calling DatasetHandler.findResourceControl(uri). Up to now, we have changed every URL of the form XYZ.nc.dods to XYZ.nc, and fed this last uri to the DatasetHandler, but this approach does not work for aggregations.
>>>> 
>>>> For example, this is what I see in the logs when making an opendap request on a single file:
>>>> 
>>>> 2011-05-16T09:50:28.655 -0700 [     53640][      14] DEBUG - esg.orp.app.tds.TDSPolicyService - URI=/esg_dataroot/obs4cmip5/observations/atmos/hus/mon/grid/NASA-JPL/AQUA/AIRS/r1i1p1/hus_AQUA_AIRS_L3_RetStd-v5_200209-201006.nc.dods resource control=null
>>>> 2011-05-16T09:50:28.656 -0700 [     53641][      14] DEBUG - esg.orp.app.tds.TDSPolicyService - Uri changed.
>>>> 2011-05-16T09:50:28.656 -0700 [     53641][      14] DEBUG - esg.orp.app.tds.TDSPolicyService - URI=/esg_dataroot/obs4cmip5/observations/atmos/hus/mon/grid/NASA-JPL/AQUA/AIRS/r1i1p1/hus_AQUA_AIRS_L3_RetStd-v5_200209-201006.nc resource control=esg-user is secure=true
>>>> 2011-05-16T09:50:28.656 -0700 [     53641][      14] DEBUG - esg.orp.app.AuthenticationFilter - URL=http://test-datanode.jpl.nasa.gov/thredds/dodsC/esg_dataroot/obs4cmip5/observations/atmos/hus/mon/grid/NASA-JPL/AQUA/AIRS/r1i1p1/hus_AQUA_AIRS_L3_RetStd-v5_200209-201006.nc.dods?hus[0:1:0][0:1:0][0:1:0][0:1:0] is secure
>>>> 
>>>> You'll notice that the original URI */hus_AQUA_AIRS_L3_RetStd-v5_200209-201006.nc.dods is NOT secure, but after dropping the last extension, the URI hus_AQUA_AIRS_L3_RetStd-v5_200209-201006.nc IS secure.
>>>> 
>>>> Off course I might be doing something wrong here, but before digging any further I wanted to make sure that you think dods requests are treated just like normal file requests as far as security is concerned. FYI the catalog I am using to test is:
>>>> 
>>>> http://test-datanode.jpl.nasa.gov/thredds/esgcet/1/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.v1.xml
>>>> 
>>>> thanks a lot, it was great seeing you at the workshop,
>>>> Luca
>>> ok, now that im testing this, im convincing myself that this does work
>>> correctly.
>>> 
>>> when one makes a request for<opendap>.dods, TDS has to open the file
>>> <opendap>, and so DatasetHandler.findResourceControl(<opendap>) will be
>>> correctly restricted, even though
>>> DatasetHandler.findResourceControl(<opendap>.dods)will not.  SO i think
>>> your test has a false positive.
>>> 
>>> of course, if you can actually access a restricted dataset (eg with
>>> .dods or any prefix) then I would be wrong.
>> I did some checking... if you use the TDS TomcatAuthorizer this indeed works, because it relies on the request being redirected to the URL<opendap>  to establish access control.
>> In our case though we use a different implementation of the Authorizer interface, which relies on the method DatasetHandler.findResourceControl(url) to return the access control attribute associated with that dataset, wether it's a request for<opendap>  or for<opendap>.dods.
>> 
>> Would it be difficult to change the TDS so that DatasetHandler.findResourceControl() can be queried with URLs of any kind ? Otherwise, we could modify our Authorizer to also rely on the eventual redirection to the straight netcdf file - I am not quite sure how difficult this last thing would be.
>> 
>> thanks, Luca
> 
> Im not sure i understand. The TDS cant satisfy the request 
> <opendap>.dods without opening <opendap>. So Im not sure how one could 
> get a response to <opendap>.dods without getting authorization for 
> <opendap> ??