[Go-essp-tech] TDS access control for OpenDAP requests
John Caron
caron at unidata.ucar.edu
Fri May 20 11:30:25 MDT 2011
On 5/20/2011 8:18 AM, Cinquini, Luca (3880) wrote:
> Hi John,
> sorry it took me a couple of days to get back on this... I think I now understand more of what the problem is, and we have a solution.
>
> First of all, I may be wrong but I think that an HTTP request to a TDS aggregation does not result in a subsequent HTTP request to the individual netcdf files - perhaps the TDS forwards this request to itself internally (i.e. it does NOT redirect the browser client), or perhaps the files are simply opened locally ?
The usual case is that it opens local files. They could be remote files
on another server, but those would show up in the logs either.
> This is what I see in the logs before I get stopped by the TDS standard authorization mechanism:
>
> 2011-05-20T07:08:58.032 -0700 [ 79325][ 14] INFO - thredds.server.catalogservice.LocalCatalogServiceController - handleRequestInternal(): Remote host: 128.149.250.137 - Request: "GET /thredds/esgcet/1/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.v1.html?dataset=obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1 HTTP/1.1"
> 2011-05-20T07:08:58.036 -0700 [ 79329][ 14] INFO - thredds.server.catalogservice.LocalCatalogServiceController - handleRequestInternal(): Request Completed - 200 - 3001 - 4
> 2011-05-20T07:08:58.208 -0700 [ 79501][ 15] INFO - thredds.server.root.RootController - handleRequestInternal(): Remote host: 128.149.250.137 - Request: "GET /thredds/threddsIcon.gif HTTP/1.1"
> 2011-05-20T07:08:58.208 -0700 [ 79501][ 15] INFO - thredds.util.RequestForwardUtils - forwardRequest() Forwarding request to "threddsIcon.gif": Request Completed - 1001 - -1 - 0
> 2011-05-20T07:08:58.221 -0700 [ 79514][ 16] INFO - thredds.server.root.RootController - handleRequestInternal(): Remote host: 128.149.250.137 - Request: "GET /thredds/tds.css HTTP/1.1"
> 2011-05-20T07:08:58.221 -0700 [ 79514][ 16] INFO - thredds.util.RequestForwardUtils - forwardRequest() Forwarding request to "tds.css": Request Completed - 1001 - -1 - 0
> 2011-05-20T07:09:03.971 -0700 [ 85264][ 17] INFO - thredds.server.opendap.OpendapServlet - doGet(): Remote host: 128.149.250.137 - Request: "GET /thredds/dodsC/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.html HTTP/1.1"
> 2011-05-20T07:09:03.972 -0700 [ 85265][ 17] DEBUG - thredds.server.opendap.OpendapServlet - doGet path=/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.html
> 2011-05-20T07:09:03.973 -0700 [ 85266][ 17] DEBUG - thredds.server.opendap.OpendapServlet - doGet(): baseURI was set = http://test-datanode.jpl.nasa.gov/thredds/dodsC/
> 2011-05-20T07:09:03.974 -0700 [ 85267][ 17] DEBUG - thredds.servlet.DatasetHandler - DatasetHandler wants obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1
> 2011-05-20T07:09:03.976 -0700 [ 85269][ 17] INFO - thredds.server.opendap.OpendapServlet - Request Completed - 200 - -1 - 6
>
> So it seems to me like the standard TDS authorization is based on securing the URI "obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1" which is the id of the aggregation. It all makes sense, since this is the ID that is secured with restrictAccess="....." in the TDS catalog.
yes
> Our problems originated from the fact that we were checking the full uri: "/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods" for access restrictions. Now instead we can reduce that uri to the ID of the aggregation: "obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1", which can be fed to the DatasetHandler to obtain the correct access control.
i dont think you can use the ID, you have to use the url path, look at
DatasetHandler.putResourceControl():
static void putResourceControl(InvDatasetImpl ds) {
if (log.isDebugEnabled()) log.debug("putResourceControl " +
ds.getRestrictAccess() + " for " + ds.getName());
// resourceControl is inherited, but no guarentee that children
paths are related, unless its a
// InvDatasetScan or InvDatasetFmrc. So we keep track of all
datasets that have a ResourceControl, including children
// InvDatasetScan and InvDatasetFmrc must use a PathMatcher, others
can use exact match (hash)
if (ds instanceof InvDatasetScan) {
InvDatasetScan scan = (InvDatasetScan) ds;
if (debugResourceControl)
System.out.println("putResourceControl " +
ds.getRestrictAccess() + " for datasetScan " + scan.getPath());
resourceControlMatcher.put(scan.getPath(), ds.getRestrictAccess());
} else if (ds instanceof InvDatasetFmrc) {
InvDatasetFmrc fmrc = (InvDatasetFmrc) ds;
if (debugResourceControl)
System.out.println("putResourceControl " +
ds.getRestrictAccess() + " for datasetFmrc " + fmrc.getPath());
resourceControlMatcher.put(fmrc.getPath(), ds.getRestrictAccess());
} else { // dataset
if (debugResourceControl)
System.out.println("putResourceControl " +
ds.getRestrictAccess() + " for dataset " + ds.getUrlPath());
// LOOK: seems like you only need to add if
InvAccess.InvService.isReletive
// LOOK: seems like we should use resourceControlMatcher to make
sure we match .dods, etc
for (InvAccess access : ds.getAccess()) {
if (access.getService().isRelativeBase())
resourceControlHash.put(access.getUrlPath(),
ds.getRestrictAccess());
}
}
hasResourceControl = true;
}
it might be that in your case the id = path, but does that always hold.
> Here's what happens in the logs when I use ESGF security with the latest modifications (before I am redirected to the ORP):
>
> 2011-05-20T07:13:48.354 -0700 [ 54281][ 13] DEBUG - esg.orp.app.AuthenticationFilter - Establishing access control for URL=http://test-datanode.jpl.nasa.gov/thredds/dodsC/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods?plev[0:1:16] session id=5A10A09C78DC8AD6C98885433C54398B current authorization attribute=null
> 2011-05-20T07:13:48.354 -0700 [ 54281][ 13] DEBUG - esg.orp.app.tds.TDSPolicyService - URI=/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods resource control=null
> 2011-05-20T07:13:48.354 -0700 [ 54281][ 13] DEBUG - esg.orp.app.tds.TDSPolicyService - Uri changed.
> 2011-05-20T07:13:48.354 -0700 [ 54281][ 13] DEBUG - esg.orp.app.tds.TDSPolicyService - URI=obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1 resource control=esg-user is secure=true
> 2011-05-20T07:13:48.354 -0700 [ 54281][ 13] DEBUG - esg.orp.app.AuthenticationFilter - URL=http://test-datanode.jpl.nasa.gov/thredds/dodsC/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods?plev[0:1:16] is secure
> 2011-05-20T07:13:48.356 -0700 [ 54283][ 13] DEBUG - esg.orp.app.AuthenticationFilter - Authentication COOKIE NOT FOUND IN REQUEST, redirecting to: https://test-datanode.jpl.nasa.gov/OpenidRelyingParty/home.htm?redirect=http%3A%2F%2Ftest-datanode.jpl.nasa.gov%2Fthredds%2FdodsC%2Fobs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods%3Fplev%5B0%3A1%3A16%5D
> 2011-05-20T07:13:48.356 -0700 [ 54283][ 13] DEBUG - esg.orp.app.AuthorizationFilter - Establishing access control for URL=http://test-datanode.jpl.nasa.gov/thredds/dodsC/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods?plev[0:1:16] session id=5A10A09C78DC8AD6C98885433C54398B current authorization attribute=null
> 2011-05-20T07:13:48.356 -0700 [ 54283][ 13] DEBUG - esg.node.filters.AccessLoggingFilter - No match against: http://test-datanode.jpl.nasa.gov/thredds/dodsC/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods
> 2011-05-20T07:13:48.356 -0700 [ 54283][ 14] INFO - thredds.server.opendap.OpendapServlet - doGet(): Remote host: 128.149.250.137 - Request: "GET /thredds/dodsC/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods?plev[0:1:16] HTTP/1.1"
> 2011-05-20T07:13:48.357 -0700 [ 54284][ 14] DEBUG - thredds.server.opendap.OpendapServlet - doGet path=/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1.dods
> 2011-05-20T07:13:48.357 -0700 [ 54284][ 14] DEBUG - thredds.servlet.DatasetHandler - DatasetHandler wants obs4cmip5.NASA-JPL.AQUA.AIRS.mon.hus.1.aggregation.1
> 2011-05-20T07:13:48.357 -0700 [ 54284][ 14] DEBUG - esg.orp.app.tds.TDSAuthorizer - Authorization Request Attribute:null
> 2011-05-20T07:13:48.357 -0700 [ 54284][ 14] INFO - thredds.server.opendap.OpendapServlet - Request Completed - 200 - -1 - 1
>
>
>
> thanks for your help, I think we have a solution, at least on the data node side... (more to follow).
>
> Luca
>
>
>
> On May 17, 2011, at 9:53 AM, John Caron wrote:
>
>> On 5/17/2011 8:05 AM, Cinquini, Luca (3880) wrote:
>>> Hi John:
>>>
>>> On May 16, 2011, at 7:25 PM, John Caron wrote:
>>>
>>>> On 5/16/2011 11:01 AM, Cinquini, Luca (3880) wrote:
>>>>> Hi John,
>>>>> how's going ? I have a follow-up question to our brief conversation at GO-ESSP last week. Could you confirm that theoretically the TDS access control model should be able to secure access to http requests ending in .dods, besides those ending in .nc ?
>>>>>
>>>>> The reason I am asking is because looking at the log files it seems otherwise. The ESG filters establish whether or not a URL is secure by calling DatasetHandler.findResourceControl(uri). Up to now, we have changed every URL of the form XYZ.nc.dods to XYZ.nc, and fed this last uri to the DatasetHandler, but this approach does not work for aggregations.
>>>>>
>>>>> For example, this is what I see in the logs when making an opendap request on a single file:
>>>>>
>>>>> 2011-05-16T09:50:28.655 -0700 [ 53640][ 14] DEBUG - esg.orp.app.tds.TDSPolicyService - URI=/esg_dataroot/obs4cmip5/observations/atmos/hus/mon/grid/NASA-JPL/AQUA/AIRS/r1i1p1/hus_AQUA_AIRS_L3_RetStd-v5_200209-201006.nc.dods resource control=null
>>>>> 2011-05-16T09:50:28.656 -0700 [ 53641][ 14] DEBUG - esg.orp.app.tds.TDSPolicyService - Uri changed.
>>>>> 2011-05-16T09:50:28.656 -0700 [ 53641][ 14] DEBUG - esg.orp.app.tds.TDSPolicyService - URI=/esg_dataroot/obs4cmip5/observations/atmos/hus/mon/grid/NASA-JPL/AQUA/AIRS/r1i1p1/hus_AQUA_AIRS_L3_RetStd-v5_200209-201006.nc resource control=esg-user is secure=true
>>>>> 2011-05-16T09:50:28.656 -0700 [ 53641][ 14] DEBUG - esg.orp.app.AuthenticationFilter - URL=http://test-datanode.jpl.nasa.gov/thredds/dodsC/esg_dataroot/obs4cmip5/observations/atmos/hus/mon/grid/NASA-JPL/AQUA/AIRS/r1i1p1/hus_AQUA_AIRS_L3_RetStd-v5_200209-201006.nc.dods?hus[0:1:0][0:1:0][0:1:0][0:1:0] is secure
>>>>>
>>>>> You'll notice that the original URI */hus_AQUA_AIRS_L3_RetStd-v5_200209-201006.nc.dods is NOT secure, but after dropping the last extension, the URI hus_AQUA_AIRS_L3_RetStd-v5_200209-201006.nc IS secure.
>>>>>
>>>>> Off course I might be doing something wrong here, but before digging any further I wanted to make sure that you think dods requests are treated just like normal file requests as far as security is concerned. FYI the catalog I am using to test is:
>>>>>
>>>>> http://test-datanode.jpl.nasa.gov/thredds/esgcet/1/obs4cmip5.NASA-JPL.AQUA.AIRS.mon.v1.xml
>>>>>
>>>>> thanks a lot, it was great seeing you at the workshop,
>>>>> Luca
>>>> ok, now that im testing this, im convincing myself that this does work
>>>> correctly.
>>>>
>>>> when one makes a request for<opendap>.dods, TDS has to open the file
>>>> <opendap>, and so DatasetHandler.findResourceControl(<opendap>) will be
>>>> correctly restricted, even though
>>>> DatasetHandler.findResourceControl(<opendap>.dods)will not. SO i think
>>>> your test has a false positive.
>>>>
>>>> of course, if you can actually access a restricted dataset (eg with
>>>> .dods or any prefix) then I would be wrong.
>>> I did some checking... if you use the TDS TomcatAuthorizer this indeed works, because it relies on the request being redirected to the URL<opendap> to establish access control.
>>> In our case though we use a different implementation of the Authorizer interface, which relies on the method DatasetHandler.findResourceControl(url) to return the access control attribute associated with that dataset, wether it's a request for<opendap> or for<opendap>.dods.
>>>
>>> Would it be difficult to change the TDS so that DatasetHandler.findResourceControl() can be queried with URLs of any kind ? Otherwise, we could modify our Authorizer to also rely on the eventual redirection to the straight netcdf file - I am not quite sure how difficult this last thing would be.
>>>
>>> thanks, Luca
>> Im not sure i understand. The TDS cant satisfy the request
>> <opendap>.dods without opening<opendap>. So Im not sure how one could
>> get a response to<opendap>.dods without getting authorization for
>> <opendap> ??
More information about the GO-ESSP-TECH
mailing list