[Go-essp-tech] TDS Authorization issues
Nathan Wilhelmi
wilhelmi at ucar.edu
Wed Feb 22 07:01:36 MST 2012
Hi All,
I am pretty confident that isn't enough, in essence the problem is still
there. There is a 1-1 mapping of SOAPServiceClient to HttpClient. So
with this code you still have 1 HttpClient and ConnectionManager per
call which is the fundamental problem. I believe SoapServiceClient is
new'd per request.
For out patch we just made it a static variable. Extracting it would
have been a much better solution, but we didn't want to change that much
code....
Thanks!
-Nate
On 2/22/2012 4:56 AM, stephen.pascoe at stfc.ac.uk wrote:
> I think Luca has already fixed it in git's devel branch.
>
> SAMLAuthorizationServiceFilterCollaborator instantiates SAMLAuthorizer (final) which instantiates SAMLServiceClient (final) which instantiates HttpClient. The collaborator is a singleton created from spring so I *think* there is only one SAMLServiceClient. In Git this instantiates HttpClient like this:
>
> public SOAPServiceClient() {
> HttpConnectionManager manager = new MultiThreadedHttpConnectionManager();
> manager.getParams().setConnectionTimeout(TIMEOUT);
> manager.getParams().setSoTimeout(TIMEOUT);
> client = new HttpClient(manager);
> }
>
> As usual you can't beat Luca to the punch :-)
>
> Cheers,
> Stephen.
>
> ---
> Stephen Pascoe +44 (0)1235 445980
> Centre of Environmental Data Archival
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>
>
> -----Original Message-----
> From: Estanislao Gonzalez [mailto:gonzalez at dkrz.de]
> Sent: 22 February 2012 11:45
> To: Nathan Wilhelmi; Cinquini, Luca (3880); Pascoe, Stephen (STFC,RAL,RALSP)
> Subject: Re: [Go-essp-tech] TDS Authorization issues
>
> Hi Stephen,
>
> I guess I should have added you to this email.
>
> Thanks,
> Estani
>
> Am 22.02.2012 10:35, schrieb Estanislao Gonzalez:
>> Hi Nate,
>>
>> thanks for this... indeed it looks like the problem I've been seen...
>>
>> Could you send us the patch?
>>
>> I've tried to see it myself, but it's really hard to see what gets
>> used where, with all the Spring wiring happening dynamically...
>> I've found only 5 non-util classes with synchronized methods in
>> them... and none of them in the SOAP area... I wonder if that part of
>> the code is multi-thread tolerant at all....
>>
>> I can't patch it myself without knowing what is really being done. How
>> have you done it Nate?
>> For instance, the problem is here:
>> public SOAPServiceClient() {
>> HttpConnectionManager manager = new
>> SimpleHttpConnectionManager();
>> manager.getParams().setConnectionTimeout(TIMEOUT);
>> manager.getParams().setSoTimeout(TIMEOUT);
>> client = new HttpClient(manager);
>>
>> }
>>
>> But SOAPServiceClient is not a singleton, so potentially we could have
>> multiple objects, is that true? If So, I'd think the connection
>> manager should be extracted to a static variable and made
>> multi-threaded. If not, then The whole class should be a singleton and
>> add the multi-threaded connection manager exactly there where it is...
>> but I can't find out how exactly is it being used (I assume the later,
>> as it's in a factory class... but then again the factory itself
>> (esg.security.authz.service.impl.SAMLAuthorizationFactoryImpl) is
>> neither a singleton nor static, so Spring could potentially be crating
>> multiple factories... very confusing :-/
>>
>> Anyway, Luca, I also saw many classes that look like test classes, but
>> are on the main branch... they make the code harder to read in my
>> opinion... if that' so, could you please refactor them to the test
>> source-code branch? I've found for example these while trying to
>> figure out where SOAPServiceClient is being used:
>>
>> esg.security.attr.main.SAMLAttributeServiceSOAPClient
>> esg.security.attr.main.SecureSAMLAttributeServiceSOAPClient
>> esg.security.authz.main.SAMLAuthorizationServiceSOAPClient
>> esg.security.authz.main.SecureSAMLAuthorizationServiceSOAPClient
>> esg.security.webclient.TestController
>>
>> They all have hard coded values and just a main method.
>>
>> Thanks,
>> Estani
>> Am 22.02.2012 03:55, schrieb Nathan Wilhelmi:
>>> Hi All,
>>>
>>> In debugging the issue I think we found a couple of issues on the node
>>> that would manifest themselves under load.
>>>
>>> HttpClient may not be closing connections, it depends on the load, GC,
>>> etc. This is a pretty good summary of the problem:
>>>
>>> http://fuyun.org/2009/09/connection-close-in-httpclient/
>>>
>>> Once we patched the SOAPServiceClient.class code to use a single
>>> instance the authorization calls began working again. We also found that
>>> the authorization would work reliably, albeit very slowly, if tomcat was
>>> only given a single thread.
>>>
>>> Estani, I believe this is likely the source of dangling sockets you
>>> noted in earlier emails.
>>>
>>>
>>> Thanks!
>>> -Nate
>>>
>>>
>>>
>>>
>>> On 02/20/2012 02:20 PM, Cinquini, Luca (3880) wrote:
>>>> Hi Stephen,
>>>> we expect a release any day - Gavin needs to fix something one
>>>> the node manager, then we will build a release... we'll let you know...
>>>> thanks, Luca
>>>>
>>>> On Feb 20, 2012, at 8:49 AM,<stephen.pascoe at stfc.ac.uk> wrote:
>>>>
>>>>> Hi Luca,
>>>>>
>>>>> This is great timing -- I was just thinking about making some sort
>>>>> of cache to fix this AttributeService problem.
>>>>>
>>>>> So, I'm eager to move to this version. I have a partially-built
>>>>> test P2P node on which I last ran the script about 10 days ago.
>>>>> How should I go about installing this version with caching
>>>>> AuthzService? Our test P2P node is intended to either work in
>>>>> datanode-only mode or move swiftly to a full P2P node with index
>>>>> and web-fe, depending on how the deployment goes. My top priority
>>>>> is to make sure our production datanode is compatible with the P2P
>>>>> system.
>>>>>
>>>>> Cheers,
>>>>> Stephen.
>>>>>
>>>>> ---
>>>>> Stephen Pascoe +44 (0)1235 445980
>>>>> Centre of Environmental Data Archival
>>>>> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11
>>>>> 0QX, UK
>>>>>
>>>>> -----Original Message-----
>>>>> From: Cinquini, Luca (3880) [mailto:Luca.Cinquini at jpl.nasa.gov]
>>>>> Sent: 20 February 2012 14:40
>>>>> To: Pascoe, Stephen (STFC,RAL,RALSP)
>>>>> Cc: wilhelmi at ucar.edu; go-essp-tech at ucar.edu
>>>>> Subject: Re: [Go-essp-tech] TDS Authorization issues
>>>>>
>>>>> Hi Stephen,
>>>>> and you'll be happy to know that with the next release, there
>>>>> will be caching of user attributes by the authorization service -
>>>>> your most desired feature.
>>>>> This release will support the data-node-only configuration. I've
>>>>> been testing and it looks good, but I'd like another node to test
>>>>> it first before we ask more nodes to install it.
>>>>> thanks, Luca
>>>>>
>>>>> On Feb 20, 2012, at 3:38 AM,<stephen.pascoe at stfc.ac.uk> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> For some reason I've got these git URLs for esgf-security but I
>>>>>> think esg-repo.llnl.gov is just an alias for esgf.org:
>>>>>>
>>>>>> $ git remote -v
>>>>>> origin git at esg-repo.llnl.gov:esgf-security.git (fetch)
>>>>>> origin git at esg-repo.llnl.gov:esgf-security.git (push)
>>>>>>
>>>>>> The HttpClient code has timeouts in it now so upgrading should
>>>>>> help with SOAP calls. Our production node still has a fairly old
>>>>>> version of esgf-security but there will be an upgrade really
>>>>>> soon. It's great I've got an even bigger reason to upgrade!
>>>>>>
>>>>>> See
>>>>>> esgf-security.git/src/java/main/esg/security/common/SOAPServiceClient.java
>>>>>>
>>>>>> Cheers,
>>>>>> Stephen.
>>>>>>
>>>>>> ---
>>>>>> Stephen Pascoe +44 (0)1235 445980
>>>>>> Centre of Environmental Data Archival
>>>>>> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11
>>>>>> 0QX, UK
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: go-essp-tech-bounces at ucar.edu
>>>>>> [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Cinquini, Luca
>>>>>> (3880)
>>>>>> Sent: 17 February 2012 19:53
>>>>>> To: Nathan Wilhelmi
>>>>>> Cc: go-essp-tech at ucar.edu
>>>>>> Subject: Re: [Go-essp-tech] TDS Authorization issues
>>>>>>
>>>>>> Hi Nate,
>>>>>> it should be esgf-security.git. But again, my advice is to
>>>>>> wait till you can install the same software as al the other CMIP5
>>>>>> data nodes - it should be early next week.
>>>>>> thanks, L
>>>>>>
>>>>>> On Feb 17, 2012, at 11:14 AM, Nathan Wilhelmi wrote:
>>>>>>
>>>>>>> Hi Luca,
>>>>>>>
>>>>>>> Is esg-security available? esg-security.git doesn't seem to be
>>>>>>> available?
>>>>>>>
>>>>>>> Thanks!
>>>>>>> -Nate
>>>>>>>
>>>>>>> On 2/17/2012 11:04 AM, Cinquini, Luca (3880) wrote:
>>>>>>>> Hi Nate,
>>>>>>>> that is some seriously old code you are using...
>>>>>>>>
>>>>>>>> The ORP is available from:
>>>>>>>>
>>>>>>>> git clone git at esgf.org:esg-orp.git
>>>>>>>>
>>>>>>>> but it looks like the version you are using is the very first
>>>>>>>> tag there,
>>>>>>>> probably more than one year old.
>>>>>>>>
>>>>>>>> The esg-saml library does not even exist any more, it's been long
>>>>>>>> replaced by esgf-security.
>>>>>>>>
>>>>>>>> My suggestion would be to wait till next week when we will roll
>>>>>>>> out a
>>>>>>>> data-node only installation, for use by all CMIP5 data nodes. That
>>>>>>>> should allow you to upgrade the UCAR TDS as a data-node only.
>>>>>>>>
>>>>>>>> thanks, Luca
>>>>>>>>
>>>>>>>> On Feb 17, 2012, at 10:46 AM, Nathan Wilhelmi wrote:
>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> Version # mistake.
>>>>>>>>>
>>>>>>>>> We are looking for:
>>>>>>>>>
>>>>>>>>> esg-node.1.0.0.4.jar
>>>>>>>>> esg-node-common.1.0.0.4.jar
>>>>>>>>> esg-node-filters.1.0.0.4.jar
>>>>>>>>> esg-orp.1.1.2.2.jar
>>>>>>>>> esg-saml.1.2.1.jar
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> -Nate
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2/17/2012 10:38 AM, Nathan Wilhelmi wrote:
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> We upgraded our TDS to the newer no caching version this week.
>>>>>>>>>> This
>>>>>>>>>> version uses much less RAM and is more responsive. However
>>>>>>>>>> this appears
>>>>>>>>>> to have uncovered some issues with the authorization related node
>>>>>>>>>> components. At this point our TDS will only authorize datasets
>>>>>>>>>> for a
>>>>>>>>>> short time after a restart. Our TDS will authorize reliably if
>>>>>>>>>> only 1
>>>>>>>>>> thread is enabled in Tomcat, but this isn't really a workable
>>>>>>>>>> solution.
>>>>>>>>>>
>>>>>>>>>> After some debugging and code review we believe the problem
>>>>>>>>>> lies with
>>>>>>>>>> HttpClient usage patterns.
>>>>>>>>>>
>>>>>>>>>> To expedite getting our TDS back online and debugging the
>>>>>>>>>> problem we are
>>>>>>>>>> trying to get a hold of the source for the node components we
>>>>>>>>>> are using.
>>>>>>>>>> Where can we get the source for the following jars?
>>>>>>>>>>
>>>>>>>>>> esg-node.1.0.0.4.jar
>>>>>>>>>> esg-node-common.1.0.0.4.jar
>>>>>>>>>> esg-node-filters.1.0.0.4.jar
>>>>>>>>>> esg-orp.1.0.0.4.jar
>>>>>>>>>> esg-saml.1.0.0.4.jar
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>> -Nate
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> GO-ESSP-TECH mailing list
>>>>>>>>>> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>>>>> _______________________________________________
>>>>>>>>> GO-ESSP-TECH mailing list
>>>>>>>>> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>> _______________________________________________
>>>>>> GO-ESSP-TECH mailing list
>>>>>> GO-ESSP-TECH at ucar.edu
>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>> --
>>>>>> Scanned by iCritical.
>>>>> --
>>>>> Scanned by iCritical.
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>>
>
>
More information about the GO-ESSP-TECH
mailing list