[Go-essp-tech] TDS Authorization issues - to CMIP5 node administrators

Wed Feb 22 03:47:51 MST 2012

Hi,

until this get solved a work-around I found is to increase the number of 
allowed open files (includes tcp connections).
This will let the garbage collector act before the node runs out of them.

#as tomcat run ulimit
root#  su -c"ulimit -n"  tomcat
1024

#if it's that low try to increase it to 4096 by adding this line to /etc/security/limits.conf
tomcat               -       nofile          4096

#Check it has been changed
root
#  su -c"ulimit -n"  tomcat
4096

It's documented here: 
http://esgf.org/wiki/ESGFNode/FAQ#Tomcat_is_complaining_about_too_many_open_files

4x looked Ok, there's no other reason why it should be 4096. If anyone 
have better heuristics, please share.
The problem should be corrected asap, but that number of open files is 
too low anyway (every connection maintains at least 2 open files, one to 
the file being read and one to the TCP connection)

Thanks,
Estani

Am 22.02.2012 03:55, schrieb Nathan Wilhelmi:
> Hi All,
>
> In debugging the issue I think we found a couple of issues on the node
> that would manifest themselves under load.
>
> HttpClient may not be closing connections, it depends on the load, GC,
> etc. This is a pretty good summary of the problem:
>
> http://fuyun.org/2009/09/connection-close-in-httpclient/
>
> Once we patched the SOAPServiceClient.class code to use a single
> instance the authorization calls began working again. We also found that
> the authorization would work reliably, albeit very slowly, if tomcat was
> only given a single thread.
>
> Estani, I believe this is likely the source of dangling sockets you
> noted in earlier emails.
>
>
> Thanks!
> -Nate
>
>
>
>
> On 02/20/2012 02:20 PM, Cinquini, Luca (3880) wrote:
>> Hi Stephen,
>> 	we expect a release any day - Gavin needs to fix something one the node manager, then we will build a release... we'll let you know...
>> thanks, Luca
>>
>> On Feb 20, 2012, at 8:49 AM,<stephen.pascoe at stfc.ac.uk>   wrote:
>>
>>> Hi Luca,
>>>
>>> This is great timing -- I was just thinking about making some sort of cache to fix this AttributeService problem.
>>>
>>> So, I'm eager to move to this version.  I have a partially-built test P2P node on which I last ran the script about 10 days ago.  How should I go about installing this version with caching AuthzService?  Our test P2P node is intended to either work in datanode-only mode or move swiftly to a full P2P node with index and web-fe, depending on how the deployment goes.  My top priority is to make sure our production datanode is compatible with the P2P system.
>>>
>>> Cheers,
>>> Stephen.
>>>
>>> ---
>>> Stephen Pascoe  +44 (0)1235 445980
>>> Centre of Environmental Data Archival
>>> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>>>
>>> -----Original Message-----
>>> From: Cinquini, Luca (3880) [mailto:Luca.Cinquini at jpl.nasa.gov]
>>> Sent: 20 February 2012 14:40
>>> To: Pascoe, Stephen (STFC,RAL,RALSP)
>>> Cc: wilhelmi at ucar.edu; go-essp-tech at ucar.edu
>>> Subject: Re: [Go-essp-tech] TDS Authorization issues
>>>
>>> Hi Stephen,
>>> 	and you'll be happy to know that with the next release, there will be caching of user attributes by the authorization service - your most desired feature.
>>> This release will support the data-node-only configuration. I've been testing and it looks good, but I'd like another node to test it first before we ask more nodes to install it.
>>> thanks, Luca
>>>
>>> On Feb 20, 2012, at 3:38 AM,<stephen.pascoe at stfc.ac.uk>   wrote:
>>>
>>>> Hi All,
>>>>
>>>> For some reason I've got these git URLs for esgf-security but I think esg-repo.llnl.gov is just an alias for esgf.org:
>>>>
>>>> $ git remote -v
>>>> origin	git at esg-repo.llnl.gov:esgf-security.git (fetch)
>>>> origin	git at esg-repo.llnl.gov:esgf-security.git (push)
>>>>
>>>> The HttpClient code has timeouts in it now so upgrading should help with SOAP calls.  Our production node still has a fairly old version of esgf-security but there will be an upgrade really soon.  It's great I've got an even bigger reason to upgrade!
>>>>
>>>> See esgf-security.git/src/java/main/esg/security/common/SOAPServiceClient.java
>>>>
>>>> Cheers,
>>>> Stephen.
>>>>
>>>> ---
>>>> Stephen Pascoe  +44 (0)1235 445980
>>>> Centre of Environmental Data Archival
>>>> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Cinquini, Luca (3880)
>>>> Sent: 17 February 2012 19:53
>>>> To: Nathan Wilhelmi
>>>> Cc: go-essp-tech at ucar.edu
>>>> Subject: Re: [Go-essp-tech] TDS Authorization issues
>>>>
>>>> Hi Nate,
>>>> 	 it should be esgf-security.git. But again, my advice is to wait till you can install the same software as al the other CMIP5 data nodes - it should be early next week.
>>>> thanks, L
>>>>
>>>> On Feb 17, 2012, at 11:14 AM, Nathan Wilhelmi wrote:
>>>>
>>>>> Hi Luca,
>>>>>
>>>>> Is esg-security available? esg-security.git doesn't seem to be available?
>>>>>
>>>>> Thanks!
>>>>> -Nate
>>>>>
>>>>> On 2/17/2012 11:04 AM, Cinquini, Luca (3880) wrote:
>>>>>> Hi Nate,
>>>>>> that is some seriously old code you are using...
>>>>>>
>>>>>> The ORP is available from:
>>>>>>
>>>>>> git clone git at esgf.org:esg-orp.git
>>>>>>
>>>>>> but it looks like the version you are using is the very first tag there,
>>>>>> probably more than one year old.
>>>>>>
>>>>>> The esg-saml library does not even exist any more, it's been long
>>>>>> replaced by esgf-security.
>>>>>>
>>>>>> My suggestion would be to wait till next week when we will roll out a
>>>>>> data-node only installation, for use by all CMIP5 data nodes. That
>>>>>> should allow you to upgrade the UCAR TDS as a data-node only.
>>>>>>
>>>>>> thanks, Luca
>>>>>>
>>>>>> On Feb 17, 2012, at 10:46 AM, Nathan Wilhelmi wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> Version # mistake.
>>>>>>>
>>>>>>> We are looking for:
>>>>>>>
>>>>>>> esg-node.1.0.0.4.jar
>>>>>>> esg-node-common.1.0.0.4.jar
>>>>>>> esg-node-filters.1.0.0.4.jar
>>>>>>> esg-orp.1.1.2.2.jar
>>>>>>> esg-saml.1.2.1.jar
>>>>>>>
>>>>>>> Thanks!
>>>>>>> -Nate
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2/17/2012 10:38 AM, Nathan Wilhelmi wrote:
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> We upgraded our TDS to the newer no caching version this week. This
>>>>>>>> version uses much less RAM and is more responsive. However this appears
>>>>>>>> to have uncovered some issues with the authorization related node
>>>>>>>> components. At this point our TDS will only authorize datasets for a
>>>>>>>> short time after a restart. Our TDS will authorize reliably if only 1
>>>>>>>> thread is enabled in Tomcat, but this isn't really a workable solution.
>>>>>>>>
>>>>>>>> After some debugging and code review we believe the problem lies with
>>>>>>>> HttpClient usage patterns.
>>>>>>>>
>>>>>>>> To expedite getting our TDS back online and debugging the problem we are
>>>>>>>> trying to get a hold of the source for the node components we are using.
>>>>>>>> Where can we get the source for the following jars?
>>>>>>>>
>>>>>>>> esg-node.1.0.0.4.jar
>>>>>>>> esg-node-common.1.0.0.4.jar
>>>>>>>> esg-node-filters.1.0.0.4.jar
>>>>>>>> esg-orp.1.0.0.4.jar
>>>>>>>> esg-saml.1.0.0.4.jar
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> -Nate
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> GO-ESSP-TECH mailing list
>>>>>>>> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>>> _______________________________________________
>>>>>>> GO-ESSP-TECH mailing list
>>>>>>> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>> _______________________________________________
>>>> GO-ESSP-TECH mailing list
>>>> GO-ESSP-TECH at ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>> --
>>>> Scanned by iCritical.
>>> --
>>> Scanned by iCritical.
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

-- 
Estanislao Gonzalez

Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany

Phone:   +49 (40) 46 00 94-126
E-Mail:  gonzalez at dkrz.de