<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<html>

  <head>

    <meta content="text/html; charset=ISO-8859-1"

      http-equiv="Content-Type">

  </head>

  <body bgcolor="#ffffcc" text="#000000">

    Hans, <br>

    <br>

    I am not sure what to say... <br>

    The script command that I cut and pasted in the previous email

    clearly has the user's options coming after those that I hard coded,

    and thus indeed as you proved the last option should win.&nbsp; Are you

    sure you have the latest script?<br>

    <br>

    Hmmm.... give me a call some time tomorrow morning PDT and we can

    sort this out.<br>

    In the mean time, sanity check me and pull the latest from devel.&nbsp;

    I'd check myself but I have to get home now. :-)<br>

    <br>

    We will sort this out.<br>

    <br>

    On 12/8/11 5:44 PM, Hans Vahlenkamp wrote:

    <blockquote cite="mid:4EE16810.2070107@noaa.gov" type="cite">

      <pre wrap="">Hi Gavin,

After setting JAVA_OPTS in the shell environment, I see that the new options

are put before the options set by the esg-node script.

[root@data2 bin]# echo $JAVA_OPTS

-Xmx8G -Xms8G -XX:MaxPermSize=128m

[root@data2 bin]# ps -ef | grep jsvc

root      3845     1  0 20:24 ?        00:00:00 jsvc.exec -Djava.awt.headless=true -Dcatalina.home=/usr/local/tomcat -pidfile /var/run/tomcat-jsvc.pid -cp

/usr/local/apache-tomcat-6.0.32/bin/bootstrap.jar:/usr/local/apache-tomcat-6.0.32/bin/commons-daemon.jar:/usr/local/apache-tomcat-6.0.32/bin/tomcat-juli.jar -outfile /usr/local/tomcat/logs/catalina.out -errfile /usr/local/tomcat/logs/catalina.err -user tomcat -Xmx8G -Xms8G -XX:MaxPermSize=128m

-Xmx2048m -Xms1024m -XX:MaxPermSize=512m -Dsun.security.ssl.allowUnsafeRenegotiation=false org.apache.catalina.startup.Bootstrap

tomcat    3846  3845  8 20:24 ?        00:01:00 jsvc.exec -Djava.awt.headless=true -Dcatalina.home=/usr/local/tomcat -pidfile /var/run/tomcat-jsvc.pid -cp

/usr/local/apache-tomcat-6.0.32/bin/bootstrap.jar:/usr/local/apache-tomcat-6.0.32/bin/commons-daemon.jar:/usr/local/apache-tomcat-6.0.32/bin/tomcat-juli.jar -outfile /usr/local/tomcat/logs/catalina.out -errfile /usr/local/tomcat/logs/catalina.err -user tomcat -Xmx8G -Xms8G -XX:MaxPermSize=128m

-Xmx2048m -Xms1024m -XX:MaxPermSize=512m -Dsun.security.ssl.allowUnsafeRenegotiation=false org.apache.catalina.startup.Bootstrap

root      4005  3424  0 20:36 pts/4    00:00:00 grep jsvc

If the last option wins then the new options would not be used.  I confirmed

this by running the jmap command.  The MaxHeapSize stills shows as 2GB.

Hans

[root@data2 bin]# /usr/local/java/bin/jmap -heap 3846

Attaching to process ID 3846, please wait...

Debugger attached successfully.

Server compiler detected.

JVM version is 19.0-b09

using thread-local object allocation.

Parallel GC with 10 thread(s)

Heap Configuration:

   MinHeapFreeRatio = 40

   MaxHeapFreeRatio = 70

   MaxHeapSize      = 2147483648 (2048.0MB)

   NewSize          = 1310720 (1.25MB)

   MaxNewSize       = 17592186044415 MB

   OldSize          = 5439488 (5.1875MB)

   NewRatio         = 2

   SurvivorRatio    = 8

   PermSize         = 21757952 (20.75MB)

   MaxPermSize      = 536870912 (512.0MB)

Heap Usage:

PS Young Generation

Eden Space:

   capacity = 238682112 (227.625MB)

   used     = 46414704 (44.26451110839844MB)

   free     = 192267408 (183.36048889160156MB)

   19.4462432107187% used

>From Space:

   capacity = 238551040 (227.5MB)

   used     = 0 (0.0MB)

   free     = 238551040 (227.5MB)

   0.0% used

To Space:

   capacity = 238551040 (227.5MB)

   used     = 0 (0.0MB)

   free     = 238551040 (227.5MB)

   0.0% used

PS Old Generation

   capacity = 1431699456 (1365.375MB)

   used     = 1281466712 (1222.1018905639648MB)

   free     = 150232744 (143.27310943603516MB)

   89.50668428556% used

PS Perm Generation

   capacity = 106823680 (101.875MB)

   used     = 71632176 (68.31376647949219MB)

   free     = 35191504 (33.56123352050781MB)

   67.05645789398007% used

On 12/08/2011 12:00 PM, Gavin M. Bell wrote:

</pre>

      <blockquote type="cite">

        <pre wrap=""> Hi,

If you set JAVA_OPTS it will be respected.

Indeed java_opts does have a preset value... but then it also appends

whatever was specified for JAVA_OPTS...

Thus what you will see is two settings for java opts... but because of

the order of operations in how options are read, the last one wins...

the last ones being the ones that were specified by environment

JAVA_OPTS or by the variable java_opts.

You should not have to edit the script. :-)

    pushd $tomcat_install_dir &gt;&amp; /dev/null

    [ $((sel &amp; ALL_BIT)) != 0 ] &amp;&amp; *java_opts=" -Xmx2048m -Xms1024m

-XX:MaxPermSize=512m /$java_opts/"*

    jsvc_launch_command="JAVA_HOME=$java_install_dir

${tomcat_install_dir}/bin/jsvc -Djava.awt.headless=true

-Dcatalina.home=${tomcat_install_dir} -pidfile $tomcat_pid_file -cp

$(find $(readlink -f `pwd`/bin/) | grep jar | xargs | perl -pe 's/

/:/g') -outfile ${tomcat_install_dir}/logs/catalina.out -errfile

${tomcat_install_dir}/logs/catalina.err -user $tomcat_user $tomcat_opts

$java_opts -Dsun.security.ssl.allowUnsafeRenegotiation=false

org.apache.catalina.startup.Bootstrap"

    echo "$jsvc_launch_command"

    JAVA_HOME=${java_install_dir} ./bin/jsvc \

    -Djava.awt.headless=true \

    -Dcatalina.home=${tomcat_install_dir} \

    -pidfile ${tomcat_pid_file} \

    -cp $(find $(readlink -f `pwd`/bin/) | grep .jar | xargs | perl -pe

's/ /:/g') \

    -outfile ${tomcat_install_dir}/logs/catalina.out \

    -errfile ${tomcat_install_dir}/logs/catalina.err \

    -user $tomcat_user \

    ${tomcat_opts} \

    *$java_opts* -Dsun.security.ssl.allowUnsafeRenegotiation=false \

    org.apache.catalina.startup.Bootstrap

    if [ $? != 0 ]; then

    echo " ERROR: Could not start up tomcat"

    tail ./logs/catalina.err

    popd &gt;&amp; /dev/null

    checked_done 1

    fi

On 12/7/11 5:12 PM, Drach, Bob wrote:

</pre>

        <blockquote type="cite">

          <pre wrap="">Look in the esg-node script, in the start_tomcat routine. It may work to just add the options to the JAVA_OPTS environment variable, but it looks like the options are hardwired, so the surefire way is to edit the script.

Gavin: Is there a better way?

--Bob

________________________________________

From: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a> [<a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a>] On Behalf Of Hans Vahlenkamp [<a class="moz-txt-link-abbreviated" href="mailto:Hans.Vahlenkamp@noaa.gov">Hans.Vahlenkamp@noaa.gov</a>]

Sent: Wednesday, December 07, 2011 5:02 PM

To: Estanislao Gonzalez

Cc: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>

Subject: Re: [Go-essp-tech] Error publishing large dataset

Hello Estani,

Thanks for the advice, it looks promising.  One question...  In what file

should the "-Xmx8G -Xms8G -XX:MaxPermSize=128m" options be put?

Hans

-------- Original Message --------

Subject: Re: [Go-essp-tech] Error publishing large dataset

Date: Wed, 07 Dec 2011 12:58:13 +0100

From: Estanislao Gonzalez <a class="moz-txt-link-rfc2396E" href="mailto:gonzalez@dkrz.de">&lt;gonzalez@dkrz.de&gt;</a>

Organization: DKRZ

To: Drach, Bob <a class="moz-txt-link-rfc2396E" href="mailto:drach1@llnl.gov">&lt;drach1@llnl.gov&gt;</a>

CC: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a> <a class="moz-txt-link-rfc2396E" href="mailto:go-essp-tech@ucar.edu">&lt;go-essp-tech@ucar.edu&gt;</a>, Hans Vahlenkamp

<a class="moz-txt-link-rfc2396E" href="mailto:Hans.Vahlenkamp@noaa.gov">&lt;Hans.Vahlenkamp@noaa.gov&gt;</a>

Hi,

just one short comment on memory. The Java Virtual Machine (JVM) does

not allocate memory completely dynamically, so this parameters should be

tuned by hand.

If there's nothing else running on the machine, it's pretty safe to say

you can use 8G for the TDS. In that case use:

-Xmx8G -Xms8G -XX:MaxPermSize=128m

The Xmx is the maximal memory the JVM can grow to, the Xms the minimal.

By setting both to the same level, the Garbage collector will have

enough room to work on idle only and thus the speed of the JVM should be

boosted. The PermSize is where Java classes (not objects) are kept, this

should be enough.

And you should be fine. It is very important to setup this values, the

default of the JVM are just 64MB, and the default node installation 2G,

which it is quite low for a "normal" CMIP5 data node.

Hope this helps,

Estani

Am 07.12.2011 01:41, schrieb Drach, Bob:

</pre>

          <blockquote type="cite">

            <pre wrap="">Hi Hans, Sergey,

That error will occur if the gateway tries to reach the GFDL data node to read the catalog, but fails either because the server is down or is inaccessible. Given that your TDS was running into memory issues, it was probably that the TDS was unavailable.

I would try to back out of the latest TDS catalog update. Assuming that ONLY the most recently published datasets are in 'map_file', you can remove the catalogs with

esgunpublish --map map_file --skip-gateway

The above command will remove the catalogs related to 'map_file' and regenerate a top-level catalog. It will try (and fail) to reinit TDS. But if you then restart tomcat, the TDS may recover.

If that doesn't work, you may have to manually edit the top-level THREDDS catalog:

/usr/local/tomcat/content/thredds/esgcet/catalog.xml

Remove the links to the most recently published datasets, then restart tomcat.

Also consider giving max heap memory to Java. We include (in the jsvc command to start tomcat)

-Xmx2560m -Xms2560m -XX:MaxPermSize=128m

depending on your system you may be able to specify more memory.

--Bob

________________________________________

From: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a> [<a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a>] On Behalf Of Hans Vahlenkamp [<a class="moz-txt-link-abbreviated" href="mailto:Hans.Vahlenkamp@noaa.gov">Hans.Vahlenkamp@noaa.gov</a>]

Sent: Tuesday, December 06, 2011 4:12 PM

To: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>

Subject: [Go-essp-tech] Error publishing large dataset

Hi all,

Did anybody encountered with difficulties to publish large set of files in one

pool?

We had a problem when tried to make  about 30000 files at once.  We could write

them in db and configure TDS (esgpublish --map map_file --project cmip5;

esgpublish --map map_file --project cmip5 --noscan --thredds) without problems.

But on last stage - publishing on gateway - we've got en error just at the

beginning of process (see at the bottom).

After that jsvc process became to consume 100% of CPU and TDS was barely

responsive. Restarting ESG node didn't help. TDS is completely dead with 404

error. Also tomcat gives severe errors about possible memory leak.

Here are all the entries in the "catalina.err" log from one startup of the

ESG data node services.  Note the problems on the lines with "SEVERE:".  Are

there any suggestions about how to make TDS functional again?

Also, our ESG data node is running on a machine with 64-bit RHEL 5 and 24GB

of memory.  So, the out of memory error is rather unexpected...

Thanks,

Sergey Nikonov

Hans Vahlenkamp

GFDL Data Portal

Error from publisher script (esgpublish --map map_file --project cmip5 --noscan

--publish):

===================================================================================

INFO       2011-12-06 10:25:01,537 Publishing:

cmip5.output1.NOAA-GFDL.GFDL-ESM2M.historicalMisc.3hr.atmos.3hr.r1i1p2, parent =

pcmdi.GFDL

Traceback (most recent call last):

   File "/usr/local/cdat/bin/esgpublish", line 5, in&lt;module&gt;

     pkg_resources.run_script('esgcet==2.9.0', 'esgpublish')

   File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in run_script

   File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in run_script

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-INFO/scripts/esgpublish",

line 461, in&lt;module&gt;

     main(sys.argv[1:])

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-INFO/scripts/esgpublish",

line 447, in main

     result = publishDatasetList(datasetNames, Session, publish=publish,

thredds=thredds, las=las, parentId=parent, service=service, perVariable=perVariable)

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/publish.py",

line 308, in publishDatasetList

     dset, statusId, state, evname, status = publishDataset(datasetName,

parentIdent, service, threddsRootURL, session)

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/publish.py",

line 118, in publishDataset

     statusId = service.createDataset(parentId, threddsURL, -1, "Published")

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/hessianlib.py",

line 426, in __call__

     return self._invoker(self._method, args)

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/hessianlib.py",

line 543, in __invoke

     return self.parse_response(responseProxy)

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/hessianlib.py",

line 554, in parse_response

     raise value

esgcet.publish.hessianlib.RemoteCallException: Java ServiceException:

PublishingOperation must define at least 1 PublishingTask

         at org.springframework.util.Assert.notEmpty(Assert.java:268)

         at

sgf.gateway.metadata.complete.service.impl.PublishingOperationExectionServiceImpl.handleExecute(PublishingOperationExectionServiceImpl.java:158)

         at

sgf.gateway.metadata.complete.service.impl.PublishingOperationExectionServiceImpl.execute(PublishingOperationExectionServiceImpl.java:127)

         at

sgf.gateway.service.publishing.impl.spring.PublishingServiceImpl.publishThreddsCatalog(PublishingServiceImpl.java:141)

         at

sgf.gateway.metadata.harvest.thredds.RemotePublishingWrapper.publishThreddsCatalogFromDatasetWithPermission(RemotePublishingWrapper.java:273)

         at

sgf.gateway.metadata.harvest.thredds.RemotePublishingWrapper.publishThreddsCatalogFromDatasetWithPermission(RemotePublishingWrapper.java:251)

         at

sgf.gateway.service.publishing.impl.spring.RemotePublishingServiceImpl.createOrUpdateDataset(RemotePublishingServiceImpl.java:99)

         at

sgf.gateway.service.publishing.impl.spring.RemotePublishingServiceImpl.createDataset(RemotePublishingServiceImpl.java:84)

         at

sgf.gateway.service.publishing.impl.spring.RemotePublishingServiceSecurityProxyImpl.createDataset(RemotePublishingServiceSecurityProxyImpl.java:67)

         at sun.reflect.GeneratedMethodAccessor581.invoke(None:-1)

         at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

         at java.lang.reflect.Method.invoke(Method.java:597)

         at

org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)

         at

org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

         at

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

         at

org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)

         at

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

         at

org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

         at $Proxy25.createDataset(None:-1)

         at sun.reflect.GeneratedMethodAccessor581.invoke(None:-1)

         at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

         at java.lang.reflect.Method.invoke(Method.java:597)

         at

com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:155)

         at

org.springframework.remoting.caucho.HessianExporter.doInvoke(HessianExporter.java:198)

         at

org.springframework.remoting.caucho.HessianExporter.invoke(HessianExporter.java:118)

         at

org.springframework.remoting.caucho.HessianServiceExporter.handleRequest(HessianServiceExporter.java:66)

         at

org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:49)

         at

org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)

         at

org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)

         at

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)

         at

org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560)

         at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

         at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

         at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

         at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:366)

         at

org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)

         at

org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:187)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:187)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:89)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:109)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:167)

         at

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

         at

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

         at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

         at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

         at

org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198)

         at

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

         at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

         at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

         at

sgf.gateway.web.filters.exception.UnhandledExceptionFilter.doFilter(UnhandledExceptionFilter.java:99)

         at

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

         at

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

         at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

         at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

         at

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

         at

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

         at

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)

         at

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

         at

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

         at

org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:276)

         at

org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81)

         at

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

         at

org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)

         at

org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)

         at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)

         at java.lang.Thread.run(Thread.java:619)

=================================================================================

Error from catalina.err:

============================================================================

Dec 6, 2011 5:35:04 PM org.apache.catalina.core.AprLifecycleListener init

INFO: The APR based Apache Tomcat Native library which allows optimal

performance in production environments was not found on the java.library.path:

/usr/local/openssl/lib:/usr/local/curl/lib:/usr/local/pgsql/lib:/usr/local/cdat/Externals/lib:/usr/local/globus/lib:/usr/local/openssl/lib:/usr/local/curl/lib:/usr/local/pgsql/lib:/usr/local/cdat/Externals/lib:/usr/local/globus/lib:/usr/local/openssl/lib:/usr/local/curl/lib:/usr/local/pgsql/lib:/usr/local/cdat/Externals/lib:/usr/local/globus/lib:/usr/local/cdat/Externals/lib:/usr/local/pgsql/lib:.:/usr/local/java/jre/lib/amd64/server:/usr/local/java/jre/lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

Dec 6, 2011 5:35:04 PM org.apache.coyote.http11.Http11Protocol init

INFO: Initializing Coyote HTTP/1.1 on http-80

Dec 6, 2011 5:35:05 PM org.apache.coyote.http11.Http11Protocol init

INFO: Initializing Coyote HTTP/1.1 on http-443

Dec 6, 2011 5:35:05 PM org.apache.catalina.startup.Catalina load

INFO: Initialization processed in 966 ms

Dec 6, 2011 5:35:05 PM org.apache.catalina.realm.JAASRealm setContainer

INFO: Set JAAS app name Catalina

Dec 6, 2011 5:35:05 PM org.apache.catalina.core.StandardService start

INFO: Starting service Catalina

Dec 6, 2011 5:35:05 PM org.apache.catalina.core.StandardEngine start

INFO: Starting Servlet Engine: Apache Tomcat/6.0.32

Dec 6, 2011 5:35:05 PM org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor thredds.xml

Dec 6, 2011 5:35:05 PM org.apache.catalina.core.ApplicationContext log

INFO: Initializing Spring root WebApplicationContext

Dec 6, 2011 5:35:06 PM org.apache.catalina.core.ApplicationContext log

INFO: Set web app root system property: 'webapp.root' =

[/usr/local/tomcat/webapps/thredds/]

Dec 6, 2011 5:35:06 PM org.apache.catalina.core.ApplicationContext log

INFO: Initializing log4j from [/usr/local/tomcat/webapps/thredds/WEB-INF/log4j.xml]

Dec 6, 2011 5:39:08 PM org.apache.catalina.core.StandardContext listenerStart

SEVERE: Exception sending context initialized event to listener instance of

class thredds.server.config.TdsConfigContextListener

java.lang.OutOfMemoryError: GC overhead limit exceeded

Dec 6, 2011 5:39:12 PM org.apache.catalina.core.StandardContext start

SEVERE: Error listenerStart

Dec 6, 2011 5:39:12 PM org.apache.catalina.core.StandardContext start

SEVERE: Context [/thredds] startup failed due to previous errors

Dec 6, 2011 5:39:12 PM org.apache.catalina.core.StandardContext listenerStop

SEVERE: Exception sending context destroyed event to listener instance of class

thredds.server.config.TdsConfigContextListener

java.lang.IllegalStateException: setInstance() must be called first.

         at thredds.servlet.DataRootHandler.getInstance(DataRootHandler.java:110)

         at

thredds.server.config.TdsConfigContextListener.contextDestroyed(TdsConfigContextListener.java:93)

         at

org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4244)

         at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4879)

         at

org.apache.catalina.core.StandardContext.start(StandardContext.java:4749)

         at

org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)

         at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)

         at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)

         at

org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)

         at

org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)

         at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)

         at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)

         at

org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)

         at

org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)

         at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)

         at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)

         at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)

         at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)

         at org.apache.catalina.core.StandardService.start(StandardService.java:525)

         at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)

         at org.apache.catalina.startup.Catalina.start(Catalina.java:595)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

         at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

         at java.lang.reflect.Method.invoke(Method.java:597)

         at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

         at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

         at java.lang.reflect.Method.invoke(Method.java:597)

         at

org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:219)

Dec 6, 2011 5:39:12 PM org.apache.catalina.core.ApplicationContext log

INFO: Closing Spring root WebApplicationContext

Dec 6, 2011 5:40:17 PM org.apache.catalina.loader.WebappClassLoader

clearReferencesThreads

SEVERE: The web application [/thredds] appears to have started a thread named

[TdsScheduler_Worker-1] but has failed to stop it. This is very likely to create

a memory leak.

Dec 6, 2011 5:40:17 PM org.apache.catalina.loader.WebappClassLoader

clearReferencesThreads

SEVERE: The web application [/thredds] appears to have started a thread named

[TdsScheduler_Worker-2] but has failed to stop it. This is very likely to create

a memory leak.

Dec 6, 2011 5:40:17 PM org.apache.catalina.loader.WebappClassLoader

clearReferencesThreads

SEVERE: The web application [/thredds] appears to have started a thread named

[TdsScheduler_Worker-3] but has failed to stop it. This is very likely to create

a memory leak.

Dec 6, 2011 5:40:25 PM org.apache.catalina.loader.WebappClassLoader

clearReferencesThreads

SEVERE: The web application [/thredds] appears to have started a thread named

[Timer-0] but has failed to stop it. This is very likely to create a memory leak.

Dec 6, 2011 5:40:25 PM org.apache.catalina.loader.WebappClassLoader

clearReferencesThreads

SEVERE: The web application [/thredds] appears to have started a thread named

[TdsScheduler_QuartzSchedulerThread] but has failed to stop it. This is very

likely to create a memory leak.

Dec 6, 2011 5:40:25 PM org.apache.catalina.loader.WebappClassLoader

clearThreadLocalMap

SEVERE: The web application [/thredds] created a ThreadLocal with key of type

[org.apache.log4j.helpers.ThreadLocalMap] (value

[org.apache.log4j.helpers.ThreadLocalMap@6106dc2d]) and a value of type

[java.util.Hashtable] (value [{startTime=1323211152783, ID=2}]) but failed to

remove it when the web

application was stopped. This is very likely to create a memory leak.

Dec 6, 2011 5:40:25 PM org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor manager.xml

Dec 6, 2011 5:40:26 PM org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor host-manager.xml

Dec 6, 2011 5:40:26 PM org.apache.catalina.startup.HostConfig deployWAR

INFO: Deploying web application archive las.war

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity   [info] VelocityViewServlet: Using custom properties at

'/WEB-INF/velocity.properties'

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  :

*******************************************************************

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] Starting Apache Velocity v1.5 (compiled: 2007-02-22

08:52:29)

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  : RuntimeInstance initializing.

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] Default Properties File:

org/apache/velocity/runtime/defaults/velocity.properties

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] Trying to use logger class

org.apache.velocity.tools.view.servlet.ServletLogger

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity   [info] LogSystem has been deprecated. Please use a LogChute

implementation.

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] Default ResourceManager initializing. (class

org.apache.velocity.runtime.resource.ResourceManagerImpl)

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] ResourceLoader instantiated:

org.apache.velocity.tools.view.servlet.WebappLoader

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] WebappLoader : initialization starting.

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] WebappLoader : initialization complete.

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] ResourceCache: initialized (class

org.apache.velocity.runtime.resource.ResourceCacheImpl)

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  : Default ResourceManager initialization complete.

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log