<html>

  <head>

    <meta content="text/html; charset=ISO-8859-1"

      http-equiv="Content-Type">

  </head>

  <body bgcolor="#FFFFFF" text="#000000">

    indeed,<br>

    <br>

    the JAVA_OPTS option should have been corrected, as previously it

    wasn't honored. The 2G option is added always so if your version is

    still broken (or this wasn't corrected at al)l you'll have to change

    the script... which means also "always after an upgrade".<br>

    <br>

    Let's assume it's corrected :-) :<br>

    export JAVA_OPTS='-Xmx8G -Xms8G -XX:MaxPermSize=128m'<br>

    esg-node restart<br>

    <br>

    check the java process via:<br>

    &nbsp;$ sudo jps<br>

    30722 Jps<br>

    28432<br>

    <br>

    or<br>

    $pgrep -u tomcat<br>

    28432<br>

    <br>

    And then us jmap to get a dump of the current status:<br>

    $ sudo jmap -heap 28432<br>

    Attaching to process ID 28432, please wait...<br>

    Debugger attached successfully.<br>

    Server compiler detected.<br>

    JVM version is 19.0-b09<br>

    <br>

    using thread-local object allocation.<br>

    Parallel GC with 8 thread(s)<br>

    <br>

    Heap Configuration:<br>

    &nbsp;&nbsp; MinHeapFreeRatio = 40<br>

    &nbsp;&nbsp; MaxHeapFreeRatio = 70<br>

    <b>&nbsp;&nbsp; MaxHeapSize&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = 8589934592 (8192.0MB)</b><br>

    &nbsp;&nbsp; NewSize&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = 1310720 (1.25MB)<br>

    &nbsp;&nbsp; MaxNewSize&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = 17592186044415 MB<br>

    &nbsp;&nbsp; OldSize&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = 5439488 (5.1875MB)<br>

    &nbsp;&nbsp; NewRatio&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = 2<br>

    &nbsp;&nbsp; SurvivorRatio&nbsp;&nbsp;&nbsp; = 8<br>

    &nbsp;&nbsp; PermSize&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = 21757952 (20.75MB)<br>

    &nbsp;&nbsp; MaxPermSize&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = 536870912 (512.0MB)<br>

    <br>

    Heap Usage:<br>

    PS Young Generation<br>

    Eden Space:<br>

    &nbsp;&nbsp; capacity = 954466304 (910.25MB)<br>

    &nbsp;&nbsp; used&nbsp;&nbsp;&nbsp;&nbsp; = 618087896 (589.4545516967773MB)<br>

    &nbsp;&nbsp; free&nbsp;&nbsp;&nbsp;&nbsp; = 336378408 (320.79544830322266MB)<br>

    &nbsp;&nbsp; 64.75743495707523% used<br>

    From Space:<br>

    &nbsp;&nbsp; capacity = 954400768 (910.1875MB)<br>

    &nbsp;&nbsp; used&nbsp;&nbsp;&nbsp;&nbsp; = 49982728 (47.66724395751953MB)<br>

    &nbsp;&nbsp; free&nbsp;&nbsp;&nbsp;&nbsp; = 904418040 (862.5202560424805MB)<br>

    &nbsp;&nbsp; 5.237079608049938% used<br>

    To Space:<br>

    &nbsp;&nbsp; capacity = 954400768 (910.1875MB)<br>

    &nbsp;&nbsp; used&nbsp;&nbsp;&nbsp;&nbsp; = 0 (0.0MB)<br>

    &nbsp;&nbsp; free&nbsp;&nbsp;&nbsp;&nbsp; = 954400768 (910.1875MB)<br>

    &nbsp;&nbsp; 0.0% used<br>

    PS Old Generation<br>

    &nbsp;&nbsp; capacity = 5726666752 (5461.375MB)<br>

    &nbsp;&nbsp; used&nbsp;&nbsp;&nbsp;&nbsp; = 4270949824 (4073.0951538085938MB)<br>

    &nbsp;&nbsp; free&nbsp;&nbsp;&nbsp;&nbsp; = 1455716928 (1388.2798461914062MB)<br>

    &nbsp;&nbsp; 74.58003073966893% used<br>

    PS Perm Generation<br>

    &nbsp;&nbsp; capacity = 74055680 (70.625MB)<br>

    &nbsp;&nbsp; used&nbsp;&nbsp;&nbsp;&nbsp; = 73915776 (70.4915771484375MB)<br>

    &nbsp;&nbsp; free&nbsp;&nbsp;&nbsp;&nbsp; = 139904 (0.1334228515625MB)<br>

    &nbsp;&nbsp; 99.81108268805309% used<br>

    <br>

    Besides giving you a nice view of the JVM memory status, you'll see

    what's the current max heap size of a running JVM. If it's not what

    you expect, you'll have to manually set it in:<br>

    /usr/local/bin/esg-node:1834:&nbsp;&nbsp;&nbsp; [ $((sel &amp; ALL_BIT)) != 0 ]&nbsp;&nbsp;&nbsp;&nbsp;

    &amp;&amp; java_opts+=' -Xmx8192m -Xms8192m -XX:MaxPermSize=512m'<br>

    <br>

    Hope this helps,<br>

    Estani<br>

    <br>

    Am 08.12.2011 02:12, schrieb Drach, Bob:

    <blockquote

cite="mid:73FA9659F569E34E960EE31FC4D2D45601DCE39E9D10@NSPEXMBX-B.the-lab.llnl.gov"

      type="cite">

      <pre wrap="">Look in the esg-node script, in the start_tomcat routine. It may work to just add the options to the JAVA_OPTS environment variable, but it looks like the options are hardwired, so the surefire way is to edit the script.

Gavin: Is there a better way?

--Bob

________________________________________

From: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a> [<a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a>] On Behalf Of Hans Vahlenkamp [<a class="moz-txt-link-abbreviated" href="mailto:Hans.Vahlenkamp@noaa.gov">Hans.Vahlenkamp@noaa.gov</a>]

Sent: Wednesday, December 07, 2011 5:02 PM

To: Estanislao Gonzalez

Cc: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>

Subject: Re: [Go-essp-tech] Error publishing large dataset

Hello Estani,

Thanks for the advice, it looks promising.  One question...  In what file

should the "-Xmx8G -Xms8G -XX:MaxPermSize=128m" options be put?

Hans

-------- Original Message --------

Subject: Re: [Go-essp-tech] Error publishing large dataset

Date: Wed, 07 Dec 2011 12:58:13 +0100

From: Estanislao Gonzalez <a class="moz-txt-link-rfc2396E" href="mailto:gonzalez@dkrz.de">&lt;gonzalez@dkrz.de&gt;</a>

Organization: DKRZ

To: Drach, Bob <a class="moz-txt-link-rfc2396E" href="mailto:drach1@llnl.gov">&lt;drach1@llnl.gov&gt;</a>

CC: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a> <a class="moz-txt-link-rfc2396E" href="mailto:go-essp-tech@ucar.edu">&lt;go-essp-tech@ucar.edu&gt;</a>, Hans Vahlenkamp

<a class="moz-txt-link-rfc2396E" href="mailto:Hans.Vahlenkamp@noaa.gov">&lt;Hans.Vahlenkamp@noaa.gov&gt;</a>

Hi,

just one short comment on memory. The Java Virtual Machine (JVM) does

not allocate memory completely dynamically, so this parameters should be

tuned by hand.

If there's nothing else running on the machine, it's pretty safe to say

you can use 8G for the TDS. In that case use:

-Xmx8G -Xms8G -XX:MaxPermSize=128m

The Xmx is the maximal memory the JVM can grow to, the Xms the minimal.

By setting both to the same level, the Garbage collector will have

enough room to work on idle only and thus the speed of the JVM should be

boosted. The PermSize is where Java classes (not objects) are kept, this

should be enough.

And you should be fine. It is very important to setup this values, the

default of the JVM are just 64MB, and the default node installation 2G,

which it is quite low for a "normal" CMIP5 data node.

Hope this helps,

Estani

Am 07.12.2011 01:41, schrieb Drach, Bob:

</pre>

      <blockquote type="cite">

        <pre wrap="">Hi Hans, Sergey,

That error will occur if the gateway tries to reach the GFDL data node to read the catalog, but fails either because the server is down or is inaccessible. Given that your TDS was running into memory issues, it was probably that the TDS was unavailable.

I would try to back out of the latest TDS catalog update. Assuming that ONLY the most recently published datasets are in 'map_file', you can remove the catalogs with

esgunpublish --map map_file --skip-gateway

The above command will remove the catalogs related to 'map_file' and regenerate a top-level catalog. It will try (and fail) to reinit TDS. But if you then restart tomcat, the TDS may recover.

If that doesn't work, you may have to manually edit the top-level THREDDS catalog:

/usr/local/tomcat/content/thredds/esgcet/catalog.xml

Remove the links to the most recently published datasets, then restart tomcat.

Also consider giving max heap memory to Java. We include (in the jsvc command to start tomcat)

-Xmx2560m -Xms2560m -XX:MaxPermSize=128m

depending on your system you may be able to specify more memory.

--Bob

________________________________________

From: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a> [<a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a>] On Behalf Of Hans Vahlenkamp [<a class="moz-txt-link-abbreviated" href="mailto:Hans.Vahlenkamp@noaa.gov">Hans.Vahlenkamp@noaa.gov</a>]

Sent: Tuesday, December 06, 2011 4:12 PM

To: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>

Subject: [Go-essp-tech] Error publishing large dataset

Hi all,

Did anybody encountered with difficulties to publish large set of files in one

pool?

We had a problem when tried to make  about 30000 files at once.  We could write

them in db and configure TDS (esgpublish --map map_file --project cmip5;

esgpublish --map map_file --project cmip5 --noscan --thredds) without problems.

But on last stage - publishing on gateway - we've got en error just at the

beginning of process (see at the bottom).

After that jsvc process became to consume 100% of CPU and TDS was barely

responsive. Restarting ESG node didn't help. TDS is completely dead with 404

error. Also tomcat gives severe errors about possible memory leak.

Here are all the entries in the "catalina.err" log from one startup of the

ESG data node services.  Note the problems on the lines with "SEVERE:".  Are

there any suggestions about how to make TDS functional again?

Also, our ESG data node is running on a machine with 64-bit RHEL 5 and 24GB

of memory.  So, the out of memory error is rather unexpected...

Thanks,

Sergey Nikonov

Hans Vahlenkamp

GFDL Data Portal

Error from publisher script (esgpublish --map map_file --project cmip5 --noscan

--publish):

===================================================================================

INFO       2011-12-06 10:25:01,537 Publishing:

cmip5.output1.NOAA-GFDL.GFDL-ESM2M.historicalMisc.3hr.atmos.3hr.r1i1p2, parent =

pcmdi.GFDL

Traceback (most recent call last):

   File "/usr/local/cdat/bin/esgpublish", line 5, in&lt;module&gt;

     pkg_resources.run_script('esgcet==2.9.0', 'esgpublish')

   File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in run_script

   File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in run_script

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-INFO/scripts/esgpublish",

line 461, in&lt;module&gt;

     main(sys.argv[1:])

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-INFO/scripts/esgpublish",

line 447, in main

     result = publishDatasetList(datasetNames, Session, publish=publish,

thredds=thredds, las=las, parentId=parent, service=service, perVariable=perVariable)

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/publish.py",

line 308, in publishDatasetList

     dset, statusId, state, evname, status = publishDataset(datasetName,

parentIdent, service, threddsRootURL, session)

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/publish.py",

line 118, in publishDataset

     statusId = service.createDataset(parentId, threddsURL, -1, "Published")

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/hessianlib.py",

line 426, in __call__

     return self._invoker(self._method, args)

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/hessianlib.py",

line 543, in __invoke

     return self.parse_response(responseProxy)

   File

"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/hessianlib.py",

line 554, in parse_response

     raise value

esgcet.publish.hessianlib.RemoteCallException: Java ServiceException:

PublishingOperation must define at least 1 PublishingTask

         at org.springframework.util.Assert.notEmpty(Assert.java:268)

         at

sgf.gateway.metadata.complete.service.impl.PublishingOperationExectionServiceImpl.handleExecute(PublishingOperationExectionServiceImpl.java:158)

         at

sgf.gateway.metadata.complete.service.impl.PublishingOperationExectionServiceImpl.execute(PublishingOperationExectionServiceImpl.java:127)

         at

sgf.gateway.service.publishing.impl.spring.PublishingServiceImpl.publishThreddsCatalog(PublishingServiceImpl.java:141)

         at

sgf.gateway.metadata.harvest.thredds.RemotePublishingWrapper.publishThreddsCatalogFromDatasetWithPermission(RemotePublishingWrapper.java:273)

         at

sgf.gateway.metadata.harvest.thredds.RemotePublishingWrapper.publishThreddsCatalogFromDatasetWithPermission(RemotePublishingWrapper.java:251)

         at

sgf.gateway.service.publishing.impl.spring.RemotePublishingServiceImpl.createOrUpdateDataset(RemotePublishingServiceImpl.java:99)

         at

sgf.gateway.service.publishing.impl.spring.RemotePublishingServiceImpl.createDataset(RemotePublishingServiceImpl.java:84)

         at

sgf.gateway.service.publishing.impl.spring.RemotePublishingServiceSecurityProxyImpl.createDataset(RemotePublishingServiceSecurityProxyImpl.java:67)

         at sun.reflect.GeneratedMethodAccessor581.invoke(None:-1)

         at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

         at java.lang.reflect.Method.invoke(Method.java:597)

         at

org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)

         at

org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

         at

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

         at

org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)

         at

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

         at

org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

         at $Proxy25.createDataset(None:-1)

         at sun.reflect.GeneratedMethodAccessor581.invoke(None:-1)

         at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

         at java.lang.reflect.Method.invoke(Method.java:597)

         at

com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:155)

         at

org.springframework.remoting.caucho.HessianExporter.doInvoke(HessianExporter.java:198)

         at

org.springframework.remoting.caucho.HessianExporter.invoke(HessianExporter.java:118)

         at

org.springframework.remoting.caucho.HessianServiceExporter.handleRequest(HessianServiceExporter.java:66)

         at

org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:49)

         at

org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)

         at

org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)

         at

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)

         at

org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560)

         at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

         at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

         at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

         at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:366)

         at

org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)

         at

org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:187)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:187)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:89)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:109)

         at

org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

         at

org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:167)

         at

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

         at

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

         at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

         at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

         at

org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198)

         at

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

         at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

         at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

         at

sgf.gateway.web.filters.exception.UnhandledExceptionFilter.doFilter(UnhandledExceptionFilter.java:99)

         at

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

         at

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

         at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

         at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

         at

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

         at

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

         at

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)

         at

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

         at

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

         at

org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:276)

         at

org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81)

         at

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

         at

org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)

         at

org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)

         at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)

         at java.lang.Thread.run(Thread.java:619)

=================================================================================

Error from catalina.err:

============================================================================

Dec 6, 2011 5:35:04 PM org.apache.catalina.core.AprLifecycleListener init

INFO: The APR based Apache Tomcat Native library which allows optimal

performance in production environments was not found on the java.library.path:

/usr/local/openssl/lib:/usr/local/curl/lib:/usr/local/pgsql/lib:/usr/local/cdat/Externals/lib:/usr/local/globus/lib:/usr/local/openssl/lib:/usr/local/curl/lib:/usr/local/pgsql/lib:/usr/local/cdat/Externals/lib:/usr/local/globus/lib:/usr/local/openssl/lib:/usr/local/curl/lib:/usr/local/pgsql/lib:/usr/local/cdat/Externals/lib:/usr/local/globus/lib:/usr/local/cdat/Externals/lib:/usr/local/pgsql/lib:.:/usr/local/java/jre/lib/amd64/server:/usr/local/java/jre/lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

Dec 6, 2011 5:35:04 PM org.apache.coyote.http11.Http11Protocol init

INFO: Initializing Coyote HTTP/1.1 on http-80

Dec 6, 2011 5:35:05 PM org.apache.coyote.http11.Http11Protocol init

INFO: Initializing Coyote HTTP/1.1 on http-443

Dec 6, 2011 5:35:05 PM org.apache.catalina.startup.Catalina load

INFO: Initialization processed in 966 ms

Dec 6, 2011 5:35:05 PM org.apache.catalina.realm.JAASRealm setContainer

INFO: Set JAAS app name Catalina

Dec 6, 2011 5:35:05 PM org.apache.catalina.core.StandardService start

INFO: Starting service Catalina

Dec 6, 2011 5:35:05 PM org.apache.catalina.core.StandardEngine start

INFO: Starting Servlet Engine: Apache Tomcat/6.0.32

Dec 6, 2011 5:35:05 PM org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor thredds.xml

Dec 6, 2011 5:35:05 PM org.apache.catalina.core.ApplicationContext log

INFO: Initializing Spring root WebApplicationContext

Dec 6, 2011 5:35:06 PM org.apache.catalina.core.ApplicationContext log

INFO: Set web app root system property: 'webapp.root' =

[/usr/local/tomcat/webapps/thredds/]

Dec 6, 2011 5:35:06 PM org.apache.catalina.core.ApplicationContext log

INFO: Initializing log4j from [/usr/local/tomcat/webapps/thredds/WEB-INF/log4j.xml]

Dec 6, 2011 5:39:08 PM org.apache.catalina.core.StandardContext listenerStart

SEVERE: Exception sending context initialized event to listener instance of

class thredds.server.config.TdsConfigContextListener

java.lang.OutOfMemoryError: GC overhead limit exceeded

Dec 6, 2011 5:39:12 PM org.apache.catalina.core.StandardContext start

SEVERE: Error listenerStart

Dec 6, 2011 5:39:12 PM org.apache.catalina.core.StandardContext start

SEVERE: Context [/thredds] startup failed due to previous errors

Dec 6, 2011 5:39:12 PM org.apache.catalina.core.StandardContext listenerStop

SEVERE: Exception sending context destroyed event to listener instance of class

thredds.server.config.TdsConfigContextListener

java.lang.IllegalStateException: setInstance() must be called first.

         at thredds.servlet.DataRootHandler.getInstance(DataRootHandler.java:110)

         at

thredds.server.config.TdsConfigContextListener.contextDestroyed(TdsConfigContextListener.java:93)

         at

org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4244)

         at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4879)

         at

org.apache.catalina.core.StandardContext.start(StandardContext.java:4749)

         at

org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)

         at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)

         at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)

         at

org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)

         at

org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)

         at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)

         at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)

         at

org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)

         at

org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)

         at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)

         at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)

         at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)

         at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)

         at org.apache.catalina.core.StandardService.start(StandardService.java:525)

         at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)

         at org.apache.catalina.startup.Catalina.start(Catalina.java:595)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

         at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

         at java.lang.reflect.Method.invoke(Method.java:597)

         at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

         at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

         at java.lang.reflect.Method.invoke(Method.java:597)

         at

org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:219)

Dec 6, 2011 5:39:12 PM org.apache.catalina.core.ApplicationContext log

INFO: Closing Spring root WebApplicationContext

Dec 6, 2011 5:40:17 PM org.apache.catalina.loader.WebappClassLoader

clearReferencesThreads

SEVERE: The web application [/thredds] appears to have started a thread named

[TdsScheduler_Worker-1] but has failed to stop it. This is very likely to create

a memory leak.

Dec 6, 2011 5:40:17 PM org.apache.catalina.loader.WebappClassLoader

clearReferencesThreads

SEVERE: The web application [/thredds] appears to have started a thread named

[TdsScheduler_Worker-2] but has failed to stop it. This is very likely to create

a memory leak.

Dec 6, 2011 5:40:17 PM org.apache.catalina.loader.WebappClassLoader

clearReferencesThreads

SEVERE: The web application [/thredds] appears to have started a thread named

[TdsScheduler_Worker-3] but has failed to stop it. This is very likely to create

a memory leak.

Dec 6, 2011 5:40:25 PM org.apache.catalina.loader.WebappClassLoader

clearReferencesThreads

SEVERE: The web application [/thredds] appears to have started a thread named

[Timer-0] but has failed to stop it. This is very likely to create a memory leak.

Dec 6, 2011 5:40:25 PM org.apache.catalina.loader.WebappClassLoader

clearReferencesThreads

SEVERE: The web application [/thredds] appears to have started a thread named

[TdsScheduler_QuartzSchedulerThread] but has failed to stop it. This is very

likely to create a memory leak.

Dec 6, 2011 5:40:25 PM org.apache.catalina.loader.WebappClassLoader

clearThreadLocalMap

SEVERE: The web application [/thredds] created a ThreadLocal with key of type

[org.apache.log4j.helpers.ThreadLocalMap] (value

[org.apache.log4j.helpers.ThreadLocalMap@6106dc2d]) and a value of type

[java.util.Hashtable] (value [{startTime=1323211152783, ID=2}]) but failed to

remove it when the web

application was stopped. This is very likely to create a memory leak.

Dec 6, 2011 5:40:25 PM org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor manager.xml

Dec 6, 2011 5:40:26 PM org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor host-manager.xml

Dec 6, 2011 5:40:26 PM org.apache.catalina.startup.HostConfig deployWAR

INFO: Deploying web application archive las.war

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity   [info] VelocityViewServlet: Using custom properties at

'/WEB-INF/velocity.properties'

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  :

*******************************************************************

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] Starting Apache Velocity v1.5 (compiled: 2007-02-22

08:52:29)

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  : RuntimeInstance initializing.

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] Default Properties File:

org/apache/velocity/runtime/defaults/velocity.properties

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] Trying to use logger class

org.apache.velocity.tools.view.servlet.ServletLogger

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity   [info] LogSystem has been deprecated. Please use a LogChute

implementation.

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] Default ResourceManager initializing. (class

org.apache.velocity.runtime.resource.ResourceManagerImpl)

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] ResourceLoader instantiated:

org.apache.velocity.tools.view.servlet.WebappLoader

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] WebappLoader : initialization starting.

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] WebappLoader : initialization complete.

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  [debug] ResourceCache: initialized (class

org.apache.velocity.runtime.resource.ResourceCacheImpl)

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log

INFO:  Velocity  : Default ResourceManager initialization complete.

Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log