[Go-essp-tech] Error publishing large dataset
Estanislao Gonzalez
gonzalez at dkrz.de
Thu Dec 8 06:04:26 MST 2011
indeed,
the JAVA_OPTS option should have been corrected, as previously it wasn't
honored. The 2G option is added always so if your version is still
broken (or this wasn't corrected at al)l you'll have to change the
script... which means also "always after an upgrade".
Let's assume it's corrected :-) :
export JAVA_OPTS='-Xmx8G -Xms8G -XX:MaxPermSize=128m'
esg-node restart
check the java process via:
$ sudo jps
30722 Jps
28432
or
$pgrep -u tomcat
28432
And then us jmap to get a dump of the current status:
$ sudo jmap -heap 28432
Attaching to process ID 28432, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 19.0-b09
using thread-local object allocation.
Parallel GC with 8 thread(s)
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
* MaxHeapSize = 8589934592 (8192.0MB)*
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 536870912 (512.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 954466304 (910.25MB)
used = 618087896 (589.4545516967773MB)
free = 336378408 (320.79544830322266MB)
64.75743495707523% used
From Space:
capacity = 954400768 (910.1875MB)
used = 49982728 (47.66724395751953MB)
free = 904418040 (862.5202560424805MB)
5.237079608049938% used
To Space:
capacity = 954400768 (910.1875MB)
used = 0 (0.0MB)
free = 954400768 (910.1875MB)
0.0% used
PS Old Generation
capacity = 5726666752 (5461.375MB)
used = 4270949824 (4073.0951538085938MB)
free = 1455716928 (1388.2798461914062MB)
74.58003073966893% used
PS Perm Generation
capacity = 74055680 (70.625MB)
used = 73915776 (70.4915771484375MB)
free = 139904 (0.1334228515625MB)
99.81108268805309% used
Besides giving you a nice view of the JVM memory status, you'll see
what's the current max heap size of a running JVM. If it's not what you
expect, you'll have to manually set it in:
/usr/local/bin/esg-node:1834: [ $((sel & ALL_BIT)) != 0 ] &&
java_opts+=' -Xmx8192m -Xms8192m -XX:MaxPermSize=512m'
Hope this helps,
Estani
Am 08.12.2011 02:12, schrieb Drach, Bob:
> Look in the esg-node script, in the start_tomcat routine. It may work to just add the options to the JAVA_OPTS environment variable, but it looks like the options are hardwired, so the surefire way is to edit the script.
>
> Gavin: Is there a better way?
>
> --Bob
> ________________________________________
> From: go-essp-tech-bounces at ucar.edu [go-essp-tech-bounces at ucar.edu] On Behalf Of Hans Vahlenkamp [Hans.Vahlenkamp at noaa.gov]
> Sent: Wednesday, December 07, 2011 5:02 PM
> To: Estanislao Gonzalez
> Cc: go-essp-tech at ucar.edu
> Subject: Re: [Go-essp-tech] Error publishing large dataset
>
> Hello Estani,
>
> Thanks for the advice, it looks promising. One question... In what file
> should the "-Xmx8G -Xms8G -XX:MaxPermSize=128m" options be put?
>
> Hans
>
> -------- Original Message --------
> Subject: Re: [Go-essp-tech] Error publishing large dataset
> Date: Wed, 07 Dec 2011 12:58:13 +0100
> From: Estanislao Gonzalez<gonzalez at dkrz.de>
> Organization: DKRZ
> To: Drach, Bob<drach1 at llnl.gov>
> CC: go-essp-tech at ucar.edu<go-essp-tech at ucar.edu>, Hans Vahlenkamp
> <Hans.Vahlenkamp at noaa.gov>
>
> Hi,
>
> just one short comment on memory. The Java Virtual Machine (JVM) does
> not allocate memory completely dynamically, so this parameters should be
> tuned by hand.
> If there's nothing else running on the machine, it's pretty safe to say
> you can use 8G for the TDS. In that case use:
>
> -Xmx8G -Xms8G -XX:MaxPermSize=128m
>
> The Xmx is the maximal memory the JVM can grow to, the Xms the minimal.
> By setting both to the same level, the Garbage collector will have
> enough room to work on idle only and thus the speed of the JVM should be
> boosted. The PermSize is where Java classes (not objects) are kept, this
> should be enough.
> And you should be fine. It is very important to setup this values, the
> default of the JVM are just 64MB, and the default node installation 2G,
> which it is quite low for a "normal" CMIP5 data node.
>
> Hope this helps,
> Estani
>
> Am 07.12.2011 01:41, schrieb Drach, Bob:
>> Hi Hans, Sergey,
>>
>> That error will occur if the gateway tries to reach the GFDL data node to read the catalog, but fails either because the server is down or is inaccessible. Given that your TDS was running into memory issues, it was probably that the TDS was unavailable.
>>
>> I would try to back out of the latest TDS catalog update. Assuming that ONLY the most recently published datasets are in 'map_file', you can remove the catalogs with
>>
>> esgunpublish --map map_file --skip-gateway
>>
>> The above command will remove the catalogs related to 'map_file' and regenerate a top-level catalog. It will try (and fail) to reinit TDS. But if you then restart tomcat, the TDS may recover.
>>
>> If that doesn't work, you may have to manually edit the top-level THREDDS catalog:
>>
>> /usr/local/tomcat/content/thredds/esgcet/catalog.xml
>>
>> Remove the links to the most recently published datasets, then restart tomcat.
>>
>> Also consider giving max heap memory to Java. We include (in the jsvc command to start tomcat)
>>
>> -Xmx2560m -Xms2560m -XX:MaxPermSize=128m
>>
>> depending on your system you may be able to specify more memory.
>>
>> --Bob
>> ________________________________________
>> From: go-essp-tech-bounces at ucar.edu [go-essp-tech-bounces at ucar.edu] On Behalf Of Hans Vahlenkamp [Hans.Vahlenkamp at noaa.gov]
>> Sent: Tuesday, December 06, 2011 4:12 PM
>> To: go-essp-tech at ucar.edu
>> Subject: [Go-essp-tech] Error publishing large dataset
>>
>> Hi all,
>>
>> Did anybody encountered with difficulties to publish large set of files in one
>> pool?
>>
>> We had a problem when tried to make about 30000 files at once. We could write
>> them in db and configure TDS (esgpublish --map map_file --project cmip5;
>> esgpublish --map map_file --project cmip5 --noscan --thredds) without problems.
>> But on last stage - publishing on gateway - we've got en error just at the
>> beginning of process (see at the bottom).
>>
>> After that jsvc process became to consume 100% of CPU and TDS was barely
>> responsive. Restarting ESG node didn't help. TDS is completely dead with 404
>> error. Also tomcat gives severe errors about possible memory leak.
>> Here are all the entries in the "catalina.err" log from one startup of the
>> ESG data node services. Note the problems on the lines with "SEVERE:". Are
>> there any suggestions about how to make TDS functional again?
>>
>> Also, our ESG data node is running on a machine with 64-bit RHEL 5 and 24GB
>> of memory. So, the out of memory error is rather unexpected...
>>
>> Thanks,
>> Sergey Nikonov
>> Hans Vahlenkamp
>> GFDL Data Portal
>>
>> Error from publisher script (esgpublish --map map_file --project cmip5 --noscan
>> --publish):
>> ===================================================================================
>> INFO 2011-12-06 10:25:01,537 Publishing:
>> cmip5.output1.NOAA-GFDL.GFDL-ESM2M.historicalMisc.3hr.atmos.3hr.r1i1p2, parent =
>> pcmdi.GFDL
>> Traceback (most recent call last):
>> File "/usr/local/cdat/bin/esgpublish", line 5, in<module>
>> pkg_resources.run_script('esgcet==2.9.0', 'esgpublish')
>> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in run_script
>> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in run_script
>> File
>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-INFO/scripts/esgpublish",
>> line 461, in<module>
>> main(sys.argv[1:])
>> File
>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-INFO/scripts/esgpublish",
>> line 447, in main
>> result = publishDatasetList(datasetNames, Session, publish=publish,
>> thredds=thredds, las=las, parentId=parent, service=service, perVariable=perVariable)
>> File
>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/publish.py",
>> line 308, in publishDatasetList
>> dset, statusId, state, evname, status = publishDataset(datasetName,
>> parentIdent, service, threddsRootURL, session)
>> File
>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/publish.py",
>> line 118, in publishDataset
>> statusId = service.createDataset(parentId, threddsURL, -1, "Published")
>> File
>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/hessianlib.py",
>> line 426, in __call__
>> return self._invoker(self._method, args)
>> File
>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/hessianlib.py",
>> line 543, in __invoke
>> return self.parse_response(responseProxy)
>> File
>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet/publish/hessianlib.py",
>> line 554, in parse_response
>> raise value
>> esgcet.publish.hessianlib.RemoteCallException: Java ServiceException:
>> PublishingOperation must define at least 1 PublishingTask
>> at org.springframework.util.Assert.notEmpty(Assert.java:268)
>> at
>> sgf.gateway.metadata.complete.service.impl.PublishingOperationExectionServiceImpl.handleExecute(PublishingOperationExectionServiceImpl.java:158)
>> at
>> sgf.gateway.metadata.complete.service.impl.PublishingOperationExectionServiceImpl.execute(PublishingOperationExectionServiceImpl.java:127)
>> at
>> sgf.gateway.service.publishing.impl.spring.PublishingServiceImpl.publishThreddsCatalog(PublishingServiceImpl.java:141)
>> at
>> sgf.gateway.metadata.harvest.thredds.RemotePublishingWrapper.publishThreddsCatalogFromDatasetWithPermission(RemotePublishingWrapper.java:273)
>> at
>> sgf.gateway.metadata.harvest.thredds.RemotePublishingWrapper.publishThreddsCatalogFromDatasetWithPermission(RemotePublishingWrapper.java:251)
>> at
>> sgf.gateway.service.publishing.impl.spring.RemotePublishingServiceImpl.createOrUpdateDataset(RemotePublishingServiceImpl.java:99)
>> at
>> sgf.gateway.service.publishing.impl.spring.RemotePublishingServiceImpl.createDataset(RemotePublishingServiceImpl.java:84)
>> at
>> sgf.gateway.service.publishing.impl.spring.RemotePublishingServiceSecurityProxyImpl.createDataset(RemotePublishingServiceSecurityProxyImpl.java:67)
>> at sun.reflect.GeneratedMethodAccessor581.invoke(None:-1)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>> at
>> org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
>> at $Proxy25.createDataset(None:-1)
>> at sun.reflect.GeneratedMethodAccessor581.invoke(None:-1)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:155)
>> at
>> org.springframework.remoting.caucho.HessianExporter.doInvoke(HessianExporter.java:198)
>> at
>> org.springframework.remoting.caucho.HessianExporter.invoke(HessianExporter.java:118)
>> at
>> org.springframework.remoting.caucho.HessianServiceExporter.handleRequest(HessianServiceExporter.java:66)
>> at
>> org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:49)
>> at
>> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
>> at
>> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
>> at
>> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
>> at
>> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:366)
>> at
>> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
>> at
>> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>> at
>> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>> at
>> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>> at
>> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>> at
>> org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>> at
>> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:187)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>> at
>> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:187)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>> at
>> org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:89)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>> at
>> org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>> at
>> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>> at
>> org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:109)
>> at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>> at
>> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:167)
>> at
>> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
>> at
>> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>> at
>> org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198)
>> at
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>> at
>> sgf.gateway.web.filters.exception.UnhandledExceptionFilter.doFilter(UnhandledExceptionFilter.java:99)
>> at
>> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
>> at
>> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>> at
>> org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:276)
>> at
>> org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>> at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
>> at
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
>> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
>> at java.lang.Thread.run(Thread.java:619)
>> =================================================================================
>>
>>
>> Error from catalina.err:
>> ============================================================================
>>
>> Dec 6, 2011 5:35:04 PM org.apache.catalina.core.AprLifecycleListener init
>> INFO: The APR based Apache Tomcat Native library which allows optimal
>> performance in production environments was not found on the java.library.path:
>> /usr/local/openssl/lib:/usr/local/curl/lib:/usr/local/pgsql/lib:/usr/local/cdat/Externals/lib:/usr/local/globus/lib:/usr/local/openssl/lib:/usr/local/curl/lib:/usr/local/pgsql/lib:/usr/local/cdat/Externals/lib:/usr/local/globus/lib:/usr/local/openssl/lib:/usr/local/curl/lib:/usr/local/pgsql/lib:/usr/local/cdat/Externals/lib:/usr/local/globus/lib:/usr/local/cdat/Externals/lib:/usr/local/pgsql/lib:.:/usr/local/java/jre/lib/amd64/server:/usr/local/java/jre/lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>> Dec 6, 2011 5:35:04 PM org.apache.coyote.http11.Http11Protocol init
>> INFO: Initializing Coyote HTTP/1.1 on http-80
>> Dec 6, 2011 5:35:05 PM org.apache.coyote.http11.Http11Protocol init
>> INFO: Initializing Coyote HTTP/1.1 on http-443
>> Dec 6, 2011 5:35:05 PM org.apache.catalina.startup.Catalina load
>> INFO: Initialization processed in 966 ms
>> Dec 6, 2011 5:35:05 PM org.apache.catalina.realm.JAASRealm setContainer
>> INFO: Set JAAS app name Catalina
>> Dec 6, 2011 5:35:05 PM org.apache.catalina.core.StandardService start
>> INFO: Starting service Catalina
>> Dec 6, 2011 5:35:05 PM org.apache.catalina.core.StandardEngine start
>> INFO: Starting Servlet Engine: Apache Tomcat/6.0.32
>> Dec 6, 2011 5:35:05 PM org.apache.catalina.startup.HostConfig deployDescriptor
>> INFO: Deploying configuration descriptor thredds.xml
>> Dec 6, 2011 5:35:05 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Initializing Spring root WebApplicationContext
>> Dec 6, 2011 5:35:06 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Set web app root system property: 'webapp.root' =
>> [/usr/local/tomcat/webapps/thredds/]
>> Dec 6, 2011 5:35:06 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Initializing log4j from [/usr/local/tomcat/webapps/thredds/WEB-INF/log4j.xml]
>> Dec 6, 2011 5:39:08 PM org.apache.catalina.core.StandardContext listenerStart
>> SEVERE: Exception sending context initialized event to listener instance of
>> class thredds.server.config.TdsConfigContextListener
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>> Dec 6, 2011 5:39:12 PM org.apache.catalina.core.StandardContext start
>> SEVERE: Error listenerStart
>> Dec 6, 2011 5:39:12 PM org.apache.catalina.core.StandardContext start
>> SEVERE: Context [/thredds] startup failed due to previous errors
>> Dec 6, 2011 5:39:12 PM org.apache.catalina.core.StandardContext listenerStop
>> SEVERE: Exception sending context destroyed event to listener instance of class
>> thredds.server.config.TdsConfigContextListener
>> java.lang.IllegalStateException: setInstance() must be called first.
>> at thredds.servlet.DataRootHandler.getInstance(DataRootHandler.java:110)
>> at
>> thredds.server.config.TdsConfigContextListener.contextDestroyed(TdsConfigContextListener.java:93)
>> at
>> org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4244)
>> at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4879)
>> at
>> org.apache.catalina.core.StandardContext.start(StandardContext.java:4749)
>> at
>> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
>> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
>> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
>> at
>> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
>> at
>> org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
>> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
>> at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
>> at
>> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
>> at
>> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
>> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
>> at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
>> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
>> at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
>> at org.apache.catalina.core.StandardService.start(StandardService.java:525)
>> at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
>> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:219)
>> Dec 6, 2011 5:39:12 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Closing Spring root WebApplicationContext
>> Dec 6, 2011 5:40:17 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/thredds] appears to have started a thread named
>> [TdsScheduler_Worker-1] but has failed to stop it. This is very likely to create
>> a memory leak.
>> Dec 6, 2011 5:40:17 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/thredds] appears to have started a thread named
>> [TdsScheduler_Worker-2] but has failed to stop it. This is very likely to create
>> a memory leak.
>> Dec 6, 2011 5:40:17 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/thredds] appears to have started a thread named
>> [TdsScheduler_Worker-3] but has failed to stop it. This is very likely to create
>> a memory leak.
>> Dec 6, 2011 5:40:25 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/thredds] appears to have started a thread named
>> [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
>> Dec 6, 2011 5:40:25 PM org.apache.catalina.loader.WebappClassLoader
>> clearReferencesThreads
>> SEVERE: The web application [/thredds] appears to have started a thread named
>> [TdsScheduler_QuartzSchedulerThread] but has failed to stop it. This is very
>> likely to create a memory leak.
>> Dec 6, 2011 5:40:25 PM org.apache.catalina.loader.WebappClassLoader
>> clearThreadLocalMap
>> SEVERE: The web application [/thredds] created a ThreadLocal with key of type
>> [org.apache.log4j.helpers.ThreadLocalMap] (value
>> [org.apache.log4j.helpers.ThreadLocalMap at 6106dc2d]) and a value of type
>> [java.util.Hashtable] (value [{startTime=1323211152783, ID=2}]) but failed to
>> remove it when the web
>> application was stopped. This is very likely to create a memory leak.
>> Dec 6, 2011 5:40:25 PM org.apache.catalina.startup.HostConfig deployDescriptor
>> INFO: Deploying configuration descriptor manager.xml
>> Dec 6, 2011 5:40:26 PM org.apache.catalina.startup.HostConfig deployDescriptor
>> INFO: Deploying configuration descriptor host-manager.xml
>> Dec 6, 2011 5:40:26 PM org.apache.catalina.startup.HostConfig deployWAR
>> INFO: Deploying web application archive las.war
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [info] VelocityViewServlet: Using custom properties at
>> '/WEB-INF/velocity.properties'
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity :
>> *******************************************************************
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Starting Apache Velocity v1.5 (compiled: 2007-02-22
>> 08:52:29)
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity : RuntimeInstance initializing.
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Default Properties File:
>> org/apache/velocity/runtime/defaults/velocity.properties
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Trying to use logger class
>> org.apache.velocity.tools.view.servlet.ServletLogger
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [info] LogSystem has been deprecated. Please use a LogChute
>> implementation.
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Default ResourceManager initializing. (class
>> org.apache.velocity.runtime.resource.ResourceManagerImpl)
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] ResourceLoader instantiated:
>> org.apache.velocity.tools.view.servlet.WebappLoader
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] WebappLoader : initialization starting.
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] WebappLoader : initialization complete.
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] ResourceCache: initialized (class
>> org.apache.velocity.runtime.resource.ResourceCacheImpl)
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity : Default ResourceManager initialization complete.
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Loaded System Directive:
>> org.apache.velocity.runtime.directive.Literal
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Loaded System Directive:
>> org.apache.velocity.runtime.directive.Macro
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Loaded System Directive:
>> org.apache.velocity.runtime.directive.Parse
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Loaded System Directive:
>> org.apache.velocity.runtime.directive.Include
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Loaded System Directive:
>> org.apache.velocity.runtime.directive.Foreach
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Created '20' parsers.
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity : Velocimacro : initialization starting.
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Velocimacro : adding VMs from VM library :
>> /WEB-INF/VM_global_library.vm
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [info] Velocimacro : added #quietnull( a ) : source =
>> /WEB-INF/VM_global_library.vm
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [info] Velocimacro : added #recurse( a ) : source =
>> /WEB-INF/VM_global_library.vm
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [info] Velocimacro : added #callrecurse( ) : source =
>> /WEB-INF/VM_global_library.vm
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [info] Velocimacro : added #testbool( b ) : source =
>> /WEB-INF/VM_global_library.vm
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] ResourceManager : found /WEB-INF/VM_global_library.vm
>> with loader org.apache.velocity.tools.view.servlet.WebappLoader
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity : Velocimacro : VM library registration complete.
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Velocimacro : allowInline = true : VMs can be defined
>> inline in templates
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Velocimacro : allowInlineToOverride = false : VMs
>> defined inline may NOT replace previous VM definitions
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Velocimacro : allowInlineLocal = false : VMs defined
>> inline will be global in scope if allowed.
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [debug] Velocimacro : autoload off : VM system will not
>> automatically reload global library macros
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity : Velocimacro : Velocimacro : initialization complete.
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity : RuntimeInstance successfully initialized.
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Velocity [info] VelocityViewServlet: Default content-type is: text/html
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.startup.HostConfig deployDirectory
>> INFO: Deploying web application directory ROOT
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.startup.HostConfig deployDirectory
>> INFO: Deploying web application directory docs
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.startup.HostConfig deployDirectory
>> INFO: Deploying web application directory OpenidRelyingParty
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>> INFO:
>> validateJarFile(/usr/local/tomcat/webapps/OpenidRelyingParty/WEB-INF/lib/servlet-api-2.3.jar)
>> - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class:
>> javax/servlet/Servlet.class
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Set web app root system property: 'esg-orp.root' =
>> [/usr/local/tomcat/webapps/OpenidRelyingParty/]
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Initializing log4j from
>> [/usr/local/tomcat/webapps/OpenidRelyingParty/WEB-INF/log4j.xml]
>> Dec 6, 2011 5:40:27 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Initializing Spring root WebApplicationContext
>> Dec 6, 2011 5:40:29 PM org.apache.catalina.core.ApplicationContext log
>> INFO: Initializing Spring FrameworkServlet 'DispatcherServlet'
>> Dec 6, 2011 5:40:29 PM org.apache.catalina.startup.HostConfig deployDirectory
>> INFO: Deploying web application directory esgf-node-manager
>> Dec 6, 2011 5:40:29 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>> INFO:
>> validateJarFile(/usr/local/tomcat/webapps/esgf-node-manager/WEB-INF/lib/servlet-api-2.3.jar)
>> - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class:
>> javax/servlet/Servlet.class
>> Dec 6, 2011 5:40:30 PM org.apache.coyote.http11.Http11Protocol start
>> INFO: Starting Coyote HTTP/1.1 on http-80
>> Dec 6, 2011 5:40:30 PM org.apache.coyote.http11.Http11Protocol start
>> INFO: Starting Coyote HTTP/1.1 on http-443
>> Dec 6, 2011 5:40:30 PM org.apache.jk.common.ChannelSocket init
>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>> Dec 6, 2011 5:40:30 PM org.apache.jk.server.JkMain start
>> INFO: Jk running ID=0 time=0/22 config=null
>> Dec 6, 2011 5:40:30 PM org.apache.catalina.startup.Catalina start
>> INFO: Server startup in 325459 ms
>>
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
> --
> Estanislao Gonzalez
>
> Max-Planck-Institut f?(1/4)r Meteorologie (MPI-M)
> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>
> Phone: +49 (40) 46 00 94-126
> E-Mail: gonzalez at dkrz.de
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: gonzalez at dkrz.de
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111208/72cc33c6/attachment-0001.html
More information about the GO-ESSP-TECH
mailing list