[Go-essp-tech] output1 vs. output2
Serguei Nikonov
serguei.nikonov at noaa.gov
Wed Feb 29 09:55:20 MST 2012
Hi all,
can somebody explain me meaning of messages in THREDDS log file
(threddsServlet.log). We have a LOT messages like
2012-02-29T11:00:26.752 -0500 [ 7201683][ 5663] INFO -
thredds.servlet.FileServerServlet - Remote host: 134.157.176.252 - Request: "GET
/thredds/fileServer/gfdl_dataroot/NOAA-GFDL/GFDL-ESM2M/rcp26/mon/atmos/Amon/r1i1p1/v20110601/rlds/rlds_Amon_GFDL-
SM2M_rcp26_r1i1p1_206101-206512.nc HTTP/1.0"
2012-02-29T11:00:26.752 -0500 [ 7201683][ 5663] INFO -
thredds.servlet.FileServerServlet - Request Completed - -1 - 0 - 0
What does it mean "-1 - 0 - 0"? Does it mean that this request was not
actually done?
Actually, most of time we have these kind of messages on GFDL data node, and not
too much happy ones like
2012-02-29T11:46:03.995 -0500 [ 9938926][ 9347] INFO -
thredds.servlet.ServletUtil - returnFile(): Request Completed - 206 - 0 - 27
For example, for last 50 mins:
bad one - 3876
good one - 43.
It means that there are only 1% of successful requests. Not too much...
Is this rate common for other data nodes?
Also, we have a lot java errors (see at the bottom). What the cause of them and
how serious they are?
Thanks,
Sergey Nikonov
2012-02-29T11:00:27.095 -0500 [ 7202026][ 5654] WARN -
esg.orp.app.SAMLAuthorizationServiceFilterCollaborator -
org.apache.commons.httpclient.ProtocolException: The server pcmdi3.llnl.gov
failed to respond with a valid HTTP response
java.lang.RuntimeException: org.apache.commons.httpclient.ProtocolException: The
server pcmdi3.llnl.gov failed to respond with a valid HTTP response
at esg.security.common.SOAPServiceClient.doSoap(SOAPServiceClient.java:77)
at
esg.orp.app.SAMLAuthorizationServiceFilterCollaborator.authorize(SAMLAuthorizationServiceFilterCollaborator.java:78)
at
esg.orp.app.AuthorizationFilter.attemptValidation(AuthorizationFilter.java:60)
at
esg.orp.app.AccessControlFilterTemplate.doFilter(AccessControlFilterTemplate.java:62)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
esg.orp.app.AccessControlFilterTemplate.doFilter(AccessControlFilterTemplate.java:66)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
eske.web.filters.security.AuthorizationTokenValidationFilter.doFilter(AuthorizationTokenValidationFilter.java:84)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.commons.httpclient.ProtocolException: The server
pcmdi3.llnl.gov failed to respond with a valid HTTP response
at
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1987)
at
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at esg.security.common.SOAPServiceClient.doSoap(SOAPServiceClient.java:56)
... 22 more
2012-02-29T11:00:27.096 -0500 [ 7202027][ 5655] WARN -
esg.orp.app.SAMLAuthorizationServiceFilterCollaborator - java.io.IOException:
Stream closed
java.lang.RuntimeException: java.io.IOException: Stream closed
at esg.security.common.SOAPServiceClient.doSoap(SOAPServiceClient.java:81)
at
esg.orp.app.SAMLAuthorizationServiceFilterCollaborator.authorize(SAMLAuthorizationServiceFilterCollaborator.java:78)
at
esg.orp.app.AuthorizationFilter.attemptValidation(AuthorizationFilter.java:60)
at
esg.orp.app.AccessControlFilterTemplate.doFilter(AccessControlFilterTemplate.java:62)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
esg.orp.app.AccessControlFilterTemplate.doFilter(AccessControlFilterTemplate.java:66)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
eske.web.filters.security.AuthorizationTokenValidationFilter.doFilter(AuthorizationTokenValidationFilter.java:84)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:145)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at
org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at esg.security.common.SOAPServiceClient.doSoap(SOAPServiceClient.java:56)
... 22 more
On 02/29/2012 05:09 AM, Kettleborough, Jamie wrote:
> Hello Karl,
> I think its an error if someone tries to publish an unknown diagnostic with
> product=output. *But* do we know the scale of the problem of data already in the
> system? How much data has the wrong product assigned (either in the DRS id, or
> in the files netCDF attribute)? Is there an expectation that this data will be
> reassigned to the correct product? Or will we just live with it? If its
> sufficiently large volume/number of files *and* we are going to 'live with' the
> legacy problem then I don't see a strong reason to fix new data - users already
> have to deal with the wrong products. (but happy to be wrong on this)
> Jamie
> --------------------------------------------------------------------------------
> *From:* go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] *On
> Behalf Of *Karl Taylor
> *Sent:* 28 February 2012 16:51
> *To:* stephen.pascoe at stfc.ac.uk
> *Cc:* Drach, Bob; go-essp-tech at ucar.edu
> *Subject:* Re: [Go-essp-tech] output1 vs. output2
>
> Dear all,
>
> thanks for identifying the likely bug.
>
> While fixing this problem, we should check whether the publisher correctly
> treats product="unsolicited" data sets correctly. Is there an "if test" for
> this case? What should we do with data where product as been set by the data
> writer to "output", but "the product cannot be determined" by the publisher?
> Should we set product to "output" as the publisher apparently now does, or
> should be throw an error, or should be set it to "unsolicited"?
>
> Please vote. (and give reasons for your view).
>
> thanks,
> Karl
>
> On 2/28/12 3:14 AM, stephen.pascoe at stfc.ac.uk wrote:
>>
>> Karl, Bob,
>>
>> I think this is a bug in ESG publisher. I've just taken a look at the code
>> and it converts the variable to lower-case before looking it up in an
>> internal table, however the internal table has "sfcWind" so it never finds
>> the variable. Other variables that may be affected are tasAdjust, tsAdjust
>> and parasolRefl
>>
>> ### In esgcet.config.cmip5_product
>>
>> def getProduct(cmor_table, variable, experiment, year1, year2):
>>
>> """Get the DRS product value associated with the file.
>>
>> Returns
>>
>> 'output1' for datasets to be replicated,
>>
>> 'output2' for datasets outside the replicated datasets,
>>
>> 'output' if the product cannot be determined.
>>
>> """
>>
>> cmor_table = cmor_table.lower()
>>
>> variable *= variable.lower()*
>>
>> ### In esgcet.config.cmip5_tables
>>
>> # cmor_variables : cmor_table => { variable => (priority, dimension_names)}
>>
>> cmor_variables = { '3hr': { 'clt': (1, ['longitude', 'latitude', 'time']),
>>
>> # ...
>>
>> 'amon': { 'ccb': (1, ['longitude', 'latitude', 'time']),
>>
>> # ...
>>
>> *'sfcWind'*: (1, ['longitude', 'latitude', 'time', 'height10m']),
>>
>> 'ta': (1, ['longitude', 'latitude', 'plevs', 'time']),
>>
>> 'tas': (1, ['longitude', 'latitude', 'time', 'height2m']),
>>
>> 'tasAdjust': (1, ['longitude', 'latitude', 'time', 'height2m']),
>>
>> Cheers,
>>
>> Stephen
>>
>> ---
>>
>> Stephen Pascoe +44 (0)1235 445980
>>
>> Centre of Environmental Data Archival
>>
>> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>>
>> *From:*go-essp-tech-bounces at ucar.edu
>> [mailto:go-essp-tech-bounces at ucar.edu] *On Behalf Of *Karl Taylor
>> *Sent:* 28 February 2012 00:52
>> *To:* go-essp-tech at ucar.edu
>> *Subject:* [Go-essp-tech] output1 vs. output2
>>
>> Hi all,
>>
>> I have noticed (using the new p2p search engine, which works much faster,
>> and I'm pretty sure is accurate) that sfcWind_Amon_historical output is
>> designated as "output 1" for some models (from IPSL, MOHC, MPI-M, and
>> NOAA-GFDL), but "output2" for the rest. I think "output1" is correct.
>>
>> Can anyone explain why the publisher correctly characterized the output
>> from some groups, but not for the rest?
>>
>> thanks,
>> Karl
>>
>>
>> --
>> Scanned by iCritical.
>>
>>
>
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
More information about the GO-ESSP-TECH
mailing list