[Go-essp-tech] threads
Serguei Nikonov
serguei.nikonov at noaa.gov
Wed Feb 29 15:55:41 MST 2012
Hi Stephen,
it would be great if we had not have too few "206" messages
INFO - thredds.servlet.ServletUtil - returnFile(): Request Completed - 206 - 0
- 1. I assume that this is standard HTTP message from "2xx" family of HTTP
messages meaning success.
We had yesterday several hours of normal work - I could download ~4000 files for
40 mins (just for testing) from our data node and rate was
98/4712 (Request Completed - -1 - 0 - ... / Request Completed - 206 - 0 - ...).
And I assumed that it indicates a problem cause we had problem starting
yesterday evening again.
Sorry, I left wrong subject and didn't check recipients that spawned requests in
helpdesk. Actually, I was assured by mail server that this email was rejected by
by stfc.ac.uk domain.
Thanks,
Sergey
On 02/29/2012 04:55 PM, stephen.pascoe at stfc.ac.uk wrote:
> Hi Serguei,
>
> I don't think those messages indicate failure. I get these messages for downloading a test file that worked:
>
> [2012-02-29T21:47:06] Request Completed - -1 - 0 - 0
> [2012-02-29T21:47:16] Remote host: 130.246.132.177 - Request: "GET /thredds/fileServer/esg_testroot/test/sftlf_2.nc HT
> TP/1.0"
> [2012-02-29T21:47:16] Request Completed - -1 - 0 - 0
> [2012-02-29T21:47:17] Remote host: 130.246.132.177 - Request: "GET /thredds/fileServer/esg_testroot/test/sftlf_2.nc HT
> TP/1.0"
>
> So each download triggers 2 sets of log message (probably something to do with the redirect) and "-1 - 0 - 0" is either meaningless or suggests success.
>
> Cheers,
> Stephen
>
> P.S. in general could you please not email cmip5-helpdesk when also emailing others, particularly lists. Replies tend to trigger new queries in our system.
>
> ---
> Stephen Pascoe +44 (0)1235 445980
> Centre of Environmental Data Archival
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>
>
> -----Original Message-----
> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Serguei Nikonov
> Sent: 29 February 2012 16:55
> To: CMIP5-Helpdesk
> Cc: go-essp-tech at ucar.edu
> Subject: Re: [Go-essp-tech] output1 vs. output2
>
> Hi all,
>
> can somebody explain me meaning of messages in THREDDS log file
> (threddsServlet.log). We have a LOT messages like
>
> 2012-02-29T11:00:26.752 -0500 [ 7201683][ 5663] INFO -
> thredds.servlet.FileServerServlet - Remote host: 134.157.176.252 - Request: "GET
> /thredds/fileServer/gfdl_dataroot/NOAA-GFDL/GFDL-ESM2M/rcp26/mon/atmos/Amon/r1i1p1/v20110601/rlds/rlds_Amon_GFDL-
> SM2M_rcp26_r1i1p1_206101-206512.nc HTTP/1.0"
>
> 2012-02-29T11:00:26.752 -0500 [ 7201683][ 5663] INFO -
> thredds.servlet.FileServerServlet - Request Completed - -1 - 0 - 0
>
> What does it mean "-1 - 0 - 0"? Does it mean that this request was not
> actually done?
>
> Actually, most of time we have these kind of messages on GFDL data node, and not
> too much happy ones like
> 2012-02-29T11:46:03.995 -0500 [ 9938926][ 9347] INFO -
> thredds.servlet.ServletUtil - returnFile(): Request Completed - 206 - 0 - 27
>
> For example, for last 50 mins:
> bad one - 3876
> good one - 43.
>
> It means that there are only 1% of successful requests. Not too much...
> Is this rate common for other data nodes?
>
> Also, we have a lot java errors (see at the bottom). What the cause of them and
> how serious they are?
>
> Thanks,
> Sergey Nikonov
>
> 2012-02-29T11:00:27.095 -0500 [ 7202026][ 5654] WARN -
> esg.orp.app.SAMLAuthorizationServiceFilterCollaborator -
> org.apache.commons.httpclient.ProtocolException: The server pcmdi3.llnl.gov
> failed to respond with a valid HTTP response
> java.lang.RuntimeException: org.apache.commons.httpclient.ProtocolException: The
> server pcmdi3.llnl.gov failed to respond with a valid HTTP response
> at esg.security.common.SOAPServiceClient.doSoap(SOAPServiceClient.java:77)
> at
> esg.orp.app.SAMLAuthorizationServiceFilterCollaborator.authorize(SAMLAuthorizationServiceFilterCollaborator.java:78)
> at
> esg.orp.app.AuthorizationFilter.attemptValidation(AuthorizationFilter.java:60)
> at
> esg.orp.app.AccessControlFilterTemplate.doFilter(AccessControlFilterTemplate.java:62)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> esg.orp.app.AccessControlFilterTemplate.doFilter(AccessControlFilterTemplate.java:66)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> eske.web.filters.security.AuthorizationTokenValidationFilter.doFilter(AuthorizationTokenValidationFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.commons.httpclient.ProtocolException: The server
> pcmdi3.llnl.gov failed to respond with a valid HTTP response
> at
> org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1987)
> at
> org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
> at
> org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> at esg.security.common.SOAPServiceClient.doSoap(SOAPServiceClient.java:56)
> ... 22 more
> 2012-02-29T11:00:27.096 -0500 [ 7202027][ 5655] WARN -
> esg.orp.app.SAMLAuthorizationServiceFilterCollaborator - java.io.IOException:
> Stream closed
> java.lang.RuntimeException: java.io.IOException: Stream closed
> at esg.security.common.SOAPServiceClient.doSoap(SOAPServiceClient.java:81)
> at
> esg.orp.app.SAMLAuthorizationServiceFilterCollaborator.authorize(SAMLAuthorizationServiceFilterCollaborator.java:78)
> at
> esg.orp.app.AuthorizationFilter.attemptValidation(AuthorizationFilter.java:60)
> at
> esg.orp.app.AccessControlFilterTemplate.doFilter(AccessControlFilterTemplate.java:62)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> esg.orp.app.AccessControlFilterTemplate.doFilter(AccessControlFilterTemplate.java:66)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> eske.web.filters.security.AuthorizationTokenValidationFilter.doFilter(AuthorizationTokenValidationFilter.java:84)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
> at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Stream closed
> at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:145)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
> at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
> at
> org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
> at
> org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
> at
> org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
> at
> org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> at esg.security.common.SOAPServiceClient.doSoap(SOAPServiceClient.java:56)
> ... 22 more
>
>
> On 02/29/2012 05:09 AM, Kettleborough, Jamie wrote:
>> Hello Karl,
>> I think its an error if someone tries to publish an unknown diagnostic with
>> product=output. *But* do we know the scale of the problem of data already in the
>> system? How much data has the wrong product assigned (either in the DRS id, or
>> in the files netCDF attribute)? Is there an expectation that this data will be
>> reassigned to the correct product? Or will we just live with it? If its
>> sufficiently large volume/number of files *and* we are going to 'live with' the
>> legacy problem then I don't see a strong reason to fix new data - users already
>> have to deal with the wrong products. (but happy to be wrong on this)
>> Jamie
>> --------------------------------------------------------------------------------
>> *From:* go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] *On
>> Behalf Of *Karl Taylor
>> *Sent:* 28 February 2012 16:51
>> *To:* stephen.pascoe at stfc.ac.uk
>> *Cc:* Drach, Bob; go-essp-tech at ucar.edu
>> *Subject:* Re: [Go-essp-tech] output1 vs. output2
>>
>> Dear all,
>>
>> thanks for identifying the likely bug.
>>
>> While fixing this problem, we should check whether the publisher correctly
>> treats product="unsolicited" data sets correctly. Is there an "if test" for
>> this case? What should we do with data where product as been set by the data
>> writer to "output", but "the product cannot be determined" by the publisher?
>> Should we set product to "output" as the publisher apparently now does, or
>> should be throw an error, or should be set it to "unsolicited"?
>>
>> Please vote. (and give reasons for your view).
>>
>> thanks,
>> Karl
>>
>> On 2/28/12 3:14 AM, stephen.pascoe at stfc.ac.uk wrote:
>>>
>>> Karl, Bob,
>>>
>>> I think this is a bug in ESG publisher. I've just taken a look at the code
>>> and it converts the variable to lower-case before looking it up in an
>>> internal table, however the internal table has "sfcWind" so it never finds
>>> the variable. Other variables that may be affected are tasAdjust, tsAdjust
>>> and parasolRefl
>>>
>>> ### In esgcet.config.cmip5_product
>>>
>>> def getProduct(cmor_table, variable, experiment, year1, year2):
>>>
>>> """Get the DRS product value associated with the file.
>>>
>>> Returns
>>>
>>> 'output1' for datasets to be replicated,
>>>
>>> 'output2' for datasets outside the replicated datasets,
>>>
>>> 'output' if the product cannot be determined.
>>>
>>> """
>>>
>>> cmor_table = cmor_table.lower()
>>>
>>> variable *= variable.lower()*
>>>
>>> ### In esgcet.config.cmip5_tables
>>>
>>> # cmor_variables : cmor_table => { variable => (priority, dimension_names)}
>>>
>>> cmor_variables = { '3hr': { 'clt': (1, ['longitude', 'latitude', 'time']),
>>>
>>> # ...
>>>
>>> 'amon': { 'ccb': (1, ['longitude', 'latitude', 'time']),
>>>
>>> # ...
>>>
>>> *'sfcWind'*: (1, ['longitude', 'latitude', 'time', 'height10m']),
>>>
>>> 'ta': (1, ['longitude', 'latitude', 'plevs', 'time']),
>>>
>>> 'tas': (1, ['longitude', 'latitude', 'time', 'height2m']),
>>>
>>> 'tasAdjust': (1, ['longitude', 'latitude', 'time', 'height2m']),
>>>
>>> Cheers,
>>>
>>> Stephen
>>>
>>> ---
>>>
>>> Stephen Pascoe +44 (0)1235 445980
>>>
>>> Centre of Environmental Data Archival
>>>
>>> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>>>
>>> *From:*go-essp-tech-bounces at ucar.edu
>>> [mailto:go-essp-tech-bounces at ucar.edu] *On Behalf Of *Karl Taylor
>>> *Sent:* 28 February 2012 00:52
>>> *To:* go-essp-tech at ucar.edu
>>> *Subject:* [Go-essp-tech] output1 vs. output2
>>>
>>> Hi all,
>>>
>>> I have noticed (using the new p2p search engine, which works much faster,
>>> and I'm pretty sure is accurate) that sfcWind_Amon_historical output is
>>> designated as "output 1" for some models (from IPSL, MOHC, MPI-M, and
>>> NOAA-GFDL), but "output2" for the rest. I think "output1" is correct.
>>>
>>> Can anyone explain why the publisher correctly characterized the output
>>> from some groups, but not for the rest?
>>>
>>> thanks,
>>> Karl
>>>
>>>
>>> --
>>> Scanned by iCritical.
>>>
>>>
>>
>>
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
More information about the GO-ESSP-TECH
mailing list