<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffcc" text="#000000">
We are not behind a proxy here for this application, anymore.<br>
<br>
On 3/11/11 3:17 PM, Nathan Wilhelmi wrote:
<blockquote cite="mid:4D7AADA0.1000907@ucar.edu" type="cite">
<pre wrap="">Hi Bob,
Aren't you also behind some sort of proxy? Didn't you have a proxy that
was caching requests which lead to a mysterious publishing problem a
while ago? Catalogs were changed on the TDS but the gateway wasn't
seeing the changes.
-Nate
On 03/11/2011 04:13 PM, Robert S. Drach wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Cinquini, Luca (3880) wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Nathan,
we'll be happy to help. The person at JPL who is most familiar with the Apache configuration is Paul Zimdars, since he set it up, he is our lead system administrator and he is cc-ed in this email, so please contact him for any information you might need.
That said... it seems to me like there are other sites that do NOT have an Apache server front end, and use the same ESG truststore, yet they do not show any problems with the Safari browser. For example, isn't PCMDI such a site ? Bob, is PCMDI using the standard ESGF truststore ?
</pre>
</blockquote>
<pre wrap="">Probably not. It hasn't been updated for a while. I'll be glad to send
it if that would help debug the problem.
--Bob
</pre>
<blockquote type="cite">
<pre wrap="">thanks, Luca
On Mar 11, 2011, at 3:08 PM, Nathan Hook wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Everyone,
Work is still ongoing for finding a long term fix for the discovered
problem with large truststores.
That said...
It is my understanding that the folks at JPL have possibly solved the
problem by proxying their tomcat server with an apache http server.
Is anyone at JPL that is personally/intimately familiar with the apache
http server configuration available to discuss their current
solution/configuration?
Thank you for your time,
Nathan H.
On 3/4/2011 11:02 AM, <a class="moz-txt-link-abbreviated" href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I am extremely relieved you've got to the bottom of this issue. I spent a lot of time trying different truststores without truly getting to the bottom of the problem but your diagnosis makes sense. Thanks to everyone who's been working on this.
When ever possible I have used the esgf.org truststore but have always had to "tweek" it to resolve the Safari issue. Until your diagnosis the tweek has been frustrating trial and error. Until we have a "bare minimum" truststore that works I am reluctant to mess with our deployed truststores because something always breaks when I do. At present there are some authz issues I still have to resolve but most people seem to be getting through.
Cheers,
Stephen.
---
Stephen Pascoe +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
-----Original Message-----
From: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a> [<a class="moz-txt-link-freetext" href="mailto:go-essp-tech-bounces@ucar.edu">mailto:go-essp-tech-bounces@ucar.edu</a>] On Behalf Of Eric Nienhouse
Sent: 03 March 2011 16:07
To: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>; rachana ananthakrishnan
Subject: [Go-essp-tech] ESG Safari Browser Login Issue / truststore
Hi All,
We're still experiencing a problem with Safari browser login at several
ESG federation gateways (and possibly some data nodes.)
In particular, Safari SSL connections (eg: https requests) may fail with
an error "Safari can't open page." This prevents Safari browser users
from logging in to affected ESG Gateways. We've altered the NCAR
gateway configuration to allow Safari login in the short term, however,
this setting disables the ability to publish new datasets. Please let
me know if you need the details of this configuration change.
Included below is a recent thread describing this problem and suggested
solutions. Thus far, a recommended solution has not been identified.
This issue currently affects NCAR, NCI and DKRZ and possibly certain
data nodes.
Following is a summary:
The Safari browser mail fail to establish and SSL connection with
certain gateway servers depending on the size in bytes of the SSL
handshake response. Responses of> ~16K bytes cause Safari to fail to
establish an SSL connection. The current ESG federation truststore file
(esg-truststore.ts) which includes CA certificates for all federation
gateways (and some anticipated participants) is of sufficient size to
result in a Safari-refusing response size when combined with the server
SSL certificate (and other information.) Since each federation gateway
has a slightly differing configuration (and SSL certificate) some
gateways cause the Safari error while others do not for a particular
esg-truststore.ts file.
We're working on a "bare-minimum" truststore file to remedy this issue
in the short term, however this is likely not sustainable as the number
of federation gateways grow. Further, is it not clear a minimum
truststore can be developed given the current set of federation gateways
and required CA certificates.
This issue is related to the gateway requiring a client certificate for
authenticating certain service requests, such as publishing from
datanodes. In order to configure the gateway tomcat application server
to accept these certificates *and* to disable unsafe SSL renegotiation
gateway tomcat servers should currently be configured to want client
certificates. When tomcat is configured in this mode, the SSL handshake
response grows in size causing the Safari connection problem.
I'll keep you posted on progress with this issue. We're also looking
into several other options to remedy the problem.
Regards,
-Eric
Eric Nienhouse wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Rachana, All,
Some findings from today re. Safari and SSL failures.
1) Truststore files producing an SSL handshake of> ~16752 bytes
(including server certificate+chain, CA DN list) fail on Safari
5.0.3/OSX10.5.8 *always*.
2) The JPL gateway does indeed have clientAuth="want". It does not
present the CA DN list on SSL connect like other (tomcat only)
Gateways with this setting.
*the above is likely due to the fact that Apache is front-ending the
Gateway tomcat application. Apache could be one solution to this
issue, though I have not researched this deeply. Publishing
operations requiring client certificate (and all aspects of the
gateway) are functioning properly at JPL.
3) Truststores are not consistent at least for parts of the
federation. BADC, JPL and DKRZ all have different CA DN lists based
on SSL handshake output in openssl, indicating differing trust
stores. I am still researching this area across the federation.
Number (1) above seems to explain why a certain truststore works in
one gateway and not in another. The SSL handshake includes the
gateway server SSL certificate and the CA DN list. Certain gateway
certificates (plus chain) are "large" and presumably overflow the
Safari SSL buffer when combined with the CA DN list, whereas others
with "small" certs do not.
I'd appreciate some validation/counters to my thinking as this is not
an expert area of mine. This is my understanding thus far :-)
There may be hope to find a "bare minimum" truststore in the
meanwhile. The TS configured at BADC is a good candidate basis for
this. I'll see if I can get a reasonable one worked out by tomorrow
morning for further testing.
-Eric
Eric Nienhouse wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Rachana,
Comments inline below.
-Eric
Rachana Ananthakrishnan wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Nate, Eric,
Sounds good.
I need to look at the safe renegotiation, haven't had a chance to
delve into that.
Few points:
1. We should definitely ensure that all sites are using the same
trust store, and we can check the CA DN list that is returned using
openssl s_client, and that prints the size of the returned DNs. That
can help us atleast validate the theory on size limitations, and why
it works with some Gateways. I am happy to help with this testing,
but we need to first ensure that the Gateways where this works have
the same trust store as the others. Eric, is this something you can
help drive? Even one or two as an experiment would suffice.
</pre>
</blockquote>
<pre wrap="">I'll work on driving a consistent configuration (SSL connector,
truststore file) across the federation, starting with information
gathering on the working gateway truststores.
</pre>
<blockquote type="cite">
<pre wrap="">2. While a subclassed valve is a good fix, it is going to add Tomcat
version dependency and support burden. But maybe there are already
other Gateway pieces for which we do this.
3. I also like the idea of trimming trust store if we can. The
reason is some of the other notes I see of the inherent buffer
limitations other browsers might also have, which means this might
be an issue for us with other scenarios, and other browsers and
versions.
</pre>
</blockquote>
<pre wrap="">I'm looking at pruning the truststore currently - I'll report back
what I learn. I agree, other browsers seem to have limitations in
this area as well with Safari seeming to be the most constrained.
</pre>
<blockquote type="cite">
<pre wrap="">Please let me know if there are pieces we can test or investigate.
Thanks,
Rachana
On Mar 1, 2011, at 10:13 AM, Eric Nienhouse wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi All,
Thank you for your attention to this issue. I feel the more we can
learn about this the better. It also makes sense to me to consider
other options as well, including Nate's suggestion below regarding
enabling SSL renegotiation.
I am convinced there is something related to the number of CA DNs
sent to Safari and/or possibly the order of them. My own
experimentation with this has shown a dependency on the size of the
DN list and Safari failures.
Perhaps the duplicate DNs are not a significant issue as we've
thought and the Safari SSL failures are due to an absolute
byte-size limit in the certificate request DN list.
What is odd to me is that certain federation sites are working
correctly with Safari and others are not and we're presumably all
using the same truststore file. (Will tomcat "add to" the DN
response in some way that is dependent on the server
configuration? Eg: ciphers, SSL version data, etc.?)
I suggest we continue to dig into this problem and also pursue
Nate's suggested direction as an alternative solution.
I will spend some time this morning looking to a "bare minimum"
truststore as an interim option. This may not be sustainable in
the long term as any number of federation DNs may be needed in the
future.
I'll experiment with the NCAR "staging2" site
(esg2.prototype.ucar.edu.)
Neill, please pass on anything you need regarding the NCAR staging
(esg.prototype.ucar.edu) site.
FWIW, for those testing without Mac Safari access, I was able to
repro the SSL issue on my son's Ipod Touch Safari version :-)
Thanks all,
-Eric
Neill Miller wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hello Nathan,
I'm not sure how to do what you're saying, but if you're able to
set up a test (or need help doing so), let me know.
Rachana was correct that disabling SSLv3, or even only specifying
the SSLv2 handshake, makes no difference when Safari is
restarted. I still see the usual error in the server logs:
"http-128.117.12.102-443-5, WRITE: SSLv3 Handshake, length = 16384
*** ServerHelloDone
http-128.117.12.102-443-5, WRITE: SSLv3 Handshake, length = 2258
http-128.117.12.102-443-5, received EOFException: error
http-128.117.12.102-443-5, handling exception:
javax.net.ssl.SSLHandshakeException: Remote host closed connection
during handshake
http-128.117.12.102-443-5, SEND SSLv3 ALERT: fatal, description =
handshake_failure
http-128.117.12.102-443-5, WRITE: SSLv3 Alert, length = 2"
-Neill.
----- Original Message -----
From: "Nathan Wilhelmi"<a class="moz-txt-link-rfc2396E" href="mailto:wilhelmi@ucar.edu"><wilhelmi@ucar.edu></a>
To: "Rachana Ananthakrishnan"<a class="moz-txt-link-rfc2396E" href="mailto:ranantha@mcs.anl.gov"><ranantha@mcs.anl.gov></a>
Cc: "Neill Miller"<a class="moz-txt-link-rfc2396E" href="mailto:neillm@mcs.anl.gov"><neillm@mcs.anl.gov></a>, "Eric Nienhouse"
<a class="moz-txt-link-rfc2396E" href="mailto:ejn@ucar.edu"><ejn@ucar.edu></a>, "Dean N. Williams"<a class="moz-txt-link-rfc2396E" href="mailto:williams13@llnl.gov"><williams13@llnl.gov></a>,
<a class="moz-txt-link-abbreviated" href="mailto:vets-sgg@ucar.edu">vets-sgg@ucar.edu</a>, "Philip Kershaw (BADC)"
<a class="moz-txt-link-rfc2396E" href="mailto:philip.kershaw@stfc.ac.uk"><philip.kershaw@stfc.ac.uk></a>, "Don Middleton"<a class="moz-txt-link-rfc2396E" href="mailto:don@ucar.edu"><don@ucar.edu></a>
Sent: Tuesday, March 1, 2011 9:43:08 AM
Subject: Re: Truststore issue
Hi,
IMO as the truststore works more then it doesn't I would like to
see if the problem can be fixed, or at definitely identify the
problem. We do have a possible plan b that I started to look into
a bit. Now that Java supports safe renegotiation we could look at
going away from want in the SSL connector and go back to
renegotiating for a client cert in the cases where it's needed. I
would *not* recommend we add the old realm back in, it triggered a
number of very difficult to debug problems on it's own. I need to
do a little more research but we might be able to use a valve or
implement a realm that does nothing more the trigger the
renegotiation.
Thanks!
-Nate
Rachana Ananthakrishnan wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Neill,
I looked at this some more, and I don't think move to disable
SSLv3 is going to help. The issue seems to be the size of the
list of CA DNs we get from the server, and Safari having a lower
buffer size to store it, when compared to other browsers. It
might be worth looking at the trust store to see if there are CAs
we can eliminate - not ideal, but one option there is rather than
add CAs from ESGF to stock trust store that already has a bunch
of CAs, just create a trust store with only CAs from ESGF and
hopefully that should reduce the size some. I looked around some
more to see if I can find a definitive byte size limit that
Safari can handle, but no luck.
But if you would like to test SSLv3, here is what I tried:
- there is nothing in Safari to turn off SSL negotiation versions.
- Under Utilities and Java Application, you can look at Security
tab, and remove the selected SSLv3 version.
- Then restart browser and try to connect. I am not sure if that
disabled SSLv3 for Safari, but based on search this is the only
suggested option I found. Rachana
On Mar 1, 2011, at 9:13 AM, Neill Miller wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hello Rachana,
This is useful information. Where I left off last night was
wanting to try to disable SSLv3 on Safari, but I couldn't figure
out how! If you're able to do that, or have done it already,
let's coordinate a test where I watch the logs and you do it so
that I can see what's going on with the handshake.
The consensus from both of our findings is that it's a problem
specific to Safari, and if the truststore size is an issue as
indicted below, there may be no obvious fix. Let's try the
above and a couple more tests to be sure though.
thanks,
-Neill.
----- Original Message -----
From: "Rachana Ananthakrishnan"<a class="moz-txt-link-rfc2396E" href="mailto:ranantha@mcs.anl.gov"><ranantha@mcs.anl.gov></a>
To: "Neill Miller"<a class="moz-txt-link-rfc2396E" href="mailto:neillm@mcs.anl.gov"><neillm@mcs.anl.gov></a>
Cc: "Eric Nienhouse"<a class="moz-txt-link-rfc2396E" href="mailto:ejn@ucar.edu"><ejn@ucar.edu></a>, "Nathan Wilhelmi"
<a class="moz-txt-link-rfc2396E" href="mailto:wilhelmi@ucar.edu"><wilhelmi@ucar.edu></a>, "Dean N. Williams"<a class="moz-txt-link-rfc2396E" href="mailto:williams13@llnl.gov"><williams13@llnl.gov></a>,
<a class="moz-txt-link-abbreviated" href="mailto:vets-sgg@ucar.edu">vets-sgg@ucar.edu</a>, "Philip Kershaw (BADC)"
<a class="moz-txt-link-rfc2396E" href="mailto:philip.kershaw@stfc.ac.uk"><philip.kershaw@stfc.ac.uk></a>
Sent: Monday, February 28, 2011 7:55:26 PM
Subject: Re: Truststore issue
Hi,
I have tried a bunch of things here, but without server side
logs I am a little blinded. Nothing worked end to end, but I
found a plausible theory on failure. 1. Turned off the SSLv3
usage in my Java application as suggested by some thread. It
doesn't seem to affect Safari, which surprisingly has poor
logging of its SSL capabilities. I am convinced Safari has no
way to pick SSL ciphers or protocol versions, which is really
disappointing! 2. WIreshark is pretty hard to get up and running
on Mac OS, so I am not able to tell what was on the wire. I'll
try to coordinate this test with Neill to see if indeed aTLSv1
was attempted.
3. Of course, a openssl s_client works well, and is able to
establish a context with no issues. 4. I tried to explicitly set
a client certificate to the URL in my KeyChain to force Safari
to use it, but no such luck.
5. Many other trivial tweaks and posts of complaints about
Safari, but this one is the most clear explanation I could see:
<a class="moz-txt-link-freetext" href="http://restlet-discuss.1400322.n2.nabble.com/SSL-modifications-since-2-0-RC-1-td4962485.html">http://restlet-discuss.1400322.n2.nabble.com/SSL-modifications-since-2-0-RC-1-td4962485.html</a>.
On that thread post from Bruno Harbulot Apr 30, 2010; 10:06am.
Given it is many email deep, pasting relevant responses below,
pardon the long email. Boils down to whether SSL stacks can
handle the large number of CA DNs that are sent to the client
when a client authentication is required. Based on the openssl
s_client command I ran the NCAR prototype site returns 19131
bytes. Here is relevant paste:
--------- OpenSSL Command -------------
[ranantha@veenaciuchicagoedu:~] openssl s_client -connect
esg.prototype.ucar.edu:443
CONNECTED(00000003)
depth=3 /C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by
ref. (limits liab.)/OU=(c) 1999 Entrust.net
Limited/CN=Entrust.net Secure Server Certification Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0
<snip>
--
SSL handshake has read 19131 bytes and written 297 bytes
---
New, TLSv1/SSLv3, Cipher is EDH-RSA-DES-CBC3-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : EDH-RSA-DES-CBC3-SHA
Session-ID:
4D6C4DE08ABFC460971771528FCCF8D55B111D8BD83ADC1AF81E2D3A1CD0BD25
Session-ID-ctx: Master-Key:
4931083F443BAE48DF6F0AAEA08941F5C92345BD829FD7EFCE4F686D1FBA4479D16A5ADE9769C45BBB9141AF130CAA1D
Key-Arg : None
Start Time: 1298943456
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
--------- OpenSSL Command -------------
The above seems consistent with the behavior Eric was seeing,
with a smaller trust store working. I'll search to see if there
are workaround, but I doubt we will have a way out of trying to
slim down our trust store.
Rachana
--------- Relevant post messages -------------
"I've looked a bit more into this issue, and it doesn't seem
OSX-specific (on the server side) unfortunately. It's not even
specific to self-signed certificates (I've tried with a trusted
cert), the Simple connector or Restlet. I've managed to
reproduce it with a basic Jetty server [1] running with the Sun
JVM on a Linux box. It has similar symptoms to the buffer issue
we had a few weeks ago [2], but I don't know whether it's a Java
or a Safari problem at this stage. It appears with Simple,
because Simple always requests (wants) a client certificate. It
also happens with Jetty if it's configured to do so. To (try to)
keep it short, when a TLS server requests a client certificate
during the handshake, and extra TLS message is sent
(CertificateRequest), which contains a list of the names of the
CAs from which the server would be willing to accept client
certificates. In Java, this list is automatically populated by
the default SSLContext with the list of names corresponding to
the certificates held in the truststore. The more certificates
in the truststore, the longer the list, and thus the bigger the
packet. On the current OSX 10.6, the default truststore [3]
seems to have about 160 certificates, whereas on an Ubuntu 9.10
(Sun JVM) the default trust store contains about 75. When I use
the Mac's truststore on the Linux box, this fails too. When I
use a smaller truststore on the Mac, it works there. I've tried
to debug the SSL handshake on the Java side and it looks like it
works fine, internally; however, when looking at the packets
with WireShark, not everything sent by the server according to
the Java debugging logs seems to be effectively sent. I suspect
this is a buffer size problem similar to [2], although in this
case, it happens with other servers, using NIO or BIO.
A workaround would be to specify a smaller truststore or to turn
off the optional client-certificate authentication.
Best wishes, Bruno. "
"It looks like it's a more general problem, with some buffer
stack settings in various TLS stacks. I've just tried to set up
Apache Httpd 2.2 with a large number of certificates too. Above
about 150 DNs, it doesn't even accept connections from either
Firefox or the OpenSSL test client (I suppose this number may
vary slightly depending the length of the distinguished names in
the certificates). This makes the server send something around
22000 bytes during the handshake. When Apache Httpd is
configured with 118 CA certificates (again the exact number may
vary), Firefox and the OpenSSL client can connect to it, but not
Safari. (The server sends about 18000 bytes.)
Best wishes, Bruno. "
--------- Relevant post messages -------------
On Feb 28, 2011, at 5:56 PM, Neill Miller wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Thanks, no change in Safari, but do note the previous log
difference that I sent (regarding TLSv1 and SSLv3). I'll keep
looking around.
-Neill.
----- Original Message -----
From: "Eric Nienhouse"<a class="moz-txt-link-rfc2396E" href="mailto:ejn@ucar.edu"><ejn@ucar.edu></a>
To: "Neill Miller"<a class="moz-txt-link-rfc2396E" href="mailto:neillm@mcs.anl.gov"><neillm@mcs.anl.gov></a>
Cc: "Nathan Wilhelmi"<a class="moz-txt-link-rfc2396E" href="mailto:wilhelmi@ucar.edu"><wilhelmi@ucar.edu></a>, "Dean N. Williams"
<a class="moz-txt-link-rfc2396E" href="mailto:williams13@llnl.gov"><williams13@llnl.gov></a>, <a class="moz-txt-link-abbreviated" href="mailto:vets-sgg@ucar.edu">vets-sgg@ucar.edu</a>, "Rachana
Ananthakrishnan"<a class="moz-txt-link-rfc2396E" href="mailto:ranantha@mcs.anl.gov"><ranantha@mcs.anl.gov></a>
Sent: Monday, February 28, 2011 5:54:24 PM
Subject: Re: Truststore issue
Neill,
The esg.prototype.ucar.edu site is back up w/ TLSv1 change.
-Eric
Eric Nienhouse wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Neill,
OK - I'll change the connector to the below for a test. I
think the server (already) does support TLSv1 based on the SSL
logging.
<Connector port="443" protocol="HTTP/1.1" SSLEnabled="true"
address="128.117
.12.102"
maxThreads="150" scheme="https" secure="true"
keystoreFile="/usr/local/certificates/star_prototype_ucar_edu.jks
" keypass="*****"
clientAuth="want" sslProtocol="TLSv1"
/>
I'll let you know once the server is back up...
-Eric
Neill Miller wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Thanks Eric (and Nate),
Can you humor me and change "TLS" to "TLSv1"? The Safari
connection is coming in as SSLv3 and I suspect that's the
problem. I've found some information that claims there's a
safari problem if TLSv1 isn't supported and I'd like to
eliminate that possibility, or see if it fixes the issue.
thanks,
-Neill.
----- Original Message -----
From: "Eric Nienhouse"<a class="moz-txt-link-rfc2396E" href="mailto:ejn@ucar.edu"><ejn@ucar.edu></a>
To: "Neill Miller"<a class="moz-txt-link-rfc2396E" href="mailto:neillm@mcs.anl.gov"><neillm@mcs.anl.gov></a>
Cc: "Nathan Wilhelmi"<a class="moz-txt-link-rfc2396E" href="mailto:wilhelmi@ucar.edu"><wilhelmi@ucar.edu></a>, "Dean N. Williams"
<a class="moz-txt-link-rfc2396E" href="mailto:williams13@llnl.gov"><williams13@llnl.gov></a>, <a class="moz-txt-link-abbreviated" href="mailto:vets-sgg@ucar.edu">vets-sgg@ucar.edu</a>, "Rachana
Ananthakrishnan"<a class="moz-txt-link-rfc2396E" href="mailto:ranantha@mcs.anl.gov"><ranantha@mcs.anl.gov></a>
Sent: Monday, February 28, 2011 5:34:47 PM
Subject: Re: Truststore issue
Hi Neill,
The server.xml is in:
/usr/local/esg-staging/conf/server.xml. Here's the SSL
connector section:
<Connector port="443" protocol="HTTP/1.1" SSLEnabled="true"
address="128.117
.12.102"
maxThreads="150" scheme="https" secure="true"
keystoreFile="/usr/local/certificates/star_prototype_ucar_edu.jks"
keypass="*****"
clientAuth="want" sslProtocol="TLS"
/>
This file is root read only. I can make a readable copy of
the file for you if you need it.
-Eric
Neill Miller wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hello Eric,
Where is the running tomcat installation's server.xml config
file? I would have guessed /usr/local/apache-tomcat/conf,
but there is no conf directory there. I want to verify if
it supports TLSv1, in addition to either SSL2.0/SSL3.0. It
does appear that some versions of Safari has issues if the
server does not also support TLSv1. If you know it supports
it already, just let me know and I'll look at other possible
options.
thanks,
-Neill.
----- Original Message -----
From: "Eric Nienhouse"<a class="moz-txt-link-rfc2396E" href="mailto:ejn@ucar.edu"><ejn@ucar.edu></a>
To: "Neill Miller"<a class="moz-txt-link-rfc2396E" href="mailto:neillm@mcs.anl.gov"><neillm@mcs.anl.gov></a>
Cc: "Nathan Wilhelmi"<a class="moz-txt-link-rfc2396E" href="mailto:wilhelmi@ucar.edu"><wilhelmi@ucar.edu></a>, "Dean N.
Williams"<a class="moz-txt-link-rfc2396E" href="mailto:williams13@llnl.gov"><williams13@llnl.gov></a>, <a class="moz-txt-link-abbreviated" href="mailto:vets-sgg@ucar.edu">vets-sgg@ucar.edu</a>, "Rachana
Ananthakrishnan"<a class="moz-txt-link-rfc2396E" href="mailto:ranantha@mcs.anl.gov"><ranantha@mcs.anl.gov></a>
Sent: Monday, February 28, 2011 5:02:30 PM
Subject: Re: Truststore issue
Hi Neill,
You should have what you need to view these logs in real
time on the test server.
You'll need to (ssh) login to: vetswebdev.ucar.edu
Let me know if you need the steps to do so - you likely need
to connect first to the NCAR gate machine using your
crypto-card.
The log file of interest is at:
/usr/local/esg-staging/logs/catalina.out
Please let me know of anything else you may need.
Thanks,
-Eric
Neill Miller wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hello Eric,
Perfect, that's what I needed to know, thanks :-)
I am seeing the error now. I have a cryptocard and NCAR
account -- would I have access to the test installation to
view logs in real-time?
-Neill.
----- Original Message -----
From: "Eric Nienhouse"<a class="moz-txt-link-rfc2396E" href="mailto:ejn@ucar.edu"><ejn@ucar.edu></a>
To: "Neill Miller"<a class="moz-txt-link-rfc2396E" href="mailto:neillm@mcs.anl.gov"><neillm@mcs.anl.gov></a>
Cc: "Nathan Wilhelmi"<a class="moz-txt-link-rfc2396E" href="mailto:wilhelmi@ucar.edu"><wilhelmi@ucar.edu></a>, "Dean N.
Williams"<a class="moz-txt-link-rfc2396E" href="mailto:williams13@llnl.gov"><williams13@llnl.gov></a>, <a class="moz-txt-link-abbreviated" href="mailto:vets-sgg@ucar.edu">vets-sgg@ucar.edu</a>,
"Rachana Ananthakrishnan"<a class="moz-txt-link-rfc2396E" href="mailto:ranantha@mcs.anl.gov"><ranantha@mcs.anl.gov></a>
Sent: Monday, February 28, 2011 4:43:44 PM
Subject: Re: Truststore issue
Hi Neill,
The <a class="moz-txt-link-abbreviated" href="http://www.earthsystemgrid.org">www.earthsystemgrid.org</a> site (NCAR production) is no
longer showing the Safari SSL connection issue. In order
to allow Safari users to login in the short term, we've
changed the tomcat SSL configuration to
'clientAuth="false"' on this server. This resolves the
Safari SSL issue, but renders publishing disabled.
The esg.prototype.ucar.edu site demonstrates the problem
currently (as well as the production DKRZ site:
<a class="moz-txt-link-freetext" href="http://ipcc-ar5.dkrz.de">http://ipcc-ar5.dkrz.de</a> afaik.)
I've been testing with Safari 5.0.3 on OS X 10.5. I can
get you the exact versions if you need them.
I can trigger this error when clicking on the "Login" tab
at the top of the home page at the NCAR staging (eg: test)
site:
<a class="moz-txt-link-freetext" href="https://esg.prototype.ucar.edu/ac/guest/secure/sso.htm">https://esg.prototype.ucar.edu/ac/guest/secure/sso.htm</a>
Sorry I didn't make the earthsystemgrid.org configuration
clear. The SSL log sent earlier was based on testing of
esg.prototype.ucar.edu.
Thanks,
-Eric
Neill Miller wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hello Eric,
Hrm, I'm wondering if I've missed something on this
discussion, so please bear with me. My impression is that
there's a problem from the earthsystemgrid.org Tomcat
installation when a client is connecting via Safari. What
are the Safari and OS specs? I am using Safari 5.0.3
(6533.19.4) on OS X 10.6.6 and was able to get to the
login screen. Attached is the security exception
confirmation I had to click through in order to get there.
Can you describe exactly what you're doing to trigger the
error and on what setup?
thanks,
-Neill.
----- Original Message -----
From: "Neill Miller"<a class="moz-txt-link-rfc2396E" href="mailto:neillm@mcs.anl.gov"><neillm@mcs.anl.gov></a>
To: "Eric Nienhouse"<a class="moz-txt-link-rfc2396E" href="mailto:ejn@ucar.edu"><ejn@ucar.edu></a>
Cc: "Nathan Wilhelmi"<a class="moz-txt-link-rfc2396E" href="mailto:wilhelmi@ucar.edu"><wilhelmi@ucar.edu></a>, "Dean N.
Williams"<a class="moz-txt-link-rfc2396E" href="mailto:williams13@llnl.gov"><williams13@llnl.gov></a>, <a class="moz-txt-link-abbreviated" href="mailto:vets-sgg@ucar.edu">vets-sgg@ucar.edu</a>,
"Rachana Ananthakrishnan"<a class="moz-txt-link-rfc2396E" href="mailto:ranantha@mcs.anl.gov"><ranantha@mcs.anl.gov></a>
Sent: Monday, February 28, 2011 1:49:43 PM
Subject: Re: Truststore issue
Hello Eric,
Sorry for the delay, I was out last Friday and have been
looking into other issues. This is certainly a priority,
and I'll be taking a look at this later today.
-Neill.
----- Original Message -----
From: "Eric Nienhouse"<a class="moz-txt-link-rfc2396E" href="mailto:ejn@ucar.edu"><ejn@ucar.edu></a>
To: "Rachana Ananthakrishnan"<a class="moz-txt-link-rfc2396E" href="mailto:ranantha@mcs.anl.gov"><ranantha@mcs.anl.gov></a>,
"Neill Miller"<a class="moz-txt-link-rfc2396E" href="mailto:neillm@mcs.anl.gov"><neillm@mcs.anl.gov></a>
Cc: "Nathan Wilhelmi"<a class="moz-txt-link-rfc2396E" href="mailto:wilhelmi@ucar.edu"><wilhelmi@ucar.edu></a>, "Dean N.
Williams"<a class="moz-txt-link-rfc2396E" href="mailto:williams13@llnl.gov"><williams13@llnl.gov></a>, <a class="moz-txt-link-abbreviated" href="mailto:vets-sgg@ucar.edu">vets-sgg@ucar.edu</a>
Sent: Monday, February 28, 2011 1:02:30 PM
Subject: Re: Truststore issue
Hi Rachana, Neill,
We're still experiencing this issue (eg: Safari can't
connect over https) at NCAR. We've temporarily disabled
publishing at NCAR as a work-around to allow Safari users
to login.
Have you been able to review the SSL logging output?
Anything jumping out?
It is still unclear to me why Safari is unable to connect
to the <a class="moz-txt-link-abbreviated" href="http://www.earthsystemgrid.org">www.earthsystemgrid.org</a> (and DKRZ) server using SSL.
I've been looking at the standard truststore
(esg-truststore.ts from the pcmdi rainbow service) as well
as the SSL log output. Here are some thoughts:
1) The SSL log indicates at least 2 Cert Authorities sent
in response to the client CertificateRequest DN strings
are duplicates. For example:
<CN=UTN-USERFirst-Hardware, OU=<a class="moz-txt-link-freetext" href="http://www.usertrust.com">http://www.usertrust.com</a>,
O=The USERTRUST Network, L=Salt Lake City, ST=UT, C=US>
<OU=Go Daddy Class 2 Certification Authority, O="The Go
Daddy Group, Inc.", C=US>
I'm not sure if this is a problem for Safari (tho, I
suspect it is.) Note simply removing the duplicate certs
still produces the Safari connection error.
2) After the CertificateRequest DNs are sent, the remote
host (Safari) appears to close the connection. It is
unclear as to the reason:
http-128.117.12.102-443-1, received EOFException: error
http-128.117.12.102-443-1, handling exception:
javax.net.ssl.SSLHandshakeException: Remote host closed
connection during handshake
http-128.117.12.102-443-1, SEND SSLv3 ALERT: fatal,
description = handshake_failure
3) I've been testing federation certificates found in the
truststore (actually from the rainbow certificates tar
file.) Below are some raw notes from this effort.
So far only a small number of certificates seem to work
with Safari 5.0.3 *always* (these are labeled as "GOOD").
The "BAD" certificates cause failures w/ Safari in certain
combinations with others. This has me wondering if there
are other dependencies on the TS, such as certificate
order, cipher support, and possibly missing related CA
certs. I can only offer speculation at this point and
need to dig in further.
Your help on this is greatly appreciated.
-Eric
#
# ESGF certificate study notes. Certificates added to
Java 1.6.0_23-b05 default cacerts file to build truststore:
#
0084963c OK - vetsdevprod/NCAR prod
96e2ef2f OK - ceda.ac.uk
cf22df3a OK - ceda #2
1c3f2ca8 OK - DOE Grids
9490b52e OK - ORNL
/O=ESG/OU=ESG-ORNL/OU=NCCS/CN=esg2-gw.ccs.ornl.gov
a03bd34c OK - ORNL Already exists in cacerts.good
7395b665 OK - DKRZ /C=DE/O=Deutsches
Klimarechenzentrum GmbH/CN=DKRZ CA -
<a class="moz-txt-link-abbreviated" href="mailto:G02/emailAddress=pki@dkrz.de">G02/emailAddress=pki@dkrz.de</a> d1b603c3 BAD - ESNet
root CA 053e3ae0 BAD
1149214e BAD
159117b6 BAD
1e12d831 BAD
97f17aeb BAD - PCMDI SSL/Verisign issued
219d9499 OK
367b75c3 BAD
3c58f906 OK
49929bb3 BAD
49bc0f05 BAD
4d241d64 BAD
4d654d1d OK
4e18c148 OK
594f1775 OK
5aa2594b BAD
72fa7371 BAD
7ffb3ace BAD
9388e5cb BAD
97552d04 BAD
98ef0ee5 BAD
9df51c42 BAD
aaa0e946 BAD
ae9c66bf BAD
bc3f2570 BAD
c331edde BAD
c4a11bb8 BAD
c6d11c74 BAD
ece35fd4 BAD
faa5efcb BAD
ff783690 BAD
Nathan Wilhelmi wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Rachana& Neill,
Here is the tomcat log file from the this morning. We put
the latest truststore from
<a class="moz-txt-link-freetext" href="https://rainbow.llnl.gov/dist/certs/esg-truststore.ts">https://rainbow.llnl.gov/dist/certs/esg-truststore.ts</a> in
place this morning. The log should represent starting
tomcat and then trying to access the login page with Safari.
Thanks for the help!!!
-Nate
Rachana Ananthakrishnan wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Nate, Eric,
Neill and I are happy to help with this issue - please
let us know. At this point my best bet is to instrument
the NCAR Gateway to dump SSL dumps when the handshake
fails, so we can see what the specific issue is.
We can try to replicate it here, but I know it will
probably be more of your time to just get a Gateway
running at our end? If this has been isolated as Tomcat
issue, and if you can send us specific configuration of
the trust store and other related parameters for Tomcat
we can replicate. I hesitate to do this last option,
because there are many parameters on this that we might
no be capturing correctly.
Either way, I am willing to put some resource on this
and help, so please let us know.
Thanks,
Rachana
Rachana Ananthakrishnan
Argonne National Lab | University of Chicago
</pre>
</blockquote>
</blockquote>
<pre wrap="">------------------------------------------------------------------------
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">Rachana Ananthakrishnan
Argonne National Lab | University of Chicago
</pre>
</blockquote>
<pre wrap="">Rachana Ananthakrishnan
Argonne National Lab | University of Chicago
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
</blockquote>
<pre wrap="">Rachana Ananthakrishnan
Argonne National Lab | University of Chicago
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<pre wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<pre wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<pre wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<pre wrap="">
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Gavin M. Bell
Lawrence Livermore National Labs
--
"Never mistake a clear view for a short distance."
-Paul Saffo
(GPG Key - <a class="moz-txt-link-freetext" href="http://rainbow.llnl.gov/dist/keys/gavin.asc">http://rainbow.llnl.gov/dist/keys/gavin.asc</a>)
A796 CE39 9C31 68A4 52A7 1F6B 66B7 B250 21D5 6D3E
</pre>
</body>
</html>