Tuesday, May 11, 2010

How to debug SSL issues with weblogic server

First of all I would suggest using the following debug flag in case of any kind of SSL issue on Weblogic server:


-Dweblogic.StdoutDebugEnabled=true


-Dssl.debug=true


 


If you need to debug any SSL issue with the standalone JAVA class then you can use the following debug flag:


-Djavax.net.debug=all :                                This is for turning all debugging.


-Djavax.net.debug=ssl :                This is for turning SSL debugging.


Then In case if you are getting the following exception in the debug trace:



Errors:



FINE: Cannot complete the certificate chain: No trusted cert found


FINE: Validating certificate 0 in the chain: Serial number: 1398096
Issuer:C=CZ, CN=I.CA - Standard root certificate, O=Prvni certifikacni autorita a.s.
Subject:C=SK, CN=damas.sepsas.sk, ST=Slovakia, L=Bratislava, O=Slovenska elektrizacna prenosova sustava, a.s., OU=Damas Energy, ?=ICA - 595029
Not Valid Before:Tue Aug 11 12:07:51 CEST 2009
Not Valid After:Wed Aug 11 12:07:51 CEST 2010
Signature Algorithm:SHA1withRSA


FINE: validationCallback: validateErr = 16


Then you can follow the following steps to resolve this issue:
1: If you are running your java code as a standalone class then you should use the following to run the java code:
java -Djavax.net.ssl.trustStore=trustStore -Djavax.net.ssl.trustStorePassword=trustStorePassword YouJAVAclass


where,
trustStore: is the keystore which should contain the root certificate from the server hosting the service.
trustStorePassword: keystore password for the trustStore.


2: If the java code is running on the WLS server then WLS server being the client you should use:
-Djava.protocol.handler.pkgs=weblogic.net
-Dweblogic.security.SSL.trustedCAKeyStore=trustStore


-------------------------------------------------------------------------------------------------------------------------------------------------------


Error:


FINE: ........... Eating Exception ..........
java.security.NoSuchAlgorithmException: Algorithm ECDH not available
+ at javax.crypto.KeyAgreement.getInstance(DashoA13*..)+


Reason:


This is something like both the Client and the Server is not able to negotiate on a common Cipher (Algorithm) and hence the handshake is failing.


Just to give you a brief idea: the Cipher is initialized by the Client end and Sever has to choose any one of the Ciphers presented by the Client to communicate on SSL.
If the Server does not supports any such Cipher algorithm which is common to Client then the communication will not happen.


So, If can force the Client (Weblogic) to use the weaker ciphers and the Server does not have any constraints on using the limited ciphers then we can make the connection over SSL.


Resolution:


As by default Weblogic Server uses the certicom implementation of SSL.


The above exception is because the certicom implementation of SSL is not able get a common cipher negotiation.


We can tell Weblogic server to use the SUN implementation of SSL to solve the issue.


In order to use the SUN implementation of SSL we can use the following properties in the Weblogic server:


-Djavax.net.ssl.keyStore
-Djavax.net.ssl.keyStoreType
-Djavax.net.ssl.keyStorePassword
-Djavax.net.ssl.trustStore
-Djavax.net.ssl.trustStoreType
-Djavax.net.ssl.trustStorePassword
-Djava.protocol.handler.pkgs=com.sun.net.ssl.internal.www.protocol


all the above properties can be used as JAVA properties so that the WLS JVM uses these values rather than using its own implementation.


We can also use the above properties in our code as mentioned below:


System.setProperty( "javax.net.ssl.keyStore", "***" );
System.setProperty( "javax.net.ssl.keyStoreType", "JKS" );
System.setProperty( "javax.net.ssl.keyStorePassword", "***" );
System.setProperty( "javax.net.ssl.trustStore", "***" );
System.setProperty( "javax.net.ssl.trustStoreType", "JKS" );
System.setProperty( "javax.net.ssl.trustStorePassword", "***" );
Security.addProvider( new com.sun.net.ssl.internal.ssl.Provider() );
System.setProperty( "java.protocol.handler.pkgs", "com.sun.net.ssl.internal.www.protocol" );


-------------------------------------------------------------------------------------------------------------


Let us understand one way SSL behavior:


WLS as Client:


Remote Service as Server to be accessed over SSL:


This is broader picture of SSL.



  1. Client (WLS) Initiates the Handshake with the server (Remote Service).

  2. Server (Remote Service) will present its certificate to the Client (WLS).

  3. Client (WLS) will verify the certificate by looking into the Trust Keystores.

  4. That is why there is a need to import the Server Certificate (Remote Service certificate) here into WLS Trust Keystore.

  5. Since by default DemoTrust.jks is used as the Trust keystore by WLS hence the Remote Service certificate is imported generally into the DemoTrust.jks file.

  6. Then Client will send the list of Ciphers to be used for encryption over the network.

  7. If the Server supports some of the cipher present by the Client then Handshake is successful and SSL communication will happen.

  8. If there is no common ciphers between the Client (WLS) and the Server then SSL will not happen.


How to debug:



  1. Try testing the Remote Service connectivity with the standalone client to test whether the service is accessible over SSL or not. This is clear whether the issue is with the server or the client.

  2. If the service is accessible with standalone client and it is not accessible from WLS then we should try to run the standalone class with the same JDK/JRE which is being used by the WLS.

  3. This is clear up whether the issue is with the JDK/JRE or it is the issue with the WLS configuration.


--------------------------------------------------------------------------------------------------------------------


javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target


How to debug above Error:


The above error simply means that the URL that you are trying to access does not have a Valid Certificate or the Certificate used by the URL provider is not issue by a Trusted Certificate Authority.


in Order to remove the above error message, there can be two secnarios:


1: When the Client which is trying to access the URL is a Java Standalone Client then we have to import the root certificate of the URL into the cacerts file present in the JAVA/jdk/jre/lib/security/ directory.


2: If the Client is the WLS server trying to access the URL, then we need to determine what is the Trust Store used by the Weblogic Server.


If the Weblogic server is using DemoTrust then we can import the root certificate of the URL in the cacerts file of the JAVA used by the WLS server.


If the Weblogic Server is using Custom Trust then we can import the root certificate of the URL in the Custom Trust keystore used by the weblogic server.


How to get the root certificate of the URL:


1: Copy the URL and paste in the browser address bar , let us say IE and press Enter.


2: It will show a waring message showing the certificate is not valid, we have to click on the view certificate option of the pop-up and then we have to click on the Install certificate.


3: Then we need to open the IE >>> Internet Options >>> contents >>> certificates >>> Trusted Certificate option.


Find the certificate just installed and export that certificate into DER encoded binary format and store it as C:/root.cer


Now in order to import this certificate into the cacert file or into any keysttore we have to use the following keytool utility from JAVA :


keytool -importcert -v -alias root_cert -file c:/root.cer -keystore c:/java/jdk/jre/lib/security/cacerts -storepass changeit


-alias: this can be anything .


-file: this is the file needs to be imported.


-keysotore: this is the keystore file where we need to import the certificate.


-storepass: this is the password for the keystore file, which is "changeit" by default for cacerts.

18 comments:

Ravish Mody said...

Hi Sandeep,

Can you please have a look at the below comment:

Issue with the node manager and SSL

weblogictips said...

Hi ,

We will have to turn of host name verification and the basic validation check of the certificates.

We can do it by specifying the following flag in startWeblogic.sh

-Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.SSL.enforceConstraints=off

And the following flag in startNodeManager.sh

-Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.security.SSL.enforceConstraints=off

thanks, sandeep

Ajeeb Peter said...

Sandeep,

I am using WebLogic Server Version: 10.3.1.0. and and we do have a DemoTrust weblogic keystore. We have added many certificates and its all working fine but recently we have added a new certificate with our new vendor which seems not working. and I am getting error :

"stackTrace:javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"

Below some facts on my testcases
1)
I could verify the same certificate using below curl command, which I added successfully to my DemoTrust Keystore.

2)
curl -k --cacert /oss/app/weblogic/WLS10_3_1/user_projects/domains/xxx/certificates/wsgateway-ext-test.xxx.com.crt --data-binary ./activateNewCustomer.xml -v https://wsgateway-ext-test.xxx.com/TEST3/public/interfaces/wholesale/CustomerProvisioning

3) I verified that the certificate loaded to key store by listing it

4) I could see my DemoTrust key store being loaded whenever the first SSL request being initiated. See the logs Below




5)
I added the same certificate under java keystore "cacerts" then the SSL connection working successfully. but not from the DemoTrust.jks


In nutshell The certificate (Both root & intermediate) are working fine from java keystore but not from DemoTrust. same time all existing certificates on the Demotrust working fine as well. We are not enabled our "SSL Listen Port Enabled" option since we are not expecting any secured requests.

Any help on this issue greatly appreciated.

weblogictips said...

Hi Ajeeb,

As per the scenario mentioned by you, I can only assume that the vendor application must have mentioned the Trust Keystore location explicitly and while processing the request Weblogic is not able to find that mentioned keystore location.

Also, if the Keystore location is explictuly mentioned then WLS is not going to check the DemoTrust.jks for the Trust certificate.
It is working with cacerts because all the certificates present in the cacerts are trusted by default.

thanks,
Sandeep

Ajeeb Peter said...

Sandeep,

I am confused with the term Vendor Application. here we are invoking the vendor application and our WLS have the certificate and as I mentioned we were able curl the vendor application with same certificate.

I even tried starting the server with below options but no luck. and moreover all my other vendor certificates working whcih added in same Demotrust.

-Djava.protocol.handler.pkgs=weblogic.net -Dweblogic.security.SSL.trustedCAKeyStore=/oss/app/weblogic/WLS10_3_1/wlserver_10.3/server/lib/DemoTrust.jks

Thanks
A Peter

Sukrut Wagh said...

Hello Sandeep,
I am trying to invoke the microsoft exchange webservice within weblogic 10.3 server.
I performed the following steps:-
1.Got the certificate from server
2.Generated the stubs(JAX-WS) using 'weblogic.wsee.tools.anttasks.ClientGenTask' & deployed on weblogic
3.Added the certificate to jdk160_05jrelibsecuritycacerts. domain is configured with this JDK

I get the exception:-
com.sun.xml.ws.client.ClientTransportException: request requires HTTP authentication: Unauthorized
at com.sun.xml.ws.transport.http.client.HttpClientTransport.checkResponseCode(HttpClientTransport.java:232)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:149)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:86)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
at com.sun.xml.ws.client.Stub.process(Stub.java:248)
at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)

I have set the credentials are set as follows:-
requestContext.put(BindingProvider.USERNAME_PROPERTY, domainuser);
requestContext.put(BindingProvider.PASSWORD_PROPERTY, password);

The webservice client works well when invoked as stanalone app with weblogic's JDK/JRE

I am not sure if the handshake was successful as the standalone client printed the handshake info such as
*** ClientHello, TLSv1
*** ServerHello, TLSv1
***
Found trusted certificate:
*** ServerHelloDone
*** ClientKeyExchange, RSA PreMasterSecret, TLSv1
...
*** Finished

When the same webservice was invoked within weblogic sever, none of the above was printed.
I have set the -Djavax.net.debug=ssl in weblogic server. So, i am assuming the handshake was not sucessful.

Am i missing any configuration here?

weblogictips said...

Hi Reena,

java.security.cert.CertificateException: Could not get public key for: C=US, ST=MyState, L=MyTown, O=MyOrganization, OU=FOR TESTING ONLY, CN=CACERT
This exception shows that the file client certificate file that you are using "ClientCert.pem" does not contain the public certificate . It only contains the certificate of the CA, which is CertGenCA.

FINE: Problem accessing private key
java.security.KeyManagementException: Could not find certificate chain and/or private key
This exception shows that the private key is missing from ClientKey.pem.

thanks,
sandeep

Reena said...

Hi Sandeep,

Thanks for your response. I created the ClientCert.pem and ClientKey.pem using the commands that you had mentioned:

java utils.CertGen -certfile ClientCert -keyfile ClientKey -keyfilepass keypass
and
pkcs12 -export -in C:ServerCertClientCert.pem -inkey C:ServerCertClientKey.pem -out C:ServerCertclient-pkcs-12-cert

and then added the certificate to the Browser.

I also checked the certificate chain of ClientCert.pem and that shows:

C:ServerCert>keytool -printcert -file ClientCert.pem
Owner: CN=mytesthost, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Issuer: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Serial number: 62c7930a3f11e93b944ac728ab4d8183
Valid from: Sun Dec 11 12:52:14 GMT 2011 until: Sat Dec 12 12:52:14 GMT 2026
Certificate fingerprints:
MD5: B5:AA:A7:28:44:44:8A:A3:09:01:EF:80:60:95:DB:4A
SHA1: A8:17:E9:CC:DD:EE:C8:90:F0:46:A5:4E:C3:1D:09:05:CD:76:2F:19
Signature algorithm name: MD5withRSA
Version: 1

which means it has the public certificate issued by CertGenCA.

I verified this by checking the certificate chain of ServerCert.pem and myCert.pem (created by appending CertGenCA.pem to ServerCert.pem):

C:ServerCert>keytool -printcert -file ServerCert.pem
Owner: CN=mytesthost, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Issuer: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Serial number: -233b3b3e85f40bd5485214121f8c9d1d
Valid from: Sun Dec 11 12:51:00 GMT 2011 until: Sat Dec 12 12:51:00 GMT 2026
Certificate fingerprints:
MD5: 5C:F5:9A:74:DC:92:D2:0D:86:22:65:10:0F:9B:83:85
SHA1: A1:72:B2:51:5A:B9:46:3E:3A:52:4F:40:D4:05:55:17:54:8D:28:B6
Signature algorithm name: MD5withRSA
Version: 1

C:ServerCert>keytool -printcert -file myCert.pem
Certificate[1]:
Owner: CN=mytesthost, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Issuer: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Serial number: -233b3b3e85f40bd5485214121f8c9d1d
Valid from: Sun Dec 11 12:51:00 GMT 2011 until: Sat Dec 12 12:51:00 GMT 2026
Certificate fingerprints:
MD5: 5C:F5:9A:74:DC:92:D2:0D:86:22:65:10:0F:9B:83:85
SHA1: A1:72:B2:51:5A:B9:46:3E:3A:52:4F:40:D4:05:55:17:54:8D:28:B6
Signature algorithm name: MD5withRSA
Version: 1

Certificate[2]:
Owner: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Issuer: CN=CertGenCAB, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Serial number: 234b5559d1fa0f3ff5c82bdfed032a87
Valid from: Thu Oct 24 16:54:45 BST 2002 until: Tue Oct 25 16:54:45 BST 2022
Certificate fingerprints:
MD5: A2:18:4C:E0:1C:AB:82:A7:65:86:86:03:D0:B3:D8:FE
SHA1: F8:5D:49:A4:12:54:78:C7:BA:42:A7:14:3E:06:F5:1E:A0:D4:C6:59
Signature algorithm name: MD5withRSA
Version: 3

which has 2 certificates, a public Certificate (Certificate[1]) and a certifying authority certificate (Certificate[2]).

We use a similar command to create the ServerCert and ServerKey ( java utils.CertGen -certfile ServerCert -keyfile ServerKey -keyfilepass keypass) , so how does it know that when we issue this command, that it should create a private Certificate and when we issue command (java utils.CertGen -certfile ClientCert -keyfile ClientKey -keyfilepass keypass), that it should create a public Certificate. Is that because how we configure it in WebLogic Server?

Could you please tell me how I can resolve the original issue? Thanks in advance,

Regards,
Reena.

Reena said...

Hello,

Sorry, in my earlier post, I meant that myCert.pem has the private (not public*) certificate (Certificate[1]). That was a typo.

Thanks & Regards,
Reena.

weblogictips said...

Hi Reena,

I am really confused with the certificates files your are using with the server and the client.
So, I will try to explain using general file name for your scenario:

JMX CLient -------> Two way SSL ------->WLS Server.
Now Files used by your JMX client :
jmx-cert.pem ----> public certificate for your JMX client.
jmx-key.pem ---> private key for your JMX client.
CertGenCA.der ---> CA file used by JMX client.

wls-cert.pem ----> public cert for wls server
wls-key.pem ----> private key for wls server.
CertGenCA.der ---> CA file used by wls server.

Now, as wls server uses jks keystore configuration files for ssl configuration. So, you must be having two jks files configured on wls server:
identity.jks ----> this file will store wls-cert.pem and wls-key.pem
DemoTrust.jks ---> as you are using demo trust for wls. this must have CertGenCA.der file as the CA root certificate.

and in you client code : you will be using jmx-cert.pem and jmx-key.pem for client cert and client key. and apart from the above files used in the code you will be passing the DemoTrust.jks file as command line argument while executing the JMX client with the flag : -Dweblogic.security.TrustKeyStore=DemoTrust

let me know if this clears your confusion .

thanks,
sandeep

weblogictips said...

Hi Raj,

This is really unexpected.
would ask you to go for oracle support. This is something unexplainable.
Sorry :(

weblogictips said...

Hi Raj,

Can you try the below flag :
-Dweblogic.security.SSL.hostnameVerifier=CustomHostnameVerifierClassName

weblogictips said...

Hi Raj,

What is the version of weblogic server you are using ?
And the web service client is JAX-RPC or JAX-WS based ?

Raj said...

Hi Sandeep,
First , Thanks for your quick response.

Weblogic Version is 10.3.2.0.
Client is JAX-WS.

Yes, I have tried supplying Custom HostNameVerifier via both ways (console as well -D option) but same results.

Out of curiosity, I have also started digging deeper and will share my findings with you (if i get lucky).

Once again, Thank you and keep up the Great work.

Regards,
Raj

Raj said...

Hi Sandeep,

Just wanted to share so that others can benefit if they are also struggling with the same issue.

Although it took time BUT I am able to make my code run successfully after digging deeper in the weblogic code.

Firstly, There is NO Issue with the weblogic default HostnameVerifier and that one is working fine. SO we don't need to override that one by supplying our Custom Hostname Verifier (which extends "weblogic.security.SSL.HostnameVerifier") either by -D option OR Admin console.

The real issue is :
Not sure if its a BUG or normal behaviour of weblogic BUT Unfortunately its NOT documented anywhere.

Somewhere in weblogic code , Weblogic's defualt hostname Verifier (which extends "weblogic.security.SSL.HostnameVerifier") is getting overridden with the ONE which extends from ("javax.net.ssl.HostnameVerifier") and the default implementation of SUN based hostname verifier is to compare "localhost" Otherwise return false (nothing but hostname verification fail).

So In order to overcome with this issue, We got to following steps.
a) NO NEED to implement Weblogic Specific Custom HostnameVerifier.
b) Write a Class with extends "javax.net.ssl.HostnameVerifier" and implement "verify" method.
c) In JAX-WS client code, set below property on Stub as follows:
((BindingProvider)port_stub).getRequestContext().put(com.sun.xml.ws.developer.JAXWSProperties.HOSTNAME_VERIFIER,new JDKHostNameVerifier());

Where JDKHostNameVerifier is my Custom class as mentioned in (b).

NOTE : Another point of confusion : JAXWSProperties.HOSTNAME_VERIFIER class is a part of Weblogic kit bundled under "modules\glassfish.jaxws.rt_1.1.0.0_2-1-4.jar", NOT the ONE which is part of JDK.

Hope it may help others.

I am not sure if its a HACK or the way to solve this issue. Please let me know if you find some other solution to same issue.

Regards,
Raj

Raj said...

Just FYI:

Somehow my response / resolution to that issue shows up under your message NOT the latest one @ bottom.

So please read the one updated @
February 15, 2012 at 4:33 pm

Regards
Raj

Raj said...

Hi Sandeep,

Another basic question

Is it possible to enable SSL related Debug statements @Runtime ?

We have weblogic 10.3.2 and sometime there is a need to enable SSL debugs to troubleshoot the connectivity or some other issues.

In order to do that, Unfortunately it needs -D debug parameters needs to be added to startup script and bounce Weblogic server (which is kind of pain in production environment).

We don't have admin previllege to Admin Console as well.

I tried to add Debug statements via System.setProperty @Runtime (whenever needed) But it seems its kind of static parameters which works only @Startup.

Followings are the debug parameters are tried.
Properties prop=System.getProperties();
prop.setProperty("weblogic.debug.DebugSecuritySSL","true");
prop.setProperty("weblogic.debug.DebugSecuritySSLEaten","true");
prop.setProperty("ssl.debug","true");
prop.setProperty("weblogic.StdoutDebugEnabled","true");
prop.setProperty("weblogic.security.SSL.verbose","true");

Any Help would be highly appreciated.

Regards,
Raj

weblogictips said...

Hi Raj,

Thanks a lot for sharing this information. I will try to create a page for the same so that other users do not have to struggle.

Also for SSL runtime debug, you can use the Admin Console .
For server log on to Admin Console >>>> Servers >>> MS1 >>> debug >>>> weblogic tree >>>> ssl option .

Check the option and click enable.

Thanks,
sandeep