1 2 Previous Next 15 Replies Latest reply on Mar 4, 2015 2:46 PM by Scott Dunbar

    SSL_connect Errors During Compliance

    Cody Dean

      Good Afternoon,

      We have imported the latest compliance content for 8.6P1 that we are running in our BSA environment and we started noticing weird things when trying to run the compliance content.

       

      An example, is when trying to run the PCI Compliance for Windows 2008, it fails at seemingly random points with errors similar to this:

      Error 02/23/2015 11:50:32 com.bladelogic.om.infra.app.collector.AssetCollectionException: SSL_connect2

      SSL_connect2

      SSL_connect

      SSL_connect

      SSL_connect2

      SSL_connect2

      SSL_connect

      SSL_connect

      (component=PCI Data Security Standard v3 - Windows Server 2008 (SERVERNAME), selector=Extended Object:User Rights Policy-2.2.3.89)

      It seems like I can run the old content just fine.

       

      I know SSL_connect has a "fix" of refreshing the certificates.pem file, and we tried that with no effect in this case.


      I also noticed the "BMCCacheCreator" jobs that the import created - but there isn't one for all options (such as PCI Win 2k8).

       

      Anyone with 8.6 or 8.6P1 that has imported the content having similar issues?


      Thanks,
      Cody

        • 1. Re: SSL_connect Errors During Compliance
          Dale Reagan

          We are seeing a similar issue with 8.6 and NSH - I will guess that they are related (i.e. a tuning issue.)

           

          I am looking at these values (which we previously changed when hitting a similar NSH issue with 8.2 - and noting that your issue many NOT be related.)

           

          One way to review values is to dump them to a text file via blasadmin, i.e.

           

          export BL_SERVER=Name_of_your_BL_server

          export JUl=$(date +%j) # set Julian date

          # get the data

          nexec ${BL_SERVER}  blasadmin.exe show all >  ${BL_SERVER}.all.vars.${JUL}.txt

          # show the file - anything there?

          ls -l ${BL_SERVER}.all.vars.${JUL}.txt

          # search for relevant values

          egrep -i "MaxWorkItemThreads|MaxNshProxyThreads|MaxNshProxyContexts|MaxNshProxyContexts" ${BL_SERVER}.all.vars.${JUL}.txt

           

          We are not seeing errors in the App or Console log, however we are seeing these in ~/NSH/br/AppServerLauncher.log:

           

          [23 Feb 2015 15:52:35,961] [RMI TCP Connection(45459)-xx.yy.zz.00] [WARN] [::] [] received serviceTicket with invalid signature from System

          [23 Feb 2015 15:52:35,961] [RMI TCP Connection(45459)-xx.yy.zz.00] [INFO] [::] [] BlSession authentication failed. java.lang.SecurityException: Failed to establish session

          • 2. Re: SSL_connect Errors During Compliance
            richard mcleod

            Sounds like BMC needs to replace your JRE folders... I think they messed up the available/acceptable encryption methods in 8.5 SP1 P4 + ... most likely it was necessary due to heartbleed or poodle but I think they're denying connections to the appserver using a previously acceptable form of encryption.

            • 3. Re: SSL_connect Errors During Compliance
              Dale Reagan

              Could be related, however, I noted the problem when I used Nsh to check RSCD agents, if I run the same set of agents then the error point is not consistent (i.e. my set of 'problem hosts' changes with each 'run'.)  Also, now thinking that the APP Launcer log info may not be related...

              • 4. Re: SSL_connect Errors During Compliance
                Bill Robinson

                can you attach the entire log that shows the ssl_connect errors so we can see which asset it's failing on ?

                • 5. Re: SSL_connect Errors During Compliance
                  Dale Reagan

                  Sorry, which log?  Noting that I have not seen any issue in the apppserver log that seems to correlate w/these errors.  Today, I ran a single agentinfo command - it failed the 1st time w/SSL errors and worked as expected on the 2nd try...  The SSL errors that I am seeing are via commnd line or re-directed from NSH scripts.

                  • 6. Re: SSL_connect Errors During Compliance
                    Bill Robinson

                    this one:

                     

                    An example, is when trying to run the PCI Compliance for Windows 2008, it fails at seemingly random points with errors similar to this:

                    Error02/23/2015 11:50:32com.bladelogic.om.infra.app.collector.AssetCollectionException: SSL_connect2

                    SSL_connect2

                    SSL_connect

                    SSL_connect

                    SSL_connect2

                    SSL_connect2

                    SSL_connect

                    SSL_connect

                    (component=PCI Data Security Standard v3 - Windows Server 2008 (SERVERNAME), selector=Extended Object:User Rights Policy-2.2.3.89)

                    • 7. Re: SSL_connect Errors During Compliance
                      Dale Reagan

                      Ummm, That was from the original poster - runing a compliance job (guessing from the console); my errors are showing up via NSH command-line or NSH scripts run from the command line.  Also, I am using a 'paralell process' where I kick off up to ~120 simultaneous agentinfo queries - my captured Error info:

                       

                      ### ai_scan.RMTH.nsh | Tue Feb 24 07:54:06 2015 | Errors    ###

                      ###  ###

                      SSL_connect2

                      SSL_connect2

                      SSL_connect

                      SSL_connect

                      Can't access host "error-host-001": Error in TLS protocol

                      SSL_connect2

                      SSL_connect2

                      SSL_connect

                      SSL_connect

                      Can't access host "error-host-002": Error in TLS protocol

                      SSL_connect2

                      SSL_connect2

                      SSL_connect

                      SSL_connect

                      Can't access host "error-host-003": Error in TLS protocol

                      • 8. Re: SSL_connect Errors During Compliance
                        Bill Robinson

                        Ah – sorry.  Is the nsh client going through a nsh proxy ? and what version of blade ?

                        • 9. Re: SSL_connect Errors During Compliance
                          Dale Reagan

                          8.6, of course!   Same process working with 8.2 and 8.3;   Blasadmin tuning settings the same as with 8.2 and 8.3.

                           

                          Also, I did try reducing the max paralell down - still get errors.

                          • 10. Re: SSL_connect Errors During Compliance
                            Bill Robinson

                            ?we had a couple customers on 8.5.01 getting these and it was related to using the nsh proxy and the ciphers used.  we altered the ciphers (code change + jars) and that seems to have resolved it.  i'd open a ticket (both of you)

                            • 11. Re: SSL_connect Errors During Compliance
                              Cody Dean

                              Thanks Bill, I submitted a support ticket yesterday.  Hopefully we can get it resolved

                              • 12. Re: SSL_connect Errors During Compliance
                                Cody Dean

                                Interesting finding yesterday.  I imported the content into our fresh lab environment which has a simplistic network layout (no proxies, ect) and ran compliance against the app server itself, and still got SSL_connect errors.

                                 

                                They seem to relate to this in the application log, same thing we are seeing in production.

                                 

                                [25 Feb 2015 17:42:01,052] [Nsh-Proxy-Thread-12] [WARN] [Anonymous:Anonymous:0:0:0:0:0:0:0:1] [BLSSOPROXY] javax.net.ssl.SSLHandshakeException: Invalid Padding length: 52

                                1. com.bladelogic.sso.engine.ClientDisconnectedException: javax.net.ssl.SSLHandshakeException: Invalid Padding length: 52

                                                at com.bladelogic.sso.engine.StreamMessaging.receiveMessage(StreamMessaging.java:123)

                                                at com.bladelogic.sso.engine.StreamMessaging.receiveToken(StreamMessaging.java:203)

                                                at com.bladelogic.om.infra.mfw.net.BlSessionServerConnection.authenticate(BlSessionServerConnection.java:186)

                                                at com.bladelogic.om.infra.mfw.net.BlSessionNshServerConnection.internalHandshake(BlSessionNshServerConnection.java:78)

                                                at com.bladelogic.om.infra.mfw.net.BlSessionNshServerConnection.doHandshake(BlSessionNshServerConnection.java:50)

                                                at com.bladelogic.om.infra.mfw.fw.BlSessionNshProxyPair.setupClient(BlSessionNshProxyPair.java:104)

                                                at com.bladelogic.om.infra.mfw.fw.BlSessionNshProxyPair.init(BlSessionNshProxyPair.java:75)

                                                at com.bladelogic.om.infra.mfw.fw.NshProxyWorkerThread.execute(NshProxyWorkerThread.java:106)

                                                at com.bladelogic.om.infra.mfw.fw.NshProxyWorkerThread.execute(NshProxyWorkerThread.java:17)

                                                at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                                Caused by: javax.net.ssl.SSLHandshakeException: Invalid Padding length: 52

                                                at sun.security.ssl.Alerts.getSSLException(Unknown Source)

                                                at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)

                                                at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

                                                at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)

                                                at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)

                                                at sun.security.ssl.AppInputStream.read(Unknown Source)

                                                at java.io.BufferedInputStream.fill(Unknown Source)

                                                at java.io.BufferedInputStream.read(Unknown Source)

                                                at com.bladelogic.sso.engine.StreamMessaging.receiveMessage(StreamMessaging.java:119)

                                                ... 9 more

                                Caused by: javax.crypto.BadPaddingException: Invalid Padding length: 52

                                                at sun.security.ssl.CipherBox.removePadding(Unknown Source)

                                                at sun.security.ssl.CipherBox.decrypt(Unknown Source)

                                                at sun.security.ssl.InputRecord.decrypt(Unknown Source)

                                                ... 16 more

                                [25 Feb 2015 17:42:01,068] [Nsh-Proxy-Thread-12] [INFO] [Anonymous:Anonymous:0:0:0:0:0:0:0:1] [BLSSOPROXY] failure establishing session with proxy service

                                [25 Feb 2015 17:42:01,099] [Nsh-Proxy-Thread-9] [INFO] [BLAdmin:BLAdmins:0:0:0:0:0:0:0:1] [BLSSOPROXY] copy data stop: Connection closed

                                 

                                I've been working with BMC support and hopefully it progresses to a resolution eventually.

                                • 13. Re: SSL_connect Errors During Compliance
                                  Bill Robinson

                                  what's the issue # ?

                                  • 14. Re: SSL_connect Errors During Compliance
                                    Cody Dean

                                    Support provided us an altered jar file that they indicated will be implemented in BSA 8.6 SP1 and it seemed to fix the problem. The issue number is ISS04431642.

                                     

                                    Thank You,

                                    Cody

                                    1 2 Previous Next