13 Replies Latest reply on Jul 26, 2019 9:45 AM by S Crawford

    Peer Not Authenticated in the ro-adapter-ws adapter

    S Crawford
      Share This:

      Hi - We run BLCLI calls through BAO using the ro-adapter-ws adapter that makes SOAP requests to the Truesight (BladeLogic) webservice on port 9843.  We intermittently are seeing 'Peer Not Authenticated' errors in the adapter logs.  The request usually times out with a 504 gateway_timeout error and we are seeing these Peer Not Authenticated errors.  Not sure if they are related but a few years ago we were seeing the peer errors.  The previous fix was to upgrade to the next version of BladeLogic, but now we are several version past that old version. We are now running version 8.9.03.  Is it possible this issue has been reinserted in the newer version of BL?

       

      See previous communities post where we ran into this...
      Peer Not Authenticated ro-adapter-ws

       

      Bill Robinson, have you seen this from other users lately in the newer version of Truesight?

        • 1. Re: Peer Not Authenticated in the ro-adapter-ws adapter
          Aryan Anantwar

          Hi,

           

          As you mentioned the error you seen is like - 504 gateway_timeout error.

          Which indicates, that the AO WS adapter is trying to get authenticated BSA Application in respective WS call and failing to get it done due to gateway timeout.

          It would be great if you can share more details -

          - AO Platform Version

          - AO Content Version

          - BSA Version

          - Brief of the AO - BSA network comunication path.

           

          Regards,

          Aryan Anantwar

          2 of 2 people found this helpful
          • 2. Re: Peer Not Authenticated in the ro-adapter-ws adapter
            S Crawford

            Hi Aryan,

             

            Yes, that seems to be the issue we are experiencing.  We tried adding the BSA SSL cert into the AO CDP and AP keystores (/opt/bmc/atrium/ap/tomcat/conf/.keystore and /opt/bmc/atrium/cdp/tomcat/conf/.keystore).  This seemed to work in our test environment but not in our prod environment. 

             

            Here are the other details you requested:

            AO Platform Version - 7.9.01.003
            AO Content Version - ro-adapter-ws - 20.16.03.00

            BSA Version - 8.9.03.162
            The AO environment is on AWS and sits behind an ELB.  A module in AO accepts the BLCLI info and that module processes and sends SOAP requests to the BSA web service using the ro-adapter-ws.  The BSA servers sit behind a local F5 load balancer. 

             

            Thanks,

            Scott

            • 3. Re: Peer Not Authenticated in the ro-adapter-ws adapter
              Aryan Anantwar

              Hi,

               

              You don't need to add BSA SSL Certs into AO CDP and AP keystores, instead you need to add them to truststore.

              i.e. (/opt/bmc/atrium/ap/jvm/lib/security/cacerts and /opt/bmc/atrium/cdp/jvm/lib/security/cacerts).

               

              Also you can use below in your WebService Adapter request:

              <use-ssl-certificate>true</use-ssl-certificate>
              <install-certificate>true</install-certificate>

               

              If the issue is intermittent, then it is more related to 504 gateway_timeout error, then I would suggest to check in BSA logs when you see this error again.

               

              Regards,

              Aryan Anantwar

              • 4. Re: Peer Not Authenticated in the ro-adapter-ws adapter
                Deepak Bhola

                I would recommend to use latest content version of adapter i.e. 20.19.02.

                 

                If you want to manually install the certificates then install the certificates in below truststore file, please note this change requires AO peer service restart.

                <BAO_HOME>\jvm\lib\security\jssecacerts.

                 

                please enable below parameters in adapter request (as suggested by Aryan) and adapter will automatically install the certificates, in this case you don't need to install certificates manually as described above.

                1 of 1 people found this helpful
                • 5. Re: Peer Not Authenticated in the ro-adapter-ws adapter
                  S Crawford

                  Hi,

                   

                  Thanks for the replies.  I have a few other follow-up questions if you don't mind:

                   

                  For the use-ssl-certificate and install-certificate parameters, can these be added to the adapter configuration (via Grid Manager) to ensure they are always set, or do they have to be set in the AO workflow that makes the SOAP request?

                   

                  As far as adding the BSA cert to the keystore, we were instructed to add it to the /opt/bmc/atrium/ap/tomcat/conf/.keystore.  Would you expect the issue to still come up if it wasn't added to the jssecacerts and cacerts keystore?  It has never needed to be added before.  We started running into this issue a couple months ago, after a BSA upgrade from 8.6 to 8.9.

                   

                  Do you know if there anything new in the 20.19.02 adapter that would specifically address this issue?

                   

                  Thanks

                  • 6. Re: Peer Not Authenticated in the ro-adapter-ws adapter
                    Aryan Anantwar

                    Hi,

                     

                    For the use-ssl-certificate and install-certificate parameters, can these be added to the adapter configuration (via Grid Manager) to ensure they are always set, or do they have to be set in the AO workflow that makes the SOAP request?

                    - any such parameter value passed in workflow will override the adapter configuration, so it's up to you that how you want to pass these parameter values. completely based on requirement.

                     

                    As you mentioned, the issue arises only after BSA upgrades means there is some new change sin BSA which might be impacting the BSA WebServices authentication mechanism.

                    Whenever you see the same error again, jump to BSA logs and investigate there.

                     

                    Regards,

                    Aryan Anantwar

                    • 7. Re: Peer Not Authenticated in the ro-adapter-ws adapter
                      S Crawford

                      It looks like the workflow's default value for use-ssl and installing the cert are set to false if they are initially blank, so we will set the adapter configuration values to true to ensure those values are never blank.

                       

                      This morning, we are seeing these errors come through spoardically.  We believe these errors are contributing to the gateway_timeout errors.  Have you see these before:

                       

                      18 Jul 2019 10:00:27,872 [Thread=AMP - Perform Action Executor - 2183] ERROR SoapRpcActorAdapter [PeerName=AP 1] [JobID=f3c79208af7e7429:3e845c1c:16bf83ca329:-7ff81-1563458126201] [AdapterName=BLADELOGIC-SOAP-ACTOR]  BLADELOGIC-SOAP-ACTOR: Error occurred processing request data  Summary: Exception occurred in Adapter BLADELOGIC-SOAP-ACTOR. Certificate downloaded from <bladelogic-url>.com:9843 is invalid.

                      • 8. Re: Peer Not Authenticated in the ro-adapter-ws adapter
                        S Crawford

                        Aryan Anantwar and Matthew Highcove - I noticed this issue was previously encountered by someone else on here and you both responded to the issue.  See Webservice adapter SSL Connectiion failure .

                         

                        I am running on 7.9 for BAO and 7.9 for BSA. If I look in my BSA app server option settings, these are the ciphers and TLS protocols that are configured:

                         

                        EnabledPkiProtocols=TLSv1

                        EnabledCipherSuites=TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA

                        EnabledAppserverClientProtocols=TLSv1,TLSv1.2

                        EnabledSecureProtocols=TLSv1,TLSv1.2

                        EnabledTlsContextProtocol=TLSv1

                        EnabledRscdProtocols=TLSv1,TLSv1.2

                        EnabledCipherSuitesForWebservices=TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA

                         

                        If I look in BAO at my AP tomcat server.xml connector settings, I see the right cipher but I only see the protocol set to 'TLS'.  Would 'TLS' include all support for the ones that are listed in the BSA server settings above, or does it need to be changed?

                         

                        Connector SSLEnabled="true" URIEncoding="UTF-8" ciphers="SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_anon_WITH_AES_128_CBC_SHA,TLS_ECDH_anon_WITH_AES_256_CBC_SHA" clientAuth="false" keystoreFile="/opt/bmc/atrium/ap/tomcat/conf/.keystore" maxSwallowSize="-1" maxThreads="150" port="38080" protocol="org.apache.coyote.http11.Http11NioProtocol" scheme="https" secure="true" sslProtocol="TLS" useServerCipherSuitesOrder="true"

                        • 9. Re: Peer Not Authenticated in the ro-adapter-ws adapter
                          Matthew Highcove

                          The value "TLS" covers different TLS versions on different versions of Java. If you want to tell BAO explicitly to use all versions of TLS, you can change this to SSLProtocol="TLSv1+TLSv1.1+TLSv1.2".

                           

                          However, the settings in server.xml pertain to BAO's webserver port, 38080. Web service calls from BAO do not use this port. The settings in [BAO_HOME]/jvm/lib/security/java.security are more relevant for outgoing adapter calls. What values do you have for jdk.tls.disabledAlgorithms in this file?

                          • 10. Re: Peer Not Authenticated in the ro-adapter-ws adapter
                            S Crawford

                            Here are the ones that are set on the AP1 peer:

                             

                            jdk.tls.disabledAlgorithms=SSLv3, RC4, DHE, DH keySize < 768

                            • 11. Re: Peer Not Authenticated in the ro-adapter-ws adapter
                              Deepak Bhola

                              Does the gateway also requires latest certificate of upgraded BSA server endpoint? I would suggest to use 20.19.01 version of adapter as 20.16.03 is pretty old version.

                              java SSL debug logging would provide more information is there is any certificate related issues, you need to add -Djavax.net.debug=ssl entry in java options in BAO/TSO peer where adapter is enabled.

                              Are you getting this error intermittently of constantly? Can you share the adapter request which were failed?. 

                              1 of 1 people found this helpful
                              • 12. Re: Peer Not Authenticated in the ro-adapter-ws adapter
                                S Crawford

                                Deepak Bhola, yes I have SSL and handshake set to debug level in the Java logging in BAO.  I have an existing ticket opened for this issue (00694161) and I uploaded the logs from a failed request there. 

                                • 13. Re: Peer Not Authenticated in the ro-adapter-ws adapter
                                  S Crawford

                                  FYI - We were able to work around this issue by setting use-ssl-certificate and install-certificate to false in our SOAP request that our AO workflow generates.  We are releasing a new API that will go directly to the TSA webservice so it will only be needed in this state for a week or two at most and then we will move off it.