13 Replies Latest reply on Mar 4, 2015 9:44 AM by Bill Robinson

    Unexpected exception in the agent communication layer when verifiying the server

      Hi,

       

      For a solaris server for which a communication is through proxy i get an error "Unexpected exception in the agent communication layer" when i verify the server but agentinfo runs fine below is the entries in app log.

       

      Unexpected exception in the agent communication layer


      1. com.bladelogic.om.infra.app.remote.BlRemoteException: Unexpected exception in the agent communication layer

              at com.bladelogic.om.infra.app.service.agentservice.AgentConnectionImpl.execute(AgentConnectionImpl.java:226)

              at com.bladelogic.om.infra.app.service.agentservice.AgentConnectionImpl.sendIntroInternal(AgentConnectionImpl.java:546)

              at com.bladelogic.om.infra.app.service.agentservice.AgentConnectionImpl.sendIntro(AgentConnectionImpl.java:143)

              at com.bladelogic.om.infra.app.service.agentservice.AgentConnectionImpl.execute(AgentConnectionImpl.java:187)

              at com.bladelogic.om.infra.app.service.agentservice.AgentMethodInvocationProvider.executeRequest(AgentMethodInvocationProvider.java:80)

              at com.bladelogic.om.infra.app.service.agentservice.AgentMethodInvocationProvider.invoke(AgentMethodInvocationProvider.java:50)

              at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invokeMethodLocally(RoutingServiceImpl.java:317)

              at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invoke(RoutingServiceImpl.java:244)

              at com.bladelogic.om.infra.app.service.agentservice.AgentRequestManager.invoke(AgentRequestManager.java:156)

            at $Proxy32.getAsset(Unknown Source)  

              at com.bladelogic.om.infra.daal.DAALService.getAsset(DAALService.java:164)

              at com.bladelogic.om.infra.model.server.ServerImpl.updatePropertyValuesFromAgent(ServerImpl.java:1435)

              at com.bladelogic.om.infra.model.server.ServerImpl.updatePropertyValuesFromAgent(ServerImpl.java:1406)

              at com.bladelogic.om.infra.model.server.ServerImpl.updatePropertyValuesFromAgent(ServerImpl.java:1396)

              at com.bladelogic.om.infra.model.server.ServerServiceImpl.updatePropertyValuesFromAgent(ServerServiceImpl.java:568)

              at com.bladelogic.om.infra.model.server.ServerServiceRequestHandlerImpl.updatePropertyValuesFromAgent_20(ServerServiceRequestHandlerImpl.java:1241)

              at com.bladelogic.om.infra.model.server.ServerServiceRequestHandlerImpl.execute(ServerServiceRequestHandlerImpl.java:408)

              at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.executeRequest(APIMethodInvocationProvider.java:54)

              at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.invoke(APIMethodInvocationProvider.java:39)

              at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invokeMethodLocally(RoutingServiceImpl.java:317)

              at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invoke(RoutingServiceImpl.java:244)

              at com.bladelogic.om.infra.app.api.manage.APIRequestManager.executeRequest(APIRequestManager.java:30)

              at com.bladelogic.om.infra.message.app.frame.APIMessageFrame.service(APIMessageFrame.java:113)

              at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processRequest(ClientWorkerThread.java:385)

              at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processClientRequest(ClientWorkerThread.java:282)

              at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.handleClientRequest(ClientWorkerThread.java:182)

              at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:101)

              at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:27)

              at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

      Caused by: java.util.NoSuchElementException

              at java.util.StringTokenizer.nextToken(StringTokenizer.java:332)

              at com.bladelogic.om.infra.app.service.agentservice.XmlRpcBLHttpTransport.getInputStream(XmlRpcBLHttpTransport.java:367)

              at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:152)

              at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143)

              at com.bladelogic.om.infra.app.service.agentservice.XmlRpcBLHttpTransport.sendRequest(XmlRpcBLHttpTransport.java:284)

              at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56)

              at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167)

              at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137)

              at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126)

              at com.bladelogic.om.infra.app.service.agentservice.AgentConnectionImpl.execute(AgentConnectionImpl.java:222)

              ... 28 more

       

      is this because of $Proxy32.getAsset(Unknown Source) ?

        • 1. Re: Unexpected exception in the agent communication layer when verifiying the server
          Abhishek Rai

          Hi Manju,

           

          In most of the cases, restarting Appserver services resolved this issue but I won't recommend until we are sure that it requires.

           

          If log4j.logger.com.bladelogic.om.infra.app.jni.JNILogger is set to "debug", in log4j.properties then it can print more information about the communication between Appserver and Agent but again this requires a restart of appserver services.

           

          Anything in RSCD logs on proxy and target vm?

           

          Regards,

          Abhishek

          • 2. Re: Unexpected exception in the agent communication layer when verifiying the server
            Bill Robinson

            i'm assuming you mean a socks proxy?

             

            your agentinfo is run from where?  does it use a nsh proxy and the socks proxy ?

            • 3. Re: Unexpected exception in the agent communication layer when verifiying the server

              Yes Socks proxy,

               

              I ran the agent info from my BL app server.

              • 4. Re: Unexpected exception in the agent communication layer when verifiying the server
                Bill Robinson

                You started nsh from the appserver (not ‘nsh here’ from a gui on another box) – and is that nsh instance using the nsh proxy or communicating directly to the agent ?

                • 5. Re: Unexpected exception in the agent communication layer when verifiying the server

                  Bill,

                   

                  There is a Socks proxy in between app server and the target machine.

                  • 6. Re: Unexpected exception in the agent communication layer when verifiying the server
                    Bill Robinson

                    Right, but when you ran ‘agentinfo’ via nsh, if there was not a nsh proxy in use the communication would not have used the socks proxy.

                    • 7. Re: Unexpected exception in the agent communication layer when verifiying the server

                      Bill,

                       

                      Yeah we do have NSH proxy set on our app servers.

                      • 8. Re: Unexpected exception in the agent communication layer when verifiying the server
                        Bill Robinson

                        Ok, so is the nsh client on the appservers setup to use it ?  and the only way to talk to the remote server is via the socks proxy ?

                        • 9. Re: Unexpected exception in the agent communication layer when verifiying the server

                          This is the app server debug log,

                           

                              Connecting to host: chzn05dbs037d

                                Launching remote command: INTRO_CMD

                                Connecting to host: chzn05dbs037d returned 1 and took 457(ms)

                                Launching remote command: LOG_TRANSACTION: args =  Retrieving property values

                                Launching remote command: RSC_UNAME

                                Launching remote command: GET_AGENT_VERSION

                                Launching remote command: RSC_CALL_SHARED_OBJ

                                Launching remote command: RSC_CALL_SHARED_OBJ

                                Launching remote command: GET_PATH_TO_NSH

                                Launching remote command: GET_AGENT_VERSION

                                Launching remote command: GET_LICENSE_STATUS

                                Launching remote command: RSC_GET_FQDN

                                Launching remote command: LOG_TRANSACTION

                                Launching remote command: R_EXEC_CMD: args = uname -v

                                Launching remote command: RETURN_ASIS_OUTPUT

                                Launching remote command: LOG_TRANSACTION

                                Launching remote command: R_EXEC_CMD: args = zonename

                                Launching remote command: RETURN_ASIS_OUTPUT

                                Launching remote command: GET_AGENT_INSTALL_DIR

                               Unexpected exception in the agent communication layer

                          com.bladelogic.om.infra.app.remote.BlRemoteException: Unexpected exception in the agent communication layer

                                  at com.bladelogic.om.infra.app.service.agentservice.AgentConnectionImpl.execute(AgentConnectionImpl.java:226)

                                 at com.bladelogic.om.infra.app.service.agentservice.AgentConnectionImpl.sendIntroInternal(AgentConnectionImpl.java:546)

                                  at com.bladelogic.om.infra.app.service.agentservice.AgentConnectionImpl.sendIntro(AgentConnectionImpl.java:143)

                                  at com.bladelogic.om.infra.app.service.agentservice.AgentConnectionImpl.execute(AgentConnectionImpl.java:187)

                                  at com.bladelogic.om.infra.app.service.agentservice.AgentMethodInvocationProvider.executeRequest(AgentMethodInvocationProvider.java:80)

                                  at com.bladelogic.om.infra.app.service.agentservice.AgentMethodInvocationProvider.invoke(AgentMethodInvocationProvider.java:50)

                                  at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invokeMethodLocally(RoutingServiceImpl.java:317)

                                  at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invoke(RoutingServiceImpl.java:244)

                                  at com.bladelogic.om.infra.app.service.agentservice.AgentRequestManager.invoke(AgentRequestManager.java:156)

                                  at $Proxy32.getAsset(Unknown Source)

                                  at com.bladelogic.om.infra.daal.DAALService.getAsset(DAALService.java:164)

                                  at com.bladelogic.om.infra.model.server.ServerImpl.updatePropertyValuesFromAgent(ServerImpl.java:1435)

                                  at com.bladelogic.om.infra.model.server.ServerImpl.updatePropertyValuesFromAgent(ServerImpl.java:1406)

                                  at com.bladelogic.om.infra.model.server.ServerImpl.updatePropertyValuesFromAgent(ServerImpl.java:1396)

                                  at com.bladelogic.om.infra.model.server.ServerServiceImpl.updatePropertyValuesFromAgent(ServerServiceImpl.java:568)

                                  at com.bladelogic.om.infra.model.server.ServerServiceRequestHandlerImpl.updatePropertyValuesFromAgent_20(ServerServiceRequestHandlerImpl.java:1241)

                                  at com.bladelogic.om.infra.model.server.ServerServiceRequestHandlerImpl.execute(ServerServiceRequestHandlerImpl.java:408)

                                  at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.executeRequest(APIMethodInvocationProvider.java:54)

                                  at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.invoke(APIMethodInvocationProvider.java:39)

                                  at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invokeMethodLocally(RoutingServiceImpl.java:317)

                                  at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invoke(RoutingServiceImpl.java:244)

                                  at com.bladelogic.om.infra.app.api.manage.APIRequestManager.executeRequest(APIRequestManager.java:30)

                                  at com.bladelogic.om.infra.message.app.frame.APIMessageFrame.service(APIMessageFrame.java:113)

                                  at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processRequest(ClientWorkerThread.java:385)

                                  at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processClientRequest(ClientWorkerThread.java:282)

                                  at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.handleClientRequest(ClientWorkerThread.java:182)

                                  at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:101)

                                  at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:27)

                                  at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                          Caused by: java.util.NoSuchElementException

                                  at java.util.StringTokenizer.nextToken(StringTokenizer.java:332)

                                  at com.bladelogic.om.infra.app.service.agentservice.XmlRpcBLHttpTransport.getInputStream(XmlRpcBLHttpTransport.java:367)

                                  at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:152)

                                  at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143)

                                  at com.bladelogic.om.infra.app.service.agentservice.XmlRpcBLHttpTransport.sendRequest(XmlRpcBLHttpTransport.java:284)

                                  at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56)

                                  at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167)

                                  at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137)

                                  at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126)

                                  at com.bladelogic.om.infra.app.service.agentservice.AgentConnectionImpl.execute(AgentConnectionImpl.java:222)

                                  ... 28 more

                          • 10. Re: Unexpected exception in the agent communication layer when verifiying the server
                            Bill Robinson

                            Ok – that doesn’t really answer my question about the nsh client and why that might work…

                             

                            Do you see the appserver connecting through the socks proxy when you do the ‘verify’ ?  do you see it connecting to the target ?

                            • 11. Re: Unexpected exception in the agent communication layer when verifiying the server

                              I am not sure on that i ned to check it I need to put capture in proxy to check that.

                               

                              I have seen the target machine rscd log where if try to connect its NSH  it logs in the rscd.log of target machine

                              • 12. Re: Unexpected exception in the agent communication layer when verifiying the server

                                Bill,

                                 

                                Yes we can see the traffic through the socks proxy when we verify the server.

                                • 13. Re: Unexpected exception in the agent communication layer when verifiying the server
                                  Bill Robinson

                                  can you attach the entire appserver log from when this happens and indicate around what time ?

                                   

                                  also - do you have the case where the 'resolve dns' option is set on the socks proxy and the appserver can locally resolve the target but to a different ip than the socks proxy resolves to ?