9 Replies Latest reply on May 12, 2011 9:13 PM by Simon Marko

    Unexpected exception in the agent communication layer

      Dear All,

       

           While taking the snapshot of the server I got the below error:

       

      com.bladelogic.app.service.agentservice.AgentConnectionException: com.bladelogic.app.remote.BlRemoteException: Unexpected exception in the agent communication layer

       

      Service running status of the target server:

       

      # ps -ef |grep rscd

          root   738     1   0   Apr 01 ?           0:00 bin/rscd

          root   743   738   0   Apr 01 ?           0:00 bin/rscd

          root   744   738   0   Apr 01 ?           0:00 bin/rscd

          root 21984 21594   0 18:47:01 pts/2       0:00 grep rscd

       

      #agentinfo <target_server>

      Agent Release   : 8.0.7.722
      Hostname        : <Hostname>

      Operating System: SunOS 5.10
      User Permissions: 0/0 (root/root)
      Security        : Protocol=5, Encryption=TLS1
      Host ID         : <Host ID>
      # of Processors : 2
      License Status  : Licensed for NSH/CM

       

       

      Please help me to resolve this issue.

       

      Regards,

      Raghava

        • 1. Unexpected exception in the agent communication layer
          Bill Robinson

          what parts are you taking a snapshot of ?

          • 2. Unexpected exception in the agent communication layer

            I'm getting the same error on one host.  But I'm just trying to run the update properties job against the server.

            • 3. Re: Unexpected exception in the agent communication layer
              Bill Robinson

              Can you post the full error from the appserver log?

              • 4. Re: Unexpected exception in the agent communication layer

                Here's the log from my job run.  Update properties is the first step in a multi-job batch run.

                 

                [18 Apr 2011 16:45:42,179] [Job-Execution-4] [INFO] [MLanghorst@OA.CAISO.COM:BLAdmins:] [Batch] Started running member job update properties

                [18 Apr 2011 16:45:42,280] [Job-Execution-3] [INFO] [MLanghorst@OA.CAISO.COM:BLAdmins:] [ServerProperty] Started running the job 'update properties' on application server 'fpapblog1'(2,000,000)

                [18 Apr 2011 16:45:46,988] [WorkItem-Thread-136] [INFO] [MLanghorst@OA.CAISO.COM:BLAdmins:] [ServerProperty] Executing work item Update Server Properties Job:update properties; Server:fpapsas1.wepex.net;  on application server: fpapblog1

                [18 Apr 2011 16:45:52,393] [WorkItem-Thread-136] [WARN] [MLanghorst@OA.CAISO.COM:BLAdmins:] [ServerProperty] Unexpected exception in the agent communication layer

                com.bladelogic.app.remote.BlRemoteException: Unexpected exception in the agent communication layer

                        at com.bladelogic.app.service.agentservice.AgentConnectionImpl.execute(AgentConnectionImpl.java:189)

                        at com.bladelogic.app.service.agentservice.AgentConnectionImpl.sendIntroInternal(AgentConnectionImpl.java:427)

                        at com.bladelogic.app.service.agentservice.AgentConnectionImpl.sendIntro(AgentConnectionImpl.java:119)

                        at com.bladelogic.app.service.agentservice.AgentConnectionImpl.execute(AgentConnectionImpl.java:154)

                        at com.bladelogic.app.service.agentservice.AgentMethodInvocationProvider.executeRequest(AgentMethodInvocationProvider.java:82)

                        at com.bladelogic.app.service.agentservice.AgentMethodInvocationProvider.invoke(AgentMethodInvocationProvider.java:49)

                        at com.bladelogic.app.service.routing.RoutingServiceImpl.invokeMethodLocally(RoutingServiceImpl.java:319)

                        at com.bladelogic.app.service.routing.RoutingServiceImpl.invoke(RoutingServiceImpl.java:246)

                        at com.bladelogic.app.service.agentservice.AgentRequestManager.invoke(AgentRequestManager.java:156)

                        at $Proxy35.getAsset(Unknown Source)

                        at com.bladelogic.daal.DAALService.getAsset(DAALService.java:164)

                        at com.bladelogic.model.server.ServerImpl.updatePropertyValuesFromAgent(ServerImpl.java:1246)

                        at com.bladelogic.model.job.updateserver.UpdateServerPropertyJobWorkItemImpl.execute(UpdateServerPropertyJobWorkItemImpl.java:125)

                        at com.bladelogic.app.service.workitem.WorkItem.doExecute(WorkItem.java:111)

                        at com.bladelogic.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:171)

                        at com.bladelogic.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:48)

                        at com.bladelogic.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:92)

                Caused by: java.util.NoSuchElementException

                        at java.util.StringTokenizer.nextToken(StringTokenizer.java:332)

                        at com.bladelogic.app.service.agentservice.XmlRpcBLHttpTransport.getInputStream(XmlRpcBLHttpTransport.java:312)

                        at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:141)

                        at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:94)

                        at com.bladelogic.app.service.agentservice.XmlRpcBLHttpTransport.sendRequest(XmlRpcBLHttpTransport.java:229)

                        at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:53)

                        at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:166)

                        at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:136)

                        at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:125)

                        at com.bladelogic.app.service.agentservice.AgentConnectionImpl.execute(AgentConnectionImpl.java:185)

                        ... 16 more

                [18 Apr 2011 16:45:52,397] [WorkItem-Thread-136] [WARN] [MLanghorst@OA.CAISO.COM:BLAdmins:] [ServerProperty] com.bladelogic.app.remote.BlRemoteException: Unexpected exception in the agent communication layer

                com.bladelogic.app.service.agentservice.AgentConnectionException: com.bladelogic.app.remote.BlRemoteException: Unexpected exception in the agent communication layer

                        at com.bladelogic.app.service.agentservice.AgentConnectionImpl.execute(AgentConnectionImpl.java:213)

                        at com.bladelogic.app.service.agentservice.AgentConnectionImpl.sendIntroInternal(AgentConnectionImpl.java:427)

                        at com.bladelogic.app.service.agentservice.AgentConnectionImpl.sendIntro(AgentConnectionImpl.java:119)

                        at com.bladelogic.app.service.agentservice.AgentConnectionImpl.execute(AgentConnectionImpl.java:154)

                        at com.bladelogic.app.service.agentservice.AgentMethodInvocationProvider.executeRequest(AgentMethodInvocationProvider.java:82)

                        at com.bladelogic.app.service.agentservice.AgentMethodInvocationProvider.invoke(AgentMethodInvocationProvider.java:49)

                        at com.bladelogic.app.service.routing.RoutingServiceImpl.invokeMethodLocally(RoutingServiceImpl.java:319)

                        at com.bladelogic.app.service.routing.RoutingServiceImpl.invoke(RoutingServiceImpl.java:246)

                        at com.bladelogic.app.service.agentservice.AgentRequestManager.invoke(AgentRequestManager.java:156)

                        at $Proxy35.getAsset(Unknown Source)

                        at com.bladelogic.daal.DAALService.getAsset(DAALService.java:164)

                        at com.bladelogic.model.server.ServerImpl.updatePropertyValuesFromAgent(ServerImpl.java:1246)

                        at com.bladelogic.model.job.updateserver.UpdateServerPropertyJobWorkItemImpl.execute(UpdateServerPropertyJobWorkItemImpl.java:125)

                        at com.bladelogic.app.service.workitem.WorkItem.doExecute(WorkItem.java:111)

                        at com.bladelogic.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:171)

                        at com.bladelogic.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:48)

                        at com.bladelogic.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:92)

                Caused by: com.bladelogic.app.remote.BlRemoteException: Unexpected exception in the agent communication layer

                        at com.bladelogic.app.service.agentservice.AgentConnectionImpl.execute(AgentConnectionImpl.java:189)

                        ... 16 more

                Caused by: java.util.NoSuchElementException

                        at java.util.StringTokenizer.nextToken(StringTokenizer.java:332)

                        at com.bladelogic.app.service.agentservice.XmlRpcBLHttpTransport.getInputStream(XmlRpcBLHttpTransport.java:312)

                        at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:141)

                        at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:94)

                        at com.bladelogic.app.service.agentservice.XmlRpcBLHttpTransport.sendRequest(XmlRpcBLHttpTransport.java:229)

                        at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:53)

                        at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:166)

                        at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:136)

                        at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:125)

                        at com.bladelogic.app.service.agentservice.AgentConnectionImpl.execute(AgentConnectionImpl.java:185)

                        ... 16 more

                [18 Apr 2011 16:45:52,399] [WorkItem-Thread-136] [ERROR] [MLanghorst@OA.CAISO.COM:BLAdmins:] [ServerProperty] com.bladelogic.app.service.agentservice.AgentConnectionException: com.bladelogic.app.remote.BlRemoteException: Unexpected exception in the agent communication layer

                [18 Apr 2011 16:45:52,719] [Job-Execution-2] [INFO] [MLanghorst@OA.CAISO.COM:BLAdmins:] [ServerProperty] The job 'update properties' has failed

                [18 Apr 2011 16:45:52,737] [Job-Execution-3] [ERROR] [MLanghorst@OA.CAISO.COM:BLAdmins:] [Batch] Member job update properties failed

                [18 Apr 2011 16:45:52,763] [Job-Execution-3] [INFO] [MLanghorst@OA.CAISO.COM:BLAdmins:] [Batch] The job 'Server Commisioning - Install Infosec Software 2' has failed

                • 5. Re: Unexpected exception in the agent communication layer

                  Hi Bill,

                   

                       Thanks for your response. Actually I'm taking snapshot of Hardware Information and System Info parts on solaris servers. Please find the below error log:

                   

                  Job Log:

                   

                  Info Apr 18, 2011 5:27:49 PM Snapshot job 'All Servers Snapshot 18-Apr-2011' Started taking snapshot of server objects on '<TARGET_HOST>'

                  Info Apr 18, 2011 5:27:49 PM Snapshot job 'All Servers Snapshot 18-Apr-2011' Started taking snapshot of 'System Info' on server '<TARGET_HOST>'

                  Info Apr 18, 2011 5:27:50 PM Snapshot job 'All Servers Snapshot 18-Apr-2011' succeeded for 'System Info' on server '<TARGET_HOST>'

                  Info Apr 18, 2011 5:27:50 PM Snapshot job 'All Servers Snapshot 18-Apr-2011' Started taking snapshot of '/' on server '<TARGET_HOST>'

                  Error Apr 18, 2011 5:27:51 PM com.bladelogic.app.service.agentservice.AgentConnectionException: com.bladelogic.app.remote.BlRemoteException: Unexpected exception in the agent communication layer

                  Info Apr 18, 2011 5:27:51 PM Snapshot job 'All Servers Snapshot 18-Apr-2011' succeeded for '/' on server '<TARGET_HOST>'

                   

                   

                  When I opened the live browse of the server and selected the Hardware Information , it throws the below error:

                   

                  Error on Console:

                   

                  com.bladelogic.mfw.util.BlException: Unexpected exception while handling request.com.bladelogic.app.api.BlAssetService_getBlAssetChildrenFromContainer(com.bladelogic.app.api.factory.AssetContainer, com.bladelogic.model.assetselector.IncludeExcludeAssetSelector)com.bladelogic.mfw.util.BlException: Unexpected exception while handling request.com.bladelogic.app.api.BlAssetService_getBlAssetChildrenFromContainer(com.bladelogic.app.api.factory.AssetContainer, com.bl

                  adelogic.model.assetselector.IncludeExcludeAssetSelector)

                   

                   

                  Regards,

                  Raghava

                  • 6. Re: Unexpected exception in the agent communication layer
                    Bill Robinson

                    can you login to the target system and run the sysinfo command, it's in the agent install dir /nativetool/bin.  see if you get an error from that.

                    • 7. Re: Unexpected exception in the agent communication layer

                      Hi Guys (first post here)

                      I'm having similar issues with agent comms on several HP-UX targets.

                      The nativetool/sysinfo runs OK

                      I can telnet through to port 4750 from the socks proxy

                      cd //hostname in NSH fails with I/O error: //hostname as does agentinfo, ie.

                      % agentinfo hostname

                      Can't access host "hostname": I/O error

                       

                      The hosts were added to BladeLogic about six weeks ago (21 March) and the daily Inventory Snapshot job started failing on March 29.

                       

                      This was in the rscd.log1

                       

                      05/11/11 14:24:30.006 WARN     rscd -  139.168.41.142 992 -1/-1 (???): ???: TLS setup failed for agent: Protocol mismatch. Check that client and server "secure" files match. Exiting and terminating connection.

                       

                      Target's secure file says:

                       

                      rscd:port=4750:protocol=5:tls_mode=encryption_only:encryption=tls:

                      default:port=4750:protocol=5:tls_mode=encryption_only:encryption=tls:

                       

                      App Server's secure file says:

                       

                      rscd:port=4750:protocol=5:tls_mode=encryption_only:encryption=tls:

                      default:port=4750:protocol=5:tls_mode=encryption_only:auth_profiles_file=/opt/bmc/BladeLogic/8.0/NSH/br/authenticationProfiles.xml:auth_profile=defaultProfile:appserver_protocol=ssoproxy:encryption=tls:

                       

                      rscd entries seems to match, but defaults are a bit different...

                       

                      Message was edited by: Simon Marko

                      • 8. Re: Unexpected exception in the agent communication layer
                        Bill Robinson

                        Is the appserver configured to run a NSH Proxy?  I believe that is the only way to communicate via nsh to your target, eg nsh client -> nsh proxy -> socks proxy -> agent.

                        • 9. Re: Unexpected exception in the agent communication layer

                          Hi All,

                          The problem was resolved by bouncing the app server

                           

                          There were active routing rules in the app server to use a socks proxy, however we could see that various operations (live browse and verify in particular) were not accessing the socks proxy at all.

                           

                          We tail'ed the sockd.log on the proxy and found no connections occurring.  Bouncing the app server seemed to restore all the targets which used the proxy - all the directly connected hosts were fine throughout.

                           

                          It doesn't explain why the rscd agent complained about TLS setup failing - perhaps some connections were getting through before we started concentrating on the sockd.log. There were certainly fewer TLS setup failures in the rscd.log than there were failed attempts to contact the agent.

                           

                          If it occurs repeatedly, we'll certainly be talking to support but for now it's fixed