10 Replies Latest reply on Feb 18, 2016 7:53 PM by Bill Robinson

    How to debug hanging blcli_execute command?

    michael huttner

      A very useful method I'd like to use is "Profiler getCompleteStatusReport" - which executes perfectly on two of our three environments... but on one of them it just hangs.. is there a way to debug/troubleshoot what's going on with it?

       

      I am using something like this:

       

      #!/bin/nsh

      #set -x

      blcli_setjvmoption -Xms1024M

      blcli_setjvmoption -Xmx1024M

      blcli_init

      blcli_setoption authType BLSSO

      blcli_setoption serviceProfilesFile [profile path to xml file]

      blcli_setoption serviceProfileName [profile name]

      blcli_setoption roleName BLAdmins

      blcli_connect || exit 2

      blcli_execute Profiler getCompleteStatusReport

      exit 0

        • 1. Re: How to debug hanging blcli_execute command?
          Bill Robinson

          How can you tell it’s hanging on the profiler command ?

          • 2. Re: How to debug hanging blcli_execute command?
            michael huttner

            it sits there in the script (or in the interactive nsh shell if I invoke it that way) and when the command "blcli_execute Profiler getCompleteStatusReport" is invoked it sits there on the terminal, hung.

            • 3. Re: How to debug hanging blcli_execute command?
              Santhosh Kurimilla

              One or the other AppServer/it's instances might be down while your command trying to get the profiler info. Please make sure of it.

              • 4. Re: How to debug hanging blcli_execute command?
                Bill Robinson

                Do you see the request running on the appserver it’s connecting to ?  there should be a client connection thread running the status report.  what state is that thread in ?

                • 5. Re: How to debug hanging blcli_execute command?
                  michael huttner

                  Apparently, we do see it on the appserver, and it's RUNNABLE.. eg:

                   

                  Client-Connections-Thread-43 (State = RUNNABLE)

                  ------------------------------------------------------

                  Stack Trace:

                      java.net.PlainSocketImpl.socketConnect(Native Method)

                      java.net.PlainSocketImpl.doConnect(Unknown Source)

                      java.net.PlainSocketImpl.connectToAddress(Unknown Source)

                      java.net.PlainSocketImpl.connect(Unknown Source)

                      java.net.SocksSocketImpl.connect(Unknown Source)

                      java.net.Socket.connect(Unknown Source)

                      java.net.Socket.connect(Unknown Source)

                      java.net.Socket.<init>(Unknown Source)

                      java.net.Socket.<init>(Unknown Source)

                      sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(Unknown Source)

                      sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(Unknown Source)

                      sun.rmi.transport.tcp.TCPEndpoint.newSocket(Unknown Source)

                      sun.rmi.transport.tcp.TCPChannel.createConnection(Unknown Source)

                      sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown Source)

                      sun.rmi.server.UnicastRef.newCall(Unknown Source)

                      sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)

                      com.bladelogic.om.infra.app.service.directory.DirectoryServiceImpl.getRemoteObject(DirectoryServiceImpl.java:213)

                      com.bladelogic.om.infra.app.service.directory.DirectoryServiceImpl.getRemoteService(DirectoryServiceImpl.java:656)

                      com.bladelogic.om.infra.app.service.ProfilerServiceImpl.getAppServerStatusInfo(ProfilerServiceImpl.java:348)

                      com.bladelogic.om.infra.app.service.ProfilerServiceImpl.findAllPxeServerStatus(ProfilerServiceImpl.java:251)

                      com.bladelogic.om.infra.app.service.ProfilerServiceImpl.getCompleteStatusReport(ProfilerServiceImpl.java:443)

                      com.bladelogic.om.infra.app.service.ProfilerServiceRequestHandlerImpl.getCompleteStatusReport_13(ProfilerServiceRequestHandlerImpl.java:504)

                      com.bladelogic.om.infra.app.service.ProfilerServiceRequestHandlerImpl.execute(ProfilerServiceRequestHandlerImpl.java:155)

                      com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.executeRequest(APIMethodInvocationProvider.java:54)

                      com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.invoke(APIMethodInvocationProvider.java:39)

                      com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invokeMethodLocally(RoutingServiceImpl.java:317)

                      com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invoke(RoutingServiceImpl.java:244)

                      com.bladelogic.om.infra.app.api.manage.APIRequestManager.executeRequest(APIRequestManager.java:30)

                      com.bladelogic.om.infra.message.app.frame.APIMessageFrame.service(APIMessageFrame.java:113)

                      com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processRequest(ClientWorkerThread.java:385)

                      com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processClientRequest(ClientWorkerThread.java:282)

                      com.bladelogic.om.infra.mfw.net.ClientWorkerThread.handleClientRequest(ClientWorkerThread.java:182)

                      com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:101)

                      com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:27)

                      com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                  • 6. Re: How to debug hanging blcli_execute command?
                    Bill Robinson

                    ok, so it's running and getting the profile data.  like someone else asked - are a lot of your appservers offline ?  have you tried to let it run to completion ? how long does that take ?  how many appservers in the hanging env ?  and this is running from one of the appservers itself, connecting to the appserver, not through a vip, etc ?

                    • 7. Re: How to debug hanging blcli_execute command?
                      michael huttner

                      hi bill,  we have 23 online app servers (one needs restart), and 23 offline (apparently earmarked for failover).   Apparently we don't know how to configure these to fix the performance problem without breaking failover.  I'm getting this info from UI server console.  thoughts?

                      • 10. Re: How to debug hanging blcli_execute command?
                        Bill Robinson

                        yeah - so probably it's hanging trying to pull the status report from the offline appservers - if you let it sit it would eventually finish.  i think the question is why do the offline appservers need to show up in the db ?