13 Replies Latest reply on Jul 24, 2014 9:00 AM by Mark Casey

    Compliance Job Stuck in console

    Mark Casey

      Hi all,


      I'm currently trying to run a compliance job against a large group of servers. I've run this job in the past, and everything went fine.


      The past couple times, I would run it against a "All Active Windows" server group, which I assumed only would list servers that the agent was alive on. The smart group was configured incorrectly, so I would get Red X's on the servers which were offline, which was normal.


      I reconfigured the smart group to only target live servers, and while I got no more red X's, which shouldn't have stalled the job any way, the job is still stuck,


      The last entry in the log is just 1 minute after launch, and it appears it launched the job on all servers, and subsequently completed.


      Not sure why the job is stuck.


      Any ideas?




        • 1. Re: Compliance Job Stuck in console
          Mark Casey

          After I posted this, I went back into the console again.


          The progress show 1720/1722. I opened the log once more, and another server name just popped up as completing 30 minutes after the launch of the main job. I guess I will leave the job running to see if it finally finishes, but I thought it should have gone quicker then this.



          • 2. Re: Compliance Job Stuck in console
            Mark Casey

            Well, no movement on the Jobs in the last 2 hours. I sent another job to the same group of servers, and it too is stuck.

            • 3. Re: Compliance Job Stuck in console
              Bill Robinson

              Get a thread dump from the appserver – ‘Configuration | generate support data’ choose all your appservers, and the ‘appserver status report’.

              • 4. Re: Re: Compliance Job Stuck in console
                Mark Casey

                Here you go. Thanks for responding.

                • 5. Re: Re: Compliance Job Stuck in console
                  Bill Robinson

                  you've got a few wit all doing pretty much the same thing.  so what are those jobs looking at?  looks like they are hitting a 'server object' (processes, unix users, etc) and waiting for a return.  or it's possible a firewall dropped the connect and they are waiting for nothing.



                  Line 4712: WorkItem-Thread-20,EXECUTING,2h:8m:1s456ms,Compliance Job:IBM Director Compliance; Template:IBM Director; Component:IBM Director (PSDPHPRDAP02); Server:PSDPHPRDAP02;

                      Line 4715: WorkItem-Thread-23,EXECUTING,1h:24m:14s696ms,Compliance Job:Dell Open Manage Compliance; Template:Dell Open Manage; Component:Dell Open Manage (PSDPHPRDRPT01); Server:PSDPHPRDRPT01;

                      Line 4722: WorkItem-Thread-30,EXECUTING,2h:7m:49s646ms,Compliance Job:IBM Director Compliance; Template:IBM Director; Component:IBM Director (PSDPHPRDRPT01); Server:PSDPHPRDRPT01;

                      Line 4726: WorkItem-Thread-34,EXECUTING,1h:24m:39s360ms,Compliance Job:Dell Open Manage Compliance; Template:Dell Open Manage; Component:Dell Open Manage (PSDPHPRDCTX01); Server:PSDPHPRDCTX01;

                      Line 4736: WorkItem-Thread-44,EXECUTING,1h:24m:24s10ms,Compliance Job:Dell Open Manage Compliance; Template:Dell Open Manage; Component:Dell Open Manage (PSDPHPRDAP02); Server:PSDPHPRDAP02;








                  WorkItem-Thread-44 (State = RUNNABLE)


                  Stack Trace:

                      java.net.SocketInputStream.socketRead0(Native Method)

                      java.net.SocketInputStream.read(Unknown Source)

                      java.net.SocketInputStream.read(Unknown Source)

                      sun.security.ssl.InputRecord.readFully(Unknown Source)

                      sun.security.ssl.InputRecord.read(Unknown Source)

                      sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

                      sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)

                      sun.security.ssl.AppInputStream.read(Unknown Source)

                      java.io.BufferedInputStream.fill(Unknown Source)

                      java.io.BufferedInputStream.read(Unknown Source)
















                      com.sun.proxy.$Proxy39.getAsset(Unknown Source)















                  • 6. Re: Re: Compliance Job Stuck in console
                    Mark Casey



                    I logged back into the console to find that both of the jobs I had ran had both finished. I checked the logs, and they finished after about 8.5 hours. In both cases, the jobs are checking for the presence of IBM Director or Dell Open Manage on a Windows Server. They've been run in the past with minimal issues, so I was curious as to why all of a sudden they are taking a long time.


                    Is there any way to tell what is tying them up in the process?



                    • 7. Re: Compliance Job Stuck in console
                      Bill Robinson

                      How are they checking ?

                      • 8. Re: Compliance Job Stuck in console
                        Mark Casey

                        The Dell Open Manage job grabs the following parts:


                        Three different versions of the Application as it may appear in the Add/Remove Programs. (By different names)

                        A windows service

                        A registry key

                        Hardware information in /System


                        The compliance test is as follows:



                           (  "Hardware Information.Machine Summary:/System".Manufacturer = "Dell Inc."  OR

                              "Hardware Information.Machine Summary:/System".Manufacturer = "Unisys Corp."

                           )  AND

                           "Hardware Information.Machine Summary:/System".Model does not contain "ES7000"


                           "Windows Application:Dell OpenManage Server Administrator" exists  OR

                           "Windows Application:Dell OpenManage Systems Management Software" exists  OR

                           "Windows Application:Server Administrator" exists



                        The IBM Director job is even smaller, containing the following parts:


                        Hardware Information is /System

                        Two different application titles


                        The test for compliance looks like:



                           (  "Hardware Information.Machine Summary:/System".Manufacturer = "Unisys Corporation"  OR

                              "Hardware Information.Machine Summary:/System".Manufacturer = "IBM"

                           )  AND

                           "Hardware Information.Machine Summary:/System".Model does not contain "7000"


                           "Windows Application:IBM Director Agent Ext" exists  OR

                           "Windows Application:IBM Systems Director Platform Agent" exists


                        • 9. Re: Compliance Job Stuck in console
                          Bill Robinson

                          Hardware info is another CO and is probably the one that’s hanging.  If you live browse that on the target servers how long does it take to respond ?

                          1 of 1 people found this helpful
                          • 10. Re: Compliance Job Stuck in console
                            Mark Casey

                            Yeah, I doubled clicked on the server and it came up quickly, however, when I tried to expand "Hardware Information", it's now stuck at the hourglass.

                            • 11. Re: Compliance Job Stuck in console
                              Mark Casey

                              What would cause the console to respond to slowly to trying to expand the Hardware Information?

                              • 12. Re: Compliance Job Stuck in console
                                Ravi Shankar

                                There could be many reason


                                1) Target server is over WAN

                                2) RAM and CPU utilization on app-server

                                3) You are browsing server when there is maximum load on app-server

                                4) Target server response time (please check rscd log of target server)


                                Try browsing Hardware Information on other servers, check this is specific to few servers or this is generic?

                                • 13. Re: Compliance Job Stuck in console
                                  Mark Casey

                                  I believe I have found the issue. While browsing through the fixes in SP1 for BSA 8.5, I came across this:



                                  Winscan.exe hangs on server when opening hardware information via live browse

                                  8.2.04 8.5.01

                                  To test this, I went out to two servers, one of the ones which hung the console, and another which worked fine.


                                  I live browsed both, and the working one, had winscan.exe come up and then promptly go away. The one which got stuck, winscan.exe came up and never went away.


                                  To double check, I performed the same test with my compliance job and experienced the same hang with winscan.exe.