9 Replies Latest reply: May 18, 2012 3:05 PM by Vinnie Lima RSS

    Job Hangs when creating VM

    Nick Rawlins

      Hi,


      After requesting a service offering instance it executes the BBSA job but it just sits there and doesn't seem to be doing anything.

       

      Its been running for over 24 hours now and has no activity.

       

      Does anyone know what could be causing this?

       

      Thanks

        • 1. Job Hangs when creating VM
          Aryan Anantwar

          Hi

           

          Check VGP job Status in BBSA.

           

          is VGP job runnig or failed?

           

          if failed the the service offering failed.

           

          if running then @ whivch stage it is hung.

           

           

          regards

           

          Dnyaneshwar

          • 2. Job Hangs when creating VM
            Nick Rawlins

            Hi,

             

            Yes the job is just stuck in BBSA, it says Running... Its been doing it for about 48 hours now so its defanately stuck!

             

            CLM reports it as Running.

             

            Thanks

            • 3. Job Hangs when creating VM
              Aryan Anantwar

              HI,

               

              If Possible then check is the VM is created or not in Datastore configured in VGP package.

               

              also verify the values mentioned in VGP package.

               

              can u paste here the status details in BBSA of VGP job which is stuck.

               

               

              regards

               

              Dnyaneshwar

              • 4. Job Hangs when creating VM
                Nick Rawlins

                Hi,

                 

                No the VM hasn't created in the Datastore.

                 

                Here are the values from the job itself:

                 

                Running          May 1, 2012 4:22:23 PM          testNICK-12_VG_Job_1335886266859          Virtual Guest Job Run          BLAdmin          BLAdmins          NORMAL          job_deployment_clm-bbsa

                 

                 

                Any idea's would be great! Thanks

                • 5. Job Hangs when creating VM
                  Nick Rawlins

                  Hi,

                   

                  This seems to be the error!

                   

                  [01 May 2012 17:25:11,857] [http-10843-Processor3] [ERROR] [BLAdmin:BLAdmins:192.168.19.207] [] Resource not found

                  com.bladelogic.om.infra.mfw.util.NotFoundException: could not find asset /Virtual Machines/testNICK-12/BMC_VMware_VirtualMachineInfo on host 192.168.20.20 with asset type BMC_VMware_VirtualMachineInfo

                            at com.bladelogic.om.infra.model.asset.generic.GenericAssetFactory.loadValuesAndPopulateAsset(GenericAssetFactory.java:298)

                            at com.bladelogic.om.infra.model.asset.generic.GenericAssetFactory.getBlAssetFromContainer(GenericAssetFactory.java:282)

                            at com.bladelogic.om.infra.app.collector.LiveAssetContainer.getBlAsset(LiveAssetContainer.java:72)

                            at com.bladelogic.om.infra.app.collector.ServerAssetContainer.getBlAsset(ServerAssetContainer.java:109)

                            at com.bladelogic.om.infra.app.api.factory.BlAssetServiceImpl.getBlAssetFromContainer(BlAssetServiceImpl.java:424)

                            at com.bladelogic.om.infra.model.generic.rest.assets.RESTAssetRequestHandler.handleAssetRequest(RESTAssetRequestHandler.java:294)

                            at com.bladelogic.om.infra.model.generic.rest.RESTfulRequestServiceImpl.handleIdRequest(RESTfulRequestServiceImpl.java:751)

                            at com.bladelogic.om.infra.model.generic.rest.RESTfulRequestServiceImpl.processGet(RESTfulRequestServiceImpl.java:238)

                            at com.bladelogic.webservices.rest.RESTServlet.handleGet(RESTServlet.java:635)

                            at com.bladelogic.webservices.rest.RESTServlet.doGet(RESTServlet.java:292)

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

                            at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                            at java.lang.reflect.Method.invoke(Method.java:597)

                            at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)

                            at java.security.AccessController.doPrivileged(Native Method)

                            at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

                            at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)

                            at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)

                            at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)

                            at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)

                            at java.security.AccessController.doPrivileged(Native Method)

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)

                            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

                            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)

                            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

                            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

                            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)

                            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)

                            at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)

                            at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)

                            at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)

                            at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)

                            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

                            at java.lang.Thread.run(Thread.java:619)

                  [01 May 2012 17:25:11,868] [http-10843-Processor11] [ERROR] [::] [] Unable to find corresponding context for the serviceGroupId: urn:uuid:4C0FED0F283B75F8A11335888761510

                  org.apache.axis2.AxisFault: Unable to find corresponding context for the serviceGroupId: urn:uuid:4C0FED0F283B75F8A11335888761510

                            at org.apache.axis2.context.ConfigurationContext.getServiceGroupContextFromSoapSessionTable(ConfigurationContext.java:541)

                            at org.apache.axis2.context.ConfigurationContext.fillServiceContextAndServiceGroupContext(ConfigurationContext.java:249)

                            at com.bladelogic.webservices.framework.WSInFlowDispatchPhase.loadContexts(WSInFlowDispatchPhase.java:256)

                            at com.bladelogic.webservices.framework.WSInFlowDispatchPhase.checkPostConditions(WSInFlowDispatchPhase.java:146)

                            at org.apache.axis2.engine.Phase.invoke(Phase.java:334)

                            at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:251)

                            at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:160)

                            at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)

                            at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

                            at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                            at java.lang.reflect.Method.invoke(Method.java:597)

                            at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)

                            at java.security.AccessController.doPrivileged(Native Method)

                            at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

                            at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)

                            at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)

                            at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)

                            at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)

                            at java.security.AccessController.doPrivileged(Native Method)

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)

                            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

                            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)

                            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

                            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

                            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)

                            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)

                            at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)

                            at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)

                            at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)

                            at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)

                            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

                            at java.lang.Thread.run(Thread.java:619)

                  [01 May 2012 17:25:11,880] [http-10843-Processor7] [INFO] [::] [] LoginUsingUserCredentialResponse | Login Request with session id urn:uuid:4C0FED0F283B75F8A11335889520656

                  [01 May 2012 17:25:12,022] [http-10843-Processor7] [INFO] [BLAdmin:Anonymous:192.168.19.203] [] Assuming the default role: BLAdmins for the current soap session.

                  [01 May 2012 17:25:12,022] [http-10843-Processor7] [INFO] [BLAdmin:BLAdmins:192.168.19.203] [] User(BLAdmin) is authenticated with the authentication service.

                  [01 May 2012 17:25:12,070] [http-10843-Processor7] [INFO] [BLAdmin:BLAdmins:192.168.19.203] [] Creating connection pool for urn:uuid:4C0FED0F283B75F8A11335889520656

                  [01 May 2012 17:25:12,207] [Client-Connections-Thread-9] [INFO] [BLAdmin:BLAdmins:192.168.19.203] [Client] User 'BLAdmin' assumed the role 'BLAdmins'

                  [01 May 2012 17:25:12,211] [http-10843-Processor7] [ERROR] [BLAdmin:BLAdmins:192.168.19.203] [] testNICK-12 is not enrolled

                  com.bladelogic.om.infra.mfw.util.BlException: testNICK-12 is not enrolled

                            at com.bladelogic.om.virtualization.model.depot.vgp.VirtAssetServiceHelper.getVirtualGuestKeyByServerName(VirtAssetServiceHelper.java:171)

                            at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceImpl.java:625)

                            at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceRouterStubImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceRouterStubImpl.java:654)

                            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                            at java.lang.reflect.Method.invoke(Method.java:597)

                            at com.bladelogic.om.infra.cli.factory.APICommand.executeCommandOnSingleObject(APICommand.java:127)

                            at com.bladelogic.om.infra.cli.factory.APICommand.executeCommand(APICommand.java:106)

                            at com.bladelogic.om.infra.cli.factory.ComplexCommand.executeCommand(ComplexCommand.java:72)

                            at com.bladelogic.om.infra.cli.CLIEngine.executeCommand(CLIEngine.java:116)

                            at com.bladelogic.om.infra.model.clitunnel.CLITunnelServiceImpl.executeCommand(CLITunnelServiceImpl.java:134)

                            at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceSkeleton.executeCommandByParamList(CLITunnelServiceSkeleton.java:174)

                            at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceMessageReceiverInOut.invokeBusinessLogic(CLITunnelServiceMessageReceiverInOut.java:102)

                            at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)

                            at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:114)

                            at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:173)

                            at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)

                            at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

                            at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                            at java.lang.reflect.Method.invoke(Method.java:597)

                            at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)

                            at java.security.AccessController.doPrivileged(Native Method)

                            at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

                            at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)

                            at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)

                            at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)

                            at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)

                            at java.security.AccessController.doPrivileged(Native Method)

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)

                            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

                            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)

                            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

                            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

                            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)

                            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)

                            at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)

                            at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)

                            at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)

                            at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)

                            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

                            at java.lang.Thread.run(Thread.java:619)

                  [01 May 2012 17:25:12,212] [http-10843-Processor7] [ERROR] [BLAdmin:BLAdmins:192.168.19.203] [] [CLI Tunnel Service] testNICK-12 is not enrolled

                  com.bladelogic.om.infra.mfw.util.BlException: testNICK-12 is not enrolled

                            at com.bladelogic.om.virtualization.model.depot.vgp.VirtAssetServiceHelper.getVirtualGuestKeyByServerName(VirtAssetServiceHelper.java:171)

                            at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceImpl.java:625)

                            at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceRouterStubImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceRouterStubImpl.java:654)

                            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                            at java.lang.reflect.Method.invoke(Method.java:597)

                            at com.bladelogic.om.infra.cli.factory.APICommand.executeCommandOnSingleObject(APICommand.java:127)

                            at com.bladelogic.om.infra.cli.factory.APICommand.executeCommand(APICommand.java:106)

                            at com.bladelogic.om.infra.cli.factory.ComplexCommand.executeCommand(ComplexCommand.java:72)

                            at com.bladelogic.om.infra.cli.CLIEngine.executeCommand(CLIEngine.java:116)

                            at com.bladelogic.om.infra.model.clitunnel.CLITunnelServiceImpl.executeCommand(CLITunnelServiceImpl.java:134)

                            at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceSkeleton.executeCommandByParamList(CLITunnelServiceSkeleton.java:174)

                            at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceMessageReceiverInOut.invokeBusinessLogic(CLITunnelServiceMessageReceiverInOut.java:102)

                            at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)

                            at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:114)

                            at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:173)

                            at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)

                            at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

                            at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                            at java.lang.reflect.Method.invoke(Method.java:597)

                            at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)

                            at java.security.AccessController.doPrivileged(Native Method)

                            at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

                            at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)

                            at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)

                            at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)

                            at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)

                            at java.security.AccessController.doPrivileged(Native Method)

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)

                            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

                            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)

                            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

                            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

                            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)

                            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)

                            at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)

                            at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)

                            at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)

                            at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)

                            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

                            at java.lang.Thread.run(Thread.java:619)

                  • 6. Job Hangs when creating VM
                    Aryan Anantwar

                    Hi Friend,

                     

                    here is the ERROR from the log u have mentioned here:

                     

                     

                    [ERROR] [BLAdmin:BLAdmins:192.168.19.203] [] [CLI Tunnel Service] testNICK-12 is not enrolled

                    com.bladelogic.om.infra.mfw.util.BlException: testNICK-12 is not enrolled.

                     


                    testNICK-12 is the instance name which you are going to create through CLM.

                     

                    it occures when the VGP job creates a VM on Datastore & install OS with RSCD agent.

                    After all installations BBSA enrolls the RSCD agent. if it fails then the VGP job get failed & it rollbacks the things, during rollback it deletes the created VM.

                    This job stuck in running state, that means it have created VM & failed to enroll the Agent.

                    But While Rollback it get stuck.

                     

                    regards,

                    Dnyaneshwar.

                    • 7. Job Hangs when creating VM
                      Nick Rawlins

                      Thats great, thanks Dnyaneshwar. It makes sense and that would explain why the error occurred after about 45 minutes of it running... which is around the time it takes to deploy it manually.

                       

                      So how could I troubleshoot why the RSCD agent is failing to install?

                       

                      Thanks

                      • 8. Re: Job Hangs when creating VM
                        Ahmad Ramzi Jamjoom

                        As a hint, check the agent being installed on the OS, if it's comptabile with the OS or not!

                         

                        We had a similar error, but the difference is that in our enviornment the agents are pre-installed in the VM Templates and a 64-bit agent was installed on 32-bit VM template, so we couldn't connect to the provisioned machine and continue service request.

                        • 9. Job Hangs when creating VM
                          Vinnie Lima

                          There could be various causes to that symptom, which I've experienced as well.  Are you using DHCP or Static IP for your Container?  Is CPM configured to use IP address as VM names or have you switched to use name?

                           

                          We've had issues with the RSCD Vcenter agent that is part of CLM 2.1 SP1. Once and a while we've had to restart the agent on the Vcenter. Take a look at the BSA config server logfile for any errors there.

                           

                          Also make sure that a) the RSCD agent on your VM Template is licensed correctly, and that in BSA a machine with that same name (or IP if you are using IP for names) does NOT exist in the Server workspace (BSA).  If it does (because the last provisioning job was unsuccessful) all hell breaks loose.  Also make sure VCenter doesn't have a VM already with that same name from a previous provisioning.