9 Replies Latest reply: May 18, 2012 3:05 PM by Vinnie Lima RSS

Job Hangs when creating VM

realtek

Hi,


After requesting a service offering instance it executes the BBSA job but it just sits there and doesn't seem to be doing anything.

 

Its been running for over 24 hours now and has no activity.

 

Does anyone know what could be causing this?

 

Thanks

  • 1. Job Hangs when creating VM
    Aryan Anantwar

    Hi

     

    Check VGP job Status in BBSA.

     

    is VGP job runnig or failed?

     

    if failed the the service offering failed.

     

    if running then @ whivch stage it is hung.

     

     

    regards

     

    Dnyaneshwar

  • 2. Job Hangs when creating VM
    realtek

    Hi,

     

    Yes the job is just stuck in BBSA, it says Running... Its been doing it for about 48 hours now so its defanately stuck!

     

    CLM reports it as Running.

     

    Thanks

  • 3. Job Hangs when creating VM
    Aryan Anantwar

    HI,

     

    If Possible then check is the VM is created or not in Datastore configured in VGP package.

     

    also verify the values mentioned in VGP package.

     

    can u paste here the status details in BBSA of VGP job which is stuck.

     

     

    regards

     

    Dnyaneshwar

  • 4. Job Hangs when creating VM
    realtek

    Hi,

     

    No the VM hasn't created in the Datastore.

     

    Here are the values from the job itself:

     

    Running          May 1, 2012 4:22:23 PM          testNICK-12_VG_Job_1335886266859          Virtual Guest Job Run          BLAdmin          BLAdmins          NORMAL          job_deployment_clm-bbsa

     

     

    Any idea's would be great! Thanks

  • 5. Job Hangs when creating VM
    realtek

    Hi,

     

    This seems to be the error!

     

    [01 May 2012 17:25:11,857] [http-10843-Processor3] [ERROR] [BLAdmin:BLAdmins:192.168.19.207] [] Resource not found

    com.bladelogic.om.infra.mfw.util.NotFoundException: could not find asset /Virtual Machines/testNICK-12/BMC_VMware_VirtualMachineInfo on host 192.168.20.20 with asset type BMC_VMware_VirtualMachineInfo

              at com.bladelogic.om.infra.model.asset.generic.GenericAssetFactory.loadValuesAndPopulateAsset(GenericAssetFactory.java:298)

              at com.bladelogic.om.infra.model.asset.generic.GenericAssetFactory.getBlAssetFromContainer(GenericAssetFactory.java:282)

              at com.bladelogic.om.infra.app.collector.LiveAssetContainer.getBlAsset(LiveAssetContainer.java:72)

              at com.bladelogic.om.infra.app.collector.ServerAssetContainer.getBlAsset(ServerAssetContainer.java:109)

              at com.bladelogic.om.infra.app.api.factory.BlAssetServiceImpl.getBlAssetFromContainer(BlAssetServiceImpl.java:424)

              at com.bladelogic.om.infra.model.generic.rest.assets.RESTAssetRequestHandler.handleAssetRequest(RESTAssetRequestHandler.java:294)

              at com.bladelogic.om.infra.model.generic.rest.RESTfulRequestServiceImpl.handleIdRequest(RESTfulRequestServiceImpl.java:751)

              at com.bladelogic.om.infra.model.generic.rest.RESTfulRequestServiceImpl.processGet(RESTfulRequestServiceImpl.java:238)

              at com.bladelogic.webservices.rest.RESTServlet.handleGet(RESTServlet.java:635)

              at com.bladelogic.webservices.rest.RESTServlet.doGet(RESTServlet.java:292)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

              at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)

              at java.security.AccessController.doPrivileged(Native Method)

              at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

              at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)

              at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)

              at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)

              at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)

              at java.security.AccessController.doPrivileged(Native Method)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)

              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)

              at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)

              at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)

              at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)

              at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

              at java.lang.Thread.run(Thread.java:619)

    [01 May 2012 17:25:11,868] [http-10843-Processor11] [ERROR] [::] [] Unable to find corresponding context for the serviceGroupId: urn:uuid:4C0FED0F283B75F8A11335888761510

    org.apache.axis2.AxisFault: Unable to find corresponding context for the serviceGroupId: urn:uuid:4C0FED0F283B75F8A11335888761510

              at org.apache.axis2.context.ConfigurationContext.getServiceGroupContextFromSoapSessionTable(ConfigurationContext.java:541)

              at org.apache.axis2.context.ConfigurationContext.fillServiceContextAndServiceGroupContext(ConfigurationContext.java:249)

              at com.bladelogic.webservices.framework.WSInFlowDispatchPhase.loadContexts(WSInFlowDispatchPhase.java:256)

              at com.bladelogic.webservices.framework.WSInFlowDispatchPhase.checkPostConditions(WSInFlowDispatchPhase.java:146)

              at org.apache.axis2.engine.Phase.invoke(Phase.java:334)

              at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:251)

              at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:160)

              at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)

              at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

              at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)

              at java.security.AccessController.doPrivileged(Native Method)

              at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

              at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)

              at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)

              at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)

              at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)

              at java.security.AccessController.doPrivileged(Native Method)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)

              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)

              at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)

              at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)

              at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)

              at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

              at java.lang.Thread.run(Thread.java:619)

    [01 May 2012 17:25:11,880] [http-10843-Processor7] [INFO] [::] [] LoginUsingUserCredentialResponse | Login Request with session id urn:uuid:4C0FED0F283B75F8A11335889520656

    [01 May 2012 17:25:12,022] [http-10843-Processor7] [INFO] [BLAdmin:Anonymous:192.168.19.203] [] Assuming the default role: BLAdmins for the current soap session.

    [01 May 2012 17:25:12,022] [http-10843-Processor7] [INFO] [BLAdmin:BLAdmins:192.168.19.203] [] User(BLAdmin) is authenticated with the authentication service.

    [01 May 2012 17:25:12,070] [http-10843-Processor7] [INFO] [BLAdmin:BLAdmins:192.168.19.203] [] Creating connection pool for urn:uuid:4C0FED0F283B75F8A11335889520656

    [01 May 2012 17:25:12,207] [Client-Connections-Thread-9] [INFO] [BLAdmin:BLAdmins:192.168.19.203] [Client] User 'BLAdmin' assumed the role 'BLAdmins'

    [01 May 2012 17:25:12,211] [http-10843-Processor7] [ERROR] [BLAdmin:BLAdmins:192.168.19.203] [] testNICK-12 is not enrolled

    com.bladelogic.om.infra.mfw.util.BlException: testNICK-12 is not enrolled

              at com.bladelogic.om.virtualization.model.depot.vgp.VirtAssetServiceHelper.getVirtualGuestKeyByServerName(VirtAssetServiceHelper.java:171)

              at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceImpl.java:625)

              at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceRouterStubImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceRouterStubImpl.java:654)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at com.bladelogic.om.infra.cli.factory.APICommand.executeCommandOnSingleObject(APICommand.java:127)

              at com.bladelogic.om.infra.cli.factory.APICommand.executeCommand(APICommand.java:106)

              at com.bladelogic.om.infra.cli.factory.ComplexCommand.executeCommand(ComplexCommand.java:72)

              at com.bladelogic.om.infra.cli.CLIEngine.executeCommand(CLIEngine.java:116)

              at com.bladelogic.om.infra.model.clitunnel.CLITunnelServiceImpl.executeCommand(CLITunnelServiceImpl.java:134)

              at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceSkeleton.executeCommandByParamList(CLITunnelServiceSkeleton.java:174)

              at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceMessageReceiverInOut.invokeBusinessLogic(CLITunnelServiceMessageReceiverInOut.java:102)

              at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)

              at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:114)

              at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:173)

              at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)

              at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

              at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)

              at java.security.AccessController.doPrivileged(Native Method)

              at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

              at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)

              at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)

              at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)

              at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)

              at java.security.AccessController.doPrivileged(Native Method)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)

              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)

              at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)

              at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)

              at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)

              at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

              at java.lang.Thread.run(Thread.java:619)

    [01 May 2012 17:25:12,212] [http-10843-Processor7] [ERROR] [BLAdmin:BLAdmins:192.168.19.203] [] [CLI Tunnel Service] testNICK-12 is not enrolled

    com.bladelogic.om.infra.mfw.util.BlException: testNICK-12 is not enrolled

              at com.bladelogic.om.virtualization.model.depot.vgp.VirtAssetServiceHelper.getVirtualGuestKeyByServerName(VirtAssetServiceHelper.java:171)

              at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceImpl.java:625)

              at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceRouterStubImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceRouterStubImpl.java:654)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at com.bladelogic.om.infra.cli.factory.APICommand.executeCommandOnSingleObject(APICommand.java:127)

              at com.bladelogic.om.infra.cli.factory.APICommand.executeCommand(APICommand.java:106)

              at com.bladelogic.om.infra.cli.factory.ComplexCommand.executeCommand(ComplexCommand.java:72)

              at com.bladelogic.om.infra.cli.CLIEngine.executeCommand(CLIEngine.java:116)

              at com.bladelogic.om.infra.model.clitunnel.CLITunnelServiceImpl.executeCommand(CLITunnelServiceImpl.java:134)

              at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceSkeleton.executeCommandByParamList(CLITunnelServiceSkeleton.java:174)

              at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceMessageReceiverInOut.invokeBusinessLogic(CLITunnelServiceMessageReceiverInOut.java:102)

              at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)

              at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:114)

              at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:173)

              at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)

              at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

              at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)

              at java.security.AccessController.doPrivileged(Native Method)

              at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

              at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)

              at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)

              at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)

              at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)

              at java.security.AccessController.doPrivileged(Native Method)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)

              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)

              at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)

              at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)

              at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)

              at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

              at java.lang.Thread.run(Thread.java:619)

  • 6. Job Hangs when creating VM
    Aryan Anantwar

    Hi Friend,

     

    here is the ERROR from the log u have mentioned here:

     

     

    [ERROR] [BLAdmin:BLAdmins:192.168.19.203] [] [CLI Tunnel Service] testNICK-12 is not enrolled

    com.bladelogic.om.infra.mfw.util.BlException: testNICK-12 is not enrolled.

     


    testNICK-12 is the instance name which you are going to create through CLM.

     

    it occures when the VGP job creates a VM on Datastore & install OS with RSCD agent.

    After all installations BBSA enrolls the RSCD agent. if it fails then the VGP job get failed & it rollbacks the things, during rollback it deletes the created VM.

    This job stuck in running state, that means it have created VM & failed to enroll the Agent.

    But While Rollback it get stuck.

     

    regards,

    Dnyaneshwar.

  • 7. Job Hangs when creating VM
    realtek

    Thats great, thanks Dnyaneshwar. It makes sense and that would explain why the error occurred after about 45 minutes of it running... which is around the time it takes to deploy it manually.

     

    So how could I troubleshoot why the RSCD agent is failing to install?

     

    Thanks

  • 8. Re: Job Hangs when creating VM
    Ahmad Ramzi Jamjoom

    As a hint, check the agent being installed on the OS, if it's comptabile with the OS or not!

     

    We had a similar error, but the difference is that in our enviornment the agents are pre-installed in the VM Templates and a 64-bit agent was installed on 32-bit VM template, so we couldn't connect to the provisioned machine and continue service request.

  • 9. Job Hangs when creating VM
    Vinnie Lima

    There could be various causes to that symptom, which I've experienced as well.  Are you using DHCP or Static IP for your Container?  Is CPM configured to use IP address as VM names or have you switched to use name?

     

    We've had issues with the RSCD Vcenter agent that is part of CLM 2.1 SP1. Once and a while we've had to restart the agent on the Vcenter. Take a look at the BSA config server logfile for any errors there.

     

    Also make sure that a) the RSCD agent on your VM Template is licensed correctly, and that in BSA a machine with that same name (or IP if you are using IP for names) does NOT exist in the Server workspace (BSA).  If it does (because the last provisioning job was unsuccessful) all hell breaks loose.  Also make sure VCenter doesn't have a VM already with that same name from a previous provisioning.