9 Replies Latest reply on May 18, 2012 3:05 PM by Vinnie Lima

Job Hangs when creating VM

Nick Rawlins May 2, 2012 4:40 AM

Hi,

After requesting a service offering instance it executes the BBSA job but it just sits there and doesn't seem to be doing anything.

Its been running for over 24 hours now and has no activity.

Does anyone know what could be causing this?

Thanks

1. Job Hangs when creating VM

Aryan Anantwar May 3, 2012 2:24 AM (in response to Nick Rawlins)

Hi

Check VGP job Status in BBSA.

is VGP job runnig or failed?

if failed the the service offering failed.

if running then @ whivch stage it is hung.

regards

Dnyaneshwar
Like Show 0 Likes(0)

Actions
2. Job Hangs when creating VM

Nick Rawlins May 3, 2012 4:03 AM (in response to Aryan Anantwar)

Hi,

Yes the job is just stuck in BBSA, it says Running... Its been doing it for about 48 hours now so its defanately stuck!

CLM reports it as Running.

Thanks
Like Show 0 Likes(0)

Actions
3. Job Hangs when creating VM

Aryan Anantwar May 3, 2012 4:09 AM (in response to Nick Rawlins)

HI,

If Possible then check is the VM is created or not in Datastore configured in VGP package.

also verify the values mentioned in VGP package.

can u paste here the status details in BBSA of VGP job which is stuck.

regards

Dnyaneshwar
Like Show 0 Likes(0)

Actions
4. Job Hangs when creating VM

Nick Rawlins May 3, 2012 4:38 AM (in response to Aryan Anantwar)

Hi,

No the VM hasn't created in the Datastore.

Here are the values from the job itself:

Running May 1, 2012 4:22:23 PM testNICK-12_VG_Job_1335886266859 Virtual Guest Job Run BLAdmin BLAdmins NORMAL job_deployment_clm-bbsa

Any idea's would be great! Thanks
Like Show 0 Likes(0)

Actions
5. Job Hangs when creating VM

Nick Rawlins May 3, 2012 4:54 AM (in response to Nick Rawlins)

Hi,

This seems to be the error!

[01 May 2012 17:25:11,857] [http-10843-Processor3] [ERROR] [BLAdmin:BLAdmins:192.168.19.207] [] Resource not found
com.bladelogic.om.infra.mfw.util.NotFoundException: could not find asset /Virtual Machines/testNICK-12/BMC_VMware_VirtualMachineInfo on host 192.168.20.20 with asset type BMC_VMware_VirtualMachineInfo
          at com.bladelogic.om.infra.model.asset.generic.GenericAssetFactory.loadValuesAndPopulateAsset(GenericAssetFactory.java:298)
          at com.bladelogic.om.infra.model.asset.generic.GenericAssetFactory.getBlAssetFromContainer(GenericAssetFactory.java:282)
          at com.bladelogic.om.infra.app.collector.LiveAssetContainer.getBlAsset(LiveAssetContainer.java:72)
          at com.bladelogic.om.infra.app.collector.ServerAssetContainer.getBlAsset(ServerAssetContainer.java:109)
          at com.bladelogic.om.infra.app.api.factory.BlAssetServiceImpl.getBlAssetFromContainer(BlAssetServiceImpl.java:424)
          at com.bladelogic.om.infra.model.generic.rest.assets.RESTAssetRequestHandler.handleAssetRequest(RESTAssetRequestHandler.java:294)
          at com.bladelogic.om.infra.model.generic.rest.RESTfulRequestServiceImpl.handleIdRequest(RESTfulRequestServiceImpl.java:751)
          at com.bladelogic.om.infra.model.generic.rest.RESTfulRequestServiceImpl.processGet(RESTfulRequestServiceImpl.java:238)
          at com.bladelogic.webservices.rest.RESTServlet.handleGet(RESTServlet.java:635)
          at com.bladelogic.webservices.rest.RESTServlet.doGet(RESTServlet.java:292)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
          at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
          at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
          at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)
          at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)
          at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)
          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
          at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
          at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
          at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
          at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
          at java.lang.Thread.run(Thread.java:619)
[01 May 2012 17:25:11,868] [http-10843-Processor11] [ERROR] [::] [] Unable to find corresponding context for the serviceGroupId: urn:uuid:4C0FED0F283B75F8A11335888761510
org.apache.axis2.AxisFault: Unable to find corresponding context for the serviceGroupId: urn:uuid:4C0FED0F283B75F8A11335888761510
          at org.apache.axis2.context.ConfigurationContext.getServiceGroupContextFromSoapSessionTable(ConfigurationContext.java:541)
          at org.apache.axis2.context.ConfigurationContext.fillServiceContextAndServiceGroupContext(ConfigurationContext.java:249)
          at com.bladelogic.webservices.framework.WSInFlowDispatchPhase.loadContexts(WSInFlowDispatchPhase.java:256)
          at com.bladelogic.webservices.framework.WSInFlowDispatchPhase.checkPostConditions(WSInFlowDispatchPhase.java:146)
          at org.apache.axis2.engine.Phase.invoke(Phase.java:334)
          at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:251)
          at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:160)
          at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)
          at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
          at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
          at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
          at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)
          at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)
          at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)
          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
          at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
          at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
          at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
          at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
          at java.lang.Thread.run(Thread.java:619)
[01 May 2012 17:25:11,880] [http-10843-Processor7] [INFO] [::] [] LoginUsingUserCredentialResponse | Login Request with session id urn:uuid:4C0FED0F283B75F8A11335889520656
[01 May 2012 17:25:12,022] [http-10843-Processor7] [INFO] [BLAdmin:Anonymous:192.168.19.203] [] Assuming the default role: BLAdmins for the current soap session.
[01 May 2012 17:25:12,022] [http-10843-Processor7] [INFO] [BLAdmin:BLAdmins:192.168.19.203] [] User(BLAdmin) is authenticated with the authentication service.
[01 May 2012 17:25:12,070] [http-10843-Processor7] [INFO] [BLAdmin:BLAdmins:192.168.19.203] [] Creating connection pool for urn:uuid:4C0FED0F283B75F8A11335889520656
[01 May 2012 17:25:12,207] [Client-Connections-Thread-9] [INFO] [BLAdmin:BLAdmins:192.168.19.203] [Client] User 'BLAdmin' assumed the role 'BLAdmins'
[01 May 2012 17:25:12,211] [http-10843-Processor7] [ERROR] [BLAdmin:BLAdmins:192.168.19.203] [] testNICK-12 is not enrolled
com.bladelogic.om.infra.mfw.util.BlException: testNICK-12 is not enrolled
          at com.bladelogic.om.virtualization.model.depot.vgp.VirtAssetServiceHelper.getVirtualGuestKeyByServerName(VirtAssetServiceHelper.java:171)
          at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceImpl.java:625)
          at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceRouterStubImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceRouterStubImpl.java:654)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at com.bladelogic.om.infra.cli.factory.APICommand.executeCommandOnSingleObject(APICommand.java:127)
          at com.bladelogic.om.infra.cli.factory.APICommand.executeCommand(APICommand.java:106)
          at com.bladelogic.om.infra.cli.factory.ComplexCommand.executeCommand(ComplexCommand.java:72)
          at com.bladelogic.om.infra.cli.CLIEngine.executeCommand(CLIEngine.java:116)
          at com.bladelogic.om.infra.model.clitunnel.CLITunnelServiceImpl.executeCommand(CLITunnelServiceImpl.java:134)
          at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceSkeleton.executeCommandByParamList(CLITunnelServiceSkeleton.java:174)
          at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceMessageReceiverInOut.invokeBusinessLogic(CLITunnelServiceMessageReceiverInOut.java:102)
          at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
          at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:114)
          at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:173)
          at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)
          at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
          at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
          at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
          at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)
          at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)
          at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)
          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
          at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
          at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
          at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
          at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
          at java.lang.Thread.run(Thread.java:619)
[01 May 2012 17:25:12,212] [http-10843-Processor7] [ERROR] [BLAdmin:BLAdmins:192.168.19.203] [] [CLI Tunnel Service] testNICK-12 is not enrolled
com.bladelogic.om.infra.mfw.util.BlException: testNICK-12 is not enrolled
          at com.bladelogic.om.virtualization.model.depot.vgp.VirtAssetServiceHelper.getVirtualGuestKeyByServerName(VirtAssetServiceHelper.java:171)
          at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceImpl.java:625)
          at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceRouterStubImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceRouterStubImpl.java:654)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at com.bladelogic.om.infra.cli.factory.APICommand.executeCommandOnSingleObject(APICommand.java:127)
          at com.bladelogic.om.infra.cli.factory.APICommand.executeCommand(APICommand.java:106)
          at com.bladelogic.om.infra.cli.factory.ComplexCommand.executeCommand(ComplexCommand.java:72)
          at com.bladelogic.om.infra.cli.CLIEngine.executeCommand(CLIEngine.java:116)
          at com.bladelogic.om.infra.model.clitunnel.CLITunnelServiceImpl.executeCommand(CLITunnelServiceImpl.java:134)
          at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceSkeleton.executeCommandByParamList(CLITunnelServiceSkeleton.java:174)
          at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceMessageReceiverInOut.invokeBusinessLogic(CLITunnelServiceMessageReceiverInOut.java:102)
          at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
          at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:114)
          at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:173)
          at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)
          at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
          at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
          at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
          at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)
          at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)
          at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)
          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
          at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
          at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
          at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
          at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
          at java.lang.Thread.run(Thread.java:619)
Like Show 0 Likes(0)

Actions
6. Job Hangs when creating VM

Aryan Anantwar May 3, 2012 6:46 AM (in response to Nick Rawlins)

Hi Friend,

here is the ERROR from the log u have mentioned here:

[ERROR] [BLAdmin:BLAdmins:192.168.19.203] [] [CLI Tunnel Service] testNICK-12 is not enrolled
com.bladelogic.om.infra.mfw.util.BlException: testNICK-12 is not enrolled.

testNICK-12 is the instance name which you are going to create through CLM.

it occures when the VGP job creates a VM on Datastore & install OS with RSCD agent.
After all installations BBSA enrolls the RSCD agent. if it fails then the VGP job get failed & it rollbacks the things, during rollback it deletes the created VM.
This job stuck in running state, that means it have created VM & failed to enroll the Agent.
But While Rollback it get stuck.

regards,
Dnyaneshwar.
Like Show 0 Likes(0)

Actions
7. Job Hangs when creating VM

Nick Rawlins May 3, 2012 7:05 AM (in response to Aryan Anantwar)

Thats great, thanks Dnyaneshwar. It makes sense and that would explain why the error occurred after about 45 minutes of it running... which is around the time it takes to deploy it manually.

So how could I troubleshoot why the RSCD agent is failing to install?

Thanks
Like Show 0 Likes(0)

Actions
8. Re: Job Hangs when creating VM

Ahmad Ramzi Jamjoom May 6, 2012 7:49 AM (in response to Nick Rawlins)

As a hint, check the agent being installed on the OS, if it's comptabile with the OS or not!

We had a similar error, but the difference is that in our enviornment the agents are pre-installed in the VM Templates and a 64-bit agent was installed on 32-bit VM template, so we couldn't connect to the provisioned machine and continue service request.
Like Show 0 Likes(0)

Actions
9. Job Hangs when creating VM

Vinnie Lima May 18, 2012 3:05 PM (in response to Ahmad Ramzi Jamjoom)

There could be various causes to that symptom, which I've experienced as well. Are you using DHCP or Static IP for your Container? Is CPM configured to use IP address as VM names or have you switched to use name?

We've had issues with the RSCD Vcenter agent that is part of CLM 2.1 SP1. Once and a while we've had to restart the agent on the Vcenter. Take a look at the BSA config server logfile for any errors there.

Also make sure that a) the RSCD agent on your VM Template is licensed correctly, and that in BSA a machine with that same name (or IP if you are using IP for names) does NOT exist in the Server workspace (BSA). If it does (because the last provisioning job was unsuccessful) all hell breaks loose. Also make sure VCenter doesn't have a VM already with that same name from a previous provisioning.
Like Show 0 Likes(0)

Actions