14 Replies Latest reply: May 3, 2012 5:04 PM by Joe Piotrowski RSS

Hung deployment job in BBSA

realtek

Hi,

 

I have a VM deployment which has hung for over 24 hours, which logs would I check to see whats going on?

 

Thanks!

  • 1. Hung deployment job in BBSA
    Omkar Hajare

    Hi,

     

    What exactly you are trying to deploy ? BLPackage / Software Package / Patches ?

    Check transaction log (%RSCD home%\Transaction\log\bldeploy..) on target server.

     

    Regards,

    Omkar

  • 2. Hung deployment job in BBSA
    Prabhat Handoo

    The job logs and the appserver.log to begin with.

  • 3. Hung deployment job in BBSA
    realtek

    Thanks, I will look here.

     

    I am deploying a VGP job.

     

    Thanks

  • 4. Hung deployment job in BBSA
    realtek

    Hi,

     

    How can I terminate this job?

     

    Thanks

  • 5. Hung deployment job in BBSA
    Prabhat Handoo

    You can cancel the job which is running. What is the state of job: Running? Check the properties of the job which has the entries for the JOB_TIMEOUT and JOB_PART_TIMEOUT

     

    Also what do you see in job log for this VGP job? See the appserver.log as well. What is the version of the BBSA? there was a defect in 8.0 which did not allow any special characters in a VGP job.

  • 6. Hung deployment job in BBSA
    realtek

    This is Bladelogic 8.1 and the JOB_TIMEOUT and JOB_PART_TIMEOUT are both set to 0.

     

    The Job Status is Running... i'll check the logs.

  • 7. Hung deployment job in BBSA
    realtek

    Hi,

     

    This seems to be the error:

     

    [01 May 2012 17:25:11,857] [http-10843-Processor3] [ERROR] [BLAdmin:BLAdmins:192.168.19.207] [] Resource not found

    com.bladelogic.om.infra.mfw.util.NotFoundException: could not find asset /Virtual Machines/testNICK-12/BMC_VMware_VirtualMachineInfo on host 192.168.20.20 with asset type BMC_VMware_VirtualMachineInfo

              at com.bladelogic.om.infra.model.asset.generic.GenericAssetFactory.loadValuesAndPopulateAsset(GenericAssetFactory.java:298)

              at com.bladelogic.om.infra.model.asset.generic.GenericAssetFactory.getBlAssetFromContainer(GenericAssetFactory.java:282)

              at com.bladelogic.om.infra.app.collector.LiveAssetContainer.getBlAsset(LiveAssetContainer.java:72)

              at com.bladelogic.om.infra.app.collector.ServerAssetContainer.getBlAsset(ServerAssetContainer.java:109)

              at com.bladelogic.om.infra.app.api.factory.BlAssetServiceImpl.getBlAssetFromContainer(BlAssetServiceImpl.java:424)

              at com.bladelogic.om.infra.model.generic.rest.assets.RESTAssetRequestHandler.handleAssetRequest(RESTAssetRequestHandler.java:294)

              at com.bladelogic.om.infra.model.generic.rest.RESTfulRequestServiceImpl.handleIdRequest(RESTfulRequestServiceImpl.java:751)

              at com.bladelogic.om.infra.model.generic.rest.RESTfulRequestServiceImpl.processGet(RESTfulRequestServiceImpl.java:238)

              at com.bladelogic.webservices.rest.RESTServlet.handleGet(RESTServlet.java:635)

              at com.bladelogic.webservices.rest.RESTServlet.doGet(RESTServlet.java:292)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

              at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)

              at java.security.AccessController.doPrivileged(Native Method)

              at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

              at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)

              at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)

              at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)

              at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)

              at java.security.AccessController.doPrivileged(Native Method)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)

              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)

              at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)

              at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)

              at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)

              at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

              at java.lang.Thread.run(Thread.java:619)

    [01 May 2012 17:25:11,868] [http-10843-Processor11] [ERROR] [::] [] Unable to find corresponding context for the serviceGroupId: urn:uuid:4C0FED0F283B75F8A11335888761510

    org.apache.axis2.AxisFault: Unable to find corresponding context for the serviceGroupId: urn:uuid:4C0FED0F283B75F8A11335888761510

              at org.apache.axis2.context.ConfigurationContext.getServiceGroupContextFromSoapSessionTable(ConfigurationContext.java:541)

              at org.apache.axis2.context.ConfigurationContext.fillServiceContextAndServiceGroupContext(ConfigurationContext.java:249)

              at com.bladelogic.webservices.framework.WSInFlowDispatchPhase.loadContexts(WSInFlowDispatchPhase.java:256)

              at com.bladelogic.webservices.framework.WSInFlowDispatchPhase.checkPostConditions(WSInFlowDispatchPhase.java:146)

              at org.apache.axis2.engine.Phase.invoke(Phase.java:334)

              at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:251)

              at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:160)

              at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)

              at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

              at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)

              at java.security.AccessController.doPrivileged(Native Method)

              at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

              at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)

              at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)

              at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)

              at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)

              at java.security.AccessController.doPrivileged(Native Method)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)

              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)

              at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)

              at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)

              at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)

              at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

              at java.lang.Thread.run(Thread.java:619)

    [01 May 2012 17:25:11,880] [http-10843-Processor7] [INFO] [::] [] LoginUsingUserCredentialResponse | Login Request with session id urn:uuid:4C0FED0F283B75F8A11335889520656

    [01 May 2012 17:25:12,022] [http-10843-Processor7] [INFO] [BLAdmin:Anonymous:192.168.19.203] [] Assuming the default role: BLAdmins for the current soap session.

    [01 May 2012 17:25:12,022] [http-10843-Processor7] [INFO] [BLAdmin:BLAdmins:192.168.19.203] [] User(BLAdmin) is authenticated with the authentication service.

    [01 May 2012 17:25:12,070] [http-10843-Processor7] [INFO] [BLAdmin:BLAdmins:192.168.19.203] [] Creating connection pool for urn:uuid:4C0FED0F283B75F8A11335889520656

    [01 May 2012 17:25:12,207] [Client-Connections-Thread-9] [INFO] [BLAdmin:BLAdmins:192.168.19.203] [Client] User 'BLAdmin' assumed the role 'BLAdmins'

    [01 May 2012 17:25:12,211] [http-10843-Processor7] [ERROR] [BLAdmin:BLAdmins:192.168.19.203] [] testNICK-12 is not enrolled

    com.bladelogic.om.infra.mfw.util.BlException: testNICK-12 is not enrolled

              at com.bladelogic.om.virtualization.model.depot.vgp.VirtAssetServiceHelper.getVirtualGuestKeyByServerName(VirtAssetServiceHelper.java:171)

              at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceImpl.java:625)

              at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceRouterStubImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceRouterStubImpl.java:654)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at com.bladelogic.om.infra.cli.factory.APICommand.executeCommandOnSingleObject(APICommand.java:127)

              at com.bladelogic.om.infra.cli.factory.APICommand.executeCommand(APICommand.java:106)

              at com.bladelogic.om.infra.cli.factory.ComplexCommand.executeCommand(ComplexCommand.java:72)

              at com.bladelogic.om.infra.cli.CLIEngine.executeCommand(CLIEngine.java:116)

              at com.bladelogic.om.infra.model.clitunnel.CLITunnelServiceImpl.executeCommand(CLITunnelServiceImpl.java:134)

              at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceSkeleton.executeCommandByParamList(CLITunnelServiceSkeleton.java:174)

              at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceMessageReceiverInOut.invokeBusinessLogic(CLITunnelServiceMessageReceiverInOut.java:102)

              at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)

              at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:114)

              at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:173)

              at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)

              at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

              at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)

              at java.security.AccessController.doPrivileged(Native Method)

              at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

              at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)

              at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)

              at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)

              at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)

              at java.security.AccessController.doPrivileged(Native Method)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)

              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)

              at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)

              at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)

              at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)

              at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

              at java.lang.Thread.run(Thread.java:619)

    [01 May 2012 17:25:12,212] [http-10843-Processor7] [ERROR] [BLAdmin:BLAdmins:192.168.19.203] [] [CLI Tunnel Service] testNICK-12 is not enrolled

    com.bladelogic.om.infra.mfw.util.BlException: testNICK-12 is not enrolled

              at com.bladelogic.om.virtualization.model.depot.vgp.VirtAssetServiceHelper.getVirtualGuestKeyByServerName(VirtAssetServiceHelper.java:171)

              at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceImpl.java:625)

              at com.bladelogic.om.virtualization.app.service.VirtualGuestPackageServiceRouterStubImpl.getVirtualGuestKeyByServerName(VirtualGuestPackageServiceRouterStubImpl.java:654)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at com.bladelogic.om.infra.cli.factory.APICommand.executeCommandOnSingleObject(APICommand.java:127)

              at com.bladelogic.om.infra.cli.factory.APICommand.executeCommand(APICommand.java:106)

              at com.bladelogic.om.infra.cli.factory.ComplexCommand.executeCommand(ComplexCommand.java:72)

              at com.bladelogic.om.infra.cli.CLIEngine.executeCommand(CLIEngine.java:116)

              at com.bladelogic.om.infra.model.clitunnel.CLITunnelServiceImpl.executeCommand(CLITunnelServiceImpl.java:134)

              at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceSkeleton.executeCommandByParamList(CLITunnelServiceSkeleton.java:174)

              at com.bladelogic.webservices.skeleton.clitunnel.CLITunnelServiceMessageReceiverInOut.invokeBusinessLogic(CLITunnelServiceMessageReceiverInOut.java:102)

              at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)

              at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:114)

              at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:173)

              at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:167)

              at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:142)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

              at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)

              at java.security.AccessController.doPrivileged(Native Method)

              at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

              at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)

              at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:262)

              at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:52)

              at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:171)

              at java.security.AccessController.doPrivileged(Native Method)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)

              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)

              at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)

              at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)

              at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)

              at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

              at java.lang.Thread.run(Thread.java:619)

  • 8. Hung deployment job in BBSA
    Prabhat Handoo

    There are 3 errors that I see in these logs:

     

    Unable to find corresponding context for the serviceGroupId: urn:uuid:4C0FED0F283B75F8A11335888761510 -->> This can be ignored. This should ideally be a warning and not an error

     

    The rest 2 are interesting:

     

    [ERROR] [BLAdmin:BLAdmins:192.168.19.203] [] testNICK-12 is not enrolled

    com.bladelogic.om.infra.mfw.util.BlException: testNICK-12 is not enrolled

     

    AND

     

    Resource not found

    com.bladelogic.om.infra.mfw.util.NotFoundException: could not find asset /Virtual Machines/testNICK-12/BMC_VMware_VirtualMachineInfo on host 192.168.20.20 with asset type BMC_VMware_VirtualMachineInfo

     

    I am not sure about the 2nd error, but is testNICK-12 the VM that you are trying to provision? Maybe it has hung at the state of enrolling it. Can you ping testNICK-12 from the Appserver?

  • 9. Hung deployment job in BBSA
    realtek

    Hi,

     

    No the VM hasn't been provisioned... nothing for it in the datastore or anywhere so I don't think it got that far.

     

    I agree, very odd!

     

    I'm trying to manually provision it and see what happens!

  • 10. Hung deployment job in BBSA
    Jim Wilson

    Shot in the dark: Did the VMWare environment go into maintenance mode?  Check that you cab create a new VM outside BSA/CLM

  • 11. Hung deployment job in BBSA
    realtek

    Hi Jim,

     

    No not in maintenance mode unfortunately but good idea! perhaps I will escalate to support

     

     

    Thanks!

  • 12. Hung deployment job in BBSA
    Prabhat Handoo

    Realtek: I saw your other post- https://communities.bmc.com/communities/message/241278#241278

     

    I believe these both are for the same error. As I said earlier, This is a problem with agent enrollment. I have seen that happening if you are not able to reach to the agent via hostname. Do you think name resolution could be a problem here?

  • 13. Hung deployment job in BBSA
    realtek

    Hi prabhathandoo,

     

    Thank you, yes I think you may have hit the nail on the head.

     

    In my CLM environment I am using the Hosts file as no DNS is currently installed.

     

     

    Very good point.

     

    Thank you!

  • 14. Hung deployment job in BBSA
    Joe Piotrowski

    Marking the thread as Answered. Please edit if this issue is still not resolved.