Provisioning request intermittently fails with "The virtual guest creation failed:Error retrieving serial number for virtual guest" error

Version 2
    Share:|

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    Cloud Lifecycle Management


    COMPONENT:

    Cloud Lifecycle Management


    APPLIES TO:

    BMC Cloud Lifecycle Management 4.6.x



    PROBLEM:

    Provisioning request intermittently fails with the following error either on a single server request or multiple server requests:

    16 Jan 2018 19:42:59,130 [DEBUG] TASK - [Thread=9b78244a-3864-4b7a-916a-ed69099c14a8::3946d69e-1a62-414e-b556-8001e120e80f::f22f2360-f850-4ef0-be24-5af8a78ec585(227)] [Class=TaskManagerImpl:handleCallback] - Got a callback OR a task propagation event for UUID:::com.bmc.cloud.model.beans.Task@ff10c860
    isCallout = false
    creationTime = 2018/01/16 19:28:44 CST
    taskSubState = task_fail
    className = VirtualGuest
    operationName = constructor
    transactionID = 3946d69e-1a62-414e-b556-8001e120e80f
    completionTime = 2018/01/16 19:42:59 CST
    taskResultURI = null
    parentTaskInternalUUID = 9f379a9b-247b-411b-b6ff-587fed101eeb
    isError = true
    taskState = FAILED
    taskInternalUUID = f22f2360-f850-4ef0-be24-5af8a78ec585
    guid = 3d2637d0-0d5d-474f-a88d-ccb3b3e7c319
    taskProgress = 100.0
    errors = [com.bmc.cloud.model.beans.TaskError@c0c0adb2
         severity = CRITICAL
         hostname = PlatformManager_Hostname
         errorCause = The virtual guest creation failed:Error retrieving serial number for virtual guest : <target VM hostname> . BBSA response: Resource not found. Invalid URI: Failed to connect to <target VM hostname>:4750.
         taskInternalUUID = f22f2360-f850-4ef0-be24-5af8a78ec585
         errorTrace = java.lang.IllegalArgumentException: Error retrieving serial number for virtual guest : <target VM hostname> . BBSA response: Resource not found. Invalid URI: Failed to connect to <target VM hostname>:4750
        at com.bmc.cloud.provider.bbsaprovider.task.VirtualGuestTaskExecHelper.setSerialNumberForVG(VirtualGuestTaskExecHelper.java:638)
        at com.bmc.cloud.provider.bbsaprovider.task.VirtualGuestTaskExecHelper.populateVirtualGuest(VirtualGuestTaskExecHelper.java:532)
        at com.bmc.cloud.provider.bbsaprovider.task.VirtualGuestTaskExecHelper.execute(VirtualGuestTaskExecHelper.java:187)
        at com.bmc.cloud.provider.bbsaprovider.task.VirtualGuestTaskExecHelper.execute(VirtualGuestTaskExecHelper.java:135)
        at com.bmc.cloud.provider.bbsaprovider.task.VirtualGuestConstructorExecHelper.execute(VirtualGuestConstructorExecHelper.java:82)
        at com.bmc.cloud.task.scxml.actions.ExecAction.doExecute(ExecAction.java:74)
        at com.bmc.cloud.task.scxml.actions.CSMBaseAction.execute(CSMBaseAction.java:110)
        at org.apache.commons.scxml.semantics.SCXMLSemanticsImpl.executeActions(SCXMLSemanticsImpl.java:239)
        at org.apache.commons.scxml.SCXMLExecutor.reset(SCXMLExecutor.java:252)
        at org.apache.commons.scxml.SCXMLExecutor.go(SCXMLExecutor.java:351)
        at com.bmc.cloud.task.impl.SCXMLTaskImpl.executeTask(SCXMLTaskImpl.java:164)
        at com.bmc.cloud.task.impl.TaskImpl.fireEvent(TaskImpl.java:675)
        at com.bmc.cloud.taskmanager.impl.SCXMLEventWorker.process(SCXMLEventWorker.java:58)
        at com.bmc.cloud.taskmanager.impl.SCXMLEventWorker.process(SCXMLEventWorker.java:17)
        at com.bmc.cloud.worker.WorkQueue$WorkerThread.run(WorkQueue.java:430)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

         guid = 318b9f07-f060-4874-8f7d-06f27a428d7e
         errorID = BMC-CLMT-10100019
         errorAction = Troubleshooting information is available in the CLM logs (Needs Administrator Access).
         transactionID = 3946d69e-1a62-414e-b556-8001e120e80f]
    isSuccess = false
    taskStatusURI = https://PlatformManager:9443/csm/task/f22f2360-f850-4ef0-be24-5af8a78ec585/status


    CAUSE:

    This failure is due to BladeLogic Config Server is not able to resolve a target server hostname when trying to retrieve serial number information using BSA REST API


    SOLUTION:

    BSA REST API call to retrieve a target VM's serial number is done post- USP Job success and prior to create_user_job nsh script and is executed by BladeLogic Config Server that is configured as your Platform Manager's Provider instead of BladeLogic Job Server.

    In this scenario, you can see this error when the following REST API call is triggered:

    16 Jan 2018 19:42:56,149 [ERROR] BBSA - [Thread=9b78244a-3864-4b7a-916a-ed69099c14a8::3946d69e-1a62-414e-b556-8001e120e80f::f22f2360-f850-4ef0-be24-5af8a78ec585(225)] [Class=VirtualGuestTaskExecHelper:setSerialNumberForVG] - Request /id/SystemObject/Server/bb3f6b28-3bf4-4d10-a1d9-31129f448af9/Assets/SystemInfo/89000001/System/AssetAttributeValues/Serial Number failed. BBSA response: Resource not found. Invalid URI: Failed to connect to <target VM hostname>:4750
    16 Jan 2018 19:42:56,149 [ERROR] BBSA - [Thread=9b78244a-3864-4b7a-916a-ed69099c14a8::3946d69e-1a62-414e-b556-8001e120e80f::f22f2360-f850-4ef0-be24-5af8a78ec585(225)] [Class=VirtualGuestTaskExecHelper:execute] - Could not create Server (Virtual Guest): <target VM hostname>
    java.lang.IllegalArgumentException: Error retrieving serial number for virtual guest : <target VM hostname> . BBSA response: Resource not found. Invalid URI: Failed to connect to <target VM hostname>:4750
        at com.bmc.cloud.provider.bbsaprovider.task.VirtualGuestTaskExecHelper.setSerialNumberForVG(VirtualGuestTaskExecHelper.java:638)
        at com.bmc.cloud.provider.bbsaprovider.task.VirtualGuestTaskExecHelper.populateVirtualGuest(VirtualGuestTaskExecHelper.java:532)
        at com.bmc.cloud.provider.bbsaprovider.task.VirtualGuestTaskExecHelper.execute(VirtualGuestTaskExecHelper.java:187)
        at com.bmc.cloud.provider.bbsaprovider.task.VirtualGuestTaskExecHelper.execute(VirtualGuestTaskExecHelper.java:135)
        at com.bmc.cloud.provider.bbsaprovider.task.VirtualGuestConstructorExecHelper.execute(VirtualGuestConstructorExecHelper.java:82)
        at com.bmc.cloud.task.scxml.actions.ExecAction.doExecute(ExecAction.java:74)
        at com.bmc.cloud.task.scxml.actions.CSMBaseAction.execute(CSMBaseAction.java:110)
        at org.apache.commons.scxml.semantics.SCXMLSemanticsImpl.executeActions(SCXMLSemanticsImpl.java:239)
        at org.apache.commons.scxml.SCXMLExecutor.reset(SCXMLExecutor.java:252)
        at org.apache.commons.scxml.SCXMLExecutor.go(SCXMLExecutor.java:351)
        at com.bmc.cloud.task.impl.SCXMLTaskImpl.executeTask(SCXMLTaskImpl.java:164)
        at com.bmc.cloud.task.impl.TaskImpl.fireEvent(TaskImpl.java:675)
        at com.bmc.cloud.taskmanager.impl.SCXMLEventWorker.process(SCXMLEventWorker.java:58)
        at com.bmc.cloud.taskmanager.impl.SCXMLEventWorker.process(SCXMLEventWorker.java:17)
        at com.bmc.cloud.worker.WorkQueue$WorkerThread.run(WorkQueue.java:430)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

    If you have multiple BladeLogic Job Servers and the Job Server (e.g.: BSA Server2) that completes the USP Job successfully is different server than BladeLogic Config Server (e.g.: BSA Server1), you have to make sure that the both of these BladeLogic servers  (e.g.: BSA Server1 and BSA Server2) are able to resolve your target VM hostnames at the same time.

    Since the REST API above is triggered after Post-USP job and it is triggered by BladeLogic Config Server, if there is a delay at least 2 or 3 minutes for BSA Server1 to be able to resolve your target VM hostname compare to BSA Server2 that completes the execution of the USP Job, then you will see the error above due to unresolved target VM hostname.

    For every failure REST API, Platform Manager will retry 5 times with 30 seconds interval (e.g.: 150 seconds = 2.5 minutes)

    There are 2 solutions to address the above error:

       
    • Make sure that your multiple BladeLogic Server is able to resolve your target VM hostnames at the same time or at least your configured BladeLogic Config Server that is configured with Platform Manager's Provider has to be able to resolve your target VM hostnames sooner compare to BladeLogic Job Server that will execute your Virtual Guest Job or USP Job.
    •  
    • or configure or consolidate all activities (Virtual Guest Job, USP Job, BSA REST API, and etc) in a single BladeLogic Config Server/Job Server. In this case, it will ensure that BSA REST API is triggered from the same BladeLogic Server that completes your execution of USP Job.
      
    List of extra validations that can be done after following either of the solutions above:  
       
    • Run "ping <target VM hostname>" from BladeLogic Config Server
    •  
    • Run "agentinfo <target VM hostname>" from BladeLogic Config Server
    •  
    • Verify that your target VM's RSCD agent is listening on the correct port (e.g.: port 4750)
    •  
    • Verify that your target VM's RSCD agent service is up and running
    Please contact BMC Customer Support for more details information on this discussion.  

    Additional information: Refer How to Videos - Cloud Lifecycle Management playlist for How-to videos created by BMC Support.

     


    Article Number:

    000148056


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles