9 Replies Latest reply on Jul 18, 2013 11:24 AM by David NameToUpdate

    BSA Error BMC-BSA-99999999

      I am seeing this error (BMC-BSA-99999999) in BSA 8.3 (as part of a CLM 3.1 stack)

      It occurs while running a vgp deploy job - no BSA error log (the console says the job is cancelled/aborted) - a blank error is displayed when attempting to view the log.  I am using a VM Hypervisor -- after the vgp dies the vm cloning process continues and when finished the OS in the new vm is corrupted (the base template can be cloned in vSphere successfully)

       

       

      This error originally happened when run from CLM so I have posted there as well (see CLM 3.1 / BSA provisioning problem - BMC-BSA-99999999) for more info.

        • 1. Re: BSA Error BMC-BSA-99999999
          Bill Robinson

          can you get the bldeploy log and blcoserver log from this deploy that will be on the vcenter server in the <install>/RSCD/Transactions/log directory ?

          • 2. Re: BSA Error BMC-BSA-99999999

            I will pull those logs, but I may have found the issue (or at least an issue).

            I ran the application server diagnostics from the console and got failures on the appserver .

             

            All three failed (they are all running on the same linux box)

            I restarted the blappserv - no change

            I also restarted the linux server - no change.

             

            What else could cause this?

             

             

            • 3. Re: BSA Error BMC-BSA-99999999
              Bill Robinson

              is the bladelogic.keystore file the same between all deployments?  are those services actually running - check the appserver logs for each deployment or look in the infra mgr.

              • 4. Re: BSA Error BMC-BSA-99999999

                Yes the keystores are all the same and the services are running (checked the logs - no errors and checked the status of PIDs listed in the logs).

                 

                Is it normal in BSA 8.3 infr manager to show the appservers as being stopped even though the logs say running?

                • 5. Re: BSA Error BMC-BSA-99999999
                  Bill Robinson

                  No – that would usually indicate:

                  - time is out of sync (but not here since it’s the same box)

                  - keystores don’t match – how exactly did you check that the keystore files were the same ?

                  - registry port is not open

                   

                  Can you look in the appserverlauncher.log – do you see any messages there when you try and browse the other instances ?

                  • 6. Re: BSA Error BMC-BSA-99999999

                    I checked the keystores by doing a side by side file compare - they all appeared to be identical.

                    I tailed the appserverlauncher log and browsed the other instances in the Infrastructure manager - no entries appeared in the log.

                     

                    I did find entries from the start of of the appservers in that log - but no errors reported

                    • 7. Re: BSA Error BMC-BSA-99999999
                      Bill Robinson

                      You should do a md5sum on the keystore files, not really sure what ‘side by side’ means.

                      • 8. Re: BSA Error BMC-BSA-99999999

                        The md5sums are all identical

                        The 'side-by-side' comparison was a ascii file compare - character to

                        character

                         

                        I also dug up the rscd logs from vCenter - the *last *bldeploy entry for

                        the vgp deploy job occurs about 20 seconds after the job starts (while

                        performing file locks) - no entries concerning the failure.  The logging

                        stops and the file says in the tmp directory.

                         

                        There is an entry in the blcoserver log for this job -

                        06/21/13 13:35:44.819 INFO     blcoserver -  LabVC 75552 BladeLogicRSCD

                        (Not_available): (Not_available): Processing modify asset attribute :

                        VMware_Memory_Shares to : normal

                        06/21/13 14:02:42.724 INFO     blcoserver -  LabVC 75552 BladeLogicRSCD

                        (Not_available): (Not_available): Changed the configuration for BusSRV3

                        06/21/13 14:02:58.773 INFO     blcoserver -  LabVC 75552 BladeLogicRSCD

                        (Not_available): (Not_available): Is the request is to fetch VM Template?:

                        false

                        06/21/13 14:02:59.417 INFO     blcoserver -  LabVC 75552 BladeLogicRSCD

                        (Not_available): (Not_available): Is the request is to fetch VM Template?:

                        false

                        06/21/13 14:03:02.142 INFO     blcoserver -  LabVC 75552 BladeLogicRSCD

                        (Not_available): (Not_available): Processing delete asset : BusSRV3 of type

                        : BMC_VMware_VirtualMachine

                        06/21/13 14:03:02.168 WARN     blcoserver -  LabVC 75552 BladeLogicRSCD

                        (Not_available): (Not_available): Asset not modified:

                        BMC_VMware_VirtualMachineHardwareCPU:/Virtual Machines/BusSRV3/Hardware/CPU

                        06/21/13 14:03:02.171 WARN     blcoserver -  LabVC 75552 BladeLogicRSCD

                        (Not_available): (Not_available): Asset not modified:

                        BMC_VMware_VirtualMachineHardwareMemory:/Virtual

                        Machines/BusSRV3/Hardware/Memory

                        06/21/13 14:03:17.257 INFO     blcoserver -  LabVC 75552 BladeLogicRSCD

                        (Not_available): (Not_available): Deleted the virtual machine BusSRV3

                        06/21/13 14:03:27.263 ERROR    blcoserver -  LabVC 75552 BladeLogicRSCD

                        (Not_available): (Not_available): Commit failed

                        06/21/13 14:03:27.267 ERROR    blcoserver -  LabVC 75552 BladeLogicRSCD

                        (Not_available): (Not_available): BMC-BSA-00400000:Failed to power on the

                        VM after reconfiguration. Task completed with an error. Insufficient

                        resources.

                        06/21/13 14:03:27.270 ERROR    blcoserver -  LabVC 75552 BladeLogicRSCD

                        (BladeLogicRSCD): (Not_available): Exception occurred in plugin:

                        com.bladelogic.om.infra.daal.jco.CustomObjectException: Failed to power on

                        the VM after reconfiguration. Task completed with an error. Insufficient

                        resources.

                        06/21/13 14:03:27.273 ERROR    blcoserver -  LabVC 75552 BladeLogicRSCD

                        (BladeLogicRSCD): (Not_available): Failed to power on the VM after

                        reconfiguration. Task completed with an error. Insufficient resources.

                        06/21/13 14:03:27.278 ERROR    blcoserver -  LabVC 75552 BladeLogicRSCD

                        (BladeLogicRSCD): (Not_available): Exception occurred in plugin:

                        com.bladelogic.om.infra.daal.jco.CustomObjectException: Failed to power on

                        the VM after reconfiguration. Task completed with an error. Insufficient

                        resources.

                        com.bladelogic.virtualization.jpa.VirtCustomObject.bulkPutAttributeValues(VirtCustomObject.java:519)

                        com.bladelogic.om.infra.daal.jco.GenericPlugin.notify(GenericPlugin.java:203)

                        com.bladelogic.om.infra.daal.jco.JPAShim.bulkPutAttributeValues(JPAShim.java:693)

                        Caused by: com.bladelogic.virtualization.ServiceException: Failed to power

                        on the VM after reconfiguration. Task completed with an error. Insufficient

                        resources.

                        com.bladelogic.vmware.VMwareJpaAdapter.processVMException(VMwareJpaAdapter.java:212)

                        com.bladelogic.vmware.deployer.VMwareLegacyDeployer.deploy(VMwareLegacyDeployer.java:346)

                        com.bladelogic.vmware.deployer.VMwareLegacyDeployer.bulkPutAttributeValues(VMwareLegacyDeployer.java:108)

                        com.bladelogic.vmware.deployer.VMwareLegacyDeployer.bulkPutAttributeValues(VMwareLegacyDeployer.java:39)

                        com.bladelogic.virtualization.jpa.VirtCustomObject.bulkPutAttributeValues(VirtCustomObject.java:511)

                        com.bladelogic.om.infra.daal.jco.GenericPlugin.notify(GenericPlugin.java:203)

                        com.bladelogic.om.infra.daal.jco.JPAShim.bulkPutAttributeValues(JPAShim.java:693)

                        Caused by: com.bladelogic.om.virtualization.vmware.VMException: Failed to

                        power on the VM after reconfiguration. Task completed with an error.

                        Insufficient resources.

                        com.bladelogic.om.virtualization.vmware.deploy.AbstractDeployer.checkTaskForErrors(AbstractDeployer.java:274)

                        com.bladelogic.om.virtualization.vmware.VMWareDeployService.commit(VMWareDeployService.java:296)

                        com.bladelogic.om.virtualization.vmware.VMWareDeployService.deploy(VMWareDeployService.java:169)

                        com.bladelogic.vmware.deployer.VMwareLegacyDeployer.deploy(VMwareLegacyDeployer.java:342)

                        com.bladelogic.vmware.deployer.VMwareLegacyDeployer.bulkPutAttributeValues(VMwareLegacyDeployer.java:108)

                        com.bladelogic.vmware.deployer.VMwareLegacyDeployer.bulkPutAttributeValues(VMwareLegacyDeployer.java:39)

                        com.bladelogic.virtualization.jpa.VirtCustomObject.bulkPutAttributeValues(VirtCustomObject.java:511)

                        com.bladelogic.om.infra.daal.jco.GenericPlugin.notify(GenericPlugin.java:203)

                        com.bladelogic.om.infra.daal.jco.JPAShim.bulkPutAttributeValues(JPAShim.java:693)

                         

                        06/21/13 14:03:27.295 ERROR    blcoserver -  LabVC 75552 BladeLogicRSCD

                        (BladeLogicRSCD): (Not_available): Error returned from plug-in ; Plug-in:

                        /BMC_VMware_VirtualInfrastructureManager_win64 ; Plug-in function:

                        blAsset_PutAll ; Plug-in asset:

                        BMC_VMware_VirtualMachineTemplate:10.0.10.51:/Templates/W2k8_64_STD_template

                        ; Plug-in error code: 800 ; Plug-in error message: Failed to power on the

                        VM after reconfiguration. Task completed with an error. Insufficient

                        resources. ; Plug-in detail message: Exception occurred in plugin:

                        com.bladelogic.om.infra.daal.jco.CustomObjectException: Failed to power on

                        the VM after reconfiguration. Task completed with an error. Insufficient

                        resources.

                         

                         

                        The job was started at 13:35:23 and was canceled (apparently by BSA - I

                        have not found a log entry for this action only appears in the console) at

                        13:54:30

                         

                        The insufficient resources error shows in the log about 9 minutes after the

                        BSA server had canceled the job.

                         

                         

                        *DAVID PURKEY

                        *Senior Consultant - Advanced Solutions Group

                        E: david@belmont-ss.com

                        C: 615.509.6991

                        O: 615.340.7744

                        W: http://belmont-ss.com/

                         

                         

                         

                        <http://blog.belmont-ss.com/>

                        <http://www.facebook.com/pages/Belmont-Software-Services/100721798986?ref=hl>

                           <http://www.linkedin.com/company/398875?goback=%2Efcs_GLHD_belmontsoftwareservices_false_2_2_2_2_2_2_2_2_2_2_2_2&trk=ncsrch_hits>

                         

                         

                         

                        • 9. Re: BSA Error BMC-BSA-99999999

                          Problem solved!

                           

                          The application diagnostics wizard gave us a clue - but not the exact problem.

                          All of our application servers are running on the same virtual machine so the clocks could not be out of sync - however it was an issue of the clocks being out of sync - but it was between the database and the application server!

                          It was a time zone problem - the database clock was set to Eastern time zone while the application server was set to the Central Time zone.