10 Replies Latest reply on May 15, 2014 7:42 AM by Jared Murrell

    BladeLogic Performance Issues

    Jared Murrell

      I have a BSA 8.3.03 environment with over 5000 managed objects, and performance is becoming an issue. We're running SQL Server 2008R2 and I can see LCK_M_IX having AvgWaitTime of 180000+ and MaxWaitTime of 770000+. What can I look at to figure out what's causing the Intent Exclusive wait locks?

        • 1. Re: BladeLogic Performance Issues
          Bill Robinson

          are you running any of the blcli database cleanup commands when this happens ?

           

          how large is the database?  can you run the attached script ? and also "select * from delete_tasks' ?

           

          have you opened a ticket on this w/ support yet ?

          1 of 1 people found this helpful
          • 2. Re: BladeLogic Performance Issues
            Jim Wilson

            Are you successfully running cleanup on a regular basis?

             

            I think I would start with suggesting you run the the assessment tool:

             

            Assessment Tool for BSA version 1.02 [Deprecated, Use Health & Value Dashboards 1.4.00]

            1 of 1 people found this helpful
            • 3. Re: BladeLogic Performance Issues
              Jared Murrell

              I was not running a cleanup when this happens... just been running them on a schedule. There hasn't been a ticket opened yet, since I'm pretty sure there are things I can do before going down that path. I'll run that script. Thanks!

              • 4. Re: BladeLogic Performance Issues
                Jared Murrell

                Didn't know about that tool. Thanks!

                • 5. Re: BladeLogic Performance Issues
                  Jared Murrell

                  When I try to run DB Cleanup with the "CLEAN_DB" parameter I get an error that says it failed to delete objects for SBLGroup:

                   

                  Failed to delete objects for class:SBLGroup at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteObjectsByClass(CleanupOperationImpl.java:621) at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllObjectsByClass(CleanupOperationImpl.java:492) at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllTopLevelObjects(CleanupOperationImpl.java:158) at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllObjects(CleanupOperationImpl.java:94) at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.internal_hardDeleteAllObjects(BaseCleanupServiceImpl.java:1427) at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.hardDeleteAllObjects(BaseCleanupServiceImpl.java:1369) at com.bladelogic.om.infra.app. service.cleanup.BaseCleanupServiceImpl.cleanupDatabase(BaseCleanupServiceImpl.java:1283) at com.bladelogic.om.infra.app.api.factory.DeleteServiceImpl.cleanupDatabase(DeleteServiceImpl.java:338) at com.bladelogic.om.infra.app.api.DeleteServiceRequestHandlerImpl.cleanupDatabase_22(DeleteServiceRequestHandlerImpl.java:1013) at com.bladelogic.om.infra.app.api.DeleteServiceRequestHandlerImpl.execute(DeleteServiceRequestHandlerImpl.java:246) at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.executeRequest(APIMethodInvocationProvider.java:54) at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.invoke(APIMethodInvocationProvider.java:39) at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invokeMethodLocally(RoutingServiceImpl.java:317) at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invoke(RoutingServiceImpl.java:244) at com.bladelogic.om.infra.app.api.manage.APIRequestManager.executeRequest(APIRequestManager.java:30) at com.bladelogic.om.infra.message.app.frame.APIMessageFrame.service(APIMessageFrame.java:113) at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processRequest(ClientWorkerThread.java:385) at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processClientRequest(ClientWorkerThread.java:282) at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.handleClientRequest(ClientWorkerThread.java:182) at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:101) at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:27) at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                   

                   

                  Failed to delete objects for class:SBLGroup at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteObjectsByClass(CleanupOperationImpl.java:621) at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllObjectsByClass(CleanupOperationImpl.java:492) at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllTopLevelObjects(CleanupOperationImpl.java:158) at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllObjects(CleanupOperationImpl.java:94) at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.internal_hardDeleteAllObjects(BaseCleanupServiceImpl.java:1427) at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.hardDeleteAllObjects(BaseCleanupServiceImpl.java:1369) at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.cleanupDatabase(BaseCleanupServiceImpl.java:1283) at com.bladelogic.om.infra.app.api.factory.DeleteServiceImpl.cleanupDatabase(DeleteServiceImpl.java:338) at com.bladelogic.om.infra.app.api.DeleteServiceRequestHandlerImpl.cleanupDatabase_22(DeleteServiceRequestHandlerImpl.java:1013) at com.bladelogic.om.infra.app.api.DeleteServiceRequestHandlerImpl.execute(DeleteServiceRequestHandlerImpl.java:246) at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.executeRequest(APIMethodInvocationProvider.java:54) at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.invoke(APIMethodInvocationProvider.java:39) at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invokeMethodLocally(RoutingServiceImpl.java:317) at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invoke(RoutingServiceImpl.java:244) at com.bladelogic.om.infra.app.api.manage.APIRequestManager.executeRequest(APIRequestManager.java:30) at com.bladelogic.om.infra.message.app.frame.APIMessageFrame.service(APIMessageFrame.java:113) at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processRequest(ClientWorkerThread.java:385) at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processClientRequest(ClientWorkerThread.java:282) at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.handleClientRequest(ClientWorkerThread.java:182) at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:101) at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:27) at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                   

                   

                  [Tue May 13 14:27:04 EDT 2014] Cleanup encoutered an error during execution.

                   

                  [Tue May 13 14:27:04 EDT 2014] Executing available patches.

                   

                  [Tue May 13 14:27:17 EDT 2014] Successfully executed available patches.

                   

                   

                  Command execution failed. com.bladelogic.om.infra.mfw.util.BlException: Failed to delete objects for class:SBLGroup: com.bladelogic.om.infra.mfw.util.BlException: Descendents of SBLGroup are: [SDisplayString, SAssetSelectorPropertyReference, SAssetSelectorInclude, SDisplayStringSet, SAssetSelectorExclude, SAssetSelectorAssetAttribute, SPropertyCondition, SAssetAttributeCondition, SPatchCujFilterOptions, SSelectorCondition, SAssetSelector, SConditionSubCondition, SObjectOverflowDetail, SSmartGroupQuery, SObjectOverflow, SPatchCujFilter, SBLGroupAuditTrail, SCatalogUpdateJobAssociation, SBLGroupCondition, SCondition, SBLGroupDevice, SBLGroupComponent, SACEMaskValue, SListBlValueElement, SPropertyReferenceBlValue, SPropertyReferenceProperty, SPropertyReference, SAceMaskBlValue, SACEValue, SPropertySetInstanceOverriddenValueMap, SPropertySetInstanceAuditTrail, SPropertySetInstance, SListBlValue, SPrimitiveBlValue, SEnumeratedBlValue, SRangeBlValue, SAceBlValue, SBlValue]: com.bladelogic.om.infra.mfw.util.BlException: Failed to update class for deletion :SCondition query id STREAMABLE_DELETION_CONDITION_DELETED_OBJECTS_MARKED_FOR_DELETION: com.microsoft.sqlserver.jdbc.SQLServerException: The DELETE statement conflicted with the REFERENCE constraint "FKX1_CONDITION". The conflict occurred in database "Bladelogic_V8", table "dbo.blgroup_condition", column 'condition_id'.

                  • 6. Re: BladeLogic Performance Issues
                    Bill Robinson

                    There should be a fix for this, I suggest opening a support ticket.

                    • 7. Re: BladeLogic Performance Issues
                      Jared Murrell

                      Ok, so on the DB cleanup I see a failure on the following 2 items:

                       

                      SBLGroup

                      SJobRunOfProvisionJob

                       

                      I tried running this against the SQL database, but the SBLGroup error is still there:

                       

                      UPDATE selector_condition SET is_deleted = 0 WHERE is_deleted = 1;

                      UPDATE blgroup_condition SET is_deleted = 0 WHERE is_deleted = 1;

                      UPDATE property_condition SET is_deleted = 0 WHERE is_deleted = 1;

                      UPDATE asset_attribute_condition SET is_deleted = 0 WHERE is_deleted = 1;

                      UPDATE condition_sub_condition SET is_deleted = 0 WHERE is_deleted = 1;

                      • 9. Re: BladeLogic Performance Issues
                        Bill Robinson

                        Please do not manually run sql updates against the database w/o the guidance of support.

                        • 10. Re: BladeLogic Performance Issues
                          Jared Murrell

                          It was based on BMC docs referencing ISS03635259/QM001684972, which is the same error I was getting. I've opened a case now and am waiting for direction. So far I've only been told to run cleanupHistoricalData first, but that had no effect and was already being run. Thanks for the assistance!