13 Replies Latest reply on Apr 6, 2016 1:10 PM by Luciano Spagnuolo

    DB cleanup error Failed to delete objects for class:SComponent

    Luciano Spagnuolo

      So I've seen this error off and on and usually for not a long time but this time it doesn't seem to go away.  Has anyone come across this?

      This is running the weekly database clean up job as per recommendation from the Docs.

       

      Error below

      Info 03/31/2016 14:10:41 Command execution failed.  Failed to delete objects for class:SComponent

      Info 03/31/2016 14:10:41 [Thu Mar 31 14:10:40 EDT 2016] Successfully executed available patches.

      Info 03/31/2016 14:10:23 [Thu Mar 31 14:10:23 EDT 2016] Executing available patches.

      Info 03/31/2016 14:10:23 [Thu Mar 31 14:10:23 EDT 2016] Cleanup encoutered an error during execution.

      Info 03/31/2016 14:10:23 [Thu Mar 31 14:10:23 EDT 2016] Skipping class: SExitCode

      Info 03/31/2016 14:10:23 [Thu Mar 31 14:10:23 EDT 2016] Skipping class: SProgressStatus

      Info 03/31/2016 14:10:23 [Thu Mar 31 14:10:23 EDT 2016] Skipping class: SBLOption

       

      Lou Spagnuolo

        • 1. Re: DB cleanup error Failed to delete objects for class:SComponent
          Bill Robinson

          There should be a stack trace in the appserver log at the same time as the error there, do you see one ?

          • 2. Re: DB cleanup error Failed to delete objects for class:SComponent
            Luciano Spagnuolo

            I only found a nsh.exe.stack.dump dated 2/6/2016 (not current).  The application log also only has what the job log states. nothing more from what I see.

             

            \===AppServer Log event for DB cleanup===/

            [31 Mar 2016 08:46:09,414] [WorkItem-Thread-36] [INFO] [xxxxxxxxxxxxx:BLAdmins:] [NSHScript] Started pid 19112: "C:/Program Files/BMC Software/BladeLogic/NSH/bin/nsh" --norc -c "/C/Program Files/BMC Software/BladeLogic/NSH/tmp/syswnjpblaap01_maint_job/scripts/job__6139bb23-7d2a-4efd-968f-c6ea269a032f/master_14e57aa6-d8ff-4501-be1c-5fd6d0e820ce"

            [31 Mar 2016 11:53:21,196] [Job-Execution-4] [ERROR] [xxxxxxxxxxxxx:BLAdmins:] [NSHScript] The job 'BSA Cleanup - Database' has failed
            [31 Mar 2016 11:53:21,274] [Job-Execution-2] [ERROR] [xxxxxxxxxxxxx:BLAdmins:] [Batch] Member job BSA Cleanup - Database failed
            [31 Mar 2016 11:53:21,477] [Job-Execution-2] [INFO] [xxxxxxxxxxxxx:BLAdmins:] [Batch] Started running member job BSA Cleanup - SharedObjects
            [31 Mar 2016 11:53:21,523] [Job-Execution-3] [INFO] [xxxxxxxxxxxxx.bloomberg.com:BLAdmins:] [NSHScript] Started running the job 'BSA Cleanup - SharedObjects' with priority 'NORMAL' on application server 'syswnjpblaap01_maint_job'(38)
            [31 Mar 2016 11:53:25,423] [WorkItem-Thread-11] [INFO] [xxxxxxxxxxxxx:BLAdmins:] [NSHScript] Executing work item NSH Script Job:BSA Cleanup - SharedObjects;  on application server: syswnjpblaap01_maint_job
            [31 Mar 2016 11:53:26,656] [WorkItem-Thread-11] [INFO] [xxxxxxxxxxxxx:BLAdmins:] [NSHScript] Started pid 12976: "C:/Program Files/BMC Software/BladeLogic/NSH/bin/nsh" --norc -c "/C/Program Files/BMC Software/BladeLogic/NSH/tmp/syswnjpblaap01_maint_job/scripts/job__f31449d8-24a8-409d-a65a-289b6fd0454d/master_8398255d-04db-4e39-9e6a-f92fc785c02d"

            [31 Mar 2016 14:05:05,029] [Job-Execution-3] [INFO] [xxxxxxxxxxxxx:BLAdmins:] [NSHScript] Started running the job 'BSA Cleanup - Database' with priority 'NORMAL' on application server 'syswnjpblaap01_maint_job'(38)
            [31 Mar 2016 14:05:17,587] [WorkItem-Thread-44] [INFO] [xxxxxxxxxxxxx:BLAdmins:] [NSHScript] Executing work item NSH Script Job:BSA Cleanup - Database;  on application server: syswnjpblaap01_maint_job
            [31 Mar 2016 14:05:18,882] [WorkItem-Thread-44] [INFO] [xxxxxxxxxxxxx:BLAdmins:] [NSHScript] Started pid 17536: "C:/Program Files/BMC Software/BladeLogic/NSH/bin/nsh" --norc -c "/C/Program Files/BMC Software/BladeLogic/NSH/tmp/syswnjpblaap01_maint_job/scripts/job__fa58ab6d-8f45-4536-a4d4-d6eef4b91789/master_3f55ac6f-ef68-4d9b-ba36-275f141f0ccd"
            [31 Mar 2016 14:10:42,394] [Job-Execution-2] [ERROR] [xxxxxxxxxxxxx:BLAdmins:] [NSHScript] The job 'BSA Cleanup - Database' has failed

             

            \===From the nsh.exe.stackdump===/

            Stack trace:

            Frame     Function  Args

            00284528  6102F9AB  (00284528, 00000000, 00000000, 00000000)

            00284818  6102F9AB  (6119FE20, 00008000, 00000000, 611A1C4F)

            00285848  6100665C  (611DC60C, 00285874, 76471148, 00000130)

            00285868  61006698  (611DC60C, 611DC6BD, 00000000, 00000000)

            002858A8  61076CB0  (61203020, 00000000, 00000000, 00000000)

            002858E8  61076E47  (00285910, 002858F0, 40000000, 00010000)

            00285A78  61077FB9  (00000003, 00000000, 00000000, 00000000)

            00289AD8  610D6745  (8011FA68, 00000B01, 000001B6, 0028A048)

            00289F88  004416BB  (8011FA50, 0028A048, 00000001, 00000000)

            0028A118  00443DB1  (0028ABA0, 0000000B, 00000000, 00000012)

            0028A178  0044117D  (0028ABA0, 00000883, 00000012, 0000000B)

            0028A1D8  00441409  (0028ABA0, 000008A3, 00000012, 00000000)

            0028A248  00440902  (0028ABA0, 00003D02, 00000012, 00000000)

            0028A298  00440397  (0028ABA0, 00000001, 00000000, 00000001)

            0028A2C8  0045FE07  (0028ABA0, 00000000, 00000000, 00000000)

            0028A458  004440C1  (0028ABA0, 00000000, 00000000, 00000012)

            End of stack trace (more stack frames may be present)

             

            Lou Spagnuolo

            • 3. Re: DB cleanup error Failed to delete objects for class:SComponent
              Bill Robinson

              was the stack dump generated at the same time as the error?

               

              what about in the blcli.log in the user's home dir the appserver is running as ?

               

               

              also can you attach the entire job run log from the cleanup job ?

              • 4. Re: DB cleanup error Failed to delete objects for class:SComponent
                Luciano Spagnuolo

                No, the stack dump was dated last month.  The blcli.log was dated 3/21/2016 and the last entry was 3/21/2016 so that can't be from this job. I have attached the job run log for your review.

                 

                Thanks,

                Lou Spagnuolo

                • 5. Re: DB cleanup error Failed to delete objects for class:SComponent
                  Bill Robinson

                  Can you attach the job run log for the nsh job, not the batch job ?

                   

                  Run at 03/31/2016 08:45:43,Info,03/31/2016 08:46:05,Started running member job BSA Cleanup - Database

                  Run at 03/31/2016 08:45:43,Error,03/31/2016 11:53:21,Member job BSA Cleanup - Database failed

                  • 6. Re: DB cleanup error Failed to delete objects for class:SComponent
                    Luciano Spagnuolo

                    If this isn't it, let me know where I can find it.

                     

                     

                    Lou Spagnuolo

                    • 7. Re: DB cleanup error Failed to delete objects for class:SComponent
                      Bill Robinson

                      yes - that run has the error:

                      BSA Cleanup - Database Run at 03/31/2016 08:46:05,Info,03/31/2016 11:52:53,Failed to delete objects for class:SComponent   

                      at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteObjectsByClass(CleanupOperationImpl.java:618)   

                      at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllObjectsByClass(CleanupOperationImpl.java:494)   

                      at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllTopLevelObjects(CleanupOperationImpl.java:160)   

                      at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllObjects(CleanupOperationImpl.java:96)   

                      at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.internal_hardDeleteAllObjects(BaseCleanupServiceImpl.java:1616)   

                      at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.hardDeleteAllObjects(BaseCleanupServiceImpl.java:1558)   

                      at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.cleanupDatabase(BaseCleanupServiceImpl.java:1489)   

                      at com.bladelogic.om.infra.app.api.factory.DeleteServiceImpl.cleanupDatabase(DeleteServiceImpl.java:515)   

                      at com.bladelogic.om.infra.app.api.DeleteServiceRequestHandlerImpl.cleanupDatabase_29(DeleteServiceRequestHandlerImpl.java:1362)   

                      at com.bladelogic.om.infra.app.api.DeleteServiceRequestHandlerImpl.execute(DeleteServiceRequestHandlerImpl.java:292)   

                      at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.executeRequest(APIMethodInvocationProvider.java:54)   

                      at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.invoke(APIMethodInvocationProvider.java:39)   

                      at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invokeMethodLocally(RoutingServiceImpl.java:267)   

                      at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invoke(RoutingServiceImpl.java:210)   

                      at com.bladelogic.om.infra.app.api.manage.APIRequestManager.executeRequest(APIRequestManager.java:30)   

                      at com.bladelogic.om.infra.message.app.frame.APIMessageFrame.service(APIMessageFrame.java:113)   

                      at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processRequest(ClientWorkerThread.java:385)   

                      at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processClientRequest(ClientWorkerThread.java:282)   

                      at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.handleClientRequest(ClientWorkerThread.java:182)   

                      at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:101)   

                      at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:27)   

                      at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                       

                      the job run time does not match the appserver log snippets:

                      Start Time,03/31/2016 08:46:05

                      End Time,03/31/2016 11:53:21

                       

                      can you attach the entire appserver log ?  it looks like you filtered out the entries ?

                      • 8. Re: DB cleanup error Failed to delete objects for class:SComponent
                        Luciano Spagnuolo

                        sorry Bill, I couldn't get back to this earlier. I plan on running the job tomorrow morning fresh and get you the blcli log, appserver log, and job export for review.

                         

                        Lou Spagnuolo

                        • 9. Re: DB cleanup error Failed to delete objects for class:SComponent
                          Luciano Spagnuolo

                          Bill, attached are the logs from this mornings run.

                           

                          Thanks

                          Lou Spagnuolo

                          • 10. Re: DB cleanup error Failed to delete objects for class:SComponent
                            Bill Robinson

                            hmm - a couple things - the export of the job run got cut off:

                            BSA Cleanup - Database Run at 04/06/2016 07:58:14,Info,04/06/2016 08:33:05,[Wed Apr 06 08:33:05 EDT 2016] Hard deleting objects of class 'SACEValue'

                            BSA Cleanup - Database Run at 04/06/2016 07:58:14,Info,04/06/2016 08:33:05,[Wed Apr 06 08:33:05 EDT 2016] Delete count was : 0

                            BSA Cleanup - Database Run at 04/06/2016 07:58:14,Info,04/06/2016 08:33:0

                            is how the file ends.  though the job ends some time later.

                             

                            is the 'appservice url' setting set on the syswnjpblaap01_maint_job instance?  normally that's blank and any blcli process spun up in a nsh job on the instance will just connect to the same instance it's running in for the appservice (client connection).  if the appservice url is set to something else then the blcli will connect there, and that is what should have more info about the actual error from the blcli.  and that's the log we actually need.  if it's set to a vip, maybe just get the appserver logs from all the other appservers behind the vip from the same time as the job run and we can sort it out.

                            • 11. Re: DB cleanup error Failed to delete objects for class:SComponent
                              Bill Robinson

                              actually - nevermind - it's in the blcli logs you provided..

                               

                              06 Apr 2016 08:47:43,650 INFO [MESSAGE] Command execution failed.  [/MESSAGE]

                              com.bladelogic.om.infra.mfw.util.BlException: Failed to delete objects for class:SComponent

                                  at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteObjectsByClass(CleanupOperationImpl.java:618)

                                  at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllObjectsByClass(CleanupOperationImpl.java:494)

                                  at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllTopLevelObjects(CleanupOperationImpl.java:160)

                                  at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllObjects(CleanupOperationImpl.java:96)

                                  at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.internal_hardDeleteAllObjects(BaseCleanupServiceImpl.java:1616)

                                  at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.hardDeleteAllObjects(BaseCleanupServiceImpl.java:1558)

                                  at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.cleanupDatabase(BaseCleanupServiceImpl.java:1489)

                                  at com.bladelogic.om.infra.app.api.factory.DeleteServiceImpl.cleanupDatabase(DeleteServiceImpl.java:515)

                                  at com.bladelogic.om.infra.app.api.DeleteServiceRequestHandlerImpl.cleanupDatabase_29(DeleteServiceRequestHandlerImpl.java:1362)

                                  at com.bladelogic.om.infra.app.api.DeleteServiceRequestHandlerImpl.execute(DeleteServiceRequestHandlerImpl.java:292)

                                  at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.executeRequest(APIMethodInvocationProvider.java:54)

                                  at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.invoke(APIMethodInvocationProvider.java:39)

                                  at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invokeMethodLocally(RoutingServiceImpl.java:267)

                                  at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invoke(RoutingServiceImpl.java:210)

                                  at com.bladelogic.om.infra.app.api.manage.APIRequestManager.executeRequest(APIRequestManager.java:30)

                                  at com.bladelogic.om.infra.message.app.frame.APIMessageFrame.service(APIMessageFrame.java:113)

                                  at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processRequest(ClientWorkerThread.java:385)

                                  at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processClientRequest(ClientWorkerThread.java:282)

                                  at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.handleClientRequest(ClientWorkerThread.java:182)

                                  at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:101)

                                  at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:27)

                                  at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                              Caused by: com.bladelogic.om.infra.mfw.util.BlException: Descendents of SComponent are: [SAssetSelectorPropertyReference, SAssetSelectorExclude, SAssetSelectorAssetAttribute, SAssetSelectorInclude, SDisplayString, SComponentExceptionRuleSelector, SAssetSelector, SComponentExceptionComplianceRuleListElement, SObjectOverflowDetail, SDisplayStringSet, SDeployJobRunEvent, SJobRunEvent, SObjectOverflow, SComponentAuditTrail, SComponentComponentException, SComponentException, SACEMaskValue, SListBlValueElement, SPropertyReferenceBlValue, SPropertyReferenceProperty, SPropertyReference, SAceMaskBlValue, SACEValue, SPropertySetInstanceOverriddenValueMap, SPropertySetInstanceAuditTrail, SPropertySetInstance, SListBlValue, SPrimitiveBlValue, SEnumeratedBlValue, SRangeBlValue, SAceBlValue, SBlValue]

                                  at com.bladelogic.om.infra.app.service.cleanup.DefaultObjectDeleter.hardDeleteObjectAndChildren(DefaultObjectDeleter.java:830)

                                  at com.bladelogic.om.infra.app.service.cleanup.DefaultObjectDeleter.deleteCompositeObjects(DefaultObjectDeleter.java:122)

                                  at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl$2.doDBOperation(CleanupOperationImpl.java:591)

                                  at com.bladelogic.om.infra.app.db.TransactionalOperation.manageTransaction(TransactionalOperation.java:182)

                                  at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:136)

                                  at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteObjectsByClass(CleanupOperationImpl.java:603)

                                  ... 21 more

                              Caused by: com.bladelogic.om.infra.mfw.util.BlException: Failed to update class for deletion :SComponent query id STREAMABLE_DELETION_COMPONENT_DELETED_OBJECTS_MARKED_FOR_DELETION

                                  at com.bladelogic.om.infra.app.service.cleanup.DefaultObjectDeleter.hardDeleteObjectsForClass(DefaultObjectDeleter.java:1168)

                                  at com.bladelogic.om.infra.app.service.cleanup.DefaultObjectDeleter.hardDeleteObjectAndChildren(DefaultObjectDeleter.java:794)

                                  ... 26 more

                              Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The DELETE statement conflicted with the REFERENCE constraint "FK2758_CR_GROUP_RESULT". The conflict occurred in database "Bladelogic_V8", table "dbo.comp_rule_group_result_xref", column 'cr_group_result_id'.

                                  at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:216)

                                  at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1515)

                                  at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:404)

                                  at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:350)

                                  at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:5696)

                                  at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1715)

                                  at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:180)

                                  at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:155)

                                  at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:314)

                                  at com.bladelogic.om.infra.shared.db.BlPreparedStatement.executeUpdate(BlPreparedStatement.java:63)

                                  at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)

                                  at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)

                                  at com.bladelogic.om.infra.app.service.db.DefaultDatabaseServiceImpl.executeUpdate(DefaultDatabaseServiceImpl.java:1579)

                                  at com.bladelogic.om.infra.app.service.db.DefaultDatabaseServiceImpl.executeUpdate(DefaultDatabaseServiceImpl.java:1438)

                                  at com.bladelogic.om.infra.app.service.db.DefaultDatabaseServiceImpl.executeUpdate(DefaultDatabaseServiceImpl.java:1429)

                                  at com.bladelogic.om.infra.app.service.cleanup.DefaultObjectDeleter.hardDeleteObjectsForClass(DefaultObjectDeleter.java:1154)

                                  ... 27 more

                              • 12. Re: DB cleanup error Failed to delete objects for class:SComponent
                                Bill Robinson

                                so that constraint: FK2758_CR_GROUP_RESULT isn't listed in the script you previously ran afaict so you should open a support ticket w/ that trace above.

                                • 13. Re: DB cleanup error Failed to delete objects for class:SComponent
                                  Luciano Spagnuolo

                                  Will do, thank you for taking the time to determine cause.  Much appreciated

                                   

                                  Lou Spagnuolo