4 Replies Latest reply on Sep 2, 2011 2:35 PM by Bill Robinson

    BLCLI cleanupDatabase / cleanupFileserver

      I've tried to cleanup my BladeLogic Database using the BLCLI command cleanupDatabase in the Delete namespace. The manual is saying that you can use the following SQL Statement to verify which database objects will be deleted:

       

      select count (*) from depot_object where is_deleted = ‘1’;

       

      It also says that after running the cleanupDatabase command, the result should be "0". For me that's not the case. After running the command the count is still 39. How can I debug this?

       

      It also appears to me that cleanupFileserver isn't working properly too. It doesn't delete any files on the fileserver.

       

      The logs of those commands do look good, no errors appearing!

       

      Does anyone have a hint how I can resolve this? We're using BMC BladeLogic Server Automation 8.1 SP2.

        • 1. BLCLI cleanupDatabase / cleanupFileserver
          Bill Robinson

          what do you see in the output of the db cleanup run?  does it show any depot objects being deleted ?

          • 2. BLCLI cleanupDatabase / cleanupFileserver

            Hello Bill,

             

            just noticed that there is an error in the log output for the depot objects. In the appserver.log the following stacktrace and oracle error is logged:

             

            [25 Aug 2011 15:20:52,780] [Client-Connections-Thread-9] [INFO] [BLAdmin:BLAdmins:149.224.10.204] [Client] Deleting top level objects of type : SDepotObject

            [25 Aug 2011 15:20:52,781] [Client-Connections-Thread-9] [INFO] [BLAdmin:BLAdmins:149.224.10.204] [Client] 40 entries were marked as deleted. We are deleting 1000 per transaction.

            [25 Aug 2011 15:20:52,781] [Client-Connections-Thread-9] [INFO] [BLAdmin:BLAdmins:149.224.10.204] [Client] Starting hard delete for class type: SDepotObject

            [25 Aug 2011 15:20:52,785] [Client-Connections-Thread-9] [INFO] [BLAdmin:BLAdmins:149.224.10.204] [Client] Delete in the range of 2000000 to 2013992

            [25 Aug 2011 15:20:52,836] [Client-Connections-Thread-9] [INFO] [BLAdmin:BLAdmins:149.224.10.204] [Client] Updated 20 file location entries for deletion

            [25 Aug 2011 15:20:52,871] [Client-Connections-Thread-9] [INFO] [BLAdmin:BLAdmins:149.224.10.204] [Client] Updated 0 file location entries for deletion

            [25 Aug 2011 15:20:55,486] [Client-Connections-Thread-9] [INFO] [BLAdmin:BLAdmins:149.224.10.204] [Client] Updating deletion status for Anonymous BlPackage Builder Job

            [25 Aug 2011 15:20:55,492] [Client-Connections-Thread-9] [INFO] [BLAdmin:BLAdmins:149.224.10.204] [Client] Updated 0 Anonymous BlPackage Builder Jobs.

            [25 Aug 2011 15:20:56,969] [Client-Connections-Thread-9] [INFO] [BLAdmin:BLAdmins:149.224.10.204] [Client] SQLException: Sql State (23000) Vendor Code (2292)

            [25 Aug 2011 15:20:57,163] [Client-Connections-Thread-9] [WARN] [BLAdmin:BLAdmins:149.224.10.204] [Client]

            com.bladelogic.om.infra.mfw.util.BlException: Failed to delete objects for class:SDepotObject

                at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteObjectsByClass(CleanupOperationImpl.java:761)

                at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllObjectsByClass(CleanupOperationImpl.java:664)

                at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllTopLevelObjects(CleanupOperationImpl.java:160)

                at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteAllObjects(CleanupOperationImpl.java:97)

                at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.internal_hardDeleteAllObjects(BaseCleanupServiceImpl.java:1324)

                at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.hardDeleteAllObjects(BaseCleanupServiceImpl.java:1270)

                at com.bladelogic.om.infra.app.service.cleanup.BaseCleanupServiceImpl.cleanupDatabase(BaseCleanupServiceImpl.java:1203)

                at com.bladelogic.om.infra.app.api.factory.DeleteServiceImpl.cleanupDatabase(DeleteServiceImpl.java:336)

                at com.bladelogic.om.infra.app.api.factory.DeleteServiceImpl.cleanupDatabase(DeleteServiceImpl.java:328)

                at com.bladelogic.om.infra.app.api.DeleteServiceRequestHandlerImpl.cleanupDatabase_17(DeleteServiceRequestHandlerImpl.java:781)

                at com.bladelogic.om.infra.app.api.DeleteServiceRequestHandlerImpl.execute(DeleteServiceRequestHandlerImpl.java:211)

                at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.executeRequest(APIMethodInvocationProvider.java:57)

                at com.bladelogic.om.infra.app.api.manage.APIMethodInvocationProvider.invoke(APIMethodInvocationProvider.java:42)

                at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invokeMethodLocally(RoutingServiceImpl.java:319)

                at com.bladelogic.om.infra.app.service.routing.RoutingServiceImpl.invoke(RoutingServiceImpl.java:246)

                at com.bladelogic.om.infra.app.api.manage.APIRequestManager.executeRequest(APIRequestManager.java:30)

                at com.bladelogic.om.infra.message.app.frame.APIMessageFrame.service(APIMessageFrame.java:101)

                at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processRequest(ClientWorkerThread.java:369)

                at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.processClientRequest(ClientWorkerThread.java:279)

                at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.handleClientRequest(ClientWorkerThread.java:179)

                at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:98)

                at com.bladelogic.om.infra.mfw.net.ClientWorkerThread.execute(ClientWorkerThread.java:24)

                at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

            Caused by: com.bladelogic.om.infra.mfw.util.BlException: Descendents of SDepotObject are: [SDnsServers, SDefaultGateway, SIpAddress, SDisplayString, SVGPZoneResourceControl, SVGPLparVirtualEthAdp, SVGPCitrixXenNetwork, SVGPRHELKVMNetworkAddress, SSystemPackageNimDiskStanza, SVGPCitrixXenDisk, SPropertySetClassAuditTrail, SVGPLparIOSlot, SPropertySetClassDefaultValueMap, SProperty, SVGPZoneDevice, SVGPLparLHEALogicalPort, SSystemPackageJumpstartAttrs, SVGPZoneFileSystem, SVGPZoneAttribute, SVGPZoneInheritedPkgDir, SVGPRHELKVMStorage, SVGPLparVirtualScsiAdp, SVGPLparVirtualSerialAdp, SSystemPackageNimScript, SSystemPackageNimBosinstattr, SSystemPackageIgniteScript, SVGPCitrixXen, SVGPLparVirtualFCAdp, SDepotSWSuppFileLoc, SVGPZoneDataSet, SPropertySetClassSuperClassList, SVGPZoneNetworkAddress, SVGPackagePropRef, SSystemPackageIgnite, SVGPackageDiskInfo, SSystemPackageScript, SVGPackageVMWarePropRef, SNTBootOrder, SSystemPackageLog, SSystemPackageJumpstart, SSystemPackageNim, SNTVMNicConfig, SDepotSoftwarePart, SNTPolicy, SDepotSoftwareHotfix, SSysPackPreInstlScripts, SDepotSoftwareErrata, SVirtualGuestPackageRHELKVM, SListDataType, SSystemPackageAuditTrail, SVGPackageVMNicInfo, SVirtualGuestPackage, SVGPackageCustomConfig, SNTLanNic, SDepotSoftwareSoltcluster, SSystemPackageOsCompIds, SSystemPackagePropRef, SDepotSoftwareSolpatch, SDepotSWSuppFile, SDepotSoftwareAixFileset, SDepotSoftwareAixContainer, SVirtualGuestPackageLpar, SPrimitiveDataType, SEnumeratedDataType, SDepotSoftwareRpm, SNTSanNic, SRangeDataType, SPartitions, SDisplayStringSet, SSystemPackageNetworkAdapter, SNetworkAdapters, SVirtualGuestPackageZone, SDepotSoftwareWinBulletin, SSystemPackageLocale, SDepotSoftwarePropertyValue, SDepotSoftware, SDepotObjectLocation, SBLPkgDepotObjectSoftlink, SSystemPackage, SVirtualGuestPackageBase, SNimbusTemplate, SBlPackageActionRef, SDepotObjectAuditTrail, SBlPackagePropertyReference, SUcsStorageEntry, SNimbusMacRange, SNimbusUuidRange, SBlPackageAttributeRef, SDepotObjectOptionValue, SDepotObjectRef, SBlPackageClassRef, SNshParameter, SACEMaskValue, SListBlValueElement, SPropertyReferenceBlValue, SPropertyReferenceProperty, SPropertyReference, SAceMaskBlValue, SACEValue, SPropertySetInstanceOverriddenValueMap, SPropertySetInstanceAuditTrail, SPropertySetInstance, SPropertySetClass, SListBlValue, SPrimitiveBlValue, SEnumeratedBlValue, SRangeBlValue, SAceBlValue, SBlValue, SDataType]

                at com.bladelogic.om.infra.app.service.cleanup.DefaultObjectDeleter.hardDeleteObjectAndChildren(DefaultObjectDeleter.java:772)

                at com.bladelogic.om.infra.app.service.cleanup.DefaultObjectDeleter.deleteCompositeObjects(DefaultObjectDeleter.java:118)

                at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl$2.doDBOperation(CleanupOperationImpl.java:734)

                at com.bladelogic.om.infra.app.db.TransactionalOperation.manageTransaction(TransactionalOperation.java:174)

                at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:136)

                at com.bladelogic.om.infra.app.service.cleanup.CleanupOperationImpl.hardDeleteObjectsByClass(CleanupOperationImpl.java:748)

                ... 22 more

            Caused by: com.bladelogic.om.infra.mfw.util.BlException: Failed to update class for deletion :SVGPackageCustomConfig query id STREAMABLE_DELETION_VGPACKAGE_CUSTOM_CONFIG_DELETED_OBJECTS_MARKED_FOR_DELETION

                at com.bladelogic.om.infra.app.service.cleanup.DefaultObjectDeleter.hardDeleteObjectsForClass(DefaultObjectDeleter.java:1061)

                at com.bladelogic.om.infra.app.service.cleanup.DefaultObjectDeleter.hardDeleteObjectsForClass(DefaultObjectDeleter.java:994)

                at com.bladelogic.om.infra.app.service.cleanup.DefaultObjectDeleter.hardDeleteChildObject(DefaultObjectDeleter.java:847)

                at com.bladelogic.om.infra.app.service.cleanup.DefaultObjectDeleter.hardDeleteObjectAndChildren(DefaultObjectDeleter.java:707)

                ... 27 more

            Caused by: java.sql.SQLException: ORA-02292: integrity constraint (BLADELOGIC.FK1_VIRTUAL_GUEST_PACKAGE) violated - child record found

             

                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)

                at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)

                at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)

                at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:745)

                at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216)

                at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:966)

                at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1170)

                at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3339)

                at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3423)

                at com.bladelogic.om.infra.shared.db.BlPreparedStatement.executeUpdate(BlPreparedStatement.java:63)

                at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)

                at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)

                at com.bladelogic.om.infra.app.service.db.DefaultDatabaseServiceImpl.executeUpdate(DefaultDatabaseServiceImpl.java:1358)

                at com.bladelogic.om.infra.app.service.db.DefaultDatabaseServiceImpl.executeUpdate(DefaultDatabaseServiceImpl.java:1280)

                at com.bladelogic.om.infra.app.service.db.DefaultDatabaseServiceImpl.executeUpdate(DefaultDatabaseServiceImpl.java:1271)

                at com.bladelogic.om.infra.app.service.cleanup.DefaultObjectDeleter.hardDeleteObjectsForClass(DefaultObjectDeleter.java:1047)

                ... 30 more

             

            Any clue what happened here with the Virtual Guest Packages?

            • 3. Re: BLCLI cleanupDatabase / cleanupFileserver

              Bill,

               

              is this a known issue? The environment where this happened is not really much used, just for testing some functionalitys. Recently we did some practicing with Virtual Guest Packages which were also deleted.

               

              I've fixed the issue manually by cleaning up all the (still existing) foreign key relationships of the datasets in depot_objects table in the database. I wouldn't have done this on a production environment, though.

               

              I had to manually delete related datasets in the following tables: depot_object_location, bl_pkg_property_reference, depot_object_option_value, bl_package_class_ref, bl_package_attribute_ref, blpkg_depot_object_softlink, vgpackage_vmnic_info, virtual_guest_package

               

              Addition: Unfortunately I can't reproduce this behaviour.

              • 4. Re: BLCLI cleanupDatabase / cleanupFileserver
                Bill Robinson

                Not sure if this is a known issue or not.  I would follow up w/ support on this.  I’d advise against making changes to the db w/o the assistance of bmc support.