Skip navigation

Server Automation

6 Posts authored by: John O'Toole Moderator
Share:|

The BMC Customer Support team wish to notify customers of an issue affecting the download of Redhat 4, 5 and 6 data from both the BSA Offline Downloader and a BSA Online Catalog Update job.

 

The issue is described in full in this Knowledge Base article along with instructions on how to obtain and apply the fix for your BSA Environment.

 

Redhat 7 downloads/updates are not affected by this issue and the issue is not confined to specific versions of BSA.

Share:|

A couple of critical broken authentication vulnerabilities were disclosed to BMC by ERNW Gmbh (an independent research company) and they will be disclosed publicly at the Troopers 2016 conference in Heidelberg on March 16th, 2016. These vulnerabilities allow remote unauthorized access to Linux/Unix RSCD agents using the agents’ RPC API. Windows agents are not affected. For more-detailed information, please see the following Flash Notification.

 

The security of our solutions is of the utmost importance to BMC. Once we were made aware of the issue, we investigated, developed a fix, and began the process of notifying customers.

 

As part of the notification process, BMC has published a Knowledge Article with patching information as well as general recommendations for reducing the likelihood of successful exploitation. We strongly recommend that BSA customers follow the instructions within the article. Should you have questions or need assistance, please contact BMC Customer Support.

Share:|

BMC Server Automation (BSA) 8.5.1 Patch 5 officially released on 04/17/2015. This post presents a video which walks the user through the steps required to upgrade a BSA 8.5.x environment to BSA 8.5.1 Patch 5.

 

The video follows the steps listed in the Patch 5 upgrade documentation and demonstrates the steps on both Linux and Windows Application Servers as well as Oracle and SQL Server Databases.e

 

Note that the youtube video also contains a Table of Contents so the viewer can skip to specific sections e.g. Env Shutdown, DB Upgrade, Postmigration etc.


 

 

Table of Contents: (visit YouTube link for the TOC to be clickable)

 

00:00 - Introduction

01:26 - Documentation

02:31 - 1 Shutdown

03:39 - 1a Linux/Unix

04:45 - 1b Windows

05:28 - 2 Backup

06:03 - 3 AppServer Upgrade

06:43 - 3a Linux/Unix

09:26 - 3b  Windows

09:56 - 4 Database Upgrade

10:28 - 4a Oracle

12:38 - 4b  SQL Server

13:43 - 5 Postmigration

16:03 - 6 Appserver Agent

17:20 - 7 FileServer Agent

21:24 - 8 AppServer Restart

22:55 - 9 PXE Server

23:13 - 10 BSA Console

26:44 - 11 Additional AppServers

27:38 - 12 Target Agents

28:34 - Conclusion


The top-level documentation for BSA 8.5.1 Patch 5 can be found here.


I hope this video provides the information you need to plan and execute your BSA 8.5.x to 8.5.x Patch 5 upgrade.
If you have any questions or comments on this topic, please feel free to use the comments section below. Also, please take a moment to rate the article so we can continue to monitor which topics and formats users find most useful.

To see more like this, see BMC BladeLogic Support Blogs

Share:|

At BMC Engage, IT management professionals connect with peers, BMC product experts, and industry insiders to share insights and best practices for BMC solutions.

Do you understand the challenges and see the future of the delivery of digital services?

Do you have a viewpoint on how IT can help drive business transformation?

Do you have a unique solution or technical expertise to share?

Do you have an innovation story to tell?


Be a part of BMC Engage 2015, September 8-11 in Las Vegas, and help us bring IT to life.

Customers, partners, and IT and product experts are needed to deliver an agenda that’s topical and valuable to other BMC users.

We are now accepting submissions for technical, business, and hands-on labs breakout sessions. Submit your session for the Cloud Management & DevOps, BladeLogic Data Center Automation and Security and Compliance tracks.

 
The submission window won’t stay open long – submit now. Deadline is February 28, 2015.

Share:|

In an earlier BladeLogic Server Automation (BSA) Support Blog, we looked at how to avoid database referential integrity errors when running the BSA database cleanup commands. This post will look at how another issue sometimes encountered during BSA database cleanup, table-locking and the job hangs it can cause, can be avoided with some recent enhancements to the list of BSA database indexes.

 

 

Problem Summary

 

It is generally advisable to try to schedule the BSA database cleanup commands to run during off-peak hours as they are database-intensive tasks and have the potential to affect the performance of other BSA jobs running at the same time. However, in a real-world environment, it is, of course, seldom possible for the cleanup jobs to run completely in isolation and not overlap with other BSA jobs. While a slight hit on the performance of other jobs during the execution of BSA DB Cleanup may be acceptable, what we have occasionally encountered in the past is BSA jobs effectively becoming hung/blocked while the BSA database cleanup commands are running. The jobs may wait in the "Waiting to Run" state or they may reach a certain point in their execution before seeming to pause with no obvious progress being displayed in the "Tasks in Progress" pane of the BSA console. Since the cleanup commands may take hours to complete, a restart of the BSA Application Servers is often required to resolve this issue and allow the jobs to execute as normal. The problem may also present itself as hung login attempts to the BSA console.

 

Root Cause


In most cases, the BSA database cleanup will first delete data from the child tables and then from the parent tables. In certain cases, where the foreign key definition has been changed to "on-delete cascade" in order to avoid foreign key violations, the BSA database cleanup commands will trigger cascading deletes which means that if a record in the parent table is deleted, then the corresponding records in the child table will also automatically be deleted.

 

If the foreign key on the child table is not indexed, undesirable and expensive table scans may occur on the child table which can cause lengthy table-level locking on the parent table. This topic is explained very well in the following Oracle and SQL Server articles:

 

https://asktom.oracle.com/pls/asktom/f?p=100:11:::::P11_QUESTION_ID:292016138754

http://blogs.msdn.com/b/dbrowne/archive/2010/03/29/indexes-supporting-foreign-keys.aspx

 

Solution

 

Similar to the solution for the Referential Integrity errors, the solution here is to run a SQL script on the BSA DB/schema to create new indexes on the foreign keys which the BSA Development team have deemed to be appropriate. As discussed in the above articles, not every foreign key will have an index added. The SQL script to run depends on your version of BSA and whether you are running Oracle or SQL Server.

 

 

Obtaining the Solution

 

BSA 8.2.X:

8.2.X Oracle: ftp://ftp.bmc.com/pub/BSACleanupIndexes/82/add82xIdx_oracle.rar

8.2.X SQL Server: ftp://ftp.bmc.com/pub/BSACleanupIndexes/82/add82xIdx_sqlserver.rar

 

BSA 8.3.X:

8.3.X Oracle: ftp://ftp.bmc.com/pub/BSACleanupIndexes/83/add83xIdx_oracle.rar

8.3.X SQL Server: ftp://ftp.bmc.com/pub/BSACleanupIndexes/83/add83xIdx_sqlserver.rar

 

*BSA 8.5.X:

8.5 Oracle: ftp://ftp.bmc.com/pub/BSACleanupIndexes/85/add85xIdx_oracle.rar

8.5 SQL Server: ftp://ftp.bmc.com/pub/BSACleanupIndexes/85/add85xIdx_sqlserver.rar

 

*Note - Although many of the new indexes were added "out of the box" in BSA 8.5.1, we do update the scripts occasionally and add new indexes so it is still a good idea to run the latest 8.5.X version of the script even if running BSA 8.5.1.


The above .rar files are up-to-date as of 10/21/2014 and will be updated should the BSA development team add any new indexes in the future.

 

Each .rar file contains the SQL Script to be run and a readme.txt file. The Add Index SQL scripts do not require the BSA Application Servers to be down while they are running though they can take over an hour to complete depending on the size of the environment.

 

Verification


Once the appropriate script has been executed on the BSA DB/Schema, the BSA database cleanup commands should be rerun and monitored. In many cases, a performance improvement may be observed due to the addition of the indexes and the likelihood of table-locking resulting in BSA jobs being blocked should be greatly reduced. If BSA job hangs are still observed during the execution of the cleanup commands, further analysis will be required and the following should be captured and provided when opening a ticket with BMC Support:

 

Oracle DB:

  1. BSA Job Run Log for the Cleanup NSH Script Job.
  2. BSA Job Run Logs for the hung jobs.
  3. Export of Appserver Details from the BSA UI at the time of the hang. (Configuration - Infrastructure Management - Export Detail Report)
  4. Oracle AWR, ADDM and ASH reports for the timeframe from when cleanup was kicked off until the job hangs were observed. These can be captured after the appservers have been restarted and do not need to be captured at the time of the hang. For AWR reports, it is preferable to gather hourly reports for the entire duration of the problem rather than a single report for the entire duration.
  5. Output of the appropriate attached SQL Query run as the system or sys user. These must be run at the time of the hang:
    • GetOracleLocks.txt for non-RAC environments
    • GetOracleRACLocks.txt for Oracle RAC environments.

SQL Server DB

  1. BSA Job Run Log for the Cleanup NSH Script Job.
  2. BSA Job Run Logs for the hung jobs.
  3. Export of Appserver Details from the BSA UI at the time of the hang. (Configuration - Infrastructure Management - Export Detail Report)
  4. The output of the following SQL Query run on the BladeLogic DB. These must be run at the time of the hang:
    • sp_whoisactive @get_locks = 1
    • Note: The sp_whoisactive procedure may need to be installed by the DBA
  5. From Microsoft SQL Server Management Studio, click on "Activity Monitor" from the toolbar. Click on the "Recent Expensive Queries" Panel.  Copy the top 5 longest executing queries from the grid and paste to a file.

 

 

Additional recommendations for avoiding blocking during cleanup:

 

  1. In Oracle environments, ensure that the BMC-recommended gather_schema_stats routine is being run frequently in place of the Oracle out of the box procedure. See KA288648 for details.
  2. When performing the initial run of BSA database cleanup, the cleanupHistoricalData commands should all be run to completion before the cleanupDatabase or hardDeleteAllSharedObjects commands are attempted. The cleanupHistoricalData commands are db-side stored procedures and are better-equipped to handle very large data loads. Allowing the cleanupHistoricalData commands to reduce the size of the tables as much as possible first will greatly reduce the workload to be performed by cleanupDatabase and hardDeleteAllSharedObjects.
  3. In BSA 8.3.2, 8.3.3 and 8.5.X, an offline database cleanup utility was added to the product. As the name suggests, this utility requires the BSA Application Servers to be down but it is able to perform very efficient cleanup on very large tables which may take the regular online cleanup many iterations to get through. If running cleanup for the first time on a large environment, or the first time after a lengthy gap, consider first running the offline database cleanup utility to efficiently reduce the size of some of the largest tables.

    BSA 8.3.2 documentation for the offline Database Maintenance Wizard
    BSA 8.3.3 documentation for the offline Database Maintenance Wizard
    BSA 8.5 documentation for the offline Database Maintenance Wizard
  4. The Cleaning up the BMC Server Automation database BMC Contributor documentation page is a very useful source for various BSA db cleanup tips and trips and is updated frequently.

I hope this post provides the information you need to help prevent db table-locking issues during the BSA DB cleanup commands. If you have any questions or comments on this topic, please feel free to use the comments section below. Also, please take a moment to rate the article so we can continue to monitor which topics and formats users find most useful.

 

To see more like this, see BMC BladeLogic Pulse Blog Posts


Join the Customer Support Community and give us feedback if our efforts in Communities are helpful, and how we can better serve you.
Connect with BMC Support Team over Chat sessions

Share:|

This post will share information regarding recent improvements in BladeLogic Server Automation (BSA) designed to greatly reduce the likelihood of referential integrity errors occurring during the cleanupDatabase command. We will discuss the referential integrity errors and options for applying the solution to your environment depending on the version of BSA you have installed.


Problem Summary

 

Many BSA customers who run the cleanupDatabase CLI command will be familiar with referential integrity errors, such as the below examples, appearing in their appserver logs.

 

Oracle:

 

The reference SUpdateServerPropertyJob.SUpdateServerPropertyJob could not be resolved. Cause: java.sql.SQLIntegrityConstraintViolationException: ORA-02292: integrity constraint (BLADELOGIC.FK458_JOB) violated - child record found

 

Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-02292: integrity constraint (BLADELOGIC.FK674_JOB_RUN) violated - child record found

 

Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-02292: integrity constraint (BLADELOGIC.FK499_JOB) violated - child record found

 

SQL Server:

 

Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The DELETE statement conflicted with the REFERENCE constraint "FK2403_JOB". The conflict occurred in database "BladeLogic", table "dbo.batch_job_job".

 

Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The DELETE statement conflicted with the REFERENCE constraint "FK1_COMP_ENTRY_RESULT". The conflict occurred in database "bladelogic", table "dbo.comp_entry_result", column 'compliance_rule_result_id'.

 

Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The DELETE statement conflicted with the REFERENCE constraint "FK1_RSLVDSYSPKCDC_JOB_RESULT". The conflict occurred in database "bladelogic", table "dbo.prov_job_rslvd_sys_pkg_cdc", column 'result_id'.

 

Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The DELETE statement conflicted with the REFERENCE constraint "FK767_COMPONENT". The conflict occurred in database "bladelogic",

 

 

 

The name of the constraint listed in the error message can vary greatly but the error message will always contain one of the following depending on the DB Type:

 

Oracle: ORA-02292: integrity constraint violated

SQL Server: The DELETE statement conflicted with the REFERENCE constraint

 

 

Root Cause

 

The root cause of these referential integrity errors during cleanup can usually be attributed to order of deletion issues. There are occasional, and very specific, cases where errors such as these can occur during cleanupDatabase due to other reasons ( e.g. BSA job activity while cleanup is running, Smart Group issue KA404005) but the vast majority of these errors occur due to an attempt to delete a record from Table A which is still referenced by records in Table B.

 

Solution

 

The solution to these issues is to modify the foreign key constraints which encounter this issue most frequently to perform cascading deletes. A foreign key with "On Delete Cascade" means that if a record in the parent table is deleted, then the corresponding records in the child table will automatically be deleted. BMC now provides SQL scripts, for both Oracle and SQL Server, which modify these constraints accordingly to help avoid these issues during cleanupDatabase.

 

 

Obtaining the Solution

 

BSA 8.5:

BSA 8.5.1 contains most of the constraint modifications out of the box. Some additional constraints were modified after BSA 8.5.1 was released. So BSA 8.5.1 customers should still run the appropriate 85 SQL script for their database platform to pick up the complete list of constraint modifications. The solution can be applied by downloading the appropriate zip file for your database type and following the instructions in the readme.txt file. The steps involve running a single SQL script on the bladelogic schema/db however this does require the BSA Appservers to be stopped while the script is being run. The script does not typically take more than a few minutes to complete.

 

8.5.X Oracle: ftp://ftp.bmc.com/pub/BSACleanupConstraints/85/alter85xConstraints_oracle.rar

 

8.5.X SQL Server: ftp://ftp.bmc.com/pub/BSACleanupConstraints/85/alter85xConstraints_sqlserver.rar

 

 

 

BSA 8.3.3:

BSA 8.3.3 contains a subset of the constraint modifications out of the box. See the "Complete List of Modified Constraints" section below for specifics. However, some additional constraints were modified after BSA 8.3.3 was released. So BSA 8.3.3 customers should still run the appropriate 83 SQL script for their database platform to pick up the complete list of constraint modifications. The solution can be applied by downloading the appropriate zip file for your database type and following the instructions in the readme.txt file. The steps involve running a single SQL script on the bladelogic schema/db however this does require the BSA Appservers to be stopped while the script is being run. The script does not typically take more than a few minutes to complete.

 

8.3.X Oracle: ftp://ftp.bmc.com/pub/BSACleanupConstraints/83/alter83xConstraints_oracle.rar

8.3.X SQL Server: ftp://ftp.bmc.com/pub/BSACleanupConstraints/83/alter83xConstraints_sqlserver.rar

 

 

 

BSA 8.3.0, 8.3.1 or 8.3.2: BSA 8.3 versions prior to 8.3.3 will not have any of the modified constraint definitions out of the box. The solution can be applied by downloading the appropriate zip file for your database type and following the instructions in the readme.txt file. The steps involve running a single SQL script on the bladelogic schema/db however this does require the BSA Appservers to be stopped while the script is being run. The script does not typically take more than a few minutes to complete.

 

8.3.X Oracle: ftp://ftp.bmc.com/pub/BSACleanupConstraints/83/alter83xConstraints_oracle.rar

8.3.X SQL Server: ftp://ftp.bmc.com/pub/BSACleanupConstraints/83/alter83xConstraints_sqlserver.rar

 

 

BSA 8.2.X: BSA 8.2 will also not have any of the required constraint modifications out of the box. The solution can be applied by downloading the appropriate zip file for your database type and following the instructions in the readme.txt file. The steps involve running a single SQL script on the bladelogic schema/db however this does require the BSA Appservers to be stopped while the script is being run. The script does not typically take more than a few minutes to complete.

 

8.2.X Oracle: ftp://ftp.bmc.com/pub/BSACleanupConstraints/82/alter82xConstraints_oracle.rar

8.2.X SQL Server: ftp://ftp.bmc.com/pub/BSACleanupConstraints/82/alter82xConstraints_sqlserver.rar

 

 

Complete List of Modified Constraints:

 

The following 26 constraints are fixed "Out of the Box" in BSA 8.3.3:

 

FK458_JOB

FK674_JOB_RUN

FK683_JOB_RUN

FK1_COMP_ENTRY_RESULT

FK1_COMP_ENTRY_RESULT_DETAIL

FK653_COMPONENT

FK2449_AUDIT_OBJECT

FK1034_AUDIT_OBJECT_PART

FK1025_AUDIT_OBJECT_PART

FK1014_AUDIT_OBJECT_PART

FK2654_JOB

FK2658_JOB

FK2328_JOB_RUN

FK2_PATCHING_JOB_RUN_PART

FK269_COMPONENT_EXCEPTION

FK2403_JOB

FKX3_DEPOT_OBJECT_LOCATION

FK2262_JOB_RUN -> Oracle Only

FK2119_JOB_RESULT

FK1_JOB_RUN_EVENT

FK2652_SCHEDULE

FK767_COMPONENT

FK648_COMPONENT

FK248_COMPONENT

FK51_BL_VALUE

FK1346_BLGROUP

 

The following additional constraints are not fixed out of the box in BSA 8.3.3 but are fixed both in the standalone SQL scripts for 8.2.X and 8.3.X and also fixed out of the box in 8.5 SP1:

FK499_JOB

FK1_VGPACKAGE_VMNIC_INFO

FK1_RSLVDSYSPKCDC_JOB_RESULT

FK459_JOB

FK249_CR_GROUP_RESULT

FK1014_AUDITRSLT_2_RSLT

FK1450_SCHEDULE

FK305_DEPOT_OBJECT
FK1_JOB_RUN*
FK1013_AUDIT_OBJECT_PART*

 

* - Added to package on 06/10/2014. If previous version of the package has been applied, the latest package can be downloaded and rerun without any issue to pick up the two additional indexes.

 

Conclusion

 

Applying these constraint modifications to your BSA Database is important to prevent referential integrity errors during the BSA cleanupDatabase process. A referential integrity error itself does not cause any DB problems but it does prevent cleanup from running to completion therefore preventing DB and FileServer space from being reclaimed.
If a referential integrity error matching one of the constraints listed above is encountered, the appropriate fix can be downloaded and applied as described above (note: does require BSA Appservers to be stopped). Likewise, these can be applied proactively to avoid such errors in the future. Applying these scripts to BSA 8.2.X or 8.3.X will not affect the upgrade to either 8.3.3 or 8.5.X and no steps need to be taken differently during those upgrades as a result of previously running these scripts.
If a constraint violation is encountered during cleanup with a constraint which is not listed in the above list, then please let us know via a support ticket. The zip files in the FTP locations referenced above will be updated should any modifications be released by BMC.

 

Please also see the following Pulse article on a related topic - how to avoid database locking when running BSA DB cleanup. After applying the constraint medications described in this article, it is important to also apply the new indexes described in this secondary article.

 

I hope this post provides the information you need to prevent and resolve referential integrity errors during the BSA cleanupDatabse command. If you have any questions or comments on this topic, please feel free to use the comments section below. Also, please take a moment to rate the article so we can continue to monitor which topics and formats users find most useful.


Join the Customer Support Community and give us feedback if our efforts in Communities are helpful, and how we can better serve you.
Connect with BMC Support Team over Chat sessions

To see more like this, see BMC BladeLogic Support Blogs

Filter Blog

By date:
By tag: