A) Useful High-Level questions to answer to help BMC Customer Support understand the problematic behavior
1) Is the performance problem confined to specific console actions (e.g. saving certain Component Templates, browsing Server Groups etc) or is it slowness across the board? If confined to specific actions, what are these actions and how severe is the performance issue? How does the performance of these operations differ from previous experience?
2) Is the performance problem consistent or does it appear to be intermittent? If intermittent, does there seem to be any pattern to the performance problem? e.g. always slower at certain times of the day or when certain jobs or Database Maintenance Tasks are running
3) Is the performance the same for members of the BLAdmins role as it is for members of other roles? Or is it specific to certain roles?
4) When did the performance problem(s) begin to occur in the BSA environment and were there any noticeable environment changes made at that time? e.g. BSA Upgrade, Database migration, Network change, changes to the virtualization infrastructure on which the BSA env resides etc.
5) Is the problem confined to specific BSA consoles or do BSA consoles installed on different servers/workstation all have the same issue?
6) Is the BSA Console installed in the same subnet as the BSA Application Server?
7) Is the BSA Console running on user desktops or on a centralized Console Server e.g. Citrix
8) Is there anti-virus software running on the BSA Console? Can that be temporarily disabled and the operation retested to determine whether this is affecting performance?
9) If the BSA environment uses an Oracle Database, is the BMC-recommended gather_schema_stats job running weekly and has it been succeeding? This can be confirmed via a DBA, the Health and Value Dashboard or running a DB Diagnostic as described in the "Verifying that statistics are current" section of this page.
10) Has the Oracle Out of Box gather statistics job been disabled as recommended in the BSA documentation? Running both the BMC-recommended gather_schema_stats job and the Out Of Box Oracle version, even at different times, has been known to cause performance problems which is why the BSA documentation recommends disabling the Out Of Box Oracle version.
11) If the BSA environment uses a SQL Server Database, is the SQL Server sp_updatestats procedure scheduled to run weekly? And does an index rebuild occur on a regular schedule?
12) Is the BSA Database Cleanup Job running regularly and successfully to completion in this environment?
13) Run the Database Diagnostic under BSA Appserver Diagnostics. This tests DB latency by running many queries and returning their response time to the expected response time.
14) Are the BSA Appservers and BSA DB in the same network location? If Virtual Machines, are the BSA Application Servers and DB running on hosts in the same data center? We have seen issues where the DB Server is running on a host on a different, albeit close by, data center. Once the DB Server host was migrated to the same data center as the BSA Application Servers, the performance issue was resolved. The VM team should be consulted to verify this.
B) What items should be gathered and provided to BMC Customer Support?
In addition to answering the above questions, the following data should be provided to BMC Customer Support to help with problem diagnosis.
1) Job Run Logs of recent BSA DB Cleanup Jobs. There may be multiple jobs configured to run on different days e.g. Daily Cleanup and Weekly Cleanup. Please gather the job run logs of each Cleanup Job. This will help BMC Support confirm what cleanup commands are being run and whether they are running to completion.
2) Database Reports:
- Oracle: Request that the site DBA provide Oracle AWR, ADDM and ASH reports for a time period during which the performance problems was reproduced. It is important that the reports are captured for a timeframe during which the problems were observed. Rather than a single report covering a 12 hour period, it is better to generate multi hourly reports which can then be correlated to Job Run logs, known times when Console performance was slow etc.
- SqlServer: Sql Server usually requires setting up a SQL Server Profiler session and capturing data, so that may require reproducing the problem again when the profiler session is running.
3) Health and Value Dashboard results zip file: The BSA Health and Value dashboard stores its results in the <File Server Storage>\dashboards\reports\latest directory. See below example. Please zip up the most-recent directory and provide the resultant zip file:
4) Debug Logging - setup the following Debug Logging on both the BSA Console Workstation and the BSA Configuration Appserver
a) Console Logs - perform the following steps on the BSA Console workstation:
- Backup the existing <BSA_Console_Dir>/NSH/br/rcp.cf file, and then replace with the attached version.
- Close the BSA Console
- Rename the Workspace directory: %APPDATA%\Bladelogic\1_2_1\
- Restart the BSA console
- After the issue has been reproduced (step c below), capture and provide the following logs from the BSA Console Workstation:
- BLWorkbenchPlugin.log: From %APPDATA%\Bladelogic\1_2_1\Workspace1\.metadata\.plugins\com.bladelogic.client.ui\
- .log file from %APPDATA%\Bladelogic\1_2_1\1_2_1\Workspace1\.metadata
- Restore the original rcp.cf with the backup after the testing.
b) Configuration Apperver Logs - perform the following steps on the BSA Application Server which the console is connecting to:
- Backup the following file: "<TSSA_INSTALL_DIR>/NSH/br/deployments/<deployments>/lob4j.properties
- Edit the following line:
You don’t have to restart the application server. After 30 to 40 seconds it will start logging debug entries.
c) Reproduce the issue and carefully note the timeframe during which the issue was reproduced.
Undo the lob4j.properties
change from "b" above after reproducing the issue
d) Generate Support Data: Run the “Generate Support Data” with the below options checked/selected for each Application Server in the environment. Note that the tool can be run once with all application server selected.This video discusses how to run the “Generate Support Data” utility.
In addition to collecting the Appserver and Console logs (including rolled over log files) selecting the below options will also provide BSA Customer Support with extensive information about the BSA environment including:
- Version information
- Number of Application Servers in the BSA Environment
- OS of Application Servers
- Database Type and Version
- Memory configuration
- Application Servers configuration settings e.g. MaxWorkItemThreads
e) Database Reports - Oracle
AWR, ADDM and ASH report for the hour duringwhich issue was reproduced.
Also, short AWR report for example 10 mins by following below link and steps
- Have the DBA setup to create manual AWR snapshot on BladeLogic database.
- Run the actions that are slow in that GUI, note what actions are slow.
- Create another manual AWR snapshot.
- Send AWR report
f) Database Reports - SQL Server
Run the actions that are slow in that GUI, note what actions are slow, then stop the DB trace and send the trace and the actions/times.
- Have the DBA setup a trace session on the BladeLogic database.
g) Database maintenance job history. Please follow below steps.
- Go to SQLServer Management Console
- Login using sysdba user
- Under Object Explorer, expand to ‘SQL Server Agent’
- Right click on ‘Jobs’ and click ‘View History’
- Select all history for last 60 days and export to excel.
h) Oracle environments - while the issue is being reproduced please capture the output of these queries. These may need to be run by the DBA as the sys users.
SELECT r.start_time [Start Time],session_ID [SPID],
CASE WHEN statement_end_offset=-1 OR statement_end_offset=0
END) [Executing SQL],
FROM sys.dm_exec_requests r
OUTER APPLY sys.dm_exec_sql_text(sql_handle) t
WHERE session_id != @@SPID -- don't show this query
AND session_id > 50 -- don't show system queries
ORDER BY r.start_time;
SELECT TOP 50
WHEN -1 THEN DATALENGTH(eqt.TEXT)
END - eqs.statement_start_offset)/2)+1)
,eqs.total_elapsed_time/1000000 AS Total_elapsed_time_Secs
,eqs.last_elapsed_time/1000000 AS Last_elapsed_time_Secs
FROM sys.dm_exec_query_stats eqs
CROSS APPLY sys.dm_exec_sql_text(eqs.sql_handle) eqt
CROSS APPLY sys.dm_exec_query_plan(eqs.plan_handle) eqp
ORDER BY eqs.total_worker_time DESC;
SELECT TOP 10
FROM sys.[dm_os_wait_stats] AS [ws]
ORDER BY [wait_time_ms] DESC;