Share This:

In this post we are going to talk about end to end integration of BMC TrueSight with Change Management.

 

The integration of BMC TrueSight with Change Management enables you to create an automated blackout policy and schedule downtime for a Configuration Item (CI) in BMC TrueSight

When BMC TrueSight receives a change or a task request for scheduled maintenance, the CI associated with it goes into blackout state, as a result preventing the generation of false alerts, alarms, and data for the CI.

 

There are many advantages of integrating BMC TrueSight with Change Management, some of which include

  • Enables BMC TrueSight to receive notifications of change requests configured on specific CIs from BMC Change Management.
  • Enables the BMC TrueSight Presentation Server console and TrueSight Operations Console to display the attributes of the change requests that are configured in BMC Change Management. You can view these attributes in the Details pane of the change event in the BMC TrueSight Presentation Server console and BMC TrueSight Operations Console.
  • Creates an automated blackout policy in the BMC TrueSight cell for the CI that you configure for maintenance in BMC Change Management. The Blackout policy make sure that no false alarms are generated for the CI that is scheduled for maintenance.

HOW DOES IT WORK

To understand how the integration works, consider the following scenario

Jim is a network operations Center (NOC) operator at a company that has multiple monitoring tools for its IT environment.

The monitoring tools generate events for all changes that occur in the environment.
Due to lack of communication between users who introduce these changes and the NOC operators, the NOC operators end up troubleshooting events generated by known change requests, wasting time and resources.


The integration of BMC TrueSight with BMC Change Management allows Jim to reduce the number of events generated by changes introduced in the environment.

The integration allows him to automatically block and notify users of any change including:

For example, Roger, the distributed server operations user, requests Rahul, the Change Coordinator, to schedule a security patch update for selected servers.
Jim has configured an automated CI blackout policy for change requests received through BMC Remedy ITSM.
This policy:

  • Marks all events occurring the in scheduled time period for that CI to blackout
  • Automatically schedules downtime of TSIM Schedule downtime in TSIM for change requests received through ITSM.
  • Stops data collection
  • Stops Alarms and abnormalities generation
  • Sets CI to maintenance mode

Becky, the Event and Impact Management – Service Model administrator, requests that the status of CIs changed to Scheduled Downtime in the Service Model.
For this use case, the following procedure is followed:

  • Roger contacts Rahul about scheduling maintenance of the selected group of servers to apply the latest security patches.
  • Rahul schedules the server for the maintenance 2 weeks from the date of the request for 5 hours.
  • Jim receives an event in the BMC TrueSight Presentation Server and TrueSight Infrastructure Management GUI notifying him of the upcoming maintenance. He also receives a notification that the Cell Blackout Policy and TSIM downtime schedule was created.
  • When the date of maintenance arrives, the event status is modified in the BMC TrueSight Presentation Server and BMC TrueSight Infrastructure Management GUI to notify him of that fact that now the designated servers are in down for scheduled maintenance. He also sees that all events received from the CIs that represent the servers that are now in maintenance, are set to blackout state.
  • Becky sees that the CIs status representing servers is changed to Blackout in the Service Model.
  • When at the end of the scheduled down period, the Change request is marked as completed, Jim observes that the event status is modified to Completed and validates that the Blackout Policy and TSIM Schedule downtime are removed. Events received for the CIs reflect the actual state of the CIs. Jim also receives Event Status Closed for the event that was originally generated from the change request.
  • Becky observes that CIs status is back to normal state in the Service Model

ARCHITECTURE

Communicating between BMC Change Management and Operations Management

BMC Change Management sends change and task requests to Operations Management through RESTful web services.

Rest Webservice EventProvider component:

This component will receive the event notifications from the ITSM change management in JSON format over rest webservice, and this component convert this event notification into the  event and send it to Jserver (event processor component in TrueSight)

EventProcessor:

This component receives the event from the webservices Eventprovider component and do any pre-processing required on the event then send the event to cell.

Event Selector:

This Component comes into picture once event is received by Gateway Event handler, Event selector is a thread pool, its purpose is to receives the event from gatewayEvent handler and if this event is a Change Event or some other event of concern, event Selector will send this event to RuleConfigurator asynchronously.

RuleConfigurator:

This Component itself have two major components, RuleConfiguratorConsumer, and RuleConfiguratorWorker.
RuleConfiguratorConsumer's are controlled by RuleConfiguratorConsumerManager, which gets started while the initialization of JSERVER.

This RuleConfiguratorManager starts the different RuleConfiguratorConsumer's thread, which in itself feeds the data to RuleConfigurator's i.e PolicyConfigurator and ScheduleConfigurator. PolicyConfigurator and Scheduleconfigurator uses BlackOutPolicyCrud and ScheduleConfiguratorCrud API's to create policy and schedule down time respectively.

ITSM End

 

 

From ITSM end, The CAI RESTful plug-in establishes communication between the Change Management application and BMC TrueSight. It gets data from CAI and pushes data to a RESTful web service exposed by BMC TrueSight. This plug-in refers to the configuration data for communicating with RESTful web services.

 

 

Subscription call from True Sight

 

 

 

Pre-requisite before activating the Change Management Integration with TrueSight.

 

  • Make sure that the required CMDB extensions are installed on box where the CMDB is installed. Perform following steps:
  • Perform search on SHARE:Application_Interface form where ‘Providing Application Name = BMC_PN: Class and Attribute Definitions for Service Impact Management’.
  • Make sure that the status of the chgbpm.jar file is set as Installed = 'Yes'
  • Make sure that forms starting PN: are present (e.g. PN:Published Data etc.).
  • Ensure ITSM Change management related properties are enabled in %BMC_TRUESIGHT_HOME%\pw\pronto\conf\.pronet.conf file

     -Ensure that the flag pronet.jserver.ruleconfigurator.enable in the pronet.conf file is enabled.

     -Ensure that the flag pronet.jserver.ruleconfigurator.eventcollectiondowntime.enable in the pronet.conf file isset to true.

     -Ensure that the flag pronet.jserver.ruleconfigurator.datacollectiondowntime.enable in the pronet.conf file is set to true.

  • Make sure BPPM/TSIM Server is properly integrated with CMDB

Verify  if BPPM/TSIM Server is properly integrated with CMDB. To verify this please check cmdbcorsslaunch.properties from %BMC_TRUESIGHT_HOME%\custom\conf\ for UDDI and CMDB details. File and data should exist in order to enable ITSM change integration.

 

Install ITSM change management application and verify the step 1 from above section to ensure the UDDI is configured.

 

ENABLING THE INTEGRATION

 

From the TrueSight Presentation Server select Administration --> Integrations

Under Change Management Integration, click the action menu and select Edit.

 

 

 

Enter the CMDB and UDDI details.Select Activate Integration and click Save.

VALIDATING THE INTEGRATION ENABLED

 

Once the integration activated from TSPS console, Make sure that the integration is active on the box where ITSM is installed.

  • Go     to Application Administration Console > Custom Configuration     > Change Management > Advanced Options > Change CI Event     Notification to BPPM > BPPM Subscription.
  • Click Test     Connection to ensure connectivity between Change Management and     Operations Management.
  • Click Save.

 

Flow from ITSM to TrueSight end to end

 

 

1-Submit the CRQ from Change Management System and status should be in Scheduled State

 

 

2- ITSM will send notifications for the submitted change by the user and the entry will be created in Form - CHG:CHGBPM:Change Adapter and there are two escalations triggers INT:CHGBPM:SendEvent and INT:CHGBPM:ProcessEvent  to process this events from ITSM to TrueSight.

 

3-In order to check the Event has received from the ITSM, we can check the form CAI:Events.

Search for “Change_Out_CreateBPPMEvent

The expectation here is the event should be process from here and event API call will land to TrueSight Server for creating event.

 

If the event remains in CAI:Events form then verify the return code

 

 

4-Once the event process from ITSM end, the Web Service API call will go to BMC TrueSight Presentation Sever and internally call will from TrueSight Presentation Sever to BMC TrueSight Infrastructure Management Server.

 

RestServiceAudit.log

INFO  09/18 22:01:51.264 [https-jsse-nio-443-exec-1] RestServiceAudit Target is: InvocationTarget [authRequired=false, httpMethod=POST, targetURI=/api/CI/OI-7DAC07EA4AAD4899B3B196270433D34C/event]

 

DEBUG 09/18 22:02:13.029 [Thread-115,EXECUTOR_POOL] RestServiceAudit Calling process request...

DEBUG 09/18 22:02:13.029 [Thread-115,EXECUTOR_POOL] RestServiceAudit About to process the event request...................

DEBUG 09/18 22:02:13.029 [Thread-115,EXECUTOR_POOL] RestServiceAudit  processRequest is requested for resource size 1

DEBUG 09/18 22:02:13.045 [Thread-115,EXECUTOR_POOL] RestServiceAudit done with create uri

DEBUG 09/18 22:02:13.045 [Thread-115,EXECUTOR_POOL] RestServiceAudit created execution units list

DEBUG 09/18 22:02:13.045 [Thread-115,EXECUTOR_POOL] RestServiceAudit about to add execution unit...................

uri:http://ab.yy.com:80/bppmws/api/CI/OI-7DAC07EA4AAD4899B3B196270433D34C/event

 

ngphost:ab.yy.com

 

5-Example Event created in TSPS

 

 

6- Status in Services Page on TSPS

 

 

7-Blackout Policy created in TSIM Admin Console. If we have submitted the CR for the Patrol Agent then Blackout Policy will be created in Infrastructure Policies under Configuration Page on TSPS

 

 

8- Sent an event during blackout period and event status are blackout

 

9-Once the CR will go from Scheduled to Completed state, then events will be closed. After closing the CR the configured blackout will be deleted from TSIM Admin Console.

 

 

 

Troubleshooting Change Management Issues on TSPS/TSIM

1) Unable to register the TSPS instance

  • Check TrueSight.log about the error coming.
  • Check if AR Web service is publish and available with UDDI.

  • Make sure that correct UDDI credentials are provided.
  • Make sure that WSDL,End Point URL UDDI is reachable and links are accessible from TSPS.

  • Make sure that following forms are available on AR side. if not, check if extension installation is proper.

A) CAI:Events
B) CAI:AdapterEventConfiguration
C) CAI:AdapterConfiguration

  • Make sure that default itsm change instance is also available.
  • If you face time-out issue, please check arerror.log
  • If you face key violation(unique index) issue, check if is there already any integration present. There is one-one relationship present. (1 ITSM to 1 TSIM)
  • Check CAI: Events form for the error you are getting.

2) Registration successful, but unable to see events in TSPS and thus to TSIM

A) If you do not see events pending there, check arjavaplugin.log from AR side.

B) If event is pending in the CAI:Events Form,there check the error you are receiving

  • For 401: PLease check if you have manually change the password of default user (csm_user)
    You can change the password from CAI:AdapterConfiguration form.
  • For 400: Check arjavaplugin.log for detailed error. TSPS will not process the event if it found multiple entries of resource (CI) present.
  • For 404: Check if you have CI available on TSPS side.
  • Make sure that the TSIM to which the CI belongs is properly registered to TSPS and in connected state.


C) You can also check TrueSight.log and InfraApp.log for the error coming in TSPS if failure lands in TSPS side.
Files to check on TSIM : bppmws.debug.log, eventws.debug.log, TrueSight.log

3) Unable to see CMA policy on TSPS but Event is listed on UI.

  • Check if you have Automated Blackout Policy created on TSIM admin console.
  • Check the jserver.log (debug level) for any error coming on while sending message to CMA.
  • Check for the messages coming for RuleConfigurator.
  • You may need to check messaging logs from TSIM and TSPS side for "ITSM" Blackout message communication.

 

 

·       EventProvider (EventConverter)

Any Error occured in this component will be sent back to the Webservice component, which will map that error into some error code.

1> Invalid Data Error: This error will occur when some of the manadatory parameter would not be present or some data would be in invalid format.

2> Not Connected Error: This error will occur when cell will be down, and connection would not be possible with cell.

3> Processing Error: This error will occur while sending the event to cell or error occured at cell while creating the event.

·      Event Processor Component

Any Error occured in this component will be sent back to the Webservice component, which will map that error into some error code.

1> Invalid Data Error: This error will occur when some of the mandatory parameter would not be present or some data would be in invalid format.

2> Not Connected Error: This error will occur when cell will be down, and connection would not be possible with cell.

3>  Processing Error: This error will occur while sending the event to cell or error occured at cell while creating the event.

·      Rule Configurator Component

No error will be sent back to any component, only logged into the Jserver log file.

1> One of the rule configurator failed i.e policy creater or pnet scheduler. We will revert back the successful rule on the failure of other rule.

 

Some Use Cases:

Problem: Blackout policy and schedule downtime are not created for events that are received successfully
by BMCProactiveNet Performance Management

  • Ensure that the flag pronet.jserver.ruleconfigurator.enable in the pronet.conf file is enabled.
  • If a blackout policy is not created, check if the flag pronet.jserver.ruleconfigurator.eventcollectiondowntime.enable=true is
    set to true.
  • If schedule downtime is not created, check if the flag pronet.jserver.ruleconfigurator.datacollectiondowntime.enable=true is
    set to true.
  • If the problem persists even after the flags are set to true, check the
    JServer logs for any errors.

Some of the possible error conditions are:
- All mandatory slots are not getting propagated from the BMC ProactiveNet cell to the gatewayevent handler.
  - Scheduled date and time entered in BMC Change Management are not valid.
  - The CI does not exist.


Problem: Event is not created in the BMC ProactiveNet cell

  • Check the web services logs for any change notifications. If the change notification is present, check the web service logs for any error conditions.

Some of the possible error conditions are:
  - The event is invalid.
  - Some mandatory slots are missing.
  - Other internal error to the server.

  • If JServer receives the event, check the JServer logs if the event is available. If the event is available, check for any error.
    Some of the possible error conditions are:
    - The BMC ProactiveNet cell is not connected.
    - An exception has occurred in the BMC ProactiveNet cell, due to invalid data.
  • If the BMC ProactiveNet cell receives the event, check the cell logs for any errors.

Problem: Event Not received to TSPS/TSIM and no log entry in any of the TSPS/TSIM logs

Check Configuration on the ITSM change management :

  • Check if Change-BPPM integration is active. To verify, go to Application Administration Console > Custom Configuration > Change Management > Advanced Options > Change CI Event Notification to BPPM > BPPM Subscription. Status of BPPM instance should be active.
  • Click Test Connection button on same page. Check server, check user name password.
  • Check AR Java Plugin.log file, BPPM error logs for any errors.
  • Make sure the Escalations are enabled in AR Server Setting.

        If still Events are not seen on BPPM console, then perform following steps:

  • Check CAI:Events form for any Error out events. Check the error message code defined in this event, check the details mentioned in the BPPM error message guide and take appropriate actions.
  • The integration will only process the Events for SIEM Enabled CI's with all allowed relationships (Only "Impacted" relationship will not be processed for event generation). So check whether CI's related to the Change Request are SIEM enabled or not.

 

 

 

 

 

Sample Logging in the Jserver:

 

To Check Event reach the jserver, and to check the values of attributes, search following log
----------------------------------------------------------------------------
com.proactivenet.api.aspen.ws.EventDelegate: createEvent: Attributes:

DEBUG 09/19 03:29:46 jserver              [Thread-54,EXECUTOR_POOL] com.proactivenet.api.aspen.ws.EventDelegate: createEvent: NGPEvent: Event id: null, severity: null, date reception: null, status: null

INFO  09/19 03:29:46 jserver              [Thread-54,EXECUTOR_POOL] 302868 com.proactivenet.api.aspen.ws.EventDelegate: createEvent: Receieved event: Event id: null, severity: null, date reception: null, status: null, Attributes: {msg=Testing, requestedForName=Allen Allbrook, changeNotes=, ciAssociationType=Related to, riskLevel=Risk Level 5, changeModifiedDate=1537352956, ciAssociationDate=1537349970, mc_smc_id=OI-7DAC07EA4AAD4899B3B196270433D34C, changeId=CRQ000000000202, changeCoordinator=Mary Mann, scheduledEndDate=1537353000, requestedForCompanyId=Calbro Services, mc_owner=Mary Mann, mc_smc_alias=Sharad_Service, mc_host=clm-pun-030033.bmc.com, statusReason=, urgency=LOW, itsm_company=Calbro Services, mc_priority=PRIORITY_5, itsm_location=clm-pun-030033.bmc.com, severity=INFO, changeReason=, impact=MINOR_LOCALIZED, mc_host_address=, ciClass=BMC_BUSINESSSERVICE, changeLocationCompanyId=Calbro Services, scheduledStartDate=1537351200, changeCategory=Normal, dataSetId=BMC.ASSET, changePriority=Low, changeStatus=Implementation In Progress}

 

For status of send event check following logs:

 

INFO  09/19 03:29:46 jserver              [Thread-54,EXECUTOR_POOL] 302868 com.bmc.sas.ngp.eventprocessor.util.EventSender: sendEvent: Sending Event succeed with handle: 40534

  1. com.bmc.sas.ngp.eventprocessor.util.EventSender: sendEvent: error: (Detail message)

When Event gets propagates from cell to Jserver we can see the following logs:

DEBUG 09/19 03:29:46 jserver              [EventSelectorWorker#5] com.bmc.sas.ngp.server.components.EventSelectorWorker: run: event id: mc.pncell_VW-PUN-APM-SP01.1ba21ebb.0, time for submmiting the event(millis): 1

INFO  09/19 03:29:46 jserver              [ChangeRuleConfiguratorConsumer] 302868 com.bmc.sas.ngp.ruleconfigurator.change.ChangeRuleConfiguratorQueue: poll: resuming from wait state

To Search any logs corresponding to blackoutpolicy, refer thread
-------------------------------------------------------------------------------------------------
RuleConfiguratorWorker#PolicyConfigurator

To Search any logs corresponding to Schedule downtime, refer thread
-----------------------------------------------------------------------------------------------------
RuleConfiguratorWorker#ScheduleConfigurator

Log statement to confirm Schedule downtime is created successfully

INFO  09/19 03:29:46 jserver [RuleConfiguratorWorker#ScheduleConfigurator] 600002 com.bmc.sas.ngp.ruleconfigurator.change.ScheduleConfigurator: delegateConfigure: event id: mc.pncell_VW-PUN-APM-SP01.1ba21ebb.0, configuration is Successful: true, ruleConfigurator: ScheduleConfigurator

Log statement to confirm Blackout is created successfully
-------------------------------------------------------------------------------------
INFO  09/19 03:29:46 jserver [RuleConfiguratorWorker#PolicyConfigurator] 600002 com.bmc.sas.ngp.ruleconfigurator.change.PolicyConfigurator: delegateConfigure: event id: mc.pncell_VW-PUN-APM-SP01.1ba21ebb.0, configuration isSuccessful: true, ruleConfigurator: PolicyConfigurator

Logs to check the status of schedule and policy together:
----------------------------------------------------------------------------------
INFO  09/19 03:29:46 jserver [ChangeRuleConfiguratorConsumer] 600002 com.bmc.sas.ngp.ruleconfigurator.AbstractRuleConfigurator: submit: event id: mc.pncell_VW-PUN-APM-SP01.1ba21ebb.0, All the rule configurator workers succeed: [PolicyConfigurator, ScheduleConfigurator]

INFO 09/19 03:29:46 jserver [ChangeRuleConfiguratorConsumer] 600002 com.bmc.sas.ngp.ruleconfigurator.AbstractRuleConfigurator: submit: event id: mc.pncell_VW-PUN-APM-SP01.1ba21ebb.0, Failed rule configurator workers:

In Case CI is not a BMC computer system, we skip schedule down time.
----------------------------------------------------------------------------------------------------------
jserver [RuleConfiguratorWorker#ScheduleConfigurator] com.bmc.sas.ngp.ruleconfigurator.scheduleddtcrud.ScheduledDownTimeCRUD: scheduledDTConfigure: Given CI is not a ComputerSystem CI, hence not configuring the schedule downtime

 

 

Known Issues and Fixes

1-Product defect SW00525875 - was raised for Atrium WebServices, where If customers want to have the WS published and facing the behaviour reported in the defect they need to publish the WS manually, that involves using scripts as a workaround.

 

This same workaround, is impacting the configuration of Change Management, in TrueSight Presentation Server, which is not being pushed to Application Administration Console > Custom Configuration > Change Management > Advanced Options > Change CI Event Notification to BPPM > BPPM Subscription.

 

Currently the defect will possibly be considered in 9.1SP4 release.

 

Workaround:

Force the registration, from TrueSight Presentation Server

 

a) Open the AR System Web Services Registry form in search mode.

b) In the Web Service drop-down menu, select CAI_ChangeManagementEventsSubscription_WS and click Search

c) Copy the End Point URL.

d) Go back to TrueSight Presentation Server, and open an Command Prompt window, type:

 

tssh properties set defaultItsmSoapWSURL <End Point URL>

This property should now be inserted, into properties configuration.

 

Reload the properties

tssh properties reload

This properties should now be reloaded

 

Check if the property "defaultItsmSoapWSURL" can now be found in the configuration:

tssh properties list

 

e) Go back to Change Management Integration in TrueSight TSOM, and press Save, this should no longer fail.

 

f) The BPPM/TSOM integration should be active in ITSM.

 

Go to Application Administration Console > Custom Configuration > Change Management > Advanced Options > Change CI Event Notification to BPPM > BPPM Subscription.

The BPPM status should be Active.    

 

2-The integration between BPPM/Truesight and Remedy Change Management will not work if there is a colon character (:) in the summary of the Change Request.

 

Refer Article:000136490

 

3-After successful integration with TrueSight (10.7) and Remedy Change Management (v9.x) was confirmed with successful publishing of CI from Remedy, there was no automatic blackout policy being created and no notifications being received, as expected with scheduling of a Change Request (CR) and relating the relevant CI.

 

Refer Article:000144136

 

What Logs and Information To Send BMC Support

 

1. Version of TSPS,TSIM and ITSM

 

2. From TrueSight Presentation Server below is the list of important logger to troubleshoot issue and command to turn on/off debug

 

tssh logger debug on -n RestServiceAudit

tssh logger debug on -n ROOT

tssh logger debug on -n InfraApp

 

Logs folder from  %TRUESIGHTPSERVER_HOME%\truesightpserver

 

Note: Make sure you disable debug as soon as issue is reproduced

 

3.From TrueSight Infrastructure Management Server important logger to troubleshoot issue and command to turn on/off debug

 

 

pw debug on -p jserver

pw debug off -p jserver

 

Logs folder from <BMC_TRUESIGHT_HOME>\pw\pronto\lpg

 

Note: Make sure you disable debug as soon as issue is reproduced

 

4-From ITSM Side.

 

 

Enable API/SQL/FILTER/ESCALATION debug with below steps.

 

 

In AR User tool go to,

AR System Administration Console  -> System -> General -> Server Information Logs Files tab

 

 

Select 'Filter Log', 'API Log', 'SQL Log' and modify the 'Name:' fields so they all write to the same file..

Click Apply > OK

 

Note: Make sure you disable API/SQL/FILTER/ESCALATION debug as soon as issue is reproduced so the size of log file is minimum.After this, wait for the problem to reproduce and collect below logs immediately.

 

What am I missing here? Let me know in the comments and I'll add it in!