Share This:

BMC IT Data Analytics (ITDA) is a powerful solution that allows to mine, index and analyze semistructured data, typically coming from log files, traces, syslog streams and other sources that provide data in a time stamped text format.

Among its various capabilities, ITDA allows to derive metrics from the text it analyzes, extracting the desired values from the text entries or counting the entries resulting from a search.

 

This post aims to explain and provide few examples on how to extract metrics and pass them to other applications.

 

Assumptions: in this post I only cover the data extraction and integration aspects, so I am assuming you are already familiar with ITDA setup and configuration, searches, patterns and all the other core product areas.

 

The key ITDA capabilities we'll use are:

 

Use Case: Execute a search, count the number of returned lines and send the number to a desired target.

 

The first thing to have is, of course, the search that will provide us with the lines matching our criteria in a given timeframe. This could be used for getting all the lines reporting failed actions, connection retries, etc. The time context will give us the measure of how significant or impacting that behaviour is from a time perspective.

In my example, I am counting lines representing tweets and those lines contain information like evaluated sentiment. So i'm basically counting the number of positive (or negative, or neutral) tweets  received in the last 5 minutes.

 

The saved search.

So the first step is to have the search you want use to count the relevant entries saved so that it can be called from other functions in and outside ITDA, like notification sin our case but a saved search could also be called by the ITDA REST API.

The saved search in my example is configured to return matching lines in the last five minutes but this can be overridden later.

saved_search_window.png

 

This search returns all the tweet lines tagged as "positive" in the last 5 minutes:

search_outcome.png

 

 

So now we can count them. This step is not actually needed but clarifies what we are going to get as a value we'll send out to the target system we want to integrate.

count_on_field.png

 

 

The notification.

Now the key part. Here we define a notification of type "alert" (1) that triggers a command line (5) when the defined conditions are met (2). This is done every 5 minutes (3) and as stated before, the saved search (2) time frame can be modified overriding the one defined in the saved search (4).

notification_details.png

In the above picture, step (2) shows 2 conditions:

 

number of saved search results > 0

OR

number of saved search results = 0

 

The reason for this is that with just the "> 0" condition we would trigger the script (5) and therefore send the data only when there are entries to count. But for a polling based monitoring system this would result in "data gaps" for all the intervals where the count "= 0" as the script wouldn't be executed and therefore no data would be sent to the target system.

Adding the "= 0" condition we're basically ensuring that when there are no entries to count in the 5 minutes polling time frame (3) instead of doing nothing we send a "0" data point to the target system avoiding misleading data representations.

 

Using Macros

Macros are more often used in internal actions like sending an email or an event. In those scenarios macros are pretty useful to dynamically build mail bodies or event messages.

 

In such context, a notification message that uses macros would look like this:

Saved search ${QUERYNAME} has result count:${COUNT} for duration: [${STARTTIME}] to [${ENDTIME}]

and would produce a message text like this:

Saved search ITDA_Log_Monitoring has result count: 3567 for duration: 01/30/2015 11:30:30 GMT to 02/06/2015 11:30:30 GMT

(I added the bold formatting)

 

The available macros are listed in this doc page.

 

In our example we are using the ${COUNT} macro which provides the number of search results returned by a search query.

if in an internal notification like an email the macro would be referenced as described above, when using an external script it can be referenced as an OS environment variable which in this case, on a Linux based ITDA system, it would be $COUNT without curly brackets but it would be %COUNT% on a windows ITDA system.

 

this means that in my case the Linux OS shell script I'm referencing at point (5) looks like this:

 

[root@itda-tsi TrueSight]# more itda_positweets_2_tsi.sh

 

#!/bin/bash

python /opt/bmc/TrueSight/ITDA_2_TSI_tweetscount.py -p $COUNT

 

So it is extremely straightforward, in my example I'm just passing the count of the log lines representing the positive tweets which will be a value "> 0" when there is data or "= 0" when there is no data so that we can avoid data gaps and actually have a "0" value when there are no lines to count.

 

The script called in this example sends the tweets count to TrueSight Intelligence. The specifics of the TS Intelligence integration will be covered in another post.

TSI_sshot.png

 

This is just an example of how ITDA can be used as a "KPI collector" and feed other monitoring, reporting, Dashboarding or SLM solutions.

 

I hope you'll find this useful.

 

Gianpaolo Pagano Mariano

gpaganom@bmc.com