4 Replies Latest reply on May 6, 2015 5:39 AM by Julien CANDOLINI

    IDA - Personal review after few days of use

    Julien CANDOLINI

      Hi Guys,

       

      So i started to use this product few days ago and there is some points i wanted to speak about.

       

      1/ Content Packs

       

      I downloaded some useful content packs (Vcenter, BPPM, Oracle, MSSQL) but OOTB all the environment variables use the $ sign (for unix systems). As we use almost only windows systems, i needed to modify all the collection profile directory path to fit the windows system variable.

       

      It would be maybe be fine to modify the OOTB Collection Profile with an "operating system" drop down list that will automatically modify the env variable depending on the OS (or add a second Path filed, like "Windows Patch" / Unix path that will be selected by the OS drop down list).

       

      Actually if i want to apply an Oracle profile that run on windows and linux, i need to recreate a second Collection profile and modify all the variables.

       

      Also a "copy to new" button would be useful.

       

      2/ Data Collectors

      On the Administration -> Data Collectors views you can just see 20 data collectors.

       

      If you have thousand of data collectors, for a datacenter infrastructure use in my case, this is not enough (event to see the status of all the failed collectors)

       

      The would be fine to have another drop down list with "show 20 / show 100 / show 200 / show 500".

       

      Also you can only delete the collectors one by one ! Try to select 2 collectors and the delete button became grey. I mean if you create a Collection Profile with a lot of things in it to fit the maximum monitoring needs, but some Data Collectors can be removed cause the server don't need everything, you need to take some time to delete the unnecessary ones.

       

      3/ Credential Problems

       

      I never successfully enter credentials

       

      I tried to add a Windows Events log monitoring on a collection station that is on the same subnet and domain that my IDA server, using domain admin, i get :

      Collection Station is not reachable

       

      My agent status is green, windows firewall is no activated.

       

      I tried to add a Windows Events log monitoring on a collection station that is on a different subnet and domain that my IDA server, using domain admin, i get :

      The username or password is incorrect.

      all FW flows are allowed in both direction between those 2 subnets, no windows FW.

       

      For the second case i want to monitor a Vcenter vpxd.log.

      So this is not working with the OOTB VCenter collection profile, as he cannot connect using the provided credentials to the shared drive (\\xxxx.xx.int\shared), so i needed to create a new "monitor file on Collection Agent" with path C:\ProgramData\VMware\VMware VirtualCenter\Logs and filename vpxd*.log

       

      4/ Content Pack roadmap

       

      I wanted to know if you have some roadmap for the Content Pack.

      For BMC Products, like ITSM suite (atrium, sd, incidents etc...), BAO, Portal, Entuity, Bladelogic Server/Network, ADDM

      or other products.

       

       

      That's it for the moment

      Any feedback is welcome.

       

      best regards

      JC

        • 1. Re: IDA - Personal review after few days of use
          Scott Bleasdell

          Julien,

           

          Thank you so much for posting all this detailed feedback!  I don't know how I missed seeing it until now - I must have had my eyes closed.  This is fantastic stuff.

           

          The $variable notation used in Content Packs is the Linux environment variable notation, but our agent is smart enough to know that if it is installed on a Windows machine it will convert that notation to %variable% automatically.  You didn't have to edit all the data collectors because of this.  We need to make that clearer in the product, I have added that to our to-do list.

           

          There is a "Clone Data Collector" option for copying a data collector.  If you check a data collector in the Administration / Data Collectors page, the icons at the top of the data collector list will enable and there is one there that you can use to copy/clone the selected data collector.  Maybe you missed that.  We need to make that easier to find.

           

          I completely agree with you that we need to make the Data Collectors view much more scalable, and we are working on that.  It will be coming after our 2.0 release.

           

          Again, I completely agree with you about deleting data collectors one by one.  We didn't expect that people would need to do this very often initially, and that's why the feature isn't there yet, but I agree we need to add it now.  It's in our backlog and will be coming soon.

           

          Regarding the credentials problem... this one is a little tricky.  We have improved this significantly in the upcoming 2.0 release, but until then, what I recommend you do is leave the "Host" entry blank and enter the FQDN of the server name you want to pull the event log data from in the Server Name entry.  If that doesn't magically make it work, reach out to me directly and we'll hop on a WebEx and I can help you troubleshoot.

           

          Finally, we do have a roadmap for Content Packs, which include BMC products like the ITSM suite and much more.  That roadmap is not quite ready for external sharing, but your feedback here is good confirmation of what we have planned for the future of Content Packs.

           

          Thank you!

          Scott

          • 2. Re: IDA - Personal review after few days of use
            Julien CANDOLINI

            Hi Scott, thanks for your response, I know that this product is still young and am sure that with the future improvements, he will be able to replace all my splunk servers and other logs analysers

             

            Some other things I tried :

            - Setup a syslog receiver and I start to receive the logs from some of my equipements (clearswift appliances).

            the timestamp of those logs are not in the long list of the one supported out of the boy by ITDA. I created a new data pattern following the video tutorial (the same that in the documentation).

             

            Sample text :

             

            <190>1 2015-05-04T11:08:41Z xxx.xxx.int sys - - - ntpd[4466]: unable to create socket on eth1 (513845) for fe80::225:xxxxxx:1006%xxx#123

            <190>1 2015-05-04T11:09:11Z xxx.xxx.int sys - - - ntpd[4466]: bind(23) AF_INET6 fe80::xxxx:100e%xxx%253#123 flags 0x11 failed: No such device

            <190>1 2015-05-04T11:10:50Z xxx.xxx.int audit - - - INFO  [Consolidator]- Time in Disposer:225688 Records: 8183 rps:36

             

            I created a "ClearswiftSyslogDate" subpattern : %{Year}-%{MonthNum}-%{MonthDay}T%{Hour}:%{Minute}:%{Second}Z    that match the timestamp in the syslog

             

            1.png

             

            when I want to preview, I always get this message :/

             

            2.png

             

            - Another point, in the Search page, I would find more helpful a panel with "Hosts collectors" instead of   "Data collectors" (and with the list limited to 10 items), cause if you have just 1 Pnet server, you already have +- 10 data collectors on it. Immagine with 5 pnet servers, vcenter etc ...

            So from this search page I would directly click on the collector where I want to see all the logs instead of enter a search like (COLLECTOR_NAME="Syslog Server - RSM DMZ TCP 514") each time.

            But maybe I can play with the "workspace" for that ? I didn't had time to go more deep for the moment.

             

            - I created some collectors with OOTB collection profiles (like Pnet Server, also Vcenter), I never was able to get anything in the "View": I can go in Vsphere:vCenter, Vsphere:storage, BPPM Events ... I just can see "no data available", just the OOTB BMC Truesight ITDA view is working. Maybe I miss some configurations to make in the documentation? cause the log analysis is working fine for my Pnet or Vcenter.

             

            Best regards

            • 3. Re: IDA - Personal review after few days of use
              Scott Bleasdell

              Julien,

               

              Thank you again for your feedback.  On first glance, I am not seeing anything wrong with the data pattern you defined, but I am going to try to reproduce this using the sample text you supplied and see if I can figure out what's going on.  I'll get back to you as soon as I can.

               

              I like your feedback on having host collectors on the search page.  We simplified things in the upcoming ITDA 2.0 release and removed the list of data collectors from the search page and will be adding better/smarter ways of searching the collected data in the following release.  Host collectors is one of the options we are considering.  We also removed the Workspaces pane by default in 2.0 as well because it was causing a lot of confusion for users.  I don't believe it would serve as a good solution for a list of host collectors anyway.  You could fairly easily create a saved search for each of the common host collectors you are likely to want to search within as a temporary workaround until we deliver a better solution.

               

              The saved searches (in the Cabinet page) for BPPM and VCenter should work right out of the box, and therefore the charts in the View page should be showing data.  Can you manually execute the saved search that is associated with the chart that is reporting "No data available" and see if you can detect why it's not finding the data?  On the surface, it seems that either the search is somehow wrong or the data isn't being collected for some reason.  Just need to figure out which is the case.

               

              Thanks...

              Scott

              • 4. Re: IDA - Personal review after few days of use
                Julien CANDOLINI

                Hi Scott,

                 

                About the View Page and No Data :

                 

                Can you manually execute the saved search that is associated with the chart that is reporting "No data available" and see if you can detect why it's not finding the data?

                > I recreated my VSphere collector (Monitor file on Collection Agent - C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd*.log with vSphere:VpxdLog pattern)

                Looks like almost all the Cabinet/Saved Search didnt fit those vpxd logs.

                All the saved search with this VmwareModule search string didnt exists in the vpx logs thats why the views don't have any data

                 

                From the Data Patterns there is the vSphere:VpxSyslog pattern that contain this VmwareModule

                (?:|<%{VPriority:Priority}>)\s*%{VUtcTimestamp:timestamp}\s*%{vSphere_IpOrHost:VmwareHost}\s*%{Word:VmwareModule}:\s*%{MultilineEntry:details}

                 

                But the pattern from vSphere:VpxdLog just consider the text after the date as a MultilineEntry.

                 

                So if i try to search vSphere:Host: Tasks: Initiated per ESX

                your query   VmwareModule="Vpxa" && BEGIN && "task-internal" didnt return any result as the entry in my logs is :

                 

                2015-05-05T12:27:41.247+02:00 [13800 info 'commonvpxLro' opID=18daa2bd] [VpxLRO] -- BEGIN task-internal-906933049 --  -- vmodl.query.PropertyCollector.retrieveContents -- ba615f1f-c20e-6421-6de4-04bb82cda912(521d3c40-90d1-1112-b15d-81df96fd10d4)

                 

                But a query like : BEGIN && "task-internal" will work.

                 

                So i assume those search are only for vSphere:VpxSyslog :

                 

                  vSphere:Host Warnings

                  vSphere:Host: Errors

                  vSphere:Host: Events (from Log) per ESX

                  vSphere:Host: Events of VMs per ESX

                  vSphere:Host: HTTP Requests: Completed

                  vSphere:Host: HTTP requests: Initiated

                  vSphere:Host: Tasks: Finished per ESX

                  vSphere:Host: Tasks: Initiated per ESX

                  vSphere:Host: VOB Events per Host

                  vSphere:Number of vSphere Entities

                  vSphere:Storage: Latency Deterioration

                  vSphere:Storage: Latency Improvement

                  vSphere:Storage: SCSI/iSCSI Failures

                  vSphere:Storage: VMFS heartbeat timeouts

                  vSphere:VMkernel: Warnings

                  vSphere:VM Operations Rate: Clone

                  vSphere:VM Operations Rate: Create

                  vSphere:VM Operations Rate: Power Off

                  vSphere:VM Operations Rate: Reconfiguration

                 

                  And just those one are for vSphere:VpxdLog pattern :

                  vSphere:vCenter: Client Connection Requests

                  vSphere:vCenter: Host Communication Errors

                  vSphere:vCenter: Invalid Event Filter

                  vSphere:vCenter: Login Failures

                  vSphere:vCenter: New Sessions

                 

                  Thats why almost all my views are empty, as i don't use VpxdSysLog.

                 

                 

                  Following that, i created a new Syslog Data Collector and configure some of my esx to this syslog.

                 

                  ** About that, this is a little bit sad that one ITDA Syslog Collector can be linked to just one data pattern.

                  remember my Syslog collector i use for my ClearSwift appliances- I set the data pattern as "Free Text".

                  But i i want to use this same syslog collector for my Esx for exemple, i don't have the choice and the data pattern will be free text, and i'll not have any view.

                  My only choice is to create on the same ITDA Collector server anoter Syslog Collector on port 515 for exemple and set VpxSysLog as a data pattern

                  Then create a third collector on port 516 for my Cisco switches/routers equipments and use the Cisco pattern, and another one for alcatel, adva etc ...

                  I think that this Syslog Collector "flexibility" can maybe be improved.

                  **

                 

                  I finally get the syslog with many problems.

                  Creating a TCP syslog receiver with vSphere:VpxSyslog didnt work and i take many time to figure out why.

                 

                  After enabling the debug log on my collector i notice that none of my logs send by the esxi was processed and thats why there was no data for my collector

                 

                WARN: Collector=Syslog Server - RSM02 - TCP 514, CollectionPollId=1, Completed reading source=Syslog Server - RSM02 - TCP 514, Events - success=0, ignored=9114 ,rejected=49

                 

                After setting up the "free text" pattern, i can see my esxi logs

                 

                WARN: Collector=Syslog - RSM02 tcp514, CollectionPollId=1, Completed reading source=Syslog - RSM02 tcp514, Events - success=1775, ignored=0 ,rejected=0

                 

                here are some sample syslog from my ESXi (V5.5), looks like it didnt fit your vSphere:VpxSyslog pattern.

                <166>2015-05-06T10:04:41.578Z l1sesxoxxx Vpxa: [2DB68B70 verbose 'VpxaHalCnxHostagent' opID=WFU-ac6e6b46] [WaitForUpdatesDone] Completed callback

                <166>2015-05-06T10:04:41.850Z l1sesxoxxx Vpxa: [2DB89B70 verbose 'halservices' opID=WFU-c9d6cf90] [VpxaHalServices] VmGuestDiskChange Event for vm(56) 55

                <166>2015-05-06T10:04:41.850Z l1sesxoxxx Vpxa: [2DB89B70 verbose 'VpxaHalCnxHostagent' opID=WFU-c9d6cf90] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd

                 

                Then i just tried to re-modify the pattern configuration of my syslog collector  from Free Text to VpxSyslog and i direclty get hundred of those entries in collection.log :

                 

                WARN: Collector=Syslog - RSM02 tcp514, CollectionPollId=1,  Buffer is full, write cannot proceed. Dropping the entry: com.bmc.ola.collection.source.portreader.DataEntry@4bef141a

                May 06, 2015 12:09:18 PM com.bmc.ola.collection.source.portreader.BufferDataStore write():101

                WARN: Collector=Syslog - RSM02 tcp514, CollectionPollId=1,  Buffer is full, write cannot proceed. Dropping the entry: com.bmc.ola.collection.source.portreader.DataEntry@3d10dde8

                May 06, 2015 12:09:18 PM com.bmc.ola.collection.source.portreader.BufferDataStore write():101

                WARN: Collector=Syslog - RSM02 tcp514, CollectionPollId=1,  Buffer is full, write cannot proceed. Dropping the entry: com.bmc.ola.collection.source.portreader.DataEntry@1e58c9aa

                 

                then

                INFO: loading configurations for type = logsource

                May 06, 2015 12:11:16 PM com.bmc.ola.config.source.HttpConfigurationSource load():99

                INFO: Requesting configuration , URL used is configuration/collection-agent_L1VWPIRSM02.dtc.int/logsource

                May 06, 2015 12:11:16 PM com.bmc.ola.config.source.HttpConfigurationSource load():105

                INFO: Number of Configuration received of type logsource is 0

                May 06, 2015 12:11:16 PM com.bmc.ola.bwagent.controller.CollectionController$CollectionConfigTask run():735

                INFO: Number of collector configurations received is = 0

                May 06, 2015 12:11:17 PM com.bmc.ola.collection.collector.BaseCollector logMetricsEvent():683

                WARN: Collector=Syslog - RSM02 tcp514, CollectionPollId=1, Completed reading source=Syslog - RSM02 tcp514, Events - success=0, ignored=6522 ,rejected=77

                 

                And no more data was collected.

                 

                 

                  About the empty view for the BPPM Collection profile, i applied this profile on my PNET central server (Win2k8 R2 - v9.6) where ITDA Agent is installed.

                  The Data Collectors was automatically created and the polling status is green.

                  If i take the exmple of BPPM Server:CellServiceTemplate where directory is %BMC_PROACTIVENET_HOME%/pronto/logs and log files regex is cellservice*.log*, on my system

                  D:\BMC\ProactiveNet\pw\pronto\logs contains many logs like cellservice_agent.log, cellservice_agent_cntl.log, cellservice_jserver.log

                  but on the search view, there is no results found for my collection adapter COLLECTOR_NAME="BPPM Server:CellTemplate_xxxx.xxx.xxx"

                  Same problem for IBSRD logs (the data collector configuration looks correct with %IBRSD_HOME% including subfolder and *.log* regex )

                  Same for Pronet_db

                 

                  Some other collectors like the BPPM Server:ProactiveNetLogTemplate that collect the ProactiveNet.log* is working, same for WSLog, StormDB,CellService

                 

                  From the Views, all the BPPM Server : xxxx have no data available.

                 

                 

                  About MS SQL collection profile, i applied this profile on one of my host

                  Just changed the Data Collector path to the right one :

                  E:\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Log

                  This is collecting data but the Data Collector don't appear in the search view.

                  the Cabinet search string match some entries in my in my ERROR*.log

                 

                  2015-02-15 03:06:38.21 Logon       Error: 17803, Severity: 20, State: 13.

                 

                 

                  So if you have some ideas about the Vsphere and Pnet collectors, how to get the view and everything working, you are welcome.

                  If you want also we can make a webex.

                 

                  And congratulation if you read everything