7 Replies Latest reply on Mar 24, 2017 4:34 PM by Peter Lundqvist

    Apache Access

    Peter Lundqvist

      Hi people,

       

      As of yesterday I started using IT Data Analytics so I am very new to this product

      It was really easy to set up - I am impressed!

       

      So I've set up a collector of Apache Access logs from a tomcat server and it is collecting logs just fine.

      Now I wanted to search it, in perticular I wanted to search for Response codes and I was kind of under the impression that the Pattern (Appache Access) would help with this as the Pirmary Pattern described contains a field representation of it (num1, strange name - but so what).

       

      I have tried to search for this field, but I can't get it to work in any other way except for doing a text search of the code itself. E.g.: " 200 " (note the spaces).

      I feel that I am missing something obvious here. Is the pattern not matching correctly maybe?

        • 1. Re: Apache Access
          Peter Lundqvist

          I verified that the grok pattern worked in grok constructor (also noticed that there actually is a variant with response). So the logs do match the pattern.

          • 2. Re: Apache Access
            Scott Bleasdell

            Peter Lundqvist, you should be able to search on that value with, for example, a search that looks like this:

             

            COLLECTOR_NAME="whatever you named your data collector" && num1=200

             

            You should not need quotes around it, and certainly not spaces.  You can also use > and < signs, etc.  Here is the doc that has all those details: Search string syntax - BMC TrueSight IT Data Analytics 2.5 - BMC Documentation

             

            Let me know if you are not seeing this work as expected.

            • 3. Re: Apache Access
              Peter Lundqvist

              Yes,  I've read that but it does not seem to work, I've tried both with the fields named num1 and response.

               

              If I look at the collected logs, these fields do not show up. I only get the COLLECTOR_NAME and HOST fields:

              This is what made me investigate if the pattern matched the logs in the first place.

               

              The documentation for the 2.5 patch mention a configuration in an agent - but this is a simple one server install. Have I missed something?

              Is there a way to view logs and/or increase log level for the grok parser?

              • 4. Re: Apache Access
                Scott Bleasdell

                Peter Lundqvist, The fields are there, but in the "Optimized View" (the default view) they don't appear.  Change the "Optimized View" to "Detailed View" and you will see all the fields, both explicitly extracted by the data pattern and automatically extracted.

                 

                You will then be able to click on the bookmark icon to add the Num1 field to your filter pane (on the left-hand side of the screen) to see the various values for that field within the search results.

                 

                Let me know if that doesn't answer all of your questions.  I'd be very happy to hop on a call with you and go through this in more detail if you like.

                 

                Thanks!

                • 5. Re: Apache Access
                  Peter Lundqvist

                  I think I have solved it now, but thank you for your offer to help!

                  I will admit, this was partly a user error - but there was also a pattern mismatch.

                   

                  Having some (limited) previous experience with grok I embarked on trying to create my own pattern. I quickly realized that the ITDA grok parser was more strict than the grok tool I was using which led me to stop using Apache Access log and switch to Access Log - Common (reference) Now some parts of the example log worked. There was only one issue: logs where the size of the document was reported as a hyphen (representing unavailable information, not a number) were not recognized, so I modified the pattern from this (Access Log - Common, remove line breaks if you use it)

                  %{IpOrHost:ipaddress}\s+%{Data:rfc931}\s+%{Data:username}\s+\[%{AccessCommonTimestamp:timestamp}\]\s+
                  "%{RequestType:type}\s+%{GreedyData:imageurl}\s+%{Data:protocol}"\s+
                  %{PosInt:statuscode}\s+%{PosInt:size}(?:|\s*%{MultilineEntry:details})
                  

                   

                  to this

                  %{IpOrHost:ipaddress}\s+%{Data:rfc931}\s+%{Data:username}\s+\[%{AccessCommonTimestamp:timestamp}\]\s+
                  "%{RequestType:type}\s+%{GreedyData:document}\s+%{Data:protocol}"\s+
                  %{PosInt:statuscode}\s+(?:%{PosInt:size}|-)(?:|\s*%{MultilineEntry:details})
                  

                   

                  and everything worked, the fields appeared and they were searchable.

                  I made two modifications, but only one is important - the non capturing group for the size field that now also allows a hyphen instead of a positive integer.

                  The second one was only cosmetic, I renamed imageurl to document.

                   

                  For completeness, below is the log line that was not accepted:

                  143.237.33.30 - - [17/Mar/2017:21:33:57 +0100] "GET /arsys/ HTTP/1.1" 302 -

                   

                  Now my follow up question, how do I know if there were log entries that failed to match the grok pattern?

                  • 6. Re: Apache Access
                    Scott Bleasdell

                    Peter Lundqvist, sorry for taking so long to get back to you.  That's great that you figured this pattern issue out.  Great job!  You are correct... that grok parser is slightly different than the parsing that ITDA does.  You did exactly the right thing with the non-capturing group, too.  Well done.

                     

                    As for knowing whether log entries are failed or not, there are a couple of things you can do.  My first recommendation would be to:

                    • Make a copy of the "ITDA collection polls with no data" saved search (this is an out-of-the-box saved search)
                    • Modify that copy to replace the " | timechart span=2h count(events)" with "&& rejected > 0"
                    • Save this search as "Rejected data"

                     

                    You can now set up a dashboard and/or a notification which will show/notify you if any data isn't matching the data pattern you specify.  Note that if Best Effort Collection is enabled (and it is by default) on each data collector, and the timestamp matches, the data will not get rejected.

                     

                    My second recommendation would be to use the Collection History window.  Select the data collector you want to check out and click the Collection History toolbar (the icon that looks like a book).  This will bring up a window that shows you the last 10 polls and the status of that polling.  If any messages are getting rejected, you will see that show up in the table of message counts.  This is easier for a quick check against a data collector that you just created or suspect may have a pattern issue.

                     

                    I hope this helps.  Let us know if you need anything else.

                    1 of 1 people found this helpful
                    • 7. Re: Apache Access
                      Peter Lundqvist

                      No worries, and thanks for the tips!