7 Replies Latest reply on Apr 19, 2018 9:20 AM by Scott Bleasdell

    ITDA Search - How to search for records which does not contain a particular string

    Sameer Salve

      I want to create a saved search to look for records which does not contain a particular string.

      The data is captured as free text, so there is no field as such where I can use the "<>" operator.

       

      E.g search data containing "400" but not "test" ( There might be instances where "400" and "test" will be in same line. )

        • 1. Re: ITDA Search - How to search for records which does not contain a particular string
          Scott Bleasdell

          Sameer Salve,

           

          * | extract field=_raw "(?<MSGS>.*)" | filter regex(MSGS, “.*400\s+(?!test).*")

           

          This will look for all messages you collected (that's the * at the beginning of the line) and then filter those to only messages that contain “400” that are NOT followed by “test”, assuming there is at least one space character in between.

           

          Let me explain how this works.  The * at the beginning says "find all messages".  The result of that is then piped (that's the | character) to the extract search command.  The extract search command will extract data from the field you specify based on the regex you enter.  In this example, I told it to extract everything in the raw message (that's the _raw field).  The regex there, "(?<MSGS>.*)", says to create a new field (also known in the regex world as a capturing group) named "MSGS" from the _raw field that matches the regex .*.  Well, as you likely know, .* means zero or more of any character - which basically means everything.  So it returns the entire raw message and assigns it to a new field called MSGS.

           

          Why did I do that?  Because I need to use the filter search command in order to do the negative search you are looking for - where the word "test" does not exist.

           

          So I pipe the output of the extract search command to the filter search command.  I use the regex option of the filter command so I can specify a negative lookahead regex.  This regex says match anything, followed by 400, followed by one or more spaces, followed by anything except the word "test", followed by anything.

           

          If in your case you have the word "test" appear before the number 400, then your search command would be:

           

          * | extract field=_raw "(?<MSGS>.*)" | filter regex(MSGS, “.*test\s+(?!400).*")

           

          I hope that make sense.  Let me know if you need anything else!

           

          Scott

          • 2. Re: ITDA Search - How to search for records which does not contain a particular string
            Sameer Salve

            Thanks Scott for your response.

             

            I tried above regex, but I am not seeing the expected result. Not sure if I am doing something wrong.

             

            I used below query

            COLLECTOR_NAME="Coll" |  extract field=_raw "(?<MSGS>.*)" | filter regex(MSGS, ".*400\s+(?!test).*")

             

            I have uploaded below lines in the above collector (Coll), so the pipe should send below lines to the next extract command....

            After executing above query, it still outputs all the lines.

             

             

            Mar 28, 2018 9:31:55 AM

            400 with username, ?invalid_grant Bad credentials" ? error description then do not create any alert.

             

            Mar 28, 2018 9:31:55 AM

            400 Hi this is test result

             

            Mar 28, 2018 9:31:55 AM

            [400] : If you get 400 with ?unauthorized Cannot find user with uid? error description

            • 3. Re: ITDA Search - How to search for records which does not contain a particular string
              Scott Bleasdell

              Sameer Salve,

               

              I'm glad you sent me a few examples.  I have fixed the search query.  To match those lines properly, you would run the following query:

               

              COLLECTOR_NAME="Coll" | extract field=_raw "(?<MSGS>.*)" | filter regex(MSGS, ".*400[\]\w]+(?!test).*")

               

              The difference is in the regex for the filter search command.  First, I assumed that there was a space character after the 400.  That's why I had "400\s+" originally.  In your case, you had two different characters - a tab (I think), and a "]".  So, I updated the regex to handle these with "400[\]\w]+".  This means 400, followed by one or more of either a white space (any non-visible, or whitespace character) or a "]".  If you have more lines in your actual source log file that may have other characters after the 400 - something like a colon ( : ), semi-colon ( ; ), dash ( - ), etc. - then you need to add those inside the square brackets of the regex I mentioned above.  And you will need to escape those characters that are regex special characters, if any.

               

              Please let me know if you need any more help.

               

              Scott

              • 4. Re: ITDA Search - How to search for records which does not contain a particular string
                Sameer Salve

                Hi Scott,

                 

                COLLECTOR_NAME="SysLog_TCP_XXX" && "Http Status=[400]"  | extract field=_raw "(?<MSGS>.*)" | filter regex(MSGS, ".*(?!Bad credentials).*")

                 

                I was trying above and I still see records containing "Bad credential" being returned by the query. I want the result with http status 400 which does not contain string "Bad credentails" anywhere in it .

                 

                 

                 

                Thanks,

                Sameer Salve

                • 5. Re: ITDA Search - How to search for records which does not contain a particular string
                  Scott Bleasdell

                  Sameer,

                   

                  Can you paste a few example lines from your logs that have what you are looking for and what you want excluded so I can do some debugging?  I'm guessing it's just something similar to what we saw before where you have to put in some additional regex to describe things like white spaces.  In fact, I am going to guess right now that the following might work:

                   

                  COLLECTOR_NAME="SysLog_TCP_XXX" && "Http Status=[400]"  | extract field=_raw "(?<MSGS>.*)" | filter regex(MSGS, ".*(?!Bad[\]\w]+credentials).*")

                   

                  Let me know.

                  Scott

                  • 6. Re: ITDA Search - How to search for records which does not contain a particular string
                    Sameer Salve

                    Thanks Scot for you help.

                    Below is the solution. If you see error related to double quotes, type them again.

                     

                    COLLECTOR_NAME=“test" && "Http Status=[400]" | extract field=_raw "(?<MSGS>.*)" | filter regex(MSGS, "^((?!Bad\s+credentials).)*$”)

                    1 of 1 people found this helpful
                    • 7. Re: ITDA Search - How to search for records which does not contain a particular string
                      Scott Bleasdell

                      Thanks for updating the community, Sameer.  Yes, if you look close at the query you posted above, the last double-quote looks different - it's a different font.  sometimes copy/paste gets funky...