8 Replies Latest reply on Dec 11, 2017 12:46 PM by Scott Bleasdell

    Is it possbile to capture multiline records including timestamp?

    Sudeep Bajpai

      Hello,

       

      We have an application log in the following format -

       

      start: <timestamp>

      record 1

      record 2

      record 3

      .

      .

      end: <timestamp>

       

      Is it possible to create a pattern to capture the 'start: <timestamp>' and the subsequent entries in ITDA?

       

      Regards,

       

      SB.

        • 1. Re: Is it possbile to capture multiline records including timestamp?
          Scott Bleasdell

          Sudeep Bajpai,

           

          I think this should be possible, but will require a custom field.  Can you copy/paste a few lines from your log file into a reply here and I'll create a custom data pattern for you.  If you have any specific fields you want to extract in this data pattern, note those and I'll include that in the data pattern I create for you.

           

          Scott

          • 2. Re: Is it possbile to capture multiline records including timestamp?
            Sudeep Bajpai

            Scott Bleasdell,

             

            Thanks for getting back on this.. I have attached few entries from the log and the data pattern I created to capture the 'records' in between the 'start' and 'end' tags.

            Just to test the pattern, I had to use the timestamp present in the 'record' entries. The correct timestamp is present in the 'start' tag and the problem is I don't know how to capture that and group all these records or events at that 'timestamp'..

             

            Regards,

             

            S.B

            • 3. Re: Is it possbile to capture multiline records including timestamp?
              Scott Bleasdell

              Sudeep Bajpai,

               

              This is a little bit tricky - I can completely understand why you had trouble.  I have escalated this to some internal experts in this area for help.  Please bear with us as we try to figure this one out for you.

               

              Scott

              • 4. Re: Is it possbile to capture multiline records including timestamp?
                Sudeep Bajpai

                Thanks Scott... Much appreciated! I was also looking at the 'group' command if it could be used in some form..?

                 

                Also, another example of this type of log is the PatrolAgentCache.dump file found in TS Integration Service Hosts (...\TSIMAgent\Agent\pronto\logs\debug) to test this in ITDA. It has the timestamp at the start of the file and status of the Agents after that.

                 

                Regards,

                 

                SB.

                • 5. Re: Is it possbile to capture multiline records including timestamp?
                  Scott Bleasdell

                  Sudeep Bajpai, this is a bit of a tricky bit of regex, to be sure.  I've got a couple of people looking at this.  Give us some more time to be sure we can do this and get it right for you.

                   

                  Thanks!

                  Scott

                  • 6. Re: Is it possbile to capture multiline records including timestamp?
                    Scott Bleasdell

                    Sudeep Bajpai,

                     

                    Can you try the following data pattern?:

                    start:\s*%{ITDADatePattern736:timestamp}\n*(?:PID:\s*%{Number:pid}\nDW Duration:\s*%{Data:dwDuration}\nstart Check:\s*%{ITDADatePattern736:startCheck}\n(?:Content%{Data:_ignore}\n)*%{Base10Num:wallClockSeconds}\s*wallclock secs\s*\(\s*%{Base10Num:userTime}\s*usr\s*\+\s*%{Base10Num:systemTime}\s*sys\s*=\s*%{Base10Num:cpu}\s*CPU\)\n\s*end:\s*%{ITDADatePattern736:end})?%{MultilineEntry:details}

                     

                    Note, some of the lines in your sample data started with "Content...", such as "Content   check PASSED".  There are multiple of these in between the start and end timestamps.  This pattern does not parse those lines.  It appears that the number of those lines can vary.  If you need those lines parsed, we could do it, but not for an unlimited number of those messages.  If you can give me some guidance on how many of those messages you may see, I can look at updating this data pattern accordingly.

                     

                    Please let me know how well this works.

                     

                    Thanks!

                    Scott

                    1 of 1 people found this helpful
                    • 7. Re: Is it possbile to capture multiline records including timestamp?
                      Sudeep Bajpai

                      Hi Scott Bleasdell

                       

                      Apologies for getting back on this.. I have been trying to get the Team to change the log format to make this easier for us..   so it is under consideration.. (and so is TSI instead of ITDA)

                      Meanwhile to answer your question.. yes this log will have unlimited number of entries with 'Content...' after the 'start' entry. Also, I don't care about the 'end:' entry so what if I exclude that from the pattern and just go with something like this --

                       

                      start:\s*%{ITDADatePattern736:timestamp}\n*(?:PID:\s*%{Number:pid}\nDW Duration:\s*%{Data:dwDuration}\nstart Check:\s*%{ITDADatePattern736:startCheck}\n(?:Content\s\s\scheck\s%{Data:checkStatus}\.\s\s%{Data:reportStatus}\s\s\s\[%{Data:_ignore}\]\s\/nfs\/.*\/Staging\/%{Data:stageMNE}\/%{Data:stageService}\/%{Year}%{MonthNum}%{MonthDay}\/%{Data:stage1C}\-%{Data:stageEnvSvc}\-%{Data:stageRepMNE}\-%{LCH_PortalChecker_DatePattern:timestamp}\_%{Year}%{MonthNum}%{MonthDay}\_%{Data:stageRepFName}\s\-\s%{Data:stageRepLName}\sto\s\/nfs\/report_prod\/%{Data:pubMNE}\/%{Data:pubService}\/%{Year}%{MonthNum}%{MonthDay}\/%{Data:pub1C}\-%{Data:pubEnvSvc}\-%{Data:pubRepMNE}\-%{Checker_DatePattern:pubTime}\_%{Year}%{MonthNum}%{MonthDay}\_%{Data:pubRepFName}\s\-\s%{Data:pubRepLName}\n)*?%{MultilineEntry:details}

                       

                      So ignore the 'footer' after all the 'Content...' entries and let ITDA determine the next 'start'.. entry..

                       

                      4.0431 wallclock secs ( 3.29 usr +  1.28 sys =  4.57 CPU)
                        end: 2017/10/10 01:27:36

                       

                      I'll update on how this goes...

                       

                      Thanks,

                       

                      Sudeep.

                       

                      • 8. Re: Is it possbile to capture multiline records including timestamp?
                        Scott Bleasdell

                        I don't think that is going to work as you expect because ITDA doesn't know to re-read each of those "Content" lines and parse them individually.  It's going to consider the second and subsequent "Content" lines to be part of the MultilineEntry:details field.  With the way ITDA collects data today, you are going to have to define a pattern that defines N number of those "Content" lines .