11 Replies Latest reply on Apr 20, 2010 12:26 PM by Catherine Thomas

    Log file analysis - MRM portal and DMV

      Share This:

      The customer is having an issue with their application Portal.exe.  For this particular issue they are seeing the following:

       

      MRM portal and DMV (This is just an exe that we launch from portal through Process.start and the exe name)- The user is trying to switch between MRM and DMV and MRM freezes.  In this log for the DMV issue the freeze happens in the first half of the Visual log. Approx around Log Id: 107708 you would see a freeze.

       

      For this log they would like to know why the freeze happens.

       

      Thanks,
      Remco Hermes
      Senior Technical Support Engineer
      BMC Software

        • 1. Re: Log file analysis - MRM portal and DMV

          I noticed the following:

           

          There is a 13 seconds delay between lines 107637 and 107649 and following that another 11 seconds delay until line 107653

          Then in line 107775 the Portal.exe exitst with an error, but there is no stack info; the operatgion following that shows that

          ACSC.Portal.Updater.Client.exe is launched - what is the purpose of this module? it takes over 20 seconds before the next operation executes, so it might be interesting to record this EXE as well.

          I will also turn application configuration on for this recording

           

          Ariel

          • 2. Re: Log file analysis - MRM portal and DMV

            The only thing I can see is that they kill the Portal.exe process in line 107775.
            Right before that the process (107767) receives a 503 service unavailable from the server. Maybe this is causing it to wait/freeze.

             

            It might be helpful to trace the server to get more information regarding the 503 event.  Usually a 503 event from the server means that the server is down / IIS is stopped or maybe frozen.

            • 3. Re: Log file analysis - MRM portal and DMV

              In line 107775 portal does not actually exit with an error. The user actually ENDS the process because of the freeze.

               

              The AutoUpdater Client is a program that runs when you launch portal to automatically get updates for the MRM application. It just checks the files on the machine against the server and makes sure that the user has the latest version of MRM portal.

              • 4. Re: Log file analysis - MRM portal and DMV

                That is correct. The user actually ended the process "portal.exe" in line 107775.

                 

                The 503 server Unavailable error happened only on this day. We had a production issue. But this is not an error that appears on every log. Its just a coincidence that on this day we had another issue going on. The 503 error was resolved on the same day.

                • 5. Re: Log file analysis - MRM portal and DMV

                  The same applies for this log file analysis.  I think at this point a new recording on the client and server will be most beneficial in order to continue analyzing this log.  If your server has no more issues then those operations will not show up in the log file anymore and it will be a cleaner log.  Please modify your recording profile when doing a recording on the client and server.  Adding automatically attach to child processes and record application configuration will be very helpful.

                   

                  Please place any related files in a Zip named "Log file analysis - MRM portal and DMV.zip" and upload the Zip file to our Anonymous FTP site; please respond to this email to notify us once the file is uploaded.


                     "ftp://ftp.bmc.com/incoming/"

                   

                  Thanks,
                  Remco Hermes
                  Senior Technical Support Engineer
                  BMC Software

                  • 6. Re: Log file analysis - MRM portal and DMV

                     

                     

                     

                    Hello Catherine,

                     

                    I have not forgotten about this issue.  I am going to forward the logs to Professional Services so they can look at them.  Are you comfortable posting them on the BMC FTP site?

                     

                    Thanks,

                     

                    Remco

                    • 7. Re: Log file analysis - MRM portal and DMV

                      Hello Catherine,

                       

                      Please place any related files in a Zip named "AAA20100415.zip" and upload the Zip file to our Anonymous FTP site; please respond to this email to notify us once the file is uploaded.

                       

                         "ftp://ftp.bmc.com/incoming/"

                       

                      Thanks,

                       

                      Remco

                      • 8. Re: Log file analysis - MRM portal and DMV

                        Hello Catherine,

                         

                        Log analysis has started on your logs, but can you explain why no server recording has been done.  The logs you provided are almost identical when we started this discussion and not a lot of new information has surfaced from these logs.  Doing a client - server recording would most likely give us much more detail.

                         

                        Is it not possible to install a Black Box Service on the server?

                         

                        Thanks,

                         

                        Remco

                        • 9. Re: Log file analysis - MRM portal and DMV

                          We did find some useful data in your logs and we will have to modify your Recording Profile again to capture more data.  We see the

                          10+ second delays in the logs.  I will update you later on today with the details.

                           

                          Remco

                          • 10. Re: Log file analysis - MRM portal and DMV

                            In log..3706 we are able to see operations that take longer than 10 seconds.  Take a look at operation 9732, 9732 and 9741.  The call (Close COM Library) only takes milliseconds, but the actual operation took much longer.  In order to find out more the RPR needs to be modified.

                             

                            1. In AppSight System you can open up the RPR on the active Black Box that does the recording and you need to select the icon on top - Refresh COM Information.  This will take several minutes.  When this is complete you will notice that COM+ Calls and COM Calls are available.  You need to select the COM objects related to the application you are tracing.

                             

                            Screenshot - 4_20_2010 , 9_47_59 AM.jpg

                             

                            2. Since the Close COM Library takes so long it will be very useful to get a stack dump on the Close COM Library operation.  In the RPR being used select Advanced Operations Set => Expand COM Operations => Expand Uninitialize COM Library and set a new alert (stack dump) on Close COM Library.  See attached screen shot.

                            Screenshot - 4_20_2010 , 10_00_32 AM.jpg

                             

                            3. Your dninfo.nef file needs to be updated.  Black Box is unable to record andy framework operations.  Follow instructions in the following BMC Document on how to update your dninfo.nef file: Recording .NET Applications With New Versions Of The .Net Framework

                             

                            After you have completed the above steps you will need to re-record.  The new log files should show us much more information why the Close COM Library is taking so long.

                             

                            Thanks,

                             

                            Remco

                            • 11. Re: Log file analysis - MRM portal and DMV

                              Hi Remco,

                               

                                      Thanks for the quick response and all the useful information. I am going to try these out, create another updated RPR and re record.

                               

                              We also found something on our end that we think might be causing the issue. We are going to work on that as well.

                               

                              Again, Thanks so much for all your help and support in trying to get this issue resolved.

                               

                              Thanks

                              Catherine