1 2 Previous Next 29 Replies Latest reply on Feb 20, 2018 9:41 AM by Michael Evans

    TrueSight v10.7 Agent Deployments

    Daric Smith

      Anyone have experience performing agent upgrade deployments from v10 to v10.7?  We are getting about a 10% success rate with the main problem being the patrolagent.exe is locked by another process.

       

      It isn't documented but on a bunch of test systems in our pre-prod environment I added an exclusion to the A/V software and that fixed it.  This doesn't appear to be working in our Production Environment as we keep getting this and this is after disabling the A/V software completely and rebooting the server.  I have a ticket opened with BMC but checking the community for any tips to make this better.  The process seems far to sensitive for mass use.

       

      Found service: patrolagent

      Stopping service: patrolagent

      Stopping service: patrolagent, waiting 15 seconds

      Stopping service: patrolagent, waiting 30 seconds

      Stopping service: patrolagent, waiting 45 seconds

      Service: patrolagent stopped successfully.

      Installation/upgrade is failed.

       

      The user log contains the following fatal errors:

       

      C:\Program Files\BMC Software\Patrol3/bin\PatrolAgent.exe is in use

       

      The PATROL installer upgrade process failed while installing due to existing files being used by active processes on the system.

       

      Found service: patrolagent

      Starting service: patrolagent

      Service starting: patrolagent, waiting 5 seconds

      Service starting: patrolagent, waiting 10 seconds

      Service: patrolagent started successfully.

      [INFO]: Waiting for 2 mins to bring Patrol Agent services to resume

      [INFO]: More than half of registered Patrol Agent are up & running

      Installation failed (exit code: 181 ). See the RunSilentInstall log file stored in: C:\PROGRA~1\BMCSOF~2\Patrol3\\tmp\Install_Log\2017_06_06_03_42_16_057d5be9-d56f-421c-8211-e013b90a6e67 for detailed information.

        • 1. Re: TrueSight v10.7 Agent Deployments
          Patrick Mischler

          Hi Daric

           

          We have successfully upgraded several PATROL Agents running on Windows.

          Are there multiple agents started?

           

          Regards,

          Patrick

          • 2. Re: TrueSight v10.7 Agent Deployments
            Amitai Bar

            Hi

            Can you please open a support ticket for this?

            Thanks

            Amitai

            • 3. Re: TrueSight v10.7 Agent Deployments
              Maneesh Kumar

              Hi Daric,

               

              I am getting the same errors while upgrading the agents on many Windows servers? I have applied the repository fix pack 001. Tried installing the old package, then created  a new package, but all failed with same errors.

               

              Along with the errors you mentioned above, we are also getting errors on some servers saying C:\Program Files\BMC Software\Patrol3\bin\msvcr100.dll file is in use. Uninstall does not work either. These files cannot be deleted until the server is restarted, which is not the solution.

               

              Have you found a solution for it? Did BMC provide a solution for this?

               

              Thanks,

              Maneesh

               

              Daric Smith

              Amitai Bar

              Mudassir Parwez

              • 4. Re: TrueSight v10.7 Agent Deployments
                Daric Smith

                I mentioned that I have an open support ticket with BMC in the original post.

                • 5. Re: TrueSight v10.7 Agent Deployments
                  Daric Smith

                  I am doing a webex with development early next week to troubleshoot.  Following are all of the things I have tried (and am going to try).  Perhaps you have other troubleshooting that you can add in case I've missed something.

                   

                  • Ran ProcMon during
                    install.  You can see the PML file in the root of D.  I could see the
                    FILE LOCK error but I couldn’t see what was locking it.
                  • Ran Process Explorer
                    checking for open handles to patrolagent.exe
                  • I ran sysinternals
                    “handles64.exe” utility  BUT I was getting access denied viewing the
                    handles owned by SYSTEM.  Maybe you could get that to work.
                    • The package we create doesn't leverage an account so only uses local system.  I have read some things about the SYSTEM account keeping handles open even when the process is dead.  Because it is SYSTEM, you can't view or kill them.
                  • I read stuff about
                    the Application Experience service needing to be set to Automatic.  Didn’t
                    help.
                  • On <servername> I stopped
                    and disabled the following to no avail:
                    • DynamicCache Service
                    • Enstart64
                    • Networker
                    • OfficeScan
                    • VMTools
                    • Windows Defender
                  • I also created a new
                    Dev VM and successfully ran the installer just to make sure it worked.
                    • I have not yet installed the v10 agent then done an
                      upgrade on a new VM.
                  • I was able to
                    successfully deploy the same package to the TDMZ servers (2012).

                   

                  The error seen is:

                   

                  [15:42:03]
                  global_BackupRestoreInstall command is
                  [C:\Users\\Desktop\WinX64_Agent_EDEL_v10_7\bmc_products\Install\instbin\getDiskSpace.bat
                  "C:\Program Files\BMC Software\\Uninstall" "C:\Program Files\BMC
                  Software\Patrol3\tmp" "C:\Program Files\BMC Software" "Patrol3"]

                  [15:42:03]
                  RedirectSTDinoutAndCreateProcess:Created Process
                  [C:\Users\\Desktop\WinX64_Agent_EDEL_v10_7\bmc_products\Install\instbin]

                  [15:42:05]
                  ExecuteScriptNT:external process output = [OUTPUT]C:\Program Files\BMC
                  Software\Patrol3/bin\PatrolAgent.exe is in use

                  [15:42:05] C:\Program Files\BMC
                  Software\Patrol3/bin\PatrolAgent.exe is in use

                  [15:42:05]
                  ExecuteScriptNT:external process output = [OUTPUT] The PATROL installer upgrade
                  process failed while installing due to existing files being used by active
                  processes on the system.

                  [15:42:05]  The PATROL
                  installer upgrade process failed while installing due to existing files being
                  used by active processes on the system.

                   

                  Path to installer log files: C:\Program
                  Files\BMC Software\log

                  • 6. Re: TrueSight v10.7 Agent Deployments
                    Maneesh Kumar

                    Hi Daric,

                     

                    Thanks for sharing the details.

                     

                    I tried to install the agent, but some files including patrolagent.exe were not removed because they were in use. PatrolAgent.exe was in use by DHCP Client process. I stopped the DHCP client process to release the lock but did not work. I tried some other ways to release the lock on the file but nothing worked. Eventually, I had to restart one of the servers, which released the lock on the files, then reinstalled the agent. But its not possible to restart the production servers just to upgrade the agent.

                     

                    I faced similar issue with msvcr100.dll file being in use.

                     

                    The issue seems related to how the upgrade is handled in the installation utility of 10.7 repository. I have never faced these issues with agent upgrade before.

                     

                    Thanks,

                    Maneesh

                    Daric Smith

                    Mudassir Parwez

                    • 7. Re: TrueSight v10.7 Agent Deployments
                      Daric Smith

                      Oh, that happened to us: It said the DHCP Client was locking the process.  We found it to be a red herring because when we stopped the DHCP Client, it would go on to another service and stopping that would tell us another service.

                       

                      There is definitely something on the OS side that is causing the lock.

                       

                      I have built a 2008 R2 VM and so far have installed a bunch of our utilities such as backup, forensic and A\V but have not run across the problem.

                       

                      The final step is to fully Windows Patch the server to see if that breaks it.

                      • 8. Re: TrueSight v10.7 Agent Deployments
                        Daric Smith

                        What I am really curious about: How many deployments have been done by BMC in their on corporate network?  I know they QAed it in the development but are they actually using v10.7 in their production environment that has thousands of nodes?  It can't just be us that are experiencing this in our production environments.

                         

                        Andy Abrams

                        • 9. Re: TrueSight v10.7 Agent Deployments
                          Andy Abrams

                          Daric,

                           

                          BMC-IT has been running TrueSight 10.7 in Production since April 20th (~24 days after it was released GA).  I will check with the Assurance team manager to see if they have had any of the issues upgrading PATROL Agents as described in this thread.

                           

                          -Andy

                          • 10. Re: TrueSight v10.7 Agent Deployments
                            Daric Smith

                            Thank you very much for the response Andy.  If  you could provide some more information on how your agents are handled as it will help the BMC support and engineers to figure out why Maneesh and I have this issue:

                             

                            1. Are 100% of the agents deployed in your environment created through the TrueSight Repository (so not using the old Patrol installer utility)?
                            2. Have you converted most of your agent configurations over to Infrastructure Policies or is it still mostly done through PCM?
                            3. Have you attempted v10.7 agent deployments through TrueSight to a large number of production systems?
                            4. What deployment methods to you leverage for your agents (manual install / Blade Logic)?

                             

                            With the exception of Monitoring Studio that still requires some level of PCM and that is only on about 4 nodes out of 1200, everything we configure is through TrueSight.

                             

                            Again, I am trying to gather as much data as possible to determine the differences and why you environment with thousands more nodes wouldn't experience this at all.

                            • 11. Re: TrueSight v10.7 Agent Deployments
                              Daric Smith

                              New information breakthrough for this as I've worked with support.

                               

                              I learned something new for running Process Explorer as Local System: Using psexec with the switches -i -s -d will run the target process as Local System.

                               

                              So, executing: psexec -i -s -d procexp.exe will open it and now we can see all of the information for open handles.

                               

                              Now, I can see that svchost -k LocalServiceNetworkRestricted is what keeps the handle open:

                               

                              Locking Process.png

                               

                              This also matches up with the "services" that were keeping the process open: DHCP, Event Log and TCP/IP Helper:

                               

                              sRegisteredServices.png

                               

                              Can you see if you get the same results on a troubled server?

                               

                              We are now waiting for BMC to provide more information on why patrolagent.exe handle gets locked for that svchost.

                              • 12. Re: TrueSight v10.7 Agent Deployments
                                Maneesh Kumar

                                Hi Daric,

                                 

                                Thank you very much for sharing this information. I can confirm that we have the exact scenario which you have described.

                                 

                                Maneesh Kumar

                                • 13. Re: TrueSight v10.7 Agent Deployments
                                  David Kokes

                                  Hi Daric,

                                   

                                  Thank you for the useful information on this. We are also experiencing same issue as you (installation fails because patrol process is being used).

                                   

                                  I have also some other thing. If I run the update on more agents (for example 100) it will start with first batch of 40 and once that is done it will continue with another 40 and then it will do the 20. It will not run simultaneously on all of them. Info from support is that it should be possible to run 500 simultaneously. Have you seen this?

                                   

                                  Thank you

                                  • 14. Re: TrueSight v10.7 Agent Deployments
                                    Patrick Mischler

                                    Hi all,

                                     

                                     

                                    The maximum numbers of paralell deployments can be configured in the tssh properties. For example 'tssh properties list' does return this value:

                                     

                                    MAX_PARALLEL_PA_DEPLOYMENT                                40

                                     

                                    A value of '40' seems to be the default.

                                    You can set this to another value witht the following command:

                                     

                                    tssh properties set MAX_PARALLEL_PA_DEPLOYMENT 500

                                     

                                    But I am not sure what BMC recommends. To be sure, ask the support.

                                     

                                    Regards,

                                    Patrick

                                    1 2 Previous Next