I mentioned that I have an open support ticket with BMC in the original post.

I am doing a webex with development early next week to troubleshoot.  Following are all of the things I have tried (and am going to try).  Perhaps you have other troubleshooting that you can add in case I've missed something.

• Ran ProcMon during
  install.  You can see the PML file in the root of D.  I could see the
  FILE LOCK error but I couldn’t see what was locking it.
• Ran Process Explorer
  checking for open handles to patrolagent.exe
• I ran sysinternals
  “handles64.exe” utility  BUT I was getting access denied viewing the
  handles owned by SYSTEM.  Maybe you could get that to work.
  • The package we create doesn't leverage an account so only uses local system.  I have read some things about the SYSTEM account keeping handles open even when the process is dead.  Because it is SYSTEM, you can't view or kill them.
• I read stuff about
  the Application Experience service needing to be set to Automatic.  Didn’t
  help.
• On <servername> I stopped
  and disabled the following to no avail:
  • DynamicCache Service
  • Enstart64
  • Networker
  • OfficeScan
  • VMTools
  • Windows Defender
• I also created a new
  Dev VM and successfully ran the installer just to make sure it worked.
  • I have not yet installed the v10 agent then done an
    upgrade on a new VM.
• I was able to
  successfully deploy the same package to the TDMZ servers (2012).

The error seen is:

[15:42:03]
global_BackupRestoreInstall command is
[C:\Users\\Desktop\WinX64_Agent_EDEL_v10_7\bmc_products\Install\instbin\getDiskSpace.bat
"C:\Program Files\BMC Software\\Uninstall" "C:\Program Files\BMC
Software\Patrol3\tmp" "C:\Program Files\BMC Software" "Patrol3"]

[15:42:03]
RedirectSTDinoutAndCreateProcess:Created Process
[C:\Users\\Desktop\WinX64_Agent_EDEL_v10_7\bmc_products\Install\instbin]

[15:42:05]
ExecuteScriptNT:external process output = [OUTPUT]C:\Program Files\BMC
Software\Patrol3/bin\PatrolAgent.exe is in use

[15:42:05] C:\Program Files\BMC
Software\Patrol3/bin\PatrolAgent.exe is in use

[15:42:05]
ExecuteScriptNT:external process output = [OUTPUT] The PATROL installer upgrade
process failed while installing due to existing files being used by active
processes on the system.

[15:42:05]  The PATROL
installer upgrade process failed while installing due to existing files being
used by active processes on the system.

Path to installer log files: C:\Program
Files\BMC Software\log

Oh, that happened to us: It said the DHCP Client was locking the process.  We found it to be a red herring because when we stopped the DHCP Client, it would go on to another service and stopping that would tell us another service.

There is definitely something on the OS side that is causing the lock.

I have built a 2008 R2 VM and so far have installed a bunch of our utilities such as backup, forensic and A\V but have not run across the problem.

The final step is to fully Windows Patch the server to see if that breaks it.

What I am really curious about: How many deployments have been done by BMC in their on corporate network?  I know they QAed it in the development but are they actually using v10.7 in their production environment that has thousands of nodes?  It can't just be us that are experiencing this in our production environments.

Andy Abrams

Thank you very much for the response Andy.  If  you could provide some more information on how your agents are handled as it will help the BMC support and engineers to figure out why Maneesh and I have this issue:

1. Are 100% of the agents deployed in your environment created through the TrueSight Repository (so not using the old Patrol installer utility)?
2. Have you converted most of your agent configurations over to Infrastructure Policies or is it still mostly done through PCM?
3. Have you attempted v10.7 agent deployments through TrueSight to a large number of production systems?
4. What deployment methods to you leverage for your agents (manual install / Blade Logic)?

With the exception of Monitoring Studio that still requires some level of PCM and that is only on about 4 nodes out of 1200, everything we configure is through TrueSight.

Again, I am trying to gather as much data as possible to determine the differences and why you environment with thousands more nodes wouldn't experience this at all.

New information breakthrough for this as I've worked with support.

I learned something new for running Process Explorer as Local System: Using psexec with the switches -i -s -d will run the target process as Local System.

So, executing: psexec -i -s -d procexp.exe will open it and now we can see all of the information for open handles.

Now, I can see that svchost -k LocalServiceNetworkRestricted is what keeps the handle open:

This also matches up with the "services" that were keeping the process open: DHCP, Event Log and TCP/IP Helper:

s

Can you see if you get the same results on a troubled server?

We are now waiting for BMC to provide more information on why patrolagent.exe handle gets locked for that svchost.