10 Replies Latest reply on Nov 17, 2014 3:35 PM by Jim Campbell

    Auto-reboot unable to initiate reboot

    Jim Campbell

      Checking to see if anyone else has this problem with bldeploy job reboots on Windows.  As a preface, this is a problem we have had for the past 2+ years that occurs randomly and is not reproducible.  It occurs with any bldeploy that has an item with Reboot='After Item Deployment'.  We have seen this error with agent versions 8.0sp7patch4 and 8.1sp3patch1.  It occurs (again, randomly) on Windows 2003x86, Windows 2003x64, Windows 2008x86, Windows 2008x64, and Windows 2008R2.  My experience is somewhat limited, but I have yet to see it occur on Windows 2012 (where we use an unsupported configuration of 8.2sp2 agents with an 8.1sp3 application server).  The only information I could find was a KB article that indicated the problem could be with early 8.0 agents on Windows 2008 servers - however, we see this issue with all of the OSes we typically build and with agents that are from 8.1.


      Our jobs run primarily using User Principal mapping with the bladelogicrscd user mapped to the local administrator.  However, the problem also occurs with seemingly the same frequency with the subset of jobs we run as domain users with an automation principal (all of which are members of the local administrators group).  Again, this problem is not reproducible - a second attempt of the job with no changes will almost always succeed.  Our standard 'fix' for users has always been 'just re-run the job using execute against failed target' and this almost invariably works.  Nothing needs to be changed for the second attempt to work; the job just has to be re-run.


      The error message is always the same:


      Error Jun 7, 2013 6:47:03 PM Auto-reboot unable to initiate reboot, please reboot manually. Error code:5 Access is denied.

      Error Jun 7, 2013 6:47:04 PM  Could not reboot as required

      Error Jun 7, 2013 6:47:04 PM  Failed to reboot for item EXTERNALCMD


      The transaction log looks like:


      06/07/13 18:47:03.975 DEBUG    bldeploy - Successfully paused the agent before shutting down the system
      06/07/13 18:47:03.991 INFO     bldeploy - Attempting shutdown REBOOT=true TIMEOUT=0 MSG=System is rebooting
      06/07/13 18:47:03.991 ERROR    bldeploy - Auto-reboot unable to initiate reboot, please reboot manually. Error code:5 Access is denied.

      06/07/13 18:47:03.991 DEBUG    bldeploy - Reset process token privleges
      06/07/13 18:47:04.022 ERROR    bldeploy - [3][Run Install] Could not reboot as required
      06/07/13 18:47:04.037 DEBUG    bldeploy - Metabase initialization skipped because it was not needed
      06/07/13 18:47:04.037 ERROR    bldeploy - [3][Run Install] Failed to reboot for item EXTERNALCMD


      This problem does not occur with great frequency but given the number of reboots we have in our build process it can be annoying when building large numbers of servers.

        • 1. Re: Auto-reboot unable to initiate reboot
          Siddu angadi

          HI Jim,


          There are so many reasons.  As you said, if it is happening randomly, if it better to rerun against servers.


          The reasons may be:


          Server is logged with other users, current user may not have permission to kill other user's active login

          any other applications/ patch / any other setup is running in parallel





          • 2. Re: Auto-reboot unable to initiate reboot
            Bill Robinson

            do you see anything in the event logs that correspond to the access denied ?

            • 3. Re: Auto-reboot unable to initiate reboot
              Jim Campbell

              Nothing relevant in application/system logs.  Security log shows 2 identical failure entries at the same time:


              An operation was attempted on a privileged object.

              Security ID:  SERVER_NAME\BladeLogicRSCD
              Account Name:  BladeLogicRSCD
              Account Domain:  SERVER_NAME
              Logon ID:  0x85b7e

              Object Server: Win32 SystemShutdown module
              Object Type: -
              Object Name: -
              Object Handle: 0x0

              Process Information:
              Process ID: 0x2b0
              Process Name: C:\Windows\System32\wininit.exe

              Requested Operation:
              Desired Access: 0
              Privileges:  SeShutdownPrivilege

              • 4. Re: Auto-reboot unable to initiate reboot
                Yanick Girouard

                The SeShutdownPrivilege priviledge corresponds to the "Local Policy/User Right Assignment/Shut down the system" policy. I would check that policy on the servers in question and make sure the user BladeLogic maps to is in the list of allowed users (member of one of the groups listed).


                You can launch secpol.msc to view this.


                Alternatively you can Live Browse the servers through the automation console and look here:


                • 5. Re: Auto-reboot unable to initiate reboot
                  Jim Campbell

                  The administrators group has that permission set in policy.  All of the users that generate this random error are members of the administrator group (usually the bladelogicrscd user with User Principal Mapping to the local administrator account, but also occurs with automation principals for domain users that are members of domain groups that are included in the local administrator group).


                  Also again, this problem is always resolved by simply re-running the job with no other action taken (i.e. no intervention on the server itself or any change in the agent user mapping).  If there is a permissions issue it is obviously not systemic as it only rarely occurs and only intermittently.

                  • 6. Re: Auto-reboot unable to initiate reboot
                    Steven Wyns

                    Hi Jim,


                    Maybe you've found a solution already?


                    We see the same problem since we began with BL (on 7.6). Altough it is much better since we're on 8.2.4 we still occasionally see the issue. And indeed the rerun works perfectly almost every time.


                    We use agents.

                    • 7. Re: Auto-reboot unable to initiate reboot
                      Jim Campbell

                      No I have not found a solution.  Given that there is no way to reproduce the error on demand the issue is really impossible to troubleshoot.  Any ticket would just be a waste of time proving that it wasn't a systemic error from the common causes of permissions failures so we have just learned to live with it.

                      • 8. Re: Auto-reboot unable to initiate reboot
                        Robert Woodward

                        I am seeing the exact same issue in our environment - very hard to debug as a retry works fine.  Any updates on this issue?

                        • 9. Re: Auto-reboot unable to initiate reboot
                          Steve Cupp

                          Put me down as seeing the same type of reboot problems and I must say that it is reeking havoc with our more complex software installation stacks, i.e. XenApp 6.5. What I have to do sometimes is manually reboot the target to get the jobs once again able to communicate with the target.


                          Could the shutdown tracker be somehow related to the reboots not being performed?

                          • 10. Re: Auto-reboot unable to initiate reboot
                            Jim Campbell

                            Xenapp 6.5 installation is one our primary culprits because of the large number of reboots though we do see this elsewhere.  We disable the shutdown tracker so it doesn't seem to be the problem.  I'm not sure its a 100% correlation but it does seem to occure more frequently when the server is rebooted rapidly in sequence ( as is the case with enabling roles prior to installing Xenapp 6.5 ).