12 Replies Latest reply on Jun 23, 2014 10:48 PM by Deepak Rajput

    Server got rebooted after patch deployment

    Deepak Rajput

      Hi Team,

       

      I have deployed the patches on 35 server for month of April and May 2014 Below are the patches.

      Q2936068,Q2922229,Q2964358,Q2928120,Q2926765,Q2953522,

       

      Out of 35 servers 25 servers were rebooted automatically. Even afer selecting "Ignore item defined reboot setting"

       

      And on few the windows system log says the patches is not applicable still BBSA install that patches(Q2926765).

      Please find the attached log and snapshot from rebooted servers and BBSA console.

       

      I know this not happened by BBSA please tell me how i can prove the Windows sysadmin guys. There are just trying push the blame on Tools

       

      Find the attached screenshot given by sysadmin.

       

      In bldeploy job it says that manual reboot required. The servers have rebooted after 12hrs of patch deployment.

        • 1. Re: Server got rebooted after patch deployment
          Jim Wilson

          The actions performed by BSA Patch Deploy will be logged in the <rscd_install_dir>\RSCD\Transactions\log

          The log files are bldeploy*.log and bltargetjobmanager*.log

           

          You will be able to see if a reboot occurred as part of the deploy processing, but it doesn't look like it was initiated by BSA.

           

          The action of installing the patches but not allowing the option to reboot if required may have cause the "Reboot Pending" to be set, which has then caused another process to perform the reboot.

           

          Please be aware that if a patch requires a reboot, this should be completed before attempting any other patch activity (analysis or remediation)

          1 of 1 people found this helpful
          • 2. Re: Server got rebooted after patch deployment
            Bill Robinson

            There’s another setting on the deploy job – ‘ignore copy on boot files’ that will also govern the reboot for windows.  what was that set to.  and we’ve seen some cases where the patch will initiate a reboot no matter what we tell it – we should be able to figure out if that’s happening in the bldeploy log jim mentioned.

            1 of 1 people found this helpful
            • 3. Re: Re: Server got rebooted after patch deployment
              Deepak Rajput

              Hi,

              Please find the attached bldeploy log and screenshot of "ignore copy on boot files".

              • 4. Re: Re: Server got rebooted after patch deployment
                Jim Wilson

                No attachment found :-(

                • 6. Re: Server got rebooted after patch deployment
                  Steffen Kreis

                  Hi Bill,

                   

                  you mentioned that the ‘ignore copy on boot files’ also has an influence if a server reboots or not.

                   

                  That confused me a little, as i thought this setting would cause the "Commit" not to begin in case the server has a Pending Reboot.

                   

                  Steffen

                  • 7. Re: Re: Server got rebooted after patch deployment
                    Bill Robinson

                    do you have the bldeploy log ?

                    • 8. Re: Re: Server got rebooted after patch deployment
                      Bill Robinson

                      'ignore copy on boot' in the job prevents the job from performing a reboot due to the registry key being present (from the thing blade just installed or pre-existing).  for the RSCD installer you can pass an option to have it ignore a system w/ the pending reboot state set.  it's likely other MSI installers either ignore that or have a similar option.

                      • 10. Re: Server got rebooted after patch deployment
                        Bill Robinson

                        So this is from a system the rebooted on its own?

                         

                        In the logs I see that a reboot was denied:

                         

                        06/13/14 19:05:53.307 DEBUG    bldeploy - [Windows6.0-2008-SP2-KB2922229-x64.msu-MS14-019-en-WINDOWS SERVER 2008, ENTERPRISE EDITION (X64)-SP2] In RunProcess: exitCode = 3010

                        06/13/14 19:09:28.986 DEBUG    bldeploy - [Windows6.0-2008-SP2-KB2936068-x64.msu-MS14-018-en-INTERNET EXPLORER 7 (X64)-GOLD] In RunProcess: exitCode = 3010

                        06/13/14 19:16:03.931 DEBUG    bldeploy - [Windows6.0-2008-SP2-KB2964358-x64.msu-MS14-021-en-INTERNET EXPLORER 7 (X64)-GOLD] In RunProcess: exitCode = 3010

                        06/13/14 19:22:41.371 DEBUG    bldeploy - [Windows6.0-2008-SP2-KB2953522-x64.msu-MS14-029-en-INTERNET EXPLORER 7 (X64)-GOLD] In RunProcess: exitCode = 3010

                        06/13/14 19:29:38.999 DEBUG    bldeploy - [Windows6.0-2008-SP2-KB2926765-x64.msu-MS14-027-en-WINDOWS SERVER 2008, ENTERPRISE EDITION (X64)-SP2] In RunProcess: exitCode = 3010

                        06/13/14 19:29:40.045 DEBUG    bldeploy - Bldeploy done - nRet = 1 (Apply successful; Reboot required to complete) exitCode = -4002 (Deployment succeeded, but requires manual reboot)

                         

                        I don’t see a reboot happening in that deploy. I would expect to see some errors.  I do see some long gaps there in the logs – a few ~5min gaps but those seem related to a patch install.

                         

                        For the issue of Q2926765 being installed when it was not applicipable we’d need to see the trace.txt from the analysis.  Do you still have any systems that show that patch as missing ?

                        • 11. Re: Re: Server got rebooted after patch deployment
                          Yanick Girouard

                          You already have the proof with the original screenshots you provided. The Windows event log shows that the user account that rebooted the system is the SYSTEM account. Any command run through the BladeLogicRSCD agent will be run as the BladeLogicRSCD unprivileged account (with privilege mapping to local administrator account), unless you changed the name of the service user in the registry, or that you are using automation principals. If you run an agentinfo command against the agent, you will see the name of the account it's using in the User Permissions line:

                           

                          Example:

                           

                          User Permissions: BladeLogicRSCD@COMPUTER_NAME->ADMIN_ACCOUNT@COMPUTER_NAME:PrivilegeMapped (Identity via trust)

                           

                          You can't use the SYSTEM account with automation principals, so that rules that out, you also cannot use the SYSTEM account as the unprivileged account for the BladeLogic RSCD service. So... it's not BladeLogic and cannot be. If the sysadmin argues that the RSVDscv service (the watchdog process) and RSCD.exe (listener) runs as SYSTEM, explain him that these do not call or spawn commands as SYSTEM, but always as the BladeLogicRSCD (or automation principal) account, and the watchdog process's only role is to restart the listener, so that does not spawn any commands other than RSCD.exe.

                           

                          Furthermore, during a patch deployment, the account running the actual MS patch files is also the BladeLogicRSCD account, and not SYSTEM. So even if one of the patches would override the "ignore item defined reboot settings" option somehow, and reboot the system anyway, the Windows Event Log would show BladeLogicRSCD as the account that started it, and not the SYSTEM account. and it would look like this in the event log:

                           

                           

                          On top of it, you say it happened 12 hours after the patch deployment was completed with BladeLogic. Simply use the Activity view of the server (in Live Browse) and look at all jobs that run against it between the patching deployment and the reboot and see if any jobs in there could potentially initiate a reboot. Chances are you will not find anything there and that will be your proof.

                           

                          Seriously, I think you have more than enough proof already that BladeLogic did not cause this, and if your sysadmin still doesn't believe it, have him prove that it was BladeLogic. he won't be able to.