4 Replies Latest reply on Jun 26, 2014 7:52 AM by Bill Robinson

    Optimizing our Patch Cycle

    Mark Casey

      Hello all!

       

      Here is my latest question of the week.   

       

      I'm currently working on our currently patching procedures, with the end goal being to turn much of the monitoring and escalation of any issues during the actual Maintenance Window over to our Operations Department. I've piloted the new BladeLogic Portal application, and while a very good concept and on paper would fit our need rather well, I couldn't get it to work the way I needed to in the short window I have to finish this project.

       

      I've been pouring over all the Best Practices documents and videos out there, searching for ideas, but I'm still confused. First a little about our environment.

       

      We have a handful of Windows Admins, who all handle patching for their respective customers.  Each tech does their patching a little differently.  Some use very specific Patch Jobs with specific targets identified, while others have very generic jobs, that they just re-run each time on the different target groups as needed.  I originally thought I would be able to baseline all the patch jobs into one or two, and then run specific Execution Tasks pointing to the individual target groups, each with a easily identifiable name, but that wouldn't work either.

       

      A lot of months, we have other items we lump into our patching windows, like agent updates, virusscan agent updates, etc.  That got me going down the path of lumping them all into a batch job and kicking it off that way, but I think that has it's downfalls also.  For instance, would I still be able to run the Analyze, simulate and stage the patches, and then run the other needed product updates, and then come back after that's all done and kick off the commit? 

       

      As you can see, my head is whirling with all the possibilities out there in BSA, but since I'm reinventing the whole process from the ground up, I want to make sure I do it in a way that is simple and easy to understand.

       

      Also, when it's all said and done, the managers and other folks up there in the tower, want to be able to have documented proof of the success and/or failure of these jobs, and that's causing me some pain also.  I can't seem to find the perfect report to give them what they need, and I'm in no way a BDSSA guru, so trying to build the report from scratch is proving daunting as well. 

       

      Any ideas out there?

       

      Thanks,

      Mark

        • 1. Re: Optimizing our Patch Cycle
          Bill Robinson

          Why wouldn’t the ETs work ?

           

          What was the issue getting the portal to work ?

           

          Do you want the users to use the portal or the bsa gui ?

           

          So your goal is to provide users w/:

          - a way to execute their patching jobs, over and over again, against the same target servers?

          - and a way to execute their patching jobs w/ an ad-hoc target list ?

           

          What issues are you having building the report?  what specifically are you trying to show here?  there should be some out of box reports that are patching ‘by policy’ where the policy is the job name.  is that a starting point and you would just show all of the patching jobs ?  or do you want the latest patching results across all your servers no matter what ‘policy’ was used ?

          • 2. Re: Optimizing our Patch Cycle
            Mark Casey

            Thanks for you response Bill, my answers are below:

            Bill Robinson wrote:

             

            Why wouldn’t the ETs work ?

            When I was looking at using ET's to do the work, I thought I hit a snag at a point where they wouldn't work, but I can't seem to remember what it was right now.  I'm going to test it again and see if I can make that work.  The batch job seemed to be the better scenario because we could control the entire window more then using ET's. Like scheduling other installs of clients, before finally following up with the patching itself. But the question came up, would it be possible to wrap up the whole process in a batch, but all keep the capability to simulate and scan ahead of time to keep our times down.

             

             

            What was the issue getting the portal to work ?

             

            Well, as with any 1.0 product, we hit a couple snags. I had no issues standing up the product and getting it to work with the BLAdmins account, but when I started provisioning the Operations users, I had a heck of a time getting the patching to work. It seemed we would hit a road block, clear that, and then immediately hit another one. At first, I couldn't get them to be able to run the jobs. Then once we fixed that issue, I was able to launch jobs through the BSA Console, but if I ran that same analysis job on the console, it would run fine, but when I went to the results to install the patches, I could not see the "Actions" button on any of the screens when logged in as the user. After hitting a couple snags, my boss put my off of the Portal and wanted me to move ahead with getting them in to the GUI.

             

             

            Do you want the users to use the portal or the bsa gui ?

             

            Well, I thought the Portal would be the perfect solution, but I'm not going to lie, there seemed to be a couple shortcomings we were prepared to work around. For instance, we have quite a few operators covering the 3 shifts, and not to mention quite a few patching windows, which are scattered all over the month. To have the operator run an operation, there was no way to "pre-populate" the console with the jobs they would need to run. Since the job definitions are per user, we would have had to type up instructions for each operation we wanted them to run, so they could create the operation the first time for future runs. Not a huge deal, but would have been nice to set all that up for them beforehand. Ideally, I thought the simpler interface would be easier to train them on, but I'm working under a time constraint from management and didn't have the time to spend to try to iron out all the issues unfortunately. I'm still open to the possibility of using it down the road.

             

             

            So your goal is to provide users w/:

            - a way to execute their patching jobs, over and over again, against the same target servers?

            - and a way to execute their patching jobs w/ an ad-hoc target list ?

             

            Well, the above is more of an OR then an AND statement. If we used the same target servers for the jobs, we would have to create a job for each server group we needed to patch, which is upwards of probably 20 to 30. On the other hand, if we created a couple baseline Windows Patching Jobs, that we could reuse and just change the targets, that would probably end up being a solution also. Not sure of the best or cleanest way to do it.

             

            What issues are you having building the report?  what specifically are you trying to show here?  there should be some out of box reports that are patching ‘by policy’ where the policy is the job name.  is that a starting point and you would just show all of the patching jobs ?  or do you want the latest patching results across all your servers no matter what ‘policy’ was used ?

            What the boss was looking for was a report that would be run to show the results of the job, kind of like a verification that everything happened the way it should. And if Operations had any issues, it could be followed up on by an analyst after the fact. We did some brainstorming yesterday and ended up going into the Windows Patching Job, expanding the Remediation Run, and then the Deploy Run, and exporting the list from there. This gave us a high level indication that the job ran and what the results for each server, as well as where those servers stood in the process. As I'm typing this, I'm guessing the server would show "Reboot Required" if we manually rebooted the servers outside of the Patching Job though, so we would have to also have a report to make sure the group of servers all got rebooted. I believe Sean Berry mentioned in his Patching Best Practices Video, that there was a way to create a Component Template to view servers reboots in 1, 4 and 24 hour increments. Will have to follow up on that to see if that would solve that problem.

             

            Sorry for the long winded response. Thank you for taking the time to help me with this!

             

            -Mark

            • 3. Re: Optimizing our Patch Cycle
              Mark Casey

               

              Why wouldn’t the ETs work ?

               

              I remembered why I was having issues using ETs. When creating them, I'm not able to edit the Deploy Options of the Windows Patching Job. Because we have such a wide variety of Patching Windows, I'm not sure how I would go about creating all those in a way that would still allow me to Simulate and Stage prior to the beginning of the window.

              • 4. Re: Optimizing our Patch Cycle
                Bill Robinson

                The concern w/ having a set of baseline jobs is that if you have multiple people trying to use those jobs at the same time and editing the targets and options they will step on each other.  so it would be better to have 1 job per target group.  that way each admin should be able to setup the job the way they want for those target.  and it may be easier to report on if you need to report on each group of servers.