5 Replies Latest reply on Aug 4, 2014 3:04 PM by Daniel Goetzman

    How do you test BSA / BDSSA upgrades ?

    Steffen Kreis

      Hi guys,


      just wanted to collect some input here on how you are testing BladeLogic or BDSSA upgrades for your PROD environments.


      We run in total up to 4 BSA environments (LAB / DEV / TEST / PROD), which we upgrade one after the other when we are going to a new release.

      Some content (SysPackages / Component Templates / ...) might be the same in these systems, but the real targets that we manage are only registered in the PROD system. Due to that fact, only PROD contains a huge amount of data (Job-Runs, Jobs created by the BSA users and and and)


      So testing an upgrade in the Non-Prod environments is no real test if the upgrade in Prod will work.


      In some occasions we have created a dump of the PROD DB and tweaked the names and stuff so we can at least simulate a Data-Migration Manager with it, but we still wouldn't be able to fire up such a migrated system and test the result against targets etc.


      Only yesterday we ran into an issue were the ETL is failing on the PATCH scenario after we upgraded the PROD BDSSA to 8.5.1. Just because of the huge amount of data in the PROD BSA Database which we couldn't simulate beforehand when testing the BDSSA upgrdaes in DEV, Test etc.


      So i just wanted to ask around how you get around that issue ?




        • 1. Re: How do you test BSA / BDSSA upgrades ?
          Jonas Valkiunas

          This has less to do with BSA/BDSA and more with test methodology.  It's usually a good practice to have your pre-prod environment match prod.  In the case of LAB->DEV->TEST->PROD, TEST should have the same hardware and data as PROD.

          • 2. Re: How do you test BSA / BDSSA upgrades ?
            Steffen Kreis

            I fully agree,


            However the challenge with BSA is the fact that we manage a lot of real world targets, which we can't do in any other stage than PROD.



            • 3. Re: How do you test BSA / BDSSA upgrades ?
              Daniel Goetzman

              We have this same challenge here at BMC IT!


              Our current process is to clone/refresh both DEV and QA to be the SAME as PROD, this allows us to upgrade, test and qualify upgrades targeted towards PROD using as close to the same total configuration as we are able. All the target servers are defined in our DEV and QA, as it is a clone of PROD. However...


              To prevent DEV and QA from impacting the same set of target servers, our datacenter targets, we do the following;

              • Delete ALL scheduled JOBs in DEV and QA in the OM Core DB BEFORE starting up DEV or QA
              • Set IS_ONLINE=false on ALL target servers in DEV and QA

              This pretty much blocks DEV and QA from having any change impact to the same set of target servers as defined in our PROD BSA. We run some basic functionality tests that represent all the common use cases we use in both DEV and QA, and this is possible by setting IS_ONLINE=true on a select set of identified test servers that we allow to be live targets in the DEV and QA envs. As DEV and QA are a clone of PROD, we have all the exact same jobs we can run if needed to test exactly that same automation we use daily in PROD.


              Once we have upgraded DEV and QA, run our basic functionality test and have qualified to a large degree confidence we can upgrade PROD, then we are able to schedule a CR and provide acceptance test results required to gain change management approval to upgrade our prod env.


              So, that's how we do it here!


              • 4. Re: How do you test BSA / BDSSA upgrades ?
                Steffen Kreis

                Hi Dan,


                thx for your input on that one.

                I like to idea of the IS_ONLINE property and the deletion of the schedules.


                When you say "clone". What exactly do you do ?

                Cause over here we are unable to keep hostnames the same when doing such a thing.

                So we probably would need to change File-Server locations in the DB etc.



                • 5. Re: How do you test BSA / BDSSA upgrades ?
                  Daniel Goetzman

                  I left that part out...

                  We use a common name for our File Server name, for example you could use blfilestore...

                  To allow each node direct access we simply define it as an alias in /etc/hosts (we run Linux app servers) so the exact same hostname (DNS end point name to be more accurate) will be valid and work on all app server nodes. Using the local direct connection based on the local /etc/hosts definition of the common "blfilestore" hostname!