1 2 Previous Next 23 Replies Latest reply on May 18, 2018 9:37 AM by Sandeep Nagi

    9.1.04 upgrade complains about previous steps

    Peter Lundqvist

      First of all, sorry if this should be posted in another group, but the issue seems to cover a few products.

      I am upgrading 9.1.03.001 to 9.1.04 using a staged upgrade on RHEL 7.4/Oracle 12c.

       

      The system I am upgrading has two AR servers in a server group, the primary server upgrades just fine (ARS, Atrium, RSSO, ITSM, SLM, SRM).

      The secondary server starts out fine up to ITSM. The ITSM upgrade aborts and complains:

       

      (Apr 25 2018 04:44:00.928 PM +0200),SEVERE,com.bmc.install.product.bmcremedyitsmsuite.core.tasks.BMCRemedyITSMSuiteApplicationDependencyTask,

        LOG EVENT {Description=[Complete the BMC Remedy AR System, BMC Atrium CMDB, and Atrium Integrator installation on all the servers. Make sure BMC Remedy AR System upgrade status is 'Done' on AR Server Information Page, before proceeding with BMC IT Service Management installation.],Detail=[Complete the BMC Remedy AR System, BMC Atrium CMDB, and Atrium Integrator installation on all the servers. Make sure BMC Remedy AR System upgrade status is 'Done' on AR Server Information Page, before proceeding with BMC IT Service Management installation.]}

       

      The complaint is correct. the system upgrade status is 'Pending'. But in CCS it is 'Done'. And in the database (control table) it is 'Pending'.

      Checking the RIK logs, it would seem that everything else is OK. I have tried modifying the value manually - it did not help, I got the same message (which is really curious to me).

       

      I have a support ticket going for this issue, but I figured it would be worth checking here if anyone else had run into this or a similar issue.

        • 1. Re: 9.1.04 upgrade complains about previous steps
          Ashutosh Deshpande

          Hi Peter,

           

          In 9.1.04, zero dowtime upgrade for AR Platform (AR, CMDB, AI) with automatic rollback was implemented. And unless AR Platform all the servers in server group is upgraded, ITSM upgrade cannot be performed. The check you are seeing for upgrade status is this check.

           

          Now, the reasons it can remain as "Pending" can be:

          1. Primary server checks for AR, CMDB and AI version for all servers in the server group. Ranking form has new fields for this in 9.1.04. If your system has some dead / unaccounted servers in the ranking form and their version still shows up as null or older version, that's a problem. If they are really unused server and never come back in 48 hours since primary server restart, system will mark them as dead and will ignore those and perform some post-upgrade tasks and then mark status as done.

           

          2. If you have all the servers accounted for and upgraded to reflect correct 9.1.04 version for all 3 fields in all entries of the ranking form, post-upgrade processing starts and it may take some time. This performs things like ft_pending optimization and unique index creation, sets some configurations, etc. that can otherwise not be performed if even one of your AR server is older version than 9.1.04.

          To check this, your arerror.log should show log statements prefixed with "[Post Upgrade]" word. It will show you the progress or error, if any.

           

          Once all of above is done, the upgrade status is marked as "Done". Only then ITSM installer will continue.

           

          Hope this helps.

           

          Thanks and Regards,

          Ashutosh.

          3 of 3 people found this helpful
          • 2. Re: 9.1.04 upgrade complains about previous steps
            Ashutosh Deshpande

            This ITSM installer behavior is documented here: Upgrading the applications - Documentation for BMC Remedy ITSM Deployment 9.1 - BMC Documentation  in a note added for 9.1.04.

             

            Thanks and Regards,

            Ashutosh.

            • 3. Re: 9.1.04 upgrade complains about previous steps
              Peter Lundqvist

              Hi Ashutosh,

               

              We are not doing a zero downtime upgrade, but your second  reason for the upgrade not completing gave me something.

               

              As I said before, the primary server upgrade completed just fine. However, looking in the arerror.log for the primary server I noticed this as the last "[Post Upgrade]" event:

              [Post Upgrade] Exception in executing task: CMDBPostUpgradeTask, exception: ERROR (120000): The CMDB Engine is not initialized.; Unexpected Error during CMDB Initialization!

               

              And CMDB ERROR 120000 seems to be

              120000=The Class Manager system is not initialized.

               

              The rest of the log looks just fine, and one hour later the system was restarted after ITSM had been installed on the primary server.

              • 4. Re: 9.1.04 upgrade complains about previous steps
                Ashutosh Deshpande

                Ok, thanks Peter for additional info.

                 

                I should have been clear. Platform 9.1.04 and later installers always assume Zero Downtime upgrade by default, hence this behavior.

                 

                About CMDB error, I am not sure what's wrong - +Shaktisinh Jhala, can you please check.

                 

                But that confirms that Post Upgrade tasks were yet to begin or in progress (there is also a wait time of up to 30 minutes in 9.1.04 for those to get triggered), hence ITSM installer got this error. Once the tasks were complete, status would have changed to "Done".

                 

                Thanks and Regards,

                Ashutosh.

                1 of 1 people found this helpful
                • 5. Re: 9.1.04 upgrade complains about previous steps
                  Shaktisinh Jhala

                  Hi Peter, can you provide following log files from %AR_HOME%\ARServer\Db (windows) or %AR_HOME%\db (linux)

                  arerror.log

                  ardebug.log

                  cmdb_eng_debug_log.log

                   

                  In case there are multiple files due to roll over (for example cmdb_eng_debug_log.log.0), please provide all of them.

                   

                  Thanks,

                  Shakti

                  • 6. Re: 9.1.04 upgrade complains about previous steps
                    Peter Lundqvist

                    Thanks for helping, here are the logs.

                    • 7. Re: 9.1.04 upgrade complains about previous steps
                      Shaktisinh Jhala

                      Thanks Peter for the logs.

                      We identified the issue with error and fixed it: [Post Upgrade] Exception in executing task: CMDBPostUpgradeTask, exception: ERROR (120000): The CMDB Engine is not initialized.; Unexpected Error during CMDB Initialization!

                       

                      However, even with this issue we should not have got the error during ITSM upgrade. We will get back on this.

                       

                      Thanks,

                      Shakti

                      • 8. Re: 9.1.04 upgrade complains about previous steps
                        Peter Lundqvist

                        Also, remember that we could proceed with installation of ITSM, SLM and SRM on the primary server.

                        The issue only appared once we reached ITSM on the secondary server.

                        • 9. Re: 9.1.04 upgrade complains about previous steps
                          Ashutosh Deshpande

                          Thanks Peter for additional detail, but this is very strange. If your primary ITSM already upgraded that means upgrade status was already "Done" at this stage. There should be no reason for it to go back to "Pending".

                           

                          Just to confirm our understanding:

                          Platform on all servers in server group must be upgraded first starting 9.1.04 before ITSM. So you performed upgrade of platform (AR+CMDB+AI) on primary, then platform (AR+CMDB+AI) upgrade on secondary - this is where status should become "Done" from "Pending". After this you were able to successfully upgrade ITSM on primary without seeing an error - this is good and expected. But now when you try to upgrade ITSM on secondary, you see status as Pending and you see error in ITSM installer.

                           

                          Is this correct understanding? If yes, something very strange happened after primary upgrade of ITSM.

                           

                          If possible, can you share all logs zipped via maintenance utility for all the servers in server group? That can give us more clues as to what could cause this.

                           

                          Thanks and Regards,

                          Ashutosh.

                          • 10. Re: 9.1.04 upgrade complains about previous steps
                            Peter Lundqvist

                            There is a difference,

                             

                            When we clone the database we clear all server references (we don't reuse the servers - we create a staged copy).

                            So in essence, we have a standalone server that we upgrade (base and applications). After that we create a server group and install a new AR Server.

                             

                            I will go ahead and create the secondary server before upgrading ITSM on the primary server, but if doing the upgrade this way is not possible - how do you then remove/add AR servers to the server group? Is it only possible to achieve through making a clone of a filesystem and modify the configuration on another server?

                             

                            Many thanks for all the help!

                            • 11. Re: 9.1.04 upgrade complains about previous steps
                              Ashutosh Deshpande

                              Thanks Peter for clarifying.

                               

                              >> So in essence, we have a standalone server that we upgrade (base and applications). After that we create a server group and install a new AR Server.

                              If this is the case, it's perfectly valid. You can always add new servers in server group starting with AR server later on. And in this case, you should not get an issue in ITSM installer, as the upgrade status will be already "Done". But looks like it didn't happen in your case. If possible, I request you to please share those logs from both primary and secondary servers so that we can see the exact sequence and what caused upgrade status to change after primary apps were upgraded.

                               

                              Rahul Palnitkar, let's try this case in-house again.

                               

                              Thanks and Regards,

                              Ashutosh.

                              • 13. Re: 9.1.04 upgrade complains about previous steps
                                Rahul Palnitkar

                                Yes Ashutosh. I will try this scenario in-house.

                                • 14. Re: 9.1.04 upgrade complains about previous steps
                                  Rahul Palnitkar

                                  Hi Peter,

                                   

                                  I tried the same case in-house and it did NOT reproduce.

                                   

                                  Could you help me with following things: -

                                   

                                  1. Installation Logs + arerror.log from Primary as well as the new server

                                  2. What did the "AR System Server Group Operation Ranking" Form display before and after adding the new server

                                  3. Any updates done to the database directly

                                   

                                  -Rahul Palnitkar

                                  1 2 Previous Next