7 Replies Latest reply on Aug 6, 2018 4:00 AM by Michael Aligbe

    Upgrade From 8.1.02 to 9.1.04

    Michael Aligbe

      Good afternoon people,

       

      We are in the process of upgrading from  Version 8.1.02 to Version 9.1.04. My environment details is stated below:

       

      OS: Windows Server 2012 R2

       

      DB: SQL 2014

       

      BMC Applications: AR 8.1.02(Its an AR environment, no OOTB Applications, all our applications are custom built on AR).

       

      We are not in a Server Group

       

      When i ran the first upgrade it failed, got below in the error log:

       

        LOG EVENT {Description=[[resultType=COMPLETED,exitCode=2]],Detail=[RIK command ( returned a RETURN code of 2, which was interpreted as a failure from RIKJ  [RIKCOMMAND] LOAD_APPLICATION -C true -f D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\adf_ServerApps_en_def.xml -G 14400 -k 0 -l D:\Program Files\BMC Software\ARSystem -L true -m nosamp -n ARServer-RIK_PostUpgrade -N 7200 -t 9998 -u Action Request Installer Account -x Server.com -X 21600  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\Group.def [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\UserPasswordChange.def [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\AlertList.def]}

       

      We restarted the system and ran the upgrade again, the result said the upgrade was successful but it still showed the following errors:

       

      Sample plugin call check failed : ERROR (93): Timeout during data retrieval due to busy server -- retry the operation; Server:9998 ONC/RPC call timed out

       

      Installer detected that the AR Server is a non-Admin Server and skipped the form and data imports.

       

      After restarting the AR Service starts well but we couldn't login to the system. We had to roll back. Now we are about to start the upgrade again and i need to know if there are things i should do before we start based on the errors in the log.
      In this environment, we have only made modifications on the Group and User forms(We just created a number of fields and they were created on base mode).
      Please assist with any docs or directions on what we need to do to avoid another failed upgrade.
      Regards,
      Mike.
        • 1. Re: Upgrade From 8.1.02 to 9.1.04
          LJ LongWing

          Michael,

          You may want to check and make sure that your Server Group Operational Ranking forms are setup properly...SP4 requires it to be in a server group and the upgrade requires that it be an admin server....so, stand-alone servers that were refreshed from server groups quite often have problems in this scenario.

          1 of 1 people found this helpful
          • 2. Re: Upgrade From 8.1.02 to 9.1.04
            Francois Seegers

            Michael

             

            Try to import this file (D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\UserPasswordChange.def ) manually onto your test where you testing the upgrade to see why it gives error, then remediate that first then do the upgrade, also consider the 3 way reconciliation process due to the customization's.

             

            Francois

            1 of 1 people found this helpful
            • 3. Re: Upgrade From 8.1.02 to 9.1.04
              Michael Aligbe

              Thanks for your responses,

               

              Please see my questions below:

               

              This environment doesn't have server groups. The server has always been a stand-alone server. Please what should I do to avoid the failure again?

               

              Secondly, if 9.1.04 requires it to be in a server group, should i upgrade with 9.1.03?

               

              Please assist?

               

              Regards,

              Mike.

              • 4. Re: Upgrade From 8.1.02 to 9.1.04
                Ashutosh Deshpande

                Hi Michael,

                 

                There is no reason to go to 9.1.03, you can certainly upgrade to 9.1.04 or 18.05 releases even when you have a standalone server. Here are a few guidelines for you.

                 

                Since you have only AR platform and no apps, you need to go through documentation and best practices mentioned in documentation: Upgrading the platform - Documentation for BMC Remedy ITSM Deployment 9.1 - BMC Documentation

                 

                Now, coming to the problems you are facing.

                Problem 1: UserPasswordChange.def has failed. Since you may not have checked additional errors in arerror.log and ardebug.log, I can only guess it to be related to the customization you made in Base mode. And if User form didn't get imported properly, that can surely cause problems with authentication. So, this time you need to ensure this file will get imported successfully. Things that you should do:

                  - Move your existing customization in User and Group (and any other forms, workflows) by creating overlays in existing version 8.1 itself using Developer Studio Best Practices Customization mode instead of Base mode. Once you have done that, import existing UserPasswordChange.def file from the 8.1 file system into the 8.1 server in base mode so that your base mode forms are exactly as they were shipped by BMC. This will hopefully avoid any conflicts while importing 9.x copy of the DEF file during upgrade.

                  - In your overlay layer, avoid illegal customization e.g. field ids must not be in BMC range, they must be in custom range. Refer to this doc for more details on reserved range: Reserved field ranges - Documentation for BMC Remedy Action Request System 9.1 - BMC Documentation

                 

                Problem 2: server:9998 ONC/RPC call timed out seems like an intermittent or network issue to me. You should check plugin server and armonitor logs to see if there was a problem in starting up plugin server. If yes, you will also find a error code and cause in the logs.

                 

                Problem 3: Installer detected that the AR Server is a non-Admin Server seems to me that you didn't revert the database since earlier failure. This could cause db changes to tell installer that it dbversion is already upgraded. Hence installer may be treating this as a secondary server. The best practice when performing a standlone server upgrade is to take back up of database and also a snapshot of file system machine/vm. If the upgrade fails, you should revert database backup and restore the vm snapshot, fix issues first and then perform another upgrade.

                 

                Hope this helps and you get a successful upgrade.

                 

                Thanks and Regards,

                Ashutosh.

                1 of 1 people found this helpful
                • 5. Re: Upgrade From 8.1.02 to 9.1.04
                  Michael Aligbe

                  Thanks People,

                   

                   

                   

                  I did exactly as Ashutosh advised on the test server and it was successful, but the user still deleted all the custom fields, no big deal, i recreated them in Best Practice Mode and it is fine now.

                   

                   

                   

                  With the success on the test environment, I went ahead and repeated the same steps on the new server and it returned a message that the upgrade failed. However, after restarting the server, it seems to be working well and the Server Information is showing: Server Version: 9.1.04 201711272256. And the custom fields on the User form were deleted again while the ones on the Group form were not deleted. I have now recreated the fields in best practice mode and imported the user data and it seems to be working well.

                   

                  I have put together the error message from the log:

                   

                   

                   

                  LOG EVENT {Description=[AR System Server  failed],Detail=[featureARServer]}

                  ********************************************************************************************************************

                    LOG EVENT {Description=[[resultType=COMPLETED,exitCode=2]],Detail=[RIK command ( returned a RETURN code of 2, which was interpreted as a failure from RIKJ  [RIKCOMMAND] LOAD_APPLICATION -C true -f D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\adf_ServerApps_en_def.xml -G 14400 -k 0 -l D:\Program Files\BMC Software\ARSystem -L true -m nosamp -n ARServer-RIK_PostUpgrade -N 7200 -t 0 -u Action Request Installer Account -x Server.com -X 21600  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\Group.def [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\UserPasswordChange.def [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\AlertList.def]}

                  ********************************************************************************************************************

                    THROWABLE EVENT {Description=[Failed to execute Rule Engine],Detail=[[RIKCOMMAND] LOAD_APPLICATION -C true -f D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\adf_ServerApps_en_def.xml -G 14400 -k 0 -l D:\Program Files\BMC Software\ARSystem -L true -m nosamp -n ARServer-RIK_PostUpgrade -N 7200 -t 0 -u Action Request Installer Account -x Server.com -X 21600

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\Group.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\UserPasswordChange.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\AlertList.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\SearchesPref.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\DataVisForms.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\visualizer_forms.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\HomePage.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\reportforms.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\reportvendor.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\SystemWebService.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\ARTemplateDef.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\ActorViewForms.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\ARSkins.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\CustomizableHomePage.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\ARSystemReporting.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\ARJobScheduler.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\ARSystemKeyStore.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\MFSMultiFormSearch.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\ARQueryWidget.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\RSSFeed.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\ARChat.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\CallHomeCustomerDetails.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\ARSystemServerGroupAdministration.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\AdminPref.def

                  [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - D:\Program Files\BMC Software\ARSystem\ARServer\installforms\en\UserPref.def]}

                  ******************************************************************************************************************

                  Throwable=[com.bmc.smbu.install.common.rule.engine.RuleEngineExecutionException: com.bmc.smbu.install.common.rule.engine.CommandExecutionException: Failure during execution of RIK command

                  1. com.bmc.smbu.install.common.rule.engine.Stage.execute(Stage.java:160)
                  2. com.bmc.smbu.install.common.rule.engine.StageGroup.execute(StageGroup.java:137)
                  3. com.bmc.smbu.install.common.rule.engine.Installer.execute(Installer.java:103)
                  4. com.bmc.smbu.install.common.rule.engine.RuleEngineExecutor.execute(RuleEngineExecutor.java:56)
                  5. com.bmc.install.product.arsuitekit.AbstractPostInstallerInstallationTask.executeRuleEngine(AbstractPostInstallerInstallationTask.java:341)
                  6. com.bmc.install.product.arsuitekit.platforms.arsystemservers.arserver.ARServerPostUpgradeInstallationTask.execute(ARServerPostUpgradeInstallationTask.java:208)
                  7. com.bmc.install.task.InstallationTask.run(InstallationTask.java:93)
                  8. java.lang.Thread.run(Thread.java:745)],

                  Throwable=[com.bmc.smbu.install.common.rule.engine.CommandExecutionException: Failure during execution of RIK command

                  1. com.bmc.smbu.install.common.rule.engine.rik.RikCommand.execute(RikCommand.java:463)
                  2. com.bmc.smbu.install.common.rule.engine.Stage.execute(Stage.java:124)
                  3. com.bmc.smbu.install.common.rule.engine.StageGroup.execute(StageGroup.java:137)
                  4. com.bmc.smbu.install.common.rule.engine.Installer.execute(Installer.java:103)
                  5. com.bmc.smbu.install.common.rule.engine.RuleEngineExecutor.execute(RuleEngineExecutor.java:56)
                  6. com.bmc.install.product.arsuitekit.AbstractPostInstallerInstallationTask.executeRuleEngine(AbstractPostInstallerInstallationTask.java:341)
                  7. com.bmc.install.product.arsuitekit.platforms.arsystemservers.arserver.ARServerPostUpgradeInstallationTask.execute(ARServerPostUpgradeInstallationTask.java:208)
                  8. com.bmc.install.task.InstallationTask.run(InstallationTask.java:93)
                  9. java.lang.Thread.run(Thread.java:745)]

                   

                   

                  LOG EVENT {Description=[AR System Server  failed],Detail=[featureARServer]}

                   

                  Please kindly take a look and advice if there are any further things i need to do achieve a smoother upgrade as i still have main production server to upgrade.

                   

                  Regards,

                  Mike.

                  • 6. Re: Upgrade From 8.1.02 to 9.1.04
                    Michael Aligbe

                    Hello People,

                     

                    Thanks for assistance so far. I have done some further tests and found that Email engine is not working, I have the entries in the log below:

                     

                    SEVERE: Cannot establish a network connection to the AR System server ONC/RPC program not registered Server.com

                     

                    I think my email engine is still on v8. I got this when i started the Email Engine:                                                                         

                     

                    D:\Program Files\BMC Software\ARSystem\AREmail>echo off

                     

                    Checking connection to server: Server.com ...

                    Application has started Version: 8.1.02 201408260235

                    Using JRE: 1.6.0_31

                     

                    Email Service has been suspended.

                    thread is still alive

                     

                    Email Engine service is not running since Jul 21, 2018 6:55:17 PM

                     

                    Please kindly assist.

                     

                    Regards,

                    Mike.

                    • 7. Re: Upgrade From 8.1.02 to 9.1.04
                      Michael Aligbe

                      Good Morning All,

                       

                      Thanks for your support so far.

                       

                      We had to change our approach. Decided to leave our old data on the current server and start V9 afresh on the new environment. So we only brought in the definitions and configuration data.

                       

                      We cut over this morning and everything was working well until an hour ago, the system wont allow anybody log in. Even on the application server. Both Web and BMC User Tool cant login to the system.

                       

                      All services are started but we cant login:

                       

                      From User tool:

                      ARERR [93] Timeout during data retrieval due to busy server -- retry the operation : Server Name

                       

                      The Server is relatively new and the utilization on it is at the minimal.

                       

                      Kindly assist as is now the production environment.

                       

                      Regards,

                      Mike