1 2 3 Previous Next 31 Replies Latest reply on Oct 10, 2017 2:32 AM by Ashutosh Deshpande Go to original post
      • 15. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
        Marek Ceizel

        Hi Ravi,

         

        Finally I found time to check this again.

        Actually the problem is pretty clear an as you wrote, visible in RIK log. As I wrote it is postUpgrate phase. In the file:

             ARServer-RIK_PostUpgrade_error.log

         

        is written lot of:

        ...

        [Thu Aug 17 20:29:36.138] ERROR RIKMain  - Definition import failed, code 2 , filename - /opt/app_aio/bmc/ARSystem/installforms/en/RSSFeed.def

        [Thu Aug 17 20:29:36.140] ERROR RIKMain  - 26 Administrator operations are disabled on this server

        [Thu Aug 17 20:29:36.140] ERROR RIKMain  - Definition import failed, code 2 , filename - /opt/app_aio/bmc/ARSystem/installforms/en/ARChat.def

        [Thu Aug 17 20:29:36.142] ERROR RIKMain  - 26 Administrator operations are disabled on this server

        ......

         

        So now I know why and I can also enable the admin operations to go through the installation. But that is not answering my question. Why he is doing this on secondary server ???

         

        This is probably worth opening a case on support.....

         

        thanks

        regards

        Marek

        • 16. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
          Ashutosh Deshpande

          Hi Marek,

           

          You are right, installer should detect that it's a secondary server and should not perform any imports. However, it appears that installer identified this as a primary server. Logic of installer identifying a server as a primary is based on value of "dbversion" column in "control" table. So, looks like when installation on secondary started, dbversion was not updated in control table.

           

          Now, here are a few questions and things that you should check:

          1. AR installer supports parallel installations on multiple servers, but only on secondary servers after primary server is upgraded. Did you deliberately or accidentally start primary and secondary installations in parallel? If yes, that explains why secondary server also tried to become primary and tried imports.

          2. If not above, you will need to check the installation logs on primary and secondary:

            - on primary you will see existing dbversion in install logs

                 LOG EVENT {Description=[SET PROPERTY EXISTING_AR_SERVER_DBVERSION],Detail=[53]}   --- 53 is dbversion for 9.1.02

            - on primary server towards the upgrade is completion log, you should see a log message something like:

                 LOG EVENT {Description=[SET PROPERTY BMC_AR_SERVER_DBVERSION],Detail=[54]}       --- 54 is dbversion for 9.1.03

            - in the secondary server installation log, if you see 53 again:

                   LOG EVENT {Description=[SET PROPERTY EXISTING_AR_SERVER_DBVERSION],Detail=[53]}   --- if you see 53, this is a problem - dbversion wasn't updated before you started secondary upgrade.

           

           

          Refer to BMC Remedy Release Version Strings for dbversion values for various AR versions.

           

          Hope this helps.

           

          Thanks and Regards,

          Ashutosh.

          1 of 1 people found this helpful
          • 17. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
            Marek Ceizel

            Thanks a lot Ashutosh. I'll check this next week and write more / answer.

             

            best regards

            Marek

            • 18. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
              Marek Ceizel

              Hello Ashutosh,

               

              Finally I got the time to take a look on this. And Houston, we have a problem!

               

              - These are lines from Server1 arsystem_install_log.txt (is searched _DBVERSION]):

              LOG EVENT {Description=[SET PROPERTY BMC_AR_SERVER_DBVERSION],Detail=[53]}

              ...

              LOG EVENT {Description=[SET PROPERTY EXISTING_AR_SERVER_DBVERSION],Detail=[30]}

              ...

              LOG EVENT {Description=[SET PROPERTY BMC_AR_SERVER_DBVERSION],Detail=[54]}

               

              I don't understand the 30, however the DBVERSION should be updated.

               

              - These are lines from Server2 arsystem_install_log.txt (is searched _DBVERSION]):

                LOG EVENT {Description=[SET PROPERTY BMC_AR_SERVER_DBVERSION],Detail=[53]}

              ...

              LOG EVENT {Description=[SET PROPERTY EXISTING_AR_SERVER_DBVERSION],Detail=[53]}

              ...

                LOG EVENT {Description=[SET PROPERTY BMC_AR_SERVER_DBVERSION],Detail=[54]}

               

              Now it's clear. The second server thought db is not updated and tried again. As I started the second server installation several times, it's clear he is not able to update this. why ?

              And what does mean the 30 ?

               

              Can you please write me how can I check this value now ? probably it's still 53.

               

               

              PS-I didn't start the update parallel.

               

              thanks a lot

              Marek

              • 19. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
                Andreas Mitterdorfer

                I think a "select dbversion from control" should give you the actual value.

                • 20. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
                  Ashutosh Deshpande

                  Response by Andreas is right. SELECT dbVersion FROM control will give you the version.

                   

                  I am not 100% sure where this 30 is coming from. But can you check file named ARSystemInstalledConfiguration.xml in your installation folder to see if that contains this property? May be it stayed there from one of the earlier upgrades that you performed.

                   

                  Thanks and Regards,

                  Ashutosh.

                  • 21. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
                    Marek Ceizel

                    Hi Ashutosh,

                     

                    You're right with the 30. It is really present in ARSystemInstalledConfiguration.xml:

                     

                    Anyway let's make a walkthrough in ARSystemInstalledConfigurations.xml of both nodes and parameter DBVERSION:

                    Node 1 contains:

                         <name>BMC_AR_SERVER_DBVERSION</name>

                         <value>54</value>

                         <name>EXISTING_AR_SERVER_DBVERSION</name>

                         <value>30</value>

                     

                    Node 2 contains:

                         <name>BMC_AR_SERVER_DBVERSION</name>

                         <value>54</value>

                         <name>BMC_AR_SERVER_INSTALLED_DBVERSION</name>

                         <value>53</value>

                         <name>EXISTING_AR_SERVER_DBVERSION</name>

                         <value>53</value>

                     

                    - I don't touch them so why they are so different ?

                    - Why there is this 30 in Node 1 file ?

                    - Why there is parameter INSTALLED with value 53 in Node 2 file ?

                     

                    So much questions I checked the DB and it is 54.

                     

                    Should I try to change 30 to 54 and remove the INSTALLED parameter and try to run the Node 2 again ?

                     

                    regards

                    Marek

                    • 22. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
                      Ashutosh Deshpande

                      Ok, thanks for these additional details. Unfortunately I don't have all the answers for this at this time. It's some legacy code and I will get some experts opinion, once they are available next week (this is a long weekend in India, so Monday is a holiday, so expect some delays)

                       

                      Having said that, here is why you are seeing different versions in 2 servers:

                      1. Node1 was on dbVersion 30 i.e. AR 8.0 or AR 8.1, which you upgraded to dbVersion 53 i.e. AR 9.1.02. So on primary, control.dbVersion was still 30 when you upgraded to 9.1.02 and that value seems to be still preserved in the file. On successful completion of AR 9.1.02 upgrade, control.dbVersion was updated in database to 53.

                      2. When Node2 was being upgraded from AR 8.0/8.1 to AR 9.1.02, control.dbVersion was already updated to 53, so it detected existing dbversion as 53. 9.1.02 installer code also has hardcoded version as 53, thus matching the versions. This indicates that it's a secondary server upgrade.

                       

                      3. I am not sure about BMC_AR_SERVER_INSTALLED_DBVERSION property at this time, so I will check and get back to you as soon as I can, but it's possible that one of these versions are causing installer to think that dbversion is still 53, instead of reading 54 from the database. This may be a bug, but I am not sure yet.

                       

                      If you want to try, you need to change this file only in Node2. No changes needed in Node1. You can change these values to 54 in Node 2 and start upgrade, see if it passes. If it does, it looks like a defect to me. We can try to reproduce it inhouse based on your finding.

                       

                      Hope this helps.

                       

                      Thanks and Regards,

                      Ashutosh.

                      2 of 2 people found this helpful
                      • 23. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
                        Marek Ceizel

                        Hi Ashutosh,

                         

                        I'm pretty sure this 30 has something with our "installation". Actually we use quite often DB Dump from Production Database. This group is a Test System. Therefore these inconsistencies could appear.

                         

                        However the status is clear. dbversion in Database is 54 , but the Node 2 is still trying to import defs

                        I tried to change all the DBVERSIONs to 54 in the ARSystemInstalledConfiguration.xml and started the Upgrade again. Unfortunatelly the result is the same :

                        ....

                        [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - /opt/app_aio/bmc/ARSystem/installforms/en/CallHomeCustomerDetails.def

                        [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - /opt/app_aio/bmc/ARSystem/installforms/en/AdminPref.def

                        [RIKERROR]  Exception from RIKJ command Definition import failed, code 2 , filename - /opt/app_aio/bmc/ARSystem/installforms/en/UserPref.def

                        ]},

                          Throwable=[com.bmc.smbu.install.common.rule.engine.RuleEngineExecutionException: com.bmc.smbu.install.common.rule.engine.CommandExecutionException: Failure durin

                        g execution of RIK command

                        .....

                         

                        He is trying to import but Admin Ops are disabled on Node 2. So the result:

                         

                          --------------------------------------------------------

                          BMC Remedy Action Request System 9.1.03 install failed.

                          --------------------------------------------------------

                          Feature AR System Server  failed

                          --------------------------------------------------------

                         

                        Anyway thank you for your interest and that you really try to help

                         

                        Have a nice weekend!

                         

                        regards

                        Marek

                        • 24. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
                          Ravindrakumar Rodge

                          ARServer-RIK_PostUpgrade log file will give you more clarity why the RIK are failing.

                           

                          Thanks.

                           

                          Regards,

                          Ravi

                          • 25. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
                            Marek Ceizel

                            Hi Ravi,

                             

                            we know already. this was discussed

                             

                            thanks

                            Marek

                            • 26. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
                              Ashutosh Deshpande

                              Hi Marek,

                               

                              Sorry for the delayed response. I talked to someone else and they want to check if you see a line such as this in your secondary install log:

                               

                              (Sep 29 2017 01:31:25.601 PM +0530),INFO,com.bmc.install.product.arsuitekit.platforms.arsystemservers.arserver.ARServerDatabaseInformationValidationTask,

                                PROGRESS EVENT {Description=[initialization.arsuitekit.product.arsystemservers.arserver.database.information.description],Progress=[75],Detail=[initialization.arsuitekit.product.arsystemservers.arserver.database.information.validateTableExists]}

                              ...

                              (Sep 29 2017 01:31:26.751 PM +0530),CONFIG,com.bmc.install.task.InstallationPropertiesHelper,

                                LOG EVENT {Description=[SET PROPERTY BMC_AR_SERVER_DBVERSION],Detail=[54]}

                              ...

                                LOG EVENT {Description=[SET PROPERTY BMC_IS_SECONDARY_SERVER],Detail=[true]}

                               

                               

                              Value should be true for secondary server and it is set by the installer by reading dbVersion from control table after it finds version as 54.

                               

                              If you don't see this line at all, something else is wrong. If the line is present, but it says it's "false", that's a problem. We don't know why that would happen. If you change the value for this property in BMC_IS_SECONDARY_SERVER installed config xml before starting upgrade, does that work?

                               

                              If still not solved, is it possible to share full install log for secondary server here?

                               

                              Thanks and Regards,

                              Ashutosh.

                              • 27. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
                                Marek Ceizel

                                Hi Ashutosh,

                                 

                                Firstly thanks for taking care. Now I am really confused from the ServerGroup and scared to update the PRoduction Server group. Here Is what I found:

                                 

                                1) I found this line in my install log exactly as you expected:

                                          LOG EVENT {Description=[SET PROPERTY BMC_AR_SERVER_DBVERSION],Detail=[54]}

                                 

                                2) I DIDN'T find this line in the log at all:

                                          LOG EVENT {Description=[SET PROPERTY BMC_IS_SECONDARY_SERVER],Detail=[true]}

                                     BUT i found this line:

                                          LOG EVENT {Description=[Server is an Admin Server],Detail=[Found property BMC_IS_SECONDARY_SERVER : false]}

                                 

                                3) I DIDN'T find this parameter in ARSystemInstalledConfiguration.xml AT ALL:

                                          BMC_IS_SECONDARY_SERVER

                                 

                                Now tell me your opinions. Where the heck did he find the parameter false ???

                                 

                                I can send you both - install log and installed config xml. just not here. please write me some other way you prefer (email?)

                                 

                                Thank you once again.

                                Marek

                                 

                                • 28. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
                                  Ashutosh Deshpande

                                  Thanks Marek for the feedback.

                                  Ok, that's very strange and seems to be the issue. It's recognizing secondary server as admin, mostly due to lack of this property. But we can confirm looking at the full log and those XML files. I will send you a one-to-one message for log sharing.

                                   

                                  Thanks and Regards,

                                  Ashutosh.

                                  1 of 1 people found this helpful
                                  • 29. Re: 9.1.02 -> 9.1.03 Secondary Server Upgrade Failed
                                    Marek Ceizel

                                    one more note. I don't see the "SECOND" parameter neither on Productive slave server arsysteminstalledconfig xml. Only parameter with SECOND in name there is:

                                     

                                    <name>BMC_FULL_TEXT_SEARCH_SERVER_PORT_SECONDARY</name>

                                          <value>9977</value>