Share:|

Introduction : This blog will give you the brief idea about the prerequisite needs to consider before performing AR system platform upgrade using installer. Also included prerequisite needs to consider while applying any patch/hotfix through d2p.

 

Basic configuration checks to be performed before upgrading the Remedy platform:

 

1. Validate ARSystemInstalledConfiguration.xml file. Perform the following steps:

  1. Verify that the ARSystemInstalledConfiguration.xml file exists in the <AR Installation Directory> folder. If the file does not exist, do not proceed with the upgrade. You can copy the ARSystemInstalledConfiguration.xml file from another server having the same version and make the server-specific changes (such as host name) in the file.

 

    b. Check the Product feature map section. The product feature map section contains all the features that are installed.
        If a feature that is currently running on the system, is missing from the list, you must add it to the ARSystemInstalledConfiguration.xml file.

        For example:
                 <productFeature backupOnUpgrade="false" id="featureARSystemServers"

 

<productFeature backupOnUpgrade="false" id="featureARServer"

 

<productFeaturebackupOnUpgrade="false" id="featureAREALDAPDirectoryServiceAuthentication"

 

<productFeaturebackupOnUpgrade="false" id="featureARDBCLDAPDirectoryServiceAuthentication" independentOfChildren="false" parent="featureARServer" rebootRequiredOnInstall="false" rebootRequiredOnUninstall="false" rebootRequiredOnUpgrade="false" requiredDiskSpaceMode="default.linux" state="INSTALLED" visible="true">

 

     c. Before starting the upgrade, verify the release version, major and minor version.

         For example, <version majorVersion="1" minorVersion="00" releaseVersion="9"/>,

         which indicates that the current version installed is 9.1.00.

 

     d. Verify that the following properties have correct hostnames/IP addresses.

 

<name>BMC_AR_SERVER_NAME</name>

<name>BMC_AR_SERVER_HOST_NAME</name>

<name>BMC_AR_SERVER_CONNECT_NAME</name>

<name>BMC_AR_SERVICE_NAME</name>

<name>BMC_EMAIL_SERVICE_NAME</name>

<name>BMC_MIDTIER_TOMCAT_SERVICE_NAME</name>

<name>BMC_MIDTIER_INSTANCE_NAME</name>

 

      e. If the installed version is 9.1.04 or later, verify the 'BMC Remedy MidTier File Deployer' property.

          This property should exist for 9.1.04 or later versions.

<property>

      <name>BMC Remedy MidTier File Deployer - </name>

      <environmentVariable scope="SYSTEM">

      <name>BMC_JAVA_HOME</name>

                  <value>C:\Java\jdk1.8.0_45\jre</value>

      </environmentVariable>

</property>

 

2. Verify if the service name in the system registry is same as 'AR_Server_Host_Name' property in 'ARSystemInstalledConfiguration.xml' (WINDOWS SPECIFIC)

 

When you create a clone of an existing environment, you must also update the Windows registry in the cloned environment. If you do not update the registry, the following issues may occur:

  • Throwable=[java.io.FileNotFoundException: D:\Program Files\BMC Software\ARSystem\armonitor.exe (The process cannot access the file because it is being used by another process)Verify the following
  • Throwable=[java.io.FileNotFoundException: D:\Program Files\BMC Software\ARSystem\arcatalog_eng_W_win64.dll (The requested operation cannot be performed on a file with a user-mapped section open)  java.io.FileOutputStream.open0(Native Method)

 

  1. Verify the service name from the registry.

HKEY_LOCAL_MACHINE\SOFTWARE\Remedy\ARServer\<BMC_AR_SERVER_NAME>\ServiceName

    b. Verify the service name from the registry

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy Action Request System Server <BMC_AR_SERVER_NAME>

 

    c. Under registry, verify the JVM related options

    • JARS are pointing to the correct path
    • All required JARS exist.

 

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy Action Request System Server <BMC_AR_SERVER_NAME>\Parameters

 

    d. Verify the email engine service name from in registry against the one present inside ARSystemInstalledConfiguration.xml

 

        HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy Email Engine - <BMC_AR_SERVER_NAME> 1

    e. Verify the email engine parameters are:

    • Pointing to correct paths.
    • All required JARS exist in these paths.

 

 

f. Verify the flashboard service name in registry against the one present inside ARSystemInstalledConfiguration.xml

           HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy Flashboards Server - <BMC_AR_SERVER_NAME>

 

 

g. Under registry, verify the JVM related options

    • JARS are pointing to the correct path
    • All required JARS exist

           HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy Flashboards Server - <BMC_AR_SERVER_NAME>\Parameters

 

 

        If the current installed version is 9.1.04 or later, verify the following:

        h. Verify the file deployer service name from registry against the one present in ARSystemInstalledConfiguration.xml

        HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy MidTier File Deployer - <BMC_AR_SERVER_NAME> 1

 

i. Under registry, verify the JVM related options

    • Jars are pointing to correct path
    • All appropriate jars exist.

          HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy MidTier File Deployer - <BMC_AR_SERVER_NAME> 1\Parameters

 

 

3. Check servgrp_board table. opFlag should be “1” for Administrator server.

4. Make sure that Object modification logs is disabled. If turned ON, it causes slowness during installation because the entire AR Server metadata undergoes changes during installation.

    Link for Object modification logs:

    https://docs.bmc.com/docs/display/ars91/Using+the+object+modification+log

 

5.If you are upgrading from 7.x/8.x to 9.x or later, check the ‘Application Statistics Configuration’ form. All forms listed here must be part of some application.

 

6. Check DB Version in Control table. Refer to the following link to verify if the DBVersion column of the Control table has the value same as that of the AR Server version.

   https://communities.bmc.com/docs/DOC-37267

 

7. Execute the following queries to find out if there are invalid views in the database. Work with your DBA to fix/remove invalid views. This is one of the sure shot reasons of upgrade            failure.

Step 1: Execute the following SQL

select count(1) from    user_objects  where  status = 'INVALID' and object_type = 'VIEW'

Step 2: If the output is greater than 0 , execute following script to identify list of forms associated with invalid views :

select name, schemaid, case

        when overlayprop = 0 then 'Unmodified'

when overlayprop = 1 then 'Base'

when overlayprop =  2  then 'Overlay'

when overlayprop = 4 then 'Custom' end as CustomizationType

from arschema  where 'T' || schemaid in (select OBJECT_NAME  from  user_objects  where status = 'INVALID'  and object_type = 'VIEW')

Step 3 (Compile invalid views) : Execute following script to compile invalid views, change schema name if it is different from ARADMIN

EXEC DBMS_UTILITY.COMPILE_SCHEMA( schema => 'ARADMIN', compile_all => FALSE)

 

8. Make sure that the ‘Temp’ directory is clean before trying to perform a subsequent upgrade attempt.

 

9. Metadata inconsistency is one of the reasons for upgrade failure. Make sure you run checkdb utililty before the upgrade and resolve relevant inconsistencies.

 

Example:

 

 

Following error was seen in an upgrade attempt for one of the customers:

 

(Apr 11 2019 04:36:52.311 PM +0200),SEVERE,com.bmc.install.product.arsuitekit.platforms.arsystemservers.arserver.ARServerOracleManageUpgradeDatabaseTask,

  LOG EVENT {Description=[[SQLERROR] [DESCRIPTION] Failed to upgrade the database schema],Detail=[[SQLERRORCODE]=0 [SQLMESSAGE]=Failed to run SQL statement [ALTER TABLE SERVGRP_RESOURCES MODIFY ( JMSRESOURCES CLOB NOT NULL )] Due to [ORA-22296: invalid ALTER TABLE option for conversion of LONG datatype to LOB

][SQLSTATEMENT]=]}

(Apr 11 2019 04:36:52.312 PM +0200),SEVERE,com.bmc.install.product.arsuitekit.platforms.arsystemservers.arserver.ARServerOracleManageUpgradeDatabaseTask,

  THROWABLE EVENT {Description=[Failed to upgrade the database schema]},

Throwable=[java.sql.SQLException: Failed to run SQL statement [ALTER TABLE SERVGRP_RESOURCES MODIFY ( JMSRESOURCES CLOB NOT NULL )] Due to [ORA-22296: invalid ALTER TABLE option for conversion of LONG datatype to LOB

]

 

The failed SQL statements are built by the upgrade installer using the information in ‘ReadARServerDatabaseModel.xml’ and ‘TransformedARServerDatabaseModel.xml’. These files are created by the upgrade installer in the ‘Temp’ directory during installation.

 

There were two issues in this scenario:

 

  • Installer was trying to alter a column in SERVGRP_RESOURCES table, but the column was already altered. This could be due to multiple attempts of upgrades without cleanly reverting database and the file system.
  • SQL statement was trying to update Datatype CLOB with ‘NOT NULL’ option (Oracle has some issues with it) which was not allowing as per oracle. On a successful installation environment, we don’t have NULL values ) Inhouse we haven’t seen NULL value, but for this customer it was NULL. Therefore, we manually updated same values (CLOB and NOT NULL)here, deleted everything from ‘Temp’ directory before upgrade.

 

 

NOTE: If you run into these issues, contact BMC support to get appropriate assistance. Do not perform any manual changes on the database without consulting BMC Support.

 

 

Basic Configuration checks to be performed before applying a patch / hotfix using D2P:

1. Make sure that correct server entries in ‘AR System Monitor’ form exist. Delete the orphan or duplicate entries and restart the file deployer service of each of the servers in the server group.

 

If the issue still persists, go to AR System Installation Directory. Make sure that ‘monitor-ARServer-guid.properties’ file is present. It should have unique GUID for each server.

 

If the GUIDs are not unique, you must create entries with unique GUIDs. Perform the following steps:

    • Delete the file monitor-ARServer-guid.properties
    • Remove all entries from AR System Monitor form
    • Restart file deployer service.

AR System Monitor Form

 

 

2. Make sure to wait for few mins after you import the package, before clicking on ‘Deploy’ button.

 

3. Make sure that ‘AR System Single Deployment Payload’ form and ‘AR System Single Deployment Status’ form should not have any orphan records for the payload which we are           trying to deploy.

 

4. Make sure that File deployer service is running on all servers in the server group.

 

5. Make sure all the processes that are part of the AR System Service in armonitor.cfg file, are started without any issue. Please check armonitor.log file to verify this.

     For one of the customers was having duplicate entries for pluginsvr process. This caused errors “Address already in use” error, visible in armonitor.log. Deployment for one of                the payloads failed due to the error.

 

6. During deployment if any process fails to start / stop then you need to check following.

     Following error will be visible in file deployer log.

     ITSM Deployment Rollback caused by error: com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:NormalizationEngine failed to start.

  • FileDeployer signals the ARMonitor to start/stop Payload associated processes. During deployment if a specific process failed to start, then the related logging will be available in armonitor.log file with signal / log stmt saying … Starting Process <processName>  … (basic logging should be enough.)
  • If the process can be manually restarted then to troubleshoot from d2p perspective, execute the ARMonitor_Admin.bat file located in ARSystem directory manually to start / stop the specific process.

 

7. Prior to 1808 , it was mandatory to have process started for which we want to do payload deployment. Otherwise payload used to fail.

     For. E.g Email Engine process, DSO process etc.

     We can use following workaround to bypass this without starting that particular process.

  • import d2p package
  • click on deploy (This will create the Payload entries)
  • Run the below query (To remove the associated process entry for ex |;BMC:EmailEngine )
  • UPDATE < schemaid of ‘AR System Single Point Deployment Payload’> SET C49110 = 'BMC:ARServer|;BMC:JavaPluginServer|;BMC:DSOJServer|;BMC:CarteServer'          WHERE C49102 = '<Payload GUID>'

        Please get value of C49110 by viewing the payload ’Process Type’, copy it and remove only email engine from it.

        e.g. Screenshot is of different d2p package

 

  • Start the utility (arpayloadutility.bat) for deployment .