Share This:

Welcome to July’s new AR Server Blog post and we are discussing Remedy AR Server Upgrade Log Analysis.

 

When upgrading the AR Server to 18.05 or higher version if you face any issues, the following logs which are automatically generated can be used to troubleshoot the problem:

 

File Name
File Path Location
arsystem_install_log.txtlocated under System %temp% directory
armonitor.log

Windows: <Install directory>ARSystem\Arserver\Db

Linux: /opt/bmc/ARSystem/db

upgradeutil.log

Windows: <Install directory>ARSystem\Arserver\Db

Linux: /opt/bmc/ARSystem/db

RIK Logs

Windows: <Install directory>ARSystem\Logs

Linux: /opt/bmc/ARSystem/Logs

arerror.log

Windows: <Install directory>ARSystem\Arserver\Db

Linux: /opt/bmc/ARSystem/db

 

 

Now let's have a look at what keywords you can search in these logs to find the symptoms of the Upgrade failure.

 

  1. arsystem_install_log.txt

    Keywords and Parameters

    Purpose

    Next Action

    SEVERE,

    To find the failure error message

     

    Example: SEVERE,com.bmc.smbu.install.common.rule.engine.rik.RIKUtilJava,

    THROWABLE EVENT {Description=[Error from RIKJ command ],Detail=[ERROR (623): Authentication failed; aradmin]}

     

    NOTE: Look at the preceding line immediately after the SEVERE msg to find details on the actual error message

    Once you have the SEVERE errors,

    See Troubleshooting AR System Server Upgrade issues

    WARNING,

    To find the warning message

     

    NOTE: If Installer completed with any Warning message, it does not mean the upgrade has failed. You can fix the warning message issue post-upgrade.

    Once you have the WARNING message,
    See Troubleshooting AR System Server Upgrade issues

    descriptionStarting

    To find each Installer Utility started during the upgrade

     

    NOTE: When upgrading AR Server, you will find following Utility 'starting' in the log:

                arserver.installer.encryption.descriptionStarting Encryption Utility

                arserver.installer.preupgrade.descriptionStarting Boulder Database Upgrade Utility

                arserver.installer.preupgrade.descriptionStarting FTS Index Migration Utility

                arServerReStartARServerBusinessTask.descriptionStarting execution of pluginsvrclient utility

    descriptionCompleted

    To find each Installer Utility completed during the upgrade

     

    NOTE: For a successful upgrade, you will find the following Utility 'Completed' in the log:

                arserver.installer.encryption.descriptionCompleted Encryption Utility

                arserver.installer.preupgrade.descriptionCompleted Database Upgrade Utility

                arserver.installer.preupgrade.descriptionCompleted FTS Index Migration Utility

                arServerCheckPluginServerStatusBusinessTask.validatePluginServerPluginsvrclient utility execution completed

    If either of these Utility failed to complete,

    See Troubleshooting upgrade startup, connection, and timeout issues


    Troubleshooting database configuration issues

    Execution startedTo find the number of Upgrade attempts
    BMC_ARSYSTEM_INSTALL_OPTION

    To find if the installer ran as 'INSTALL' or 'UPGRADE'

     

    NOTE: You will find multiple entries for this parameter. Always look for what option the last entry has.

                Example: LOG EVENT {Description=[SET PROPERTY BMC_ARSYSTEM_INSTALL_OPTION],Detail=[Upgrade]}

    BMC_IS_SECONDARY_SERVER

    To find if the installer ran as 'Primary' or 'Secondary' server

    Value: true - Installer ran as Primary Server

                false - Installer ran as Secondary Server

     

    Example: LOG EVENT {Description=[SET PROPERTY BMC_IS_SECONDARY_SERVER],Detail=[false]}

     

    NOTE:

    - Primary Server is not always the Admin Server, its the first server using which the installer will perform the database upgrade

    - When the installer runs on Admin Server, you will always find the following line in the log:

       Server is an Admin Server

    - Secondary Server installation only updates the filesystem. No form or data will be imported.

    SET PROPERTY

    To find the list of all parameter setting properties

    You can bookmark all the line with this keyword >> remove duplicate and look for following common property names:

     

               BMC_AR_SERVER_HOST_NAME : Host Name of the current server

               BMC_AR_SERVER_HOST_NAME_PRIMARY : Host Name of the Primary Server

               BMC_AR_SERVER_NAME : Host Name of the current server or Load Balancer Name

               BMC_DATABASE_HOST : Host Name of the Database Server

               BMC_DATABASE_INSTANCE : Instance Name of the Database Server

    If either of these Property settings is incorrect,

    See Pre-Upgrade Configuration Checklist for AR Server

    Starting the AR System Server services

    To find when AR Server got the signal to Start the Service

     

    NOTE: If AR Server is able to start the service successfully, you will see the following message:
                Started AR Server service

    If AR Server Service failed to Start,

    See Troubleshooting upgrade startup, connection, and timeout issues

    Restarting AR Server service

    To find when AR Server got the signal to Restart the Service

     

    NOTE: If AR Server is able to restart the service successfully, you will see the following message:

               Restarted AR Server service




  2. armonitor.log

    • You will review this log to understand the status and progress of your AR Server Service ‘START’ and ‘STOP’ during an upgrade
    • AR Server gets the signal to ‘START’ the services two times and to ‘RESTART’ the services one time
    • Search for "Starting the AR System Server services" or "Restarting AR Server service" in the Install Log to note the timeframe. Search the same timeframe in armonitor.log to find more details on server startup

 

KeywordsPurposeNext Action
Initializing ARMonitor Version

To find the binary version of AR Server

  • This is the first step of initializing armonitor process
  • Review a couple of lines from this step to find more details on Server startup
  • Each Initializing statement will end with the following message:
    • Successful start: Initialization of ARMonitor     
                                    completed
    • Fail to start: Initialization of ARMonitor failed

armonitor ini fail.PNG

Starting Server-Process

To find when AR Server main process got the signal to start

  • This is the second step where armonitor will attempt to start the AR Server main process followed by the other processes defined in armonitor.cfg(armonitor.conf) file

 

  • When AR Server main process starts successfully, you will see the following message:
    • Server started with message - [ ArServer started. ]
  • When the AR Server main process fails to start, you will see the following message:
    • Process [ ARSERVER ] stopped with return code - [  ]
      • NOTE: Return code other than 0 means AR Server process failed to start

 

  • For each process it attempts to starts, you will see the following message:
    • Starting service - [ BMC:<process name> ] attempt [ < > ]
    • ProcessMonitor starting process [ BMC:<process name> ]

 

  • If armonitor if able to start the process successfully, you will see the following message:
    • Processes Started : [ <process name> ]
    • Example: Processes Started : [ ARSERVER BMC:DefaultJavaPluginServer BMC:NativePluginServer BMC:ARDispatcher BMC:CarteServer BMC:SLMBRService BMC:SLMCollectorService BMC:CMDBDispatcher BMC:ReconcilationEngine BMC:NormalizationEngine BMC:AtriumSharedPluginServer BMC:FileDeployer ]

 

  • If armonitor fails to start any process, you will see the following message:
    • Processes failed to Start : [ <process name>  ]
      • NOTE: If no process name is present inside the brackets, then all processes started successfully
    • Process [ BMC:<process name> ] stopped with return code - [ < > ] and error -

armonitorfull.PNG

If AR Server failed to start, find the error and

See Troubleshooting upgrade startup, connection, and timeout issues

 

 

  3. upgradeutil.log

 

    • You will review this log to find more details on 'Database Upgrade Utility'
    • Database Upgrade Utility updates the Meta-Data on your database server
    • Search for "descriptionStarting Boulder Database Upgrade Utility" in the Install Log to note the timeframe. Search the same timeframe in upgradeutil.log

 

KeywordsSymptomsNext Action
ServerJ Metadata Upgrade Utility version/ Build timestamp

To find when the utility started with the upgraded binary version

Example: ServerJ Metadata Upgrade Utility version/ Build timestamp 9.1.10 202002090135

ServerJ Metadata Upgrade Utility returning < >

To find if the utility completed successfully or not

 

NOTE: The returning code will be between 0 - 2

   Value 0 - Utility completed successfully

   Value 1 or 2 - Utility failed to complete

If Utility failed to complete,
See Troubleshooting database configuration issues

 

 

Pre-Upgrade Configuration Checklist for AR Server

 

 

  4. RIK Logs

 

    • If any AR Server component failed to upgrade, it will create a log file with name: <Component-name>-RIK_PostUpgrade_error.log file
      • Example:  ARServer-RIK_PostUpgrade_error.log or EmailEngine-RIK_PostInstall_error.log
    • The SEVERE error message in the arsystem_install_log.txt will point to which component failed to upgrade, after which you can review the RIK error.log file to get more details on the failure

 

KeywordsSymptomsNext Action
ERROR RIKMain

To find the RIK error message

 

Example:

ERROR RIKMain - 552 The SQL database operation failed. The column safeGuard in table actlink does not allow null values.

ERROR RIKMain - Definition import failed, code 2 , filename - /data1/ar/installforms/en/Group.def

See Troubleshooting AR System Server Upgrade issues

 

 

  5. arerror.log

 

    • You will review arerror.log to confirm if AR Server started with the upgraded binary version and what all AR System Objects were imported with the upgrade
    • You will find more details related to AR Server Service if it failed to start during the upgrade

 

 

KeywordsPurposeNext Action
Action Request System(R) Server Version

To find if AR Server received the signal to start with the upgraded binary version

Example: Action Request System(R) Server Version 9.1.10 202002090135

If AR Server failed to import objects or failed to start,
See Troubleshooting upgrade startup, connection, and timeout issues

 

BMC Remedy Release Version Strings

Loading AR System objects information

To find which all AR Object definitions are getting imported during the upgradearerrorlog.PNG

Normal ARServer shutdown process Initiated

To find when AR Server received the signal to shut down during the upgrade

 

NOTE: AR Server will receive signal from Installer to stop or if the user manually stopped the services. In either case, you will see the same message on ARServer shutdown