Skip navigation
1 2 3 Previous Next

Remedy AR System

99 posts
Share This:

The Remedy Management Console can be used to manage server group operations. This console simplifies the server group management by providing a single location to perform many of the server group configuration related activities.

 

The Remedy Management Console also known as the Server Group Administration Console was initially introduced in Remedy 9.1.04 (1802) as an unsupported version available from the BMC Communities. In Remedy 1805 release, a supported version was introduced.

Key Benefits

  • Ease of monitoring operations and administration in a server group without having remote access to actual servers
  • Visibility into server queues for critical processes
  • One-click operation to start or stop server processes and enable or disable logs
  • Apply configuration settings consistently to all ARServers in a server group

 

Introduction to different options on the Remedy Management Console

 

Server Group Dashboard – 6 flashboards showing status on important ARServer activities for monitoring purposes. Shows number of records FTS indexing, CAI events, Application Pending, SYS:Action, Pending outgoing emails and Number of days for oldest email message. Please see Connect with Remedy webinar for more information on customization.

 

 

Server group Configuration – Shows ARServer configurations same as Centralized Configuration form. Helps to manage all ARServer configurations of server group at one location. User can add, update configuration as it is allowed from Centralized Configuration form. You can use this interface to ensure that your settings are consistent across all the servers in the group. It can be also user to setup Global settings that can be set once and then adopted on all servers

 

 

FTS Management – Allows you to view and update ARServer configurations for Full Text Index and Full text search. User can view and update FTS configuration for all ARServers in group. User can perform complete and form level re-indexing and monitor the status

 

 

 

Manage Processes - User can manage different ARSystem processes without having remote access to actual host server. Form allows to perform stop, start, refresh, restart, add, update actions for specific process. This can be useful when it is required to make a change such as increasing Java Heap for plugin servers across the server group and you do not want to restart all the ARServers. A specific process can be modified and restarted without having whole ARServer restarted.

 

 

Logs – Form helps to Activate and deactivate log files for each ARServer in group. Also allows to download log files from respective ARServer. Helps to capture longest running API and SQL statistics quickly. During troubleshooting logs can easily be enabled and disabled on multiple servers in the group.

 

 

 

Manage User Licenses- View and manage server, application and bundled licenses in server group. This is the view same as License Review form except that it shows data from all the servers in the group at once. You can use this interface to see which users are consuming various licenses and allow to release active user sessions.

 

 

 

Current Operation Owner – View ARServer name owning AR System operations in server group in real time. Useful while checking failover activity of specific operations in server group

 

 

Ranking – Redirects to the AR System Server Group Operation Ranking form. User can manage operation ranking for each ARServer in group

 

Failover – Redirects to the AR System Service Failover Ranking form. User can view service failover ranking for different operations. Service failover is used to manage Email Engine and Normalization operations

 

Group – Redirects to the Group form. View, add, update all groups configured

 

Users – Redirects to the User form. User can manage all Remedy users

 

Configuration – Allow to customize environment such as customizing server group dashboard, setting configuration preferences, creating server name list, component list and settings list

 

 

  • Documentation

Configuring and monitoring AR System server groups using the Remedy Management Console

 

  • Communities

Remedy 9.1.04: Server Group Administration Console Overview Video

 

  • Webinar

Connect with Remedy - Using the Server Group Dashboard Webinar Recorded Session - YouTube

Share This:

After you’ve enabled log forwarding, and you’ve identified your critical events needing to be captured, the next phase is to use the Splunk application to extract usable fields so you can sort, report, search on.  This is a quick summary on how to extract some basic fields from the armonitor.log (which is used to monitor the Remedy server processes, including AR System Server). This assumes that you’ve set up Forwarding and Collecting Data from the previous article.

 

Configuring Source Types

 

1. Log into the Splunk Enterprise Administration Console


2. Click “Settings -> Source Type” to open the “Source Type” console.


3. Click on the Source type that was created with the Add Data Wizard.  In this case it was “AR Monitor Log”.


4. In the Edit Source Type console, you’ll want to configure a few items, which include event breaks, gathering time stamps, and the categories.  Select the Category as needed and if you need any Indexed Extractions.


5. Event Breaks are the first option that will need to be selected.  You can choose either Every Line or Regex, based on your requirements.


6. The next thing to configure is the Timestamps.  It is recommended that you set the timestamp configuration to “Advanced” and configure it as below.

Timezone

Set to AR System Server Time Zone

Timestamp Format

%a %b %d %Y %H:%M:%S.%Q

Timestamp Prefix

/.

Lookahead

180

7. Once that has been configured, you can click “Save”.  This has Configured the source type so it will start to parse the logs correctly.

 

 

Configuring Field Extractions

 

1. To configure field extractions click “Settings -> Fields” to open the “Fields” console.


2. Then click “Field extractions” to view and edit field extractions.


3. Click “New Field Extraction” in the top right-hand corner to open up the New Field Extraction wizard.


4. You can now configure the extraction how you’d like, for this example we’re going to create the extraction in the “Search” application, give it a name of “AR Monitor Extraction” and apply it to the REM 1908 Source.  This would be considered a “inline” extraction.  Then you’ll need to put in your expression to extract the fields. See below for a example expression to extract AR Monitor logs.

<(?P<Type>\w+)> <TNAME: (?P<Thread>[^>]*)\s*> <(?P<LogLevel>\w+)\s*> <(?P<Class>[^>]*)> <\s*(?P<ClassLine>[^>]*)> .................................... (?P<Details>.*)

5. Once you’ve configured this, Save the new extraction.  You can now modify the permissions as needed, contact your Splunk Admin if you have questions about permission.  Once you’ve done this, you should be able to open up a search and start reviewing the index created earlier to view the data needed.

 

 

Once you’ve been able to perform these actions, you can start creating Dashboards, Alerts, and other Splunk activities.  If you need help with this, then please contact your Splunk Admin for more information.  If for any reason the Splunk Server is not working, please contact Splunk as this is their application.  BMC will not provide support for this product and this documentation is provided AS IS, as a courtesy.

Adam Lawson

Log Forwarding With Splunk

Posted by Adam Lawson Employee Jan 10, 2020
Share This:

Splunk is a Enterprise Application that will collect data from different sources, and aggregate them under one console allowing for a more complete knowledge when troubleshooting or analyzing environments for problems.  Splunk has the ability to ingest several log types, including Windows Server Event logs, Linux System logs, and application logs.  BMC Remedy products have their own logs that are logged to files found on the application server independent of the system logs itself. Because of this, there are additional steps that might be needed as compared to other systems to allow Splunk to consume BMC Remedy logs.  Splunk provides all of the tools needed to do this, and BMC does not provide support to execute any of these actions.

The following steps are some high-level steps that you can perform to collect logs from your BMC Remedy servers and push them into Splunk.  If there are any questions about these steps first please view Splunk’s Universal Forwarder and Splunk Enterprise documentation (link below), or contact Splunk for more information.

https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/HowtoforwarddatatoSplunkEnterprise

Configure Splunk to Receive Logs (Enable a receiver)

 

1. Log into the Splunk Enterprise Administration Console

2. Click on “Settings -> Forwarding and Receiving”

3. Click “Add New” Under the Receive Data section.

4. Provide a port to receive the logs with, click “Save”

5. It will then bring up the Receive summary page.  You can add a new receiver here and review others.  (In a typical Production Environment, you’ll have several Receive ports configured.  Consult your Splunk Administrator if you have questions about which one to use)

 

Splunk Enterprise is now ready to receive data from a new Splunk Forwarder.  If for any reason the Splunk Server is not working, please contact Splunk as this is their application.  BMC will not provide support for this product and this documentation is provided AS IS, as a courtesy.

 

 

Install Splunk Forwarder On Same Server As Remedy AR System Server

 

1. Start Splunk Forwarder Installer.  Read and Accept the License Agreement from Splunk.  Choose if you’re using Splunk On-Premise or Splunk Cloud and click “Customize Options”

2. Choose install location and click “Next”.

3. Configure the SSL Certificates based on your organizations configuration, click “Next” to continue. If you have questions, ask your Splunk Admin.

4. Select the type of account to use by the Splunk Forwarder application.

5. Choose what data you want to be sent to the Splunk Server by the Forwarder.  For the purposes of Remedy AR System, you’ll need to select the DIRECTORY of your “DB” folder. After this is configured, click “Next”.

6. Configure the Splunk Administrator account, click “Next”.

7. Configured your Deployment Server.  This should be the Splunk Server’s configuration process.  If you don’t know this information, contact your Splunk Administrator. Click “Next” to continue.

8. Configure your Splunk Forwarder to send the data to the Receiver we created before.  Click “Next” to continue.

9. Click “Install” to proceed with the installation of the Splunk Forwarder.

10. Once the installation is complete, the forwarder is configured and running as needed for a Remedy AR System Server instance.  Click “Finish” to end the installation.

 

If for any reason the forwarder is not working, please contact Splunk as this is their utility. BMC will not provide support for this product and this documentation is provided AS IS, as a courtesy.

Consume Remedy Logs in Splunk

 

1. Log into the Splunk Enterprise Administration Console


2. Click “Settings -> Add Data” to open the “Add Data” wizard.


3. Choose the method to gather logs from.  To collect Logs from Remedy AR System Server, use a Splunk Forwarder by clicking “Forward” from the “Or get data in with the following methods” section.


4. On the “Select Forwarders” section, choose the forwarder that you want to collect, and give it a New Server Class Name.  In this example, we’re going to Select the server “clm-aus-tt8f8w” and then give the new name of “REM1908” as this is a Remedy 19.08 System.  Click “Next” to move to the next section.  You can obviously choose the Server Class name as whatever you want it to be, just make sure it is relevant to your scenario and that you can easily remember and use it.


5. Choose the source as “Files & Directories”, and choose the file you want to consume. In this example, we’re going to consume the armonitor.log file.  You can choose to setup a whitelist or blacklist if desired.  Click “Next” to continue to the next section.


6. Choose the Source Type as a “New” option and then fill out the source information as needed, if you have questions about what to select contact your Splunk Admin. Then choose a Index for use with this application.  If you need to create one, contact your Splunk Administrator.  Click “Review” to continue to the next section.


7. Once you review and confirm the settings, click “Submit” to finish the log collection.


8. Once you’ve clicked “Submit” it will start indexing the logs.  This might take some time the first time you do it. Additionally, you can start to groom the data to make better use of it.

This includes Extracting Fields, adding additional data, and building Dashboards.  You can do all of these from the final page.

Splunk Enterprise is now receiving data from the Splunk Forwarder installed on the Remedy Server. If for any reason the Splunk Server is not working, please contact Splunk as this is their application.  BMC will not provide support for this product and this documentation is provided AS IS, as a courtesy.

Share This:

Identifying Critical Events

The first step to being able to manage the logs in your environment is understanding the messages that the Remedy AR System will produce.  Please see the link below for the logs found in Remedy AR System Server:

 

https://docs.bmc.com/docs/ars1908/bmc-remedy-ar-system-error-messages-866877923.html

 

Once you’ve reviewed and understand the log events that are recorded in the AR System Logs, then you can review your system requirements to understand what is important to you. Here is a example of a requirement that might have logging requirements:

 

  • The Remedy ITSM System must be available during all working hours, Monday-Friday from 6AM-6PM.

 

This requirement indicates that you would have a service level agreement to provide this service regularly.  This would indicate that you would want to be alerted if the server process ever terminated. In the documentation above there are several messages that would be thrown based on a action like this, here are a few:

 

  1. 20 Note - BMC Remedy AR System server terminated when a signal/exception was received by the server.
  2. 21 Note - BMC Remedy AR System server terminated — fatal error occurred in ARSERVER.
  3. 32 Note - BMC Remedy AR System server terminated normally.

 

These three messages would indicate that the system server stopped due to a signal being sent to stop the server and then a fatal internal error while the server was running. Through this process of knowing and understanding the messages, understanding your requirements, and then relating the two, you can start to identify which log events are important to your organization to meet your business requirements.

 

BMC Support can not help identify what events are important to your organization.  This is specifically in relation to the requirements for your system.  If there are any questions about which log events relate to which requirements, please consult the documentation above which review all known error messages and your system requirements.

 

Common Critical Events in Remedy AR System

 

This is a list of common events that customers have wanted to identify in the past:

  1. When the system shuts down:
    1. 20 Note - BMC Remedy AR System server terminated when a signal/exception was received by the server
    2. 21 Note - BMC Remedy AR System server terminated — fatal error occurred in ARSERVER
    3. 32 Note - BMC Remedy AR System server terminated normally
  2. When the system has a non-fatal internal error:
    1. 35 Error - Another copy of the server is already running on the same RPC socket
    2. 36 Error - The database is not the expected version (may need to run upgrade program)
    3. 41 Warning - Unrecognized command line argument — ignoring command line and continuing
    4. 48 Error - Insufficient space for the response to be returned (minimum of 4,096 bytes)
    5. 49 Error - Internal error: The request ID is invalid
    6. 68 Warning - RPC environment variable is out of legal range (390600, 390603, 390619 - 390669, 390680 - 390694)
    7. 90 Error - Cannot establish a network connection to the BMC Remedy AR System server
    8. 91 Error - RPC call failed
    9. 92 Error - Timeout during database update
    10. 93 Error - Timeout during data retrieval due to busy server — retry the operation
    11. 94 Error - Timeout during database query — consider using more specific search criteria to narrow the results, and retry the operation
    12. 95 Error - RPC and sockets initialization failure
    13. 99 Error - The same AR System Server ID is found on two different servers — neither will be accessible until one is shut down and tools restarted
  3. When a user logs in:
    1. <USER> FIXED GRANT WRITE <LogOnName>
  4. When a user consumes a application license:
    1. 31 Note - The new user was issued a fixed license of the following type: <licenseType>
    2. 75 Warning - No floating write license tokens are available. Currently accessing the system in read-only mode
    3. 76 Note - A write token has become available and has been allocated to you — access has been upgraded to write access
    4. 77 Warning - No free floating full text license tokens are available. Currently accessing the system without full text search capability
    5. 78 Note - A full text token has become available and has been allocated to you — access has been upgraded to allow full text searching
  5. When a user enter incorrect credentials:
    1. 59 Warning - Your login failed, but connected as a guest user
  6. When a user attempts to access information they don’t have access to:
    1. 50 Warning - You have no permission to this field
    2. 53 Warning - Administrator access required to get permission information
    3. 60 Warning - You do not have read access to field
    4. 61 Warning - You do not have read access to this field on this record
    5. 62 Warning - You do not have access to the requested record
    6. 71 Warning - Only the Administrator has access to this active link
  7. When the system has a file system error:
    1. 22 Error - Failure during open/write to the filter/escalation log file, logging to the log file is suspended
    2. 23 Note - Open/write to the filter/escalation log file resumed successfully
    3. 34 Error - Error while opening the BMC Remedy AR System server lock file
    4. 37 Error - Error while accessing one of the debug trace files. Debug tracing to the file is disabled
    5. 64 Warning - The filter/escalation action cannot write to the specified log file — action created but not logged in the log file
  8. Workflow Errors:
    1. 24 Error - Failure while trying to run the filter/escalation process
    2. 25 Error - Fields in 'set fields' action do not exist in the target form
    3. 38 Error - Filter/escalation 'set field' process returned an error. The current transaction is rejected. No update to the database occurred.
    4. 39 Error - Filter/escalation 'set field' process timed out before completion
    5. 43 Error - Unable to retrieve the menu for character field "field name"
    6. 65 Warning - Multiple form links are not supported in new filter definition syntax. Only the first link is loaded
    7. 67 Warning - Not able to open the menu
    8. 70 Warning - The character menu referenced by one or more fields in this form does not exist
  9. When the system encounters a unknown error:
    1. 1 Error - Message not in catalog
Share This:

Welcome to December’s new Server Blog in which we will be discussing the AR System Java Driver

 

The driver program which is delivered as a Java program and includes source, is a great API learning/reference tool for AR System API users. The driver program provides a command line interface for calling every API function, and viewing return values.

 

Use cases can include:

 

  • Quick prototyping
  • API testing
  • Running simple batch jobs
  • AR server connection testing

 

It also includes outputs for every data structure in the API, making it a useful manual debugging tool. The driver program is interactive and you can use it to log in to the BMC Remedy AR System server and observe the results and behavior of the equivalent function calls that you would make in your custom API programs.

 

A driver script typically begins with below commands:

 

  • Init (initializes the user session)
  • Log (gathers the login information)
    • authentication string =
    • login = <your login>
    • password = <your password>
    • locale =
    • timezone =
    • server = <your AR Server>
  • ssp ("set server port" is optional and provides the TCP when needed)
    • specific TCP port = <your TCP port>
    • private queue =
  • ver ("verify user" validates that the user correctly logged in)

 

A driver script ends with below commands:

 

  • term (terminates the connection and logs the user out)
  • q (quits the session and exists drivers)

 

To know the full list of command on this tool type the commands ‘?’ or ‘h’ and check documentation related, where you will be able to find the command with their purpose.

 

Examples of the use of the Java Driver utility

 

  1. Get a list of groups that a user belongs to with GLG command.
  2. Find out what the FQDN of AR Server is with GSI command.
  3. Test a run process command like an Application-Business-Time-Add2.  If you think you got bad results in workflow, you can test it manually with different values using the PROC command in driver.
  4. Perform a SQL statement like the one we do to check if Oracle Case Insensitivity is enabled.   And then perform a SQL to see it return Case Insensitive results.

 

In the below example we can confirm that user Demo is able to log into the system and user is receiving Admin privileges. This could help us to confirm rights that users are receiving when login.stats. The command used is ver.

 

  

In next example the command glg (GetList Group) was used to confirm groups that a user ‘Allan’ belongs 

 

 

 

Basically, Java Driver utility can be used to see the behavior of your API calls in your system and is very useful to anticipate and resolve problems you may have. When Java Driver utility is launched there is a list of basic commands than can be used. In case more information is needed Please check below link for official documentation

 

 

 

Reference for additional information.

 

Communities

 

Documentation

  • Overview

https://docs.bmc.com/docs/ars1908/bmc-remedy-ar-system-c-api-overview-866353008.html

 

  • Create API Programs

https://docs.bmc.com/docs/ars1908/creating-and-executing-bmc-remedy-ar-system-c-api-programs-866877592.html

 

  • Functions

https://docs.bmc.com/docs/ars1908/bmc-remedy-ar-system-c-api-functions-866353122.html

 

  • Requirements

https://docs.bmc.com/docs/ars1908/bmc-remedy-ar-system-c-api-installation-and-compilation-requirements-866353019.html

Share This:

Welcome to November’s new AR Server Blog post and we are discussing D2P.

 

 

What is D2P? This stands for Development to Production (Dev to Prod).  This action is done via the Deployment Management Application that is included with the AR Server since 9.1.03.

 

The Deployment Application includes, the Deployment Management Console is used for all deployment functions –Import, Export, Create, Deploy, and Rollback. Let’s look at more specific examples for using this console and D2P packages, you can perform the following activities:

  • Apply BMC Hotfix and Patches
  • Installation of Applications like ITSM, SLM, SRM and Smart IT
  • If you performed Reconciliation, migrate Overlays between environments
  • Create and Deploy your own Custom Packages to Migrate data/definitions

 

For applications ‘ITSM, SRM, SLM  & Smart IT’, D2P packages are used as an Installer from version 18.08 (9.1.06) and higher.

 

To upgrade these applications there is a requirement to have these at version 18.05 (9.1.05); post which using D2P packages you can upgrade to a higher version.

 

For Midtier, AR and Atrium D2P packages are used to patch and it's available from 18.02 (9.1.04 002) onwards.

 

From version 18.02 onward, BMC also ships Hotfix through a D2P package.

 

Another main benefit you can get from this console is, you can create your own Custom Packages which can have a set of workflow, object definitions, data, or Service Request Management objects that you created in a development environment which, you can promote across environments, such as QA or production.

 

Using this console, you can patch, migrate data or upgrade all your servers in Server Group in a single go. You will IMPORT the D2P package only once and can DEPLOY the contain across all servers in a server group environment.

 

From Remedy version 19.02 onwards, you can perform Simultaneous Deployment or package Rollback.

 

The following diagram depicts the functionality of the BMC Remedy Deployment Application utility:

 

 

Reference Information for additional information

 

Communities

 

Documentation

 

 

  • Deploy Packages using Command Line Interface

     https://docs.bmc.com/docs/ars1808/using-a-command-line-interface-to-manage-a-package-821049028.html#Usingacommand-lineinterfacetomanageapackage-UsingCLI

 

  • Different Status of Binary Payload

      https://docs.bmc.com/docs/ars1808/viewing-the-status-of-a-binary-payload-820498092.html

 

 

Videos

 

 

Share This:

At a recent customer site (the Oracle version was 18c i.e. 12.2.0.2) we noticed that if Remedy was being run in Case Insensitive mode certain use cases would not return expected rows. Case in point is the "Site" drop down menu in the CTM:People form that returned the expected 72 rows (for this client) when Remedy was in Case Sensitive mode but not when in Case Insensitive mode.

 

One of the SQL statements associated with the population of the field is shown below (there were other SQLs where a value was entered for the Company field or the Region field or the Site Groupg field - all of them failed to return rows)

 

          SELECT DISTINCT T538.C260000001 FROM T538 WHERE ((((T538.C1000000001 = ' ') OR (' ' = ' ')) AND ((T538.C200000012 = ' ') OR (' ' = ' '))

          AND ((T538.C200000007 = ' ') OR (' ' = ' ')) AND (T538.C1000000073 = 0) AND (T538.C1000000081 = 1) AND (T538.C7 = 1)))

          AND ( ROWNUM <= 20101 ) ORDER BY T538.C260000001 ASC

 

We eventually realized that we may be encountering Oracle Bug 27416997 (check Metalink Doc ID 2390584.1).

 

The Bug has been resolved in Oracle 19.1. For the customer we set one of the parameters mentioned in the Doc (shown below) and the use case started working instantaneously.

       alter system set "_optimizer_generate_transitive_pred"=FALSE scope=both

Share This:

In a recent engagement with a Remedy customer we looked into their performance issue and discovered that the slowdown in the database was stemming from "enq:Index contention" waits.

 

Drilling down further into the waits we found that the waits were on the "S" tables corresponding to the Remedy forms "SMT: Social_FollowConfig" and "HPD: WorkLog". These "S" tables were introduced in Remedy's newest implementation of RLS that was CA (Controlled Availability = limited to  few chosen customers) in version 1902 and GA (General Availability) in version 1908.

 

The SQL statements that were waiting for an ITL (Interested Transaction List - at Oracle's block level) slot to open up were performing INSERTS into the two tables. The number of initial ITL slots when an index is created is specified by its INITRANS parameter. That parameter defaults to 1 for tables and 2 for indexes.

 

The default value of 2 for an index means that TWO transactions can each take one slot and perform an INSERT/UPDATE/DELETE operation on rows in the block.

 

If a third or fourth transaction comes along and needs to work on rows in the block and both the slots are taken Oracle can allocate additional slots, up to the table's/index's MAXTRANS parameter, provided there is space available in the block.

 

For the customer in question we had them increase the value of INITRANS for the indexes that were experiencing concurrency waits. The increase was from 2 to 10 in one case and 15 in another index.

 

The change can be accomplished by executing the following SQL command:

SQL> alter index <index name here> INITRANS <new value here>

Share This:

In a recent engagement with a Remedy customer we looked into their performance issue and discovered that the slowdown in the database was stemming from "enq:Index contention" waits.

 

Drilling down further into the waits we found that the waits were on the "S" tables corresponding to the Remedy forms "SMT: Social_FollowConfig" and "HPD: WorkLog".

 

Drilling down further into the waits we found that the waits were on the "S" tables corresponding to the Remedy forms "SMT: Social_FollowConfig" and "HPD: WorkLog". These "S" tables were introduced in Remedy's newest implementation of RLS that was CA (Controlled Availability = limited to  few chosen customers) in version 1902 and GA (General Availability) in version 1908.

 

The SQL statements that were waiting for an ITL (Interested Transaction List - at Oracle's block level) slot to open up were performing INSERTS into the two tables. The number of initial ITL slots when an index is created is specified by its INITRANS parameter. That parameter defaults to 1 for tables and 2 for indexes.

 

The default value of 2 for an index means that TWO transactions can each take one slot and perform an INSERT/UPDATE/DELETE operation on rows in the block.

 

If a third or fourth transaction comes along and needs to update rows in the block and both the slots are taken Oracle can allocate additional slots, up to the table's/index's MAXTRANS parameter, provided there is space available in the block.

 

For the customer in question we had them increase the value of INITRANS for the indexes that were experiencing concurrency waits. The increase was from 2 to 10 in one case and 15 in another index.

 

The change can be accomplished by executing the following SQL command:

SQL> alter index <index name here> INITRANS <new value here>

Share This:

BMC Software has identified an unauthenticated Remote Code Execution (RCE) vulnerability in Remedy Mid Tier.

Mid Tier versions 9.1, 18.05, 18.08, and 19.02 service packs, and patches are affected by this vulnerability.

For more information about this issue and the resolution, see the following links:

 

Thanks to Raphaël Arrouas and Stephane Grundschober for responsibly disclosing this vulnerability to BMC.

 

Best regards,

 

John Weigand
R&D Program Manager
BMC Software

Share This:

Introduction

 

In my last blog post I wrote about how to use a Tomcat container and war files for mid-tier testing.  In this one I'd like to show you how the combination of a database container and the Remedy silent install process can be used to speed up test server deployment.  Other advantages of this approach include

 

  • being able to consistently reproduce a system in the same state.
  • less concern about disposing of a system after testing as it is easily recreated.
  • use of database backups provides options for quickly restoring test systems to known good states.

 

The option to run Remedy with a flat file database went away many moons ago (bonus points if you can name the last version to offer this) and the current versions require either MS-SQL or Oracle, both of which are large and complex pieces of software.  The effort required to download, install, and configure these databases can add significantly to the time taken to create a test environment.  Wouldn't it be nice to be able to run a few commands and have a new database instance up and running, ready for use?  Containers to the rescue!

 

Both MS-SQL and Oracle are available as containers which means that a lot of the work needed to get them set up has already been done.  The Oracle container is more complex to manage, as well as being larger, so this article will focus on using MS-SQL.

 

Requirements

 

If you've read any of my previous articles you won't be surprised to find that we're going to be using a Linux system for our tests, specifically a CentOS 7 virtual machine.  At this point those of you that are familiar with the Remedy compatibility documents may be wondering about the combination of MS SQL and Linux.  There are two things to consider, firstly the use of Linux for the database platform and, secondly, whether a Linux based AR Server can talk to an MS SQL database.

 

Remedy has always been very platform agnostic with regards to the OS used to host the database.  If it looks like an MS SQL server, runs like an MS SQL server and squeaks like an MS SQL server, there's a very good chance that Remedy will run just fine.   On the second point, one of the consequences of the move to Java for the AR platform in version 9.0, was that the AR Server started using JDBC in place of the native database drivers of earlier releases.  This means that it is possible BUT NOT SUPPORTED for an AR Server running on Linux to use an MS SQL database.  Please note the highlighted comments in the previous sentence!  Yes, a Linux AR Server will work with MS SQL, but this should only be used for test systems as, at the time of writing, BMC DO NOT support this combination and you use it at your own risk.

 

OS Update and Docker Install

 

Start by making sure that the operating system is up-to-date, rebooting if many packages or the kernel are refreshed.

 

# yum update

 

Create working directories to store our files.  If you don't use /docker you will need to substitute your choice in some of the later commands.

 

# mkdir -p /docker/mssql

# cd /docker

 

If you haven't already installed Docker use these steps to add the software repository, install, and start the Docker engine.

 

# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# yum -y install docker-ce docker-ce-cli containerd.io

# systemctl start docker

 

Confirm that Docker is running with:

 

# docker version

Client:

Version: 18.03.1-ce

API version: 1.37

Go version: go1.9.5

Git commit: 9ee9f40

Built: Thu Apr 26 07:20:16 2018

OS/Arch: linux/amd64

Experimental: false

Orchestrator: swarm

 

We're also going to use a tool called docker-compose to help manage the database container configuration.  Note the version used in the command below may not be the latest, check the documentation if you want the most recent.

 

# curl -L https://github.com/docker/compose/releases/download/1.20.1/docker-compose-$(uname -s)-$(uname -m) -o /bin/docker-compose

# chmod a+x /bin/docker-compose

# docker-compose -version

docker-compose version 1.20.1, build 5d8c71b

 

MS-SQL Container

 

Microsoft publish container images for SQL 2017 on Linux in a public repository and there are some additional command line tools for MS-SQL that we will use.


# curl https://packages.microsoft.com/config/rhel/7/prod.repo > /etc/yum.repos.d/msprod.repo

# ACCEPT_EULA=Y yum -y -q install mssql-tools unixODBC-devel

 

To help make the management of the container a little easier we're going to use docker-compose.  This allows us to put all of the configuration options in a file rather than having to remember them each time we want to run a container.  Here's an example, copy and save this as a file called mssql2017.yml in the /docker directory:

 

version: '3'

services:

  mssql2017:

    image: mcr.microsoft.com/mssql/server:2017-latest

    container_name: mssql2017

    hostname: mssql2017

    ports:

      - 1433:1433

    volumes:

      - /docker/mssql:/var/opt/mssql

    environment:

      - ACCEPT_EULA=Y

      - MSSQL_SA_PASSWORD=P@ssw0rd

 

The various options we've used are:

 

Option

Purpose

image: mcr.microsoft.com/mssql/server:2017-latestThe image used to create the container.
container_name: mssql2017A friendly name for the container.
hostname: mssql2017Sets the hostname rather than using one that is automatically generated.  This will help us when we come to install AR as this is also used to name the database.

ports:

  - 1433:1433

Maps container ports to the outside world.

volumes:

  - /docker/mssql:/var/opt/mssql

Creates a shared volume to allow the database files to be stored on the docker host file system rather than inside the container.

environment:

   - ACCEPT_EULA=Y

   - MSSQL_SA_PASSWORD=P@ssw0rd

Pass environment variables used to set up the database.  Please note these requirements for the password from the Microsoft documentation :

The password should follow the SQL Server default password policy, otherwise the container can not setup SQL server and will stop working. By default, the password must be at least 8 characters long and contain characters from three of the following four sets: Uppercase letters, Lowercase letters, Base 10 digits, and Symbols.

 

All that is now required to start a database container is a single command:

 

# docker-compose -f mssql2017.yml up -d

Creating mssql2017 ... done

 

We can check that the container is running using the docker ps command and then query the database using the sqlcmd tool we installed earlier:

 

# docker ps

CONTAINER ID IMAGE                                      COMMAND                    CREATED     STATUS       PORTS                  NAMES

0ed2f8602ce9 mcr.microsoft.com/mssql/server:2017-latest "/opt/mssql/bin/sqlserver" 1 hours ago Up 8 seconds 0.0.0.0:1433->1433/tcp mssql2017

 

# /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P P@ssw0rd -Q "select getdate()"

-----------------------

2019-03-04 09:42:55.583

(1 rows affected)

 

We can also see that the container has created the default database files under the /docker/mssql directory:

 

# tree /docker/mssql/

/docker/mssql/

├── data

│   ├── master.mdf

│   ├── mastlog.ldf

│   ├── modellog.ldf

│   ├── model.mdf

│   ├── msdbdata.mdf

│   ├── msdblog.ldf

│   ├── tempdb.mdf

│   └── templog.ldf

├── log

│   ├── errorlog

│   ├── errorlog.1

│   ├── HkEngineEventFile_0_131963630182660000.xel

│   ├── log.trc

│   ├── sqlagentstartup.log

│   └── system_health_0_131963630191840000.xel

└── secrets

    └── machine-key

 

Believe it or not, that's all it takes to get a running MS-SQL Server instance on Linux!

 

The database container can be stopped with:

 

# docker-compose -f mssql2017.yml stop

Stopping mssql2017 ... done

 

Installing Remedy (quietly...)

 

One of the challenges of setting up a Remedy test system is the time taken to run all of the installers, particularly if you're also installing the ITSM Suite.  There's an added complication if you're using a headless Linux server in that using the installer GUI requires a suitable X-Windows server running on a PC to display the interface.  One of the under-used (I think) Remedy features is the ability for the installers to run in a silent mode, using a configuration file to provide all of the inputs that you usually type in using the GUI interface.  Using this option has a number of benefits:

  • no need for a GUI environment on Linux.
  • can be scripted so installations may be run remotely/unattended.
  • consistent and repeatable which makes it easier to recreate environments for testing.

All of the Remedy components can be installed using this silent mode and the process is covered in the relevant pages on the BMC docs website: 

 

Installing BMC Remedy AR System using silent mode - Documentation for Remedy Deployment 9.1 - BMC Documentation

Installing BMC Atrium Core using silent mode - Documentation for Remedy Deployment 9.1 - BMC Documentation

Performing the installation in silent mode - Documentation for Remedy Deployment 9.1 - BMC Documentation

 

So how does it work?  When you unpack the installer you will find an example options file, usually in a directory called utility.  This is a text file template that provides details of how to run the installer in silent mode and lists all of the options, and values where appropriate, that you need to provide.  The exact contents will vary depending on the product but the minimum required set of options may be a lot less than you expect.

 

Let's see what we would need to have in the silent options file to install an AR Server on the Linux system where we're running our MS-SQL container.

 

# cat silent_ar.txt

-J BMC_AR_SYSTEM_64_BIT_OR_32_BIT_JRE=64

-J BMC_JAVA_JRE_64_BIT_HOME_PATH=/opt/jre8

-J BMC_JAVA_EMAIL_ENGINE_SELECTED_FOR_32BIT=false

-J BMC_JAVA_AR_SERVER_SELECTED_FOR_32BIT=false

-J BMC_AR_APPLICATION_PASSWORD=arsystem

-J BMC_MIDTIER_PASSWORD=arsystem

-J BMC_AR_DSO_PASSWORD=arsystem

-J BMC_ARSYSTEM_INSTALL_OPTION=Install

-J BMC_DBONLY_UPGRADE_CONFIRM=false

-J BMC_USER_SELECTED_VIEW_LANGUAGES=en

-J BMC_USER_SELECTED_DATA_LANGUAGE=en

-A featureARSystemServers

-J BMC_DATABASE_TYPE=SQL_SERVER

-J BMC_DATABASE_UTF=true

-J BMC_DATABASE_HOST=sql2017

-J BMC_DATABASE_PORT=1433

-J BMC_DATABASE_LOGIN=ARAdmin

-J BMC_DATABASE_PASSWORD=arsystem

-J BMC_DATABASE_CONFIRM_PASSWORD=arsystem

-J BMC_SQLSERVER_WINDOWSAUTH_OR_SQLAUTH=SQLAUTH

-J BMC_DATABASE_INSTANCE=sql2017

-J BMC_DATABASE_DBA_LOGFILE_NAME=/var/opt/mssql/data/ARSysLog

-J BMC_DATABASE_DBA_LOGFILE_SIZE=2048

-J BMC_DATABASE_DBA_TABLESPACE_NAME=ARSystem

-J BMC_DATABASE_DBA_DATAFILE_NAME=/var/opt/mssql/data/ARSys

-J BMC_DATABASE_DBA_DATAFILE_SIZE=2048

-J BMC_DATABASE_DBA_LOGIN=sa

-J BMC_DATABASE_DBA_PASSWORD=P@ssw0rd

-J BMC_AR_USER=Demo

-J BMC_AR_PASSWORD=P@ssw0rd

-J BMC_AR_CONFIRM_PASSWORD=P@ssw0rd

-J BMC_AR_SERVER_NAME=arserver01

-J BMC_AR_SERVER_HOST_NAME=arserver01.bmc.com

-J BMC_JAVA_PLUGIN_PORT=9999

-J BMC_PORT_MAPPER_ENABLED=false

-J BMC_AR_PORT=46262

-J BMC_AR_PLUGIN_PORT=46276

-J BMC_ARSERVER_SAMPLE_DATA=true

 

You can see we're providing the Java path, product options such as language choice and sample data, database and Demo user credentials - everything that you would usually enter via the GUI.  A similar file could be used for an Oracle database but there would be some different options, which are explained in the sample file, that would need to be used.  Once this file has been created we just have to include it on the command line when running the installer:

 

# ./setup.bin -i silent -DOPTIONS_FILE=/path/to/silent_ar.txt

 

The installer will run, displaying some output on the screen and creating the usual log files, and setup the server.  Note that the file above does not include the mid-tier so you'll either have to add the options to install this or use a container based one created using the earlier blog post.

 

Once your server is up and running you can use the same process to install CMDB, AI, ITSM, SLM, SRM and so on...  With a bit of practice you could set up a script to create a full ITSM system from scratch with a single command.  Set it running as you leave for the the day and have it ready in the morning, perfect for testing.

 

There are other ways to use the MS-SQL container without installing Remedy from scratch, the next section looks at backing up and restoring the database from an existing AR Server which can then be configured to use the container based database.

 

Restore an Existing Remedy Database

 

If you don't want to wait for the silent install steps, and happen to have an existing Remedy system using MS-SQL, you could take a database backup and restore it into the container.  Transfer the .Bak file from the original system and copy it to the /docker/mssql/data directory - let's assume it is called ARSystem.Bak.    The sqlcmd utility is then used to perform the restore:

 

# /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P P@ssw0rd -Q "restore database ARSystem from disk='ARSystem.Bak' with replace, move 'ARSystem_data' to '/var/opt/mssql/data/arsys.mdf', move 'ARSystem_log' to '/var/opt/mssql/data/arsyslog.ldf'"

 

You may need to change the highlighted values depending on what your current system uses.  Progress will be shown as the restore takes place:

 

Processed 189816 pages for database 'ARSystem', file 'ARSystem_data' on file 1.

Processed 184 pages for database 'ARSystem', file 'ARSystem_log' on file 1.

Converting database 'ARSystem' from version 706 to the current version 869.

Database 'ARSystem' running the upgrade step from version 706 to version 770.

<lines snipped>

Database 'ARSystem' running the upgrade step from version 867 to version 868.

Database 'ARSystem' running the upgrade step from version 868 to version 869.

RESTORE DATABASE successfully processed 190000 pages in 3.662 seconds (405.344 MB/sec).

 

Then we need to create an ARAdmin account and make it the owner of the newly restored database, again some changes may be necessary to reflect your local names:

 

# /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P P@ssw0rd -Q "create login ARAdmin with password='arsystem', default_database=ARSystem, check_policy=off"

# /opt/mssql-tools/bin/sqlcmd -S localhost -d ARSystem -U sa -P P@ssw0rd -Q "exec sp_changedbowner ARAdmin, true"

 

Query the database as the ARAdmin user to confirm it has been restored as expected:

 

# /opt/mssql-tools/bin/sqlcmd -S localhost -U ARAdmin -P arsystem -Q "select schemaid, serverid, currdbversion from control"

schemaid serverid currdbversion

----------- ----------- -------------

4088      2            58

(1 rows affected)

 

That's it, we've created a new MS-SQL instance in a container and restored a Remedy database that's ready to use.  If you change the Db-Host-Name (and Db-user/Db-password if required)  in the ar.cfg of the AR Server that was using the original database system, setting it to the name of your docker host machine, it should be able to run using the restored copy.

 

Conclusion

I've started using this type of set up for most of my test environments and find it very flexible.  The ease and speed of creating a new database, along with the ability to install various combinations of Remedy products with minimal interaction, means that I no longer find it necessary to keep many different VMs lying around just in case I need a particular version.  Of course there are cases where this approach is not suitable, longer term test and development systems for example, but I'd encourage you to give it a go next time you need a system for a quick test or have a new version you want to evaluate.

 

Questions, comments & feedback are welcome.

Share This:

Introduction : This blog will give you the brief idea about the prerequisite needs to consider before performing AR system platform upgrade using installer. Also included prerequisite needs to consider while applying any patch/hotfix through d2p.

 

Basic configuration checks to be performed before upgrading the Remedy platform:

 

1. Validate ARSystemInstalledConfiguration.xml file. Perform the following steps:

  1. Verify that the ARSystemInstalledConfiguration.xml file exists in the <AR Installation Directory> folder. If the file does not exist, do not proceed with the upgrade. You can copy the ARSystemInstalledConfiguration.xml file from another server having the same version and make the server-specific changes (such as host name) in the file.

 

    b. Check the Product feature map section. The product feature map section contains all the features that are installed.
        If a feature that is currently running on the system, is missing from the list, you must add it to the ARSystemInstalledConfiguration.xml file.

        For example:
                 <productFeature backupOnUpgrade="false" id="featureARSystemServers"

 

<productFeature backupOnUpgrade="false" id="featureARServer"

 

<productFeaturebackupOnUpgrade="false" id="featureAREALDAPDirectoryServiceAuthentication"

 

<productFeaturebackupOnUpgrade="false" id="featureARDBCLDAPDirectoryServiceAuthentication" independentOfChildren="false" parent="featureARServer" rebootRequiredOnInstall="false" rebootRequiredOnUninstall="false" rebootRequiredOnUpgrade="false" requiredDiskSpaceMode="default.linux" state="INSTALLED" visible="true">

 

     c. Before starting the upgrade, verify the release version, major and minor version.

         For example, <version majorVersion="1" minorVersion="00" releaseVersion="9"/>,

         which indicates that the current version installed is 9.1.00.

 

     d. Verify that the following properties have correct hostnames/IP addresses.

 

<name>BMC_AR_SERVER_NAME</name>

<name>BMC_AR_SERVER_HOST_NAME</name>

<name>BMC_AR_SERVER_CONNECT_NAME</name>

<name>BMC_AR_SERVICE_NAME</name>

<name>BMC_EMAIL_SERVICE_NAME</name>

<name>BMC_MIDTIER_TOMCAT_SERVICE_NAME</name>

<name>BMC_MIDTIER_INSTANCE_NAME</name>

 

      e. If the installed version is 9.1.04 or later, verify the 'BMC Remedy MidTier File Deployer' property.

          This property should exist for 9.1.04 or later versions.

<property>

      <name>BMC Remedy MidTier File Deployer - </name>

      <environmentVariable scope="SYSTEM">

      <name>BMC_JAVA_HOME</name>

                  <value>C:\Java\jdk1.8.0_45\jre</value>

      </environmentVariable>

</property>

 

2. Verify if the service name in the system registry is same as 'AR_Server_Host_Name' property in 'ARSystemInstalledConfiguration.xml' (WINDOWS SPECIFIC)

 

When you create a clone of an existing environment, you must also update the Windows registry in the cloned environment. If you do not update the registry, the following issues may occur:

  • Throwable=[java.io.FileNotFoundException: D:\Program Files\BMC Software\ARSystem\armonitor.exe (The process cannot access the file because it is being used by another process)Verify the following
  • Throwable=[java.io.FileNotFoundException: D:\Program Files\BMC Software\ARSystem\arcatalog_eng_W_win64.dll (The requested operation cannot be performed on a file with a user-mapped section open)  java.io.FileOutputStream.open0(Native Method)

 

  1. Verify the service name from the registry.

HKEY_LOCAL_MACHINE\SOFTWARE\Remedy\ARServer\<BMC_AR_SERVER_NAME>\ServiceName

    b. Verify the service name from the registry

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy Action Request System Server <BMC_AR_SERVER_NAME>

 

    c. Under registry, verify the JVM related options

    • JARS are pointing to the correct path
    • All required JARS exist.

 

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy Action Request System Server <BMC_AR_SERVER_NAME>\Parameters

 

    d. Verify the email engine service name from in registry against the one present inside ARSystemInstalledConfiguration.xml

 

        HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy Email Engine - <BMC_AR_SERVER_NAME> 1

    e. Verify the email engine parameters are:

    • Pointing to correct paths.
    • All required JARS exist in these paths.

 

 

f. Verify the flashboard service name in registry against the one present inside ARSystemInstalledConfiguration.xml

           HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy Flashboards Server - <BMC_AR_SERVER_NAME>

 

 

g. Under registry, verify the JVM related options

    • JARS are pointing to the correct path
    • All required JARS exist

           HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy Flashboards Server - <BMC_AR_SERVER_NAME>\Parameters

 

 

        If the current installed version is 9.1.04 or later, verify the following:

        h. Verify the file deployer service name from registry against the one present in ARSystemInstalledConfiguration.xml

        HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy MidTier File Deployer - <BMC_AR_SERVER_NAME> 1

 

i. Under registry, verify the JVM related options

    • Jars are pointing to correct path
    • All appropriate jars exist.

          HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\BMC Remedy MidTier File Deployer - <BMC_AR_SERVER_NAME> 1\Parameters

 

 

3. Check servgrp_board table. opFlag should be “1” for Administrator server.

4. Make sure that Object modification logs is disabled. If turned ON, it causes slowness during installation because the entire AR Server metadata undergoes changes during installation.

    Link for Object modification logs:

    https://docs.bmc.com/docs/display/ars91/Using+the+object+modification+log

 

5.If you are upgrading from 7.x/8.x to 9.x or later, check the ‘Application Statistics Configuration’ form. All forms listed here must be part of some application.

 

6. Check DB Version in Control table. Refer to the following link to verify if the DBVersion column of the Control table has the value same as that of the AR Server version.

   https://communities.bmc.com/docs/DOC-37267

 

7. Execute the following queries to find out if there are invalid views in the database. Work with your DBA to fix/remove invalid views. This is one of the sure shot reasons of upgrade            failure.

Step 1: Execute the following SQL

select count(1) from    user_objects  where  status = 'INVALID' and object_type = 'VIEW'

Step 2: If the output is greater than 0 , execute following script to identify list of forms associated with invalid views :

select name, schemaid, case

        when overlayprop = 0 then 'Unmodified'

when overlayprop = 1 then 'Base'

when overlayprop =  2  then 'Overlay'

when overlayprop = 4 then 'Custom' end as CustomizationType

from arschema  where 'T' || schemaid in (select OBJECT_NAME  from  user_objects  where status = 'INVALID'  and object_type = 'VIEW')

Step 3 (Compile invalid views) : Execute following script to compile invalid views, change schema name if it is different from ARADMIN

EXEC DBMS_UTILITY.COMPILE_SCHEMA( schema => 'ARADMIN', compile_all => FALSE)

 

8. Make sure that the ‘Temp’ directory is clean before trying to perform a subsequent upgrade attempt.

 

9. Metadata inconsistency is one of the reasons for upgrade failure. Make sure you run checkdb utililty before the upgrade and resolve relevant inconsistencies.

 

Example:

 

 

Following error was seen in an upgrade attempt for one of the customers:

 

(Apr 11 2019 04:36:52.311 PM +0200),SEVERE,com.bmc.install.product.arsuitekit.platforms.arsystemservers.arserver.ARServerOracleManageUpgradeDatabaseTask,

  LOG EVENT {Description=[[SQLERROR] [DESCRIPTION] Failed to upgrade the database schema],Detail=[[SQLERRORCODE]=0 [SQLMESSAGE]=Failed to run SQL statement [ALTER TABLE SERVGRP_RESOURCES MODIFY ( JMSRESOURCES CLOB NOT NULL )] Due to [ORA-22296: invalid ALTER TABLE option for conversion of LONG datatype to LOB

][SQLSTATEMENT]=]}

(Apr 11 2019 04:36:52.312 PM +0200),SEVERE,com.bmc.install.product.arsuitekit.platforms.arsystemservers.arserver.ARServerOracleManageUpgradeDatabaseTask,

  THROWABLE EVENT {Description=[Failed to upgrade the database schema]},

Throwable=[java.sql.SQLException: Failed to run SQL statement [ALTER TABLE SERVGRP_RESOURCES MODIFY ( JMSRESOURCES CLOB NOT NULL )] Due to [ORA-22296: invalid ALTER TABLE option for conversion of LONG datatype to LOB

]

 

The failed SQL statements are built by the upgrade installer using the information in ‘ReadARServerDatabaseModel.xml’ and ‘TransformedARServerDatabaseModel.xml’. These files are created by the upgrade installer in the ‘Temp’ directory during installation.

 

There were two issues in this scenario:

 

  • Installer was trying to alter a column in SERVGRP_RESOURCES table, but the column was already altered. This could be due to multiple attempts of upgrades without cleanly reverting database and the file system.
  • SQL statement was trying to update Datatype CLOB with ‘NOT NULL’ option (Oracle has some issues with it) which was not allowing as per oracle. On a successful installation environment, we don’t have NULL values ) Inhouse we haven’t seen NULL value, but for this customer it was NULL. Therefore, we manually updated same values (CLOB and NOT NULL)here, deleted everything from ‘Temp’ directory before upgrade.

 

 

NOTE: If you run into these issues, contact BMC support to get appropriate assistance. Do not perform any manual changes on the database without consulting BMC Support.

 

 

Basic Configuration checks to be performed before applying a patch / hotfix using D2P:

1. Make sure that correct server entries in ‘AR System Monitor’ form exist. Delete the orphan or duplicate entries and restart the file deployer service of each of the servers in the server group.

 

If the issue still persists, go to AR System Installation Directory. Make sure that ‘monitor-ARServer-guid.properties’ file is present. It should have unique GUID for each server.

 

If the GUIDs are not unique, you must create entries with unique GUIDs. Perform the following steps:

    • Delete the file monitor-ARServer-guid.properties
    • Remove all entries from AR System Monitor form
    • Restart file deployer service.

AR System Monitor Form

 

 

2. Make sure to wait for few mins after you import the package, before clicking on ‘Deploy’ button.

 

3. Make sure that ‘AR System Single Deployment Payload’ form and ‘AR System Single Deployment Status’ form should not have any orphan records for the payload which we are           trying to deploy.

 

4. Make sure that File deployer service is running on all servers in the server group.

 

5. Make sure all the processes that are part of the AR System Service in armonitor.cfg file, are started without any issue. Please check armonitor.log file to verify this.

     For one of the customers was having duplicate entries for pluginsvr process. This caused errors “Address already in use” error, visible in armonitor.log. Deployment for one of                the payloads failed due to the error.

 

6. During deployment if any process fails to start / stop then you need to check following.

     Following error will be visible in file deployer log.

     ITSM Deployment Rollback caused by error: com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:NormalizationEngine failed to start.

  • FileDeployer signals the ARMonitor to start/stop Payload associated processes. During deployment if a specific process failed to start, then the related logging will be available in armonitor.log file with signal / log stmt saying … Starting Process <processName>  … (basic logging should be enough.)
  • If the process can be manually restarted then to troubleshoot from d2p perspective, execute the ARMonitor_Admin.bat file located in ARSystem directory manually to start / stop the specific process.

 

7. Prior to 1808 , it was mandatory to have process started for which we want to do payload deployment. Otherwise payload used to fail.

     For. E.g Email Engine process, DSO process etc.

     We can use following workaround to bypass this without starting that particular process.

  • import d2p package
  • click on deploy (This will create the Payload entries)
  • Run the below query (To remove the associated process entry for ex |;BMC:EmailEngine )
  • UPDATE < schemaid of ‘AR System Single Point Deployment Payload’> SET C49110 = 'BMC:ARServer|;BMC:JavaPluginServer|;BMC:DSOJServer|;BMC:CarteServer'          WHERE C49102 = '<Payload GUID>'

        Please get value of C49110 by viewing the payload ’Process Type’, copy it and remove only email engine from it.

        e.g. Screenshot is of different d2p package

 

  • Start the utility (arpayloadutility.bat) for deployment .
Share This:

We have seen requirements where Admins like to create users with limited admin privileges like give access only to selected forms like User/Group, Admin console, Server group admin console. Also give limited access to Dev studio like give both Base development and Best practice modes access or limit only to Best practice mode.

For such needs there is a feature in Remedy called Struct admin which are documented at below DOC pages.

Special groups in BMC Remedy AR System - Documentation for Remedy Action Request System 9.1 - BMC Documentation

Struct Admin group permissions - Documentation for Remedy Action Request System 9.1 - BMC Documentation

 

This blog page is created to demonstrate how struct admin users can be created and used. There are two use cases demonstrated in this blog. You may try more cases based on your need by reading documentation pages.

 

Case 1 - Full Struct Admin User:

This user should be able to access Server Information Console (AR System Administrator Console), Server Group Log Management Console, Centralized Configuration console, User and Group form. In Dev studio, can use both Best Practice Customization  Mode and Base Development mode.

Detailed steps to achieve this use case is available at Configuring Full Struct Admin.docx

 

 

Case 2 - Overlay Struct Admin User:

This user should be able to access ccess Server Information Console (AR System Administrator Console), Server Group Log Management Console. In Dev Studio only have access to Best Practice Customization Mode.

Detailed steps to achieve this use case is available at Configuring Overlay Struct Admin.docx

 

 

 

Thanks to Chandrakumar Palanisamy for his valuable time and inputs in creating this blog.

 

Hope you will find this blog useful.

 

Note: Though basic testing is done for the use cases, extensive testing is left to the user. Make sure to test your implementation and use case properly before making anything live on PROD.

Share This:

In interactions with customers it is often noted that getting table/view information is a back and forth operation between Support/PE and the customer.

 

The accompanying zip file contains SQL scripts, written for the Oracle database, will help gather information about a table or view in one fell swoop so to speak.

 

UPDATE

The Table_Info script has been updated to now show Function-Based Normal indexes too.

 

TABLE INFO

"table_info_wrapper.sql" can be run as SYS or ARADMIN (or the Remedy schema owner as the case may be).

                * * * * "table_info_wrapper.sql" accepts the table name to be queried and calls 2 other scripts, one of which gathers index statistics.

VIEW INFO

"view_info_wrapper.sql" needs to be run as SYS.

                 * * * * "view_info_wrapper.sql" also accepts the view name to be queried and gets its DDL information from dbms_metadata.get_ddl.

Share This:

‘USER THEME’ - OVERVIEW

Remedy 19.02 release includes a new feature called as “User Preference Theme”. This is a Remedy MidTier feature and as the name suggests, it allows users to set preference to visualize mid-tier UI with select theme colors.

Users now have flexibility to select a theme from list of themes available in drop down allowing users with different look and feel of the MidTier UI than just out-of-the-box one.

Remedy Customers (administrators) can to build custom themes (CSS files) and publish it to ‘User Preference theme’ drop down list. Thereby allowing Customers to define theme as per corporate branding guidelines.

This feature is user specific - so user can choose their theme or go with the default theme (the current look and feel of MidTier).

The Remedy administrator can enable themes at the company level or if the themes are disabled then the users will continue to see the default Theme which applies to all users.

 

Pre-requisites: Remedy MidTier 19.02 (and above) is required to enable ‘User Theme’

 

SKINS VS THEMES

Customers have been using the “Skins” feature in Remedy where the customer can change the look and feel of a field/form etc. With the advent of Themes feature, the administrator has a choice to either use Skins or use Themes. However, both cannot be used together.

Skins feature is applicable for all users in the system. Whereas the Themes feature is “user preference” based. So, the Remedy Admin can enable or disable Themes. Once enabled, User has the choice to select theme of his choice.

Skins requires Remedy development skills for changing the look and feel. Themes however are CSS based and have wide range of options available to change the look and feel.

User Theme can be enabled and disabled through MidTier Configuration at run time.

Theme applies to MidTier Forms only & not to any third party embedded components like BIRT or Flash player

 

STEPS TO ENABLE THEMES

Open “AR System User Preference” Form -- Create drop down Menu field with field ID “24016” in Web tab and save it to all Views, as shown in below screenshot

               

Import Menu “UserPrefTheme” and Add same into Menu name of User Theme field as show in image-1

Set Expand Box Hide available in side Display Property as shown in image -2

 

Image 1

Image 2

Change value of property arsystem.showCfgThemeField from false to true available inside MidTier config.properties.

Put all theme CSS files inside MidTier “resources\userpreftheme\stylesheets” path and restart MidTier

  1. After restart, open MidTier Config tool à AR Server Settings and click edit or add server à two new fields will appear “Enable User Theme “ check box and ”Default Theme” text Box.
  • “Enable User Theme” mark this as checked and give some default theme CSS file name in “Default Theme”  à this applies to all user if user haven’t selected any theme from user preference

Image 3

 

After applying theme landing console look like

AVAILABLE THEMES

Available sample 2 themes are attached to this blog.

 

HOW TO CREATE CUSTOM CSS THEME

There are two CSS files shared with this post. If customer want to create new theme then they can do so by changing their Background color and font color based on need.

Alternatively, Customers could use css class listed available in shared CSS file and redesign it there as per their requirement. After creating new CSS file, add the file name in “userTheme” menu list and flush MidTier cache.

 

For example, if you want to change Tab background color just make changes in below listed CSS

.OuterOuterTab,.Tab,.OuterTab .Tab, .OuterTab .TabLeft, .OuterTab .TabRight, .ScrollingTab .Tab,.OuterTab .TabRightRounded,.ScrollingTab .TabRightRounded

{

background: #f0f0f1 !important; /*change background color for Tab */

}

 

 

CC - Remedy ITSM Remedy AR System

Rahul Vedak Abhijeet Gadgil Ravi Singh Rawle Gibson

Filter Blog

By date:
By tag: