Share This:

As per my previous blog post about Queues and Threads, here is the follow up post regarding Private Queues.

 

As a reminder - the thread count (min & max number of threads that are defined against a queue) is very important because we have 1-to-1 relationship with database. In other words one thread connection equals one session with the ARSystem database by the ARAdmin user.

 

If you don’t have an adequate thread configuration, then your AR Server is at risk of utilizing all of the threads (dependent on workload) and if this happens users will start seeing timeout or RPC errors (ARERR 92, 93 or 94's).

 

Therefore, defining additional Private Queues offers more threads to do the work and more ways to divide the workload by segregating the jobs. They also provide a method of throttling down certain activities.

 

Private Queues are represented by an RPC program number in the following ranges:

 

390621 - 390634

390636 - 390669

390680 - 390694

 

 

What components of Remedy can avail of Private Queues?

 

There are certain components within a Remedy environment whereby private queues can be defined, which depending on your situation - may be beneficial for you to do.

 

These components are:

 

  • Email Engine
  • Assignment Engine
  • Approval Server
  • DSO (Distributed Server Operations)
  • Normalization Engine
  • Reconciliation Engine
  • Service Level Management (Business Rules Engine)
  • Service Request Management (Command Automation Interface)

 

 

In this blog post, we'll provide guidance on how to configure all these private queues.

 

In order for these components to use a private queue, you need to both Define the private queue, and then Assign the component to it. Without performing both actions, the private queue cannot be used.

 

Defining a private queue can be done in 2 ways:

 

  1. Manually updating the ar.cfg/conf
  2. Adding a record into the Server Information form -> Ports and Queues tab -> Server Ports table

 

When defining the private queue, you also set the minimum and maximum number of threads for the queue to use. Therefore, it’s a good idea to know what private queues you’ll use for each component prior to defining them – as there can be some restrictions on the thread numbers.

 

For simplicity, in the rest of the blog post – I will refer to defining the private queue as adding a ‘Private-RPC-Socket:  390xxx   x  y’ parameter to the ar.cfg/conf, although you can choose which way you’d like to define them.

 

Defining a private queue in the Ports and Queues tab will start the queue immediately, as a signal is sent to re-read the ar.cfg/conf. However, you still need to assign the queue to its relevant component.

 

The way in which you assign the private queue can differ per component - this will be covered in each of the component’s sections below.

Likewise the final action item to perform prior to the private queue actually being used can differ, and this will also be covered.

 

 

Email Engine

 

The RPC socket/Private Queue that the Email Engine can use is defined in the EmailDaemon.properties file located at <ARS_HOME>\AREmail\EmailDaemon.properties

 

 

To Assign - Choose a Private Queue in the range and define it in this file via the line:

 

com.bmc.arsys.emaildaemon.<serverName>.RPC=390xxx

 

To Define - Using the same value, add the following line to the ar.cfg/conf:

 

Private-RPC-Socket:  390xxx   x  y

 

The number of threads connecting to the AR System Server is the number of mailboxes. Typically, this would be 2 - 1 for the outgoing and 1 for the incoming mailbox (so the values for x and y above would be 2).

 

You will need to restart the Email Engine for the change to take effect.

 

 

Assignment Engine

 

To configure the Assignment Engine to use a private queue, you'll need to add the following 2 parameters to the ar.cfg/conf:

 

To Assign - Choose a Private Queue in the range and define it in this file via the following parameter in the ar.cfg/conf:

 

AE-RPC-Socket: 390xxx

 

To Define - Using the same value, add the following line to the ar.cfg/conf:

 

Private-RPC-Socket:  390xxx   2   2

 

You can use any private queue number, for example 390630.

Restart the AR System Server for the change to take effect

 

 

Approval Server

 

Prior to version 8.0, the Approval Server was single threaded.

 

To configure the Approval Server to use a private queue (in 8.1 it should be pre-configured after the Atrium Core installation), you need to first define the queue (min & max of 2 threads is sufficient, for versions lower than 8.0, set the man & max thread count to 1) and then either assign it via the application or via the ar.cfg/conf.

 

To Assign - You can assign the private queue via the Approval Console:

 

Approval Administration Console -> System Settings -> Basic tab -> Private AR Server RPC Socket

 

Approval Settings 8.1.png

----------------------------------

You can assign this directly in the ar.cfg/conf also, you just need to add this parameter:

Approval-RPC-Socket: 390xxx

----------------------------------

 

To Define - Using the same value, add the following line to the ar.cfg/conf:

 

Private-RPC-Socket:  390xxx   2   2

 

You can use any private queue number, for example 390680 – this is what is used Out of the Box.

 

If you do need to configure this, if it’s all done via the application – the private queue will be used automatically.

 

 

DSO (Distributed Server Operations)

 

To configure DSO to use a private queue, you can use either a local, remote or both RPC Sockets.

 

a) Local server:

To Assign - Go to AR System Administration Console -> General -> Server Information -> Connection Settings -> DSO Local RPC Program Number

 

DSO Local RPC Program Number 8.1.png

 

NOTE: When you set the above, this line is added to the ar.cfg/conf:

 

DSO-Local-RPC-Socket: 390xxx

 

 

To Define - You will need to define the private queue in the ar.cfg/conf using:

 

Private-RPC-Socket:  390xxx   x  y

 

 

b) Remote server:

To Assign - go to AR System Administration Console -> General -> Server Information -> Connection Settings -> DSO Server Setting Table (table on the left).

 

DSO Server Setting Table.png

 

NOTE: When you set the above, this line is added to the ar.cfg/conf:

 

DSO-Target-Connection: <server_name>:<RPC_number> <port>

 

To Define - Ensure the private queue that you set for the remote server is actually set on the remote server as the 'DSO Local RPC Program Number'.

 

If you configure these private queues via the application, they will be used automatically.

 

 

Normalization Engine

 

There are 2 different private queues for the Normalization Engine that can be set, the AR RPC Queue and the CMDB RPC Queue.

 

Once set, they are recorded in the Normalization Engine's pluginsvr_config.xml.

 

To Assign - To assign these private queues, go to:

 

Atrium Core Console -> Applications -> Normalization -> Edit Configuration -> System Configuration -> RPC Queue

 

For the AR RPC Queue, you can use 390621 and for the CMDB RPC Queue, you can use 390699

 

NE System Config 2.png

 

This is how they are displayed in the <ATRIUM_CORE_HOME>\cmdb\plugins\ne\pluginsvr_config.xml:

----------------------------------------

<com_bmc_ne_system_ARRpcQueue>390621</com_bmc_ne_system_ARRpcQueue>

<com_bmc_ne_system_CMDBRpcQueue>390699</com_bmc_ne_system_CMDBRpcQueue>

----------------------------------------

 

To Define - You will need to define these privates queue with min & max thread counts in the ar.cfg/conf:

 

Private-RPC-Socket: 390621 x y

Private-RPC-Socket: 390699 x y

 

The recommended min and max thread values for both of these private queues are 1* #CPUs of the machine.

 

If you configure these private queues via the application, they will be used automatically.

 

 

Reconciliation Engine

 

To configure the Reconciliation Engine to use a private queue (in 8.1 it should be pre-configured), you can either assign the queue in the ar.cfg/conf or via the application.

 

To Assign - Via the application, go to:

Atrium Core Console -> Applications -> Reconciliation -> Edit Server Configuration -> RPC Socket

 

RE Server Config Editor 2.png

 

The recommended queue number to use is 390698 – this is what is used Out of the Box.

 

To Define - Using the same value, add the following line to the ar.cfg/conf (use the following thread values):

 

Private-RPC-Socket:  390698   2   8

 

If you do need to configure this, if it’s all done via the application – the private queue will be used automatically.

 

 

Service Level Management

 

The BRIE Server is single threaded, and you can configure it to run on its own private queue.

 

That private queue will be used in effect for everything except the actual build actions, which will go via the Admin queue (390600)

 

To configure the BRIE Server to use a private queue, you need to add the following 2 parameters to the ar.cfg/conf.

 

To Assign - Choose a Private Queue in the range and define it in this file via the following parameter in the ar.cfg/conf:

 

BR-RPC-Socket: 390xxx

 

To Define - Using the same value, add the following line to the ar.cfg/conf:

 

Private-RPC-Socket:  390xxx   1   1

 

Restart the AR System Server for the change to take effect.

 

 

Service Request Management (Command Automation Interface)

 

You can configure CAI to run on its own private queue to use exclusively for its processing.

 

To Define - Choose a free private queue and define it in the AR Server (use the following thread values):

 

Private-RPC-Socket: 390xxx 2 6

 

To Assign - Navigate to the CAI:PluginRegistry form (Application Console -> Customer Configuration -> Foundation -> Advanced Options -> Command Automation Interface - PlugIn Registry) and create a new record here with the Private queue number you just defined.

 

CAI PluginRegistry.png

 

Restart the AR System Server for the change to take effect.

 

 

Verifying what Private Queues are set

 

If you want to verify what private queues are defined in your environment, an easy way to do this is to search for the string ‘RPC-Socket’ in the ar.cfg/conf, and extract the results into a separate file.

 

From here, you can see a list of:

  • default queues
  • private queues
  • components assigned to a specific private queue

 

Not all private queues can be accounted for using this method, as for some components the private queue is assigned elsewhere (Email Engine, Normalization Engine etc – see above), but it gives you a good start.

 

If you want to see what private queues are being utilized in your Remedy environment, simply enable the API log for a period of time and review it.

 

You will see two columns when looking at the log that identify which queue the API call came in on (<Client-RPC: xxxxxx >) and which queue the request was processed on (<Queue: xxxx>):

 

<API >     <TID: 0000000484>     <RPC ID: 0000000124>     <Queue: List      >     <Client-RPC: 390620   >

<API >     <TID: 0000002512>     <RPC ID: 0000000123>     <Queue: Fast     >     <Client-RPC: 390620   >

<API >     <TID: 0000002512>     <RPC ID: 0000000123>     <Queue: Fast     >     <Client-RPC: 390620   >

<API >     <TID: 0000003940>     <RPC ID: 0000000126>     <Queue: List      >     <Client-RPC: 390620   >

<API >     <TID: 0000003940>     <RPC ID: 0000000126>     <Queue: List      >     <Client-RPC: 390620   >

<API >     <TID: 0000000308>     <RPC ID: 0000000125>     <Queue: Fast     >     <Client-RPC: 390620   >

<API >     <TID: 0000001872>     <RPC ID: 0000000127>     <Queue: List      >     <Client-RPC: 390620   >

<API >     <TID: 0000000384>     <RPC ID: 0000000089>     <Queue: Prv:390680> <Client-RPC: 390680  >

<API >     <TID: 0000000308>     <RPC ID: 0000000090>     <Queue: Fast      >     <Client-RPC: 390600  >

<API >     <TID: 0000000384>     <RPC ID: 0000000089>     <Queue: Prv:390680> <Client-RPC: 390680  >

<API >     <TID: 0000002120>     <RPC ID: 0000000091>     <Queue: Prv:390680> <Client-RPC: 390680  >

  

 

As always if you need any assistance in identifying what private queues you have set, please raise a support issue and reference this blog post.