Share This:

Introduction

 

As per the FTS Fortification blog post, High Availability (HA) is now a feature of Full Text Search in 7.6.04 SP5 and 8.1 SP1.

 

There has been some confusion about how to configure this optional feature in a Server Group environment, so this month's blog post will address this. The key to knowing how to configure it lies in understanding how it works - and in order to understand this, you'll need to become familiar with a few key terms.

 

Glossary of Terms

 

TermDefinition
FTS Indexing Server

Has a ranking record for FTS in the AR System Server Group Operation Ranking form

Writes index data for FTS on its local disk

Provides FTS search services to itself and search-only servers in the group

Search-only Server

Servers in the group that are not indexing servers

Does not have a ranking record in the AR System Server Group Operation Ranking form

FTS Configuration formThe new form deployed by SP5 (and already in 8.1) that makes configuring FTS easier
Primary FTS Plugin Server

Also known as the writer

Active only on Single Server and Server Group-Primary options of the FTS Configuration form

Provides indexing capabilities to the Server Group

Provides search services to itself only

Denoted by the following line in the armonitor.cfg/conf:

"<JAVA_HOME>" -Xmx1024m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -classpath "<ARS_HOME>\pluginsvr\fts\primary;<ARS_HOME>\pluginsvr\fts\core;<ARS_HOME>\pluginsvr;<ARS_HOME>\pluginsvr\arpluginsvr7604_build002.jar" com.bmc.arsys.pluginsvr.ARPluginServerMain -x <SERVER_NAME> -i "<ARS_HOME>" -m

Secondary FTS Plugin Server

Also known as the reader or searcher

Active only when the Server Group-Primary option is selected in the FTS Configuration form

Provides search services to search-only servers

Denoted by the following line in the armonitor.cfg/conf:

"<JAVA_HOME>" -Xmx1024m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -classpath "<ARS_HOME>\pluginsvr\fts\secondary;<ARS_HOME>\pluginsvr\fts\core;<ARS_HOME>\pluginsvr;<ARS_HOME>\pluginsvr\arpluginsvr7604_build002.jar" com.bmc.arsys.pluginsvr.ARPluginServerMain -x <SERVER_NAME> -i "<ARS_HOME>" -m

FTS QueueAnother name for the ft_pending table in the ARSystem database

 

 

Background Info

 

High Availability for FTS by using multiple indexing servers

 

Before Service Pack 5, BMC Remedy Full Text Search (FTS) required that one server in a server group act as the FTS indexing server. The FTS index data had to reside on a local disk on that server.

 

With version 7.6.04 Service Pack 5, more than one FTS indexing server can exist and act independently of another FTS indexing server in a server group. This provides service failover should the active FTS indexing server go down or the search-only server is unable to communicate with it.

 

How failover occurs in a High Availability configuration

 

As before, with 7.6.04 SP5 the FTS index data must still reside on a local disk on the server that will be the FTS indexing server (and not on a NAS or NFS drive).

 

To provide failover, you can configure multiple servers in the server group where each acts as an independent FTS indexing server. Each FTS indexing server manages its own separate local replica of the full text index data.

 

When an indexing action takes place, each FTS indexing server indexes it independently. So if you create a form and then index a field on that form, each FTS indexing server will index that field. This is achieved by populating the ft_pending table with x amount of records for the indexing action, where x is the number of indexing servers in the group. This ensures that the individual FTS indexes will have the same data and the FTS searches will be consistent upon FTS failover. When failover occurs, the records for the indexing server that is currently down will backlog in the ft_pending table. When it’s back up, these records will get processed as normal.

 

As you can have multiple FTS indexing servers (each with its own current copy of the FTS data), you can fail over the FTS search operations to a second indexing server.

 

An AR Server is designated as an FTS indexing server by having a ranking record for FTS in the AR System Server Group Operation Ranking form. Each FTS server defined in the ranking form acts as an indexing server and provides FTS search services to other servers in the server group. Each defined FTS server will always index, regardless of its ranking order.

 

Servers in the server group that are not ranked for FTS are search-only servers. The search-only servers are user facing servers and they connect to one of the FTS indexing servers to search the FTS data.

 

To clarify this further, an FTS indexing server will run the Java Plugin Server processes and its FTS Java Plugins will be configured. Whereas for a search-only server, it will not run any FTS Java Plugin Server processes, and will point to an FTS indexing server where the plugins are running.

 

 

The following diagrams show the architecture of FTS in a Server Group before the High Availability feature was available and the architecture of it afterwards.

 

Before HA - big.jpg

Figure 1.1 Before HA – No failover (configured for performance in a Server Group)

 

After HA - big.jpgFigure 1.2 HA – the dotted line represents the failover option

 

 

The Server-Plugin-Alias parameter in the ar.cfg/conf files of the servers specifies the initial connection (this is configured using the new FTS Configuration form):

  • The FTS indexing servers connect to their own local indexes.
  • The other servers in the group can connect to any of the indexing servers.

 

If when a search-only server issues a search request to an indexing server, and the indexing server does not respond to the request (for whatever reason; a connection problem or the indexing server is re-indexing etc), then the search-only server that issued the request will point to the next indexing server as per the ranking form.

 

When testing this, I used a Server Group consisting of 3 servers. For simplicity I'll call them Server A, Server B & Server C. Server A & Server B are indexing servers and Server C is a search-only server.

 

On initial configuration, the Server-Plugin-Alias parameter for the FTS Java Plugin in the search-only server points to the indexing server that is ranked 1.

 

Server-Plugin-Alias: ARSYS.ARF.FTS ARSYS.ARF.FTS Server A:9977

 

If this indexing server has a problem with this request, the search request will go to the server that is ranked 2.

 

Failover occurs from a the perspective of the AR Server that initiated the Search request, rather than the FTS Plugin Server that was having problems.

 

 

Configuring FTS in a Server Group

 

An FTS indexing server uses one Java Plugin Server as the writer (primary) and another Java Plugin Server as the reader (secondary). These are both enabled on an FTS indexing server.

 

The writer (primary) FTS Java Plugin Server serves as both the reader and writer for an FTS indexing server, as well as a writer for all servers in the group.

 

The reader (secondary) FTS Java Plugin Server serves as the reader for all the other search-only servers in the group, so you must configure all other search-only servers to connect to the reader instance running on an FTS indexing server. It’s recommended that all search-only servers connect to the ranked 1 indexing server.

 

To illustrate this further, review the 2 FTS Java Plugin Server lines in the armonitor.cfg:

 

"<JAVA_HOME>" -Xmx1024m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -classpath "<ARS_HOME>\pluginsvr\fts\primary;<ARS_HOME>\pluginsvr\fts\core;<ARS_HOME>\pluginsvr;<ARS_HOME>\pluginsvr\arpluginsvr7604_build002.jar" com.bmc.arsys.pluginsvr.ARPluginServerMain -x <SERVER_NAME> -i "<ARS_HOME>" -m

 

"<JAVA_HOME>" -Xmx1024m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -classpath "<ARS_HOME>\pluginsvr\fts\secondary;<ARS_HOME>\pluginsvr\fts\core;<ARS_HOME>\pluginsvr;<ARS_HOME>\pluginsvr\arpluginsvr7604_build002.jar" com.bmc.arsys.pluginsvr.ARPluginServerMain -x <SERVER_NAME> -i "<ARS_HOME>" -m

 

In an FTS Indexing Server, both of these are uncommented, so the armonitor process starts these on AR Server startup

In a Search-only server, both of these are commented out.

 

NOTE: There is no search failover for the writer plugin running on the FTS indexing server. The secondary (reader) Plugin Server serves the search requests from search-only servers and does not serve as a backup to the primary writer plugin. So if a user is connected into this server, is running a FTS search, and the writer plugin has problems - this search request will fail. Given this, you only get High Availability from the search-only servers.

 

The secondary (reader) FTS Java Plugin Server runs on a separate port on the FTS indexing server (default is 9977).

 

As always, when an event that causes data to be written to the index is performed - a record is put into the ft_pending table. This event can originate from any server in the group.

 

Only the FTS indexing server(s) process index requests from the ft_pending table, and this table has been modified in SP5 so that each indexing server can build its own FTS Index. This is achieved by each index request populating multiple rows in the ft_pending table for a single index request, and the only difference in these rows is name of the indexing server. Therefore, do not be alarmed when you see far more rows in this table than before.

 

Each indexing server indexes simultaneously, but independently.

 

To ensure integrity of the system, the FTS Java Plugin design is such that any launched AR System server defaults to a read-only state until the primary FTS indexing server specifically initializes the primary Java Plugin Server for writing.

 

 

Steps to Configure High Availability for FTS in a Server Group

 

The steps are listed below on how to configure FTS High Availability correctly in a server group. Included between these steps is some important information to take note of.

 

1) You'll need a server group with at least 3 servers

First off, you will need a Server Group with at least 3 servers in it to avail of this feature:

  • 2 are required to be the indexing servers
  • 1 is required to be the search-only server

 

Failover occurs between the indexing servers and as noted before, you only get High Availability from the search-only servers.

High Availability for FTS cannot be utilized in a Server Group with 2 servers.

 

2) Upgrade to SP5 for AR Server on all servers in the group

This should be pretty straight forward, however there's a couple of caveats to look out for regarding FTS when upgrading to SP5:

 

a) Upon upgrade, the FTS Java Plugin Servers are disabled (the lines in the armonitor.cfg are commented out, as was the FTS Agent checkbox unchecked in the FTS Configuration form).

Therefore, the Java processes that runs the FTS Java Plugin Servers will not be running on AR Server startup and a AR Server restart is required if you want the FTS Java Plugin Servers to run (and be under the control of the armonitor process)

 

b) Also upon upgrade to SP5, FTS Searching and Indexing will be disabled. You can see this on the Server information form -> FTS tab, and by the 2 parameters in the ar.cfg:

 

    • Full-Text-Disable-Indexing: T
    • Full-Text-Disable-Searching: T

 

c) I've seen an instance whereby after an upgrade to SP5, in the Reader section of the FTS Configuration form, the Primary Server Name value was wrong. It was showing the Load Balancer name.

 

bad-server-name.jpg

 

This can occur if the Server-Plugin-Alias parameter for FTS has the incorrect server name:

 

Server-Plugin-Alias: ARSYS.ARF.FTS ARSYS.ARF.FTS bad-server-name:9977

 

In the same instance, the Load Balancer name was also being used in the server name option for the FTS Java Plugin Server lines in the armonitor.cfg:

 

"<JAVA_HOME>" -Xmx1024m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -classpath "<ARS_HOME>\pluginsvr\fts\primary;<ARS_HOME>\pluginsvr\fts\core;<ARS_HOME>\pluginsvr;<ARS_HOME>\pluginsvr\arpluginsvr7604_build002.jar" com.bmc.arsys.pluginsvr.ARPluginServerMain -x bad-server-name -i "<ARS_HOME>" -m

 

"<JAVA_HOME>" -Xmx1024m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -classpath "<ARS_HOME>\pluginsvr\fts\secondary;<ARS_HOME>\pluginsvr\fts\core;<ARS_HOME>\pluginsvr;<ARS_HOME>\pluginsvr\arpluginsvr7604_build002.jar" com.bmc.arsys.pluginsvr.ARPluginServerMain -x bad-server-name -i "<ARS_HOME>" -m

 

Defect SW00456444 has been raised to address point c).

 

Points a) and b) are as designed so that it no negative effects can occur to the environment during upgrade. This relates to the ‘How FTS Failover Works’ in the Footnotes section below.

 

All configuration steps should be performed before bringing FTS capabilities back online.

If you have already configured your Server Group environment for FTS Performance in 7.6.04 prior to applying SP5, refer to the ‘Upgrading to SP5 with FTS already configured for a Server Group’ section that is also in the Footnotes below.

 

3) Server Group Configuration and Operations

You'll need to ensure that you've configured the Server Group correctly (refer to page 33 in the BMC Remedy IT Service Management Suite 7.6.04 Installing and Configuring Server Groups White Paper for guidance)

 

Since the focus of this blog post is on FTS, we'll just look at the FTS Operation.

 

You should be seeing x amount of records for the FTS in the Server Group Operations Ranking form, where x is the number of servers in the group. (If you don't, raise a support issue for guidance on how to rectify this.)

 

In my test where I had 3 servers in the group, for the FTS Operation I ranked it as follows:

  • Server A = Rank 1 (Indexing server - contains writer & reader FTS Java Plugin Servers)
  • Server B = Rank 2 (Indexing server - contains writer & reader FTS Java Plugin Servers)
  • Server C = No Rank (Search server only - no FTS Java Plugin servers are started)

 

4) Disable FTS Indexing and Searching

Prior to configuring FTS in the Server Group, check that FTS Searching and Indexing is disabled on each server in the group (by default they are after an upgrade to SP5).

 

To do this, go to:

AR System Administration Console -> General -> Server Information form -> FTS tab

 

5) Configure FTS

To configure FTS on each server in the group, use the AR System Administration FTS Configuration form, shown below:

 

FTS Configuration form.jpg

 

To access this form:

AR System Administration Console -> General -> FTS Configuration.

 

System FTS Configuration.jpg

 

When testing, this is the order and how I configured it:

(Recalling the ranks I set in point 3) above)

 

Server C
FTS AgentDO NOT TOUCH
Server Configuration

Server Group-Secondary

WriterFTS Port1<Blanked Out>
Max JVM Memory1<Blanked Out>
ReaderPrimary Server NameServer A
FTS Port29977
Max JVM Memory21024
FTS Collection DirectoryC:\Program Files\BMC Software\ARSystem\ftsconfiguration\collection
FTS Configuration DirectoryC:\Program Files\BMC Software\ARSystem\ftsconfiguration\conf
Save the record

 

 

Server B

 

FTS Agent

Check

Server ConfigurationServer Group-Primary
WriterFTS Port19998
Max JVM Memory11024
ReaderPrimary Server NameServer B
FTS Port29977
Max JVM Memory21024
FTS Collection DirectoryC:\Program Files\BMC Software\ARSystem\ftsconfiguration\collection
FTS Configuration DirectoryC:\Program Files\BMC Software\ARSystem\ftsconfiguration\conf
Save the record

 

 

Server A
FTS AgentCheck
Server ConfigurationServer Group-Primary
WriterFTS Port19998
Max JVM Memory11024
ReaderPrimary Server NameServer A
FTS Port29977
Max JVM Memory21024
FTS Collection DirectoryC:\Program Files\BMC Software\ARSystem\ftsconfiguration\collection
FTS Configuration DirectoryC:\Program Files\BMC Software\ARSystem\ftsconfiguration\conf
Save the record

 

 

NOTE: It is important that for the indexing servers, the following are adhered to reduce any potential failover and directory searching issues:

  • The same path must be used for the collection & configuration directories on each indexing server
  • The Secondary FTS Plugin Server’s port should be the same for each indexing server

 

For example, if the collection directory on indexing server 1 is:

'C:\Program Files\BMC Software\ARSystem\ftsconfiguration\collection'

 

Then indexing server 2 should use:

'C:\Program Files\BMC Software\ARSystem\ftsconfiguration\collection'

 

The same applies for the path of the configuration directory.

Additionally, if indexing server 1’s FTS Port2 is 9977, then the same port number should be used for indexing server 2’s FTS Port2.

 

6) Perform a Server Group restart (clear out the current collection directories on the indexing servers & truncate the ft_pending table if desired)

The correct way to perform a server group restart is:

i. Stop the non-admin server(s) (if >1, perform one at a time - wait until each stops)

ii. Stop the admin server (wait until it stops)

iii. Start the admin server (wait until it starts)

iv. Start the non-admin server(s) (if >1, perform one at a time - wait until each starts)

 

In my test, Server A is the Admin server also. So I performed:

i. Stop Server C

ii. Stop Server B

iii. Stop Server A

iv. (Optional - Clear out the current collection directories on the indexing servers)

v. (Optional - Truncate the ft_pending table in the ARSystem database)

vi. Start Server A

vii. Start Server B

viii. Start Server C

 

Steps iv. and v. above are optional. If there any problems with them, and you'd like to start afresh - then this is the ideal time to perform these steps. If FTS was working fine prior to installing SP5, then there's no need to perform them.

 

7) Enable FTS Indexing on the Indexing Servers

Go to:

AR System Administration Console -> General -> Server Information form -> FTS tab

  • Disable Full Text Searching = Check
  • Disable FTS Indexer = Uncheck

 

8) Perform a Re-index on each indexing server

Go to:

AR System Administration Console -> General -> Server Information form -> FTS tab

  • Re-index = Check

---------------

To check the progress of the re-index you can:

• Enable the Full Text Indexer Log on each indexing server prior to performing the re-index

• Monitor the activity on the ft_pending table in the ARSystem database

---------------

Hit OK

 

9) Enable FTS Searching on all Servers

AR System Administration Console -> General -> Server Information form -> FTS tab

  • Disable Full Text Searching = Uncheck

 

Once you’ve completed all of these steps, the environment is now configured and ready for FTS High Availability.

If you have any problems performing any of these steps, raise a support issue and reference this blog post.

 

 

Foot Notes

 

Re-indexing

As per step 8 in the ‘Steps to Configure High Availability for FTS in a Server Group’ above, a re-index needs to be performed on each indexing server.

 

For the initial re-index, these can be performed at the same time on each indexing server, but any subsequent re-indexes should be performed separately so that the search-only servers can continue to perform FTS queries during the re-indexing process.

 

For join, vendor and view forms there’s a timed re-scan option, which performs a re-index of the form based upon its time settings (in the form properties) to pick up any changes made to the base form. These re-scans are independent and performed on a per server basis. This is because the indexing servers are working asynchronously and each server has to re-scan based upon the last time it scanned. In this case, there could be a slight 1-2 minute difference in data between indexing servers.

 

For this reason, we recommend that all Search-only servers point to a single reader FTS Java Plugin Server. All search-only servers should connect into the reader (secondary) FTS Java Plugin Server for the indexing server that is ranked 1 for FTS. From a performance perspective the reader (secondary) FTS Java Plugin Server can handle a high load of read requests and load balancing users between readers in not required.

 

Upgrading to SP5 with FTS already configured for a Server Group

If you already had FTS configured for Performance in a Server Group for 7.6.04, where there was only 1 indexing server - and are planning to/have upgraded to SP5 for each AR Server in the group – after applying SP5 and configuring the subsequent indexing servers, all you need to do is to perform a re-index on the new indexing servers. After which they will be on a par with the original indexing server and there is no need to re-index the original indexing server (assuming that you haven’t cleared out the collection directory or truncated the ft_pending table).

 

And since these new indexing servers are not ranked 1, the re-index is usually not noticeable to end-users. It may however take some time to complete depending on the amount of data to index.

 

If during the time when the new indexing servers are being re-indexed, a failover occurs – the new indexing servers may refuse the search requests from the search-only servers as it doesn’t have the data available to fulfill the request.

 

Deciding on the Ranking value for the Indexing servers

There is no real recommendation on the ranking value to set for each of the Indexing servers.

 

By giving a server a rank for the FTS Operation in the Server Group Operations Ranking form, this just identifies that the server is an indexing server.

 

The only recommendation would be that if you already had FTS configured for Performance in a Server Group and then upgraded to SP5, then choose the original indexing server as the rank 1 server. You should also configure the search-only servers to point to this server as the index is already in place and the new indexing servers need to perform a re-index.

 

It is also best practice that the indexing servers not be user facing servers.

 

How FTS Failover Works

For FTS, the operation entries in the ranking form are not used by AR Server in the traditional sense anymore.

 

An FTS operation record (that is ranked) indicates that the server is an indexing server, and this record is used by the search requests from the search-only servers to invoke failover. Invoking failover occurs at the search request level on a search-only server. Furthermore, each search-only server acts independently with regards to failing over the FTS Operation.

 

For example, referring to Figure 1.2 earlier, if a user is connected to AR Server 4 and issues a search request and there’s a problem with FTS on AR Server 1, then this search request would instigate an FTS failover to AR Server 2 to fulfill the request. All subsequent search requests from users on AR Server 4 will be serviced by AR Server 2 until set time threshold it hit, and then AR Server 1 is polled check if it can resume servicing FTS search requests.

 

Now, let’s say that 1 minute after the initial problem above is hit, and a user connected to AR Server 3 issues a search request – this will still go to AR Server 1 (as the search-only servers act independently). By this time, the problem on AR Server 1 no longer exists, and the search request is fulfilled by AR Server 1.

 

So in this example, there is a time when search requests from users on AR Server 4 are being serviced by AR Server 2, and search requests from users on AR Server 3 are being serviced by AR Server 1 - as the search-only servers act independently.

 

Upgrading from 7.6.04 SP5 in the future

After applying SP5 to 7.6.04, you cannot upgrade to 8.1 SP0 (base release) because of the changes introduced in SP5.

You would need to upgrade to 8.1 SP1 at least.