FULL TEXT SEARCH (FTS) - Deployment and Configuration Best Practices  (Whitepaper)

Version 8
    Share:|

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    Remedy AR System Server


    COMPONENT:

    AR System


    APPLIES TO:

    Remedy AR System 9.x, SmartIT, ITSM, RKM



    QUESTION:

    What are the Best Practice Configurations for FTS?


    ANSWER:

    Full Text Search (FTS) – Deployment and Configuration Best Practices

    This document is a whitepaper on FTS Deployment and Configuration Best Practices.

      For the ITSM Best Practices document, please see    KA000114508 
    For the FTS Best Practices Only (not the whitepaper) see    KA000153129 
    For specific information on setting up Load Balanced FTS Searchers, see    KA000157036
      
    Introduction
    The Full Text Search (FTS) is a feature of the ARServer that enables faster searches when the searches are performed on FTS-enabled fields of a given AR Form.
    In contrast to the standard character-based search function available in a database, FTS is a word-based search thus providing faster results when searching for words or phrases.

    FTS offers several benefits over database such as fast, indexed searching of CLOB fields, searching within attached files, word-stemming, weighting search results, and more.
    FTS is also an integral part of ITSM and Smart IT use cases. 
    Thus, an optimally configured and properly sized FTS environment will result in an optimal ITSM and Smart IT performance.

    This whitepaper shows how to optimally configure your FTS environment including FTS load distribution.




    How-To Enable FTS
    To include a field of any AR Form in Full-Text Searches, enable the field for FTS using Developer Studio.

              User-added image

    When a field is enabled for FTS, any data added, removed, or updated to the field are automatically indexed by the Lucene engine, an open source embedded FTS product.




    Overview of how an FTS server works  
      1- FTS code within ARServer provides indexing capability as well as searches for the local AR System Server. 
      2- The FTS searcher plugin- this FTS searcher is implemented as an AR plug-in running in its own plug-in server providing searches for remote ARServers.
        This FTS searcher runs on the same local machine as the parent ARServer in item 1.
     
      3- A set of FTS Index files (contained in its own set of directories) — a flat file database of indices.
      
    Another technical perspective:
    The FTS service contains an indexing service (item 1) that writes and updates a set of index files (item 3); and a search service that reads the index files and uses the data to perform fast searches.
    The ARServer together with the FTS searcher running in its own plug-in server is called the FTS Server (in the context of an ARServer group).

    Events such as create, update, and delete within the AR System (e.g. ARServer group) are queued via the ft_pending table in the database.
    The FTS Server spawns a set of indexer threads taking items from this queue to add, delete, or update the data within its index files accordingly.
    Only the ARServers designated as FTS Servers service the FTS queue. However, any ARServer within an ARServer group adds to this queue when servicing an AR API call that alters data on a field marked for FTS.


    The sequence of data flow is as follows:
    - When a non-FTS Server in an ARServer group is servicing an AR API call needing to perform an FTS search, it sends a search request to the FTS Searcher plug-in.
    - The FTS Searcher plug-in performs the search and returns the result to the caller ARServer.
    - The ARServer caller packages the search results into the AR API response of the original AR API call.

    NOTE: An FTS Server performs its own FTS searches and does not need to forward any FTS search requests.




    Configuring FTS Service in an ARServer Group
    When configuring the FTS service for an ARServer group, do the following:
    -- Identify and Rank (in the AR System Server Group Operation Ranking form) any servers in the Server Group as FTS Servers (more than 1 can be FTS Servers for HA and workload distribution purposes) --

    > Each FTS Servers will have its own local copy of the FTS index files.
    > Each FTS Server provides FTS service for itself and other members in the ARServer group.

    > The primary FTS server (Ranked #1) serves as the default Searcher server.
    > When the primary FTS Server is down or unavailable, another will be chosen as the primary FTS Server based on the configuration in the ranking form.


    You can designate any ARServer in an ARServer group as an FTS Server by ranking it in the AR System Server Group Operation Ranking form.
    Each configured FTS server always services the FTS queue regardless of its ranking order.


    IMPORTANT: A running FTS service can consume a large amount of resources (CPU, Memory, Disk Space); thus, FTS Servers are not recommended to be User-facing servers.

    For more information, see Setting failover rankings for servers and operations.




    How failover occurs in high-availability configuration
    FTS requires that at least one of the servers in a server group act as a primary FTS indexing server.
    To provide failover, you can have two or more FTS indexing servers in the server group where each server acts as an independent FTS Indexing server and maintains a local folder of the FTS Index in its Collection directory.
    When an indexing action takes place, each FTS indexing server indexes it independently.

    For example, if you create a form and then index a field on that form, each FTS indexing server indexes that field. Because you can have multiple FTS servers, each with its own current copy of the FTS data, you can fail over to a second server as needed.


    An AR System server is designated as an FTS indexing server by having a ranking record for FTS in the AR System Server Group Operation Ranking form.
    Each FTS server defined in the ranking form acts as an indexing server and can provide FTS search services to other servers in the server group.
    Each defined FTS server will always index, regardless of its ranking order.


    FTS servers can consume large amounts of resources (CPU, Memory, Disk Space) and are recommended to NOT be located on User-facing servers.
    Servers in the server group that are not ranked for FTS are Searcher servers.
    The Searcher servers are user facing or non-FTS backend servers that connect to one of the FTS indexing servers when they need to search the FTS data.


    Note: The servers that are not designated as indexing server should not be listed in the AR System Server Group Operation Ranking form.

    The following figure shows the FTS Servers in a high-availability architecture:


              BMC Remedy FTS Servers in a high-availability architecture

        
          User-added image


    The default configuration for all non-FTS servers in an ARServer group is to connect to the Primary FTS server (ranked 1 in the AR System Server Group Operation Ranking form).
    This is specified in the 
    Server-Plugin-Alias entry in the Centralized Configuration and ar.cfg [or ar.conf] files of the AR Servers.

    For example, consider a scenario where you have two FTS Servers that are ranked 1 and 2 in the AR System Server Group Operation Ranking form.
    Then your 
    Server-Plugin-Alias entry has the value of the server name that is ranked 1.
    - If a request to this primary FTS Server errs, the request is then sent to the next FTS Server that is
    ranked 2 in the AR System Server Group Operation Ranking form.
    - If the request sent to the FTS Server
    ranked 2 also errs, then an error is returned since no FTS Server is available.




    Architecture for FTS setup
    For some FTS deployment, it’s sufficient for all searches and indexing from the user facing servers be handled by one primary FTS Server with Rank 1. Therefore, the rank 2 FTS Server functions as HA and no searches are sent to this server unless the primary FTS Server is down or unavailable. This is the default behavior for FTS Servers in an ARServer Group.

    To overcome moderate heavy loads on a single FTS Server, there are two possible configurations:


    Configuring multiple FTS Searchers without a Load Balancer :
    Each user-facing ARServer is configured to direct searches to a specific FTS Server.
    An FTS Server can provide the service to one or more User Facing servers as shown in the diagram below.
    In the case that the manually configured FTS Server is down or unavailable to accept a search request, the ARServer redirects the search to another AR Server starting with Rank 1,2,3 until one of the FTS Servers is available to handle the request.
              User-added image
      
      Pros :
      
      In this setup the searcher load is manually distributed and binds with the same FTS Server. Failover still occurs normally as part of the server group configuration, based on the rank of the server.
      
        
      
      Cons :
      
      a) Load Balancing is manual and not based on any load balancing rules such as Least Used or Round Robin. Thus, there may be a load imbalance
      
      b) When a metadata operation such as Field Re-indexing or Global Re-indexing takes place, the search query will not be handled by that particular FTS Server and will fall back to a different server depending on Ranking.
      
      c) If any Indexer server is down for a significant amount of time, searches will not be in sync until all servers have caught up with the indexing request. (see Note:  What to do if your indexes become out of sync). 
     
      
       Configuring multiple FTS Searchers with a Load Balancer
    A load balancer is configured between the User Facing AR Servers and the FTS Servers. In this case if any searcher becomes unavailable the load balancer will redirect the search query to another Searcher Plugin. If the FTS Server is busy performing a re-index and cannot accept the search request, the load balancer may still direct searches to it. In this case, the ARServer automatically redirects the request to FTS Servers starting with Rank 1,2,3 until one of the FTS Server is available to handle the request.
              User-added image
      
      Pros :
      
      In this case the FTS search load gets distributed via load balancer and the FTS Servers share the load.
      
        
      
      Cons :
      
      a) When a metadata operation such as Field Re-indexing or Global Re-indexing takes place, the search query will not be handled by that particular FTS Server and will fall back to a different server depending on Ranking.
      
      b) If any Indexer server is down for a significant amount of time, searches will not be in sync until all servers have caught up with the indexing request (see Note:  What to do if your indexes become out of sync)
      
    Note:  What to do if your indexes become out of sync.
    If you have an Indexer server go down for some period of time, such as for performing routine maintenance, you may have a situation in which the ft_pending jobs for a single server are piling up. 
    When you bring that server back up, it will take time to catch up with the queue. 
    But because that server has become available, searches will still be performed, giving inconsistent results to end users.
    It is best to prevent users from searching on this indexer until it is caught up.
    As mentioned earlier, you can run the following SQL to  check how out of sync your indexers have become:

    select count(*), indexservername from ft_pending group by indexservername;

    Once you bring the server back up, immediately stop the fts searcher plugin server.
    This can be done easily in v.18.05 and v.18.08 and v.9.1.04 where the
    Server Group Administration Console has been installed.
    For 9.1.04, this is an option available from 
    https://communities.bmc.com/docs/DOC-102081

    From the Server Group Administration Console:
    1- Open the
    "Manage Processes" page.
    2- Sort by 
    Process Name and look for the BMC:FtsJavaPluginServer entry.
    3- Identify the exact line for the server that is out of Sync.
    4- Highlight that line and choose an
    Operation of "Stop"
    5- Click on the Execute button. 
        This will stop the plugin server and searches will not be performed on this Indexer server.
    6- Periodically run the SQL until your indexers are in-sync.
    7- Then highlight the stopped 
    BMC:FtsJavaPluginServer and choose Operation 'Start' and execute.

    Searches will now be directed to this indexer server.




    Load Test Results
    In BMC’s performance lab we have configured a SmartIT cluster Environment to test FTS.
    We have:
    > 6 user facing servers
    > 2 FTS indexers with Rank 1 and 2 as shown in below diagram.
    > Load Tests were conducted with a Load Balancer Configuration in place.

    All these tests were carried out for search scenarios.
              User-added image

    Test #1 – A test was performed in which we had search requests distributed across two admin indexers using LB.
    Following are the details of the tests carried out and the conclusion.
      
      Tests details
    - Number of users: 1,000.
    - Test duration: 30 minutes.
    - FTS threads: 10.
    - Searcher Threads (numCoreThreads): 10.
    - Threshold: 2K.
    - Indexer Node’s CPUs: 4.

    Results:
    The summary graph below is plotted to show the effect of Search threshold on Search Response times and transactions per second (TPS):

               User-added image
     
      
      
                Conclusions :

                1- The Response time increases marginally as the threshold increases.
                2- The Transactions Per Second is negatively impacted as the threshold increases.
                    Search’s best performance is observed with 2k Threshold.
                    With the increase of threshold, the total number of search transactions completed per second reduces along with an increasing trend in response times.

                3- There is significant difference in throughput viz. approximately additional 32k transactions per hour for 2k threshold compared to 10k threshold, and approximately additional 25k transactions per hour compared to 4k threshold.
     

    Test #2 - A comparison study on searches for Response time(s) and transactions per second was done with 1 indexer Server and 2 Indexer Servers.
      
     
      Results:
    Following is a graph of results comparison for the tests with 1 indexer (with HA) and 2 indexers (with load distributed using LB):

               User-added image

    Note: For comparison, the Search Threshold of 2k, 4 CPU and 10 searcher threads are constant.

    Conclusion:
      
      Response time and throughput has shown significant improvement when the search request load is distributed across 2 FTS indexers instead of 1 FTS indexer.
         


    Tests for Indexing
    Tests details:
                - FTS threads: 3.
                - Indexer Node’s CPUs: 4.      
     
    Test # 1: Time taken for an Incident to get indexed:
      
      The time varies for an Incident from ~50sec to 3mins. The variation is due to the polling of the FTS thread which is default set to 2mins.
    Note: The server making the data changes has a built-in delay of 10 seconds before alerting FTS to Index the data. 
    And the FTS Indexer has a built-in delay of 5 seconds before processing the indexing request.
     
      
       Test # 2: Time taken for the records to be handled from ft_pending.  
      We have created ~50K records in ft_pending while the Indexing server is down. After creation of ~50K Incidents, Indexer server is started.
    Throughput achieved for Indexing is 135 Incidents/sec.
        
    Test # 3: Measured Time taken for Global Re-index.
      
      Time taken for Global Re-indexing with maximum of 0.5M records in schema is 4hrs 15mins.
    The records vary in each schema, but we had a maximum of 0.5M in one of the schema related to Incidents.
    Average CPU Utilization on FTS Indexer during re-indexing: 48%.
         


    FTS plug-in configuration
    Documentation:
    v.9.1.x: https://docs.bmc.com/docs/ars91/en/fts-plug-in-configuration-609072952.html
    v.18.05 (9.1.05): https://docs.bmc.com/docs/ars1805/fts-plug-in-configuration-804712974.html
    v.18.08 (9.1.06): https://docs.bmc.com/docs/ars1808/fts-plug-in-configuration-820496274.html


    The FTS plug-in is used to search Full Text Search indexes.
    The FTS plug-in provides row-level and field-level security for indexed data during searches.

    This plug-in supports the multiform search used by Global Search and in applications such as BMC Remedy Knowledge Management (RKM).
    The multiform search uses a vendor form that displays an interface through which an application customizes the search to suit its requirements.
    The search can contain a request to return a specific list of fields, how the filters are returned, or terms that may or may not be available in the document, and so on. It also permits you to specify the forms to search.
    You can tailor the search to target just the indexed data rather than all data.


    Plug-in type: 
    FTS is a Java-based Filter API plug-in.

    AR System server connectivity: 
    The FTS plug-in interacts with the AR System server through the BMC Remedy AR System Java API.
    The plug-in (
    ftspluginVerNum.jarVerNum represents the release version number) is installed in the <ARInstallationFolder>\pluginsvr\fts folder.

    Configuration information
    The FTS plug-in is installed during the AR System installation and is hosted with the standard AR Java plug-in server.
    Each instance of the AR Java plug-in server is permitted to run only one instance of the FTS plug-in.
    By default, the FTS plug-in runs on port 9977. If you want to change the plug-in port value, the default port number value must be changed in both
    ar.cfg [or ar.conf] and pluginsvr_config.xml files.
    You can also use
    Centralized Configurations generic UI.
    In a single-server installation, ensure that the collection directory is on the same computer as the server. In a server group, ensure that all the FTS plug-in instances always point to the local collection directory. Configuring full text search

    Link: Configuring full text search for a server group
    The configuration information of the FTS plug-in is available in the <ARInstallationFolder>\pluginsvr\pluginsvr_config.xml file that includes the plug-in details in XML code.
    The ar.cfg/ar.conf file includes the Server-Plugin-Alias setting that points to the correct plug-in server alias:
    Server-Plugin-Alias: ARSYS.ARF.FTS ARSYS.ARF.FTS myServer:9977
    The FTS plug-in is launched from the armonitor.cfg [or armonitor.conf] file located in the <ARInstallationFolder>\conf folder.
     
    More Info:
    https://docs.bmc.com/docs/display/public/ars91/Configuring+full+text+search

    Configuration Consoles
    Most FTS Configuration can be performed from the FTS Configuration page on each individual server.


              User-added image

    This will automatically set the appropriate configurations in the ar.cfg/conf, armonitor.cfg/conf, and pluginsvr_config.xml
    In 9.1.04 and later versions, you have the option of configuration all the servers in your group from a single Server Group Administration Console

             User-added image


    Best Practice Configuration
    The following section details the FTS Best Practice Settings based on environment size.
    The configuration recommendations are specific to small, medium and large model defined as follows:

                                                                                                                               
    EnvironmentBMC Remedy ITSM (concurrent users)BMC Remedy Smart Reporting (concurrent users) Number of managed devices
    Small800502,000 or fewer
    Medium2,00010010,000 or fewer
    Large5,00025030,000 or fewer
      
    Configuration settings
                                                                                                                                                                                                                   
    ComponentSmallMediumLarge
    Number of Users8002,0005,000
    Physical RAM24 GB32 GB64 GB*
    ARServer Heap Size10 GB14 GB32 GB*
    Plugin Server Heap Size14 GB6 GB10 GB
    Number of Core Threads2101216
    FTS Queue (FTS Indexer servers)3Private-RPC-Socket: 390602 3 3Private-RPC-Socket: 390602 5 5Private-RPC-Socket: 390602 7 7
       *Recommend HA setup. 

    Note:  
      1 Plugin Server Heap Size is configured via armonitor.cfg/conf.  Set the ‘-Xmx’ parameter according to the above recommendations
    The Number of Core Threads can be set using the Plugin Server Configuration Console within the AR System Administration Console.
     
      
      
      User-added image 
      
      
      The Plugin server must be restarted for this setting to take effect.It will be restarted automatically when you restart ARServer 
      
      
      The FTS Queue is set from the ‘Ports and Queues’ tab of the AR System Administration Console:

    User-added image
      

    GC settings
    BMC recommends using these GC parameters (same as ARServer) with Plugin Server for FTS.
    .................................................
    -XX:+UseParNewGC
    -XX:+UseConcMarkSweepGC
    -XX:+DisableExplicitGC
    -XX:NewRatio=2
    -XX:+UseCompressedOops
    .................................................

     
    For Example using the Small Model (armonitor.cfg) on FTS Indexer Server:
    ........................................
    "C:\Program Files\Java\jdk1.8.0_73\jre\bin\java" -Xmx4096m
    -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC -XX:NewRatio=2 -XX:+UseCompressedOops
    -Dcom.sun.management.jmxremote.port=8004 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
    -classpath "C:\Program Files\BMC Software\ARSystem\pluginsvr\fts\secondary;C:\Program Files\BMC Software\ARSystem\pluginsvr\fts\core;C:\Program Files\BMC Software\ARSystem\pluginsvr;
    C:\Program Files\BMC Software\ARSystem\pluginsvr\arpluginsvr91_build002.jar" com.bmc.arsys.pluginsvr.ARPluginServerMain -x { FTS Indexer AR_Server_Name} -i "C:\Program Files\BMC Software\ARSystem" -m
    ........................................





    Troubleshooting
    https://docs.bmc.com/docs/ars1805/troubleshooting-fts-plug-in-issues-805339782.html




    FTS Monitoring
    1- Check AR Server Ranking form (AR+System+Server+Group+Operation+Ranking).
        Ensure that the Full Text Index is set to Rank=1 for only the Primary Index server.
        Servers which are not serving Full Text Index should have Rank as Empty/Null or the Ranking entry removed.


    2- In AR Server console under Server Information's ->FTS tab "Disable Full text searching"  and “Disable FTS  Indexer” checkbox should not be checked.

              User-added image




    3- Under FTS Configuration check that FTS Agent checkbox is enabled.
              User-added image



    4- Under Server Configuration -> “Server Group Indexer” radio button is selected for “Indexer Server” and “Server Group-Searcher” is selected for a Searcher server.

                  User-added image


    5- Monitor ft_pending table.
        
    Ensure that the entries are getting created in the ft_pending table for any FTS Operations.
         select count(*) from ft_pending;

         If this shows that the count is increasing and decreasing then it confirms that your indexing operations are being processed.
        - Ensure that the ServerName (user facing server) and IndexServerName (FTS Indexing Server ) are appropriate to the Environment and there are no other extraneous entries in the ft_pending table.
        - Any entries in the IndexServerName field other than the Indexing Server name will not be handled by ft_pending.
          Those entries can be deleted.

        - If the ft_pending table is properly handled by FTS Indexer, the updateTime will be updated with -1 if its data operation and it will be a numeric count if it’s a meta-data operation like global reindex on the schema.
          See Knowledge article 000152517 for more information.

        - You can use the Server Group Administration Console to Monitor indexing activity.


    6- Monitoring using JVisualVM: FTS Thread Monitoring.
        a) Under threads monitoring in JVisualVM, check the pool-FTS-threads count.
            These threads are used to handle data operations from FTS like CREATE_ENTRY/SET_ENTRY /   INDEX_ENTRY ...etc.


        b) There will be n+1 threads where n is the number of threads set in ar.conf for Private-RPC-Socket: 390602.
           
    If Private-RPC-Socket: 390602 is set to 3 3, you will observe 4 threads under pool-FTS-threads.
     
             User-added image

     
        c) There are also another 4 threads for FTS (pool-FTS-SERIAL-threads).
            These threads are visible or started only when metadata operations like GLOBAL_REINDEX, SCHEMA_REINDEX , FIELD_REINDEX ..etc are triggered.


        d) There will be n+1 threads where n is the number of threads set in ar.conf for Private-RPC-Socket: 390602.
           
    If Private-RPC-Socket: 390602 is set to 3 3, you will observe 4 threads under pool-FTS-SERIAL-threads.
             
              User-added image



    7- Check arftindx.log for information regarding index operation handling.  
       

     


    Article Number:

    000159890


    Article Type:

    FAQ/Procedural



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles