Share This:

Some of the common questions and answers regarding Remedy9 Full Text Search!


 

Answer

1. What's New in Remedy9 As far as Full Text Search feature is concerned? 

• Implemented FTS HA Model (Similar v7604 SP5 & v8.1 SP2)

• Apache Lucene version upgrade from 2.9 to 4.9

• Apache Tika version upgrade 1.2 to 1.6

• We broke down Monolithic FTS index file into schema specific files (a.k.a. Schema based index) using Lucene 4.9

• Developed FTS Index Migration Utility, Which will migrate Old Monolithic Indexes into Schema based Indexes & convert into Lucene 4.9 format too

•Incorporate FTS Index Migration Utility into AR Installer

2. What is Schema Specific Files (a.k.a. schema Based index) ?


  • Till Remedy9 Release, All FTS indexes gets stored into One Lucene specific File (Monolithic index file). 

      E.g. All incidents, Problems, Change, KA related indexes gets stored into this one single Index file. 

 

  • Now From 9.0 release we have broke down that file and we create separate folders inside <AR/FTSconfiguration/collection> folder for each AR Form/Schema stated above.


  • Example:

      If HPD:HelpDesk Schema has schemaId = 558 then All incidents related Indexes will get stored into one folder e.g. 558/

      <ARINSTALLDIR>/ftsconfiguration/collection/558


  • All Change Management related data will get stored into one folder e.g. 561

        similarly for rest of the AR forms, data will get stored.


Earlier Monolithic Indexes:

Monolithic.png


Schema Based Index:


SchemaBasedIndex.png



3. What's a need of FTS Index Migration Utility?

 

  • Till Remedy9, All FTS indexes gets stored into Monolithic Files, and Apache Lucene (The underline Search engine) was at 2.9.
  • In 9.0 We have upgraded Lucene Version to 4.9 and broke down the Monolithic indexes into Schema Based index.
  • The need arises to cater above two requirements
    • Convert Monolithic Indexes into Schema Based Indexes
    • Upgrade older indexes to newer format (i.e. from Lucene 2.9 to 4.9)

4. Where does FTS index Migration Utility gets deployed?

 

  • Windows:
    • <AR INSTALL DIR>\arftsutil.bat

 

  • Linux:
    • <AR INSTALL DIR>/arftsutil.sh

5. Is FTS index Migration Utility will kicked off automatically While Upgrading AR Server to 9.0?

 

  • Yes, As you start AR Server 9.0 upgrade, Index migration is part of AR Upgrade process.

6. Where does FTS Index Migration utility logs gets stored?

 

  • It gets stored under <AR INSTALL DIR>\Arserver\db

7. Can I skip the FTS Index Migration utility Execution?

 

  • If you are upgrading AR Server to 9.0 Using Installer (running UI) Then User do not have any option to disable the FTS index Migration Utility execution.
  • Out of the box, Installer will migrate old Indexes.

8. Is there any Way to Skip The FTS Index Migration Utility Execution?

 

  • If You upgrade AR Server 9.0 Using Slient Mode then one can skip the FTS index Migration Utility execution.
  • During AR Server 9.0 Upgrade if <AR Server INSTALL DIR>\ftsconfiguration\collection folder does not have any files then in this case, AR Server FTS Index migration utility will not migrate anything.

9. What is the Silent install parameter in order to SKIP FTS Index Migration process during AR Upgrade via Silent installation mode?


  • FTS index migration can be skipped if Installation done via AR Silent installation.

 

  • Using silent Install Parameter one can skip Index Migration:

                -J BMC_AR_SKIP_FTS_INDEX_MIGRATION= true

                  (Value to this parameter is case insensitive)

  • If this Parameter is present in AR Silent install File and it is set to True/TRUE/true then ONLY Index migration will skipped.
  • In any other condition, By default index migration will execute and will migrate old indexes.

10. If utility fails will it cause AR Server Upgrade failure ?

 

  • If FTS Utility fails it will cause AR Upgrade Failure.
  • Utility returns 3 return codes:
    • 0: Success
    • 1: Warning[When DB Only Upgrade OR when just adding new locale on existing AR Installation, it will return 1 but upgrade will not fail]
    • 2: Fail [AR upgrade will also fail]
  • FTS index Migration Utility Logs will be get stored under <AR INSTALL DIR>\Arserver\db

11. Shared load among  indexer servers. How does one configure that?

 

  • We have FTS Configuration plugin in place. Using that, one can dedicate the Indexer Server and Searcher.
  • E.g. If I have 4 Servers in Server Group, and I have nominated 2 as Indexer servers, and 2 as searcher.

        From FTS Config Plugin UI, One can configure Server A & Server B  as Indexer,  and Server C & Server D as Searcher.

  • To nominate as Indexer server, in the Server Group Ranking Form, that server should have rank for FTS Operation, that way AR Server will assume it is FTS Indexer Server.

12. FTS Index Migration Utility execution Failures: Possible Reasons?


  • Old indexes are corrupted
  • Not Enough space on the disk where ftsconfiguration/collection directory exists (during Index migration, it would required double the space on the disk, later while merging it will delete unwanted files present in the /collection folder.


        Example: Before upgrade if /collection dir is of 2 GB then, then while AR upgrade there must be 4 GB of free space                                        available in the /collection folder, at the end of the migration process, /collection folder will become ~1.2 GB


13. Can I execute FTS index Migration Utility Manually later some time?

  • Yes, user can but if indexes already migrated then no need to execute manually again.


14. Which Version of Luke can I use in order to debug Lucene 4.9 Index?


  • you can use Luke 4.1x onwards, can download from here


15. AR FTS Index Migration Utility - Usage?


  • <ARInstallDir>\arftsutil.bat  –d “<COLLECTION DIR PATH>”  -c “<FTS CONFIGURATION DIR PATH>”
  • arftsutil.bat -help (it will print usage)

16. How much time required to migrate Old indexes into newer format using FTS Index Migration Utility?

 

  • In R&D Performance Lab we conducted tests and here are the results (Note: timings may vary depending on the Server configurations)
  • Test I
    • Collection Folder  Size Before Conversion: 1.5 GB
    • Collection Folder Size After Conversion  :  0.8 GB
    • Total Time for Index Conversion Process : 13 Mins
    • Test Data during this test: ~100K records (Incidents, Change, Articles, RKM external docs etc)
  • Test II
    • Collection Folder  Size Before Conversion:  24.5 GB
    • Collection Folder Size After Conversion  :  15 GB
    • Total Time for Index Conversion Process : 130 Mins
    • Test Data during this test: INC ~522K, CHG 103K, SR 535K, WO 12K, PB 100K, RKM Docs 100K, AR Form w/       

                                                          Attachments 200K, AR Form w/o Attachments 300K

17. Global Re-indexing is required after I move my entire stack to Remedy 9?

 

      No, Not required.

18. How can I Re-index particular schema if, indexes got corrupted and wanted to create indexes for that schema?

 

  • One can use Process command: Application-FTS-Reindex-Form "<FORM NAME>" using driver program OR can write Filter and use this proc command to re-index particular schema.