12 Replies Latest reply on Aug 6, 2013 8:54 AM by Jonathan Drury

    FTS Re-Index Performance 7.6.04 (Lucene)

      Hi Everybody,

      we encounter long FTS re-index duration (more than 30 days) in an up-to-date environment (ITSM 7.6.04, windows).

      Does anyone has experienced better performance-values?


      -> indexing of 5-6 incidents per minute


      There are more than 280.000 incidents (HPD:HelpDesk) and more than 150.000 Tasks (TMS:Task) in the system, which leads to more than a month of re-indexing-time?


      What can be done to improve performance (a standalone plugin-server was already discussed)?



      Any hint would be helpful.


      Best regards,


        • 1. FTS Re-Index Performance 7.6.04 (Lucene)
          Robert Poulos

          A little more information about your environment would be helpful.  Specifically:


          What service pack of 7.6.04 are you on?

          What database type?

          Is the index collection directory local to the server doing the indexing?

          Is the server in a server group?

          How many FT Indexer threads are configured?


          Thank you -

          • 2. FTS Re-Index Performance 7.6.04 (Lucene)
            Laurent Matheo

            If it's genuine 7.6.04, try using the FTS plugins from 7.6.04 sp2 or sp3.

            • 3. Re: FTS Re-Index Performance 7.6.04 (Lucene)

              It's a ARS-7.6.04SP3 (ITSM-7.6.04SP2) on Windows2008 with MS-SQL as DB.

              The index-Collection directory is on a network-share due to the servergroup-configuration

              The "FT-Indexer threads" might give us a clue... Is it one of the Private-RPC-Sockets? If Yes, which one is reserved for FTS?

              Private-RPC-Socket:  390601   1   1

              Private-RPC-Socket:  390603   1   1

              Private-RPC-Socket:  390620   8  32

              Private-RPC-Socket:  390621   5  16

              Private-RPC-Socket:  390626   8  16

              Private-RPC-Socket:  390635   8  32

              Private-RPC-Socket:  390680   2   2

              Private-RPC-Socket:  390681   8  16


              Best Regards,


              • 4. FTS Re-Index Performance 7.6.04 (Lucene)
                Laurent Matheo

                Try setting more escalation threads:

                Private-RPC-Socket:  390603   1   1



                Private-RPC-Socket:  390603   1   3


                And restart ARS.

                • 5. Re: FTS Re-Index Performance 7.6.04 (Lucene/Tika)

                  Ok, this had a strong negative effect:

                  With this setting the re-index has processed 74 incidents in 182 minutes (0.4 per minute).

                  Before it was: 3,973 incidents in 741 minutes (5.4 per minute).


                  Any other suggestion? Here are my ideas:


                  1. I have seen in other threads, that the RPC-Socket 390602 was used for FTS, but I guess it was based on the former Hummingbird-technology and not the new Lucene/Tika-plugin?

                  Or is it a good idea to configure a setting like

                  Private-RPC-Socket:  390602   8   16


                  2. Is it reasonable to create the index first on a local disc (or even SSD) and copy/replicate it to the network-share afterwards, as mentioned in

                  http://wiki.apache.org/lucene-java/ImproveIndexingSpeed ?


                  3. Is there a setting to improve performance in \BMC Software\ARSystem\pluginsvr\fts\pluginsvr_config.xml

                  like "numCoreThreads" or "numSelectorThreads" ?
















                              <name>ARSYS.ARF.FTS</name> (...)


                  4. We replaced the out-of-the-box tika-jar (tika-app-0.6.jar) with the most up-to-date (tika-app-1.1.jar) from http://tika.apache.org/download.html

                  Seem to run fine so far. We just had to modify the appropriate line in \BMC Software\ARSystem\pluginsvr\fts\pluginsvr_config.xml to point to this jar.

                  We shall check the influence on the performance in the next days and discuss with BMC Support, if this is a supported environment.

                  • 6. FTS Re-Index Performance 7.6.04 (Lucene)
                    Robert Poulos

                    Since 390602 is the FT Indexer queue, that is the queue you can experiment with.  Having 2 or 3 threads configured is probably optimal.  However, I don't believe you will get much improvement there because your issue is most likely the network I/O performance for your shared index files.  I would recommend trying the option #2 you listed to get an idea of the scope of the improvement.  I can't recomment option #2 as a permanent solution because your searching performance will still be affected by the slow network I/O performance using the shared drive arrangement.  You will need to consider the alternative configuration discussed in the FTS performance document you referenced.

                    1 of 1 people found this helpful
                    • 7. Re: FTS Re-Index Performance 7.6.04 (Lucene)

                      Here are the results of our tests:


                      15% increase in Re-Index Performance: raise value „numSelectorThreads“ from 2 to 5 (in pluginsvr_config.xml in …\BMC Software\ARSystem\pluginsvr\fts)


                      annother 20% increase in Re-IndexPerformance:  raise Java-RAM for FTS „-Xmx2048m“ (in armonitor.cfg in …\BMC Software\ARSystem\Conf).


                      No or negative effects had:

                      -    change Tika-Version (from 0.6 to 1.1)

                      -    Set or raise the FTS-RPC-Threads (390602)

                      -    raise the parameter „numcore“ from 5 to 10 (in pluginsvr_config.xml unter …\BMC Software\ARSystem\pluginsvr\fts)


                      We guess we have to move the collection directory to the local filesystem and live with the needed manuell changes in case of a failure.

                      • 8. Re: FTS Re-Index Performance 7.6.04 (Lucene)
                        Markus Schindler



                        For 2 weeks we are going productive with 7.6.4 SP3 in a Servergroup with 7 AR Servers. The FTS is a horror :-((


                        My results of tests.:


                        If no user use the search plugin java port 9998 the indexer works fast . 6000-7000 entries in one hour

                        But if one user search for an fts field on any secondary server and the plugin java process 9998 started against the fts file share ( cif share ) the indexer on the primary server going down and only 2-3 entries per 5 minute will be create. When i killed the search process ( netstat -ano | findstr 9998 ) on the secondary server the indexer on the primary server going up and works fast .




                        Last sunday i install wireshark on the indexer server and trace the traffic ..


                        Result .:


                        If no user use the search function with fts the indexer works fast and no errors in wireshark , but if only one user search , indexer going down and thousands of error messages in wireshark . 




                        The cif file share is open with UNC .


                        We open a call by BMC since 10 days with prio high .. 


                        br from vienna

                        • 9. Re: FTS Re-Index Performance 7.6.04 (Lucene)
                          Laurent Matheo

                          What it the OS where the CIFS shared folder is?

                          For example there is a KB on this for Windows 2008 R2:


                          • 10. Re: FTS Re-Index Performance 7.6.04 (Lucene)
                            Markus Schindler



                            CIFS are not on a Windows Server , they have their own OS . But we install the two hotfixes for the indexer and for 1 secondary server . Same result .. We start a trace on the CIFS shares an see that the search server acess the files during the search , but after the search , the files always locked and the indexer goes down. The files are released when i kill the plugin process , in this moment indexer going up.


                            state=GRANTED mode=Read-denyN oplock=None fsid=0x096b1e97 fileid=0x00000040 type=waiting sharelock (llp=0x0000000138a21ea8 fcb=0x00000005c1d00578 tree=0x000000021c4a8038)


                            We changed the CIF properties to -> oplocks are disabled. , same result .. :-(


                            br from Vienna

                            • 11. Re: FTS Re-Index Performance 7.6.04 (Lucene)
                              Markus Schindler

                              and another weekend in the never ending FTS story


                              I've change the CIF Shares to external windows shares and point the indexer and the secondary servers to the windows shares . The same result ... when we search on the secondary server the indexer going down .


                              Then i changed the fts plugin target from the secondary server in the ar.cfg to the primary fts plugin . Now the primary plugin are the indexer and the searcher plugin . It works , indexer performance is ok , but the search performance during the day is slow ,  but it works .


                              Then i changed the external share to a local share on the primary server and config the secondary server with his own plugin to the local share on the primary server . The result .: Indexer works very fast and the search also .


                              I sent the result to BMC and waiting for response ..previously, they were not very helpful :-( 


                              br from Vienna

                              • 12. Re: FTS Re-Index Performance 7.6.04 (Lucene)
                                Jonathan Drury

                                Hi Vienna,

                                I know this was an old post, but can you point me to a post, or instructions on the details of how you setup the separate servers/plugins?


                                "Then i changed the external share to a local share on the primary server and config the secondary server with his own plugin to the local share on the primary server . The result .: Indexer works very fast and the search also ."