12 Replies Latest reply: Aug 6, 2013 8:54 AM by jdrury RSS

FTS Re-Index Performance 7.6.04 (Lucene)

Roger Jakobs

Hi Everybody,

we encounter long FTS re-index duration (more than 30 days) in an up-to-date environment (ITSM 7.6.04, windows).

Does anyone has experienced better performance-values?

 

-> indexing of 5-6 incidents per minute

 

There are more than 280.000 incidents (HPD:HelpDesk) and more than 150.000 Tasks (TMS:Task) in the system, which leads to more than a month of re-indexing-time?

 

What can be done to improve performance (a standalone plugin-server was already discussed)?

https://communities.bmc.com/communities/servlet/JiveServlet/download/249059-31229560/Configuring_FTS_for_performance_in_a_server_group.docx

 

Any hint would be helpful.

 

Best regards,

Roger

  • 1. FTS Re-Index Performance 7.6.04 (Lucene)
    Bob Poulos

    A little more information about your environment would be helpful.  Specifically:

     

    What service pack of 7.6.04 are you on?

    What database type?

    Is the index collection directory local to the server doing the indexing?

    Is the server in a server group?

    How many FT Indexer threads are configured?

     

    Thank you -

  • 2. FTS Re-Index Performance 7.6.04 (Lucene)
    Laurent Matheo

    If it's genuine 7.6.04, try using the FTS plugins from 7.6.04 sp2 or sp3.

  • 3. Re: FTS Re-Index Performance 7.6.04 (Lucene)
    Roger Jakobs

    It's a ARS-7.6.04SP3 (ITSM-7.6.04SP2) on Windows2008 with MS-SQL as DB.

    The index-Collection directory is on a network-share due to the servergroup-configuration
    (\\share\backup\failover\fts\configuration\).

    The "FT-Indexer threads" might give us a clue... Is it one of the Private-RPC-Sockets? If Yes, which one is reserved for FTS?

    Private-RPC-Socket:  390601   1   1

    Private-RPC-Socket:  390603   1   1

    Private-RPC-Socket:  390620   8  32

    Private-RPC-Socket:  390621   5  16

    Private-RPC-Socket:  390626   8  16

    Private-RPC-Socket:  390635   8  32

    Private-RPC-Socket:  390680   2   2

    Private-RPC-Socket:  390681   8  16

     

    Best Regards,

    Roger

  • 4. FTS Re-Index Performance 7.6.04 (Lucene)
    Laurent Matheo

    Try setting more escalation threads:

    Private-RPC-Socket:  390603   1   1

     

    To:

    Private-RPC-Socket:  390603   1   3

     

    And restart ARS.

  • 5. Re: FTS Re-Index Performance 7.6.04 (Lucene/Tika)
    Roger Jakobs

    Ok, this had a strong negative effect:

    With this setting the re-index has processed 74 incidents in 182 minutes (0.4 per minute).

    Before it was: 3,973 incidents in 741 minutes (5.4 per minute).

     

    Any other suggestion? Here are my ideas:

     

    1. I have seen in other threads, that the RPC-Socket 390602 was used for FTS, but I guess it was based on the former Hummingbird-technology and not the new Lucene/Tika-plugin?

    Or is it a good idea to configure a setting like

    Private-RPC-Socket:  390602   8   16

     

    2. Is it reasonable to create the index first on a local disc (or even SSD) and copy/replicate it to the network-share afterwards, as mentioned in

    http://wiki.apache.org/lucene-java/ImproveIndexingSpeed ?

     

    3. Is there a setting to improve performance in \BMC Software\ARSystem\pluginsvr\fts\pluginsvr_config.xml

    like "numCoreThreads" or "numSelectorThreads" ?

    <pluginsvr_config>

        <port>9998</port>

        <regPortMapper>false</regPortMapper>

        <encryptionPolicy>2</encryptionPolicy>

        <publicKeyAlg>4</publicKeyAlg>

        <publicKeyExpiry>86400</publicKeyExpiry>

        <dataEncryptionAlg>1</dataEncryptionAlg>

        <dataKeyExpiry>2700</dataKeyExpiry>

        <numCoreThreads>5</numCoreThreads>

        <numSelectorThreads>2</numSelectorThreads>

        <workQueueMonitorLogInterval>0</workQueueMonitorLogInterval>

        <workQueueTaskThreshold>5</workQueueTaskThreshold>

     

          <plugins>

              <plugin>

                <name>ARSYS.ARF.FTS</name> (...)

     

    4. We replaced the out-of-the-box tika-jar (tika-app-0.6.jar) with the most up-to-date (tika-app-1.1.jar) from http://tika.apache.org/download.html

    Seem to run fine so far. We just had to modify the appropriate line in \BMC Software\ARSystem\pluginsvr\fts\pluginsvr_config.xml to point to this jar.

    We shall check the influence on the performance in the next days and discuss with BMC Support, if this is a supported environment.

  • 6. FTS Re-Index Performance 7.6.04 (Lucene)
    Bob Poulos

    Since 390602 is the FT Indexer queue, that is the queue you can experiment with.  Having 2 or 3 threads configured is probably optimal.  However, I don't believe you will get much improvement there because your issue is most likely the network I/O performance for your shared index files.  I would recommend trying the option #2 you listed to get an idea of the scope of the improvement.  I can't recomment option #2 as a permanent solution because your searching performance will still be affected by the slow network I/O performance using the shared drive arrangement.  You will need to consider the alternative configuration discussed in the FTS performance document you referenced.

  • 7. Re: FTS Re-Index Performance 7.6.04 (Lucene)
    Roger Jakobs

    Here are the results of our tests:

     

    15% increase in Re-Index Performance: raise value „numSelectorThreads“ from 2 to 5 (in pluginsvr_config.xml in …\BMC Software\ARSystem\pluginsvr\fts)

     

    annother 20% increase in Re-IndexPerformance:  raise Java-RAM for FTS „-Xmx2048m“ (in armonitor.cfg in …\BMC Software\ARSystem\Conf).

     

    No or negative effects had:

    -    change Tika-Version (from 0.6 to 1.1)

    -    Set or raise the FTS-RPC-Threads (390602)

    -    raise the parameter „numcore“ from 5 to 10 (in pluginsvr_config.xml unter …\BMC Software\ARSystem\pluginsvr\fts)

     

    We guess we have to move the collection directory to the local filesystem and live with the needed manuell changes in case of a failure.

  • 8. Re: FTS Re-Index Performance 7.6.04 (Lucene)
    Vienna_Markus

    Hy

     

    For 2 weeks we are going productive with 7.6.4 SP3 in a Servergroup with 7 AR Servers. The FTS is a horror :-((

     

    My results of tests.:

     

    If no user use the search plugin java port 9998 the indexer works fast . 6000-7000 entries in one hour

    But if one user search for an fts field on any secondary server and the plugin java process 9998 started against the fts file share ( cif share ) the indexer on the primary server going down and only 2-3 entries per 5 minute will be create. When i killed the search process ( netstat -ano | findstr 9998 ) on the secondary server the indexer on the primary server going up and works fast .

     

    hmmm

     

    Last sunday i install wireshark on the indexer server and trace the traffic ..

     

    Result .:

     

    If no user use the search function with fts the indexer works fast and no errors in wireshark , but if only one user search , indexer going down and thousands of error messages in wireshark . 

     

    Error:STATUS_SHARING_VIOLATION

     

    The cif file share is open with UNC .

     

    We open a call by BMC since 10 days with prio high .. 

     

    br from vienna

  • 9. Re: FTS Re-Index Performance 7.6.04 (Lucene)
    Laurent Matheo

    What it the OS where the CIFS shared folder is?

    For example there is a KB on this for Windows 2008 R2:

    http://support.microsoft.com/kb/2707576

  • 10. Re: FTS Re-Index Performance 7.6.04 (Lucene)
    Vienna_Markus

    Hy

     

    CIFS are not on a Windows Server , they have their own OS . But we install the two hotfixes for the indexer and for 1 secondary server . Same result .. We start a trace on the CIFS shares an see that the search server acess the files during the search , but after the search , the files always locked and the indexer goes down. The files are released when i kill the plugin process , in this moment indexer going up.

     

    state=GRANTED mode=Read-denyN oplock=None fsid=0x096b1e97 fileid=0x00000040 type=waiting sharelock (llp=0x0000000138a21ea8 fcb=0x00000005c1d00578 tree=0x000000021c4a8038)

     

    We changed the CIF properties to -> oplocks are disabled. , same result .. :-(

     

    br from Vienna

  • 11. Re: FTS Re-Index Performance 7.6.04 (Lucene)
    Vienna_Markus

    and another weekend in the never ending FTS story

     

    I've change the CIF Shares to external windows shares and point the indexer and the secondary servers to the windows shares . The same result ... when we search on the secondary server the indexer going down .

     

    Then i changed the fts plugin target from the secondary server in the ar.cfg to the primary fts plugin . Now the primary plugin are the indexer and the searcher plugin . It works , indexer performance is ok , but the search performance during the day is slow ,  but it works .

     

    Then i changed the external share to a local share on the primary server and config the secondary server with his own plugin to the local share on the primary server . The result .: Indexer works very fast and the search also .

     

    I sent the result to BMC and waiting for response ..previously, they were not very helpful :-( 

     

    br from Vienna

  • 12. Re: FTS Re-Index Performance 7.6.04 (Lucene)
    jdrury

    Hi Vienna,

    I know this was an old post, but can you point me to a post, or instructions on the details of how you setup the separate servers/plugins?

     

    "Then i changed the external share to a local share on the primary server and config the secondary server with his own plugin to the local share on the primary server . The result .: Indexer works very fast and the search also ."

     

    Thanks,

    Jonathan