It's a ARS-7.6.04SP3 (ITSM-7.6.04SP2) on Windows2008 with MS-SQL as DB.
The index-Collection directory is on a network-share due to the servergroup-configuration
The "FT-Indexer threads" might give us a clue... Is it one of the Private-RPC-Sockets? If Yes, which one is reserved for FTS?
Private-RPC-Socket: 390601 1 1
Private-RPC-Socket: 390603 1 1
Private-RPC-Socket: 390620 8 32
Private-RPC-Socket: 390621 5 16
Private-RPC-Socket: 390626 8 16
Private-RPC-Socket: 390635 8 32
Private-RPC-Socket: 390680 2 2
Private-RPC-Socket: 390681 8 16
Ok, this had a strong negative effect:
With this setting the re-index has processed 74 incidents in 182 minutes (0.4 per minute).
Before it was: 3,973 incidents in 741 minutes (5.4 per minute).
Any other suggestion? Here are my ideas:
1. I have seen in other threads, that the RPC-Socket 390602 was used for FTS, but I guess it was based on the former Hummingbird-technology and not the new Lucene/Tika-plugin?
Or is it a good idea to configure a setting like
Private-RPC-Socket: 390602 8 16
2. Is it reasonable to create the index first on a local disc (or even SSD) and copy/replicate it to the network-share afterwards, as mentioned in
3. Is there a setting to improve performance in \BMC Software\ARSystem\pluginsvr\fts\pluginsvr_config.xml
like "numCoreThreads" or "numSelectorThreads" ?
4. We replaced the out-of-the-box tika-jar (tika-app-0.6.jar) with the most up-to-date (tika-app-1.1.jar) from http://tika.apache.org/download.html
Seem to run fine so far. We just had to modify the appropriate line in \BMC Software\ARSystem\pluginsvr\fts\pluginsvr_config.xml to point to this jar.
We shall check the influence on the performance in the next days and discuss with BMC Support, if this is a supported environment.
Since 390602 is the FT Indexer queue, that is the queue you can experiment with. Having 2 or 3 threads configured is probably optimal. However, I don't believe you will get much improvement there because your issue is most likely the network I/O performance for your shared index files. I would recommend trying the option #2 you listed to get an idea of the scope of the improvement. I can't recomment option #2 as a permanent solution because your searching performance will still be affected by the slow network I/O performance using the shared drive arrangement. You will need to consider the alternative configuration discussed in the FTS performance document you referenced.
Here are the results of our tests:
15% increase in Re-Index Performance: raise value „numSelectorThreads“ from 2 to 5 (in pluginsvr_config.xml in …\BMC Software\ARSystem\pluginsvr\fts)
annother 20% increase in Re-IndexPerformance: raise Java-RAM for FTS „-Xmx2048m“ (in armonitor.cfg in …\BMC Software\ARSystem\Conf).
No or negative effects had:
- change Tika-Version (from 0.6 to 1.1)
- Set or raise the FTS-RPC-Threads (390602)
- raise the parameter „numcore“ from 5 to 10 (in pluginsvr_config.xml unter …\BMC Software\ARSystem\pluginsvr\fts)
We guess we have to move the collection directory to the local filesystem and live with the needed manuell changes in case of a failure.
For 2 weeks we are going productive with 7.6.4 SP3 in a Servergroup with 7 AR Servers. The FTS is a horror :-((
My results of tests.:
If no user use the search plugin java port 9998 the indexer works fast . 6000-7000 entries in one hour
But if one user search for an fts field on any secondary server and the plugin java process 9998 started against the fts file share ( cif share ) the indexer on the primary server going down and only 2-3 entries per 5 minute will be create. When i killed the search process ( netstat -ano | findstr 9998 ) on the secondary server the indexer on the primary server going up and works fast .
Last sunday i install wireshark on the indexer server and trace the traffic ..
If no user use the search function with fts the indexer works fast and no errors in wireshark , but if only one user search , indexer going down and thousands of error messages in wireshark .
The cif file share is open with UNC .
We open a call by BMC since 10 days with prio high ..
br from vienna
CIFS are not on a Windows Server , they have their own OS . But we install the two hotfixes for the indexer and for 1 secondary server . Same result .. We start a trace on the CIFS shares an see that the search server acess the files during the search , but after the search , the files always locked and the indexer goes down. The files are released when i kill the plugin process , in this moment indexer going up.
state=GRANTED mode=Read-denyN oplock=None fsid=0x096b1e97 fileid=0x00000040 type=waiting sharelock (llp=0x0000000138a21ea8 fcb=0x00000005c1d00578 tree=0x000000021c4a8038)
We changed the CIF properties to -> oplocks are disabled. , same result .. :-(
br from Vienna
and another weekend in the never ending FTS story
I've change the CIF Shares to external windows shares and point the indexer and the secondary servers to the windows shares . The same result ... when we search on the secondary server the indexer going down .
Then i changed the fts plugin target from the secondary server in the ar.cfg to the primary fts plugin . Now the primary plugin are the indexer and the searcher plugin . It works , indexer performance is ok , but the search performance during the day is slow , but it works .
Then i changed the external share to a local share on the primary server and config the secondary server with his own plugin to the local share on the primary server . The result .: Indexer works very fast and the search also .
I sent the result to BMC and waiting for response ..previously, they were not very helpful :-(
br from Vienna
I know this was an old post, but can you point me to a post, or instructions on the details of how you setup the separate servers/plugins?
"Then i changed the external share to a local share on the primary server and config the secondary server with his own plugin to the local share on the primary server . The result .: Indexer works very fast and the search also ."