[NYF] How to read the 9.0 FTS indexes (the force is strong with him)

Version 2
    Share This:

    Hello again

     

    If you remember, I already did a document about browsing FTS indexes a while back, and I renamed it to:

    [NYF] How to read the 7.6.04 / 8.x FTS indexes (the force is strong with him)

     

    The reason is the tool used for this is not the same anymore (different version) and well it's a bit "too" different to have both versions in the same document...

     

    And anyway, it's time for a reboot

    27A5286C00000578-3042308-image-a-130_1429223616749.jpg

    Please Disney, don't mess up... Kill Jar Jar...

     

     

    So anyway, as you know ARS uses Full text search and you have some nice documents around here explaining how it is working and how to tune it:

    Trending in Support: FTS Fortification

    Connect with Remedy - FTS Configuration for Performance and High Availability Webinar

    Ash's Guide to Knowledge Management in Smart IT & RKM - Searching, Optimization, Process, Indexing & Images

    Configuring_FTS_for_performance_in_a_server_group.docx

    Trending in Support: High Availability for FTS in a Server Group

     

    But a question was going round my head, how to be actually sure that a data is indexed?

     

    A common case is when you create a knowledge article, you save it, you try to search it immediately in "Global Search" and... nothing...

    Most of the time you have to wait "some time" for your search to be successful.

    At the end of the day, you don't actually know if indexing went ok, or it's still in the indexing queue, or the FTS plugin is down

     

    So my question is, how can you check the data within the FTS indexes?

    The problem is that FTS indexes are not stored in database, but in flat files:

    2015-05-13_01-18-10.png

     

     

    The technology behind seems to be Apache Lucene 4.9 (Apache Lucene - Apache Lucene Core):

    2015-05-13_01-28-16.png

    So we saw in the previous thread that opening FTS indexes was possible using Luke (hence the bad joke ):

    [NYF] How to read the 7.6.04 / 8.x FTS indexes (the force is strong with him)

     

    The problem with Luke is the latest version (4.0) has been dropped a couple of years back and not compatible with the new Lucene libraries anymore

     

    BUT, since it was an open source project, it has been picked up and still maintained

    56812581.jpg

    So I used the Forked version from DmitryKey you can find here:

    DmitryKey/luke · GitHub

     

    You can download a "built" version here (Releases · DmitryKey/luke · GitHub), I used the version "4.10.4" and the file "luke-with-deps.jar".

     

    This way, you can check what are the current data stored in the FTS indexes and do basic searches.

    Using this tool, you can quickly see if an incident, change, article has been indexed and even do some quick searches.

     

    I made a quick video to show you how to setup and use "Luke" for ARS 9.0:

    [ARS 9.0] How to browse your FTS indexes - YouTube

     

    Note:

    Be sure to check the "old" video for more tricks on the search part too.

     

     

    Oh yes, just one more thing... If you don't look at the video there is a catch

    tumblr_m94dqe5duH1rr3e86o1_250.gif

    As you saw there are now several folders in the "collection" folder:

    2015-05-13_01-18-10.png

    When you open the "main" folder in Luke, you won't see any indexes.

    The reason is that, even if Luke is supposed to browse indexes from subdirectories, it seems it does not work.

    You have, in Luke to open a subfolder to see the indexes and their values

     

    BUT it seems (I'm not R&D remember) that each folder matches one form

    As you know, one form in ARS (let's say "HPD:Help Desk") has a "Txxxx" table in the database.

    This "xxxx" is know as the "schemaid".

    As you will see in the video, I saw that the folder is actually the "schemaid".

     

    For example, the schemaid from "HPD:Help Desk" is, for me, 2471:

    2015-05-13_01-45-55.pngAnd such a folder exists:

    2015-05-13_01-18-10.png

    And if you open it in Luke, you will see all the indexes and their values, for this form

    2015-05-13_01-47-36.png

    So the good thing in ARS 9, is that now one form has its own indexes, meaning it will be much easier for you to search and browse the FTS indexes from a specific form, you will not have to copy all files anymore.

     

     

    Note:

    Like I said, I am not BMC R&D but I think it's working this way. I made some tests and it seems so.

    It seems to be confirmed (version 4.9 and schema format) in this doc wiki page

    https://docs.bmc.com/docs/display/public/ars9000/BMC+Remedy+AR+System+server+enhancements+in+version+9.0.00#BMCRemedyARSystemserverenhancementsinversion9.0.00-Full-TextSearchenhancements

     

     

     

    Links / resources:

    Luke from DmitryKey (I used version 4.10.4):

    Releases · DmitryKey/luke · GitHub

     

    Apache documentation for Query Syntax for 4.9:

    http://lucene.apache.org/core/4_9_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description