[NYF] How to read the 9.0 FTS indexes (the force is strong with him)

Version 2
    Share This:

    Hello again


    If you remember, I already did a document about browsing FTS indexes a while back, and I renamed it to:

    [NYF] How to read the 7.6.04 / 8.x FTS indexes (the force is strong with him)


    The reason is the tool used for this is not the same anymore (different version) and well it's a bit "too" different to have both versions in the same document...


    And anyway, it's time for a reboot


    Please Disney, don't mess up... Kill Jar Jar...



    So anyway, as you know ARS uses Full text search and you have some nice documents around here explaining how it is working and how to tune it:

    Trending in Support: FTS Fortification

    Connect with Remedy - FTS Configuration for Performance and High Availability Webinar

    Ash's Guide to Knowledge Management in Smart IT & RKM - Searching, Optimization, Process, Indexing & Images


    Trending in Support: High Availability for FTS in a Server Group


    But a question was going round my head, how to be actually sure that a data is indexed?


    A common case is when you create a knowledge article, you save it, you try to search it immediately in "Global Search" and... nothing...

    Most of the time you have to wait "some time" for your search to be successful.

    At the end of the day, you don't actually know if indexing went ok, or it's still in the indexing queue, or the FTS plugin is down


    So my question is, how can you check the data within the FTS indexes?

    The problem is that FTS indexes are not stored in database, but in flat files:




    The technology behind seems to be Apache Lucene 4.9 (Apache Lucene - Apache Lucene Core):


    So we saw in the previous thread that opening FTS indexes was possible using Luke (hence the bad joke ):

    [NYF] How to read the 7.6.04 / 8.x FTS indexes (the force is strong with him)


    The problem with Luke is the latest version (4.0) has been dropped a couple of years back and not compatible with the new Lucene libraries anymore


    BUT, since it was an open source project, it has been picked up and still maintained


    So I used the Forked version from DmitryKey you can find here:

    DmitryKey/luke · GitHub


    You can download a "built" version here (Releases · DmitryKey/luke · GitHub), I used the version "4.10.4" and the file "luke-with-deps.jar".


    This way, you can check what are the current data stored in the FTS indexes and do basic searches.

    Using this tool, you can quickly see if an incident, change, article has been indexed and even do some quick searches.


    I made a quick video to show you how to setup and use "Luke" for ARS 9.0:

    [ARS 9.0] How to browse your FTS indexes - YouTube



    Be sure to check the "old" video for more tricks on the search part too.



    Oh yes, just one more thing... If you don't look at the video there is a catch


    As you saw there are now several folders in the "collection" folder:


    When you open the "main" folder in Luke, you won't see any indexes.

    The reason is that, even if Luke is supposed to browse indexes from subdirectories, it seems it does not work.

    You have, in Luke to open a subfolder to see the indexes and their values


    BUT it seems (I'm not R&D remember) that each folder matches one form

    As you know, one form in ARS (let's say "HPD:Help Desk") has a "Txxxx" table in the database.

    This "xxxx" is know as the "schemaid".

    As you will see in the video, I saw that the folder is actually the "schemaid".


    For example, the schemaid from "HPD:Help Desk" is, for me, 2471:

    2015-05-13_01-45-55.pngAnd such a folder exists:


    And if you open it in Luke, you will see all the indexes and their values, for this form


    So the good thing in ARS 9, is that now one form has its own indexes, meaning it will be much easier for you to search and browse the FTS indexes from a specific form, you will not have to copy all files anymore.




    Like I said, I am not BMC R&D but I think it's working this way. I made some tests and it seems so.

    It seems to be confirmed (version 4.9 and schema format) in this doc wiki page





    Links / resources:

    Luke from DmitryKey (I used version 4.10.4):

    Releases · DmitryKey/luke · GitHub


    Apache documentation for Query Syntax for 4.9: