13 Replies Latest reply on Jul 7, 2011 4:03 PM by Jan Lindhardsen

    ITSM 7.6 perfomance issues with 10k+ companies

    Jan Lindhardsen

      Dear Community!


      I have ran across some huge performance issues implementing ITSM 7.6 at at Telco customer site.


      This company want to use ITSM to support their  business customers, which is approx. 75.000 at the moment.
      Each customer will thereby have a record in COM:Company, and a number of records in supporting forms such as Location, People etc.


      I expected to run into some problems with performance in this, and I certainly did:


      After as few as a 1000 customers, the system becomes more or less impossible to use, when it comes to "Customer Drop-Down-List", such as the one that are present in the Application Administration Console or on forms such as Incident etc.


      I currently have my test system loaded with 26k dummy customers, and currently it takes 1 minute 15 seconds. The reason for this is that the API has to build a drop-down with 26k customers, each time we open the drop down. So I guess that there is not much that can be done to make the default behavior acceptable to the end users?
      This is on a test box, but I still have lots of both CPU and memory, so I cannot imagine that our production box will perform much better.


      Issues like these I expected (sort of), but what I want your help with, is how you have solved this before, and if these is more issues related to a 60k customer implementation, that I should be aware of, before I get there?


      I can see that changing all of the drop-downs to simple "search fields" using leading searches, will be an acceptable work around to this issue, but again, how have others handled this issue?

      What are the BMC best practice in this area, and what other workflow might we want to change to ensure best performance and user experience in a system supporting 75k customers/companies?


      I have been trying to get an statement from Customer Support, but so far with no luck.


      Best regards,

        • 1. Re: ITSM 7.6 perfomance issues with 10k+ companies

          Well, I certainly don't know what type of backend or server architecture you are running, but for something of this nature I would assume there is a db server that may actually be a cluster and the Remedy Services should probably be being offered by a server gourp with some for of load balancing.  But that really doesn't help you much with the menu's.  Depending on how dynamic the customer list and/or menu is, you may benefit from changing the menu's refresh property to have the menu refresh every 15 mins.  It may reduce the wait time when you first click on the list.  Also you could require that the user type in at least 2-3 letters and let the menu filter the values returned.

          • 2. Re: ITSM 7.6 perfomance issues with 10k+ companies
            Doug Mueller

            My answer is not going to be from an ITSM specific perspective but more from the perspective of any application and best practices around how to structure things when using menus (by the way, not saying that ITSM is doing it right or not).


            First, you need to take a step back and look at what you want to accomplish.  There is a field that has company (could be anything in general, but is company in this case).  Typically, you would select the company from a list and go.  However, in this case, you have a large list.  If you have a list of over a couple hundred, any type of a list is just not practical.  Finding something in that list is hard.


            You want to structure the application that has long lists to give assistance, but not just a blind list of all possibilities.


            So, you first don't want a field that requires the entire list to be present.  You need a field you can type in to give some assistance.  If you can do that with the drop down type list and you like that interaction, then use that field style.  If not, change to a regular text field with a menu.


            For your menu, you don't want the menu EVER allowed to bring back the entire content of the large set.  Think about it.  It is useless to have a menu with 1000s of items because it is simply not useful.


            Two major options....


            Use a menu, but qualify it for partial assistance.  Build it something like   $field$ != " " AND 'field on form being queried' LIKE "%" + $field$ + "%"

            What this does is to NOT match anything if there is no assistance given (so you don't ever try and get all things) and brings back items only that match somewhere in the company name (field on form being queried).  You could set it up for leading match or other things.  Note that in 7.5 there is even a feature in the AR System that allows you to do partial match where the drop down occcurs after entering three characters.   Anyway, the point is to limit the data retrieved and you need some level of info from the customer.  Without that, you get menus out of control.


            If the menu doesn't have these controls, change the field/adjust the menu to have this.  It does not affect the other interaction of the application -- just the loading of this field using the menu.


            Option two is for cases where there is more sophistication needed.  In that case, you don't have a menu associated but put a button near the field that the user clicks on to get help.  This pops up a dialog that may ask questions or give multiple fields of qualification and then you get items in a table or you use a set fields with popup for user selection or something to get the user to help you with qualificaiton to give them a reasonable subset.


            In the case you describe, the first option of menus with partial display seems to be the better choice.

            1 of 1 people found this helpful
            • 3. Re: ITSM 7.6 perfomance issues with 10k+ companies
              Jan Lindhardsen

              Thanks for both your insights.


              I agree with you,  that the first option is the best way to go.


              I guess what I'm really asking for here, is if somebody elsa has already done this, and want's to share thier workflow.

              Or even better if there exist an "official" BMC patch to address this issue.


              I have no problem creating this myself, but I still would like to see if somebody has done this before me. This customer cannot be the first enterprise customer with the same challenge!


              Also, my pther question was: "What kind of similar probelms can I expect to run into"? Any insights on this?


              Best regards,


              • 4. Re: ITSM 7.6 perfomance issues with 10k+ companies



                We have done this, but not in ITSM. Same issue and almost exact solution than suggested earlier.


                On the company field on our incident form, we run a search menu, on open.


                Menu qualification is :


                'Company' LIKE ($536870913$ +  "%" )


                Above, 'Company', is the field on the form that holds a list of the Companies. Your qualification can be more complex.

                Above, $536870913$, is the Company field on the incident form.

                We do not include the leading %, so the search is a bit quicker.


                Our company form is thin and quick, holds about 4500 records.


                Not sure how to force a user to type more than one character before clicking the menu.




                • 5. Re: ITSM 7.6 perfomance issues with 10k+ companies
                  Doug Mueller

                  There is no BMC patch for this issue.  It is an example where there has to be a choice.  If the typical use case is a small list, showing the full list is easier and less confusing that partial qualification but if there is a large list then partial qualification is right.  The vast majority of use cases are a small list.  It is a minor customization to adapt to the strategy described.


                  As for other customers doing this -- yes, there are a number.  pauln posted a note describing their situation.


                  From what I know there are no other major issues like this related to a large company list.


                  The product is in use in many major outsourcers -- and they all have reasonably long lists of companies -- and they are all working just fine with the solution.

                  • 6. Re: ITSM 7.6 perfomance issues with 10k+ companies

                    best practice would be to type in a persons name or at least a part of the name and then hit enter this would result in a limited popup and not the entire list?



                    • 7. Re: ITSM 7.6 perfomance issues with 10k+ companies
                      Jan Lindhardsen

                      Hi All!


                      Hope you guys are still monitoring this issue?

                      This sort of fell behind in our implementation, as I had found the answer to my question But Now when it is time to implement the solution I, have a major issue with the soultion suggested by Paul above:


                      Using a menu qualification with the LIKE  operator, seems to have a limitation, when it comes to menu pattern checking.

                      If I use the menu qualification above, and use pattern checking = $MENU$, then I get an error saying that the value does not match the Menu.


                      Any help on this?

                      • 8. Re: ITSM 7.6 perfomance issues with 10k+ companies
                        Jan Lindhardsen

                        Just forget about this... I implemented the check with active links

                        • 9. ITSM 7.6 perfomance issues with 10k+ companies



                          We have similar situation with SIte+ field.  We have 40k Sites with aliases becomes almost 150K records. When user clicks on down arrow, it gives database timeout.  The Site+ field is made up of three different fields. Remedy Users should be able to search on any values in the Site+ field.


                          If possible, we would like to avoid customization to hide menu drop down.


                          Any option with database like better indices or even partitioning the location table?


                          Any help would be greatly appreciated.





                          • 10. ITSM 7.6 perfomance issues with 10k+ companies

                            Another possibility (perhaps more forward looking) is to upgrade to ITSM Suite 7.6.04 or later and utilize the auto-complete feature.  BMC has tuned auto-complete to address this exact use case (a menu with a very large number of items – like a customer list). 


                            Auto-complete was actually introduced in AR System 7.5.00, but the performance increase for this use case was added in 7.6.04. 


                            28-Jan-2011 What’s New: BMC Remedy Action Request System7.6.04 PDF


                            Enhanced auto-completion

                            The automatic complete feature is improved to handle large data sets in search menus that are attached to a character field (for example, over 10,000 entries). This feature also minimizes unwanted searches.


                            For more information, see the Form and Application ObjectsGuide.




                            David J. Easter

                            Product Line Manager
                            Remedy Platform
                            BMC Software, Inc.

                            • 11. ITSM 7.6 perfomance issues with 10k+ companies
                              Jan Lindhardsen

                              Mukesh, as david is saying, now you have the fully functional Auto-complete function i 7.6.04. This is really the best way to go, and exactly what I was hoping for, when I originally wrote this question!

                              What is new in this version, and what really makes this useful is that you can hide the drop-down menu, and you can said taht it should first react after X characters.

                              At the customer where we had this problem we are now implementing 7.6.04 with this fucntionality (generate search result/menu after 3 charaters).


                              IF you by any chance can, go with an upgrade of the ARS server platform to 7.6.04, the applications you can always upgrade later. and the implement the functionality based on this.


                              If not, I guess your only (or best?) option is to do as we did on 7.6.03:

                              On each form/Company (Site in your case) field, remove the drop-down on the field, and replace the menu with an active link that performs a search on Return.

                              This is quite a lot of job, but absolutely doable!

                              One disadvantage is taht you cannot format the reult list the way you might want it.

                              • 12. ITSM 7.6 perfomance issues with 10k+ companies

                                Thanks David and Jan,


                                Your response to Site+ issue understood as follows.


                                -  Upgrade to ARS 7.6.04 platform only will enable Auto-Complete properties on a Menus.

                                -  Remedy developers need to customize impacted menu(s) attached to the character fields used in ITSM to take advantage of this new feature.

                                -  All character fields attached to an impacted menu will be enable automatically to take advatage of Auto-Complete feature ( after X number of character typed in the menu field).


                                This is greatly appreciated.  Please correct my understanding as needed.


                                Thanks again.

                                • 13. ITSM 7.6 perfomance issues with 10k+ companies
                                  Jan Lindhardsen

                                  You actually got it a bit wrong


                                  The functionlaity is not related to me menu itself, but to the charater field which has the menu attached.


                                  This menas that you need to open each form and modify each field you want to have this behaviour.

                                  Probably a bit more work, but the result is all worth it!