1 2 3 Previous Next 35 Replies Latest reply on Mar 27, 2019 8:12 AM by abdallah hamaydeh

    Footprints 2018 R2 still getting slowness issue!

    abdallah hamaydeh
      Share:|

      Hi All,

       

      After doing the needful checklist and all kind of testing on the NW, we still facing slowness during creating a new request, edit mode and scrolling, any ideas, and if anyone facing this kind of slowness

       

      Homepage charts take almost 1 min to appear and every login same thing.

       

      we have around 150 requests only and we plan to go live soon!?

       

       

       

      KnowledgeArticle - BMC

       

      thanks in advance!

        • 1. Re: Footprints 2018 R2 still getting slowness issue!
          Nicolas Roome

          Charts displaying in console widgets have always been slow. Generally 15-30 seconds is what I seem to see.

           

          How is your database server doing? Server RAM? SQL memory allocation? Deadlocks, etc

          • 2. Re: Footprints 2018 R2 still getting slowness issue!
            abdallah hamaydeh

            Nicolas Roome

            Unfortunately Slowness issue can be considered as public problem since lunching FP12, with all of the FP12 customers are complaining about the Slowness even the Customers using FootPrints 12 have reported dissatisfaction with the usability and performance of the product.

             

            We hopefully that Footprints team and find a permanent solution with the slowness issue that never seen on the FP11, even if we talk about Remedy smart IT.

             

            With my best Regards,

            • 3. Re: Footprints 2018 R2 still getting slowness issue!
              Jean AD

              What AB are you using? LDAP? FP internal?...

               

              I have many customers with (each) more than 200 000 tickets, Over 20 000 End Users, Over 150 Agent, More than 350 Business rule running (about 300 are On Screen BR), many workspaces, More than 60 SLTs, KB, Service Catalog, triggers in SQL...

              And we don't have performances issues!!

              • 4. Re: Footprints 2018 R2 still getting slowness issue!
                Jean AD

                How did you configure your Tomcat?

                • 5. Re: Footprints 2018 R2 still getting slowness issue!
                  Nicolas Roome

                  Out of curiosity Jean AD, are you doing anything special? And how do you define "no performance issues"? For instance, back in the SDE days it was sort of accepted that "no performance issue" meant the product was responsive within 3 seconds. Generally I know that for FootPrints, <5-10 seconds is 'normal'.

                   

                  I certainly wouldn't call FootPrints speedy compared to how SDE was back then.. but I can't say I've experienced slownesses either.

                   

                  Fun fact, when Remedyforce first came out it (as Service Desk on Force.com back then) it was extremely slow with some page loads in the 30+ seconds. BMC soon realized their fancy Visualforce pages were to blame and eventually turfed most of them. It meant that Remedyforce stopped looking like an SDE/Remedy clone and looked more like Salesforce. But it improved performance.

                   

                  FootPrints is basically a 4 year old product right now.. it's fairly on course with how Remedyforce was when it was 4 years old.

                  • 6. Re: Footprints 2018 R2 still getting slowness issue!
                    Jean AD

                    No performance Issue ==> Create/Edit ticket as agent < 5 seconds

                    For the configuration I mentionned above, it's 2-3 seconds

                    • 7. Re: Footprints 2018 R2 still getting slowness issue!
                      Steve van der Merwe

                      Hi Abdallah

                       

                      Can you share your Tomcat server.xml file as well as your Java Options?  I am more to agree that it is NOT Footprints.  The dependencies to Tomcat, the OS, server configuration and SQL is more likely points to investigate.  The out-of-the-box tomcat service install for Windows is terrible.  Have you heard about Tomcat APR? I will advise you look at making some changes in server.xml and Java Options to tell Java and Tomcat to make better use of your system.

                       

                      This looks like a beast of a server and you should not have performance issues.  What I sometimes found (in virtualized environments) is that adding more memory and CPU sometimes hide the real issue.  But I assume your install is not virtualized?  Here I advise you start your java options with minimum 4GB RAM and Max 8GB RAM as part of your test. You do not want Tomcat to go mad and try to build up a huge memory and do no Garbage Collection (i.e. java options).

                       

                      What I will start looking at is your disk configuration.  Is it SATA, SCSI or SSD?  If not using SSD, how is your disk array configured? Did you lump your OS and everything on one disk array or is it partitioned?  What AV/Malware tools are installed on this system?  You basically have to think about disk IO before memory and CPU as web applications are generally disk io intensive.

                       

                      I have a Linux based FP 12 running (my test system with Postgres on the same OS) that is using 2 virtual CPUs and 4 GB RAM in total and it is fast.

                       

                      Lastly... Since you are most likely using SQL, please visit the following link to ensure your SQL is properly configured to handle the load.  This article did make some improvement related to the reporting side for me at a customer site (where customer installed FP on their own) KnowledgeArticle - BMC

                       

                      Personally (i.e. my own opinion...)
                      My first choice (if the customer has the skill set) is to opt for Linux based OS installs as the disk IO scheduler can be changed to better suit web-based applications. My second choice is to rather use Postgres as a DB rather than SQL.

                       

                      A sideline chat will be great related those who tried to install FP on Windows 2016. So far Win2016 shows to me it is a slower OS...

                       

                      Below is my 'experimental' config of my Linux server Tomcat server.xml.  Please note that it may be your answer or it can make your system worse... :-) But it will give you an idea of where to start your trial and error test tweaking. THE MOST IMPORTANT part of this is how you can get Tomcat monitoring to work to see how Tomcat is behaving.  I installed the full JAVA JDK on my Windows workstation.  In this java JDK install is what  they call jmc.exe (Java Mission Control) The config to get JMC.exe to work is also in my config below.

                      <use at own risk.... This is hours/days of research made for my environment...>

                       

                       

                      server.xml extract

                      ______________________

                      <Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"

                      connectionTimeout="10000"

                      redirectPort="8443"

                      URIEncoding="UTF-8"

                      maxThreads="550"

                      keepAliveTimeout="5000"

                      maxKeepAliveRequests="1000"

                      acceptCount="10"

                      MaxConnections="8192"

                      minSpareThreads="80"

                      maxHttpHeaderSize="8192"

                      compression="on"

                      compressionMinSize="1024"

                      noCompressionUserAgents="gozilla, traviata"

                      compressableMimeType="text/html,text/xml,text/plain,text/css, text/javascript, application/x-javascript, application/javascript, application/xml, application/extjson, x-font/otf, application/x-font-woff, x-font/ttf, x-font/eot"

                      />

                       

                      And look for...

                      _____________________

                      <Connector port="8009" protocol="AJP/1.3" acceptorThreadCount="2" acceptCount="100" maxThreads="550" packetSize="65536" maxConnections="8192" minSpareThreads="80" keepAliveTimeout="5000" tomcatAuthentication="false" URIEncoding="UTF-8" />

                      _____________________

                       

                      Since I use a Linux based install, my Java Options is saved in tomcat.conf and not in the Java app as you see in Windows. But in essence, this is my Java Options.  Note the BOLD Highlighted sections. This will allows tomcat to give you access to monitor remotely. (

                      CATALINA_OPTS="-Xms3072m -Xmx4096m -XX:+UseParallelGC -XX:MaxPermSize=512m -XX:+DisableExplicitGC -XX:CompileThreshold=8000 -Djava.awt.headless=true -Xmn2g -XX:+UseNUMA -XX:+UseCompressedOops -XX:+AggressiveOpts -XX:+UseBiasedLocking -Djava.security.egd=file:/dev/./urandom -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5001 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=YOURSERVERIPHERE"

                       

                      [Please note that some of the above settings may cause Tomcat to not start in Windows Environments.  Simply add an option one at a time and stop and start Tomcat... Trial and error...]

                       

                      in Tomcat-Users.xml add/change the following like to ensure the remote monitoring is enabled.

                      <user username="fpadmin" password="MyPasswordHere" roles="admin-gui,manager-gui,manager-jmx" />

                       

                      To use JMC to monitor, just add a connection to IPADDRESS:5001  {5001 is in the java options above, so you can change this port} and using fpadmin user as mentioned in the previous paragraph.

                       

                      1 of 1 people found this helpful
                      • 8. Re: Footprints 2018 R2 still getting slowness issue!
                        abdallah hamaydeh

                        Hi Jean AD,

                         

                        -> Configure the Tomcat with Java:

                        C:\Program Files\Apache Software Foundation\Tomcat 7.0\conf   ---- server ---

                            -->

                            <Connector port="8080" protocol="HTTP/1.1"

                                       connectionTimeout="20000"

                                       redirectPort="8443" URIEncoding="UTF-8" maxThreads="550"/>

                            <!-- A "Connector" using the shared thread pool-->Connector connectionTimeout="20000" port="8080

                            <!--

                            <Connector executor="tomcatThreadPool"

                                       port="8080" protocol="HTTP/1.1"

                                       connectionTimeout="20000"

                                       redirectPort="8443" />

                            -->

                          -----------------------------------------------------------------------

                        C:\Program Files\Apache Software Foundation\Tomcat 7.0\bin    ---- Tomcat7w/Jav ---

                        For FP12.1.XX  “  -Xms= 4096m,  -Xmx= 8192m”

                        -Dfile.encoding=UTF-8

                        -XX:+UseParallelGC

                        • 9. Re: Footprints 2018 R2 still getting slowness issue!
                          Steve van der Merwe

                          Hi Abdallah

                           

                          Yes, with this default Tomcat Config, you will not get far...

                           

                          "<Connector port="8080" protocol="HTTP/1.1" 

                          This is using slow but stable Tomcat.

                           

                          The APR Protocol is a better/faster protocol compared to HTTP/1.1  {actually this setting should autodetect APR from what I have read...). After tomcat stop and start with this config, please verify your speed difference.

                           

                          I am assuming you installed Tomcat 7 Service Installer. if so the APR component should be loaded.  Try the following configuration and stop and start Tomcat.

                           

                           

                          <Connector port="8080" protocol="org.apache.coyote.http11.Http11AprProtocol"

                                         connectionTimeout="10000"

                                         redirectPort="8443"

                                         URIEncoding="UTF-8"

                                         maxThreads="550"

                                         maxHttpHeaderSize="8192"

                                         compression="on"

                                         compressionMinSize="2048"

                                         noCompressionUserAgents="gozilla, traviata" compressableMimeType="text/html,text/xml,text/plain,text/css, text/javascript, application/x-javascript, application/javascript, application/xml, application/extjson, x-font/otf, application/x-font-woff, x-font/ttf, x-font/eot"

                          />

                           

                           

                          Notes. 

                          ConnectionTimeout, reduce it from 20 000 to 10 000

                          Added maxHttpHeaderSize

                          Enabled compression so that the network usage between server and browser is minimised.  With the amount of CPUs you have, this should be fast to compress and send.

                           

                          Do only this change and report back on your performance finding.

                          • 10. Re: Footprints 2018 R2 still getting slowness issue!
                            Jean AD

                            abdallah hamaydeh

                            1. You need to apply Steve's recommendation! this is what I did for many customers and it enhance performances.

                            2. You have 128 Go and you just use 8 Go for Java in your Tomcat configuration? 8 Go is the minimum required!

                            • 11. Re: Footprints 2018 R2 still getting slowness issue!
                              abdallah hamaydeh

                              It gets worse, after using <Connector port="8080" protocol="org.apache.coyote.http11.Http11AprProtocol" .

                               

                              I have recorded video after modifying the Connector port, but I did restore the original server.xml file back because of the charts takes before almost 30-40 sec.

                               

                              FP2.webm - Google Drive

                               

                              Using the original <Connector port="8080" protocol="HTTP/1.1" :

                               

                              FP.webm - Google Drive

                              • 12. Re: Footprints 2018 R2 still getting slowness issue!
                                Steve van der Merwe

                                Hi Abdallah

                                 

                                Which theme are you using?  Did you change the CSS to create your own custom theme?

                                 

                                Also, let us ignore the reports section for now.  When you edit/create tickets using the config I recommended earlier... Are Footprints slower, the same or better?

                                 

                                The reason I want to forget at this stage about reporting is that it actually runs separately (or concurrently with...) from Footprints. The changes I recommend here is in general to the benefit of the Footprints application.  {Footprints runs in xxxx.html pages where reporting runs in RDPAGE.ASPX pages. I.e. 2 Apps within one Web App...}

                                 

                                Therefore, we need to know from you now if Footprints is OK in terms of performance. If Footprints is OK in terms of performance, then to fix Reporting, I will advise you visit your SQL Server as per BMC KB article I sent with the previous post. It could be that getting the data from SQL could be the issue. We may need to play around with the number of threads that Footprints spawn (Java Mission control will tell us a lot here...)

                                 

                                When I had performance issues with a customer on the reporting side, this article to change SQL (and do some SQL maintenance) helped quite a bit.  But also this customer was running in Virtualisation on an overloaded Hyper-V Cluster.  When they moved FP from a slow 1.8GHZ vCPU cluster node to 2.8GHz vCPU in another newly built cluster, their performance to view a report went from 8.5 minutes to 45 seconds (note, the FP and SQL VMs did not change. Just moving the VM to new cluster solved performance)

                                 

                                If SQL, Footprints, and Windows are on the same system using same Disks (I see a bit of BCM here... Also BCM Master installed?) Then you will see the Disk IO being the major culprit for your performance issues.  Your hardware design plays a big role.

                                 

                                On this system, due to its memory availability and number CPUs, I personally would have virtualized FP as a VM and having SQL as a separate VM on the same Host hypervisor.  But then I would have looked at the Disk Array as well to have it separated/Partitioned for IO performance.

                                 

                                As per comment from Jean, it is best not to give too much memory to Tomcat/Java. You can try to set minimum and max Memory to 8GB.  This will prevent proper garbage collection, but at least when Tomcat started (also how long does a start-up take?) you can do quick tests knowing that in the next day you will be seeing performance issues due to no garbage collection.  {i.e. plan to restart tomcat in 24 hours...}

                                • 13. Re: Footprints 2018 R2 still getting slowness issue!
                                  abdallah hamaydeh

                                  Hi Steve van der Merwe,

                                   

                                  Which theme are you using?  Did you change the CSS to create your own custom theme?

                                   

                                  • I did change a little bit to be presented as the Arabic language "left to right", and modifying CSS colors as customer prefer "Customer Standards", there is no major change on the theme, because I am using the admin theme that BMC standard with the same results.

                                   

                                  When you edit/create tickets using the config I recommended earlier... Are Footprints slower, the same or better?

                                   

                                  • footprints get slower when I did use as you mentioned "Thankfully"

                                   

                                  <Connector port="8080" protocol="org.apache.coyote.http11.Http11AprProtocol"

                                                 connectionTimeout="10000"

                                                 redirectPort="8443"

                                                 URIEncoding="UTF-8"

                                                 maxThreads="550"

                                                 maxHttpHeaderSize="8192"

                                                 compression="on"

                                                 compressionMinSize="2048"

                                                 noCompressionUserAgents="gozilla, traviata" compressableMimeType="text/html,text/xml,text/plain,text/css, text/javascript, application/x-javascript, application/javascript, application/xml, application/extjson, x-font/otf, application/x-font-woff, x-font/ttf, x-font/eot"

                                  />

                                   

                                  • As mentioned in video FP2, it took around 4 min, but we I used the default

                                   

                                      -->

                                      <Connector port="8080" protocol="HTTP/1.1"

                                                 connectionTimeout="20000"

                                                 redirectPort="8443" URIEncoding="UTF-8" maxThreads="550"/>

                                      <!-- A "Connector" using the shared thread pool-->Connector connectionTimeout="20000" port="8080

                                      <!--

                                      <Connector executor="tomcatThreadPool"

                                                 port="8080" protocol="HTTP/1.1"

                                                 connectionTimeout="20000"

                                                 redirectPort="8443" />

                                      -->

                                   

                                  • it took around 30-40 sec as mentioned in video FP.
                                  • 14. Re: Footprints 2018 R2 still getting slowness issue!
                                    Steve van der Merwe

                                    Hi

                                     

                                    The only thing I can think of is playing with Java Options here to get Java VM to perform better.

                                     

                                    Since you are on dedicated hardware, some option configuration such as the NUMA setting might assist.  The big thing, for now, is not to give too much memory to the system as you might be hiding the problem if too much memory is assigned.  Fine tuning the options will allow you to see CPU usage reducing.  This is actually good as it shows more of the app is in memory and thus using fewer hardware resources like disk IO.

                                     

                                    Adding the following options will allow you to use the Java Mission Control the ability to look at what is happening inside Tomcat and then you can make your decision.
                                    -Dcom.sun.management.jmxremote

                                    -Dcom.sun.management.jmxremote.port=5001

                                    -Dcom.sun.management.jmxremote.local.only=false

                                    -Dcom.sun.management.jmxremote.authenticate=false

                                    -Dcom.sun.management.jmxremote.ssl=false

                                    -Djava.rmi.server.hostname=YOURSERVERIPHERE

                                     

                                    Port 5001 will be the port you will tell JMC.exe to connect to.

                                     

                                    This is what I will recommend at this stage so that you can have a real-time view of what is really happening.

                                    1 2 3 Previous Next