3 Replies Latest reply: Aug 28, 2015 1:36 PM by dc NameToUpdate RSS

    Intermittent performance issue in Remedy

    S R

      Hi Team,

       

      We have faced intermittent performance issue in Remedy. Users were unable to save  incidents and SLAs were missed.

      Our Mid-Tier is installed on the separate server with Load Balancer setup.

      Server group configuration: We have 2 Mid-tiers and 1 Web load balancer and 3 application server (1 admin server)

      Please find the attached environment details and the arerror.log snippet from during the issue concurrence.



      ============

      (SQL Server 11)

      Wed Jul 10 04:20:44 2013  390620 : The SQL database operation failed. (ARERR 552)

      Wed Jul 10 04:20:44 2013     [DBNETLIB][ConnectionWrite (send()).]General network error. Check your network documentation. (SQL Server 11)

      Wed Jul 10 04:20:44 2013  390620 : SQL database is now available (ARNOTE 592)

      ============

       

      Our environment details are given below:

       

      ARS server version and patch  :  7.6.04 SP4

      ITSM version and patch  :  7.6.04 SP4

      Mid-Tier version and patch  :  7.6.04 SP4

      Mid-Tier Server OS level and patch  :  Windows Server 2008 R2

      Web server version  :  Apache Tomcat/6.0.20

      Servlet Engine version  :  1.6.0_38

      Java version and build level  :  1.6.0_38

      Database structure  :  SQL 2008 Ent (A two node cluster for database)


       

       

      Server

      OS

      RAM

      HDD

      Database

      Application Server (AR System Server)

      2 Server x Windows 2008 R2 (64-bit), 8vCPU

      32 GB

      150 GB

      NA

      Web Server (Mid-Tier)

      2 Server x Windows 2008 R2 (64-bit), 8vCPU

      32 GB

      150 GB

      NA

      Database Server

      2 node Windows 2008 R2 (64-bit) cluster, 2x4 Core

      64 GB

      Tier 1 SAN storage, 1 TB

      SQL 2008 Ent (A two node cluster for database)

       

       

      I have attached the arerror.log. Kindly assist me to resolve this issue as soon as possible and let me know for more queries.

       

      Regards

      Sreekanth

        • 1. Re: Intermittent performance issue in Remedy
          Preshit Doshi

          Hi,

           

          I have checked arerror.log and found that there is ARERR 552 is occuring continuously.

          I will suggest to check your Network setting and Firewall configuration to check why Database connection is getting lost.

          Also please provide Server side SQL and Filter Logs to Investigate more on your issue.

           

          Please be sure to mark Correct or Helpful answer if you find my inputs useful.

           

          Regards,

          Preshit Doshi

          • 2. Re: Intermittent performance issue in Remedy
            Carl Wilson

            Hi,

            the following error is thrown from the OS and recorded in the arerror.log - this is due to connectivity issues between AR > DB:

             

            [DBNETLIB][ConnectionWrite (send()).]General network error. Check your network documentation. (SQL Server 11)

             

            You should check out the following article if you are using Virtual Machines across your environment:

             

            Using Network trace to troubleshoot intermittent connectivity issues

             

            The change will need to be performed on the AR Servers.

             

            Cheers

            Carl

             

            http://www.missingpiecessoftware.com/

            • 3. Re: Intermittent performance issue in Remedy
              dc NameToUpdate

              Might be a little late to post on this..

               

               

              I had a similar issue. I have three AR Servers on a Database.

              I first targeted on SQL error "SLQ Server 11" that seemed to be the cause, but it wasnt directly related I realized later.

              Our AR servers used to log "SLQ Server 11" error once every hour in the arerror.log and the servers used to hang once in 2-3 days. Sometimes more often and sometimes not.

               

               

              After observing the trend couple of times and I started looking into the "Insufficient Memory" error and contacted BMC. They helped. We had Insufficient Memory error followed by SQL Server 11 errors.

              They observed that the servers were eating up 85% of the available 12G memory all the time and AR processes were merely taking 1.8G out of that. The rest of the memory was taken up by some back ground processed that I could not figure out.

               

               

              I brought this to my windows server admin's notice. They said that "the VM's(WindowsServer2008R2SP1) resources are restricted at the VM". I am not sure what that meant but they did something from the Task Manager, I believe (may have killed some processes and disabled them which were not required and causing problems). But soon after they did that, the servers' memory soncumption went down from 85 to 20%. Its been 2 days now, the servers are stable and the DB error(which used to log every hour) is gone!

               

               

              There is also a microsoft defect about  WindoesServer2008R2SP1 memory leak. Our windows Admin might applied the a fix around that but I am not sure.

              https://social.technet.microsoft.com/Forums/windowsserver/en-US/a5699bf3-f79a-4bf2-b14d-dedccd9742e8/memory-usage-on-2008-r2-file-server-is-high-and-doesnt-seem-to-be-accounted-for?forum=winserverfiles

               

               

               

               

              hope this helps!