8 Replies Latest reply on Sep 7, 2014 3:42 AM by Koduri Mukesh

    Remedy was not responding

    Jayesh Panchal
      Share This:

      Hi All,

       

      suddenly remedy got hanged and was not responding . users were not able to login.

       

      I tried login through user tool but found lots of delay in response.

       

      i gone through catalina logs and found below error for that exact duration and these errors popped out continuously.

       

      -----------------------------------------------------------------------

       

      12 Apr 2013 15:15:36 - Connects to remappprd:2500 through com.bmc.arsys.api.ProxyJRpc@65693291

      ERROR (94): Timeout during database query -- consider using more specific search criteria to narrow the results, and retry the operation; ONC/RPC call timed out

              at com.bmc.arsys.apitransport.ApiProxyJRpcBase.convertException(Unknown Source)

              at com.bmc.arsys.apitransport.ApiProxyJRpcBase.a(Unknown Source)

       

       

      Service: exception calling executeservice12 Apr 2013 15:28:59 - Connects to remappprd:2500 through com.bmc.arsys.api.ProxyJRpc@371070f9

       

       

      12 Apr 2013 15:38:50 - ProcNumber 0 via com.bmc.arsys.api.ProxyJRpc@60e77bc2 for remappprd:2500, converts OncRpcException to

      ARError 91 org.acplt.oncrpc.OncRpcTimeoutException: ONC/RPC call timed out

       

      --------------------------------------------------------------------------

       

      Can anybody point out the area in which we need to dig out?

       

      Regards,

      Jayesh

        • 1. Re: Remedy was not responding
          Srinivas Merugu

          This error usually occurs when there is a resourse crunch, high cpu utilization, bulk activities performed at db end, congestion in the network.

          • 2. Re: Remedy was not responding
            Jayesh Panchal

            Hi Srini ,

             

            In App server I got below error message :

            -------------------------------------------------

            Fri Apr 12 15:17:02 2013  BRIE : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (remedy-ebu-app1)  ARERR - 92

             

             

            Fri Apr 12 15:20:25 2013  Approve : Timeout during database query -- consider using more specific search criteria to narrow the results, and retry the operation (ARERR 94)

             

             

            Fri Apr 12 15:21:38 2013  Dispatch : Timeout during database query -- consider using more specific search criteria to narrow the results, and retry the operation (remedy-ebu-app1)  ARERR - 94

             

             

            Fri Apr 12 15:26:00 2013  SLMCS : Timeout during data retrieval due to busy server -- retry the operation (remedy-ebu-app1)  ARERR - 93

            -------------------------------------------------

             

            and in Mid-Tier logs I got below errors continuosuly:

             

            ---------------------------------

            12 Apr 2013 15:46:51 - Connects to remappprd:2500 through Java Rpc failed with: ERROR (91): RPC call failed; ONC/RPC call timed out

             

            12 Apr 2013 15:38:50 - ProcNumber 0 via com.bmc.arsys.api.ProxyJRpc@60e77bc2 for remappprd:2500, converts OncRpcException to

            ARError 91 org.acplt.oncrpc.OncRpcTimeoutException: ONC/RPC call timed out

             

            ----------------------------------

             

            Are both the errors are co-related???

             

            I saw CPU utlilization of DB and App servers and it was normal as usual. There is no activity performed in DB end.

             

            Dont know about Network.

             

            But after app server restart it got resolved as a last solution. Still not found RCA

             

             

            Regards,

            Jayesh


            • 3. Re: Remedy was not responding
              Srinivas Merugu

              It seems some of the processes where taking a huge amount of memory

              but I am not sure about this.

               

              You can check with your db team if they had performed any backup

              activity or anything is done at db level.

               

              Were you performing any admin activity in prod cache mode?

               

              Ask your db admins to give u a weekly performance graph from db end.

               

              Also check was there any glitch in the network?

              • 4. Re: Remedy was not responding
                Kapil Banwari

                Couple of things to check:-

                 

                -         There is a possibility that some db locking is taking place. So check from DBA if they are seeing any locks at the db level.

                 

                -         Any replication going on at the db level?

                 

                -         Any admin operations getting performed with cache mode set to 1, that way it will lock the memory until the admin operations are completed, so locks can be seen. If this is production, it should always be set to Cache mode as 0.

                 

                -         Further to this sql/api logs can be checked/analyzed to understand if there are usual long running api’s .

                 

                Hope this helps..

                Thank you.

                 

                Regards,

                Kapil Banwari

                • 5. Re: Remedy was not responding

                  Hi All,

                   

                  We have upgraded remedy from ARS 7.1.11 t0 7.6.04 sp5. And start getting the same error in catalina.out log file while creating entry in Remedy from Web Service.

                   

                  Can any one have any idea:

                   

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

                          at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

                          at java.lang.Thread.run(Unknown Source)

                  22 Aug 2014 03:20:53 - For server gemsrm-app.corp.ncr.com:32825 Exception in Proxy retrival. Release com.bmc.arsys.api.ProxyJRpc@66cd5789

                  ERROR (91): RPC call failed; ONC/RPC call timed out

                          at com.bmc.arsys.apitransport.ApiProxyJRpcBase.convertException(Unknown Source)

                          at com.bmc.arsys.api.ProxyJRpc.getRpcClient(Unknown Source)

                          at com.bmc.arsys.api.ProxyJRpc.ARSetSessionConfiguration(Unknown Source)

                          at com.bmc.arsys.apitransport.ApiProxyBase.setProxyProperties(Unknown Source)

                          at com.bmc.arsys.apitransport.connection.c.getProxy(Unknown Source)

                          at com.bmc.arsys.api.PoolingProxyManager.getProxy(Unknown Source)

                          at com.bmc.arsys.apitransport.connection.c.getProxy(Unknown Source)

                          at com.bmc.arsys.api.ARServerUser.usePrivateRpcQueue(Unknown Source)

                          at com.remedy.arsys.stubs.ServerLogin.<init>(Unknown Source)

                          at com.remedy.arsys.stubs.ServerLogin.get(Unknown Source)

                          at com.remedy.arsys.stubs.ServerLogin.getAdmin(Unknown Source)

                          at com.remedy.arsys.stubs.ServerLogin.getAdmin(Unknown Source)

                   

                  BR//

                  Sourabh Jhunjhunwala

                  • 6. Re: Remedy was not responding

                    Adding to the below error:

                     

                    22 Aug 2014 03:20:53 - ProcNumber 0 via com.bmc.arsys.api.ProxyJRpc@66cd5789 for gemsrm-app.corp.ncr.com:32825, converts OncRpcException to ARError 91

                    org.acplt.oncrpc.OncRpcTimeoutException: ONC/RPC call timed out

                     

                     

                    I am not getting what this OncRpcException is and why it's converting to ARError 91. Can any one explain why probably We are getting below error.

                    • 7. Re: Remedy was not responding
                      Jayesh Panchal

                      Hi Sourabh,

                       

                      As remedy is multi threading system. Each API call will be dispatched to specific RPC queue. If that RPC queue/thread is not responding within the defined API timeout you will get this error as per my experience.

                       

                      Even I got this error once when I restarted the ar server and whole RPC thread section was gone out from ar.conf.

                      • 8. Re: Remedy was not responding

                        Hi All,

                        i am also getting the same ONC/RPC call timed out frequently, at that time will flush the mid-tier cache and will work for a while. I know flushing the mid-tier cache and restarting the tomcat is the right way. But can anyone suggest me the permanent solutions