1 2 Previous Next 15 Replies Latest reply on Mar 21, 2019 11:55 AM by Marie Johnson

    Smart Reporting issue

    Majd Alsadi
      Share This:

      Hello guys,

       

      I have an issue with Smart reporting, i have a numbers of dashboards sometime it's working fine and some times it keeps loading and this error appear:

       

      I can't figure out what is the problem, Anyone help me??

       

      Thank you very much

       

      An error occurred retrieving the results from the database.

      The error message was: "ERROR (90): Cannot establish a network connection to the AR System server; Connection timed out: connect ******".

      Please check the SQL syntax and try again.

       

      Kindly find the below screen shots:

       

      2.PNG

       

      3.PNG

        • 1. Re: Smart Reporting issue
          Kevin Verlaine

          This error can be for the link to ARS or for Database serveur.

          try to restart ARS service on the good order.

          if is the database: the ARS can't join it and you can't restart ARS.

           

          if is the database. try to restart it.

          • 2. Re: Smart Reporting issue
            Brian Morris

            Are your AR servers and Databases on the same network, and no more than 1-2 hops away? This is just the general timeout error you'd get due to the systems just being down or the services unavailable when you tried a transaction against it.

            2 of 2 people found this helpful
            • 3. Re: Smart Reporting issue
              Marie Johnson

              I agree with Brian Morris

               

              i’d just like to add that you may want to check with your network and/or security team.  The reason is that port 2020 is typically found to be what trojans come in on so most networks block or isolate it.  You should work with this team to register a port that can be reserved for your tool.

               

              in large enterprise companies, they may manage the services file (found in etc on all OSs) which lists all registered and reserved ports in the company.  This file is typically deployed and updated through automation on all machines (even workstations).  The reason for stating this is: another tool may already be using this periodically which could be why you see it sometimes and not all the time or a company could block all ports not registered in service.   As an example port 2020 used to be the default for IBM Tivoli Provision Manager, thus it only runs periodically.

              2 of 2 people found this helpful
              • 4. Re: Smart Reporting issue
                Brian Morris

                Marie makes a good point, I didn't think of the port being seen as a potential risk by security. My normal troubleshooting process with these kinds of cases is using ping to make sure we can hit the IP, tracerout to find the hops between systems and see if there is latency or if the connection just isn't making it between systems, the maybe telnet to that port (2020) and see if it's blocked or if the connection is good. If you can get onto the system, (and it's Windows) you can see what process ID is running on that port with netstat -ano | findstr  2020. 

                 

                 

                1 of 1 people found this helpful
                • 5. Re: Smart Reporting issue
                  Majd Alsadi

                  Yes the ARs servers and Database on the same network.

                   

                  what do you mean 1-2 hops away??

                   

                  Thank you very much

                  • 6. Re: Smart Reporting issue
                    Brian Morris

                    This site provides an explaination of traceroute if you're not familiar.

                     

                    What is TraceRoute and How to Use it for Hop-by-Hop Analysis

                     

                    Basically, your DB and application server should be logically as close as possible.

                    • 7. Re: Smart Reporting issue
                      Majd Alsadi

                      I did the following :

                       

                      1) I ping the AR load balancer from smart reporting server and it's working fine

                      2) when i telnet with AR loadbalancer with the port sometimes it's working and sometimes they try to connect but at the end the message said "Could not open connection to the host ** port ** , Connect failed"

                      3)for the traceroute i did it and it bring 1 record for the AR loadbalancer...

                       

                      Can anyone suggest me...

                       

                      Thank you again...

                      • 8. Re: Smart Reporting issue
                        Ganesh Gore

                        If this is happening with particular dashboard/reports and if other reports are working fine then It looks like the issue with SQL query which trying to extract the data. if there are many columns and number of WHERE clauses in the sql statement, It takes more time to excute and returns timeout error. You need to analysis the sql statement generated by these reports. You can add some filters in the report if required.

                        • 9. Re: Smart Reporting issue
                          Brian Morris

                          So your port 2020 is connecting to a Loadbalancer? I suspect that there is a session problem that is causing the issue. The loadbalancer is not configured to keep your session alive, so sometimes it works and sometimes it doesn't. There is a sticky bit or a session persistence setting on your loadbalancer that probably needs set. Can you check with your network team on this?

                          • 10. Re: Smart Reporting issue
                            Majd Alsadi

                            It's happening with all dashboards and reports some times they working fine and some times it keeps loading and the error appear, and i already used filters in the reports and dashboards.

                             

                            and yes i'm connecting to Loadbalancer and i will check with our network team.

                             

                            Thank you very much...

                            • 11. Re: Smart Reporting issue
                              Majd Alsadi

                              The problem is not resolved tell now.

                               

                              I was check with the network team but we are unable to know the problem.

                               

                              Can anyone proivd another steps to troubleshoot ??

                               

                              Note: some times it give this error and is it the same or not? :

                               

                              Error running report

                               

                              A connection could not be established to the source database.

                               

                               

                              Thank you

                              • 12. Re: Smart Reporting issue
                                Rainer Volkmer

                                It's hard to solve this without seeing your environment.

                                Did you check with bmc-support ?

                                 

                                From my point of view, the problem is not in smartreporting, as it works sometimes. If something would be wrong in smartreporting it would never work.

                                Smartreporting connect to ARSystem over the normal JDBC-connection, using the ARS-API and this seems to work, otherwise nothing would work in smartreporting.

                                Could it be a capacity-problem ? For example not enough queues defined in ARS-server-config ? If all queues are used, you will get a timeout (ARERR 90) after a certain time.

                                Please open a ticket with bmc-support, they should be able to help you.

                                • 13. Re: Smart Reporting issue
                                  Marie Johnson

                                  Majd

                                   

                                  your issue has nothing to do with a BMC Software tool.  The fact that you cannot consistantly telnet to the port 2020 you are using for remedy, tells you there is a configuration issue within your company that is causing this issue.  This could be a few dozen things so you need to work with someone in networking at your company.

                                   

                                  to be prepared to work with them:

                                  open 6 command prompt windows with admin rights

                                  in window 1-4 run: 

                                  ping -t IP_ADDRESS-Remedy-server > c:\tmp\IPDirect.log (if Linux /tmp/IPDirect.log)

                                  ping -t IP_ADDRESS-load-balance > c:\tmp\IPLB.log (if Linux /tmp/IPLB.log)

                                  ping -t DNSNAME-Remedy-server.domain.name > c:\tmp\DNSDirect.log (if Linux /tmp/DNSDirect.log)

                                  ping -t DNSNAME-LOAD-balancer.domain.name > c:\tmp\DanSLB.log (if Linux /tmp/DNSLB.log)

                                   

                                  in the 5th window telnet to the remedy port using the DNS names for the load balancer and the remedy server.  When failures occur, note the time and look/view (don’t stop) at the DNS logs To see if there are errors at the same time. Take snippets of the failures and success So they can see it is intermitten.

                                   

                                  in the 6th window telnet to the remedy port using the IP address (ip v4) for the load balancer and remedy server.  Again, if failure occurs, note the time and look at the IP logs for errors.  If it errors, on the report server, go into the network settings turn off ipv6 and QoS, then do a ‘net use‘ and disconnect The two IPs, this will clear the connection from the cache on the report server, then try the telnet again.  Take snippets of the failures and successes so they can see it is intermitten.  FYI the net use command is like:  net use \\\\IPADDRESS /Delete (must be ran in admin mode)

                                   

                                  The logs should have have been running this entire time so you can now do a Ctrl+C on each window and send that to your network team.  You can also attach them here if you’d like, further help From the communities.  However you should send them to your network and VM admins.

                                   

                                  chances are if it is a DNS error then there maybe an issue with the primary or backup DNS server, for example maybe when it fails, you are using the secondary DNS and there isn’t an entry there.  If it is the IP failing then there could be several reasons it could even be the shared VM VIP, if you are using VMs; where maybe you are sharing with other apps using all the available bandwidth and there isn’t anything left for you to communicate over.  If both are failing (IP and DNS), then it could be many things.

                                   

                                  Im not sure that bmc support can help since this issue has nothing to do with the tools.

                                   

                                  one thing that you haven’t answered is where remedy, the Load balancer and the smart reporting tools are running.  Is something on a public cloud?  Is it On Demand? Are the internal all on the same server? Different servers? Different subnets? Different networks?  Are there firewalls?

                                  4 of 4 people found this helpful
                                  • 14. Re: Smart Reporting issue
                                    Majd Alsadi

                                    Thank you very much for the detailed information i will follow these steps with network team and i will update you if the issue is resolved.

                                     

                                    For your question  where the remedy and smart reporting running...on different servers on the same network.

                                     

                                    I asked the network team if there a firewall it maybe prevent the connection some times...they answer if the problem from the firewall it must prevent the connection all time.

                                    1 of 1 people found this helpful
                                    1 2 Previous Next