11 Replies Latest reply on Jun 8, 2016 3:23 PM by Bill Robinson

    App server cant connect to DB.

    Neal Meagher

      Of of my App servers all of a sudden cannot connect to DB. I get error below


      Getting the following error:

      java.sql.SQLException: invalid database address: jdbc:sqlserver://yke0-q12gp0103.saifg.rbc.com:41203


      I can ping both host name and IP. Telnet over port works. I use sql client to connect with username and password. Those work fine. No Iptables



      this is in console app server and app server launcher logs:


      [15 May 2016 08:12:45,553] [Scheduled-System-Tasks-Thread-6] [ERROR] [System:System:] [App Server Heartbeat] The query has timed out.

      com.microsoft.sqlserver.jdbc.SQLServerException: The query has timed out.

        at com.microsoft.sqlserver.jdbc.TDSCommand.checkForInterrupt(IOBuffer.java:5918)

        at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:70)

        at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1510)

        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:404)

        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:350)

        at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:5696)

        at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1715)

        at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:180)

        at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:155)

        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:314)

        at com.bladelogic.om.infra.shared.db.BlPreparedStatement.executeUpdate(BlPreparedStatement.java:63)

        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)

        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)

        at com.bladelogic.om.infra.app.service.db.DefaultDatabaseServiceImpl.execute(DefaultDatabaseServiceImpl.java:1078)

        at com.bladelogic.om.infra.app.service.db.DefaultDatabaseServiceImpl.execute(DefaultDatabaseServiceImpl.java:714)

        at com.bladelogic.om.infra.app.service.db.DefaultDatabaseServiceImpl.execute(DefaultDatabaseServiceImpl.java:693)

        at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl$2.doDBOperation(DbAppServerGroupCoordinatorImpl.java:241)

        at com.bladelogic.om.infra.app.db.TransactionalOperation.manageTransaction(TransactionalOperation.java:187)

        at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:136)

        at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.updateAppServerKeepAliveStatus(DbAppServerGroupCoordinatorImpl.java:256)

        at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.access$000(DbAppServerGroupCoordinatorImpl.java:29)

        at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl$1.execute(DbAppServerGroupCoordinatorImpl.java:168)

        at com.bladelogic.om.infra.app.service.timer.ScheduledTaskReference.run(ScheduledTaskReference.java:95)

        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

        at java.util.concurrent.FutureTask.runAndReset(Unknown Source)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

        at java.lang.Thread.run(Unknown Source)




        • 1. Re: App server cant connect to DB.
          Cody Dean

          Have you tried opening the blappconf tool and setting the DB connection there?

          • 2. Re: App server cant connect to DB.
            Bill Robinson

            so what exactly happened here?  the appserver was just running along and then you saw this:

            [15 May 2016 08:12:45,553] [Scheduled-System-Tasks-Thread-6] [ERROR] [System:System:] [App Server Heartbeat] The query has timed out.

            com.microsoft.sqlserver.jdbc.SQLServerException: The query has timed out.

            and then what?


            you stopped the appserver service and started it and what happened?  can you *attach* the appserver log showing that ?

            • 3. Re: App server cant connect to DB.
              Neal Meagher

              It was restarted during troubleshooting for RCP slowness. Now two app servers are doing the same thing.


              I get this error among others



              "Could not load sever profile "default" into Database


              [*** Edited post to copy long log paste into an attached file ***]

              • 4. Re: App server cant connect to DB.
                Bill Robinson

                instead of pasting in a huge log, please use the 'advanced editor' and *attach* the log file to the post.


                so, this is happening when?  on appserver start ?


                can you do this:

                stop the appserver service

                move the NSH/br/stdlib/sqlite-jdbc-3.8.6.jar out of the NSH/br/stdlib directory (keep it, we'll move it back later).

                start the appserver

                *ATTACH* the appserver log

                move the jar back into place.


                also - you still haven't explained:

                - what version of bsa is this?

                - what sequence of events led up to this problem ?

                - these appservers were working and just started having this issue ?


                is there a ticket open w/ support for this, if so, what # ?

                • 5. Re: App server cant connect to DB.
                  Neal Meagher

                  Here is the app server log with jar file moved. There was no change.  reboot took place with the first app server and now it wont connect to db. Now experience slowness on RCP client and reboot another and same issue occurred.


                  TICKET# 00148398

                  • 6. Re: App server cant connect to DB.
                    John O'Toole

                    Some additional information on this.


                    Its actually more a performance issue than a pure connectivity issue.


                    When we moved the sqllite jar file out of the way (it blocks the more useful error from SQL Server with the vague "Invalid Database Address" error, the real error is:


                    com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host <host_name>, port <port> has failed. Error: "connect timed out. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".


                    blasadmin commands which touch the DB (show database all and show file all) return the same error.


                    However, if you run the same commands repeatedly, occasionally they work. Most of the time they sit for a few minutes before displaying the error. Sometimes they also sit for a long time before returning the expected output. Really seems like a network performance issue where sometimes it sneaks in under the connection timeout and other times it does not. Even when it works though, it is too slow.


                    At one point, it was just 1 QA appserver out of 7 so it seemed like it was specific to that server.


                    Now it has started happening on others which suggests it is more likely network related. Not an issue in PROD at all.


                    Ping, Telnet to the DB server and port work ok. So, again, its not really that they cannot connect and more a case that JDBC connections to the DB server are very slow and usually timeout.


                    We downloaded and installed SQLLine 1.0.2 (free jdbc test tool) but that also could not connect even from an appserver which was working ok at the time. So that may be an unrelated issue.

                    • 7. Re: App server cant connect to DB.
                      Bill Robinson

                      where did you move the jar file?


                      because if you did we wouldn't be seeing this in the stack trace:

                      Caused by: java.sql.SQLException: invalid database address: jdbc:sqlserver://yke0-q12gp0103.saifg.rbc.com:41203

                          at org.sqlite.JDBC.createConnection(JDBC.java:111)

                          at org.sqlite.JDBC.connect(JDBC.java:88)

                          at java.sql.DriverManager.getConnection(Unknown Source)

                          at java.sql.DriverManager.getConnection(Unknown Source)



                      as john mentioned, if you remove the sqlite jar from the stdlib dir -eg move it into /tmp you should see a more detailed message that indicates the actual problem.


                      but it seems like john has already explained that to you in the ticket.


                      so can you confirm what john has said ?

                      • 8. Re: App server cant connect to DB.
                        Neal Meagher

                        Jar file has been moved to tmp. We had this problem before

                        • 9. Re: App server cant connect to DB.
                          Bill Robinson

                          The problem that john described above ?

                          • 10. Re: App server cant connect to DB.
                            Neal Meagher

                            When we move the jar file we still see that generic error in the Logs

                            • 11. Re: App server cant connect to DB.
                              Bill Robinson

                              ?so what john said above doesn't apply to your env ?