2 Replies Latest reply on Oct 1, 2015 2:34 PM by Craig Dockter

    Issue with Bladelogic App server

      Hi All,

       

      We are facing a issue where in the App servers looses communication with database with below errors, causing weird behaviour of APP server, we tried different suggestions suggested by BMC like. We use 8.2.04.531 App server version

       

      In App server,

       

      1. Increased Max Heap size to 8GB

      2. Increases servermonitorintervalcount to 180

       

      In Database.

       

      1. Increased PGA size.

      2. increased

      1. SQLNET.INBOUND_CONNECT_TIMEOUT
      2. SQLNET.INBOUND_CONNECT_TIMEOUT_<listener_name> parameter

       

      still we are facing the issue of App server losing connection with database.

       

      Error is :

       

                      ... 13 more

      [01 Oct 2015 00:33:54,432] [Scheduled-System-Tasks-Thread-5] [ERROR] [System:System:] [App Server Heartbeat] java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

      1. com.bladelogic.om.infra.mfw.util.BlException: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:145)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.updateAppServerKeepAliveStatus(DbAppServerGroupCoordinatorImpl.java:253)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.access$000(DbAppServerGroupCoordinatorImpl.java:28)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl$1.execute(DbAppServerGroupCoordinatorImpl.java:166)

                      at com.bladelogic.om.infra.app.service.timer.ScheduledTaskReference.run(ScheduledTaskReference.java:95)

                      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

                      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

                      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                      at java.lang.Thread.run(Thread.java:662)

      Caused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:389)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:382)

                      at oracle.jdbc.driver.T4C7Ocommoncall.processError(T4C7Ocommoncall.java:93)

                      at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)

                      at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)

                      at oracle.jdbc.driver.T4C7Ocommoncall.doOCOMMIT(T4C7Ocommoncall.java:75)

                      at oracle.jdbc.driver.T4CConnection.doCommit(T4CConnection.java:565)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3851)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3857)

                      at com.bladelogic.om.infra.shared.db.BlDbConnection.commit(BlDbConnection.java:83)

                      at org.apache.commons.dbcp.DelegatingConnection.commit(DelegatingConnection.java:334)

                      at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.commit(PoolingDataSource.java:211)

                      at com.bladelogic.om.infra.app.service.db.UserTransactionImpl.commit(UserTransactionImpl.java:137)

                      at com.bladelogic.om.infra.app.service.session.SessionImpl.commitUserTransaction(SessionImpl.java:352)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.manageTransaction(TransactionalOperation.java:175)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:136)

                      ... 13 more

      [01 Oct 2015 00:36:54,944] [WorkItem-Thread-54] [INFO] [BLAdmin:BLAdmins:] [ACLPush] Executing work item Acl Push Job:Enterprise Reporting - Update ACLs - Null and 0; Server:atlantagasvr;  on application server: xrxdallin005

      [01 Oct 2015 00:38:03,589] [Scheduled-System-Tasks-Thread-5] [ERROR] [System:System:] [App Server Heartbeat] java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

      1. com.bladelogic.om.infra.mfw.util.BlException: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:145)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.updateAppServerKeepAliveStatus(DbAppServerGroupCoordinatorImpl.java:253)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.access$000(DbAppServerGroupCoordinatorImpl.java:28)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl$1.execute(DbAppServerGroupCoordinatorImpl.java:166)

                      at com.bladelogic.om.infra.app.service.timer.ScheduledTaskReference.run(ScheduledTaskReference.java:95)

                      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

                      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

                      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                      at java.lang.Thread.run(Thread.java:662)

      Caused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:389)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:382)

                      at oracle.jdbc.driver.T4C7Ocommoncall.processError(T4C7Ocommoncall.java:93)

                      at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)

                      at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)

                      at oracle.jdbc.driver.T4C7Ocommoncall.doOCOMMIT(T4C7Ocommoncall.java:75)

                      at oracle.jdbc.driver.T4CConnection.doCommit(T4CConnection.java:565)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3851)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3857)

                      at com.bladelogic.om.infra.shared.db.BlDbConnection.commit(BlDbConnection.java:83)

                      at org.apache.commons.dbcp.DelegatingConnection.commit(DelegatingConnection.java:334)

                      at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.commit(PoolingDataSource.java:211)

                      at com.bladelogic.om.infra.app.service.db.UserTransactionImpl.commit(UserTransactionImpl.java:137)

                      at com.bladelogic.om.infra.app.service.session.SessionImpl.commitUserTransaction(SessionImpl.java:352)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.manageTransaction(TransactionalOperation.java:175)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:136)

                      ... 13 more

      [01 Oct 2015 00:38:03,591] [WorkItem-Thread-61] [INFO] [BLAdmin:BLAdmins:] [ACLPush] Executing work item Acl Push Job:Enterprise Reporting - Update ACLs - Null and 0; Server:uscntsqlsyn1pit;  on application server: xrxdallin005

      [01 Oct 2015 00:39:11,996] [Scheduled-System-Tasks-Thread-2] [ERROR] [BLAdmin:BLAdmins:] [Work Item Execution Timout] Work item for job (Acl Push Job:Enterprise Reporting - Update ACLs - Null and 0; Server:atlantagasvr; ) was cancelled. Reason: Work item execution timed out.

      [01 Oct 2015 00:39:12,001] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 170558840,Used JVM (B): 8253176456,VSize (B): 27485229056,RSS (B): 9370898432,Used File Descriptors: 348

      [01 Oct 2015 00:40:19,024] [Scheduled-System-Tasks-Thread-19] [ERROR] [BLAdmin:BLAdmins:] [Work Item Execution Timout] Work item for job (Acl Push Job:Enterprise Reporting - Update ACLs - Null and 0; Server:uscntsqlsyn1pit; ) was cancelled. Reason: Work item execution timed out.

      [01 Oct 2015 00:40:51,199] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 166376496,Used JVM (B): 8257358800,VSize (B): 27485229056,RSS (B): 9370853376,Used File Descriptors: 352

      [01 Oct 2015 00:40:51,207] [Scheduled-System-Tasks-Thread-5] [ERROR] [System:System:] [App Server Heartbeat] java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

      1. com.bladelogic.om.infra.mfw.util.BlException: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:145)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.updateAppServerKeepAliveStatus(DbAppServerGroupCoordinatorImpl.java:253)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.access$000(DbAppServerGroupCoordinatorImpl.java:28)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl$1.execute(DbAppServerGroupCoordinatorImpl.java:166)

                      at com.bladelogic.om.infra.app.service.timer.ScheduledTaskReference.run(ScheduledTaskReference.java:95)

                      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

                      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

                      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                      at java.lang.Thread.run(Thread.java:662)

      Caused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:389)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:382)

                      at oracle.jdbc.driver.T4C7Ocommoncall.processError(T4C7Ocommoncall.java:93)

                      at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)

                      at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)

                      at oracle.jdbc.driver.T4C7Ocommoncall.doOCOMMIT(T4C7Ocommoncall.java:75)

                      at oracle.jdbc.driver.T4CConnection.doCommit(T4CConnection.java:565)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3851)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3857)

                      at com.bladelogic.om.infra.shared.db.BlDbConnection.commit(BlDbConnection.java:83)

                      at org.apache.commons.dbcp.DelegatingConnection.commit(DelegatingConnection.java:334)

                      at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.commit(PoolingDataSource.java:211)

                      at com.bladelogic.om.infra.app.service.db.UserTransactionImpl.commit(UserTransactionImpl.java:137)

                      at com.bladelogic.om.infra.app.service.session.SessionImpl.commitUserTransaction(SessionImpl.java:352)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.manageTransaction(TransactionalOperation.java:175)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:136)

                      ... 13 more

      [01 Oct 2015 00:42:30,481] [Scheduled-System-Tasks-Thread-2] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 164512000,Used JVM (B): 8259223296,VSize (B): 27485229056,RSS (B): 9370943488,Used File Descriptors: 357

      [01 Oct 2015 00:43:37,685] [Scheduled-System-Tasks-Thread-5] [ERROR] [System:System:] [App Server Heartbeat] java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

      1. com.bladelogic.om.infra.mfw.util.BlException: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:145)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.updateAppServerKeepAliveStatus(DbAppServerGroupCoordinatorImpl.java:253)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.access$000(DbAppServerGroupCoordinatorImpl.java:28)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl$1.execute(DbAppServerGroupCoordinatorImpl.java:166)

                      at com.bladelogic.om.infra.app.service.timer.ScheduledTaskReference.run(ScheduledTaskReference.java:95)

                      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

                      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

                      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                      at java.lang.Thread.run(Thread.java:662)

      Caused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:389)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:382)

                      at oracle.jdbc.driver.T4C7Ocommoncall.processError(T4C7Ocommoncall.java:93)

                      at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)

                      at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)

                      at oracle.jdbc.driver.T4C7Ocommoncall.doOCOMMIT(T4C7Ocommoncall.java:75)

                      at oracle.jdbc.driver.T4CConnection.doCommit(T4CConnection.java:565)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3851)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3857)

                      at com.bladelogic.om.infra.shared.db.BlDbConnection.commit(BlDbConnection.java:83)

                      at org.apache.commons.dbcp.DelegatingConnection.commit(DelegatingConnection.java:334)

                      at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.commit(PoolingDataSource.java:211)

                      at com.bladelogic.om.infra.app.service.db.UserTransactionImpl.commit(UserTransactionImpl.java:137)

                      at com.bladelogic.om.infra.app.service.session.SessionImpl.commitUserTransaction(SessionImpl.java:352)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.manageTransaction(TransactionalOperation.java:175)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:136)

                      ... 13 more

      [01 Oct 2015 00:45:10,103] [Job-Execution-2] [INFO] [::] [] Created debug logger /app/bladelogic/NSH/tmp/debug/xrxdallin005/USP_TEST_Sep_30__2015_9_35_14_PM

      [01 Oct 2015 00:45:35,279] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 242272304,Used JVM (B): 8181462992,VSize (B): 27485229056,RSS (B): 9370828800,Used File Descriptors: 357

      [01 Oct 2015 00:45:35,282] [Scheduled-System-Tasks-Thread-5] [ERROR] [System:System:] [App Server Heartbeat] java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

      1. com.bladelogic.om.infra.mfw.util.BlException: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:145)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.updateAppServerKeepAliveStatus(DbAppServerGroupCoordinatorImpl.java:253)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.access$000(DbAppServerGroupCoordinatorImpl.java:28)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl$1.execute(DbAppServerGroupCoordinatorImpl.java:166)

                      at com.bladelogic.om.infra.app.service.timer.ScheduledTaskReference.run(ScheduledTaskReference.java:95)

                      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

                      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

                      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                      at java.lang.Thread.run(Thread.java:662)

      Caused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:389)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:382)

                      at oracle.jdbc.driver.T4C7Ocommoncall.processError(T4C7Ocommoncall.java:93)

                      at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)

                      at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)

                      at oracle.jdbc.driver.T4C7Ocommoncall.doOCOMMIT(T4C7Ocommoncall.java:75)

                      at oracle.jdbc.driver.T4CConnection.doCommit(T4CConnection.java:565)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3851)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3857)

                      at com.bladelogic.om.infra.shared.db.BlDbConnection.commit(BlDbConnection.java:83)

                      at org.apache.commons.dbcp.DelegatingConnection.commit(DelegatingConnection.java:334)

                      at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.commit(PoolingDataSource.java:211)

                      at com.bladelogic.om.infra.app.service.db.UserTransactionImpl.commit(UserTransactionImpl.java:137)

                      at com.bladelogic.om.infra.app.service.session.SessionImpl.commitUserTransaction(SessionImpl.java:352)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.manageTransaction(TransactionalOperation.java:175)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:136)

                      ... 13 more

      [01 Oct 2015 00:47:04,991] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 168983560,Used JVM (B): 8254751736,VSize (B): 27485229056,RSS (B): 9370836992,Used File Descriptors: 352

      [01 Oct 2015 00:47:04,995] [Scheduled-System-Tasks-Thread-5] [ERROR] [System:System:] [App Server Heartbeat] java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

      1. com.bladelogic.om.infra.mfw.util.BlException: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:145)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.updateAppServerKeepAliveStatus(DbAppServerGroupCoordinatorImpl.java:253)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.access$000(DbAppServerGroupCoordinatorImpl.java:28)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl$1.execute(DbAppServerGroupCoordinatorImpl.java:166)

                      at com.bladelogic.om.infra.app.service.timer.ScheduledTaskReference.run(ScheduledTaskReference.java:95)

                      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

                      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

                      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                      at java.lang.Thread.run(Thread.java:662)

      Caused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:389)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:382)

                      at oracle.jdbc.driver.T4C7Ocommoncall.processError(T4C7Ocommoncall.java:93)

                      at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)

                      at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)

                      at oracle.jdbc.driver.T4C7Ocommoncall.doOCOMMIT(T4C7Ocommoncall.java:75)

                      at oracle.jdbc.driver.T4CConnection.doCommit(T4CConnection.java:565)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3851)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3857)

                      at com.bladelogic.om.infra.shared.db.BlDbConnection.commit(BlDbConnection.java:83)

                      at org.apache.commons.dbcp.DelegatingConnection.commit(DelegatingConnection.java:334)

                      at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.commit(PoolingDataSource.java:211)

                      at com.bladelogic.om.infra.app.service.db.UserTransactionImpl.commit(UserTransactionImpl.java:137)

                      at com.bladelogic.om.infra.app.service.session.SessionImpl.commitUserTransaction(SessionImpl.java:352)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.manageTransaction(TransactionalOperation.java:175)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:136)

                      ... 13 more

      [01 Oct 2015 00:47:04,995] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 188052408,Used JVM (B): 8235682888,VSize (B): 27485229056,RSS (B): 9370845184,Used File Descriptors: 357

      [01 Oct 2015 00:48:12,147] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 172883168,Used JVM (B): 8250852128,VSize (B): 27485229056,RSS (B): 9370882048,Used File Descriptors: 348

      [01 Oct 2015 00:48:34,771] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 158269672,Used JVM (B): 8265465624,VSize (B): 27485229056,RSS (B): 9370882048,Used File Descriptors: 347

      [01 Oct 2015 00:48:34,772] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 263356952,Used JVM (B): 8160378344,VSize (B): 27485229056,RSS (B): 9370763264,Used File Descriptors: 348

      [01 Oct 2015 00:48:34,773] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 244670136,Used JVM (B): 8179065160,VSize (B): 27485229056,RSS (B): 9370755072,Used File Descriptors: 349

      [01 Oct 2015 00:48:34,773] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 232897592,Used JVM (B): 8190837704,VSize (B): 27485229056,RSS (B): 9370755072,Used File Descriptors: 350

      [01 Oct 2015 00:48:34,775] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 221176280,Used JVM (B): 8202559016,VSize (B): 27485229056,RSS (B): 9370734592,Used File Descriptors: 351

      [01 Oct 2015 00:48:34,776] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 197316232,Used JVM (B): 8226419064,VSize (B): 27485229056,RSS (B): 9370714112,Used File Descriptors: 351

      [01 Oct 2015 00:52:06,690] [Scheduled-System-Tasks-Thread-5] [ERROR] [System:System:] [App Server Heartbeat] java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

      1. com.bladelogic.om.infra.mfw.util.BlException: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:145)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.updateAppServerKeepAliveStatus(DbAppServerGroupCoordinatorImpl.java:253)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.access$000(DbAppServerGroupCoordinatorImpl.java:28)

                      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl$1.execute(DbAppServerGroupCoordinatorImpl.java:166)

                      at com.bladelogic.om.infra.app.service.timer.ScheduledTaskReference.run(ScheduledTaskReference.java:95)

                      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

                      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

                      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                      at java.lang.Thread.run(Thread.java:662)

      Caused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:389)

                      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:382)

                      at oracle.jdbc.driver.T4C7Ocommoncall.processError(T4C7Ocommoncall.java:93)

                      at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)

                      at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)

                      at oracle.jdbc.driver.T4C7Ocommoncall.doOCOMMIT(T4C7Ocommoncall.java:75)

                      at oracle.jdbc.driver.T4CConnection.doCommit(T4CConnection.java:565)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3851)

                      at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3857)

                      at com.bladelogic.om.infra.shared.db.BlDbConnection.commit(BlDbConnection.java:83)

                      at org.apache.commons.dbcp.DelegatingConnection.commit(DelegatingConnection.java:334)

                      at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.commit(PoolingDataSource.java:211)

                      at com.bladelogic.om.infra.app.service.db.UserTransactionImpl.commit(UserTransactionImpl.java:137)

                      at com.bladelogic.om.infra.app.service.session.SessionImpl.commitUserTransaction(SessionImpl.java:352)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.manageTransaction(TransactionalOperation.java:175)

                      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:136)

                      ... 13 more

      [01 Oct 2015 00:54:15,160] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 162312176,Used JVM (B): 8261423120,VSize (B): 27485229056,RSS (B): 9370791936,Used File Descriptors: 354

      [01 Oct 2015 00:54:15,161] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 165702576,Used JVM (B): 8258032720,VSize (B): 27485229056,RSS (B): 9370787840,Used File Descriptors: 352

      [01 Oct 2015 00:54:37,511] [Scheduled-System-Tasks-Thread-14] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 8423735296,Free JVM (B): 174726480,Used JVM (B): 8249008816,VSize (B): 27485229056,RSS (B): 9370775552,Used File Descriptors: 353

      [01 Oct 2015 00:55:32,127] [Scheduled-System-Tasks-Thread-5] [ERROR] [System:System:] [App Server Heartbeat] java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

       

      1. com.bladelogic.om.infra.mfw.util.BlException: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation
        • 1. Re: Issue with Bladelogic App server
          Bill Robinson

          how much physical memory do you have on the appserver and how many instances are on each appserver?

           

          what is the system doing during the times you see the db errors?   is it under high load ?  is the db under high load - doing a backup or something like that ?

           

          does this happen on a regular interval or randomly during the day ?

          • 2. Re: Issue with Bladelogic App server
            Craig Dockter

            TCP keepalives may help.  Since JDBC does not look at the tnsnames.ora file for connection settings, the only way to configure keepalives is to specify the entire tnsnames.ora entry as a single line for you connection string.  Here is ours as an example:

             

            ConnectionString:jdbc:oracle:thin:@(DESCRIPTION=(ENABLE=BROKEN)(ADDRESS=(COMMUNITY=tcpcom.domain.com)(PROTOCOL=TCP)(Host=db-server.domain.com)(Port=1521))(CONNECT_DATA=(SID=XXXXXX)))

             

            If the appserver is running on Linux you can use the netstat command to verify TCP keepalive packets are being used:

             

            ]# netstat -aon | grep :1521

            tcp        0      0 111.111.111.111:53166           999.999.999.999:1521            ESTABLISHED keepalive (1103.09/0/0)