6 Replies Latest reply on Aug 14, 2014 2:56 AM by Kalpa Ashhar

    java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

    Atul Matkar

      We face about once every two months an issue where BladeLogic would cancel jobs, reporting that application servers went down. Looking at the respective application server log it shows that basically there is a SQL timeout that prevents the application server to write job results into the Oracle DB. Our DBAs don't see any issue with the database as the "user requested cancel" so the DB is all fine. This usually happens when several jobs are running simultanously thus opening a lot of sessions.

      Can you point to the right direction what should be done? This time two compliance jobs were running simultanously, limiting execution to 50 servers and an agent update on all the estate. Clearly a DB such as Oracle and a system such as BladeLogic should be able to cope with that? Are we missing some settings in Oracle or the JAVA SQL timeout?

      [11 Aug 2014 12:38:16,083] [WorkItem-Thread-33] [INFO] [First.last.adm@domain.com:MEJSD-BLAdmins-PRD:] [Compliance] Executing command : nsh -c "//loc
      alhost/data/bladelogic/filestore/extended_objects/disawin-audit.nsh" wtfsql30 "Logon"
      [11 Aug 2014 12:38:16,190] [WorkItem-Thread-33] [INFO] [First.last.adm@domain.com:MEJSD-BLAdmins-PRD:] [Compliance] Started pid 25761: nsh -c "//loca
      lhost/data/bladelogic/filestore/extended_objects/disawin-audit.nsh" wtfsql30 "Logon"
      [11 Aug 2014 12:38:16,500] [Scheduled-System-Tasks-Thread-5] [INFO] [System:System:] [App Server Heartbeat] SQLException: Sql State (72000) Vendor Code (1013)
      [11 Aug 2014 12:38:16,500] [Scheduled-System-Tasks-Thread-5] [ERROR] [System:System:] [App Server Heartbeat] ORA-01013: user requested cancel of current operation

      java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
      at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837)
      at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)
      at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
      at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
      at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
      at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1010)
      at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1315)
      at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576)
      at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3657)
      at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1350)
      at com.bladelogic.om.infra.shared.db.BlPreparedStatement.executeUpdate(BlPreparedStatement.java:63)
      at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
      at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
      at com.bladelogic.om.infra.app.service.db.DefaultDatabaseServiceImpl.execute(DefaultDatabaseServiceImpl.java:1009)
      at com.bladelogic.om.infra.app.service.db.DefaultDatabaseServiceImpl.execute(DefaultDatabaseServiceImpl.java:645)
      at com.bladelogic.om.infra.app.service.db.DefaultDatabaseServiceImpl.execute(DefaultDatabaseServiceImpl.java:624)
      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl$2.doDBOperation(DbAppServerGroupCoordinatorImpl.java:234)
      at com.bladelogic.om.infra.app.db.TransactionalOperation.manageTransaction(TransactionalOperation.java:174)
      at com.bladelogic.om.infra.app.db.TransactionalOperation.execute(TransactionalOperation.java:136)
      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.updateAppServerKeepAliveStatus(DbAppServerGroupCoordinatorImpl.java:253)
      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl.access$000(DbAppServerGroupCoordinatorImpl.java:28)
      at com.bladelogic.om.infra.app.service.appserver.DbAppServerGroupCoordinatorImpl$1.execute(DbAppServerGroupCoordinatorImpl.java:166)
      at com.bladelogic.om.infra.app.service.timer.ScheduledTaskReference.run(ScheduledTaskReference.java:95)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      [11 Aug 2014 12:38:16,842] [WaitForProcessThread-PID-25712-92086] [INFO] [First.last.adm@domain.com:MEJSD-BLAdmins-PRD:] [Compliance] Process finished: 25712