11 Replies Latest reply on Apr 24, 2014 9:12 AM by Wookie Houle

    Failed to obtain high availability node's lock within the configured timeout

      Share This:

      Here's the rough backstory:

       

      AO grid functioning OK  (CDP + AP)

      BEM Monitor and Actor Adapter deployed  (HA BEM, but disabled in Actor adapter as it didnt seem to work, another story)

      Custom AO Rules and workflow setup to handle BEM Events and update Event data, all worked fine...

      AO/BEM left over night, processing a few 1000 events.

      Next day no events being processed by the BEM monitor adapter

       

      Now, the CDP log is full up with 100's of the exception below, what does this all mean?    I'm hoping someone may have an 'ah'haa' moment whilst I go digging...

      I tried a restart last time I was here, over a week ago, will try again, but im not hopeful and somewhat concered AO just stopped processing BEM events.

      (apologies for the lack of timely info)

       

       

      14 Mar 2011 14:12:23,504 DEBUG AOConnectionExpirationPolicy : [Thread: Whirlycache Tuner ( named_connection_cache-AdapterConfiguration1291969801648--1437768808 )]  Performing Maintenance at 2011-03-14 14:12:23.504

      14 Mar 2011 14:12:23,504 DEBUG AOConnectionExpirationPolicy : [Thread: Whirlycache Tuner ( named_connection_cache-AdapterConfiguration1291969801648--1437768808 )]  Cache size: 0

      14 Mar 2011 14:12:23,504 DEBUG AOConnectionExpirationPolicy : [Thread: Whirlycache Tuner ( named_connection_cache-AdapterConfiguration1291969801648--1437768808 )] Locking ManagedCache Object...........

      14 Mar 2011 14:12:23,504 DEBUG AOConnectionExpirationPolicy : [Thread: Whirlycache Tuner ( named_connection_cache-AdapterConfiguration1291969801648--1437768808 )] Unlocking ManagedCache Object...........

      14 Mar 2011 14:12:26,243 WARN  HANode          : Failed to obtain high availability node's lock within the configured timeout of 120000 ms while processing a heartbeat message from another high availability node. Cancelling this activity.

      14 Mar 2011 14:12:26,270 WARN  HANode          : Lock owner's stack trace:

          com.bmc.ao.foundation.jobmanager.DefaultLocalJobManager.uninitialize(DefaultLocalJobManager.java:1290)

          com.bmc.ao.foundation.jobmanager.DefaultSharedJobManager.onLeaveMaster(DefaultSharedJobManager.java:268)

          com.realops.foundation.gridframework.HAHub.stateChanged(HAHub.java:6215)

          com.realops.foundation.gridframework.HANode.setState(HANode.java:728)

          com.realops.foundation.gridframework.HANode.goToElectionState(HANode.java:849)

          com.realops.foundation.gridframework.HANode.heartbeat(HANode.java:361)

          com.realops.foundation.gridframework.HAHub.heartbeat(HAHub.java:95)

          sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

          sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

          sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          java.lang.reflect.Method.invoke(Method.java:597)

          com.realops.foundation.gridframework.TopologyParticipant.invoke(TopologyParticipant.java:2160)

          com.realops.foundation.gridframework.TopologyParticipant.processInboundMessage(TopologyParticipant.java:1592)

          com.realops.foundation.gridframework.TopologyParticipant$MyReceiver.processMessage(TopologyParticipant.java:409)

          com.realops.foundation.gridframework.StatsRecorderDecorator$2.processMessage(StatsRecorderDecorator.java:97)

          com.realops.foundation.gridframework.DefaultMessageDispatcher$1DispatcherTask.run(DefaultMessageDispatcher.java:124)

          java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          java.lang.Thread.run(Thread.java:619)