10 Replies Latest reply on Oct 2, 2014 5:27 AM by Sudesh Sancheti

    Process spawner error

      Hi All We are facing issue due to process spawener in Bladelogic 8.1.04 enviornment. We have two app servers running on Linux servers and pointing to same database. When we are running two different jobs at the same time it failes with the below error.

       

      Error Oct 1, 2014 2:47:40 PM Caught exception running command - /opt/nsh/NSH/bin/nsh --norc -c /opt/nsh/NSH/tmp/server1-job/scripts/job__bad6386e-2ed7-4438-835b-b9c4b1289412/master_03dbdcc2-ed7a-4743-926d-5cb24819a0d9 Error: There was an error connecting to the Process Spawner: Connection refused to host: server1.unix.xxx.com; nested exception is: java.net.ConnectException: Connection refused.

       

      Please confirm that the Process Spawner is running. When I go and check process spawener service it is running fine. I checked in spawner log for server server1 where it is failing and it shows as below.

       

      Please suggest.

       

       

      [26 Sep 2014 10:26:39,556] [Thread-0] [INFO] [::] [] Undeploying [26 Sep 2014 10:26:39,559] [Thread-0] [ERROR] [::] [] java.rmi.ConnectIOException: Exception creating connection to: server1.unix.XXX.com; nested exception is:         java.net.SocketException: Network is unreachable [26 Sep 2014 12:36:00,402] [main] [INFO] [::] [] Starting Configuration Service... [26 Sep 2014 12:36:00,436] [main] [INFO] [::] [] Configuration Service started. [26 Sep 2014 12:36:00,454] [main] [INFO] [::] [] Loading JNI libaries... [26 Sep 2014 12:36:00,462] [main] [INFO] [::] [] Loaded JNI library BlAsJni [26 Sep 2014 12:36:00,463] [main] [INFO] [::] [] Loaded JNI library XJNIService [26 Sep 2014 12:36:00,463] [main] [INFO] [::] [] Loaded JNI library Atrium Core [26 Sep 2014 12:36:00,463] [main] [INFO] [::] [] Initializing JNI Configuration... [26 Sep 2014 12:36:00,464] [main] [INFO] [::] [] Initialized JNI library BlAsJni [26 Sep 2014 12:36:00,465] [main] [INFO] [::] [] Initialized JNI library XJNIService [26 Sep 2014 12:36:00,465] [main] [INFO] [::] [] Initialized JNI library Atrium Core [26 Sep 2014 12:36:00,465] [main] [INFO] [::] [] JNI Configuration complete. [26 Sep 2014 12:36:00,534] [main] [INFO] [::] [] Starting Model Graph Service... [26 Sep 2014 12:36:02,302] [main] [INFO] [::] [] Model Graph Service started. [26 Sep 2014 12:36:02,310] [main] [INFO] [::] [] Starting Port Provider... [26 Sep 2014 12:36:02,311] [main] [INFO] [::] [] Port Provider started. [26 Sep 2014 12:36:02,495] [main] [INFO] [::] [] Starting Sizing Service... [26 Sep 2014 12:36:02,495] [main] [INFO] [::] [] Sizing Service started. [26 Sep 2014 12:36:02,496] [main] [INFO] [::] [] Starting Caching Service... [26 Sep 2014 12:36:02,499] [main] [INFO] [::] [] Caching Service started. [26 Sep 2014 12:36:02,500] [main] [INFO] [::] [] Starting Session Service... [26 Sep 2014 12:36:02,507] [main] [INFO] [::] [] Session Service started. [26 Sep 2014 12:36:02,510] [main] [INFO] [::] [] Starting Task Scheduling Service... [26 Sep 2014 12:36:02,514] [main] [INFO] [::] [] Task Scheduling Service started. [26 Sep 2014 12:36:02,516] [main] [INFO] [::] [] Starting Directory Service... [26 Sep 2014 12:36:02,525] [main] [INFO] [::] [] Starting RMI registry on host: server1.unix.XXX.com [26 Sep 2014 12:36:02,533] [main] [INFO] [::] [] Directory Service started. [26 Sep 2014 12:36:02,535] [main] [INFO] [::] [] Setting server socket factory: com.bladelogic.om.infra.app.service.client.ServerSocketFactory [26 Sep 2014 12:36:02,535] [main] [INFO] [::] [] Setting client socket factory: com.bladelogic.om.infra.app.service.client.OMClientSocketFactory [26 Sep 2014 12:36:02,535] [main] [INFO] [::] [] Setting authenticator: com.bladelogic.om.infra.app.service.manage.auth.BlSessionServerAuthenticator [26 Sep 2014 12:36:02,535] [main] [INFO] [::] [] Starting Management Service ... [26 Sep 2014 12:36:02,789] [main] [INFO] [::] [] Management Service started. [26 Sep 2014 12:36:02,794] [main] [INFO] [::] [] Starting App Server Data Source... [26 Sep 2014 12:36:03,034] [main] [ERROR] [::] [] Error installing to Start: name=bladelogic.service.AsDataSourceService state=Create java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection         at com.bladelogic.om.infra.shared.db.pool.BlDbcpConnPool.init(BlDbcpConnPool.java:147)         at com.bladelogic.om.infra.app.service.db.BlDbPooledDataSource.createConnectionPool(BlDbPooledDataSource.java:53)         at com.bladelogic.om.infra.shared.db.pool.BlPooledDbDataSource.start(BlPooledDbDataSource.java:68)         at com.bladelogic.om.infra.app.service.db.BlAsDbDataSource.start(BlAsDbDataSource.java:212)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)         at java.lang.reflect.Method.invoke(Method.java:597)         at org.jboss.joinpoint.plugins.reflect.ReflectMethodJoinPoint.dispatch(ReflectMethodJoinPoint.java:72)         at org.jboss.kernel.plugins.dependency.KernelControllerContextActions.dispatchJoinPoint(KernelControllerContextActions.java:92)         at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$LifecycleAction.installAction(KernelControllerContextActions.java:452)         at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$KernelControllerContextAction.install(KernelControllerContextActions.java:147)         at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)         at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:226)         at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:593)         at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:346)         at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:438)         at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:379)         at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:225)         at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:151)         at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBean(AbstractKernelDeployer.java:291)         at com.bladelogic.om.infra.app.service.bootstrap.ServiceBeanXMLDeployer.deployBean(ServiceBeanXMLDeployer.java:38)         at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBeans(AbstractKernelDeployer.java:261)         at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deploy(AbstractKernelDeployer.java:117)         at org.jboss.kernel.plugins.deployment.xml.BeanXMLDeployer.deploy(BeanXMLDeployer.java:91)         at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.deploy(ServiceBootstrap.java:252)         at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.deployService(ServiceBootstrap.java:199)         at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.bootstrap(ServiceBootstrap.java:153)         at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:91)         at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.startup(ServiceBootstrap.java:66)         at com.bladelogic.om.infra.mfw.fw.BlManager.startWithType(BlManager.java:590)         at com.bladelogic.om.infra.mfw.fw.BlManager.startWithType(BlManager.java:725)         at com.bladelogic.om.infra.app.process.ProcessSpawnerManager.start(ProcessSpawnerManager.java:93)         at com.bladelogic.om.infra.app.process.ProcessSpawnerManager.main(ProcessSpawnerManager.java:136) Caused by: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection         at com.bladelogic.om.infra.shared.db.BlDbConnectionFactory.createConnection(BlDbConnectionFactory.java:61)         at com.bladelogic.om.infra.shared.db.BlDbConnectionFactory.createConnection(BlDbConnectionFactory.java:23)         at com.bladelogic.om.infra.shared.db.pool.BlDbDbCpConnectionFactory.createConnection(BlDbDbCpConnectionFactory.java:22)         at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)         at com.bladelogic.om.infra.app.service.db.BlDbPoolableConnectionFactory.makeObject(BlDbPoolableConnectionFactory.java:51)         at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)         at com.bladelogic.om.infra.shared.db.pool.BlConnObjectPool$BlDbObjectPool.borrowObject(BlConnObjectPool.java:55)         at com.bladelogic.om.infra.shared.db.pool.BlConnObjectPool.borrowObject(BlConnObjectPool.java:94)         at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)         at com.bladelogic.om.infra.shared.db.pool.BlDbcpConnPool.init(BlDbcpConnPool.java:138)         ... 33 more [26 Sep 2014 12:36:03,036] [main] [INFO] [::] [] Registering remote services... [26 Sep 2014 12:36:03,036] [main] [ERROR] [::] [] com.bladelogic.om.infra.app.service.bootstrap.DeploymentException: Failed to deploy services: bladelogic.service.AsDataSourceService bladelogic.service.AuthenticationSrpService bladelogic.service.DatabaseService bladelogic.service.NotificationService bladelogic.service.ProcessSpawnerService bladelogic.service.RBACRoleService bladelogic.service.RBACService bladelogic.service.RBACUserService com.bladelogic.om.infra.app.service.bootstrap.DeploymentException: com.bladelogic.om.infra.app.service.bootstrap.DeploymentException: Failed to deploy services: bladelogic.service.AsDataSourceService bladelogic.service.AuthenticationSrpService bladelogic.service.DatabaseService bladelogic.service.NotificationService bladelogic.service.ProcessSpawnerService bladelogic.service.RBACRoleService bladelogic.service.RBACService bladelogic.service.RBACUserService         at com.bladelogic.om.infra.mfw.fw.BlManager.startWithType(BlManager.java:597)         at com.bladelogic.om.infra.mfw.fw.BlManager.startWithType(BlManager.java:725)         at com.bladelogic.om.infra.app.process.ProcessSpawnerManager.start(ProcessSpawnerManager.java:93)         at com.bladelogic.om.infra.app.process.ProcessSpawnerManager.main(ProcessSpawnerManager.java:136) Caused by: com.bladelogic.om.infra.app.service.bootstrap.DeploymentException: Failed to deploy services: bladelogic.service.AsDataSourceService bladelogic.service.AuthenticationSrpService bladelogic.service.DatabaseService bladelogic.service.NotificationService bladelogic.service.ProcessSpawnerService bladelogic.service.RBACRoleService bladelogic.service.RBACService bladelogic.service.RBACUserService         at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.checkStates(ServiceBootstrap.java:359)         at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.registerRemoteServices(ServiceBootstrap.java:366)         at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.bootstrap(ServiceBootstrap.java:159)         at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:91)         at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.startup(ServiceBootstrap.java:66)         at com.bladelogic.om.infra.mfw.fw.BlManager.startWithType(BlManager.java:590)         ... 3 more [30 Sep 2014 15:08:11,273] [Thread-0] [INFO] [::] [] Undeploying [30 Sep 2014 15:08:11,289] [Thread-0] [ERROR] [::] [] java.rmi.NotBoundException: bladelogic.service.DatabaseService