1 2 Previous Next 22 Replies Latest reply on Jun 20, 2013 9:35 AM by Mike Reider Branched to a new discussion.

    App Server doesnt start after upgrade from 8.2 to 8.3.01

    Mike Reider

      Hi all, I recently upgraded my BSA environment from 8.2 to 8.3.01, ran the Data Migration manager, no errors in upgrading the data to 8.3.01

      Rebooted the entire server, when I try to start my app server service, I am getting this error in appserver.log

       

       

      [ERROR] [::] [] Error installing to Start: name=bladelogic.service.ManagementService state=Create

      com.bladelogic.om.infra.app.service.ServiceInitializationException: Failed to start management service

        at com.bladelogic.om.infra.app.service.manage.ManagementServiceImpl.start(ManagementServiceImpl.java:92)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

        at java.lang.reflect.Method.invoke(Unknown Source)

        at org.jboss.joinpoint.plugins.reflect.ReflectMethodJoinPoint.dispatch(ReflectMethodJoinPoint.java:72)

        at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$DispatchJoinPoint.run(KernelControllerContextActions.java:631)

        at java.security.AccessController.doPrivileged(Native Method)

        at org.jboss.kernel.plugins.dependency.KernelControllerContextActions.dispatchJoinPoint(KernelControllerContextActions.java:99)

        at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$LifecycleAction.installAction(KernelControllerContextActions.java:452)

        at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$1.run(KernelControllerContextActions.java:156)

        at java.security.AccessController.doPrivileged(Native Method)

        at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$KernelControllerContextAction.install(KernelControllerContextActions.java:179)

        at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

        at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:226)

        at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:593)

        at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:346)

        at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:438)

        at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:379)

        at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:225)

        at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:151)

        at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBean(AbstractKernelDeployer.java:291)

        at com.bladelogic.om.infra.app.service.bootstrap.ServiceBeanXMLDeployer.deployBean(ServiceBeanXMLDeployer.java:38)

        at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBeans(AbstractKernelDeployer.java:261)

        at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deploy(AbstractKernelDeployer.java:117)

        at org.jboss.kernel.plugins.deployment.xml.BeanXMLDeployer.deploy(BeanXMLDeployer.java:91)

        at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.deploy(ServiceBootstrap.java:252)

        at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.deployService(ServiceBootstrap.java:199)

        at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.bootstrap(ServiceBootstrap.java:153)

        at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:91)

        at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.startup(ServiceBootstrap.java:66)

        at com.bladelogic.om.infra.mfw.fw.BlManager.start(BlManager.java:883)

        at com.bladelogic.om.infra.mfw.fw.BlManager.main(BlManager.java:1403)

      Caused by: java.io.IOException: Cannot bind to URL [rmi://BLADE:9836/jmxrmi]: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]

        at javax.management.remote.rmi.RMIConnectorServer.newIOException(Unknown Source)

        at javax.management.remote.rmi.RMIConnectorServer.start(Unknown Source)

        at com.bladelogic.om.infra.app.service.manage.ManagementServiceImpl.start(ManagementServiceImpl.java:86)

        ... 32 more

      Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]

        at com.sun.jndi.rmi.registry.RegistryContext.bind(Unknown Source)

        at com.sun.jndi.toolkit.url.GenericURLContext.bind(Unknown Source)

        at javax.naming.InitialContext.bind(Unknown Source)

        at javax.management.remote.rmi.RMIConnectorServer.bind(Unknown Source)

        ... 34 more

      Caused by: java.rmi.ConnectIOException: non-JRMP server at remote endpoint

        at sun.rmi.transport.tcp.TCPChannel.createConnection(Unknown Source)

        at sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown Source)

        at sun.rmi.server.UnicastRef.newCall(Unknown Source)

        at sun.rmi.registry.RegistryImpl_Stub.bind(Unknown Source)

       

      My IP settings look correct, I have a static IP and can ping it. Not sure why Im getting binding errors. I searched KB but no dice.

        • 1. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
          Alan Nakashian-Holsberg

          Mike,

           

          Try the following:

           

          - Launch JMX Console: ./bljconsole

          - Login against a Console appserver box

          - When JMX connection information needs to be provided you should write: Job only appserver's registryport > default port is 9136, server hostname and no user or password

          If registryport may have been changed, you can verify this value by running on the job appserver box: 'blasadmin show all | grep -i registry'


          Alan

          • 2. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
            Mike Reider

            Hi Alan, my box is win2008, is JMX console available on win?

             

            edit, nevermind found the executable in /bin folder

            • 3. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
              Alan Nakashian-Holsberg

              Mike,

               

              Yes, the JMX console is available on Windows.

               

              Alan

              • 4. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
                Mike Reider

                I ran the App server Configuration wizard, and noticed this error,

                error1.gif

                 

                apparently the DB version is different from Console version. I ran the Data Migration mgr with no errors,

                 

                ****** FINISHED ****** SINGLE STEP MIGRATION FROM 8.2.01,8.2.02,8.2.04 TO 8.3.00

                FROM CONFIG FILE C:\tmp\bl_8.3.01\db_scripts\sqlserver\upgrade\8.2.01_to_8.3.00\OM\Migration_actions\

                WITH **NO** ERRORS

                .

                Is this expected that the DB is out of sync version-wise? Wondering if I missed a step during the upgrade process

                • 5. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
                  Alan Nakashian-Holsberg

                  Mike,

                   

                  Which external files did you use?  Did you download and run 83-160-SQL_Update_Scripts?

                   

                  Alan

                  • 6. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
                    Mike Reider

                    used these

                     

                    8301-external-files.zip

                    downloaded yesterday from EPD

                     

                    I just ran the SQL update scripts so the DB is same version as OM but still having the original issue

                     

                    - Login against a Console appserver box

                     

                    I cant login using JMX to the appserver since I cant start the appserver. Also its an SRP credential and i cant have a blank user name.

                     

                     

                    I checked the registry with blasadmin, showing a 9836 port, this was the port I was using before the upgrade which worked. Should I change this to 9136?

                     

                    % blasadmin show all | grep -i "registry"

                    RegistryPort:9836

                    RegistryPort:1067

                    RegistryApplicationName:BMC_BladeLogic_Server_Automation_Suite

                    RegistryBusinessName:BMC

                    RegistryCriteria:

                    RegistryPublishedHostName:

                    RegistryServer:

                    RegistryServerPassword:

                    RegistryServerPort:

                    RegistryServerProtocol:http

                    RegistryServerUserName:

                    • 7. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
                      Mike Reider

                      I just noticed that theres a bljconsole and bljconsole-launcher

                       

                      I tried logging into both but cant login

                       

                      error2.gif

                      • 8. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
                        Alan Nakashian-Holsberg

                        Mike,

                         

                        Use the BLAdmin account to login to the JMX console.

                         

                        You should use the bljconsole.bat, it should set environmental variables.  If this does not work, I would suggest opening a support issue and continuing to work from there.

                         

                        Alan

                        • 9. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
                          Mike Reider

                          tried BLAdmin, it didnt connect. Ill open a ticket. Thanks!

                          • 10. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
                            Alan Nakashian-Holsberg

                            Mike,

                             

                            Please let me know the issue ID when it's opened, I'll provide additional assistance as I am able to for the engineer who gets assigned the issue.

                             

                            Alan

                            • 11. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
                              Alan Nakashian-Holsberg

                              Mike,

                               

                              Another thought on this.

                               

                              Can you do the following for me.

                               

                              go to services.msc and stop the appserver process.

                              go to command prompt and execute "netstat -ano |find "9836"

                              if this comes back with no results, start the appserver process and check the appserver.log for messages.

                               

                              in further review of the data that you shared it looks like that port was already in use "Caused by: java.io.IOException: Cannot bind to URL [rmi://BLADE:9836/jmxrmi]:"

                               

                              let me know how this works out.

                               

                              Alan

                              • 12. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
                                Mike Reider

                                I cant stop the appserver process since I could never get it started after the upgrade,

                                heres the full netstat return, there are no blade ports open except for rscd 4750

                                 

                                c:\tmp\>netstat -ano

                                 

                                Active Connections

                                 

                                  Proto  Local Address          Foreign Address        State           PID

                                  TCP    0.0.0.0:80             0.0.0.0:0              LISTENING       4

                                  TCP    0.0.0.0:135            0.0.0.0:0              LISTENING       868

                                  TCP    0.0.0.0:445            0.0.0.0:0              LISTENING       4

                                  TCP    0.0.0.0:1433           0.0.0.0:0              LISTENING       1752

                                  TCP    0.0.0.0:2383           0.0.0.0:0              LISTENING       1904

                                  TCP    0.0.0.0:4750           0.0.0.0:0              LISTENING       2252

                                  TCP    0.0.0.0:5357           0.0.0.0:0              LISTENING       4

                                  TCP    0.0.0.0:8009           0.0.0.0:0              LISTENING       1300

                                  TCP    0.0.0.0:8080           0.0.0.0:0              LISTENING       1300

                                  TCP    0.0.0.0:49152          0.0.0.0:0              LISTENING       548

                                  TCP    0.0.0.0:49153          0.0.0.0:0              LISTENING       952

                                  TCP    0.0.0.0:49154          0.0.0.0:0              LISTENING       1008

                                  TCP    0.0.0.0:49155          0.0.0.0:0              LISTENING       640

                                  TCP    0.0.0.0:49157          0.0.0.0:0              LISTENING       628

                                  TCP    127.0.0.1:1434         0.0.0.0:0              LISTENING       1752

                                  TCP    127.0.0.1:8005         0.0.0.0:0              LISTENING       1300

                                  TCP    192.168.34.112:139     0.0.0.0:0              LISTENING       4

                                  TCP    [::]:80                [::]:0                 LISTENING       4

                                  TCP    [::]:135               [::]:0                 LISTENING       868

                                  TCP    [::]:445               [::]:0                 LISTENING       4

                                  TCP    [::]:1433              [::]:0                 LISTENING       1752

                                  TCP    [::]:2383              [::]:0                 LISTENING       1904

                                  TCP    [::]:5357              [::]:0                 LISTENING       4

                                  TCP    [::]:8009              [::]:0                 LISTENING       1300

                                  TCP    [::]:8080              [::]:0                 LISTENING       1300

                                  TCP    [::]:49152             [::]:0                 LISTENING       548

                                  TCP    [::]:49153             [::]:0                 LISTENING       952

                                  TCP    [::]:49154             [::]:0                 LISTENING       1008

                                  TCP    [::]:49155             [::]:0                 LISTENING       640

                                  TCP    [::]:49157             [::]:0                 LISTENING       628

                                  TCP    [::1]:1434             [::]:0                 LISTENING       1752

                                  UDP    0.0.0.0:123            *:*                                    312

                                  UDP    0.0.0.0:500            *:*                                    1008

                                  UDP    0.0.0.0:3702           *:*                                    312

                                  UDP    0.0.0.0:3702           *:*                                    312

                                  UDP    0.0.0.0:4500           *:*                                    1008

                                  UDP    0.0.0.0:5355           *:*                                    488

                                  UDP    0.0.0.0:57867          *:*                                    312

                                  UDP    127.0.0.1:1900         *:*                                    312

                                  UDP    127.0.0.1:57872        *:*                                    312

                                  UDP    192.168.34.112:137     *:*                                    4

                                  UDP    192.168.34.112:138     *:*                                    4

                                  UDP    192.168.34.112:1900    *:*                                    312

                                  UDP    192.168.34.112:57871   *:*                                    312

                                  UDP    [::]:123               *:*                                    312

                                  UDP    [::]:500               *:*                                    1008

                                  UDP    [::]:3702              *:*                                    312

                                  UDP    [::]:3702              *:*                                    312

                                  UDP    [::]:5355              *:*                                    488

                                  UDP    [::]:57868             *:*                                    312

                                  UDP    [::1]:1900             *:*                                    312

                                  UDP    [::1]:57870            *:*                                    312

                                  UDP    [fe80::acc0:ca19:1102:b8f8%10]:1900  *:*

                                     312

                                  UDP    [fe80::acc0:ca19:1102:b8f8%10]:57869  *:*

                                      312

                                 

                                I am thinking of restoring my VM to an earlier snapshot and going through the upgrade again, to make sure I havent missed anything.

                                • 13. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
                                  Sean Berry

                                  What's the app server log show?

                                  • 14. Re: App Server doesnt start after upgrade from 8.2 to 8.3.01
                                    Mike Reider

                                    tried to start the app server, still getting same error,

                                     

                                     

                                    [19 Jun 2013 11:27:32,627] [main] [INFO] [::] [] Starting Management Service ...

                                    [19 Jun 2013 11:27:32,817] [main] [ERROR] [::] [] Error installing to Start: name=bladelogic.service.ManagementService state=Create

                                    com.bladelogic.om.infra.app.service.ServiceInitializationException: Failed to start management service

                                      at com.bladelogic.om.infra.app.service.manage.ManagementServiceImpl.start(ManagementServiceImpl.java:92)

                                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                                      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

                                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

                                      at java.lang.reflect.Method.invoke(Unknown Source)

                                      at org.jboss.joinpoint.plugins.reflect.ReflectMethodJoinPoint.dispatch(ReflectMethodJoinPoint.java:72)

                                      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$DispatchJoinPoint.run(KernelControllerContextActions.java:631)

                                      at java.security.AccessController.doPrivileged(Native Method)

                                      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions.dispatchJoinPoint(KernelControllerContextActions.java:99)

                                      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$LifecycleAction.installAction(KernelControllerContextActions.java:452)

                                      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$1.run(KernelControllerContextActions.java:156)

                                      at java.security.AccessController.doPrivileged(Native Method)

                                      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$KernelControllerContextAction.install(KernelControllerContextActions.java:179)

                                      at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

                                      at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:226)

                                      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:593)

                                      at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:346)

                                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:438)

                                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:379)

                                      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:225)

                                      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:151)

                                      at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBean(AbstractKernelDeployer.java:291)

                                      at com.bladelogic.om.infra.app.service.bootstrap.ServiceBeanXMLDeployer.deployBean(ServiceBeanXMLDeployer.java:38)

                                      at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBeans(AbstractKernelDeployer.java:261)

                                      at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deploy(AbstractKernelDeployer.java:117)

                                      at org.jboss.kernel.plugins.deployment.xml.BeanXMLDeployer.deploy(BeanXMLDeployer.java:91)

                                      at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.deploy(ServiceBootstrap.java:252)

                                      at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.deployService(ServiceBootstrap.java:199)

                                      at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.bootstrap(ServiceBootstrap.java:153)

                                      at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:91)

                                      at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.startup(ServiceBootstrap.java:66)

                                      at com.bladelogic.om.infra.mfw.fw.BlManager.start(BlManager.java:883)

                                      at com.bladelogic.om.infra.mfw.fw.BlManager.main(BlManager.java:1403)

                                    Caused by: java.io.IOException: Cannot bind to URL [rmi://BLADE:9836/jmxrmi]: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]

                                      at javax.management.remote.rmi.RMIConnectorServer.newIOException(Unknown Source)

                                      at javax.management.remote.rmi.RMIConnectorServer.start(Unknown Source)

                                      at com.bladelogic.om.infra.app.service.manage.ManagementServiceImpl.start(ManagementServiceImpl.java:86)

                                      ... 32 more

                                    Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]

                                      at com.sun.jndi.rmi.registry.RegistryContext.bind(Unknown Source)

                                      at com.sun.jndi.toolkit.url.GenericURLContext.bind(Unknown Source)

                                      at javax.naming.InitialContext.bind(Unknown Source)

                                      at javax.management.remote.rmi.RMIConnectorServer.bind(Unknown Source)

                                      ... 34 more

                                    Caused by: java.rmi.ConnectIOException: non-JRMP server at remote endpoint

                                      at sun.rmi.transport.tcp.TCPChannel.createConnection(Unknown Source)

                                      at sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown Source)

                                      at sun.rmi.server.UnicastRef.newCall(Unknown Source)

                                      at sun.rmi.registry.RegistryImpl_Stub.bind(Unknown Source)

                                      ... 38 more

                                    [19 Jun 2013 11:27:32,993] [main] [INFO] [::] [] Starting Service Lifecycle Service...

                                    [19 Jun 2013 11:27:32,995] [main] [INFO] [::] [] Service Lifecycle Service started.

                                    [19 Jun 2013 11:27:32,997] [main] [INFO] [::] [] Registering remote services...

                                    [19 Jun 2013 11:27:32,998] [main] [INFO] [::] [] Shutdown processes complete

                                    [19 Jun 2013 11:27:33,002] [Thread-1] [INFO] [::] [] Undeploying

                                    [19 Jun 2013 11:27:33,013] [Thread-1] [ERROR] [::] [] java.lang.NullPointerException

                                    [19 Jun 2013 11:27:33,019] [Thread-1] [ERROR] [::] [] java.rmi.ConnectIOException: non-JRMP server at remote endpoint

                                    1 2 Previous Next