1 2 Previous Next 15 Replies Latest reply on Jun 5, 2012 5:20 PM by Bill Robinson

    Cannot start appserver service after updating to 8.2.sp1

    russ mcdowell

      On a Windows 2008 R2 64bit OS, talking to SQL 2008 backend.  Upgraded from 8.0.14 to 8.2sp1.

       

      This from the appserver.log:

       

      [05 Jun 2012 14:46:41,237] [main] [INFO] [::] [] Starting Authentication Service...

      [05 Jun 2012 14:46:41,285] [main] [WARN] [::] [] Authentication method in use is Active Directory/Kerberos.

      [05 Jun 2012 14:46:41,285] [main] [WARN] [::] [] Application server is unable to log into kerberos. Verify the application servers login account / password.

      [05 Jun 2012 14:46:41,285] [main] [WARN] [::] [] Check configuration of: C:\Program Files\BMC Software\BladeLogic\8.0\NSH\br\blappserv_login.conf

      [05 Jun 2012 14:46:41,285] [main] [WARN] [::] []                    and: C:\Program Files\BMC Software\BladeLogic\8.0\NSH\br\blappserv_krb5.conf

      [05 Jun 2012 14:46:41,285] [main] [WARN] [::] [] Actual error returned: No LoginModules configured for

      [05 Jun 2012 14:46:41,285] [main] [ERROR] [::] [] ADKerberos authentication is enabled but it is not configured correctly. If you are not using kerberos authentication, please turn it off in blasadmin by running the blasadmin command "set authserver isadkauthenabled false". If you are using it, please check the configuration files to ensure it is configured correctly

      [05 Jun 2012 14:46:41,285] [main] [ERROR] [::] [] Error installing to Start: name=bladelogic.service.AuthenticationService state=Create

      com.bladelogic.om.infra.app.service.ServiceInitializationException: com.bladelogic.om.infra.mfw.util.BlException: com.bladelogic.om.infra.mfw.util.BlException: Error creating credentials: No valid credentials provided (Mechanism level: Attempt to obtain new ACCEPT credentials failed!)

      at com.bladelogic.om.infra.auth.service.AuthenticationServiceImpl.start(AuthenticationServiceImpl.java:139)

      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

      at java.lang.reflect.Method.invoke(Unknown Source)

      at org.jboss.joinpoint.plugins.reflect.ReflectMethodJoinPoint.dispatch(ReflectMethodJoinPoint.java:72)

      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$DispatchJoinPoint.run(KernelControllerContextActions.java:631)

      at java.security.AccessController.doPrivileged(Native Method)

      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions.dispatchJoinPoint(KernelControllerContextActions.java:99)

      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$LifecycleAction.installAction(KernelControllerContextActions.java:452)

      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$1.run(KernelControllerContextActions.java:156)

      at java.security.AccessController.doPrivileged(Native Method)

      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$KernelControllerContextAction.install(KernelControllerContextActions.java:179)

      at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

      at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:226)

      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:593)

      at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:346)

      at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:438)

      at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:379)

      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:225)

      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:151)

      at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBean(AbstractKernelDeployer.java:291)

      at com.bladelogic.om.infra.app.service.bootstrap.ServiceBeanXMLDeployer.deployBean(ServiceBeanXMLDeployer.java:38)

      at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBeans(AbstractKernelDeployer.java:261)

      at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deploy(AbstractKernelDeployer.java:117)

      at org.jboss.kernel.plugins.deployment.xml.BeanXMLDeployer.deploy(BeanXMLDeployer.java:91)

      at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.deploy(ServiceBootstrap.java:252)

      at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.deployService(ServiceBootstrap.java:199)

      at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.bootstrap(ServiceBootstrap.java:153)

      at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:91)

      at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.startup(ServiceBootstrap.java:66)

      at com.bladelogic.om.infra.mfw.fw.BlManager.start(BlManager.java:881)

      at com.bladelogic.om.infra.mfw.fw.BlManager.main(BlManager.java:1401)

      Caused by: com.bladelogic.om.infra.mfw.util.BlException: com.bladelogic.om.infra.mfw.util.BlException: Error creating credentials: No valid credentials provided (Mechanism level: Attempt to obtain new ACCEPT credentials failed!)

      at com.bladelogic.om.infra.auth.service.AuthSvcStateMachineFactory.<init>(AuthSvcStateMachineFactory.java:53)

      at com.bladelogic.om.infra.auth.service.AuthSvcStateMachineFactory.init(AuthSvcStateMachineFactory.java:70)

      at com.bladelogic.om.infra.auth.service.AuthenticationServiceImpl.start(AuthenticationServiceImpl.java:131)

      ... 32 more

      Caused by: com.bladelogic.om.infra.mfw.util.BlException: Error creating credentials: No valid credentials provided (Mechanism level: Attempt to obtain new ACCEPT credentials failed!)

      at com.bladelogic.sso.protocol.kerberos.KerberosServiceStateMachine.initCredentials(KerberosServiceStateMachine.java:486)

      at com.bladelogic.om.infra.auth.service.AuthSvcStateMachineFactory.<init>(AuthSvcStateMachineFactory.java:48)

      ... 34 more

      Caused by: GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new ACCEPT credentials failed!)

      at sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Unknown Source)

      at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)

      at sun.security.jgss.GSSManagerImpl.getCredentialElement(Unknown Source)

      at sun.security.jgss.GSSCredentialImpl.add(Unknown Source)

      at sun.security.jgss.GSSCredentialImpl.<init>(Unknown Source)

      at sun.security.jgss.GSSManagerImpl.createCredential(Unknown Source)

      at com.bladelogic.sso.protocol.kerberos.KerberosServiceStateMachine.initCredentials(KerberosServiceStateMachine.java:476)

      ... 35 more

      Caused by: javax.security.auth.login.LoginException: No LoginModules configured for

      at javax.security.auth.login.LoginContext.init(Unknown Source)

      at javax.security.auth.login.LoginContext.<init>(Unknown Source)

      at sun.security.jgss.GSSUtil.login(Unknown Source)

      at sun.security.jgss.krb5.Krb5Util.getKeys(Unknown Source)

      at sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Unknown Source)

      at java.security.AccessController.doPrivileged(Native Method)

      ... 42 more

      [05 Jun 2012 14:46:41,316] [main] [INFO] [::] [] Starting Service Lifecycle Service...

      [05 Jun 2012 14:46:41,316] [main] [INFO] [::] [] Service Lifecycle Service started.

      [05 Jun 2012 14:46:41,316] [main] [INFO] [::] [] Registering remote services...

      [05 Jun 2012 14:46:41,316] [main] [INFO] [::] [] Shutdown processes complete

      [05 Jun 2012 14:46:41,316] [Thread-1] [INFO] [::] [] Undeploying

      [05 Jun 2012 14:46:41,348] [Thread-1] [ERROR] [::] [] java.rmi.NotBoundException: bladelogic.service.AppServerService

       

      From the C:\Program Files\BMC Software\BladeLogic\8.0\NSH\br\blappserv_login.conf

      com.bladelogic.auth.service.ADKerberosPasswordLogin {

      com.sun.security.auth.module.Krb5LoginModule required

      doNotPrompt=false

      useTicketCache=false

      debug=false;

      };

       

      From the C:\Program Files\BMC Software\BladeLogic\8.0\NSH\br\blappserv_krb5.conf

      [libdefaults]

      default_tkt_enctypes  = rc4-hmac

      default_tgs_enctypes = rc4-hmac

      [realms]

      EDC.DS1.USDA.GOV = {

      kdc = DChostname.EDC.DS1.USDA.GOV:88

      kdc = DChostname.EDC.DS1.USDA.GOV:88

      }

      [domain_realm]

      .edc.ds1.usda.gov = EDC.DS1.USDA.GOV

       

      All of this worked prior to upgrade and is unchanged.

        • 1. Cannot start appserver service after updating to 8.2.sp1
          Bill Robinson

          can you run:

           

          blasadmin -a show auth isadk

           

          did you have domain or adk login enabled for this before ? (it looks like domain from your login.conf)

          • 2. Re: Cannot start appserver service after updating to 8.2.sp1

            I believe we were setup for Domain authentication before.

             

            palau% blasadmin -a show auth isadk blasadmin now running against the following deployments: default, _template

            default:

            IsADKAuthEnabled:true

            _template: IsADKAuthEnabled:false

            palau%

             

            Russ has to run so I have taken over troubleshooting for the evening

            • 3. Re: Cannot start appserver service after updating to 8.2.sp1
              Bill Robinson

              Can you disabled the adk?

               

              Blasadmin –a set auth isadk false

               

              And try to start?

               

              It’s possible the upgrade enabled that setting.

              • 4. Re: Cannot start appserver service after updating to 8.2.sp1

                I am unable to login to the network shell to disable the setting

                 

                palau% blcred cred -acquire

                profile name: Blapp1-Palau

                username: BLAdmin

                password:

                Authentication failure: Cannot connect to "service:authsvc.bladelogic:blauth://p alau:9840" - Connection refused: connect palau%

                • 5. Re: Cannot start appserver service after updating to 8.2.sp1
                  Bill Robinson

                  You don’t need to login to the appserver or run network shell to run blasadmin if you are on the appserver.

                  • 6. Re: Cannot start appserver service after updating to 8.2.sp1

                    Command ran, app server still will not start

                     

                    C:\Windows\system32>blasadmin -a set auth isadk false blasadmin now running against the following deployments: default, _template

                     

                     

                    [05 Jun 2012 16:00:14,744] [main] [INFO] [::] [] Validating file server host (Palau)... [05 Jun 2012 16:00:17,322] [main] [ERROR] [::] [] File server validation failed because either the file server is not specified, cannot be contacted, or sufficient permissions are not granted on the file server agent to the appserver. Please verify the connectivity between appserver and file server and ensure that the appserver has read write permissions on the file server. [05 Jun 2012 16:00:17,322] [main] [ERROR] [::] [] Error installing to Start: name=bladelogic.service.FileManagerService state=Create com.bladelogic.om.infra.app.service.ServiceInitializationException: com.bladelogic.om.infra.mfw.util.BlException: File server validation failed because either the file server is not specified, cannot be contacted, or sufficient permissions are not granted on the file server agent to the appserver. Please verify the connectivity between appserver and file server and ensure that the appserver has read write permissions on the file server. at com.bladelogic.om.infra.app.service.file.FileManagerServiceImpl.start(FileManagerServiceImpl.java:226) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.jboss.joinpoint.plugins.reflect.ReflectMethodJoinPoint.dispatch(ReflectMethodJoinPoint.java:72) at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$DispatchJoinPoint.run(KernelControllerContextActions.java:631) at java.security.AccessController.doPrivileged(Native Method) at org.jboss.kernel.plugins.dependency.KernelControllerContextActions.dispatchJoinPoint(KernelControllerContextActions.java:99) at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$LifecycleAction.installAction(KernelControllerContextActions.java:452) at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$1.run(KernelControllerContextActions.java:156) at java.security.AccessController.doPrivileged(Native Method) at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$KernelControllerContextAction.install(KernelControllerContextActions.java:179) at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:226) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:593) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:346) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:438) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:379) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:225) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:151) at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBean(AbstractKernelDeployer.java:291) at com.bladelogic.om.infra.app.service.bootstrap.ServiceBeanXMLDeployer.deployBean(ServiceBeanXMLDeployer.java:38) at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBeans(AbstractKernelDeployer.java:261) at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deploy(AbstractKernelDeployer.java:117) at org.jboss.kernel.plugins.deployment.xml.BeanXMLDeployer.deploy(BeanXMLDeployer.java:91) at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.deploy(ServiceBootstrap.java:252) at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.deployService(ServiceBootstrap.java:199) at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.bootstrap(ServiceBootstrap.java:153) at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:91) at com.bladelogic.om.infra.app.service.bootstrap.ServiceBootstrap.startup(ServiceBootstrap.java:66) at com.bladelogic.om.infra.mfw.fw.BlManager.start(BlManager.java:881) at com.bladelogic.om.infra.mfw.fw.BlManager.main(BlManager.java:1401) Caused by: com.bladelogic.om.infra.mfw.util.BlException: File server validation failed because either the file server is not specified, cannot be contacted, or sufficient permissions are not granted on the file server agent to the appserver. Please verify the connectivity between appserver and file server and ensure that the appserver has read write permissions on the file server. at com.bladelogic.om.infra.app.service.file.FileManagerServiceImpl.validateCurrentFileServer(FileManagerServiceImpl.java:406) at com.bladelogic.om.infra.app.service.file.FileManagerServiceImpl.start(FileManagerServiceImpl.java:160) ... 32 more [05 Jun 2012 16:00:17,338] [main] [INFO] [::] [] Starting SSLConnectionManager Service... [05 Jun 2012 16:00:17,354] [main] [INFO] [::] [] SSL-Connections is starting the worker threads [05 Jun 2012 16:00:17,354] [SSL-Connections-Thread-0] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,354] [SSL-Connections-Thread-1] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,354] [main] [INFO] [::] [] SSL-Connections is reserving the controlport [05 Jun 2012 16:00:17,354] [SSL-Connections-Thread-5] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,369] [SSL-Connections-Thread-3] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,385] [SSL-Connections-Thread-4] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,369] [main] [INFO] [::] [] SSL-Connections is establishing control connection... [05 Jun 2012 16:00:17,385] [SSL-Connections-Thread-8] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,385] [SSL-Connections-Thread-6] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,385] [SSL-Connections-Thread-2] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,385] [SSL-Connections-Thread-7] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,385] [Thread-8] [INFO] [::] [] SSL-Connections is waiting for control socket ... [05 Jun 2012 16:00:17,385] [SSL-Connections-Thread-9] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,385] [Thread-8] [INFO] [::] [] SSL-Connections accepted the control socket connection [05 Jun 2012 16:00:17,385] [main] [INFO] [::] [] SSL-Connections established control connection. [05 Jun 2012 16:00:17,401] [main] [INFO] [::] [] SSL-Connections is starting the selector [05 Jun 2012 16:00:17,401] [main] [INFO] [::] [] SSL-Connections is starting the acceptor [05 Jun 2012 16:00:17,401] [main] [INFO] [::] [] SSLConnectionManager Service started. [05 Jun 2012 16:00:17,401] [main] [INFO] [::] [] Starting RBAC Authentication User Service... [05 Jun 2012 16:00:17,401] [main] [INFO] [::] [] RBAC Authentication User Service started. [05 Jun 2012 16:00:17,407] [main] [INFO] [::] [] Starting Authentication Service... [05 Jun 2012 16:00:17,425] [main] [INFO] [::] [] Authentication-Service is starting worker threads [05 Jun 2012 16:00:17,425] [main] [INFO] [::] [] Authentication Service started. [05 Jun 2012 16:00:17,441] [Authentication-Service-Thread-0] [INFO] [::] [] Authentication Service thread started [05 Jun 2012 16:00:17,441] [Authentication-Service-Thread-2] [INFO] [::] [] Authentication Service thread started [05 Jun 2012 16:00:17,441] [Authentication-Service-Thread-1] [INFO] [::] [] Authentication Service thread started [05 Jun 2012 16:00:17,441] [main] [INFO] [::] [] Starting ClientConnection Service... [05 Jun 2012 16:00:17,722] [main] [INFO] [::] [] Client-Connections is starting the worker threads [05 Jun 2012 16:00:17,737] [Client-Connections-Thread-0] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,737] [Client-Connections-Thread-1] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,737] [Client-Connections-Thread-3] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,737] [main] [INFO] [::] [] ClientConnection Service started. [05 Jun 2012 16:00:17,737] [Client-Connections-Thread-2] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,737] [Client-Connections-Thread-9] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,753] [Client-Connections-Thread-7] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,753] [Client-Connections-Thread-8] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,737] [Client-Connections-Thread-4] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,737] [Client-Connections-Thread-6] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,753] [Client-Connections-Thread-5] [INFO] [::] [] Client worker thread started. [05 Jun 2012 16:00:17,784] [main] [INFO] [::] [] Starting Service Lifecycle Service... [05 Jun 2012 16:00:17,784] [main] [INFO] [::] [] Service Lifecycle Service started. [05 Jun 2012 16:00:17,784] [main] [INFO] [::] [] Registering remote services... [05 Jun 2012 16:00:17,784] [main] [INFO] [::] [] Shutdown processes complete [05 Jun 2012 16:00:17,784] [Thread-1] [INFO] [::] [] Undeploying [05 Jun 2012 16:00:17,831] [Thread-1] [ERROR] [::] [] java.rmi.NotBoundException: bladelogic.service.AppServerService

                    • 7. Re: Cannot start appserver service after updating to 8.2.sp1
                      Bill Robinson

                      Is the file server agent running?

                       

                      If it is, did you push acls to it (there would be entries in the c:\windows\rsc\users file)

                       

                      If you did, can you put this in the users.local:

                      System:System rw,map=Administrator

                       

                      (replace Administrator w/ the local admin)

                      • 8. Re: Cannot start appserver service after updating to 8.2.sp1

                        My RSCD agent is running on the file server. here are the contents of my fils.

                         

                        exports

                        * rw,user=NITCAdmin

                         

                         

                        users

                        # # This file was automatically generated by the Configuration Manager RBAC console. # Any changes to this file will be lost upon the next update by the RBAC # console. Local changes should be made in the users.local file # # The special characters listed below get automatically encoded. For example, # "Configuration Manager Administrator" becomes "Configuration%20Manager%20Administrator". # '%' --> %25 # ',' --> %2c # ':' --> %3a # '#' --> %23 # ' ' --> %20 # TAB --> %09 #

                         

                         

                         

                        users.local

                        # #  Copyright (c) 2001-2009 BladeLogic, Inc. #      -- All Rights Reserved -- # # This file contains a list of user permission overrides. The permissions # defined in this file will override any associated permissions defined in the # "exports" or "users" file. # # Please read the BladeLogicAdministration.pdf for details on how to use this # file. #

                        BLAdmins:BLAdmin    rw,map=NITCAdmin

                        System:System rw,map=NITCAdmin

                        • 9. Re: Cannot start appserver service after updating to 8.2.sp1
                          Bill Robinson

                          can you run 'agentinfo' against the file server from the appserver?

                           

                          what version is the file server agent?

                          • 10. Re: Cannot start appserver service after updating to 8.2.sp1

                            I did notice something weird with the local users in the system.

                             

                            BL.bmp

                            • 11. Re: Cannot start appserver service after updating to 8.2.sp1

                              C:\Windows\system32>agentinfo palau Can't access host "palau": Connection refused

                              • 12. Cannot start appserver service after updating to 8.2.sp1

                                My fileserver and appserver are on the same host.

                                • 13. Re: Cannot start appserver service after updating to 8.2.sp1
                                  Bill Robinson

                                  What's in the rscd.log? Is the agents oceans running?

                                  • 14. Re: Cannot start appserver service after updating to 8.2.sp1

                                    The local BladeLogicRSCD account was disabled during the upgrade. Once I enabled the account the application server started up.

                                     

                                    Thank you for your help

                                    1 2 Previous Next