8 Replies Latest reply on Aug 17, 2012 8:10 AM by Bill Robinson

    7.6 Application server certificate expired

      Hi,

       

      can anyone help with a problem were experiencing currently the certificate on the application server has expired and reading through this KM KA308901 you should be able to re-generate the certificate and resolve the problem. However after following the instructions the application server will now not start and has this error in the appserver.log, has anyone fixed this issue before?

       

      [16 Aug 2012 01:12:44,595] [main] [INFO] [::] [] Starting Agent Service...

      [16 Aug 2012 01:12:44,595] [main] [INFO] [::] [] Agent Service started.

      [16 Aug 2012 01:12:44,891] [main] [INFO] [::] [] Starting Configuration Service...

      [16 Aug 2012 01:12:44,923] [main] [INFO] [::] [] Configuration Service started.

      [16 Aug 2012 01:12:44,954] [main] [INFO] [::] [] Starting Content Importer Service...

      [16 Aug 2012 01:12:44,954] [main] [INFO] [::] [] Content Importer Service started.

      [16 Aug 2012 01:12:45,048] [main] [INFO] [::] [] Loading JNI libaries...

      [16 Aug 2012 01:12:45,110] [main] [INFO] [::] [] Loaded JNI library BlAsJni

      [16 Aug 2012 01:12:45,110] [main] [INFO] [::] [] Loaded JNI library XJNIService

      [16 Aug 2012 01:12:45,251] [main] [INFO] [::] [] Skipped Loading JNI library Atrium Core

      [16 Aug 2012 01:12:45,251] [main] [INFO] [::] [] Initializing JNI Configuration...

      [16 Aug 2012 01:12:45,251] [main] [INFO] [::] [] Initialized JNI library BlAsJni

      [16 Aug 2012 01:12:45,313] [main] [INFO] [::] [] Initialized JNI library XJNIService

      [16 Aug 2012 01:12:45,313] [main] [INFO] [::] [] Initialized JNI library Atrium Core

      [16 Aug 2012 01:12:45,313] [main] [INFO] [::] [] JNI Configuration complete.

      [16 Aug 2012 01:12:45,360] [main] [INFO] [::] [] Starting LDAP Connection Service...

      [16 Aug 2012 01:12:45,360] [main] [INFO] [::] [] LDAP Connection Service started.

      [16 Aug 2012 01:12:45,360] [main] [INFO] [::] [] Starting LDAP Service...

      [16 Aug 2012 01:12:45,485] [main] [INFO] [::] [] LDAP service is not configured.

      [16 Aug 2012 01:12:45,485] [main] [INFO] [::] [] LDAP Service started.

      [16 Aug 2012 01:12:45,501] [main] [ERROR] [::] [] Error installing to Start: name=bladelogic.service.management.ServerSocketFactory state=Create

      1. com.bladelogic.app.service.ServiceInitializationException: java.io.IOException

                      at com.bladelogic.app.service.client.ServerSocketFactory.start(ServerSocketFactory.java:71)

                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

                      at java.lang.reflect.Method.invoke(Unknown Source)

                      at org.jboss.joinpoint.plugins.reflect.ReflectMethodJoinPoint.dispatch(ReflectMethodJoinPoint.java:72)

                      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$DispatchJoinPoint.run(KernelControllerContextActions.java:631)

                      at java.security.AccessController.doPrivileged(Native Method)

                      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions.dispatchJoinPoint(KernelControllerContextActions.java:99)

                      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$LifecycleAction.installAction(KernelControllerContextActions.java:452)

                      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$1.run(KernelControllerContextActions.java:156)

                      at java.security.AccessController.doPrivileged(Native Method)

                      at org.jboss.kernel.plugins.dependency.KernelControllerContextActions$KernelControllerContextAction.install(KernelControllerContextActions.java:179)

                      at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

                      at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:226)

                      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:593)

                      at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:346)

                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:438)

                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:379)

                      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:225)

                      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:151)

                      at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBean(AbstractKernelDeployer.java:291)

                      at com.bladelogic.app.service.bootstrap.ServiceBeanXMLDeployer.deployBean(ServiceBeanXMLDeployer.java:38)

                      at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBeans(AbstractKernelDeployer.java:261)

                      at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deploy(AbstractKernelDeployer.java:117)

                      at org.jboss.kernel.plugins.deployment.xml.BeanXMLDeployer.deploy(BeanXMLDeployer.java:91)

                      at com.bladelogic.app.service.bootstrap.ServiceBootstrap.deploy(ServiceBootstrap.java:240)

                      at com.bladelogic.app.service.bootstrap.ServiceBootstrap.deployService(ServiceBootstrap.java:187)

                      at com.bladelogic.app.service.bootstrap.ServiceBootstrap.bootstrap(ServiceBootstrap.java:141)

                      at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:91)

                      at com.bladelogic.app.service.bootstrap.ServiceBootstrap.startup(ServiceBootstrap.java:66)

                      at com.bladelogic.mfw.fw.BlManager.start(BlManager.java:802)

                      at com.bladelogic.mfw.fw.BlManager.main(BlManager.java:1291)

      Caused by: java.io.IOException

                      at com.bladelogic.app.service.client.ServerSocketFactory.createSocketFactory(ServerSocketFactory.java:162)

                      at com.bladelogic.app.service.client.ServerSocketFactory.start(ServerSocketFactory.java:67)

                      ... 32 more

      Caused by: com.bladelogic.mfw.util.BlException: Failed to find resource: default/e:/BMCBladeLogic/OM/br/bladelogic.keystore

                      at com.bladelogic.app.util.AppServerUtil.loadDeploymentFile(AppServerUtil.java:296)

                      at com.bladelogic.security.cert.AppServerKeyStore.getInstance(AppServerKeyStore.java:165)

                      at com.bladelogic.app.service.client.ServerSocketFactory.getAppServerKeyStore(ServerSocketFactory.java:176)

                      at com.bladelogic.app.service.client.ServerSocketFactory.createSocketFactory(ServerSocketFactory.java:149)

                      ... 33 more

      [16 Aug 2012 01:12:45,532] [main] [INFO] [::] [] Starting Model Graph Service...

      [16 Aug 2012 01:12:47,954] [main] [INFO] [::] [] Model Graph Service started.

      [16 Aug 2012 01:12:48,048] [main] [INFO] [::] [] Starting ObjectExporter Service...

      [16 Aug 2012 01:12:48,048] [main] [INFO] [::] [] Verifying exporter classes...

      [16 Aug 2012 01:12:48,048] [main] [INFO] [::] [] Object exporters verified.

      [16 Aug 2012 01:12:48,048] [main] [INFO] [::] [] ObjectExporter Service started.

      [16 Aug 2012 01:12:48,063] [main] [INFO] [::] [] Starting Port Provider...

      [16 Aug 2012 01:12:48,063] [main] [INFO] [::] [] Port Provider started.

      [16 Aug 2012 01:12:48,110] [main] [INFO] [::] [] Starting Sizing Service...

      [16 Aug 2012 01:12:48,110] [main] [INFO] [::] [] Sizing Service started.

      [16 Aug 2012 01:12:48,110] [main] [INFO] [::] [] Starting Caching Service...

      [16 Aug 2012 01:12:48,110] [main] [INFO] [::] [] Caching Service started.

      [16 Aug 2012 01:12:48,110] [main] [INFO] [::] [] Starting Session Service...

      [16 Aug 2012 01:12:48,188] [main] [INFO] [::] [] Session Service started.

      [16 Aug 2012 01:12:48,219] [main] [INFO] [::] [] Starting Task Scheduling Service...

      [16 Aug 2012 01:12:48,219] [main] [INFO] [::] [] Task Scheduling Service started.

      [16 Aug 2012 01:12:48,219] [main] [INFO] [::] [] Starting Directory Service...

      [16 Aug 2012 01:12:48,235] [main] [INFO] [::] [] Starting RMI registry on host: CSCCBLAPS01

      [16 Aug 2012 01:12:48,251] [main] [INFO] [::] [] Directory Service started.

      [16 Aug 2012 01:12:48,251] [main] [INFO] [::] [] Registering remote services...

      [16 Aug 2012 01:12:48,251] [main] [INFO] [::] [] Shutdown processes complete

      [16 Aug 2012 01:12:48,251] [Thread-0] [INFO] [::] [] Undeploying

      [16 Aug 2012 01:12:48,251] [Thread-0] [ERROR] [::] [] java.lang.NullPointerException

      [16 Aug 2012 01:12:48,282] [Thread-0] [ERROR] [::] [] java.rmi.NotBoundException: bladelogic.service.AppServerService

       

      cheers Ben

        • 1. Re: 7.6 Application server certificate expired
          Bill Robinson

          it's funny, i was just working on updating some of these KAs. 

           

          can you run this:

           

          blasadmin -a show app certstore

           

          from the output it looks like the appserver is looking in the wrong place for the file

          • 2. Re: 7.6 Application server certificate expired

            Hi Bill,

             

            as requested:

             

            C:\Documents and Settings\adc78874>blasadmin -a show app certstore

            csccblaps01_b:

            CertStore:bladelogic.keystore

            _template:

            CertStore:bladelogic.keystore

            default:

            CertStore:e:/BMCBladeLogic/OM/br/bladelogic.keystore

             

            This is something ive seen from a different link i think you had mentioned so i ran this as well:

             

            blasadmin now running against deployment: default

            Bladelogic Admin Utility; Version 7.6.0.212.

            Copyright (c) 2001-2008 BladeLogic, Inc. All rights reserved.

            Type 'help' for Bladmin help.

            bladmin>show app all

            [AppServer]

            AppServerInstrumentationFilePath:

            AppServerInstrumentationRolloverCount:10

            AppServerInstrumentationRolloverSize:10000

            AppServerName:csccblaps01

            AppSvcPort:9841

            AssetPathCacheMaxSize:50

            AuditCacheMaxSize:50

            BlTargetJobManagerTimeout:90

            CLRProxyPort:

            CertPasswd:KNQKEPNKTWTUMLXOPUKTOXKATZKVEVKNEONWPKEWTAKABMAPTOO

            CertStore:e:/BMCBladeLogic/OM/br/bladelogic.keystore

            ComplianceResultMaxNumberOfAssets:100

            ComponentCacheMaxSize:100

            ConditionFormatLocaleCountry:

            ConditionFormatLocaleLanguage:

            ConditionFormatLocaleVariant:

            DisplayName:CSCCBLAPS01

            EnableAtriumIntegration:

            EnableInstrumentation:false

            EnableProxyInspection:true

            EnableSessionBasedCaching:true

            FileSystemObjectCacheMaxSize:50000

            HTTPProxyName:155.131.254.69

            HTTPProxyPassword:OUXQQWBOKPZWKTLULNMWTQNVBWLXPKBEQLTQXNVVPLXA

            NP

            HTTPProxyPort:9999

            HTTPProxyUser:anonymous

            Hostname:

            IdleConnectionPruneTime:120

            IdleNshProxyPruneTime:120

            KRB5ProxyPort:

            Krb5Config:

            Krb5LoginConfig:

            Krb5Port:

            MaxClientContexts:200

            MaxJMXConnections:20

            MaxJobThreads:5

            MaxJobs:20

            MaxLightweightWorkItemThreads:0

            MaxNshProxyContexts:20

            MaxNshProxyThreads:3

            MaxPort:9899

            MaxTimeForCancelToFinish:10

            MaxWorkItemThreads:50

            MaxWorkerThreads:10

            MinPort:9850

            MultiAppServerEnabled:

            NshProxyKrb5Config:

            NshProxyKrb5LoginConfig:

            NshProxyMaxThreadIdleTime:500

            NshProxySocketConnectTimeout:60

            NshProxySocketOperationTimeout:7200

            NshProxySocketsBindAddress:all

            PWDStore:

            PropagateWorkItemTimeout:true

            ProxySvcPort:

            RegistryPort:9836

            RemoteServerTimeout:60

            RequireClientAuthentication:true

            RestartIdleProvisionJobs:false

            SRPPort:9829

            SRPProxyPort:

            SSLPort:9831

            ServerMonitorInterval:10

            SnapshotCacheMaxSize:100

            SocketConnectTimeout:60

            SocketOperationTimeout:7200

            SocketTimeout:600

            SocketsBindAddress:all

            TemplateCacheMaxSize:100

            UseSSLSockets:false

            ValidateClientIpAddress:

            ValidateRequestURL:

            VersionCompatibilityCheck:minor

            bladmin>

            • 3. Re: 7.6 Application server certificate expired
              Bill Robinson

              run this:

               

              blasadmin -a set app certstore bladelogic.keystore

              blasadmin -a set proc keystorepassword bladelogic

              blasadmin -s _spawner set app certstore bladelogic.keystore

              blasadmin -s _spawner set proc keystorepassword bladelogic.keystore

               

               

              the 1st command should fix it.  the 'default' instance is looking in the wrong place for the keystore (the 7.4 location).  the other commands are just to force it for the spawner.

               

              do you have pxe installed on this box ?

              • 4. Re: 7.6 Application server certificate expired

                Hi Bill,

                 

                your a legend, thats resolved the problem and no the PXE runs on a different server.

                 

                cheers Ben

                • 5. Re: 7.6 Application server certificate expired

                  Hi Bill,

                   

                  Out of interest where did you see that the directory was wrong?

                   

                  Cheers Ben

                  • 6. Re: 7.6 Application server certificate expired
                    Bill Robinson

                    toward the bottom of that stack trace:

                     

                    Caused by: com.bladelogic.mfw.util.BlException: Failed to find resource: default/e:/BMCBladeLogic/OM/br/bladelogic.keystore

                    • 7. Re: 7.6 Application server certificate expired

                      Hi Bill,

                       

                      I saw that error and checked the file was located there so couldn't understand why this had occurred, you mentioned above that the location was 7.4 file location is there a way you could tell or somewhere you referenced for this information?

                       

                      It would be useful to know if we ever experience this issue again and thanks again for the prompt response.

                       

                       

                      Cheers Ben

                      • 8. Re: 7.6 Application server certificate expired
                        Bill Robinson

                        i think the path was not correct - default/e:/BMCBladeLogic/OM/br/bladelogic.keystore does not look like a valid path, i'm not sure why that was even working. 

                         

                        i just know from past experience - in 7.4 we did not have the concept of appserver instances so everything was kept in 'br'.  in 7.5 we added the concerpt of multiple instances on the same box and moved all of the 'deployment' information into the 'br/deployments' dir, including the certificate.  i'm sure that's documetned back in the 7.4 docs and it should be noted that in the current version all of the deployment specific files should be in the deployments dir.

                         

                        sometimes the upgrade from 7.4 to 7.6 and higher didn't convert the configuration properly.