1 2 Previous Next 16 Replies Latest reply on Apr 8, 2016 12:57 PM by Bill Robinson

    App server connection error

    Neal Meagher

      I have MAS linux environment 12 servers(first 5 all servers) other 7 are jobs ect. When I log into server 1 go to Infrastructure mgmt, all server are online.

      Server 2 shows Server 4 and 5 not connected

      Server 3 and server 4 show all ok

      server 5 shows 03 and 11 not connected

       

      The bladelogic.keystore files are all the same for all deployments. I do Have a load balancer that has a profile. Im logging into these server individually and getting these problems. All the correct ports are open either listening or established.  I can browse and verify all app servers.

       

      These are the app server log errors:

       

       

      [06 Apr 2016 10:54:45,893] [Client-Connections-Thread-3] [WARN] [Anonymous:Anonymous:127.0.0.1] [Client] Error authorizing the connection

      [06 Apr 2016 10:54:45,893] [Client-Connections-Thread-3] [INFO] [Anonymous:Anonymous:127.0.0.1] [Client] Connection disconnecting: id = 247927

      [06 Apr 2016 10:54:46,923] [Nsh-Proxy-Thread-124] [WARN] [Anonymous:Anonymous:127.0.0.1] [BLSSOPROXY] javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

      com.bladelogic.sso.engine.ClientDisconnectedException: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

        at com.bladelogic.sso.engine.StreamMessaging.receiveMessage(StreamMessaging.java:123)

        at com.bladelogic.sso.engine.StreamMessaging.receiveToken(StreamMessaging.java:203)

        at com.bladelogic.om.infra.mfw.net.BlSessionServerConnection.authenticate(BlSessionServerConnection.java:176)

        at com.bladelogic.om.infra.mfw.net.BlSessionNshServerConnection.internalHandshake(BlSessionNshServerConnection.java:78)

        at com.bladelogic.om.infra.mfw.net.BlSessionNshServerConnection.doHandshake(BlSessionNshServerConnection.java:50)

        at com.bladelogic.om.infra.mfw.fw.BlSessionNshProxyPair.setupClient(BlSessionNshProxyPair.java:104)

        at com.bladelogic.om.infra.mfw.fw.BlSessionNshProxyPair.init(BlSessionNshProxyPair.java:75)

        at com.bladelogic.om.infra.mfw.fw.NshProxyWorkerThread.execute(NshProxyWorkerThread.java:106)

        at com.bladelogic.om.infra.mfw.fw.NshProxyWorkerThread.execute(NshProxyWorkerThread.java:17)

        at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

      Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

        at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)

        at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)

        at sun.security.ssl.AppInputStream.read(Unknown Source)

        at java.io.BufferedInputStream.fill(Unknown Source)

        at java.io.BufferedInputStream.read(Unknown Source)

        at com.bladelogic.sso.engine.StreamMessaging.receiveMessage(StreamMessaging.java:119)

        ... 9 more

      Caused by: java.io.EOFException: SSL peer shut down incorrectly

        at sun.security.ssl.InputRecord.read(Unknown Source)

        ... 16 more

      [06 Apr 2016 10:54:46,923] [Nsh-Proxy-Thread-124] [INFO] [Anonymous:Anonymous:127.0.0.1] [BLSSOPROXY] failure establishing session with proxy service

      [06 Apr 2016 10:54:49,129] [Authentication-Service-Thread-1] [WARN] [::127.0.0.1] [Appserver] Connection closed by /127.0.0.1:55024 before pre-authentication handshake could be completed. This could be a probe message from Client to Authentication Server.

      [06 Apr 2016 10:54:49,129] [Authentication-Service-Thread-1] [INFO] [::127.0.0.1] [Appserver] Authentication Connection closed

      [06 Apr 2016 10:54:49,129] [Client-Connections-Thread-2] [WARN] [Anonymous:Anonymous:127.0.0.1] [Client] Connection closed by /127.0.0.1:54805 before pre-authentication handshake could be completed. This could be a probe message from Client to Authentication Server.

      [06 Apr 2016 10:54:49,129] [Client-Connections-Thread-2] [WARN] [Anonymous:Anonymous:127.0.0.1] [Client] Error authorizing the connection

      [06 Apr 2016 10:54:49,129] [Client-Connections-Thread-2] [INFO] [Anonymous:Anonymous:127.0.0.1] [Client] Connection disconnecting: id = 247929

      [06 Apr 2016 10:55:10,906] [Authentication-Service-Thread-2] [WARN] [::127.0.0.1] [Appserver] Connection closed by /127.0.0.1:55036 before pre-authentication handshake could be completed. This could be a probe message from Client to Authentication Server.

      [06 Apr 2016 10:55:10,907] [Authentication-Service-Thread-2] [INFO] [::127.0.0.1] [Appserver] Authentication Connection closed

      [06 Apr 2016 10:55:15,912] [Client-Connections-Thread-8] [WARN] [Anonymous:Anonymous:127.0.0.1] [Client] Connection closed by /127.0.0.1:54819 before pre-authentication handshake could be completed. This could be a probe message from Client to Authentication Server.

      [06 Apr 2016 10:55:15,913] [Client-Connections-Thread-8] [WARN] [Anonymous:Anonymous:127.0.0.1] [Client] Error authorizing the connection

      [06 Apr 2016 10:55:15,913] [Client-Connections-Thread-8] [INFO] [Anonymous:Anonymous:127.0.0.1] [Client] Connection disconnecting: id = 247930

      [06 Apr 2016 10:55:16,942] [Nsh-Proxy-Thread-121] [WARN] [Anonymous:Anonymous:127.0.0.1] [BLSSOPROXY] javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

      com.bladelogic.sso.engine.ClientDisconnectedException: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

        at com.bladelogic.sso.engine.StreamMessaging.receiveMessage(StreamMessaging.java:123)

        at com.bladelogic.sso.engine.StreamMessaging.receiveToken(StreamMessaging.java:203)

        at com.bladelogic.om.infra.mfw.net.BlSessionServerConnection.authenticate(BlSessionServerConnection.java:176)

        at com.bladelogic.om.infra.mfw.net.BlSessionNshServerConnection.internalHandshake(BlSessionNshServerConnection.java:78)

        at com.bladelogic.om.infra.mfw.net.BlSessionNshServerConnection.doHandshake(BlSessionNshServerConnection.java:50)

        at com.bladelogic.om.infra.mfw.fw.BlSessionNshProxyPair.setupClient(BlSessionNshProxyPair.java:104)

        at com.bladelogic.om.infra.mfw.fw.BlSessionNshProxyPair.init(BlSessionNshProxyPair.java:75)

        at com.bladelogic.om.infra.mfw.fw.NshProxyWorkerThread.execute(NshProxyWorkerThread.java:106)

        at com.bladelogic.om.infra.mfw.fw.NshProxyWorkerThread.execute(NshProxyWorkerThread.java:17)

        at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

      Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

        at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)

        at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)

        at sun.security.ssl.AppInputStream.read(Unknown Source)

        at java.io.BufferedInputStream.fill(Unknown Source)

        at java.io.BufferedInputStream.read(Unknown Source)

        at com.bladelogic.sso.engine.StreamMessaging.receiveMessage(StreamMessaging.java:119)

        ... 9 more

      Caused by: java.io.EOFException: SSL peer shut down incorrectly

        at sun.security.ssl.InputRecord.read(Unknown Source)

        • 1. Re: App server connection error
          Neal Meagher

          Does know the command to show MaxJMXConnections? I cant seem to findthe right BLASAdmin command.

           

          Where to increase?

          • 2. Re: App server connection error
            Bill Robinson

            maxjmxconnections is in the appserver section of blasadmin:

            blasadmin -s instancename show app maxjmxconnections

             

            where do you see the appservers marking each other as down?

            you should also check:

            minport->maxport and registryport are reachable by all instances from all instances for whatever the values are on each server.

            the time zones are set the same across all boxes and the db server

            the times are in sync between all appservers and db

             

            I do Have a load balancer that has a profile.

            -> what ports are load balanced ?

            -> when you login from your client (which is where?) you are going through a lb or connecting directly ?

            -> have you set the appserviceurl on any of the config/all instances ?

             

            Im logging into these server individually and getting these problems.

            -> from where ?

            All the correct ports are open either listening or established.  I can browse and verify all app servers.

            -> via the rscd ?  that has nothing to do w/ the appserver service.

             

            so:

            When I log into server 1 go to Infrastructure mgmt, all server are online.

            Server 2 shows Server 4 and 5 not connected -> from where?  when you login into server 2?  how are you doing that ?  through the lb ?  directly ? 

             

            what does "not connected" mean ?  screenshot ?

            • 3. Re: App server connection error
              Neal Meagher

              I log into app server two. And go to infrastructure mgmt and go to app servers and get a connection error.

               

              ports 9875 9840 9843 9876 are being load balanced.

               

              I log into the console through a profile setup to connect to load balancer then it it find a app server and connects to open app server. Forwarding

               

               

              When i say logging on, I mean through the console on my PC.

               

              Im logging in directly to each server checking infra mgmt

               

               

              Server Exit code Status Output

               

               

               

               

               

              -----------------------------------------------------------------------------------------

              blasadmin now running against the following deployments: _template, default

              _template:

              UseSSLSockets:false

              default:

              UseSSLSockets:false

              -----------------------------------------------------------------------------------------

              blasadmin now running against the following deployments: _template, default

              [AppServer]

              AgentInstallerJobAgentDownTimeOut:2

              AppServerInstrumentationFilePath:

              AppServerInstrumentationRolloverCount:10

              AppServerInstrumentationRolloverSize:10000

              AppServerName:

              AppSvcPort:9841

              AssetClassAclDelay:300000

              AssetClassGlobalCacheEnabled:

              AssetClassHardReferenceMaximum:100

              AssetClassJobCacheEnabled:

              AssetClassJobQueryCacheEnabled:

              AssetClassSessionCacheEnabled:

              AssetClassSessionQueryCacheEnabled:

              AssetPathCacheMaxSize:5000

              AuditCacheMaxSize:50

              BDSSAPostConfigURL:

              BDSSAServerURL:https://se113932.fg.rbc.com

              BlTargetJobManagerTimeout:90

              BroadcastMessagesPollingInterval:2

              CLRProxyPort:0

              CertPasswd:LVZOZZZLZMENQTKZLBOBTNVULVTTQUNZEPZXOTTTLKLMPZMLUXQPTQMOOLKLZXKB

              CertStore:bladelogic.keystore

              CommandTimeoutInSecForCompliance:60

              ComplianceResultMaxNumberOfAssets:100

              ComponentCacheMaxSize:100

              ComponentHeaderCacheMaxSize:500

              ConditionFormatLocaleCountry:

              ConditionFormatLocaleLanguage:

              ConditionFormatLocaleVariant:

              DaalPluginImportWaitTimeout:

              DefaultRestVersion:8.1.02

              DisableIpv6SupportForSysInfo:

              DisplayName:

              EnableARFAndASRExecution:false

              EnableAsyncExecution:

              EnableAtriumIntegration:false

              EnableDataPrunerForCompliance:true

              EnableIDRACService:false

              EnableInstrumentation:false

              EnableProxyInspection:true

              EnableSessionBasedCaching:true

              EnableTomcatService:true

              EnableWebServices:true

              FileSystemObjectCacheMaxSize:50000

              GMOCacheMaxSize:500

              HTTPProxyName:

              HTTPProxyPassword:

              HTTPProxyPort:

              HTTPProxyType:

              HTTPProxyUser:

              Hostname:

              HttpsIDRACPortNumber:4433

              HttpsPortNumber:9843

              IdleConnectionPruneTime:120

              IdleNshProxyPruneTime:120

              JMXConnectionTimeout:20

              JVMArgs:

              KeepAliveQueryTimeout:5

              LauncherStatusRefreshInterval:30

              LogItemWriterQueueSize:10000

              MaxApprovalThreads:3

              MaxClientContexts:200

              MaxConnectionAttempts:10

              MaxHeapSize:

              MaxJMXConnections:20

              MaxJobThreads:5

              MaxJobWorkItemCacheMemorySizeInkB:8000

              MaxJobs:20

              MaxLightweightWorkItemThreads:0

              MaxNshProxyContexts:50

              MaxNshProxyThreads:15

              MaxNumberOfDashboardReports:90

              MaxPort:9899

              MaxRESTNotifyThreads:12

              MaxTimeForCancelToFinish:10

              MaxWorkItemThreads:50

              MaxWorkerThreads:10

              MemoryMonitorRefreshRate:60

              MinPort:9850

              NTLMProxyDomain:

              NshProxyMaxThreadIdleTime:500

              NshProxySocketConnectTimeout:60

              NshProxySocketOperationTimeout:7200

              NumLogItemWriterThreads:5

              PSCCacheMaxSize:500

              PSICacheMaxSize:500

              PSIValueCacheMaxSize:1000

              PWDStore:

              PatternCacheMaxSize:500

              PropagateWorkItemTimeout:true

              ProxySvcPort:0

              RegistryPort:9836

              RemoteServerTimeout:300

              RequireClientAuthentication:true

              RestAssetAttributesUseInternalNames:false

              RestartIdleProvisionJobs:false

              SSCOLoaderCacheMaxSize:25

              SSLPort:0

              ServerMonitorInterval:20

              ServerSocketBacklog:50

              SnapshotCacheMaxSize:100

              SocketConnectTimeout:60

              SocketOperationTimeout:7200

              SocketTimeout:600

              SocketsBindAddress:all

              SslBackwardCompatibility:true

              TemplateCacheMaxSize:100

              UseSSLSockets:false

              ValidateClientIpAddress:true

              ValidateRequestURL:true

              VersionCompatibilityCheck:minor

              XMLConfigFileTransferChunkSize:100

              -----------------------------------------------------------------------------------------

              blasadmin now running against the following deployments: default, _template

              default:

              MaxClientContexts:200

              _template:

              MaxClientContexts:200

              -----------------------------------------------------------------------------------------

              blasadmin now running against the following deployments: _template, default

              _template:

              MaxAuthSvcThreads:3

              default:

              MaxAuthSvcThreads:3

              -----------------------------------------------------------------------------------------

              blasadmin now running against the following deployments: default, _template

              default:

              AuthSvcSocketTimeout:180

              _template:

              AuthSvcSocketTimeout:90

              -----------------------------------------------------------------------------------------

              #

              #Tue Oct 27 13:39:37 EDT 2015

              AssetPathCacheMaxSize=5000

              ConditionFormatLocaleVariant=

              ConditionFormatLocaleLanguage=

              PWDStore=

              HTTPProxyPassword=

              ComponentHeaderCacheMaxSize=500

              MaxHeapSize=6144M

              TemplateCacheMaxSize=100

              AuditCacheMaxSize=50

              ComponentCacheMaxSize=100

              HTTPProxyUser=

              MaxRESTNotifyThreads=12

              HTTPProxyName=

              PSIValueCacheMaxSize=1000

              AssetClassAclDelay=300000

              PSCCacheMaxSize=500

              PSICacheMaxSize=500

              ConditionFormatLocaleCountry=

              JMXConnectionTimeout=20

              GMOCacheMaxSize=500

              EnabledCipherSuitesForWebservices=TLS_RSA_WITH_AES_256_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA

              MaxJMXConnections=20

              LauncherStatusRefreshInterval=30

              XMLConfigFileTransferChunkSize=100

              JVMArgs=-XX\:+UseCompressedOops

              EnabledCipherSuites=TLS_RSA_WITH_AES_256_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA

              HTTPProxyPort=

              ComplianceResultMaxNumberOfAssets=100

              LogItemWriterQueueSize=10000

              NumLogItemWriterThreads=5

              FileSystemObjectCacheMaxSize=50000

              AssetClassHardReferenceMaximum=100

              SnapshotCacheMaxSize=100

              PatternCacheMaxSize=500

              SSCOLoaderCacheMaxSize=25

              • 4. Re: App server connection error
                Bill Robinson

                ports 9875 9840 9843 9876 are being load balanced

                -> ok, these correspond to what services on what server ?  normally 9840 is the auth service, 9843 is webservices and 9875/7 are part of the transitory port range used for inter-appserver communication so they would not be user facing and not part of a vip.  but they could be configured to do anything.

                 

                minport->maxport and registryport are reachable by all instances from all instances for whatever the values are on each server (also forgot 9700-9702)?

                • 5. Re: App server connection error
                  Neal Meagher

                  9876 and 9875 are mapped to 9840 and 9843 they correspond to what they are OOB.

                   

                  MinPort:9850

                  MaxPort:9899

                   

                  tcp        0      0 0.0.0.0:9854                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:17887               0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:10050               0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:5666                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9700                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9701                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9702                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:42439               0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:44616               0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:36520               0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:8649                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:1002                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:1067                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9836                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:4750                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9838                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9840                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9841                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:52081               0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9842                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9843                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9750                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:11960               0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:11961               0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9850                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9851                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9852                0.0.0.0:*                   LISTEN     

                  tcp        0      0 0.0.0.0:9853                0.0.0.0:*                   LISTEN     

                     tcp        0      0 127.0.0.1:59806             127.0.0.1:11960             ESTABLISHED

                  tcp        0      0 10.208.98.107:9702          10.208.98.96:50467          ESTABLISHED

                  tcp        0      0 10.208.98.107:9702          10.208.98.100:34689         ESTABLISHED

                   

                  tcp        0      0 10.208.98.107:9838          10.208.98.107:50170         ESTABLISHED

                  tcp        0      0 127.0.0.1:11960             127.0.0.1:37531             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:41477         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:39137         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:38577         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:45977         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:41328         TIME_WAIT  

                  tcp        0     15 10.208.98.107:46146         10.208.98.107:1067          ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:39377         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:51536         TIME_WAIT  

                  tcp        0      0 10.208.98.107:9851          10.208.98.107:47697         ESTABLISHED

                  tcp        0      0 127.0.0.1:11960             127.0.0.1:46522             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:46130         TIME_WAIT  

                  tcp        0      0 127.0.0.1:46153             127.0.0.1:11960             ESTABLISHED

                  tcp        0      0 10.208.98.107:58102         10.208.98.99:9850           TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:57904         TIME_WAIT  

                  tcp        0      0 127.0.0.1:11960             127.0.0.1:53326             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:38950         TIME_WAIT  

                  tcp        0      0 127.0.0.1:11961             127.0.0.1:47799             ESTABLISHED

                  tcp        0      0 127.0.0.1:11961             127.0.0.1:43676             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:54876         TIME_WAIT  

                  tcp        0      0 127.0.0.1:50662             127.0.0.1:11961             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:41083         TIME_WAIT  

                  tcp        0      0 10.208.98.107:9851          10.208.98.107:47698         ESTABLISHED

                  tcp        0      0 127.0.0.1:11961             127.0.0.1:35916             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:42714         TIME_WAIT  

                  tcp        0      0 10.208.98.107:9850          10.208.98.102:46067         ESTABLISHED

                  tcp        0      0 10.208.98.107:9702          10.208.98.98:55758          ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:40465         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:37548         TIME_WAIT  

                  tcp        0    212 10.208.98.107:39731         10.208.98.107:44616         ESTABLISHED

                  tcp        0      0 127.0.0.1:11961             127.0.0.1:49298             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:39942         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:40899         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:38498         TIME_WAIT  

                  tcp        0      0 10.208.98.107:38916         10.208.98.107:9853          ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:60553         TIME_WAIT  

                  tcp        0      0 10.208.98.107:42477         10.208.98.107:9854          ESTABLISHED

                  tcp        0      0 10.208.98.107:9850          10.208.98.98:32770          ESTABLISHED

                  tcp        0      0 10.208.98.107:46733         10.208.98.107:9836          TIME_WAIT  

                  tcp        0      0 10.208.98.107:54975         10.208.98.102:9850          TIME_WAIT  

                  tcp        0      0 127.0.0.1:44572             127.0.0.1:11961             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:47301         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:39006         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:47823         TIME_WAIT  

                  tcp        0      1 10.208.98.107:51060         10.208.25.238:11960         SYN_SENT   

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:39186         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:42000         TIME_WAIT  

                  tcp        0      0 10.208.98.107:41445         10.208.98.97:9836           TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:40032         TIME_WAIT  

                  tcp        0      0 127.0.0.1:33770             127.0.0.1:11961             ESTABLISHED

                  tcp        0      0 127.0.0.1:11961             127.0.0.1:44619             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:40642         TIME_WAIT  

                  tcp        0      0 127.0.0.1:56197             127.0.0.1:11960             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:51053         TIME_WAIT  

                  tcp        0      0 10.208.98.107:34803         10.208.98.104:9850          TIME_WAIT  

                  tcp        0      0 10.208.98.107:33070         10.208.98.101:9850          TIME_WAIT  

                  tcp        0      0 127.0.0.1:11961             127.0.0.1:42900             ESTABLISHED

                  tcp        0      0 10.208.98.107:55262         10.208.165.142:41206        TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:38119         TIME_WAIT  

                  tcp        0      0 10.208.98.107:49249         10.208.98.100:9850          TIME_WAIT  

                  tcp        0      0 10.208.98.107:9836          10.208.98.106:46715         ESTABLISHED

                  tcp        0      0 10.208.98.107:59428         10.208.98.105:9836          TIME_WAIT  

                  tcp        0      0 127.0.0.1:42900             127.0.0.1:11961             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:39400         TIME_WAIT  

                  tcp        0      0 10.208.98.107:45473         10.208.98.105:9850          TIME_WAIT  

                  tcp        0      0 10.208.98.107:36229         10.208.98.108:9850          TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:41744         TIME_WAIT  

                  tcp        0      0 10.208.98.107:60638         10.208.98.96:9850           TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:56692         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:40369         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:40275         TIME_WAIT  

                  tcp        0      0 127.0.0.1:11961             127.0.0.1:60037             ESTABLISHED

                  tcp        0      0 10.208.98.107:9702          10.208.98.105:32793         ESTABLISHED

                  tcp        0      0 127.0.0.1:60049             127.0.0.1:11961             ESTABLISHED

                  tcp        0      0 10.208.98.107:39270         10.208.98.103:9850          TIME_WAIT  

                  tcp        0      0 10.208.98.107:5666          10.208.28.128:38304         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:41246         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:38280         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:59082         TIME_WAIT  

                  tcp        0      0 127.0.0.1:53489             127.0.0.1:11961             ESTABLISHED

                  tcp        0      0 127.0.0.1:47799             127.0.0.1:11961             ESTABLISHED

                  tcp        0      0 127.0.0.1:35884             127.0.0.1:11960             ESTABLISHED

                  tcp        0      0 127.0.0.1:44573             127.0.0.1:11961             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:51013         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:39474         TIME_WAIT  

                  tcp        0      0 127.0.0.1:54194             127.0.0.1:11961             ESTABLISHED

                  tcp        0      0 127.0.0.1:43154             127.0.0.1:11960             ESTABLISHED

                  tcp        0      0 10.208.98.107:34828         10.208.187.174:636          ESTABLISHED

                  tcp        0      0 127.0.0.1:11961             127.0.0.1:35032             ESTABLISHED

                  tcp        0      0 10.208.98.107:9836          10.208.98.96:58978          ESTABLISHED

                   

                  tcp        0      0 10.208.98.107:9702          10.208.98.106:47501         ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:45790         TIME_WAIT  

                  tcp        0      0 10.208.98.107:9702          10.208.98.107:54442         ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:40654         TIME_WAIT  

                  tcp        0      0 10.208.98.107:9852          10.208.98.107:57708         ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:38955         TIME_WAIT  

                  tcp        0      0 127.0.0.1:46633             127.0.0.1:11961             ESTABLISHED

                  tcp        0      0 127.0.0.1:44985             127.0.0.1:11960             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:39551         TIME_WAIT  

                  tcp        0      0 10.208.98.107:9702          10.208.98.101:55285         ESTABLISHED

                  tcp        0      0 127.0.0.1:11960             127.0.0.1:39058             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:53976         TIME_WAIT  

                  tcp        0      0 127.0.0.1:40800             127.0.0.1:9836              TIME_WAIT  

                  tcp        0      0 10.208.98.107:9850          10.208.98.108:46399         ESTABLISHED

                  tcp        0      0 10.208.98.107:46403         10.208.98.97:9702           ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:52431         TIME_WAIT  

                  tcp        0      0 127.0.0.1:40834             127.0.0.1:9836              TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:39837         TIME_WAIT  

                  tcp        0      0 10.208.98.107:36270         10.208.98.108:9850          TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:38453         TIME_WAIT  

                  tcp        0      0 10.208.98.107:44616         10.208.98.107:39733         ESTABLISHED

                  tcp        0      0 127.0.0.1:54117             127.0.0.1:11960             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:41016         TIME_WAIT  

                  tcp        0   4004 10.208.98.107:46640         10.208.72.248:11960         ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:33036         TIME_WAIT  

                  tcp        0      0 10.208.98.107:42626         10.208.98.106:9850          TIME_WAIT  

                  tcp        0      0 127.0.0.1:11960             127.0.0.1:35724             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:40181         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:56804         TIME_WAIT  

                  tcp        0      0 10.208.98.107:49294         10.208.98.100:9850          TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:38960         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:45761         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:37720         TIME_WAIT  

                  tcp        0      0 10.208.98.107:39905         10.208.98.98:9702           ESTABLISHED

                  tcp        0      0 10.208.98.107:47697         10.208.98.107:9851          ESTABLISHED

                  tcp        0      0 10.208.98.107:9853          10.208.98.107:38917         ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:40400         TIME_WAIT  

                  tcp        0      0 127.0.0.1:53733             127.0.0.1:11961             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:60744         TIME_WAIT  

                  tcp        0      0 127.0.0.1:11961             127.0.0.1:54194             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:53445         TIME_WAIT  

                  tcp        0      0 10.208.98.107:50897         10.208.98.98:9850           TIME_WAIT  

                  tcp        0    209 10.208.98.107:39733         10.208.98.107:44616         ESTABLISHED

                  tcp        0      0 127.0.0.1:34812             127.0.0.1:11960             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:42462         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:57265         TIME_WAIT  

                  tcp        0      0 127.0.0.1:51366             127.0.0.1:11960             ESTABLISHED

                  tcp        0      0 10.208.98.107:55260         10.208.165.142:41206        TIME_WAIT  

                  tcp        0      0 127.0.0.1:40826             127.0.0.1:9836              TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:39301         TIME_WAIT  

                  tcp        0      0 127.0.0.1:50035             127.0.0.1:11961             ESTABLISHED

                  tcp        0      0 10.208.98.107:53314         10.208.98.103:9836          TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:38281         TIME_WAIT  

                  tcp        0      0 10.208.98.107:55963         10.208.98.102:9702          ESTABLISHED

                  tcp        0      0 10.208.98.107:33238         10.208.133.228:11960        ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:60522         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:38603         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:39745         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:41577         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:38328         TIME_WAIT  

                  tcp        0      0 10.208.98.107:9850          10.208.98.96:41838          ESTABLISHED

                  tcp        0      0 127.0.0.1:11960             127.0.0.1:54117             ESTABLISHED

                  tcp        0      0 10.208.98.107:9850          10.208.98.106:35319         ESTABLISHED

                  tcp        0      0 10.208.98.107:913           10.208.153.30:2049          ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:39667         TIME_WAIT  

                  tcp        0      0 10.208.98.107:59141         10.208.98.96:9836           TIME_WAIT  

                  tcp        0      0 10.208.98.107:34077         10.208.98.96:9702           ESTABLISHED

                  tcp        0      0 127.0.0.1:11961             127.0.0.1:53733             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:42562         TIME_WAIT  

                  tcp        0      0 127.0.0.1:11961             127.0.0.1:44572             ESTABLISHED

                  tcp        0      0 127.0.0.1:52643             127.0.0.1:4750              TIME_WAIT  

                  tcp        0      0 10.208.98.107:49833         10.208.98.101:9836          TIME_WAIT  

                  tcp        0      0 10.208.98.107:58856         10.208.98.104:9836          TIME_WAIT  

                  tcp        0      0 127.0.0.1:46565             127.0.0.1:11960             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:41771         TIME_WAIT  

                  tcp        0      0 127.0.0.1:11961             127.0.0.1:50662             ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:51752         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:39285         TIME_WAIT  

                  tcp        0      0 10.208.98.107:674           10.208.153.35:2049          ESTABLISHED

                  tcp        0      0 10.208.98.107:9854          10.208.98.107:42476         ESTABLISHED

                  tcp        0      0 10.208.98.107:9836          10.208.98.107:46780         ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:40652         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:51740         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:40460         TIME_WAIT  

                  tcp        0      0 10.208.98.107:50247         10.208.98.108:9836          TIME_WAIT  

                  tcp        0      0 127.0.0.1:57045             127.0.0.1:11961             ESTABLISHED

                  tcp        0      0 10.208.98.107:60789         10.208.98.97:9850           TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.237.196.94:34315         TIME_WAIT  

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:41411         TIME_WAIT  

                  tcp        0      0 10.208.98.107:9854          10.208.98.107:42477         ESTABLISHED

                  tcp        0      0 10.208.98.107:10050         10.208.222.24:42375         TIME_WAIT  

                  tcp        0      0 10.208.98.107:9852          10.208.98.107:57709         ESTABLISHED

                  udp        0      0 10.208.98.107:60841         10.237.159.246:8649         ESTABLISHED

                  udp        0      0 0.0.0.0:47791               0.0.0.0:*                              

                  udp        0      0 0.0.0.0:6344                0.0.0.0:*                              

                  udp        0      0 0.0.0.0:8649                0.0.0.0:*                              

                  udp        0      0 127.0.0.1:52425             0.0.0.0:*                              

                  udp        0      0 0.0.0.0:43465               0.0.0.0:*                              

                  udp        0      0 0.0.0.0:56664               0.0.0.0:*                              

                  udp        0      0 0.0.0.0:614                 0.0.0.0:*                              

                  udp        0      0 0.0.0.0:111                 0.0.0.0:*                              

                  udp        0      0 10.208.98.107:123           0.0.0.0:*                              

                  udp        0      0 127.0.0.1:123               0.0.0.0:*                              

                  udp        0      0 0.0.0.0:123                 0.0.0.0:*                              

                  udp        0      0 10.208.98.107:37138         10.208.165.127:8649         ESTABLISHED

                  udp        0      0 127.0.0.1:659               0.0.0.0:*                              

                  raw        0      0 0.0.0.0:1                   0.0.0.0:*      

                   

                   

                  All ports seem to be as they should.

                  • 6. Re: App server connection error
                    Bill Robinson

                    ok - showing netstat output doesn't tell you if the appservers can communicate w/ each other.  is there a host-based firewall running on the boxes?  is there a network firewall between the boxes ?

                     

                    the tz is set the same on all boxes and the db and the time is in sync ?

                     

                    9876 and 9875 are mapped to 9840 and 9843 -> what does that mean?  why are these ports being used for anything in the lb?

                     

                    what is the actual message you see when you connect to the 2nd appserver ?  can you show a screenshot ?

                    • 7. Re: App server connection error
                      Neal Meagher

                      No firewall. IPTable off

                       

                      i get this. When logging into console on app 5

                       

                       

                       

                      I get

                       

                      p01 all the servers look good

                      P02 shows p04 and 05 are in error

                      P03 and P04 show all OK

                      P05 shows 03 and 11 not connected....

                      • 8. Re: App server connection error
                        Bill Robinson

                        what errors are there ?  i only see a red x on 11 and 03.

                         

                        if you click on each node, what does infra mgr show?

                         

                        what's in the appserverlauncher.log on 11 and 03 when you try and get to them via infra mgr ?

                         

                        what errors are you seeing on 4 and 5 when logged into 2 ?  above all i see is the list of appservers.  no errors other than on 3 and 11.

                        • 9. Re: App server connection error
                          Neal Meagher

                          This disconnect error is what I see.. I cant go into the servers as there is not connection. I see this in all the logs of problematic servers.

                           

                           

                           

                          [06 Apr 2016 10:54:46,923] [Nsh-Proxy-Thread-124] [INFO] [Anonymous:Anonymous:127.0.0.1] [BLSSOPROXY] failure establishing session with proxy service

                          [06 Apr 2016 10:54:49,129] [Authentication-Service-Thread-1] [WARN] [::127.0.0.1] [Appserver] Connection closed by /127.0.0.1:55024 before pre-authentication handshake could be completed. This could be a probe message from Client to Authentication Server.

                          [06 Apr 2016 10:54:49,129] [Authentication-Service-Thread-1] [INFO] [::127.0.0.1] [Appserver] Authentication Connection closed

                          [06 Apr 2016 10:54:49,129] [Client-Connections-Thread-2] [WARN] [Anonymous:Anonymous:127.0.0.1] [Client] Connection closed by /127.0.0.1:54805 before pre-authentication handshake could be completed. This could be a probe message from Client to Authentication Server.

                          [06 Apr 2016 10:54:49,129] [Client-Connections-Thread-2] [WARN] [Anonymous:Anonymous:127.0.0.1] [Client] Error authorizing the connection

                          [06 Apr 2016 10:54:49,129] [Client-Connections-Thread-2] [INFO] [Anonymous:Anonymous:127.0.0.1] [Client] Connection disconnecting: id = 247929

                          [06 Apr 2016 10:55:10,906] [Authentication-Service-Thread-2] [WARN] [::127.0.0.1] [Appserver] Connection closed by /127.0.0.1:55036 before pre-authentication handshake could be completed. This could be a probe message from Client to Authentication Server.

                          [06 Apr 2016 10:55:10,907] [Authentication-Service-Thread-2] [INFO] [::127.0.0.1] [Appserver] Authentication Connection closed

                          [06 Apr 2016 10:55:15,912] [Client-Connections-Thread-8] [WARN] [Anonymous:Anonymous:127.0.0.1] [Client] Connection closed by /127.0.0.1:54819 before pre-authentication handshake could be completed. This could be a probe message from Client to Authentication Server.

                          [06 Apr 2016 10:55:15,913] [Client-Connections-Thread-8] [WARN] [Anonymous:Anonymous:127.0.0.1] [Client] Error authorizing the connection

                          [06 Apr 2016 10:55:15,913] [Client-Connections-Thread-8] [INFO] [Anonymous:Anonymous:127.0.0.1] [Client] Connection disconnecting: id = 247930

                          [06 Apr 2016 10:55:16,942] [Nsh-Proxy-Thread-121] [WARN] [Anonymous:Anonymous:127.0.0.1] [BLSSOPROXY] javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

                          com.bladelogic.sso.engine.ClientDisconnectedException: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

                            at com.bladelogic.sso.engine.StreamMessaging.receiveMessage(StreamMessaging.java:123)

                            at com.bladelogic.sso.engine.StreamMessaging.receiveToken(StreamMessaging.java:203)

                            at com.bladelogic.om.infra.mfw.net.BlSessionServerConnection.authenticate(BlSessionServerConnection.java:176)

                            at com.bladelogic.om.infra.mfw.net.BlSessionNshServerConnection.internalHandshake(BlSessionNshServerConnection.java:78)

                            at com.bladelogic.om.infra.mfw.net.BlSessionNshServerConnection.doHandshake(BlSessionNshServerConnection.java:50)

                            at com.bladelogic.om.infra.mfw.fw.BlSessionNshProxyPair.setupClient(BlSessionNshProxyPair.java:104)

                            at com.bladelogic.om.infra.mfw.fw.BlSessionNshProxyPair.init(BlSessionNshProxyPair.java:75)

                            at com.bladelogic.om.infra.mfw.fw.NshProxyWorkerThread.execute(NshProxyWorkerThread.java:106)

                            at com.bladelogic.om.infra.mfw.fw.NshProxyWorkerThread.execute(NshProxyWorkerThread.java:17)

                            at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                          Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

                            at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

                            at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)

                            at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)

                            at sun.security.ssl.AppInputStream.read(Unknown Source)

                            at java.io.BufferedInputStream.fill(Unknown Source)

                            at java.io.BufferedInputStream.read(Unknown Source)

                            at com.bladelogic.sso.engine.StreamMessaging.receiveMessage(StreamMessaging.java:119)

                            ... 9 more

                          Caused by: java.io.EOFException: SSL peer shut down incorrectly

                            at sun.security.ssl.InputRecord.read(Unknown Source)

                            ... 16 more

                          [06 Apr 2016 10:55:16,942] [Nsh-Proxy-Thread-121] [INFO] [Anonymous:Anonymous:127.0.0.1] [BLSSOPROXY] failure establishing session with proxy service

                          [06 Apr 2016 10:55:19,251] [Authentication-Service-Thread-0] [WARN] [::127.0.0.1] [Appserver] Connection closed by /127.0.0.1:55047 before pre-authentication handshake could be completed. This could be a probe message from Client to Authentication Server.

                          [06 Apr 2016 10:55:19,251] [Authentication-Service-Thread-0] [INFO] [::127.0.0.1] [Appserver] Authentication Connection closed

                          [06 Apr 2016 10:55:19,252] [Client-Connections-Thread-1] [WARN] [Anonymous:Anonymous:127.0.0.1] [Client] Connection closed by /127.0.0.1:54828 before pre-authentication handshake could be completed. This could be a probe message from Client to Authentication Server.

                          • 10. Re: App server connection error
                            Bill Robinson

                            That looks like the appserver log, not the appserverlauncher log.

                             

                            What do you mean ‘you can’t go into them’ ?  if you are logged into 2, and click on any of the other appserver launchers in infra mgr, what happens?  what’s in the appserverlauncher log on that system when you do that ?

                            • 11. Re: App server connection error
                              Neal Meagher

                              I logged into app server to from console. Tailed launcher log. clicked around in infr mgmt. Nothing logging? I am running health check on port 9840 every 10 seconds.

                              • 12. Re: App server connection error
                                Bill Robinson

                                If you are tailing the launcher log on serverB and login  w/ the gui to serverA you should see a connection from serverA to serverB in the launcher log when you click on the launcher for serverB in the inframgr.

                                 

                                What message do you get in the inframgr when you are connected to server 2 and try and expand the launcher on server 5 ?

                                • 13. Re: App server connection error
                                  Neal Meagher

                                  I log into server 2 in console. server 5 in read connection error in the infra mgmt. When i refresh server 5 in the app server launcher portion in infra mgmt it writes nothing to the log of 5

                                  • 14. Re: App server connection error
                                    Bill Robinson

                                    All these servers are on the same subnet ?

                                    1 2 Previous Next