9 Replies Latest reply on Feb 18, 2016 7:51 PM by Bill Robinson

    Intermittent connection timed out?

    Adam Khan

      I have file deploy job that is schedules and which runs every 30minutes and it targeted to 4 hosts.

      I can telnet to 4750 from my app severs to these 4 hosts no issues.

      But Intermittently i am getting the below error and out networking team cant find any issues between the app server and the target server.

       

      Error    Jan 14, 2016 9:53:14 AM    cd: connection timed out: //ASH1-CSGW-402.prod.pcln.com

       

      Failed run:

      Screen Shot 2016-01-14 at 10.39.22 AM.png

       

      Successful run:

      Screen Shot 2016-01-14 at 10.40.05 AM.png

        • 1. Re: Intermittent connection timed out?
          Davorin Stevanovic

          Can we get rscd.log  from one failed target server ? Also are these targets behind socks proxy?

          • 2. Re: Intermittent connection timed out?
            Adam Khan

            This is the rscd log during the job run time 9:50am. I am not sure about the socks proxy.

             

            ee71cea816f3a7ef22b7 0000007923 01/14/16 09:20:36.024 INFO rscd -  ash1-csgw-402 18141 -1/-1 (Not_available): (Not_available): FIPS already enabled
            1886b33a4c63fdcc49d0 0000007924 01/14/16 09:20:36.061 INFO rscd -  172.21.88.43 18141 3060/3060 (CaresRole:akhan@corp.pcln.com): mkdir: mkdir -D //ASH1-CSGW-402.prod.pcln.com/ -p /var/www/html/knowbase/PricelineCares/root
            bee02df972636423414c 0000007925 01/14/16 09:20:36.526 INFO rscd -  ash1-csgw-402 18142 -1/-1 (Not_available): (Not_available): FIPS already enabled
            03c0e648e01b36e1bb54 0000007926 01/14/16 09:20:36.564 INFO rscd -  172.21.88.43 18142 3060/3060 (CaresRole:akhan@corp.pcln.com): grep: grep -D //ASH1-CSGW-402.prod.pcln.com/ Preserve file times only
            1c84e249b2a86a499e64 0000007927 01/14/16 09:20:36.731 INFO rscd -  ash1-csgw-402 18143 -1/-1 (Not_available): (Not_available): FIPS already enabled
            6f684ec2f93ecfa64ddb 0000007928 01/14/16 09:20:36.767 INFO rscd -  172.21.88.43 18143 3060/3060 (CaresRole:akhan@corp.pcln.com): cp: cp -D //ASH1-CSGW-402.prod.pcln.com/ -rvwP -T //ASh1-CWEB-402.prod.pcln.com/T/inetpub/wwwroot/eService/knowbase/PricelineCares/root/private /var/www/html/knowbase/PricelineCares/root/private
            3078d00ef55b727a9746 0000007929 01/14/16 09:20:36.917 INFO1rscd -  172.21.88.43 18143 3060/3060 (CaresRole:akhan@corp.pcln.com): cp: > 01011 Copy file //ASh1-CWEB-402.prod.pcln.com/T/inetpub/wwwroot/eService/knowbase/PricelineCares/root/private to //ASH1-CSGW-402.prod.pcln.com/var/www/html/knowbase/PricelineCares/root/private
            2018002d2821a2614416 0000007930 01/14/16 09:23:10.673 INFO rscd -  ash1-csgw-402 19164 -1/-1 (Not_available): (Not_available): FIPS already enabled
            581d909518692786c1ca 0000007931 01/14/16 09:23:10.715 INFO rscd -  10.2.88.24 19164 3060/3060 (CaresRole:akhan@corp.pcln.com): nsh: /cygdrive/C/Program Files/BMC Software/BladeLogic/NSH/bin/nsh -c cd //ASH1-CSGW-402.prod.pcln.com/ && //@'/C/Program Files/BMC Software/BladeLogic/NSH/tmp/application_server/1d5e8571-ea16-4d21-b947-94f122e06878/script_903adaf6-0c0a-4ced-ab3c-d09f27918d85'
            e2190246dbd337893719 0000007932 01/14/16 09:23:13.143 INFO rscd -  ash1-csgw-402 19171 -1/-1 (Not_available): (Not_available): FIPS already enabled
            00b2590e723de28de8d3 0000007933 01/14/16 09:23:13.186 INFO rscd -  10.2.88.24 19171 3060/3060 (CaresRole:akhan@corp.pcln.com): nsh: /cygdrive/C/Program Files/BMC Software/BladeLogic/NSH/bin/nsh -~ //ASH1-CSGW-402.prod.pcln.com/ //@/C/Program Files/BMC Software/BladeLogic/NSH/tmp/application_server/1d5e8571-ea16-4d21-b947-94f122e06878/script_903adaf6-0c0a-4ced-ab3c-d09f27918d85
            d514503bbbb6bf9ce6f8 0000007934 01/14/16 09:23:13.330 INFO rscd -  ash1-csgw-402 19172 -1/-1 (Not_available): (Not_available): FIPS already enabled
            84f22938696b87856b36 0000007935 01/14/16 09:23:13.371 INFO rscd -  10.2.88.24 19172 3060/3060 (CaresRole:akhan@corp.pcln.com): nsh: /cygdrive/C/Program Files/BMC Software/BladeLogic/NSH/bin/nsh -~ //ASH1-CSGW-402.prod.pcln.com/ //@/C/Program Files/BMC Software/BladeLogic/NSH/tmp/application_server/1d5e8571-ea16-4d21-b947-94f122e06878/script_903adaf6-0c0a-4ced-ab3c-d09f27918d85
            a8e71f1a8c7757da28dd 0000007936 01/14/16 09:23:13.562 INFO rscd -  ash1-csgw-402 19173 -1/-1 (Not_available): (Not_available): FIPS already enabled
            7006036d82a6c1cacdcf 0000007937 01/14/16 09:23:13.606 INFO rscd -  10.2.88.24 19173 3060/3060 (CaresRole:akhan@corp.pcln.com): mkdir: mkdir -D //ASH1-CSGW-402.prod.pcln.com/ -p /var/www/html/knowbase/PricelineCares/Media
            6c09b1d18f799582d9f3 0000007938 01/14/16 09:23:13.904 INFO rscd -  ash1-csgw-402 19174 -1/-1 (Not_available): (Not_available): FIPS already enabled
            e91be645dde42afbc37e 0000007939 01/14/16 09:23:13.946 INFO rscd -  10.2.88.24 19174 3060/3060 (CaresRole:akhan@corp.pcln.com): grep: grep -D //ASH1-CSGW-402.prod.pcln.com/ Preserve file times only
            d5e71b29b9f47bc368fc 0000007940 01/14/16 09:23:14.099 INFO rscd -  ash1-csgw-402 19175 -1/-1 (Not_available): (Not_available): FIPS already enabled
            e981627d31aebf8eac2f 0000007941 01/14/16 09:23:14.141 INFO rscd -  10.2.88.24 19175 3060/3060 (CaresRole:akhan@corp.pcln.com): cp: cp -D //ASH1-CSGW-402.prod.pcln.com/ -rvwP -T //ASh1-CWEB-402.prod.pcln.com/T/inetpub/wwwroot/eService/knowbase/PricelineCares/media/Solution /var/www/html/knowbase/PricelineCares/Media/Solution
            e991babbe69fa68f0af0 0000007942 01/14/16 09:23:14.296 INFO1rscd -  10.2.88.24 19175 3060/3060 (CaresRole:akhan@corp.pcln.com): cp: > 01011 Copy file //ASh1-CWEB-402.prod.pcln.com/T/inetpub/wwwroot/eService/knowbase/PricelineCares/media/Solution to //ASH1-CSGW-402.prod.pcln.com/var/www/html/knowbase/PricelineCares/Media/Solution
            46264ede2efd1ba6566a 0000007943 01/14/16 09:50:19.711 INFO rscd -  ash1-csgw-402 22008 -1/-1 (Not_available): (Not_available): FIPS already enabled
            b19f43554b36f0a09d02 0000007944 01/14/16 09:50:19.753 INFO rscd -  172.21.88.43 22008 3060/3060 (CaresRole:akhan@corp.pcln.com): nsh: /cygdrive/C/Program Files/BMC Software/BladeLogic/NSH/bin/nsh -c cd //ASH1-CSGW-402.prod.pcln.com/ && //@'/C/Program Files/BMC Software/BladeLogic/NSH/tmp/application_server/03612f42-2bef-4898-b9e3-310d540d7aae/script_667da858-379e-43e2-9b09-a5598292efe5'
            f3b826bb0ece6276865a 0000007945 01/14/16 09:50:19.907 INFO rscd -  ash1-csgw-402 22009 -1/-1 (Not_available): (Not_available): FIPS already enabled
            2cd4a7a0ec957a8dc354 0000007946 01/14/16 09:50:19.939 INFO rscd -  172.21.88.43 22009 3060/3060 (CaresRole:akhan@corp.pcln.com): nsh: /cygdrive/C/Program Files/BMC Software/BladeLogic/NSH/bin/nsh -~ //ASH1-CSGW-402.prod.pcln.com/ //@/C/Program Files/BMC Software/BladeLogic/NSH/tmp/application_server/03612f42-2bef-4898-b9e3-310d540d7aae/script_667da858-379e-43e2-9b09-a5598292efe5
            44f09993ea2781f1247c 0000007947 01/14/16 09:50:20.076 INFO rscd -  ash1-csgw-402 22010 -1/-1 (Not_available): (Not_available): FIPS already enabled
            a83ddc4abcc13e42c45d 0000007948 01/14/16 09:50:20.111 INFO rscd -  172.21.88.43 22010 3060/3060 (CaresRole:akhan@corp.pcln.com): nsh: /cygdrive/C/Program Files/BMC Software/BladeLogic/NSH/bin/nsh -~ //ASH1-CSGW-402.prod.pcln.com/ //@/C/Program Files/BMC Software/BladeLogic/NSH/tmp/application_server/03612f42-2bef-4898-b9e3-310d540d7aae/script_667da858-379e-43e2-9b09-a5598292efe5
            83f77d71946ace66d1a2 0000007949 01/14/16 09:50:20.295 INFO rscd -  ash1-csgw-402 22011 -1/-1 (Not_available): (Not_available): FIPS already enabled
            314baa2fded4540cd37e 0000007950 01/14/16 09:50:20.329 INFO rscd -  172.21.88.43 22011 3060/3060 (CaresRole:akhan@corp.pcln.com): mkdir: mkdir -D //ASH1-CSGW-402.prod.pcln.com/ -p /var/www/html/knowbase/PricelineCares/root
            6d14612c7549e1632957 0000007951 01/14/16 09:50:20.628 INFO rscd -  ash1-csgw-402 22012 -1/-1 (Not_available): (Not_available): FIPS already enabled
            29200c9b0c26c57dfaf7 0000007952 01/14/16 09:50:20.662 INFO rscd -  172.21.88.43 22012 3060/3060 (CaresRole:akhan@corp.pcln.com): grep: grep -D //ASH1-CSGW-402.prod.pcln.com/ Preserve file times only
            cf411a758cfd9f3544ce 0000007953 01/14/16 09:50:20.815 INFO rscd -  ash1-csgw-402 22013 -1/-1 (Not_available): (Not_available): FIPS already enabled
            f6326c641058b5624389 0000007954 01/14/16 09:50:20.848 INFO rscd -  172.21.88.43 22013 3060/3060 (CaresRole:akhan@corp.pcln.com): cp: cp -D //ASH1-CSGW-402.prod.pcln.com/ -rvwP -T //ASh1-CWEB-402.prod.pcln.com/T/inetpub/wwwroot/eService/knowbase/PricelineCares/root/private /var/www/html/knowbase/PricelineCares/root/private
            56ad3fa8dd2f6b4ea1d7 0000007955 01/14/16 09:50:21.004 INFO1rscd -  172.21.88.43 22013 3060/3060 (CaresRole:akhan@corp.pcln.com): cp: > 01011 Copy file //ASh1-CWEB-402.prod.pcln.com/T/inetpub/wwwroot/eService/knowbase/PricelineCares/root/private to //ASH1-CSGW-402.prod.pcln.com/var/www/html/knowbase/PricelineCares/root/private
            f6a83750f8d88bfc5a1f 0000007956 01/14/16 10:20:35.241 INFO rscd -  ash1-csgw-402 25928 -1/-1 (Not_available): (Not_available): FIPS already enabled
            597d4c3b394a9433ed81 0000007957 01/14/16 10:20:35.277 INFO rscd -  172.21.88.43 25928 3060/3060 (CaresRole:akhan@corp.pcln.com): nsh: /cygdrive/C/Program Files/BMC Software/BladeLogic/NSH/bin/nsh -c cd //ASH1-CSGW-402.prod.pcln.com/ && //@'/C/Program Files/BMC Software/BladeLogic/NSH/tmp/application_server/cc2d919d-ac30-40a0-a41e-a4028f4c781b/script_a63d6ad8-c539-46a4-ad29-e5e6390eb767'
            afe972d93c80492950f8 0000007958 01/14/16 10:20:36.666 INFO rscd -  ash1-csgw-402 25930 -1/-1 (Not_available): (Not_available): FIPS already enabled
            • 3. Re: Intermittent connection timed out?
              Davorin Stevanovic

              Hi,

               

              So request never came to the target server. Could you please verify if there is socks proxy and if yes would it be possible to check on it log file if proxy received request. Also please attach log files as files because communities are searchable by search engines and it is in better way to post it like that instead of copy paste log content directly to thread.

              • 4. Re: Intermittent connection timed out?
                Adam Khan

                Found out from our Unix team there is no Proxy at all.. The target server are behind a firewall. But I am able to telnet fine to the hosts.

                • 5. Re: Intermittent connection timed out?
                  Bill Robinson

                  You can telnet on 4750 from the appserver and you see the connection from the appserver in the rscd log on each target?

                  • 6. Re: Intermittent connection timed out?
                    Adam Khan

                    Bill,

                     

                    I am able to telnet from my app server and i see the connection "ESTABLISHED" when i run the netstat -an | grep 4750 on my target server.

                     

                    But I am not seeing that in the rscd.log.

                    • 7. Re: Intermittent connection timed out?
                      Bill Robinson

                      you should see something like:

                       

                      01/15/16 10:36:50.201 WARN rscd -  ::ffff:192.168.50.87 7652 -1/-1 (Not_available): (Not_available): TLS setup failed for agent: Protocol mismatch. Check that client and server "secure" files match. Exiting and terminating connection.

                       

                       

                      in the rscd log when you close the telnet connection.  when you see the established connection, it's the rscd listening on 4750:

                       

                      so like:

                      tcp6   0 192.168.2.23:4750   192.168.50.87:38124 ESTABLISHED 7665/bin/rscd
                      • 8. Re: Intermittent connection timed out?
                        Adam Khan

                        sorry for the not replying back earlier. Bill I am not seeing any info in the rscd.log when I make a telnet connection or when i terminate the connection.

                        • 9. Re: Intermittent connection timed out?
                          Bill Robinson

                          that seems odd to me - that seems like the connection is not actually getting through to the agent.