11 Replies Latest reply on Oct 23, 2017 12:23 PM by Bill Robinson

    NSH Job running only against 24 target servers at a time

    sreekanth Kondaiah

      Hi All,

       

      We have 2 BSA instances

      configured 2 config and 2 job servers on each instance.

      Configured NSH proxy server on one the instance and did changes in secure file to restrict direct NSH access.

       

      Now when we execute NSH script job against 100 target server it only takes 24 at a time and giving SSL_read error.

       

      Every time I execute NSH job it takes 24 or less target servers at a time.

       

      I also configured MaxNshProxyContexts=500 and MaxNshProxyThreads=100

      but still getting the same result.

       

      PLease help on this.

       

       

      Thanks,

      Sreekanth

        • 1. Re: NSH Job running only against 24 target servers at a time
          Bill Robinson

          I also configured MaxNshProxyContexts=500 and MaxNshProxyThreads=100

          did you restart the appserver service on the system running the nsh proxy after doing this ?

           

          is NshProxyMaxThreadIdleTime 0 ?  if not, it should be.

           

          Now when we execute NSH script job against 100 target server it only takes 24 at a time and giving SSL_read error.

          where?  appserver log ?  rscd log ? can you attach the job run log, and appserver logs from the instances involved in the job covering the time of the problem

          • 2. Re: NSH Job running only against 24 target servers at a time
            sreekanth Kondaiah

            restarted the service after configuring parameters in blasadmin.

            but still the result.

             

            attached the job log and nsh proxy server logs

             

            NshProxyMaxThreadIdleTime is 500.

            changing it to 0. will share the update after restart the appserver.

             

            • 3. Re: NSH Job running only against 24 target servers at a time
              sreekanth Kondaiah

              changed NshProxyMaxThreadIdleTime  to 0. but still the sam result.

               

              Also on server where NSH proxy  is configured, NSH job is not getting executing through the job servers configured on the same server.

              proxyserviceurls are also mentioned properly thorugh the console.

               

              blcred cred -list shows empty in destination urls.

               

              NSH job is executing on job server configured on other server.

               

              I think I missed something in NSH proxy configuration.

              • 4. Re: NSH Job running only against 24 target servers at a time
                Jim Wilson

                Make each instance its own NSH_PROXY by configuring the ProxySvcPort on each and restarting.

                Setting up a Network Shell proxy server - BMC Server Automation 8.9

                 

                No other changes are needed, although you may need to review

                MaxNshProxyContexts

                MaxNshProxyThreads

                NshProxyMaxThreadIdleTime

                • 5. Re: NSH Job running only against 24 target servers at a time
                  Bill Robinson
                  Also on server where NSH proxy  is configured, NSH job is not getting executing through the job servers configured on the same server.

                  what ?

                   

                  when you run a nsh script job, a job server will pickup the job, distribute the WITs for the job to itself and other job servers.  when the WIT starts it will start a nsh process.  nsh will read the secure file and see if it should use the nsh proxy and then look for the proxy service urls passed through from the job.  then it will connect to the nsh proxy for the urls it has.

                  • 6. Re: NSH Job running only against 24 target servers at a time
                    sreekanth Kondaiah

                    I have 2 application server.

                    configured 2 config and 2 job server on each server.

                     

                    On one server configured nsh proxy.

                    So you say I  need to configure NSH proxy on another application server as well.

                    will this wont work having only one NSH proxy for both application servers.

                    • 7. Re: NSH Job running only against 24 target servers at a time
                      Bill Robinson

                      ?you have a couple options:

                       

                      set the proxy service url on the job instances to point to this nsh proxy and ensure the nsh client on the job instances is configured to use a nsh proxy.

                       

                       

                      delete and re-create the job instances as JOB+NSH_PROXY

                      • 8. Re: NSH Job running only against 24 target servers at a time
                        sreekanth Kondaiah

                        I got where I was going wrong.

                         

                        The Blasamdin configuration that I did was applied to Default  i.e "_template" and not for NSH proxy server

                         

                        Making changes it worked.

                         

                        but after executing series of jobs CPU utilisation went continuously high server went into hung state.

                        so work on this further we are planning to have 2 different NSH proxy server which will be standalone.

                         

                        so now our BSA environment will have

                        2 jobs and 2 config on one server and

                        2 jobs and 2 config on another servers

                         

                        In totallity we will have 4 deployments Plus 2 new NSH proxy servers.

                        Some times we will be executing jobs on 3000 target servers.

                        Please suggest what configuration I should have for NSH proxy servers and configuration to be made in Blasadmin utility which will prevent this kind issue in future.

                         

                         

                        Thanks,

                        Sreekanth Kondaiah

                        • 9. Re: NSH Job running only against 24 target servers at a time
                          Bill Robinson

                          but after executing series of jobs CPU utilisation went continuously high server went into hung state.

                          what bsa version is this ?  do your nsh jobs have blcli calls in them ?  that may cause this problem.

                           

                          so work on this further we are planning to have 2 different NSH proxy server which will be standalone.

                          why ?  you should have CONIG+NSH_PROXY for your users and JOB+NSH_PROXY for the job servers.  no need for standalone.

                           

                           

                          Please suggest what configuration I should have for NSH proxy servers and configuration to be made in Blasadmin utility which will prevent this kind issue in future.

                          the hang ?  that's not related to using the nsh proxy most likely.  each instance should have:

                          MaxNshProxyContexts:600

                          MaxNshProxyThreads:150

                          NshProxyMaxThreadIdleTime:0

                          • 10. Re: NSH Job running only against 24 target servers at a time
                            sreekanth Kondaiah

                            BSA version is 8.7 P2

                             

                            Yes script contains blcli command. We also observed that server went into hung state executing the sricpt which contains blcli commands.

                             

                            Can I have better explanation on CONIG+NSH_PROXY for your users and JOB+NSH_PROXY for job servers.

                            As we already have job and config servers on both the application servers can we add one nsh proxy on both application server.

                            does this will work to some extent when executing blcli commands.

                            Or what will be better options for executing blcli commands preventing this hung state.

                             

                            blasadmin parameters are in place as you suggested.

                             

                             

                            Thanks,

                            Sreekanth

                            • 11. Re: NSH Job running only against 24 target servers at a time
                              Bill Robinson

                              Yes script contains blcli command. We also observed that server went into hung state executing the sricpt which contains blcli commands.

                              yes, this is a known issue resolved in 8.9+.  when you run a 'runscript' type of nsh script/job it spawns off one nsh process for each target upto however many WITs are available or the parallelism limit set.  each nsh process will spawn off blcli jvms.  that will put a memory drain on your system and is what likely caused the hang here.  do you need the blcli in this script ?  why ?  and this has nothing to do w/ the nsh proxy.

                               

                               

                              Can I have better explanation on CONIG+NSH_PROXY for your users and JOB+NSH_PROXY for job servers.

                              yes - it's more efficient to use a combined instance instead of using a standalone proxy.  this has the benefit of:

                              - you don't need to configure the appserver to point to any proxy

                              - since the instance (config or job) uses itself for a proxy there's no issue if another instance goes down.  if you have one nsh proxy and multiple other instances pointed to it, if the proxy instance drops everything else breaks.

                              - similar to the above, you aren't overwhelming a single instance w/ tons of requests from multiple other instances.

                               

                              As we already have job and config servers on both the application servers can we add one nsh proxy on both application server.

                              does this will work to some extent when executing blcli commands.

                              nsh proxy has nothing to do w/ the blcli.

                               

                              Or what will be better options for executing blcli commands preventing this hung state.

                              see my first comment.

                              1 of 1 people found this helpful