9 Replies Latest reply on Feb 5, 2018 12:11 PM by Chad Wadkins

    BSA: Batch Job Target Server(s), 'from individual jobs' vs 'use following for all'

    Chad Wadkins

      This is more curiosity than anything, as the batch job in question is running successfully.

       

      While testing it out though, I noticed that the use of differing target server options seems to impact the time needed for it to process.

       

      For reference, this 1 batch job consists of 3 jobs run sequentially:

      1. The BSAHOST environment is v8.7 P2 (8.7.00.263) running on Linux
      2. Job 1, .nsh script, runs blcli_execute commands and, if run independent of the batch, has a target of the BSAHOST
      3. Job 2, .ps1 script, runs powershell that parses the output of Job 1 into .csv files and, if run independent of the batch, has a target of WINDOWSHOST
      4. Job 3, .ps1 script, runs powershell that adds an entry to a central process tracking SQL Server db (it does not use the data from Job 1 or 2 though) and, if run independent of the batch, has a target of WINDOWSHOST (same as Job 2)

       

      Scenario A: When I run the batch job with 'Use servers from individual jobs', I am seeing run times upwards of 20minutes.

       

      Scenario B: If I make no changes other than switching the batch job to 'Use the following server for all jobs' with WINDOWSHOST as the 1 and only target, run time goes down to 5-6minutes.

       

      Again, even with the batch job running to completion successfully and seeing the appropriate output under either scenario, I still have the following questions:

      1. Why the reduction in run time, when switching from Scenario A to Scenario B?
      2. When running under Scenario B, how does Job1 still successfully retrieve blcli_execute output from the BSAHOST when the overall batch job says to use WINDOWSHOST as a target?

       

      Thanks,

      Chad...

        • 1. Re: BSA: Batch Job Target Server(s), 'from individual jobs' vs 'use following for all'
          Bill Robinson

          Scenario A: When I run the batch job with 'Use servers from individual jobs', I am seeing run times upwards of 20minutes.

          what is the breakdown in runtime of all the member jobs?  does one run longer than the others?  do they all take the same amount of time to run ?

           

          Scenario B: If I make no changes other than switching the batch job to 'Use the following server for all jobs' with WINDOWSHOST as the 1 and only target, run time goes down to 5-6minutes.

          does one job run in less time than secnario a ?

           

          Why the reduction in run time, when switching from Scenario A to Scenario B?

          need to drill in on what's happening in the member jobs i think before we can figure that out.

          When running under Scenario B, how does Job1 still successfully retrieve blcli_execute output from the BSAHOST when the overall batch job says to use WINDOWSHOST as a target?

          if you target a nsh job against any server - appserver, target, whatever and in the job you are running blcli commands, the blcli commands don't run "against" the job target.  an appserver (not the one you targeted) picks up the job and runs it.  based on the setting of appsvcurl in that appserver/instance, the blcli will connect to that appserver, not the target.  if it's a 'runscript' type of script, then the nsh process for the job will first cd to the target and nsh commands (not blcli) will execute against the target.  if it's the 'run once' type of script there is no cd and unless you pass in something representing the target name to nsh commands in the script, the nsh commands will probably run locally on the appserver that picked up the job.

          • 2. Re: BSA: Batch Job Target Server(s), 'from individual jobs' vs 'use following for all'
            Chad Wadkins

            'what is the breakdown in runtime of all the member jobs?  does one run longer than the others?  do they all take the same amount of time to run ?'

             

            Scenario A:

            1. Job1.nsh, Run at 01/24/2018 21:21:06, NSH Script Job Run, Completed Successfully, 01/24/2018 21:21:06, 01/24/2018 21:38:07
            2. Job2.ps1, Run at 01/24/2018 21:38:07, Deploy Job Run, Completed Successfully, 01/24/2018 21:38:07, 01/24/2018 21:40:23
            3. Job3.ps1, Run at 01/24/2018 21:40:23, Deploy Job Run, Completed Successfully, 01/24/2018 21:40:23, 01/24/2018 21:42:39

             

            Scenario B:

            1. Job1.nsh, Run at 01/25/2018 21:21:06, NSH Script Job Run, Completed Successfully, 01/25/2018 21:21:06, 01/25/2018 21:28:08
            2. Job2.ps1, Run at 01/25/2018 21:28:08, Deploy Job Run, Completed Successfully, 01/25/2018 21:28:08, 01/25/2018 21:30:10
            3. Job3.ps1, Run at 01/25/2018 21:30:10, Deploy Job Run, Completed Successfully, 01/25/2018 21:30:10, 01/25/2018 21:30:24

             

            'does one job run in less time than secnario a ?'

            • It looks like Job1 has the main reduction in run time. In the above example it goes from running 17mins down to 7mins.
            • Job3 did also reduce run time from 2mins 16secs down to just 14secs, yet that could be delays to gains completely outside of BSA's control since that script is communicating from/to multiple external components (WINDOWSHOST to SQL Server).

             

            And thank you for the explanation of how blcli_execute can run on the appserver, regardless of the target set in the job.

             

            Chad...

            • 3. Re: BSA: Batch Job Target Server(s), 'from individual jobs' vs 'use following for all'
              Bill Robinson

              yeah - that's pretty weird...  i'd actually expect targeting another host (besides the appserver) to possibly incur a delay because it would need to 'cd' to the target first and that could be delayed vs the appserver which should not be because it's itself or near the other appservers.

               

              the appservers are all in the same physical location ?

               

              if you can post it here, can you create a support ticket and upload the job run logs, the scripts and the appserver logs from all appservers covering the time of the 'slow' and 'fast' job runs ?  put the ticket # here or mention the thread in the ticket and support can bug me.

              • 4. Re: BSA: Batch Job Target Server(s), 'from individual jobs' vs 'use following for all'
                Chad Wadkins

                Bill -

                 

                Your one question, 'the appservers are all in the same physical location ?', led me to maybe a probable cause.

                 

                We have 4 appservers: 2 local, 2 remote.

                 

                Upon digging down into the logs while I was gathering them, I noticed that Scenario A Job1 was sent to an appserver in our remote site.  Scenario B Job1 was sent to an appserver in our local site.

                 

                What we haven't figured out yet is why it was sent there when the local site servers don't seem to have been running any other jobs that would constitute the system thinking it would need to send this job to one of the remote ones (of course, we do not intend to pretend to know what the system is thinking at any given time).

                 

                Anyway, before I open a support case, I will monitor how this job runs over the next couple of weeks (since it runs weekly now) to see if the longer running executions translate into always being sent to one of the remote appservers.

                 

                This leads me to a further question I would like to ask you though:

                Is there a way to tell BSA to always use one of the local appservers unless they are completely unavailable, other than setting a job execution rule to pin the job to specific appservers?

                 

                I would like to retain the capability to use the remote servers in situations where the local ones are down for some reason, yet I do not want the job to go remote at all if the local ones are up and ready for work.

                 

                Thanks for your input on this and I will update with further findings after seeing some more executions of the job,

                Chad...

                • 5. Re: BSA: Batch Job Target Server(s), 'from individual jobs' vs 'use following for all'
                  Bill Robinson

                  We have 4 appservers: 2 local, 2 remote.

                  remote (from the database/other appservers) is not supported.

                   

                  What we haven't figured out yet is why it was sent there when the local site servers don't seem to have been running any other jobs that would constitute the system thinking it would need to send this job to one of the remote ones

                  because bsa has no concept of sites.  any appserver in the env is fair game to pickup jobs and WITs for a job.  unless you have job routing rules setup, but those are based on job attributes, not appserver attributes.

                   

                   

                  Is there a way to tell BSA to always use one of the local appservers unless they are completely unavailable, other than setting a job execution rule to pin the job to specific appservers?

                  no

                  I would like to retain the capability to use the remote servers in situations where the local ones are down for some reason, yet I do not want the job to go remote at all if the local ones are up and ready for work.

                  your remote site should have a database too and if there's some problem in site 1 then the entire bsa infra - including the database - should fail over to the remote site.

                  1 of 1 people found this helpful
                  • 6. Re: BSA: Batch Job Target Server(s), 'from individual jobs' vs 'use following for all'
                    Chad Wadkins

                    Thanks for the feedback Bill, I will review this with our infrastructure team to review our architecture.

                     

                    We are planning an upgrade to v8.9 P2 in the next 2 months, so now is the time to revisit how things are setup.

                     

                    We do have the database setup to fail over in the remote site if there is an issue in the local site, yet it is almost always active in the local site and the remote site is just being replicated to from the source db locally.

                     

                    And if I understand you correctly, the fact that the db is running in our local site, that is the only location that should have active appservers...am I correct in that?

                     

                    Thanks again,

                    Chad...

                    • 7. Re: BSA: Batch Job Target Server(s), 'from individual jobs' vs 'use following for all'
                      Chad Wadkins

                      Bill -

                       

                      Thanks again for all the input on this.

                       

                      It does seem that as long as the job stays on a local appserver, it runs much quicker than if it is sent to one of the remote ones.

                       

                      Also, based on the information you provided, I have initiated a conversation with the team that built the system for us, so that we can address the remote appservers.

                       

                      Thanks again again,

                      Chad...

                      • 8. Re: BSA: Batch Job Target Server(s), 'from individual jobs' vs 'use following for all'
                        Bill Robinson

                        right - the appservers should be local to the database they are using.  there is a lot of communication from the appservers to the db and to each other.  no matter how fast or how low people say the latency is to a remote site most of the time it's not what they think and you see issues like this. 

                        so if all the appservers go down at one site or the db goes down at one site, fail over everything to the other site

                        • 9. Re: BSA: Batch Job Target Server(s), 'from individual jobs' vs 'use following for all'
                          Chad Wadkins

                          Good information to have, Bill...thanks for the reply.

                           

                          We have been realigning group responsibilities here and already begun to take over ownership of the infrastructure design to address this setup as we work towards our first upgrade to 8.9.01.

                           

                          In the meantime, we will be shutting down the remote appservers since we do have the 2 local ones to handle the workload.

                           

                          Thanks again for all the information,

                          Chad...