11 Replies Latest reply on Sep 22, 2016 2:40 PM by Bill Robinson

    Agent upgrade Job performance terrible using repeater BL 8.6.1

    Vernon Whicker

      reference case :00133198

       

      We are working to perform an upgrade to the agents on roughly 20,000 Linux Servers in our environment. The performance of the BL upgrade job is poor and unusable on any targets utilizing the repeaters we have set up in remote data centers. We are utilizing a workaround provided in the reference ticket above that pushes all of the jobs through a single application server per a defect.

       

      Looking to improve performance or have another viable option to complete this upgrade. there are roughly another 20, 000 servers to do after this effort Unix alone.

       

      It almost appears that the job is pushing the package to the repeater In KDC then the repeater is sending it back to our PDC for routing to the target.

       

       

      If i run the job without the repeater the job completes in 1/3 of the time.

       

      Looking for any feedback on experirences on upgrading the  BL agent with the Agent upgrade job in BL 8.6.1.

       

       

        • 1. Re: Agent upgrade Job performance terrible using repeater BL 8.6.1
          Bill Robinson

          where is the appserver/file server?

          where is the repeater ?

          where is the payload of the agent installer stored ?

          "It almost appears that the job is pushing the package to the repeater In KDC then the repeater is sending it back to our PDC for routing to the target."

          -> how does it 'almost appear' ?  do you see the agent installer payload copied to this repeater in 'KDC' and and incoming connection from that repeater to the target in 'PDC' ?  if so that likely means your repeater routing rules are incorrect.  is there a repeater in 'PDC' that should be used instead ?

           

          "If i run the job without the repeater the job completes in 1/3 of the time. "

          -> they why are you using the repeater ?

          • 2. Re: Agent upgrade Job performance terrible using repeater BL 8.6.1
            Vernon Whicker

            Portsmouth Data Center = PDC

            Kansas City Data Center = KDC

             

            The app servers and depot are in our PDC

            the Repeater is in the KDC

            the payload is in the depot PDC

            the Targets are in the KDC

             

            The file is copied to the repeater in the KDC with the hopes of cutting the bandwidth usage and time to deploy down as the targets are in the KDC.

             

            With a large number of targets and the size of the installer one would hope to utalize the repeater to cut down on time and bandwidth usage. Would like to target hundreds/thousands of servers at a time.

             

            Have an open case with BMC:

            Case 00164494

            • 3. Re: Agent upgrade Job performance terrible using repeater BL 8.6.1
              Bill Robinson

              "It almost appears that the job is pushing the package to the repeater In KDC then the repeater is sending it back to our PDC for routing to the target."

              -> how does it 'almost appear' ?  do you see the agent installer payload copied to this repeater in 'KDC' and and incoming connection from that repeater to the target in 'PDC' ?  if so that likely means your repeater routing rules are incorrect.  is there a repeater in 'PDC' that should be used instead ?

              • 4. Re: Agent upgrade Job performance terrible using repeater BL 8.6.1
                Vernon Whicker

                based on timings...  We see in the log that its getting pushed to the repeater.. Takes like 4ish minutes, but it is taking like  another 10 minutes to go from the repeater to the target...    If we go from PDC to KDC directly its like 7 ish minutes.. Guessing on timings.

                 

                As mentioned BIll BMC has reviewed environent adn config in regard to this issue.no mention of routing rules being incorrect. I can follow up with Brian as well.

                • 5. Re: Agent upgrade Job performance terrible using repeater BL 8.6.1
                  Bill Robinson

                  if you are pushing from the appserver in PDC to the repeater in KDC and then back to a target in PDC, then you have a problem w/ your rules.  payloads destined for targets in PDC should not go through a repeater in another datacenter right ?

                   

                  "If we go from PDC to KDC directly its like 7 ish minutes."

                  -> is that for a single target ?  the copy will be appserver to repeater, then repeater to target. 

                   

                  "but it is taking like  another 10 minutes to go from the repeater to the target...  "

                  -> so that is from the appserver to the KDC repeater and a target in KDC or PDC ?

                   

                  " If we go from PDC to KDC directly its like 7 ish minutes"

                  -> that is to a single target in KDC ?

                   

                  "Guessing on timings."

                  -> why are you guessing ?  you should see what is happening - where the payload is being copied to, where the payload is coming from - appserver or repeater.

                  • 6. Re: Agent upgrade Job performance terrible using repeater BL 8.6.1
                    Vernon Whicker

                    The targets I am referencing are all in KDC. We don not want to hit PDC targets from the KDC repeater.

                     

                    "If we go from PDC to KDC directly its like 7 ish minutes."

                    -> is that for a single target ?  the copy will be appserver to repeater, then repeater to target.

                     

                    The timings are for a single target as part of the troubleshooting testing performed with BMC. In the case of the PDC to KDC directly we are not using the repeater by bypassing the rule. Straight from PDC job server to single target in KDC.

                     

                    "but it is taking like  another 10 minutes to go from the repeater to the target...  "

                    -> so that is from the appserver to the KDC repeater and a target in KDC or PDC ?

                     

                    That timing is just from the already copied payload. KDC repeater to KDC target

                     

                    " If we go from PDC to KDC directly its like 7 ish minutes"

                    -> that is to a single target in KDC ?

                     

                    Yes....see above

                     

                    "Guessing on timings."

                    -> why are you guessing ?  you should see what is happening - where the payload is being copied to, where the payload is coming from - appserver or repeater.

                     

                    Per the log the payload is being copied to repeater...

                    VXKIT-PPA00006Info9/19/2016 14:02Using standard repeater 'standard-vxkip-eblogc19' for staging. Staging directory is : '//VXKIP-EBLOGC19/BladelogicStandardRepeater/5850_78a9b620-32a1-4a68-8460-43d3fbe1820d'

                     

                    repeater to target both KDC: this case 11 min copy time

                    VXKIT-PPA00006Info9/19/2016 14:07(VXKIP-EBLOGC19) Copying config files from //VXKIP-EBLOGC19/BladelogicStandardRepeater/5850_78a9b620-32a1-4a68-8460-43d3fbe1820d to //VXKIT-PPA00006/bladelogic/stage/5850_0eb1d538-6d6c-4c17-95ad-1446489caf63/
                    VXKIT-PPA00006Info9/19/2016 14:18Staging completed successfully.

                     

                     

                    Root issue is performance time

                    • 7. Re: Agent upgrade Job performance terrible using repeater BL 8.6.1
                      Bill Robinson

                      how long does it take to copy the file from the PDC to KDC repeater?

                      how long does it take to copy the file from the KDC repeater to the target server ?

                      • 8. Re: Agent upgrade Job performance terrible using repeater BL 8.6.1
                        Vernon Whicker

                        From one example run

                         

                        how long does it take to copy the file from the PDC to KDC repeater?

                         

                        4 min

                         

                        how long does it take to copy the file from the KDC repeater to the target server ?

                         

                        11 min

                        • 9. Re: Agent upgrade Job performance terrible using repeater BL 8.6.1
                          Bill Robinson

                          that's in a job run or initiating the nsh copy directly from the repeater to the target ? (not from nsh here, or nsh on the appserver)

                          • 11. Re: Agent upgrade Job performance terrible using repeater BL 8.6.1
                            Bill Robinson

                            Ok, so take the job out of it.  the copy time in the job run should be close to the sum of the copy from the appserver to repeater and the repeater and the repeater to the target.

                             

                            In the rscd log on the target do you see the payload being copied from the repeater ?