14 Replies Latest reply on Nov 9, 2017 9:07 AM by Santhosh Kurimilla

    Patch deploy/commit errors

      Share This:

      Guys we had a weird issue with a client patch deployment job. We are running 8.6.01.66 BSA. Granted I don't know much about the inner workings of how patch jobs run behind the scenes.

       

      The client was attempting to install a set of recommended rpms on a RH6 box. He created an include list of roughly 50-70 patches. His jobs ran successfully until it got to the commit phase where it was attempting to install the patches. The actual error we were getting is "./bldeploycmd.2.bat :exit: too many arguments". It also gave an error which was rather misleading saying the job couldn't cache a particular patch,  "caching enabled but no local cache of //var/tmp/stage/.../blrepos/packages/patchname.rpm from repo

       

      We think that was due to it being the rpm after the string limit was exceeded, but that's just a guess. Which it couldn't/wouldn't download due to some issue with the committing of patches.

       

      We eventually determined that the number of patches included in the include list was the culprit. Once the client reduced the include list to 10 patches, then 30, the job ran successfully, deploying the patches.

       

       

      My questions regarding this behavior are:

      1) what is the string limit or argument limit for using patches the include list?

      2) would using a smart group with specific patches listed avoid the string limit in the include list? (if we hardcoded patches into a smart group)

      3) is there any documentation regarding limitations of using include lists?
      4) We were surmising that each object in the include list has its own xml file or possibly each one is treated individually instead of as a group when you typically only reference a catalog smart group

        • 1. Re: Patch deploy/commit errors
          Bill Robinson

          There's a patch/hotfix that should address this issue - 8.6.01 Patch1 (r71) - that you can get from support.  the problem is that when the include list has a large number of duplicate rpms (eg rpmA-1.1, rpmA-1.2, rpmA-1.3) all of them are included, but really only the latest one needs to be included.  so that will reduce the include list size and should not exceed the shell limit, unless you are including the 'RPMs' smart group (which you shouldn't be)

           

          1 - i believe it depends on the resources on the box, and it's a shell limitation, so see what redhat says

          2 - generally less, unless this list also exceeds the shell limit

          3 - this maybe noted in the defect fix notes

          4 - i'm not sure i follow - when you use an include list it's like running "yum update rpmA rpmB rpmC rpmD ...".  how else would you expect it to run ?

          • 2. Re: Patch deploy/commit errors

            Digging this back up.

             

            what if my client has an application restriction that he doesn't want the last several kernel updates installed? his application is outdated and has no future updates for later kernels and doesn't support after a certain version, which I'm sure is more than one version ago

            • 3. Re: Patch deploy/commit errors

              is there a Q number for this fix?

              • 4. Re: Patch deploy/commit errors
                Bill Robinson

                what if my client has an application restriction that he doesn't want the last several kernel updates installed? his application is outdated and has no future updates for later kernels and doesn't support after a certain version, which I'm sure is more than one version ago

                then use an exclude list w/ all the kernels or a smart group w/ all the kernels.

                 

                is there a Q number for this fix?

                Known and corrected issues - BMC Server Automation 8.6

                 

                QM001881787    Patch    Patch Analysis with smart group included will install latest RPM in catalog instead of the one specified in include list.    8.6.01    8.6.01.001

                 

                QM001886587    Patch    RPM includes the list for deploy containing duplicate RPM names.    8.6.01    8.6.01.001

                QM001871304    Patch    Performance issue with patching job after upgrade to 8.5.01    8.5.01,

                8.6.00    8.6.01

                • 5. Re: Patch deploy/commit errors

                  We are finding this issue in 8.7 Patch4 too.

                  • 6. Re: Patch deploy/commit errors

                    Contact support with full details

                    • 7. Re: Patch deploy/commit errors
                      Bill Robinson

                      what issue?  i've listed several defects above.

                      • 8. Re: Patch deploy/commit errors

                        Jim Wilson
                        Case# 00428587 has been submitted for the same.

                         

                        Bill Robinson

                        We are more concerned with the error message: "./bldeploycmd.2.bat :exit: too many arguments" which doesn't seem to be the actual issue here though. Can you please confirm what throwing "too many arguments" error?

                        • 9. Re: Patch deploy/commit errors
                          Bill Robinson

                          We are more concerned with the error message: "./bldeploycmd.2.bat :exit: too many arguments" which doesn't seem to be the actual issue here though.

                          why do you think this error isn't 'the actual issue' ?  if you are getting this error then it is the issue because it is what is breaking the job.

                           

                          Can you please confirm what throwing "too many arguments" error?

                          did you not read any of the above thread ?  the include list is too large.  the include list is passed into our yum call as shell arguments and at some point the shell can't handle it.

                          • 10. Re: Patch deploy/commit errors

                            Bill Robinson wrote:

                             

                            We are more concerned with the error message: "./bldeploycmd.2.bat :exit: too many arguments" which doesn't seem to be the actual issue here though.

                            why do you think this error isn't 'the actual issue' ?  if you are getting this error then it is the issue because it is what is breaking the job.

                             

                            Can you please confirm what throwing "too many arguments" error?

                            did you not read any of the above thread ?  the include list is too large.  the include list is passed into our yum call as shell arguments and at some point the shell can't handle it.

                            Because it was always coming with a combination of other errors like: Diskspace or rpm dependency or unpacking rpm, etc.

                            Does it cause the issue when we have Exclude list too? We don't have any include list and only Exclude list in our Analysis option.

                            • 11. Re: Patch deploy/commit errors
                              Bill Robinson
                              Because it was always coming with a combination of other errors like: Diskspace or rpm dependency or unpacking rpm, etc.

                              -> great, those can be additional issues on top of the include list size. 

                               

                              Does it cause the issue when we have Exclude list too? We don't have any include list and only Exclude list in our Analysis option.

                              -> it should not, exclude list is handled differently.

                              • 12. Re: Patch deploy/commit errors

                                Bill Robinson wrote:

                                Does it cause the issue when we have Exclude list too? We don't have any include list and only Exclude list in our Analysis option.

                                -> it should not, exclude list is handled differently.

                                So, we are finding "too many arguments" error when we don't even have an Include list. Our concern is whether or not to ignore this error. If we cannot ignore, we would like to find the preventive steps to fix the issues.

                                 

                                BLDeploy log snippet:

                                11/04/17 04:30:24.613 INFO     bldeploy - [2][New RPM Group] [stdout: 2]        Parsing include list

                                 

                                11/04/17 04:30:25.599 INFO     bldeploy - [2][New RPM Group] [stdout: 2]        Using Includes List =

                                 

                                11/04/17 04:30:25.605 INFO     bldeploy - [2][New RPM Group] [stdout: 2]        Using Excludes List = php-5.1.6-11.el5.x86_64,php-5.1.6-12.el5.x86_64,php-5.1.6-15.el5.x86_64,php-5.1.6-20.el5.x86_64,php-5.1.6-20.el5_2.1.x86_64,.... <removed_long_list_of_rpms>

                                 

                                11/04/17 04:30:25.609 INFO     bldeploy - [2][New RPM Group] [stdout: 2]        Using Deploy Type: update

                                 

                                11/04/17 04:30:25.624 INFO     bldeploy - [2][New RPM Group] [stdout: 2]        Running yum from /var/tmp/stage/504ac0b6137f3e2b855ac6eb533fb062 and yum used is blyum...

                                 

                                11/04/17 04:30:37.651 WARN     bldeploy - [2][New RPM Group] [stderr: 2]        Error unpacking rpm package httpd-2.2.3-92.el5_11.x86_64

                                11/04/17 04:30:37.651 WARN     bldeploy - [2][New RPM Group] [stderr: 2]

                                 

                                11/04/17 04:30:37.765 INFO     bldeploy - [2][New RPM Group] [stdout: 2]        yum deploy failed.

                                • 13. Re: Patch deploy/commit errors
                                  Bill Robinson

                                  I don't see 'too many arguments' in this snippet.  i see an error unpacking the rpm.

                                   

                                  in the log you attached to the ticket i see:

                                  Transaction Check Error:

                                    installing package kernel-2.6.32-696.10.3.el6.x86_64 needs 84MB on the / filesystem

                                    installing package java-1.7.0-openjdk-1:1.7.0.151-2.6.11.0.el6_9.x86_64 needs 182MB on the / filesystem

                                    installing package sssd-common-pac-1.13.3-57.el6_9.x86_64 needs 84MB on the / filesystem

                                    installing package nss-3.28.4-4.el6_9.x86_64 needs 87MB on the / filesystem

                                    installing package poppler-utils-0.12.4-12.el6_9.x86_64 needs 84MB on the / filesystem

                                  ......

                                  Disk Requirements:

                                    At least 198MB needed on the / filesystem.

                                  10/29/17 01:06:19.793 INFO bldeploy - [2][New RPM Group] [stdout: 2]     yum deploy failed.

                                  which means yum is finding that there is not enough space on / to install the rpms.

                                   

                                  the failure of the deploy triggers an undo, and that is where you are getting the 'too many arguments' message:

                                  0/29/17 01:06:21.653 INFO     bldeploy - [2][New RPM Group] Executing command: "echo "Undo is not supported";exit 1 rpcbind sssd-krb5 samba-common-0:3.6.23-45.el6_9 openssh-clients-5.3p1-123.el6_9 nss python-libipa_hbac libipa_hbac bind-utils-32:9.8.2-0.62.rc1.el6_9.4 openssh-5.3p1-123.el6_9 selinux-policy sssd-ldap nss-sysinit sssd-common-pac libsss_idmap glibc glibc-headers sssd-common python-rhsm samba4-libs-0:4.2.10-11.el6_9 sssd-ipa kernel-firmware acl sssd-proxy samba-client-0:3.6.23-45.el6_9 poppler-utils sudo-1.8.6p3-29.el6_9 libacl java-1.7.0-openjdk-1:1.7.0.151-2.6.11.0.el6_9 ca-certificates kernel samba-winbind-clients-0:3.6.23-45.el6_9 samba-winbind-0:3.6.23-45.el6_9 dhclient-12:4.1.1-53.P1.el6_9.1 glibc-devel kernel-headers poppler bind-libs-32:9.8.2-0.62.rc1.el6_9.4 sssd-client perf sssd-krb5-common sssd-ad selinux-policy-targeted procps sssd dhcp-common-12:4.1.1-53.P1.el6_9.1 python-sss-murmur glibc-common python-sssdconfig nss-tools openssh-server-5.3p1-123.el6_9 python-rhsm-certificates"
                                  10/29/17 01:06:21.653 DEBUG    bldeploy - [2][New RPM Group] chdir to PkgDir: '/opt/bmc/bladelogic/RSCD/Transactions/d0552d2c6ef0359394639d1b876b439b/'
                                  10/29/17 01:06:21.657 INFO     bldeploy - [2][New RPM Group] [stdout: 2]     Undo is not supported
                                  10/29/17 01:06:21.657 WARN     bldeploy - [2][New RPM Group] [stderr: 2]     ./bldeploycmd.2.bat: line 2: exit: too many arguments

                                   

                                  so in this case you can ignore the 'too many arguments' message because it's not relevant to the deploy failure.  the deploy failed due to lack of space on /.

                                   

                                   

                                   

                                  0/29/17 01:06:21.653 INFO     bldeploy - [2][New RPM Group] Executing command: "echo "Undo is not supported";exit 1 rpcbind sssd-krb5 samba-common-0:3.6.23-45.el6_9 openssh-clients-5.3p1-123.el6_9 nss python-libipa_hbac libipa_hbac bind-utils-32:9.8.2-0.62.rc1.el6_9.4 openssh-5.3p1-123.el6_9 selinux-policy sssd-ldap nss-sysinit sssd-common-pac libsss_idmap glibc glibc-headers sssd-common python-rhsm samba4-libs-0:4.2.10-11.el6_9 sssd-ipa kernel-firmware acl sssd-proxy samba-client-0:3.6.23-45.el6_9 poppler-utils sudo-1.8.6p3-29.el6_9 libacl java-1.7.0-openjdk-1:1.7.0.151-2.6.11.0.el6_9 ca-certificates kernel samba-winbind-clients-0:3.6.23-45.el6_9 samba-winbind-0:3.6.23-45.el6_9 dhclient-12:4.1.1-53.P1.el6_9.1 glibc-devel kernel-headers poppler bind-libs-32:9.8.2-0.62.rc1.el6_9.4 sssd-client perf sssd-krb5-common sssd-ad selinux-policy-targeted procps sssd dhcp-common-12:4.1.1-53.P1.el6_9.1 python-sss-murmur glibc-common python-sssdconfig nss-tools openssh-server-5.3p1-123.el6_9 python-rhsm-certificates"
                                  10/29/17 01:06:21.653 DEBUG    bldeploy - [2][New RPM Group] chdir to PkgDir: '/opt/bmc/bladelogic/RSCD/Transactions/d0552d2c6ef0359394639d1b876b439b/'
                                  10/29/17 01:06:21.657 INFO     bldeploy - [2][New RPM Group] [stdout: 2]     Undo is not supported

                                   

                                  10/29/17 01:06:21.657 WARN     bldeploy - [2][New RPM Group] [stderr: 2]     ./bldeploycmd.2.bat: line 2: exit: too many arguments

                                  • 14. Re: Patch deploy/commit errors

                                    Thanks for the confirmation, Bill.!