6 Replies Latest reply on Apr 15, 2010 8:26 PM by Bill Robinson

    7.6 linux patching error

      has anyone seen this error before with 7.6 suse patching? There are no scratchpad

      logs, not any transaction logs being created.

       

      -------

      Info Apr 15, 2010 1:51:45 PM Started running the job 'SLES10 x86_64 Patch Analysis Job' on application server 'u-pr-065.clear.co.nz'(2,000,000)
      Info Apr 15, 2010 1:51:46 PM Executing work item NSH Script Job:SLES10 x86_64 Patch Analysis Job;  on application server: u-pr-065.clear.co.nz
      Info Apr 15, 2010 1:51:47 PM Number of Servers to process in parallel per platform: 10
      Info Apr 15, 2010 1:51:47 PM doing analysis only: mode is "a"
      Info Apr 15, 2010 1:51:47 PM DEBUG: nsh -c 'test -d "//u-pr-065/bladelogic/suse/testing/current_patch_cycle/\$RCE/SLES10-SP3-Pool/sles-10-x86_64/"'
      Info Apr 15, 2010 1:51:47 PM DEBUG: DEBUG: child exited with return code: 0
      Info Apr 15, 2010 1:51:47 PM DEBUG: return code: 0
      Info Apr 15, 2010 1:51:47 PM DEBUG: command returned: 0
      Info Apr 15, 2010 1:51:47 PM DEBUG: nsh -c 'test -f "//u-pr-065/bladelogic/suse/testing/current_patch_cycle/\$RCE/SLES10-SP3-Pool/sles-10-x86_64//repodata.zip"'
      Info Apr 15, 2010 1:51:47 PM DEBUG: DEBUG: child exited with return code: 0
      Info Apr 15, 2010 1:51:47 PM DEBUG: return code: 0
      Info Apr 15, 2010 1:51:47 PM DEBUG: command returned: 0
      Info Apr 15, 2010 1:51:56 PM Job Unique Id: a2cd65116c6258fe9d4c2c12592ae5e1
      Info Apr 15, 2010 1:51:56 PM DEBUG: nsh -c 'mkdir -p /opt/nsh/patch/linuxpu/Work/a2cd65116c6258fe9d4c2c12592ae5e1'
      Info Apr 15, 2010 1:51:56 PM DEBUG: DEBUG: child exited with return code: 0
      Info Apr 15, 2010 1:51:56 PM DEBUG: return code: 0
      Info Apr 15, 2010 1:51:56 PM DEBUG: command returned: 0
      Info Apr 15, 2010 1:51:56 PM DEBUG: nsh -c 'rm -f /opt/nsh/patch/linuxpu/Work/job_include_list.txt'
      Info Apr 15, 2010 1:51:56 PM DEBUG: DEBUG: child exited with return code: 0
      Info Apr 15, 2010 1:51:56 PM DEBUG: return code: 0
      Info Apr 15, 2010 1:51:56 PM DEBUG: command returned: 0
      Info Apr 15, 2010 1:51:56 PM DEBUG: nsh -c 'rm -f /opt/nsh/patch/linuxpu/Work/job_exclude_list.txt'
      Info Apr 15, 2010 1:51:56 PM DEBUG: DEBUG: child exited with return code: 0
      Info Apr 15, 2010 1:51:56 PM DEBUG: return code: 0
      Info Apr 15, 2010 1:51:56 PM DEBUG: command returned: 0
      Info Apr 15, 2010 1:51:56 PM job exclude file not found: ignoring job exclude list.
      Info Apr 15, 2010 1:51:56 PM DEBUG: exclude_list:
      Info Apr 15, 2010 1:51:56 PM DEBUG: nsh -c 'rm -rf /opt/nsh/patch/linuxpu/Work/a2cd65116c6258fe9d4c2c12592ae5e1/u-pr-066 && mkdir -p /opt/nsh/patch/linuxpu/Work/a2cd65116c6258fe9d4c2c12592ae5e1/u-pr-066'
      Info Apr 15, 2010 1:51:56 PM DEBUG: DEBUG: child exited with return code: 0
      Info Apr 15, 2010 1:51:56 PM DEBUG: return code: 0
      Info Apr 15, 2010 1:51:56 PM DEBUG: command returned: 0
      Info Apr 15, 2010 1:51:56 PM DEBUG: Executing Command: "/opt/nsh/patch/Jython/bljython"  "/opt/nsh/patch/linuxpu/Work/../Scripts/Jython/linux-analysis.py" "SLES10_SP3_Pool_sles_10_x86_64_tag" "/opt/nsh/patch/linuxpu/Work/linuxrepo.conf" "/Patch Analysis Items/Linux Patch Analysis" "/Patch Analysis Jobs/Linux Patch Analysis" "/opt/nsh/tmp/application_server/scripts/job__80fdd655-7168-432a-b053-4f98208c8ea4/hosts_363f85f1-a53c-44e4-8f9e-c47a89d0c9b1" "/opt/nsh/patch/linuxpu/Work" "10" "0" "a2cd65116c6258fe9d4c2c12592ae5e1-|-DBKey:SJobKey:2000140-2" "1" "0" "24385"
      Info Apr 15, 2010 1:51:58 PM Prefix = 24385-u-pr-065
      Info Apr 15, 2010 1:51:58 PM Using yum analysis type: update
      Info Apr 15, 2010 1:51:59 PM Using Host Staging Location: /var/bladelogic/a2cd65116c6258fe9d4c2c12592ae5e1
      Info Apr 15, 2010 1:51:59 PM Update Bootloader: 0
      Info Apr 15, 2010 1:51:59 PM Internal deploy job timeout percentage: 0
      Info Apr 15, 2010 1:51:59 PM Current AppServer version is: 7.6.0.204
      Info Apr 15, 2010 1:51:59 PM AppServer version is: 7.6.0.204
      Info Apr 15, 2010 1:52:07 PM Reading tag file */opt/nsh/patch/linuxpu/Work/linuxrepo.conf*
      Info Apr 15, 2010 1:52:07 PM Scanning all hosts for applicability.
      Info Apr 15, 2010 1:52:12 PM Scanning host u-pr-066 for platform...  [SLES10x86_64]
      Info Apr 15, 2010 1:52:12 PM Generating base yum.conf for all targets
      Info Apr 15, 2010 1:52:12 PM Writting to file /opt/nsh/patch/linuxpu/Work/a2cd65116c6258fe9d4c2c12592ae5e1/yum.conf
      Info Apr 15, 2010 1:52:12 PM yum.conf generation complete.
      Info Apr 15, 2010 1:52:12 PM cd /opt/nsh/patch/linuxpu/Work && cp linux-*.sh a2cd65116c6258fe9d4c2c12592ae5e1/ && cd a2cd65116c6258fe9d4c2c12592ae5e1 && tar -czf patch-meta.tar.gz rpm-includes.lst linux-*.sh yum.conf && rm -rf rpm* linux*
      Info Apr 15, 2010 1:52:13 PM Creating meta zip... Done
      Info Apr 15, 2010 1:52:13 PM Creating custom softwares...
      Info Apr 15, 2010 1:52:13 PM Creating custom software item 24385-u-pr-065-cs-meta... ['DepotSoftware', 'addCustomSoftwareToDepotByGroupName', '/Patch Analysis Items/Linux Patch Analysis/ScratchPad', 'Linux', 'Custom Software', '/opt/nsh/patch/linuxpu/Work/a2cd65116c6258fe9d4c2c12592ae5e1/patch-meta.tar.gz', '24385-u-pr-065-cs-meta', 'Internal Custom Software', 'mkdir -p /var/bladelogic/a2cd65116c6258fe9d4c2c12592ae5e1 && mv -f ??SOURCE?? /var/bladelogic/a2cd65116c6258fe9d4c2c12592ae5e1', '', '', '', '', 'false', 'AGENT_COPY_AT_STAGING']
      Info Apr 15, 2010 1:52:17 PM Done
      Info Apr 15, 2010 1:52:17 PM Creating custom softwares...
      Info Apr 15, 2010 1:52:17 PM Creating custom software item 24385-u-pr-065-cs-SLES10_SP3_Pool_sles_10_x86_64_tag... ['DepotSoftware', 'addCustomSoftwareToDepotByGroupName', '/Patch Analysis Items/Linux Patch Analysis/ScratchPad', 'Linux', 'Custom Software', '//u-pr-065/bladelogic/suse/testing/current_patch_cycle/\\$RCE/SLES10-SP3-Pool/sles-10-x86_64//repodata.zip', '24385-u-pr-065-cs-SLES10_SP3_Pool_sles_10_x86_64_tag', 'Internal Custom Software', 'mkdir -p "/var/bladelogic/a2cd65116c6258fe9d4c2c12592ae5e1/blrepos" && rm -rf "/var/bladelogic/a2cd65116c6258fe9d4c2c12592ae5e1/blrepos/repo-SLES10_SP3_Pool_sles_10_x86_64_tag" && mv -f ??SOURCE?? "/var/bladelogic/a2cd65116c6258fe9d4c2c12592ae5e1/blrepos/repo-SLES10_SP3_Pool_sles_10_x86_64_tag"', '', '', '', '', 'false', 'AGENT_COPY_AT_STAGING']
      Info Apr 15, 2010 1:52:17 PM Done
      Info Apr 15, 2010 1:52:18 PM Creating temporary folder 24385-u-pr-065 in /Patch Analysis Jobs/Linux Patch Analysis/ScratchPad Done
      Info Apr 15, 2010 1:52:19 PM Creating BLPackage 24385-u-pr-065-blp-SLES10x86_64...  Done
      Info Apr 15, 2010 1:52:19 PM Creating deploy Job SLES10x86_64 in /Patch Analysis Jobs/Linux Patch Analysis/ScratchPad/24385-u-pr-065
      Info Apr 15, 2010 1:52:24 PM Done
      Info Apr 15, 2010 1:52:24 PM Filling Blpackages...
      Info Apr 15, 2010 1:52:26 PM Importing 24385-u-pr-065-cs-meta to 24385-u-pr-065-blp-SLES10x86_64...  Done
      Info Apr 15, 2010 1:52:26 PM Running deploy job DBKey:SJobKey:2000393-1...
      Info Apr 15, 2010 1:52:27 PM Trying to find out RunKeys for job DBKey:SJobKey:2000393-1... Couldn't find the runkeys: tryin again...
      Info Apr 15, 2010 1:52:29 PM Couldn't find the runkeys: tryin again...
      Info Apr 15, 2010 1:52:31 PM Couldn't find the runkeys: tryin again...
      Info Apr 15, 2010 1:52:33 PM Run Keys = ['DBKey:SJobRunKey:2015057', 'DBKey:SJobRunKey:2015056']
      Info Apr 15, 2010 1:52:43 PM Checking if Job Run DBKey:SJobRunKey:2015057 is completed...  Failed
      Info Apr 15, 2010 1:52:43 PM Failed deploy job is: Linux SLES10x86_64 Analysis Job
      Info Apr 15, 2010 1:52:43 PM Extracting logs for failed Analysis Deploy Job Linux SLES10x86_64 Analysis Job.
      Info Apr 15, 2010 1:52:43 PM Error: com.bladelogic.mfw.util.NotFoundException: job run with id 0 not found
      Info Apr 15, 2010 1:52:44 PM Exit Code 1
      Info Apr 15, 2010 1:52:44 PM The job 'SLES10 x86_64 Patch Analysis Job' has failed
      Error Apr 15, 2010 1:52:44 PM Traceback (innermost last):
      Error Apr 15, 2010 1:52:44 PM   File "/opt/nsh/patch/linuxpu/Work/../Scripts/Jython/linux-analysis.py", line 1317, in ?
      Error Apr 15, 2010 1:52:44 PM   File "/opt/nsh/patch/linuxpu/Work/../Scripts/Jython/linux-analysis.py", line 1298, in main
      Error Apr 15, 2010 1:52:44 PM   File "/opt/nsh/patch/linuxpu/Work/../Scripts/Jython/linux-analysis.py", line 1134, in run_deploy_job
      Error Apr 15, 2010 1:52:44 PM   File "/opt/nsh/patch/linuxpu/Work/../Scripts/Jython/linux-analysis.py", line 1227, in collect_logs
      Error Apr 15, 2010 1:52:44 PM   File "/opt/nsh/patch/linuxpu/Work/../Scripts/Jython/linux-analysis.py", line 1204, in blcli_exec
      Error Apr 15, 2010 1:52:44 PM   File "/opt/nsh/br/jython/bladelogic/cli/CLI.py", line 105, in run
      Error Apr 15, 2010 1:52:44 PM TypeError: sequence item 2: expected string, 'None' object found
      Error Apr 15, 2010 1:52:44 PM Analysis failed, execution of linux-analysis.py failed.

        • 1. Re: 7.6 linux patching error
          Bill Robinson

          you mean patching?

           

          the deploy job is probably failing.  open up the linux-analysis.py and comment out the last occurrence of the 'cleanup()' call.  that should leave a copy of the deploy job in the scratchpad folder (the cleanup function removes the scratchpad folder during run)

          • 2. Re: 7.6 linux patching error

            you mean this:

             


                    # closing
                    closing()

                finally:
                    # cleanup all intermediate data
                    #cleanup()
                    pass

                return 0

            • 3. Re: 7.6 linux patching error
              Bill Robinson

              yep.  comment that out and re-run the job, the scratchpad folder should be there and the deploy job for the metadata should be there.

              • 4. Re: 7.6 linux patching error

                Error from scratchpad job:

                 

                Handle soft link problem : BLAST Copy Failed: command=[/opt/nsh/bin/blast sync --nodelete //u-pr-065/bladelogic/suse/testing/current_patch_cycle/\$RCE/SLES10-SP3-Pool/sles-10-x86_64//repodata.zip //u-pr-066/var/tmp/stage/25238-u-pr-065-blp-SLES10x86_64-2001573.1-2000422.4/2001572.1/], exitCode=1 (cp: Unable to access file //u-pr-065/bladelogic/suse/testing/current_patch_cycle/\$RCE/SLES10-SP3-Pool/sles-10-x86_64//repodata.zip: No such file or directory
                *** blast receive (pid=25368): Error executing command: cp -rP '//u-pr-065/bladelogic/suse/testing/current_patch_cycle/\$RCE/SLES10-SP3-Pool/sles-10-x86_64//repodata.zip' '/opt/nsh/nodes/localhost/requests/u-pr-065_2010-04-15-14.33.21+25359/dataset'
                *** blast send (pid=25366): receive process exited without setting data-ready event)

                 

                the repodata.zip EXISTS, are the permissions incorrect?

                u-pr-065:/bladelogic/suse/testing/current_patch_cycle/$RCE/SLES10-SP3-Pool/sles
                -10-x86_64 # ls -lart
                total 40596
                drwxr-xr-x 4 root root     4096 2010-03-09 16:51 ..
                drwxr-xr-x 5 root root     4096 2010-03-09 16:51 rpm
                drwxr-xr-x 4 root root     4096 2010-03-16 16:27 repodata
                -rw-r--r-- 1 root root 41466449 2010-03-16 16:27 repodata.zip
                drwxr-xr-x 4 root root     4096 2010-03-16 16:27 .
                u-pr-065:/bladelogic/suse/testing/current_patch_cycle/$RCE/SLES10-SP3-Pool/sles
                -10-x86_64 # pwd
                /bladelogic/suse/testing/current_patch_cycle/$RCE/SLES10-SP3-Pool/sles-10-x86_64
                u-pr-065:/bladelogic/suse/testing/current_patch_cycle/$RCE/SLES10-SP3-Pool/sles
                -10-x86_64 #

                • 5. Re: 7.6 linux patching error
                  Bill Robinson

                  are the agent acls ok on u-pr-065?

                   

                  what are the permissions in /usr/nsh/data-cache/* and /urs/nsh/nodes/* ?

                   

                  /usr/nsh/bin/blast is sgid?

                  • 6. Re: 7.6 linux patching error
                    Stefan Kruth

                    As teh path to blast is /opt/nsh/bin/blast, did you install with -local flag? In that case LD_LIBRARY_PATH will not work for sgid. If it is the case, create the soft links under /lib to point to required libs in /opt/nsh/lib. You can figure out the reqd ones with ldd.