4 Replies Latest reply on Dec 20, 2011 9:47 PM by Avikal Jain

    Deploy jobs failing with exit code 117 at simulate phase.

    Avikal Jain

      Experts, my deploy jobs are failing with exit code 117 at simulate phase, the deploy job utilizes a blapackage to run a shell script as external command, the job used run properly some time back and suddenly I am seeing this error, here is the deploy job log of two different executions:

       

       

       

       

       

      Info Dec 19, 2011 1:22:16 PM The job '006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US->006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US' has failed on server xxxxxxxxx 
      Info Dec 19, 2011 1:20:44 PM Started running the deploy step job '006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US->006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US' on application server 'nygshspltp24a_us_db_com'(2,007,000) against target server 'xxxxxxxxx' 
      Error Dec 19, 2011 1:22:16 PM DRYRUN failed for server londbdiwpltu2.uk.db.com. Exit code = -1 
      Info Dec 19, 2011 1:20:44 PM Deploy Dry Run Job:006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US; Server:xxxxxxxx;  PkgID:"DBDI_COMPONENT_VERIFICATION_PRE_DEPLOY_SANITY-2005875.1-2036973.6"; UUID:a2c7bcbf9f833f3082edc3059fb3ba2f 
      Info Dec 19, 2011 1:22:16 PM Command received signa 
      Error Dec 19, 2011 1:22:16 PM 006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US->006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US failed on server xxxxxxxx exit code = -1 


      Info Dec 19, 2011 11:35:47 AM The job '006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US->006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US' has failed on server xxxxxxxxx 
      Info Dec 19, 2011 11:35:44 AM Started running the deploy step job '006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US->006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US' on application server 'nyc-js-01'(2,004,000) against target server 'xxxxxxxxx' 
      Error Dec 19, 2011 11:35:47 AM DRYRUN failed for server XXXXXXXX. Exit code = 117 
      Info Dec 19, 2011 11:35:44 AM Deploy Dry Run Job:006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US; Server:londbdiwpltu2.uk.db.com;  PkgID:"DBDI_COMPONENT_VERIFICATION_PRE_DEPLOY_SANITY-2005875.1-2036973.6"; UUID:b084c3b260b63e198b26abde30cc4bfa 
      Error Dec 19, 2011 11:35:47 AM 006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US->006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US failed on server XXXXXXXXXX exit code = 117 

       

      Transaction logs are not generating on the host, the appserver log has following entries:

       

      [19 Dec 2011 00:19:28,242] [Scheduled-System-Tasks-Thread-1] [INFO] [System yst
      em:] [Memory Monitor] Total JVM  : 1708851200,Free JVM  : 1418195704,Used JV
      M  : 290655496,VSize  : 1915043840,RSS  : 1909645312,Used File Descriptors
      : 383
      [19 Dec 2011 00:20:28,241] [Scheduled-System-Tasks-Thread-5] [INFO] [System yst
      em:] [Memory Monitor] Total JVM  : 1708851200,Free JVM  : 1404945008,Used JV
      M  : 303906192,VSize  : 1915043840,RSS  : 1909645312,Used File Descriptors
      : 383
      [19 Dec 2011 00:20:43,538] [WorkItem-Thread-34] [INFO] [jainavi@DBG.COM
      BDI-493-8_DPLY_sdbius1_SIT-F1-US:] [Deploy] Executing work item Deploy Dry Run J
      ob:006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US; Server:xxxxxxxx;  o
      n application server: nygshspltp24a.us.db.com
      [19 Dec 2011 00:20:45,875] [WorkItem-Thread-34] [INFO] [jainavi@DBG.COM
      BDI-493-8_DPLY_sdbius1_SIT-F1-US:] [Deploy] executing command bltargetjobmanager
      -start -cmd "bldeploy a2c7bcbf9f833f3082edc3059fb3ba2f -N="/var/tmp/stage/a2c7b
      cbf9f833f3082edc3059fb3ba2f" -P=xxxxxxxx -Q=params.txt -DryRun -X
      r -V2 -js0 -jr0 -jc0 " -me "a2c7bcbf9f833f3082edc3059fb3ba2f" -mp -sp  -ps  -h60
      on server xxxxxxxx
      [19 Dec 2011 00:21:28,241] [Scheduled-System-Tasks-Thread-1] [INFO] [System yst
      em:] [Memory Monitor] Total JVM  : 1710096384,Free JVM  : 1516949968,Used JV
      M  : 193146416,VSize  : 1919246336,RSS  : 1909653504,Used File Descriptors
      : 385
      [19 Dec 2011 00:22:16,611] [WorkItem-Thread-34] [ERROR] [jainavi@DBG.COM:
      DBDI-493-8_DPLY_sdbius1_SIT-F1-US:] [Deploy] DRYRUN failed for server xxxxxxxx. Exit code = -1
      [19 Dec 2011 00:22:16,621] [WorkItem-Thread-34] [ERROR] [jainavi@DBG.COM:
      DBDI-493-8_DPLY_sdbius1_SIT-F1-US:] [Deploy] 006_VERIFY_COMPONENTS_WEBSERVER_SIT
      -F1-US->006_VERIFY_COMPONENTS_WEBSERVER_SIT-F1-US failed on server xxxxxxxx exit code = -1

      [19 Dec 2011 00:22:28,242] [Scheduled-System-Tasks-Thread-2] [INFO] [System yst
      em:] [Memory Monitor] Total JVM  : 1711931392,Free JVM  : 1411324816,Used JV
      M  : 300606576,VSize  : 1919246336,RSS  : 1913847808,Used File Descriptors
      : 391

       

       

      Agent and app server version is 8.0.6.622.

       

      Any ideas what might have been causing the issue. Tried running a sample job using bladmins as well but that fails too. ACL push and Update server properties jobs are running fine on the host though.

       

        • 1. Re: Deploy jobs failing with exit code 117 at simulate phase.
          Bill Robinson

          What is in the rsc files on the target?  are there mapping entries for the role:user trying to run the job?

          • 2. Deploy jobs failing with exit code 117 at simulate phase.
            Avikal Jain

            exports file:

             

            #

            # Copyright (c) 2001-2009 BladeLogic, Inc.

            # -- All Rights Reserved --

            #

            # This file is read by the "rscd" to determine permissions for the given host.

            #

            # Please read the BladeLogicAdministration.pdf or "exports" man page for details

            # on how to use this file.

            #

            * rw

             

            users.local file:

             

            #

            # Copyright (c) 2001-2009 BladeLogic, Inc.

            # -- All Rights Reserved --

            #

            # This file contains a list of user permission overrides. The permissions

            # defined in this file will override any associated permissions defined in the

            # "exports" or "users" file.

            #

            # Please read the BladeLogicAdministration.pdf for details on how to use this

            # file.

            #

            BLAdmins:* rw,map=root

             

            BLAdmin rw,map=root

             

            users file looks perfect as well, all mapping are correct.

             

            And here is the transaction log of the most recent execution of the same job, I was looking under the wrong directory, sorry about that:

             

            12/19/11 15:00:41.323 DEBUG bldeploy - TJM invoked with arguments:

            12/19/11 15:00:41.327 DEBUG bldeploy - args 0 = bltargetjobmanager

            12/19/11 15:00:41.327 DEBUG bldeploy - args 1 = -start

            12/19/11 15:00:41.327 DEBUG bldeploy - args 2 = -cmd

            12/19/11 15:00:41.327 DEBUG bldeploy - args 3 = "bldeploy

            12/19/11 15:00:41.327 DEBUG bldeploy - args 4 = 8d02daf9fc3131218005deacbfb1628e

            12/19/11 15:00:41.327 DEBUG bldeploy - args 5 = -N="/var/tmp/stage/8d02daf9fc3131218005deacbfb1628e"

            12/19/11 15:00:41.327 DEBUG bldeploy - args 6 = -P=londbdiwpltu2.uk.db.com

            12/19/11 15:00:41.327 DEBUG bldeploy - args 7 = -Q=params.txt

            12/19/11 15:00:41.327 DEBUG bldeploy - args 8 = -DryRun

            12/19/11 15:00:41.327 DEBUG bldeploy - args 9 = -Xr

            12/19/11 15:00:41.327 DEBUG bldeploy - args 10 = -V2

            12/19/11 15:00:41.327 DEBUG bldeploy - args 11 = -js0

            12/19/11 15:00:41.327 DEBUG bldeploy - args 12 = -jr0

            12/19/11 15:00:41.327 DEBUG bldeploy - args 13 = -jc0

            12/19/11 15:00:41.327 DEBUG bldeploy - args 14 = "

            12/19/11 15:00:41.327 DEBUG bldeploy - args 15 = -me

            12/19/11 15:00:41.327 DEBUG bldeploy - args 16 = "8d02daf9fc3131218005deacbfb1628e"

            12/19/11 15:00:41.327 DEBUG bldeploy - args 17 = -mp

            12/19/11 15:00:41.327 DEBUG bldeploy - args 18 = -sp

            12/19/11 15:00:41.327 DEBUG bldeploy - args 19 = -ps

            12/19/11 15:00:41.327 DEBUG bldeploy - args 20 = -h60

            12/19/11 15:00:41.327 DEBUG bldeploy - BlTargetJobManager started

            12/19/11 15:00:41.327 DEBUG bldeploy - Commandline: bltargetjobmanager -start -cmd bldeploy 8d02daf9fc3131218005deacbfb1628e -N="/var/tmp/stage/8d02daf9fc3131218005deacbfb1628e" -P=londbdiwpltu2.uk.db.com -Q=params.txt -DryRun -Xr -V2 -js0 -jr0 -jc0 -me "8d02daf9fc3131218005deacbfb1628e" -mp -sp -ps -h60

            12/19/11 15:00:43.355 DEBUG bldeploy - Executing bldeploy 8d02daf9fc3131218005deacbfb1628e -N="/var/tmp/stage/8d02daf9fc3131218005deacbfb1628e" -P=londbdiwpltu2.uk.db.com -Q=params.txt -DryRun -Xr -V2 -js0 -jr0 -jc0 -StartNew

            12/19/11 15:00:43.355 DEBUG bldeploy - Invoke command: bldeploy 8d02daf9fc3131218005deacbfb1628e -N="/var/tmp/stage/8d02daf9fc3131218005deacbfb1628e" -P=londbdiwpltu2.uk.db.com -Q=params.txt -DryRun -Xr -V2 -js0 -jr0 -jc0 -StartNew

            12/19/11 15:00:44.357 DEBUG bldeploy - Compare to blp_prog=bltargetjobmanager

            12/19/11 15:00:44.357 DEBUG bldeploy - No match for PID=3055652 for blproc=bltargetjobmanager and exeName=bldeploy

            12/19/11 15:00:44.357 DEBUG bldeploy - Command has started

            12/19/11 15:00:44.358 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:45.358 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:46.358 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:47.359 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:48.359 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:49.360 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:50.360 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:51.361 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:52.361 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:53.361 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:54.362 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:55.362 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:56.363 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:57.363 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:58.364 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:00:59.364 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:00.364 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:01.365 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:02.365 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:03.366 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:04.366 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:05.367 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:06.367 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:07.368 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:08.368 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:09.368 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:10.369 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:11.369 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:12.370 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:13.370 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:14.371 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:15.371 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:16.371 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:17.372 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:18.372 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:19.373 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:20.373 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:21.374 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:22.374 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:23.374 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:24.375 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:25.375 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:26.376 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:27.376 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:28.376 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:29.377 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:30.377 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:31.378 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:32.378 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:33.379 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:34.379 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:35.380 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:36.380 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:37.380 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:38.381 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:39.381 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:40.382 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:41.382 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:42.383 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:43.383 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:44.383 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:45.384 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:46.384 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:47.385 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:48.385 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:49.385 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:50.386 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:51.386 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:52.387 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:53.387 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:54.388 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:55.388 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:56.389 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:57.389 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:58.389 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:01:59.390 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:02:00.390 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:02:01.391 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:02:02.391 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:02:03.392 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:02:04.392 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:02:05.393 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:02:06.393 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:02:07.393 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:02:08.394 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:02:09.394 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:02:10.395 INFO bldeploy - Monitor PID not yet read from event file

            12/19/11 15:02:11.285 INFO bldeploy - Command received signal: SIGTERM

            • 3. Deploy jobs failing with exit code 117 at simulate phase.
              Bill Robinson

              this looks like the targetmanager log - do you have the bldeploy-blahblah ?

              • 4. Deploy jobs failing with exit code 117 at simulate phase.
                Avikal Jain

                Thats correct, I am sorry.

                 

                There is nothing recent in the Transactions -> log folder except two files dated 8 Nov 2011 of successful deployment. Them shows that the agent version at that moment was 7.6.0.266.

                 

                Also, i mentioned that our app server and agent is both running 8.0.6.622 seems to be incorrect as well. Our app server is 8.0.11.1107 and agent is 8.0.6.622.

                 

                I think that is where the problem lies, anything else I should check before asking agent upgradation to appropriate team?