3 Replies Latest reply on Oct 5, 2015 7:36 PM by Yanick Girouard

    Deploy command fails on AIX server.

    Neal Meagher

      The Pre command "rm -rf /opt/miror/aci/prod/packages/" fails, but the BL package runs successful. I have checked permission they look ok 775. User.local is set as it should. Only have one AIX server in my environment.

       

       

       

      [29 Sep 2015 17:03:46,744] [Client-Connections-Thread-5] [INFO] [ESP:CommandCenter:10.104.76.159] [Client] User 'ESP' assumed the role 'CommandCenter'

      [29 Sep 2015 17:04:08,523] [Job-Execution-1] [INFO] [ESP:CommandCenter:] [Deploy] Started running the job 'Deploy MIR_IBM_PRD' with priority 'NORMAL' on application server 'vml150.windstream.com'(2,000,000)

      [29 Sep 2015 17:04:13,851] [WorkItem-Thread-24] [INFO] [ESP:CommandCenter:] [Deploy] Job 'Deploy MIR_IBM_PRD' is executing a pre-command of 'rm -rf /opt/miror/aci/prod/packages/*' on server vma101

      [29 Sep 2015 17:04:13,942] [WorkItem-Thread-24] [WARN] [ESP:CommandCenter:] [Deploy] Apply preCmd rm -rf /opt/miror/aci/prod/packages/* Failed ret = 13 for server vma101

      [29 Sep 2015 17:04:16,392] [WorkItem-Thread-8] [INFO] [ESP:CommandCenter:] [Deploy] Job 'Deploy MIR_IBM_PRD' is executing a pre-command of 'rm -rf /opt/miror/aci/prod/packages/*' on server vma102

      [29 Sep 2015 17:04:16,477] [WorkItem-Thread-8] [WARN] [ESP:CommandCenter:] [Deploy] Apply preCmd rm -rf /opt/miror/aci/prod/packages/* Failed ret = 13 for server vma102

      [29 Sep 2015 17:04:21,207] [Scheduled-System-Tasks-Thread-11] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 709885952,Free JVM (B): 438057624,Used JVM (B): 271828328,VSize (B): 2116091904,RSS (B): 1179578368,Used File Descriptors: 288,Used Work Item Threads: 0/50,Used Client Connections: 5/200,DB Client-Connection-Pool: 3/3/0/100,DB Job-Connection-Pool: 3/3/0/100,DB General-Connection-Pool: 1/1/0/100

      [29 Sep 2015 17:04:37,917] [Job-Execution-2] [INFO] [ESP:CommandCenter:] [Deploy] The job 'Deploy MIR_IBM_PRD' has succeeded

      [29 Sep 2015 17:04:43,574] [Client-Connections-Thread-0] [INFO] [ESP:CommandCenter:10.104.76.159] [Client] Connection disconnecting: id = 3185

      [29 Sep 2015 17:04:43,576] [Client-Connections-Thread-5] [INFO] [ESP:CommandCenter:10.104.76.159] [Client] Connection disconnecting: id = 3184

      [29 Sep 2015 17:05:21,207] [Scheduled-System-Tasks-Thread-6] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 707788800,Free JVM (B): 473548296,Used JVM (B): 234240504,VSize (B): 2116091904,RSS (B): 1189552128,Used File Descriptors: 292,Used Work Item Threads: 0/50,Used Client Connections: 3/200,DB Client-Connection-Pool: 4/4/0/100,DB Job-Connection-Pool: 3/3/0/100,DB General-Connection-Pool: 2/2/0/100

        • 1. Re: Deploy command fails on AIX server.
          Yanick Girouard

          Is there a noexec flag on the staging dir set for the target? (default /var/tmp/stage) ? Check the value of the STAGING_DIR property of the server. If it's blank, then it's /var/tmp/stage

           

          try a mount command and check the attributes of the filesystem owning the directory. This is where pre and post command scripts get copied and executed form. If there is a noexec flag on it, it will be able to copy it but then will delete it when it tries to execute, and show an error along the line of  "command not found"

          • 2. Re: Deploy command fails on AIX server.
            Neal Meagher

            The server directory is wide open – I was able to manually as user 'myuser" create a file in this directory.

            I don’t know what a noexec flag is.

             

            0]vma101:/var/tmp/stage> ls -la

            total 8

            drwxrwxrwt    2 bldeploy bldeploy        256 Oct  5 14:45 ./

            drwxrwxrwt    5 bin      bin            4096 Oct  5 14:40 ../

            [0]vma101:/var/tmp/stage> pwd

            /var/tmp/stage

            The script actually runs  if I run it from  /var/tmp/stage

            0]vma101:/var/tmp/stage> ls -la

            total 8

            drwxrwxrwt    2 bldeploy bldeploy        256 Oct  5 14:45 ./

            drwxrwxrwt    5 bin      bin            4096 Oct  5 14:40 ../

            [0]vma101:/var/tmp/stage> /opt/miror/aci/prod/packages/

            [0]vma101:/var/tmp/stage> rm -rf /opt/miror/aci/prod/packages/*



            I did notice the app server on 8.5.1 and agent is 8.2.2. I klnow this may cuase strange issues. Going to push agent upgrade  to 8.5.1.

            • 3. Re: Deploy command fails on AIX server.
              Yanick Girouard

              You say the "script" runs fine if you run it from the directory, but you are mistaking a command for a script. If you want to check fo rthe noexec flag preventing you from running a script, put that command in a file like so:

               

              #! /bin/sh

              rm -rf /opt/miror/aci/prod/packages/*

               

              Save it as test.sh, run a chmod +x on it to give it execute permissions, and then try to run the script like this from the directory:

               

              ./test.sh

               

              If this doesn't give you any error, it's not the noexec, but if it says something like "Command not found", it might be it.

               

              The noexec flag is an option you can set when mounting a filesystem. As a security measure, many admins mount the /tmp, and/or /var/tmp filesystems with a noexec flag to prevent scripts from executing from it. You will still have all the permissions you need to create and edit files there, but not execute anything. Even if the file as the +x permission, it won't let you execute it.

               

              Because the pre and post commands of deploy jobs actually create a .sh script in the staging directory and then call it, if there is a noexec flag on the filesystem where the staging directory resides, it will fail to execute, even if you have all the permissions in the world on it.

               

              To see if there's a noexec flag on the staging directory's filesystem, do this:

               

              1. Confirm which filesystem owns the staging directory:

               

              df /var/tmp/stage

               

              # make sure to add the full path of the staging dir because often, /var/tmp is a symbolic link to /tmp.

               

              2. Look at the last column, should be "Mounted On". Note the filesystem it's on.

              3. Run a mount command and grep the filesystem in question:

               

              mount | grep "/filesystem"

               

              i.e. if the filesystem was /tmp, the command would be: mount | grep "/tmp"

               

              4. Look at the options for the mount of the filesystem, if you see noexec in the list, that's your problem. If not, it's not that.

               

              5. If it is there, your options are as follow:

               

              a) change the STAGING_DIR property of the target to another filesystem/directory that was not mounted with noexec

              b) Remount the filesystem without the noexec flag and keep it that way or else BladeLogic will not be able to execute any pre and post commands.