1 2 Previous Next 20 Replies Latest reply on Oct 27, 2016 12:40 PM by Bill Robinson

    BSA 8.6.01  failing with pipe failed: too many open files.

    Neal Meagher

      I have set the Ulimit as below restarted app/proc services and I still get failing with pipe failed: too many open files when running NSH script job. I have another server running opsware and made change and they took.

       

      *               soft    nofile          16384

      *               hard    nofile          65536

       

      bladmin soft nofile 8192

      bladmin hard nofile 8192

       

       

      I changed it to:

       

       

      bladmin soft nofile 16384

      bladmin hard nofile 65536

       

       

       

      root@BLADELOGIC ~]# cat /proc/3876/limits
      Limit Soft Limit Hard Limit Units
      Max cpu time unlimited unlimited seconds
      Max file size unlimited unlimited bytes
      Max data size unlimited unlimited bytes
      Max stack size 8388608 67108864 bytes
      Max core file size unlimited unlimited bytes
      Max resident set unlimited unlimited bytes
      Max processes 1024 47554 processes
      Max open files 8192 8192 files
      Max locked memory 32768 131072 bytes
      Max address space unlimited unlimited bytes
      Max file locks unlimited unlimited locks
      Max pending signals 47554 47554 signals
      Max msgqueue size 819200 1638400 bytes
      Max nice priority 0 0
      Max realtime priority 0 0
      Max realtime timeout unlimited unlimited us


      I do not know where this is being set..... But here is Opsware agent numbers after I changed it:

      [root@OPSWARE init.d]# cat /proc/5576/limits
      Limit Soft Limit Hard Limit Units
      Max cpu time unlimited unlimited seconds
      Max file size unlimited unlimited bytes
      Max data size unlimited unlimited bytes
      Max stack size 8388608 67108864 bytes
      Max core file size unlimited unlimited bytes
      Max resident set unlimited unlimited bytes
      Max processes 47554 47554 processes
      Max open files 16384 65536 files
      Max locked memory 32768 131072 bytes
      Max address space unlimited unlimited bytes
      Max file locks unlimited unlimited locks
      Max pending signals 47554 47554 signals
      Max msgqueue size 819200 1638400 bytes
      Max nice priority 0 0
      Max realtime priority 0 0
      Max realtime timeout unlimited unlimited us


      Worked perfectly in Opsware...

       

       

      Something is overriding the ulimit

        • 1. Re: BSA 8.6.01  failing with pipe failed: too many open files.
          Bill Robinson

          Raise nproc for bladmin as well.

          • 2. Re: BSA 8.6.01  failing with pipe failed: too many open files.
            Neal Meagher

            Found the setting in BLAPPSERV startup script

            • 3. Re: BSA 8.6.01  failing with pipe failed: too many open files.
              Bill Robinson

              did you actually exceed 8192 open files?  you should see that in the 'Memory Monitor' line in the appserver logs.

              • 4. Re: BSA 8.6.01  failing with pipe failed: too many open files.
                Neal Meagher

                I  changed the /etc/security/limits.conf for bladmin to 16384 and a Hard limit of 65536.

                I changed the startup script blappserv and blprocserv to 16384 on the line we found Restarted the services... and they are still showing up as 8192. There are no complete instructions on how to do this.

                • 5. Re: BSA 8.6.01  failing with pipe failed: too many open files.
                  Neal Meagher

                  Here are the setting for the user:

                   

                  [root@ulvblgp12 br]# su - bladmin

                  -sh-4.1$ ulimit -a

                  core file size (blocks, -c) unlimited

                  data seg size (kbytes, -d) unlimited

                  scheduling priority             (-e) 0

                  file size (blocks, -f) unlimited

                  pending signals                 (-i) 95145

                  max locked memory (kbytes, -l) 32

                  max memory size (kbytes, -m) unlimited

                  open files                      (-n) 16384

                  pipe size (512 bytes, -p) 8

                  POSIX message queues (bytes, -q) 819200

                  real-time priority              (-r) 0

                  stack size (kbytes, -s) 8192

                  cpu time (seconds, -t) unlimited

                  max user processes              (-u) 8192

                  virtual memory (kbytes, -v) unlimited

                  file locks                      (-x) unlimited

                  -sh-4.1$ ulimit -Ha

                  core file size (blocks, -c) unlimited

                  data seg size (kbytes, -d) unlimited

                  scheduling priority             (-e) 0

                  file size (blocks, -f) unlimited

                  pending signals                 (-i) 95145

                  max locked memory (kbytes, -l) 128

                  max memory size (kbytes, -m) unlimited

                  open files                      (-n) 65536

                  pipe size (512 bytes, -p) 8

                  POSIX message queues (bytes, -q) 1638400

                  real-time priority              (-r) 0

                  stack size (kbytes, -s) 65536

                  cpu time (seconds, -t) unlimited

                  max user processes              (-u) 8192

                  virtual memory (kbytes, -v) unlimited

                  file locks                      (-x) unlimited

                  -sh-4.1$

                  • 6. Re: BSA 8.6.01  failing with pipe failed: too many open files.
                    Bill Robinson

                    can you show where you are getting this error - in a log file or somewhere?  it's not in the appserver log you attached to the ticket.

                    • 7. Re: BSA 8.6.01  failing with pipe failed: too many open files.
                      Neal Meagher

                      Job Name,Collect MQ files Job - Run at 10/24/2016 23:50:11

                      Start Time,10/24/2016 23:50:11

                      End Time,10/25/2016 00:37:19

                      Status,Completed with Errors

                       

                       

                      Participant,Type,Date,Message

                      Run at 10/24/2016 23:50:11,Info,10/24/2016 23:50:11,Started running the job 'Collect MQ files Job' with priority 'NORMAL' on application server 'ulvblgp02.fg.rbc.com'(13)

                      Run at 10/24/2016 23:50:11,Info,10/24/2016 23:50:15,Executing work item NSH Script Job:Collect MQ files Job; Server:ulvblgp12;  on application server: ulvblgp09.fg.rbc.com

                      ulvblgp12,Info,10/24/2016 23:50:27,BLAdmins

                      ulvblgp12,Error,10/24/2016 23:50:27,//@/opt/bmc/bladelogic/NSH/tmp/application_server/scripts/job__ede6424c-ca30-4f96-afc3-3bc7d1c193e8/1be457aa-a5ed-42e4-bd61-10007b2360c7.script_DBKey-SJobKeyImpl-7643-2_discovery_script.txt:ulimit:10: value exceeds hard limit

                      ulvblgp12,Info,10/24/2016 23:50:27,ulvblgp12

                      ulvblgp12,Info,10/24/2016 23:50:27,8192

                      ulvblgp12,Info,10/24/2016 23:50:27,8192

                      ulvblgp12,Error,10/24/2016 23:51:43,SSL_connect

                      ulvblgp12,Error,10/24/2016 23:51:43,SSL_connect

                      ulvblgp12,Error,10/24/2016 23:57:43,SSL_connect

                      ulvblgp12,Error,10/24/2016 23:57:43,SSL_connect

                      ulvblgp12,Error,10/25/2016 00:00:04,SSL_connect

                      ulvblgp12,Error,10/25/2016 00:00:04,SSL_connect

                      ulvblgp12,Error,10/25/2016 00:37:18,//@/opt/bmc/bladelogic/NSH/tmp/application_server/scripts/job__ede6424c-ca30-4f96-afc3-3bc7d1c193e8/1be457aa-a5ed-42e4-bd61-10007b2360c7.script_DBKey-SJobKeyImpl-7643-2_discovery_script.txt:47: pipe failed: too many open files

                      ulvblgp12,Info,10/25/2016 00:37:18,Exit Code 1

                      Run at 10/24/2016 23:50:11,Error,10/25/2016 00:37:19,The job 'Collect MQ files Job' has failed

                      • 8. Re: BSA 8.6.01  failing with pipe failed: too many open files.
                        Bill Robinson

                        do you have the corresponding appserver log from ulvblgp09.fg.rbc.com, and what's in the script you are running ?

                        • 9. Re: BSA 8.6.01  failing with pipe failed: too many open files.
                          Neal Meagher

                          I see nothing in the app sever log.There job is reaching it limit and it fails after. Here is the script

                           

                           

                           

                          blcli_connect

                           

                           

                          blcli_execute Utility assumeRole BLAdmins

                           

                           

                          echo ""

                          uname -n

                          id

                          ulimit -n

                          ulimit -n 16384

                          ulimit -n

                          ulimit -Hn

                           

                           

                          DATE=`date +%Y-%m-%d-%H-%M`

                          GRP="/Administration/All_Active_UNIX"

                          #GRP="/Administration/Steve/znd_servers"

                          OUTFILE="/opt/bmc/fileserver/tmp/mq.tmp"

                          ELASTIK="//ulgwjsp01.fg.rbc.com/usr/local/groundwork/rbc/mqdata2el/mq_$DATE.csv"

                           

                           

                           

                           

                           

                           

                          blcli_disconnect

                           

                           

                           

                           

                           

                           

                          and here is the actual script:

                           

                           

                          #!/bin/nsh

                           

                           

                          blcli_connect

                           

                           

                          blcli_execute Utility assumeRole BLAdmins

                           

                           

                          echo ""

                          uname -n

                          ulimit -n

                          ulimit -n 16384

                          ulimit -n

                           

                           

                          DATE=`date +%Y-%m-%d-%H-%M`

                          GRP="/Administration/All_Active_UNIX"

                          #GRP="/Administration/Steve/znd_servers"

                          OUTFILE="/opt/bmc/fileserver/tmp/mq.tmp"

                          ELASTIK="//ulgwjsp01.fg.rbc.com/usr/local/groundwork/rbc/mqdata2el/mq_$DATE.csv"

                           

                           

                          if [ -f "$OUTFILE" ]; then

                          rm -f $OUTFILE

                          fi

                           

                           

                          touch $OUTFILE

                           

                           

                           

                           

                          SERVER_LIST=`blcli Server listServersInGroup "${GRP}"`

                           

                           

                          for serverName in $SERVER_LIST

                           

                           

                          do

                          FILECHECK="//$serverName/var/tmp/mqdata.csv"

                          if [ -f "$FILECHECK" ]; then

                          OUT=`cat $FILECHECK`

                          blcli_execute Server setPropertyValueByName $serverName RBC_MQ_SERVER true >> /dev/null

                          VERSION=`echo $OUT | cut -d, -f3`

                          if [ -n "$VERSION" ]; then

                          blcli_execute Server setPropertyValueByName $serverName RBC_MQ_Version ${VERSION} >> /dev/null

                          fi

                          echo "$OUT" >> $OUTFILE

                          rm -f $FILECHECK

                          else

                          echo ""

                          fi

                           

                           

                          done

                           

                           

                          cp $OUTFILE $ELASTIK

                           

                           

                          blcli_disconnect

                          • 10. Re: BSA 8.6.01  failing with pipe failed: too many open files.
                            Bill Robinson

                            if you look on the memory monitor line of the appserver log it will show how many files are in use  when it logs.  what do you see on that line just before this job starts ?

                             

                            have you restarted the appserver service w/ the init script since you make the change in the limits.conf ?

                            • 11. Re: BSA 8.6.01  failing with pipe failed: too many open files.
                              Neal Meagher

                              It restarted services after all changes are made

                              • 12. Re: BSA 8.6.01  failing with pipe failed: too many open files.
                                Bill Robinson

                                do you have an appserver log that covers the time of the job execution and error ?  the job run log snipped you pasted above is from 24-OCT.  the appserver log you attached starts on 25-OCT.

                                 

                                also - are you getting this error only on this one script?  and this is a type 1 or type 2 nsh ?

                                • 13. Re: BSA 8.6.01  failing with pipe failed: too many open files.
                                  Neal Meagher

                                  The logs I just posted are from 25th?

                                   

                                  Error,10/25/2016 00:37:18 it errors out   Type 2

                                  • 14. Re: BSA 8.6.01  failing with pipe failed: too many open files.
                                    Bill Robinson

                                    Sorry – was looking further up in the log.  so the open files count never goes above 600.

                                     

                                    And this is the appserver log from ulvblgp09.fg.rbc.com ?  because it looks like it’s the appserver log from     [] Hostname = ulvblgp12.fg.rbc.com

                                    1 2 Previous Next