1 2 3 Previous Next 31 Replies Latest reply on Jul 7, 2016 10:07 AM by Sudesh Sancheti

    Error: fork: Cannot allocate memory

    Sudesh Sancheti

      Hi All,

      We have 1 NSH script job which calls other 15+ NSH script jobs, 10+ Deploy jobs in parallel, from past few days we are getting below error in few of the child NSH scripts job executions.

       

      Error      31-May-2016 16:20:31    Caught exception running command - /opt/nsh/NSH/bin/nsh --norc -c /opt/nsh/NSH/tmp/serverA/scripts/job__54ccdb4b-fd80-4e2b-9464-cf9f2c400676/master_eeb79a16-e4b0-4452-8732-cbc4a43391f5

      Error: fork: Cannot allocate memory

       

      here are the details of enviornment.

       

      2 application server RHEL6

      2 job server 1 on each app server

      MaxHeapsize 12G for each app server and job server

      Max FileDescripter 8192

      Max work item thread 50

      Concurrent jobs 20

       

      Earlier I was getting java memory issue(shown below) when the maxheap size was 6GB for each app server and job server which I have increased to 12GB each.

       

      Error 21-May-2016 08:45:06 Error: A fatal exception has occurred. Program will exit.

      Info 21-May-2016 08:45:06 Exit Code 1

      Info 21-May-2016 08:45:06 Could not reserve enough space for object heap

      Info 21-May-2016 08:45:06 Error occurred during initialization of VM

        • 1. Re: Error: fork: Cannot allocate memory
          Sudesh Sancheti

          the only difference in working and issuing server is " JVMArgs, why this is set as -XX:+UseCompressedOops, our system is 64 bit the why this is set by default.""

           

          We have explictly set MaxHeapsize to 12 GB, can we make JVMArgs parameter as blank ??

          • 2. Re: Error: fork: Cannot allocate memory
            Bill Robinson

            so on the failing server -XX:+UserCompressesOops is set or not set ?  and all servers were throwing that error w/ a 6gb max heap ?

             

            what version of bsa is this ?

             

            can you attach (advanced editor) the appserver logs showing the fork errors ?

            • 3. Re: Error: fork: Cannot allocate memory
              Sudesh Sancheti

              Yes on the failing server this is set  -XX:+UserCompressesOops.  With 6GB max heap also only failing server was throwing error and that time the error was the below one.

               

              Error 21-May-2016 08:45:06 Error: A fatal exception has occurred. Program will exit.

              Info 21-May-2016 08:45:06 Exit Code 1

              Info 21-May-2016 08:45:06 Could not reserve enough space for object heap

              Info 21-May-2016 08:45:06 Error occurred during initialization of VM

               

              BSA version is 8.5.1.

               

              The fork error is not coming now after updating the below parameter's.

               

              We have turn process spwaner to TRUE on problem
              servers

              >We have changed MaxWorkItem Thread to 100 on both servers

              >We have changed MaxJobExecutionConnections to 200 on both servers

              >We have changed MaxClientConnections to 200 on both the servers

              >we have changed MaxGeneralConnections to 200 on both the servers

               

              but now the different error is coming when executing bulk NSH scripts.

               

              -2016 13:38:10      NSHDIR = /nsh/scripts/prd/NSH

              Error      03-Jun-2016 13:38:10      Error: Could not create the Java Virtual Machine.

              Info        03-Jun-2016 13:38:10      Error occurred during initialization of VM

              Info        03-Jun-2016 13:38:10      Could not reserve enough space for object heap

              Info        03-Jun-2016 13:38:10      Exit Code 1

              Error      03-Jun-2016 13:38:10      Error: A fatal exception has occurred. Program will exit.

               

              The current value of JVM Max is 12G in each job and app server.

              • 4. Re: Error: fork: Cannot allocate memory
                Bill Robinson

                what do you have set for open files (nofile) and number of user processes (nproc) for the bladmin os user in /etc/security/limits.conf or /etc/security/limits.d/* ?

                • 5. Re: Error: fork: Cannot allocate memory
                  Sudesh Sancheti

                  Thanks bill for checking this.

                  Bot the working and non working server the entries are as below.

                   

                  /etc/security/limits.d/bladmin.conf

                  bladmin  soft nofile 1024

                  bladmin  hard nofile 8192

                   

                  In /etc/security/limits.conf nothing set for bladmin.

                   

                  But when I am running ulimit -n it is showing 1024. In startup script blappserv and blprocserv it is set to 8192.

                  • 6. Re: Error: fork: Cannot allocate memory
                    Bill Robinson

                    So nothing for nproc ?

                     

                    Can you attach the appserver log (using the advanced editor) that includes the errors ?

                    • 7. Re: Error: fork: Cannot allocate memory
                      Sudesh Sancheti

                      Yes for bladmin don't see anything set for nproc in both the files.

                       

                      In appserv log don't see any error.

                      • 8. Re: Error: fork: Cannot allocate memory
                        Bill Robinson

                        Then where do you see the fork errors ?

                        • 9. Re: Error: fork: Cannot allocate memory
                          Sudesh Sancheti

                          As mentioned in my above reply that after changing the above parameters the fork error stopped but now getting the below error.

                          Info        03-Jun-2016 13:38:10      NSHDIR = /nsh/scripts/prd/NSH

                          Error      03-Jun-2016 13:38:10      Error: Could not create the Java Virtual Machine.

                          Info        03-Jun-2016 13:38:10      Error occurred during initialization of VM

                          Info        03-Jun-2016 13:38:10      Could not reserve enough space for object heap

                          Info        03-Jun-2016 13:38:10      Exit Code 1

                          Error      03-Jun-2016 13:38:10      Error: A fatal exception has occurred. Program will exit.

                           

                          When this error occurred I see in the logs free JVM is very less, but as I said we have the below configuration.

                           

                          Server A:  ServerA1(Applcaition Server)---12GB Max Heap

                                            Server A1-Job(Job server on same Server A--12Gb Max Heap

                          Server B:   ServerB1(Applcaition Server)--12GB Max Heap

                                            Server B1-Job(Job server on same Server B)--12 GB Max Heap

                           

                          The issue is occurring for server B.

                           

                          03 Jun 2016 13:36:57,998] [Scheduled-System-Tasks-Thread-2] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 462422016,Free JVM (B): 57178792,Used JVM (B): 405243224,VSize (B): 14461845504,RSS (B): 842035200,Used File Descriptors: 319,Used Work Item Threads: 1/100,Used NSH Proxy Threads: 0/15,Used Client Connections: 16/200,DB Client-Connection-Pool: 4/4/0/200,DB Job-Connection-Pool: 2/2/0/200,DB General-Connection-Pool: 2/2/0/200

                          [03 Jun 2016 13:37:57,998] [Scheduled-System-Tasks-Thread-7] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 474480640,Free JVM (B): 77394328,Used JVM (B): 397086312,VSize (B): 14461845504,RSS (B): 854691840,Used File Descriptors: 368,Used Work Item Threads: 2/100,Used NSH Proxy Threads: 0/15,Used Client Connections: 16/200,DB Client-Connection-Pool: 4/4/0/200,DB Job-Connection-Pool: 4/4/0/200,DB General-Connection-Pool: 2/2/0/200

                          • 10. Re: Error: fork: Cannot allocate memory
                            Bill Robinson

                            Where are you seeing that error ?

                            • 11. Re: Error: fork: Cannot allocate memory
                              Sudesh Sancheti

                              I see the JVM error in the NSH script log run.

                               

                              Info 03-Jun-2016 13:38:10 NSHDIR = /nsh/scripts/prd/NSH

                              Error 03-Jun-2016 13:38:10 Error: Could not create the Java Virtual Machine.

                              Info 03-Jun-2016 13:38:10 Error occurred during initialization of VM

                              Info 03-Jun-2016 13:38:10 Could not reserve enough space for object heap

                              Info 03-Jun-2016 13:38:10 Exit Code 1

                              Error 03-Jun-2016 13:38:10 Error: A fatal exception has occurred. Program will exit.

                              • 12. Re: Error: fork: Cannot allocate memory
                                Bill Robinson

                                So what is this script doing ?

                                 

                                If you look in the nsh job run log there’s a line like “running work item thread for job xxx on appserver yyy”.  Can you attach the appserver log from yyy covering the time of the job run, the job run log and the script ?

                                • 13. Re: Error: fork: Cannot allocate memory
                                  Sudesh Sancheti

                                  Hi Bill,

                                  I have attached the logs to CASE, sorry can't share the logs here.

                                   

                                  CASE 00156457

                                  • 14. Re: Error: fork: Cannot allocate memory
                                    Bill Robinson

                                    Can you also attach the entire job run log showing the error, the appserver logs from the other appservers and the script to the ticket ?

                                    1 2 3 Previous Next