1 2 Previous Next 26 Replies Latest reply on May 15, 2014 8:59 AM by Mike Reider

    nsh: fork failed: resource temporarily unavailable

    Team Fujitsu

      Hi All,

       

      Sometimes I get a following error in NSH script job.

      ------------------------

      nsh:13: fork failed: resource temporarily unavailable

      3 [main] nsh 4728 child_info_fork::abort: C:\Program Files\BMC

      Software\BladeLogic\NSH\bin\cygblcrypto-0.9.8.dll: Loaded to different

      address: parent(0x570000) != child(0x910000)

      ------------------------

      Why does it happen?

      I doubted that other cygblcrypto-0.9.8.dll exists but it doesn't exist.

       

      Environment:

      - BSA: 8.2SP4

      - OS: Windows2008R2

        • 1. Re: nsh: fork failed: resource temporarily unavailable
          Bill Robinson

          What OS is your appserver ?

          • 2. Re: nsh: fork failed: resource temporarily unavailable
            Team Fujitsu

            Hi Bill,

            It is Windows2008R2.

             

            • 3. Re: nsh: fork failed: resource temporarily unavailable
              Bill Robinson

              i think this can be fixed by updating the cygwin1.dll in the NSH/bin directory to a more recent version from cygwin.

              • 4. Re: nsh: fork failed: resource temporarily unavailable
                Team Fujitsu

                Thank you for your information.

                If you know, please tell me fixed version of cygwin1.dll and BSA.

                Current version of cygwin1.dll is 1.7.17.

                • 5. Re: nsh: fork failed: resource temporarily unavailable
                  Bill Robinson

                  I know we made a change in 8.2.04 for this but I think I ran into this in 8.3.something and I just grabbed the latest cygwin1.dll from their site and that seemed to address it.  I’m not sure if there is a defect open though.

                  • 6. Re: nsh: fork failed: resource temporarily unavailable
                    Mike Jones

                    This may not apply to your issue, but we had some odd fork errors in our UAT environment with a similar error to yours above.

                     

                    We use process spawner, and at some point we had migrated to a different DB, on investigation I found that the _spawner instance was pointing to the old DB rather than the new one - when I  changed this over and restarted the app servers the fork error went away

                    • 7. Re: nsh: fork failed: resource temporarily unavailable
                      Team Fujitsu

                      I confirmed that relationship between BSA version and cygwin1.dll version.

                      Following is the result.

                       

                      BSA Version, cygwin1.dll Product Version, cygwin1.dll File Version
                      8.2.03, 1.7.10, 1007.10.0.0
                      8.2.04, 1.7.17, 1007.17.0.0
                      8.3.0,  1.7.17 1007.17.0.0
                      8.3.03  1.7.17 1007.17.0.0
                      8.5.0  1.7.17 1007.17.0.0

                       

                      And this issue occured in BSA8.2.04.

                      Is this other issue ?

                      • 8. Re: nsh: fork failed: resource temporarily unavailable
                        Team Fujitsu

                        Thank you for your information.

                        I will confirm the _spawner.

                        • 9. Re: nsh: fork failed: resource temporarily unavailable
                          Mike Jones

                          FYI, when we got this error, we were also already on 8.2.04 and the cygwin1.dll is 1007.17.0.0, the old DB that _spawner was using was pre SP 4


                          Bill Robinson did suggested upgrading cygwin1.dll to version 1007.25.0.0 but we didn't have to look at this when the problem went away after changing the _spawner DB.


                          If _spawner is fine for your environment, perhaps you can try the latest version of cygwin1.dll





                          • 10. Re: nsh: fork failed: resource temporarily unavailable
                            Team Fujitsu

                            Hi Mike,

                            I confirmed that _spawner is fine.

                            A remaining possibility  is the cygwin1.dll. Is it right?

                            • 11. Re: nsh: fork failed: resource temporarily unavailable
                              Mike Jones

                              Yes I would try the latest version of cygwin1.dll to see how that goes

                               

                              • 12. Re: nsh: fork failed: resource temporarily unavailable
                                Mike Reider

                                Im noticing similar issue running NSH job on 50-60 AIX targets, the job fails with Out of Memory errors, and it leaves zombie procs running on server (the customer gets a monitoring alert). These procs remain even if you cancel the NSH job manually, the only way to remove them is to restart appserver.

                                 

                                2 app servers (VMs) - RHEL63x64, BSA 85 00 416, Oracle 11g

                                 

                                 

                                reiderm  4433  4431  0 10:37 ?        00:00:00 sshd: reiderm@pts/0

                                reiderm  4434  4433  0 10:37 pts/0    00:00:00 -bash

                                root      4465  4434  0 10:37 pts/0    00:00:00 su -

                                root      4469  4465  0 10:37 pts/0    00:00:00 -bash

                                patrol    4597  3251  0 10:40 ?        00:00:00 /bin/bash

                                bladmin  5001    1  0 11:03 ?        00:00:00 -sh -c (cd /opt/bmc/bladelogic/NSH/br; /opt/bmc/bladelogic/NSH/br/blappserv > /dev/null 2>&1)&

                                bladmin  5002  5001  0 11:03 ?        00:00:00 /bin/sh /opt/bmc/bladelogic/NSH/br/blappserv

                                bladmin  5003  5002  2 11:03 ?        00:00:13 /opt/bmc/bladelogic/NSH/br/java/bin/java -Xss1m -Xmx1024M -Djava.security.egd=file:/dev/../dev/urandom -Dj

                                bladmin  5028  5003 16 11:03 ?        00:01:19 /opt/bmc/bladelogic/NSH/br/java/bin/java -Djava.library.path=/opt/bmc/bladelogic/NSH/lib:/usr/java/package

                                bladmin  5041  5003 15 11:03 ?        00:01:17 /opt/bmc/bladelogic/NSH/br/java/bin/java -Djava.library.path=/opt/bmc/bladelogic/NSH/lib:/usr/java/package

                                bladmin  5061  5003 16 11:03 ?        00:01:20 /opt/bmc/bladelogic/NSH/br/java/bin/java -Djava.library.path=/opt/bmc/bladelogic/NSH/lib:/usr/java/package

                                bladmin  5077  5003 14 11:03 ?        00:01:09 /opt/bmc/bladelogic/NSH/br/java/bin/java -Djava.library.path=/opt/bmc/bladelogic/NSH/lib:/usr/java/package

                                bladmin  6155  5028  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6236  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6237  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6239  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6240  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6241  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6243  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6245  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6247  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6249  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6250  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6252  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6253  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6255  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6256  5061  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6259  5041  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6260  5041  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6261  5041  0 11:07 ?        00:00:00 [nsh] <defunct>

                                bladmin  6262  5041  0 11:07 ?        00:00:00 [nsh] <defunct>

                                root      6427  4469  0 11:11 pts/0    00:00:00 ps -ef

                                 

                                Getting this from appserver log (I had the admins bump up the virtual mem to 16gigs just to make sure we have enough)

                                 

                                 

                                [ERROR] [reiderm:BLAdmins:] [NSHScript] Caught exception running command - /opt/bmc/bladelogic/NSH/bin/nsh --norc -c /opt/bmc/bladelogic/NSH/tmp/application_server/scripts/job__f73d83d7-988c-454f-b5c8-c23345f19eb3/master_afc7e3bd-09ef-4df7-a0ff-9d73cc418a83

                                Error: fork: Resource temporarily unavailable

                                com.bladelogic.om.infra.app.process.BlProcessException: fork: Resource temporarily unavailable

                                        at com.bladelogic.om.infra.app.process.BlProcess.exec(Native Method)

                                        at com.bladelogic.om.infra.app.process.BlProcess.exec(BlProcess.java:95)

                                        at com.bladelogic.om.infra.app.process.LocalProcessHandlerImpl.exec(LocalProcessHandlerImpl.java:123)

                                        at com.bladelogic.om.infra.app.nsh.NSHExec.exec(NSHExec.java:263)

                                        at com.bladelogic.om.infra.app.nsh.NSHExec.exec(NSHExec.java:165)

                                        at com.bladelogic.om.infra.model.job.script.ScriptBasedJobWorkItemImpl.executeScript(ScriptBasedJobWorkItemImpl.java:236)

                                        at com.bladelogic.om.infra.model.job.script.NSHScriptJobWorkItemImpl.executeScript(NSHScriptJobWorkItemImpl.java:724)

                                        at com.bladelogic.om.infra.model.job.script.NSHScriptJobWorkItemImpl.execute(NSHScriptJobWorkItemImpl.java:203)

                                        at com.bladelogic.om.infra.app.service.workitem.WorkItem.doExecute(WorkItem.java:114)

                                        at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:176)

                                        at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:51)

                                        at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                                [08 May 2014 11:07:41,201] [WorkItem-Thread-2] [ERROR] [reiderm:BLAdmins:] [NSHScript] Caught exception running command - /opt/bmc/bladelogic/NSH/bin/nsh --norc -c /opt/bmc/bladelogic/NSH/tmp/application_server/scripts/job__cecc4cfe-9c65-4653-867b-c35ca3465c6a/master_50dad863-c603-4efd-b979-6d0b340d1ac3

                                Error: fork: Resource temporarily unavailable

                                com.bladelogic.om.infra.app.process.BlProcessException: fork: Resource temporarily unavailable

                                        at com.bladelogic.om.infra.app.process.BlProcess.exec(Native Method)

                                        at com.bladelogic.om.infra.app.process.BlProcess.exec(BlProcess.java:95)

                                        at com.bladelogic.om.infra.app.process.LocalProcessHandlerImpl.exec(LocalProcessHandlerImpl.java:123)

                                        at com.bladelogic.om.infra.app.nsh.NSHExec.exec(NSHExec.java:263)

                                        at com.bladelogic.om.infra.app.nsh.NSHExec.exec(NSHExec.java:165)

                                        at com.bladelogic.om.infra.model.job.script.ScriptBasedJobWorkItemImpl.executeScript(ScriptBasedJobWorkItemImpl.java:236)

                                        at com.bladelogic.om.infra.model.job.script.NSHScriptJobWorkItemImpl.executeScript(NSHScriptJobWorkItemImpl.java:724)

                                        at com.bladelogic.om.infra.model.job.script.NSHScriptJobWorkItemImpl.execute(NSHScriptJobWorkItemImpl.java:203)

                                        at com.bladelogic.om.infra.app.service.workitem.WorkItem.doExecute(WorkItem.java:114)

                                        at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:176)

                                        at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:51)

                                        at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                                [08 May 2014 11:07:41,199] [WorkItem-Thread-9] [INFO] [reiderm:BLAdmins:] [NSHScript] Killing NSH process with pid -1

                                [08 May 2014 11:07:41,207] [WorkItem-Thread-9] [INFO] [reiderm:BLAdmins:] [NSHScript] Process has been killed: 0

                                [08 May 2014 11:07:41,198] [WorkItem-Thread-5] [INFO] [reiderm:BLAdmins:] [NSHScript] Process has been killed: 0

                                [08 May 2014 11:07:41,207] [WorkItem-Thread-5] [ERROR] [reiderm:BLAdmins:] [NSHScript] Caught exception running command - /opt/bmc/bladelogic/NSH/bin/nsh --norc -c /opt/bmc/bladelogic/NSH/tmp/application_server/scripts/job__b867dc8e-2097-4b84-9007-3e5f9aef2051/master_ee5746db-7838-4b9d-8698-6bf8cb03f150

                                Error: fork: Resource temporarily unavailable

                                com.bladelogic.om.infra.app.process.BlProcessException: fork: Resource temporarily unavailable

                                        at com.bladelogic.om.infra.app.process.BlProcess.exec(Native Method)

                                        at com.bladelogic.om.infra.app.process.BlProcess.exec(BlProcess.java:95)

                                        at com.bladelogic.om.infra.app.process.LocalProcessHandlerImpl.exec(LocalProcessHandlerImpl.java:123)

                                        at com.bladelogic.om.infra.app.nsh.NSHExec.exec(NSHExec.java:263)

                                        at com.bladelogic.om.infra.app.nsh.NSHExec.exec(NSHExec.java:165)

                                        at com.bladelogic.om.infra.model.job.script.ScriptBasedJobWorkItemImpl.executeScript(ScriptBasedJobWorkItemImpl.java:236)

                                        at com.bladelogic.om.infra.model.job.script.NSHScriptJobWorkItemImpl.executeScript(NSHScriptJobWorkItemImpl.java:724)

                                        at com.bladelogic.om.infra.model.job.script.NSHScriptJobWorkItemImpl.execute(NSHScriptJobWorkItemImpl.java:203)

                                        at com.bladelogic.om.infra.app.service.workitem.WorkItem.doExecute(WorkItem.java:114)

                                        at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:176)

                                        at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:51)

                                        at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                                [08 May 2014 11:07:41,197] [WorkItem-Thread-14] [INFO] [reiderm:BLAdmins:] [NSHScript] Attempting to kill process: 0

                                [08 May 2014 11:07:41,208] [WorkItem-Thread-14] [INFO] [reiderm:BLAdmins:] [NSHScript] Killing NSH process with pid -1

                                [08 May 2014 11:07:41,208] [WorkItem-Thread-14] [INFO] [reiderm:BLAdmins:] [NSHScript] Process has been killed: 0

                                [08 May 2014 11:07:41,208] [WorkItem-Thread-14] [ERROR] [reiderm:BLAdmins:] [NSHScript] Caught exception running command - /opt/bmc/bladelogic/NSH/bin/nsh --norc -c /opt/bmc/bladelogic/NSH/tmp/application_server/scripts/job__33932ea2-bba4-4f80-9bcc-70790683f588/master_6e408ca3-c839-4ebd-8777-2076d86868a2

                                Error: fork: Resource temporarily unavailable

                                com.bladelogic.om.infra.app.process.BlProcessException: fork: Resource temporarily unavailable

                                        at com.bladelogic.om.infra.app.process.BlProcess.exec(Native Method)

                                        at com.bladelogic.om.infra.app.process.BlProcess.exec(BlProcess.java:95)

                                        at com.bladelogic.om.infra.app.process.LocalProcessHandlerImpl.exec(LocalProcessHandlerImpl.java:123)

                                        at com.bladelogic.om.infra.app.nsh.NSHExec.exec(NSHExec.java:263)

                                        at com.bladelogic.om.infra.app.nsh.NSHExec.exec(NSHExec.java:165)

                                        at com.bladelogic.om.infra.model.job.script.ScriptBasedJobWorkItemImpl.executeScript(ScriptBasedJobWorkItemImpl.java:236)

                                        at com.bladelogic.om.infra.model.job.script.NSHScriptJobWorkItemImpl.executeScript(NSHScriptJobWorkItemImpl.java:724)

                                        at com.bladelogic.om.infra.model.job.script.NSHScriptJobWorkItemImpl.execute(NSHScriptJobWorkItemImpl.java:203)

                                        at com.bladelogic.om.infra.app.service.workitem.WorkItem.doExecute(WorkItem.java:114)

                                        at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:176)

                                        at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:51)

                                        at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                                [08 May 2014 11:07:41,196] [WorkItem-Thread-0] [INFO] [reiderm:BLAdmins:] [NSHScript] Attempting to kill process: 0

                                [08 May 2014 11:07:41,196] [WorkItem-Thread-7] [INFO] [reiderm:BLAdmins:] [NSHScript] Attempting to kill process: 0

                                [08 May 2014 11:07:41,209] [WorkItem-Thread-7] [INFO] [reiderm:BLAdmins:] [NSHScript] Killing NSH process with pid -1

                                [08 May 2014 11:07:41,209] [WorkItem-Thread-7] [INFO] [reiderm:BLAdmins:] [NSHScript] Process has been killed: 0

                                [08 May 2014 11:07:41,209] [WorkItem-Thread-7] [ERROR] [reiderm:BLAdmins:] [NSHScript] Caught exception running command - /opt/bmc/bladelogic/NSH/bin/nsh --norc -c /opt/bmc/bladelogic/NSH/tmp/application_server/scripts/job__ee85fea6-8d9f-41da-b33b-5464dcd2192c/master_d883c93a-b302-474c-bf0f-9ae780a2ceac

                                Error: fork: Resource temporarily unavailable

                                com.bladelogic.om.infra.app.process.BlProcessException: fork: Resource temporarily unavailable

                                        at com.bladelogic.om.infra.app.process.BlProcess.exec(Native Method)

                                        at com.bladelogic.om.infra.app.process.BlProcess.exec(BlProcess.java:95)

                                        at com.bladelogic.om.infra.app.process.LocalProcessHandlerImpl.exec(LocalProcessHandlerImpl.java:123)

                                        at com.bladelogic.om.infra.app.nsh.NSHExec.exec(NSHExec.java:263)

                                        at com.bladelogic.om.infra.app.nsh.NSHExec.exec(NSHExec.java:165)

                                        at com.bladelogic.om.infra.model.job.script.ScriptBasedJobWorkItemImpl.executeScript(ScriptBasedJobWorkItemImpl.java:236)

                                        at com.bladelogic.om.infra.model.job.script.NSHScriptJobWorkItemImpl.executeScript(NSHScriptJobWorkItemImpl.java:724)

                                        at com.bladelogic.om.infra.model.job.script.NSHScriptJobWorkItemImpl.execute(NSHScriptJobWorkItemImpl.java:203)

                                        at com.bladelogic.om.infra.app.service.workitem.WorkItem.doExecute(WorkItem.java:114)

                                        at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:176)

                                        at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:51)

                                        at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                                [08 May 2014 11:07:41,196] [WorkItem-Thread-12] [INFO] [reiderm:BLAdmins:] [NSHScript] Killing NSH process with pid -1

                                [08 May 2014 11:07:41,209] [WorkItem-Thread-12] [INFO] [reiderm:BLAdmins:] [NSHScript] Process has been killed: 0

                                [08 May 2014 11:07:41,196] [WorkItem-Thread-6] [INFO] [reiderm:BLAdmins:] [NSHScript] Attempting to kill process: 0

                                [08 May 2014 11:07:41,210] [WorkItem-Thread-6] [INFO] [reiderm:BLAdmins:] [NSHScript] Killing NSH process with pid -1

                                [08 May 2014 11:07:41,210] [WorkItem-Thread-12] [ERROR] [reiderm:BLAdmins:] [NSHScript] Caught exception running command - /opt/bmc/bladelogic/NSH/bin/nsh --norc -c /opt/bmc/bladelogic/NSH/tmp/application_server/scripts/job__22caba93-1ecb-48df-bafa-dd69c988c20e/master_bbdaeb15-d79a-44b7-aed5-fe22dfd894be

                                Error: fork: Resource temporarily unavailable

                                com.bladelogic.om.infra.app.process.BlProcessException: fork: Resource temporarily unavailable

                                        at com.bladelogic.om.infra.app.process.BlProcess.exec(Native Method)

                                        at com.bladelogic.om.infra.app.process.BlProcess.exec(BlProcess.java:95)

                                        at com.bladelogic.om.infra.app.process.LocalProcessHandlerImpl.exec(LocalProcessHandlerImpl.java:123)

                                        at com.bladelogic.om.infra.app.nsh.NSHExec.exec(NSHExec.java:263)

                                        at com.bladelogic.om.infra.app.nsh.NSHExec.exec(NSHExec.java:165)

                                        at com.bladelogic.om.infra.model.job.script.ScriptBasedJobWorkItemImpl.executeScript(ScriptBasedJobWorkItemImpl.java:236)

                                        at com.bladelogic.om.infra.model.job.script.NSHScriptJobWorkItemImpl.executeScript(NSHScriptJobWorkItemImpl.java:724)

                                        at com.bladelogic.om.infra.model.job.script.NSHScriptJobWorkItemImpl.execute(NSHScriptJobWorkItemImpl.java:203)

                                        at com.bladelogic.om.infra.app.service.workitem.WorkItem.doExecute(WorkItem.java:114)

                                        at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:176)

                                        at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:51)

                                        at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

                                [08 May 2014 11:07:41,209] [WorkItem-Thread-7] [ERROR] [reiderm:BLAdmins:] [NSHScript] Caught exception running command - /opt/bmc/bladelogic/NSH/bin/nsh --norc -c /opt/bmc/bladelogic/NSH/tmp/application_server/scripts/job__ee85fea6-8d9f-41da-b33b-5464dcd2192c/master_d883c93a-b302-474c-bf0f-9ae780a2ceac

                                Error: fork: Resource temporarily unavailable

                                • 13. Re: nsh: fork failed: resource temporarily unavailable
                                  Mike Reider

                                  INteresting, I had app server profiles enabled (Config, Job1, Job2). I disabled them, the job now runs and uses up to 9.5 gigs of memory usage (out of 16 total) but we arent seeing these errors anymore. I'll need to look into adjusting Maxheapsize of each job server profile.

                                  • 14. Re: nsh: fork failed: resource temporarily unavailable
                                    Bill Robinson

                                    what do you mean by 'appserver profiles enabled' and that you disabled them?

                                     

                                    usually the nsh fork errors are due to a low setting for the number of open files / threads in the os.

                                    1 2 Previous Next