7 Replies Latest reply on Apr 4, 2014 11:32 AM by Jason Culver

    Unable to create NSH master script folder

    Jason Culver

      Does anyone know if files can be deleted in /usr/nsh/tmp/scripts safely?  I am running Bladelogic 7.4.3 and users are reporting that they can't run jobs.  I found that we reach the limit of 32000 max files in a directory for RHEL3,4,5.  We are running RHEL4.8

       

      We are at:

      [root@ohiomgr6 tmp]# ls -l scripts | wc -l

      31999

       

      [root@ohiomgr6 scripts]# mkdir test

      mkdir: cannot create directory `test': Too many links

       

       

      BladeLogic Error message:[31 Mar 2014 16:13:14,642] [Job-WorkItem-Thread-16] [ERROR] [culverj:Application:] [NSHScript] Unable to create NSH master script folder: /usr/nsh/tmp/scripts/job_39508

      Just was looking for some guidance for what the recommended solution should be?

       


      Also, does anyone know what the recommended open files limit is for bladelogic 7.4.3 too?

       

       

       

       

       

       

      [30 Mar 2014 23:49:12,847] [Job-WorkItem-Thread-13] [ERROR] [sesettyr:Application:] [NSHScript] An error occurred while attempting to create a temporary file on the appserver: Too many open files

       

       

       

      We are at:

      [root@ohiomgr6 scripts]# su - "bladmin" -c 'ulimit -aHS'|grep open

      open files                      (-n) 16384

       

      Thanks!

        • 1. Re: Unable to create NSH master script folder
          Bill Robinson

          You can delete out of there anything that’s more than a couple days old.

          • 2. Re: Unable to create NSH master script folder
            Jason Culver

            Thanks Bill, do you know if bladelogic creates a job_#### file into the scripts directory for every bl job that is executed?  Just was trying to understand the reason for need behind the directory.  Also, did you happend to know what the recommended open files were for bl 7.4.3 too?

            [30 Mar 2014 23:49:12,847] [Job-WorkItem-Thread-13] [ERROR] [sesettyr:Application:] [NSHScript] An error occurred while attempting to create a temporary file on the appserver: Too many open

             

            We are at:

            [root@ohiomgr6 scripts]# su - "bladmin" -c 'ulimit -aHS'|grep open

            open files                      (-n) 16384

             

            Thanks!

            • 3. Re: Unable to create NSH master script folder
              Bill Robinson

              Yep, it does.  the ‘blcli Delete cleanupAllAppServerCaches’ command should also clean out these folders but I don’t recall if that’s available in 7.4

              • 4. Re: Unable to create NSH master script folder
                Jason Culver

                Thanks Bill!  That was the fix, users are now able to run their jobs again.

                • 5. Re: Unable to create NSH master script folder
                  Sean Berry

                  What can we do to help you upgrade from 7.4.3? 

                  • 6. Re: Unable to create NSH master script folder
                    Jason Culver

                    Hi Sean,

                     

                    We are targeting July to upgrade 2 bladelogic environments  running bl7.4.3 and running bl 8.1.0 and then hopefully merge both to one single file server, app console, and dbase server.  I like to upgrade our 7.4.3 instance to 7.6 ASAP.  Do you happen to have any upgrade documents for going from 7.4.3 to 7.6 by any chance?  I believe you have to be @ 7.6 before you can go to v8 right?

                     

                    Thanks,

                     

                    Jason

                    • 7. Re: Re: Unable to create NSH master script folder
                      Jason Culver

                      Hi guys, something odd is happening... after i ran 'blcli Delete cleanupAllAppServerCaches’ our blappservice is shutting down daily at night, which users can't login to perform jobs till blappserv and blprocserv is restarted on the app server.  Any idea what to check to what might be causing the blappservice from sudden shutdown?  I happen every night right before midnight since we ran the cleanupAllAppServerCaches for some reason...

                       

                      [culverj@ohiomgr6 br]$ cat appserver.log | grep down

                      [04 Apr 2014 03:40:00,212] [Thread-4] [INFO] [::] [Appserver] Shutting down the client worker threads

                      [04 Apr 2014 03:40:00,212] [Thread-4] [INFO] [::] [Appserver] Shutting down the processed nsh proxy

                      [04 Apr 2014 03:40:00,212] [Thread-4] [INFO] [::] [Appserver] Shutting down the client worker threads

                      [04 Apr 2014 03:40:00,213] [Client-Worker-Thread-9] [INFO] [::] [Client] Client worker thread is shutting down.

                      [04 Apr 2014 03:40:00,214] [Client-Worker-Thread-4] [INFO] [::] [Client] Client worker thread is shutting down.

                      [04 Apr 2014 03:40:00,215] [Client-Worker-Thread-0] [INFO] [::] [Client] Client worker thread is shutting down.

                      [04 Apr 2014 03:40:00,215] [Thread-4] [INFO] [::] [Appserver] Shutting down the acceptor

                      [04 Apr 2014 03:40:00,215] [Client-Worker-Thread-6] [INFO] [::] [Client] Client worker thread is shutting down.

                      [04 Apr 2014 03:40:00,215] [Client-Worker-Thread-2] [INFO] [::] [Client] Client worker thread is shutting down.

                      [04 Apr 2014 03:40:00,216] [Client-Worker-Thread-3] [INFO] [::] [Client] Client worker thread is shutting down.

                      [04 Apr 2014 03:40:00,215] [Client-Worker-Thread-1] [INFO] [::] [Client] Client worker thread is shutting down.

                      [04 Apr 2014 03:40:00,217] [Client-Worker-Thread-5] [INFO] [::] [Client] Client worker thread is shutting down.

                      [04 Apr 2014 03:40:00,216] [Thread-4] [INFO] [::] [Appserver] Shutting down the selector

                      [04 Apr 2014 03:40:00,218] [Client-Worker-Thread-8] [INFO] [::] [Client] Client worker thread is shutting down.

                      [04 Apr 2014 03:40:00,219] [Client-Worker-Thread-7] [INFO] [::] [Client] Client worker thread is shutting down.

                      [04 Apr 2014 03:40:00,219] [Thread-4] [INFO] [::] [Appserver] Shutting down the NSH proxy threads

                      [04 Apr 2014 03:40:00,219] [Thread-4] [INFO] [::] [Appserver] Shutting down the authentication worker threads

                      [04 Apr 2014 03:40:00,220] [AuthSvc-Thread-1] [INFO] [::] [Appserver] Authentication Service thread shutting down

                      [04 Apr 2014 03:40:00,220] [AuthSvc-Thread-2] [INFO] [::] [Appserver] Authentication Service thread shutting down

                      [04 Apr 2014 03:40:00,220] [AuthSvc-Thread-3] [INFO] [::] [Appserver] Authentication Service thread shutting down

                      [04 Apr 2014 03:40:00,220] [AuthSvc-Thread-0] [INFO] [::] [Appserver] Authentication Service thread shutting down

                      [04 Apr 2014 03:40:00,220] [AuthSvc-Thread-4] [INFO] [::] [Appserver] Authentication Service thread shutting down

                      [04 Apr 2014 03:40:00,252] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-0

                      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-1

                      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-2

                      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-3

                      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-4

                      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-5

                      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-6

                      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-7

                      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-8

                      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-9

                      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-10

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-11

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-12

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-13

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-14

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-15

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-16

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-17

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-18

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-19

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-20

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-21

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-22

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-23

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-24

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-25

                      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-26

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-27

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-28

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-29

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-30

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-31

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-32

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-33

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-34

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-35

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-36

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-37

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-38

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-39

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-40

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-41

                      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-42

                      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-43

                      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-44

                      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-45

                      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-46

                      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-47

                      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-48

                      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-49

                      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work item timeout scheduler

                      [04 Apr 2014 03:40:00,736] [Job-WorkItem-Thread-29] [INFO] [salehm:BLAdmins:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:00,856] [Job-WorkItem-Thread-3] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:01,513] [Job-WorkItem-Thread-9] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:01,541] [Select-Thread] [INFO] [::] [Appserver] Select thread shutdown complete

                      [04 Apr 2014 03:40:01,862] [Job-WorkItem-Thread-19] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:02,476] [Job-WorkItem-Thread-32] [INFO] [salehm:BLAdmins:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:02,511] [Job-WorkItem-Thread-43] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:02,794] [Job-WorkItem-Thread-41] [INFO] [salehm:BLAdmins:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:03,086] [Job-WorkItem-Thread-16] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:03,110] [Job-WorkItem-Thread-45] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:03,814] [Job-WorkItem-Thread-6] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:04,957] [Job-WorkItem-Thread-20] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:06,216] [Job-WorkItem-Thread-15] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:10,296] [Job-WorkItem-Thread-25] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:10,466] [Job-WorkItem-Thread-14] [INFO] [salehm:BLAdmins:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:12,493] [Job-WorkItem-Thread-47] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:14,967] [Job-WorkItem-Thread-7] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:41,925] [Job-WorkItem-Thread-2] [INFO] [salehm:BLAdmins:] [] Work item thread is shutting down.

                      [04 Apr 2014 03:40:58,900] [Job-WorkItem-Thread-5] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 04:05:21,962] [Job-WorkItem-Thread-44] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 04:46:11,277] [Job-WorkItem-Thread-11] [INFO] [siddiqum:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 04:52:25,273] [Job-WorkItem-Thread-36] [INFO] [siddiqum:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 06:03:05,111] [Job-WorkItem-Thread-23] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 06:08:20,766] [Job-WorkItem-Thread-18] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 06:22:41,200] [Job-WorkItem-Thread-33] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 09:30:31,752] [Job-WorkItem-Thread-34] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

                      [04 Apr 2014 11:43:50,395] [Job-WorkItem-Thread-37] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.