0 Replies Latest reply on Apr 4, 2014 1:21 PM by Jason Culver

    [Appserver] Shutting down the authentication worker threads

    Jason Culver

      Something odd is happening... after i ran 'blcli Delete cleanupAllAppServerCaches’ to do some housekeepking cleanup our blappservice is shutting down daily at night, which users can't login to perform jobs till blappserv and blprocserv is restarted on the app server.  Any idea what to check to what might be causing the blappservice from sudden shutdown?  I happen every night right before midnight since we ran the cleanupAllAppServerCaches for some reason...

       

      [culverj@ohiomgr6 br]$ cat appserver.log | grep down

      [04 Apr 2014 03:40:00,212] [Thread-4] [INFO] [::] [Appserver] Shutting down the client worker threads

      [04 Apr 2014 03:40:00,212] [Thread-4] [INFO] [::] [Appserver] Shutting down the processed nsh proxy

      [04 Apr 2014 03:40:00,212] [Thread-4] [INFO] [::] [Appserver] Shutting down the client worker threads

      [04 Apr 2014 03:40:00,213] [Client-Worker-Thread-9] [INFO] [::] [Client] Client worker thread is shutting down.

      [04 Apr 2014 03:40:00,214] [Client-Worker-Thread-4] [INFO] [::] [Client] Client worker thread is shutting down.

      [04 Apr 2014 03:40:00,215] [Client-Worker-Thread-0] [INFO] [::] [Client] Client worker thread is shutting down.

      [04 Apr 2014 03:40:00,215] [Thread-4] [INFO] [::] [Appserver] Shutting down the acceptor

      [04 Apr 2014 03:40:00,215] [Client-Worker-Thread-6] [INFO] [::] [Client] Client worker thread is shutting down.

      [04 Apr 2014 03:40:00,215] [Client-Worker-Thread-2] [INFO] [::] [Client] Client worker thread is shutting down.

      [04 Apr 2014 03:40:00,216] [Client-Worker-Thread-3] [INFO] [::] [Client] Client worker thread is shutting down.

      [04 Apr 2014 03:40:00,215] [Client-Worker-Thread-1] [INFO] [::] [Client] Client worker thread is shutting down.

      [04 Apr 2014 03:40:00,217] [Client-Worker-Thread-5] [INFO] [::] [Client] Client worker thread is shutting down.

      [04 Apr 2014 03:40:00,216] [Thread-4] [INFO] [::] [Appserver] Shutting down the selector

      [04 Apr 2014 03:40:00,218] [Client-Worker-Thread-8] [INFO] [::] [Client] Client worker thread is shutting down.

      [04 Apr 2014 03:40:00,219] [Client-Worker-Thread-7] [INFO] [::] [Client] Client worker thread is shutting down.

      [04 Apr 2014 03:40:00,219] [Thread-4] [INFO] [::] [Appserver] Shutting down the NSH proxy threads

      [04 Apr 2014 03:40:00,219] [Thread-4] [INFO] [::] [Appserver] Shutting down the authentication worker threads

      [04 Apr 2014 03:40:00,220] [AuthSvc-Thread-1] [INFO] [::] [Appserver] Authentication Service thread shutting down

      [04 Apr 2014 03:40:00,220] [AuthSvc-Thread-2] [INFO] [::] [Appserver] Authentication Service thread shutting down

      [04 Apr 2014 03:40:00,220] [AuthSvc-Thread-3] [INFO] [::] [Appserver] Authentication Service thread shutting down

      [04 Apr 2014 03:40:00,220] [AuthSvc-Thread-0] [INFO] [::] [Appserver] Authentication Service thread shutting down

      [04 Apr 2014 03:40:00,220] [AuthSvc-Thread-4] [INFO] [::] [Appserver] Authentication Service thread shutting down

      [04 Apr 2014 03:40:00,252] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-0

      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-1

      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-2

      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-3

      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-4

      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-5

      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-6

      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-7

      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-8

      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-9

      [04 Apr 2014 03:40:00,255] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-10

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-11

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-12

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-13

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-14

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-15

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-16

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-17

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-18

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-19

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-20

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-21

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-22

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-23

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-24

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-25

      [04 Apr 2014 03:40:00,256] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-26

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-27

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-28

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-29

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-30

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-31

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-32

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-33

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-34

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-35

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-36

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-37

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-38

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-39

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-40

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-41

      [04 Apr 2014 03:40:00,257] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-42

      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-43

      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-44

      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-45

      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-46

      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-47

      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-48

      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work-item thread : Job-WorkItem-Thread-49

      [04 Apr 2014 03:40:00,258] [Thread-4] [INFO] [::] [Appserver] Shutting down work item timeout scheduler

      [04 Apr 2014 03:40:00,736] [Job-WorkItem-Thread-29] [INFO] [salehm:BLAdmins:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:00,856] [Job-WorkItem-Thread-3] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:01,513] [Job-WorkItem-Thread-9] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:01,541] [Select-Thread] [INFO] [::] [Appserver] Select thread shutdown complete

      [04 Apr 2014 03:40:01,862] [Job-WorkItem-Thread-19] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:02,476] [Job-WorkItem-Thread-32] [INFO] [salehm:BLAdmins:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:02,511] [Job-WorkItem-Thread-43] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:02,794] [Job-WorkItem-Thread-41] [INFO] [salehm:BLAdmins:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:03,086] [Job-WorkItem-Thread-16] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:03,110] [Job-WorkItem-Thread-45] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:03,814] [Job-WorkItem-Thread-6] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:04,957] [Job-WorkItem-Thread-20] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:06,216] [Job-WorkItem-Thread-15] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:10,296] [Job-WorkItem-Thread-25] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:10,466] [Job-WorkItem-Thread-14] [INFO] [salehm:BLAdmins:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:12,493] [Job-WorkItem-Thread-47] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:14,967] [Job-WorkItem-Thread-7] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:41,925] [Job-WorkItem-Thread-2] [INFO] [salehm:BLAdmins:] [] Work item thread is shutting down.

      [04 Apr 2014 03:40:58,900] [Job-WorkItem-Thread-5] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 04:05:21,962] [Job-WorkItem-Thread-44] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 04:46:11,277] [Job-WorkItem-Thread-11] [INFO] [siddiqum:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 04:52:25,273] [Job-WorkItem-Thread-36] [INFO] [siddiqum:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 06:03:05,111] [Job-WorkItem-Thread-23] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 06:08:20,766] [Job-WorkItem-Thread-18] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 06:22:41,200] [Job-WorkItem-Thread-33] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 09:30:31,752] [Job-WorkItem-Thread-34] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.

      [04 Apr 2014 11:43:50,395] [Job-WorkItem-Thread-37] [INFO] [sesettyr:Application:] [] Work item thread is shutting down.