0 Replies Latest reply on Feb 6, 2015 12:33 PM by Yanick Girouard

    What is the CPU/Memory impact of doubling the number of WIT on a job server?

    Yanick Girouard

      We currently have 6 job servers, running as VMs on Red Hat Enterprise Linux 5 x86_64, with 2 x vCPU and 8Gb of RAM each. The max JVM heap is set to 4Gb on each.

       

      Each app server is set to use a maximum of 50 work item threads only. It seemed to be enough at first, but we are now noticing issues when we run multiple Windows Patch Catalogs at once, where each catalog will use 100% of the 50 threads during the processing of the depot object batches (batch size is set to 300 in Patch Global Config).

       

      As a test we're trying to double the number of WIT on a given app server and forced one CUJ to run on it.

       

      My question is what kind of impact, between memory and CPU, should we expect, and does 100 WIT seem too big for our server specs?

       

      Update: Our application server is stuck again while running the Windows CUJ. Here's a full paste of the JobManagerService showing used WITs:

       

      Job Manager Status
      ===================
      
      Currently Running Jobs:
      ========================
      Progress,Type,Name,Job Key,Job Run Key,Run Time(ms),State
      --------------------------------------------------
      /e98/f0/t306,Windows Catalog Update Job,Windows_OS-IE-.NET_CUJ-2015-02-06 11-36-09-918-0500,DBKey:SJobKeyImpl:85552-60,DBKey:SJobRunKeyImpl:1204233,6716308,EXECUTING
      
      Work Item Thread status:
      =========================
      Name,State,Time in state (ms),Additional Info
      --------------------------------------------------
      WorkItem-Thread-0,STUCK,1h:8m:10s713ms,Deploy Staging Job:Red-Hat Dev_wave - CRQ000000184962-server-3566372 @ 2015-02-06 10-07-21-432-0500; Server:server; 
      WorkItem-Thread-1,EXECUTING,1h:16m:14s52ms,Depot Object Processor WorkItem (batch 11)
      WorkItem-Thread-2,EXECUTING,1h:16m:14s53ms,Depot Object Processor WorkItem (batch 73)
      WorkItem-Thread-3,EXECUTING,1h:16m:13s869ms,Depot Object Processor WorkItem (batch 42)
      WorkItem-Thread-4,EXECUTING,1h:16m:14s65ms,Depot Object Processor WorkItem (batch 74)
      WorkItem-Thread-5,EXECUTING,1h:16m:13s910ms,Depot Object Processor WorkItem (batch 67)
      WorkItem-Thread-6,EXECUTING,1h:16m:14s51ms,Depot Object Processor WorkItem (batch 84)
      WorkItem-Thread-7,EXECUTING,1h:16m:14s7ms,Depot Object Processor WorkItem (batch 66)
      WorkItem-Thread-8,STUCK,1h:8m:27s160ms,Deploy Staging Job:Red-Hat Dev_wave - CRQ000000184962-server-3566371 @ 2015-02-06 10-07-15-439-0500; Server:server; 
      WorkItem-Thread-9,EXECUTING,1h:16m:14s122ms,Depot Object Processor WorkItem (batch 72)
      WorkItem-Thread-10,EXECUTING,1h:16m:11s326ms,Depot Object Processor WorkItem (batch 105)
      WorkItem-Thread-11,EXECUTING,1h:16m:14s100ms,Depot Object Processor WorkItem (batch 51)
      WorkItem-Thread-12,EXECUTING,1h:16m:14s45ms,Depot Object Processor WorkItem (batch 21)
      WorkItem-Thread-13,EXECUTING,1h:51m:50s421ms,Windows Catalog Patch Metadata Update Work Item
      WorkItem-Thread-14,EXECUTING,1h:16m:14s186ms,Depot Object Processor WorkItem (batch 45)
      WorkItem-Thread-15,EXECUTING,1h:16m:14s201ms,Depot Object Processor WorkItem (batch 71)
      WorkItem-Thread-16,EXECUTING,1h:16m:13s875ms,Depot Object Processor WorkItem (batch 15)
      WorkItem-Thread-17,EXECUTING,1h:16m:17s363ms,Depot Object Processor WorkItem (batch 1)
      WorkItem-Thread-18,EXECUTING,1h:16m:13s914ms,Depot Object Processor WorkItem (batch 18)
      WorkItem-Thread-19,EXECUTING,1h:16m:14s205ms,Depot Object Processor WorkItem (batch 33)
      WorkItem-Thread-20,EXECUTING,1h:16m:14s205ms,Depot Object Processor WorkItem (batch 88)
      WorkItem-Thread-21,EXECUTING,1h:16m:14s204ms,Depot Object Processor WorkItem (batch 16)
      WorkItem-Thread-22,EXECUTING,1h:16m:14s203ms,Depot Object Processor WorkItem (batch 36)
      WorkItem-Thread-23,EXECUTING,1h:16m:14s202ms,Depot Object Processor WorkItem (batch 86)
      WorkItem-Thread-24,EXECUTING,1h:16m:14s103ms,Depot Object Processor WorkItem (batch 27)
      WorkItem-Thread-25,EXECUTING,1h:16m:14s178ms,Depot Object Processor WorkItem (batch 13)
      WorkItem-Thread-26,EXECUTING,1h:16m:14s199ms,Depot Object Processor WorkItem (batch 6)
      WorkItem-Thread-27,EXECUTING,1h:16m:11s330ms,Depot Object Processor WorkItem (batch 124)
      WorkItem-Thread-28,EXECUTING,1h:16m:14s191ms,Depot Object Processor WorkItem (batch 59)
      WorkItem-Thread-29,EXECUTING,1h:16m:14s196ms,Depot Object Processor WorkItem (batch 7)
      WorkItem-Thread-30,EXECUTING,1h:16m:13s881ms,Depot Object Processor WorkItem (batch 19)
      WorkItem-Thread-31,EXECUTING,1h:16m:13s919ms,Depot Object Processor WorkItem (batch 48)
      WorkItem-Thread-32,EXECUTING,1h:16m:14s176ms,Depot Object Processor WorkItem (batch 17)
      WorkItem-Thread-33,EXECUTING,1h:16m:14s175ms,Depot Object Processor WorkItem (batch 38)
      WorkItem-Thread-34,EXECUTING,1h:16m:14s173ms,Depot Object Processor WorkItem (batch 10)
      WorkItem-Thread-35,EXECUTING,1h:16m:13s872ms,Depot Object Processor WorkItem (batch 64)
      WorkItem-Thread-36,EXECUTING,1h:16m:14s171ms,Depot Object Processor WorkItem (batch 82)
      WorkItem-Thread-37,EXECUTING,1h:16m:14s170ms,Depot Object Processor WorkItem (batch 53)
      WorkItem-Thread-38,EXECUTING,1h:16m:14s55ms,Depot Object Processor WorkItem (batch 12)
      WorkItem-Thread-39,EXECUTING,1h:16m:14s166ms,Depot Object Processor WorkItem (batch 70)
      WorkItem-Thread-40,EXECUTING,1h:16m:13s877ms,Depot Object Processor WorkItem (batch 30)
      WorkItem-Thread-41,EXECUTING,1h:16m:13s923ms,Depot Object Processor WorkItem (batch 20)
      WorkItem-Thread-42,EXECUTING,1h:16m:13s916ms,Depot Object Processor WorkItem (batch 2)
      WorkItem-Thread-43,EXECUTING,1h:16m:14s13ms,Depot Object Processor WorkItem (batch 49)
      WorkItem-Thread-44,EXECUTING,1h:16m:13s986ms,Depot Object Processor WorkItem (batch 75)
      WorkItem-Thread-45,EXECUTING,1h:16m:13s917ms,Depot Object Processor WorkItem (batch 40)
      WorkItem-Thread-46,EXECUTING,1h:16m:13s906ms,Depot Object Processor WorkItem (batch 78)
      WorkItem-Thread-47,EXECUTING,1h:16m:14s1ms,Depot Object Processor WorkItem (batch 22)
      WorkItem-Thread-48,EXECUTING,1h:16m:13s877ms,Depot Object Processor WorkItem (batch 80)
      WorkItem-Thread-49,EXECUTING,1h:16m:14s55ms,Depot Object Processor WorkItem (batch 41)
      WorkItem-Thread-50,EXECUTING,1h:16m:13s872ms,Depot Object Processor WorkItem (batch 28)
      WorkItem-Thread-51,EXECUTING,1h:16m:13s884ms,Depot Object Processor WorkItem (batch 52)
      WorkItem-Thread-52,EXECUTING,1h:16m:14s2ms,Depot Object Processor WorkItem (batch 77)
      WorkItem-Thread-53,EXECUTING,1h:16m:11s325ms,Depot Object Processor WorkItem (batch 139)
      WorkItem-Thread-54,EXECUTING,1h:16m:14s156ms,Depot Object Processor WorkItem (batch 3)
      WorkItem-Thread-55,EXECUTING,1h:16m:14s3ms,Depot Object Processor WorkItem (batch 69)
      WorkItem-Thread-56,EXECUTING,1h:16m:14s152ms,Depot Object Processor WorkItem (batch 14)
      WorkItem-Thread-57,EXECUTING,1h:16m:14s8ms,Depot Object Processor WorkItem (batch 26)
      WorkItem-Thread-58,EXECUTING,1h:16m:13s901ms,Depot Object Processor WorkItem (batch 85)
      WorkItem-Thread-59,EXECUTING,1h:16m:14s134ms,Depot Object Processor WorkItem (batch 4)
      WorkItem-Thread-60,EXECUTING,1h:16m:13s870ms,Depot Object Processor WorkItem (batch 65)
      WorkItem-Thread-61,EXECUTING,1h:16m:13s922ms,Depot Object Processor WorkItem (batch 24)
      WorkItem-Thread-62,EXECUTING,1h:16m:13s876ms,Depot Object Processor WorkItem (batch 76)
      WorkItem-Thread-63,EXECUTING,1h:16m:14s54ms,Depot Object Processor WorkItem (batch 37)
      WorkItem-Thread-64,EXECUTING,1h:16m:13s922ms,Depot Object Processor WorkItem (batch 29)
      WorkItem-Thread-65,EXECUTING,1h:16m:14s118ms,Depot Object Processor WorkItem (batch 43)
      WorkItem-Thread-66,EXECUTING,1h:16m:14s156ms,Depot Object Processor WorkItem (batch 34)
      WorkItem-Thread-67,EXECUTING,1h:16m:14s11ms,Depot Object Processor WorkItem (batch 9)
      WorkItem-Thread-68,EXECUTING,1h:16m:11s327ms,Depot Object Processor WorkItem (batch 91)
      WorkItem-Thread-69,EXECUTING,1h:16m:14s202ms,Depot Object Processor WorkItem (batch 23)
      WorkItem-Thread-70,EXECUTING,1h:16m:14s155ms,Depot Object Processor WorkItem (batch 5)
      WorkItem-Thread-71,EXECUTING,1h:16m:13s894ms,Depot Object Processor WorkItem (batch 25)
      WorkItem-Thread-72,EXECUTING,1h:16m:13s888ms,Depot Object Processor WorkItem (batch 55)
      WorkItem-Thread-73,EXECUTING,1h:16m:14s163ms,Depot Object Processor WorkItem (batch 68)
      WorkItem-Thread-74,EXECUTING,1h:16m:14s5ms,Depot Object Processor WorkItem (batch 60)
      WorkItem-Thread-75,EXECUTING,1h:16m:14s55ms,Depot Object Processor WorkItem (batch 63)
      WorkItem-Thread-76,EXECUTING,1h:16m:11s330ms,Depot Object Processor WorkItem (batch 127)
      WorkItem-Thread-77,EXECUTING,1h:16m:14s150ms,Depot Object Processor WorkItem (batch 35)
      WorkItem-Thread-78,EXECUTING,1h:16m:14s10ms,Depot Object Processor WorkItem (batch 79)
      WorkItem-Thread-79,EXECUTING,1h:16m:13s871ms,Depot Object Processor WorkItem (batch 81)
      WorkItem-Thread-80,EXECUTING,1h:16m:14s146ms,Depot Object Processor WorkItem (batch 50)
      WorkItem-Thread-81,EXECUTING,1h:16m:14s6ms,Depot Object Processor WorkItem (batch 47)
      WorkItem-Thread-82,EXECUTING,1h:16m:13s871ms,Depot Object Processor WorkItem (batch 46)
      WorkItem-Thread-83,EXECUTING,1h:16m:14s144ms,Depot Object Processor WorkItem (batch 58)
      WorkItem-Thread-84,EXECUTING,1h:16m:14s145ms,Depot Object Processor WorkItem (batch 87)
      WorkItem-Thread-85,EXECUTING,1h:16m:14s141ms,Depot Object Processor WorkItem (batch 54)
      WorkItem-Thread-86,EXECUTING,1h:16m:14s138ms,Depot Object Processor WorkItem (batch 39)
      WorkItem-Thread-87,EXECUTING,1h:16m:11s328ms,Depot Object Processor WorkItem (batch 142)
      WorkItem-Thread-88,EXECUTING,1h:16m:14s112ms,Depot Object Processor WorkItem (batch 44)
      WorkItem-Thread-89,EXECUTING,1h:16m:14s4ms,Depot Object Processor WorkItem (batch 32)
      WorkItem-Thread-90,EXECUTING,1h:16m:14s129ms,Depot Object Processor WorkItem (batch 57)
      WorkItem-Thread-91,EXECUTING,1h:16m:14s132ms,Depot Object Processor WorkItem (batch 31)
      WorkItem-Thread-92,EXECUTING,1h:16m:11s331ms,Depot Object Processor WorkItem (batch 107)
      WorkItem-Thread-93,EXECUTING,1h:16m:14s5ms,Depot Object Processor WorkItem (batch 56)
      WorkItem-Thread-94,EXECUTING,1h:16m:14s107ms,Depot Object Processor WorkItem (batch 62)
      WorkItem-Thread-95,EXECUTING,1h:16m:14s11ms,Depot Object Processor WorkItem (batch 61)
      WorkItem-Thread-96,EXECUTING,1h:16m:11s331ms,Depot Object Processor WorkItem (batch 136)
      WorkItem-Thread-97,EXECUTING,1h:16m:14s128ms,Depot Object Processor WorkItem (batch 8)
      WorkItem-Thread-98,EXECUTING,1h:16m:14s126ms,Depot Object Processor WorkItem (batch 83)
      WorkItem-Thread-99,EXECUTING,1h:16m:11s329ms,Depot Object Processor WorkItem (batch 110)
      
      Lightweight Work Item Thread status:
      =====================================
      Name,State,Time in state (ms),Additional Info
      --------------------------------------------------
      Lightweight-WorkItem-Thread-0,WAITING,1h:54m:2s894ms,
      Lightweight-WorkItem-Thread-1,WAITING,1h:53m:38s277ms,
      Lightweight-WorkItem-Thread-2,WAITING,1h:53m:38s459ms,
      Lightweight-WorkItem-Thread-3,WAITING,1h:54m:59s833ms,
      Lightweight-WorkItem-Thread-4,WAITING,1h:54m:59s833ms,
      Lightweight-WorkItem-Thread-5,WAITING,1h:54m:59s830ms,
      Lightweight-WorkItem-Thread-6,WAITING,1h:53m:31s496ms,
      Lightweight-WorkItem-Thread-7,WAITING,1h:53m:39s101ms,
      Lightweight-WorkItem-Thread-8,WAITING,1h:53m:31s455ms,
      Lightweight-WorkItem-Thread-9,WAITING,1h:53m:12s524ms,
      Lightweight-WorkItem-Thread-10,WAITING,1h:53m:12s569ms,
      Lightweight-WorkItem-Thread-11,WAITING,1h:53m:30s698ms,
      Lightweight-WorkItem-Thread-12,WAITING,1h:53m:12s532ms,
      Lightweight-WorkItem-Thread-13,WAITING,1h:53m:12s536ms,
      Lightweight-WorkItem-Thread-14,WAITING,1h:53m:41s813ms,
      Lightweight-WorkItem-Thread-15,WAITING,1h:48m:24s966ms,
      Lightweight-WorkItem-Thread-16,WAITING,1h:53m:6s348ms,
      Lightweight-WorkItem-Thread-17,WAITING,1h:47m:34s576ms,
      Lightweight-WorkItem-Thread-18,WAITING,1h:47m:25s439ms,
      Lightweight-WorkItem-Thread-19,WAITING,1h:54m:59s826ms,
      Lightweight-WorkItem-Thread-20,WAITING,1h:47m:14s763ms,
      Lightweight-WorkItem-Thread-21,WAITING,1h:47m:13s758ms,
      Lightweight-WorkItem-Thread-22,WAITING,1h:47m:8s794ms,
      Lightweight-WorkItem-Thread-23,WAITING,1h:47m:8s924ms,
      Lightweight-WorkItem-Thread-24,WAITING,1h:47m:5s396ms,
      Lightweight-WorkItem-Thread-25,WAITING,1h:47m:5s199ms,
      Lightweight-WorkItem-Thread-26,WAITING,1h:54m:59s823ms,
      Lightweight-WorkItem-Thread-27,WAITING,1h:47m:1s728ms,
      Lightweight-WorkItem-Thread-28,WAITING,1h:46m:58s484ms,
      Lightweight-WorkItem-Thread-29,WAITING,1h:45m:21s211ms,
      Lightweight-WorkItem-Thread-30,WAITING,1h:54m:59s822ms,
      Lightweight-WorkItem-Thread-31,WAITING,1h:45m:1s480ms,
      Lightweight-WorkItem-Thread-32,WAITING,1h:44m:58s576ms,
      Lightweight-WorkItem-Thread-33,WAITING,1h:54m:59s820ms,
      Lightweight-WorkItem-Thread-34,WAITING,1h:54m:59s817ms,
      Lightweight-WorkItem-Thread-35,WAITING,1h:54m:59s815ms,
      Lightweight-WorkItem-Thread-36,WAITING,1h:54m:59s817ms,
      Lightweight-WorkItem-Thread-37,WAITING,1h:54m:59s817ms,
      Lightweight-WorkItem-Thread-38,WAITING,1h:54m:59s818ms,
      Lightweight-WorkItem-Thread-39,WAITING,1h:54m:59s815ms,
      Lightweight-WorkItem-Thread-40,WAITING,1h:54m:59s818ms,
      Lightweight-WorkItem-Thread-41,WAITING,1h:54m:59s816ms,
      Lightweight-WorkItem-Thread-42,WAITING,1h:54m:59s817ms,
      Lightweight-WorkItem-Thread-43,WAITING,1h:54m:59s816ms,
      Lightweight-WorkItem-Thread-44,WAITING,1h:54m:59s816ms,
      Lightweight-WorkItem-Thread-45,WAITING,1h:54m:59s815ms,
      Lightweight-WorkItem-Thread-46,WAITING,1h:54m:59s814ms,
      Lightweight-WorkItem-Thread-47,WAITING,1h:54m:59s814ms,
      Lightweight-WorkItem-Thread-48,WAITING,1h:54m:59s814ms,
      Lightweight-WorkItem-Thread-49,WAITING,1h:54m:59s813ms,
      
      Assigned Work Items:
      =====================
      App Server,Number
      --------------------------------------------------
      whml24698_job3,0
      whml24699_job3,0
      whml24707_job3,0
      whml24700_job3,0
      whml24708_job3,55
      
      Outstanding Work Items:
      ========================
      App Server,Number
      --------------------------------------------------
      whml24698_job3,0
      whml24699_job3,0
      whml24707_job3,0
      whml24700_job3,0
      
      Unassigned Work Items:
      =======================
      0
      
      Work Item Result Objects:
      ==========================
      App Server,Number
      --------------------------------------------------
      whml24698_job3,0
      whml24707_job3,0
      whml24700_job3,0
      whml24708_job3,0