5 Replies Latest reply on Jun 16, 2015 5:22 PM by Mike Reider

    Compliance job timeouts

    Mike Reider

      BSA 8.5.1 patch 4 (redhat 6 appservers, MSSQL2012 db)


      Hi all, getting massive amounts of timeouts running compliance jobs vs 900+ redhat boxes,




      Double checked the failed boxes, no connection errors. Checked the agent logs, it shows it copying over scriptutil sh scripts over to /var/tmp/stage and then the log ends


      The compliance is a template with rules similar to CIS benchmark test, bunch of shell scripts. I checked the Compliance job, it has a Job Timeout of 90 and Job Part Timeout of 60.


      I also ran the compliance check against 1 or 2 of the failed servers and the compiance run finished successfully, no timeouts. Wondering if its the scale of # of targets (I set the Compliance job to run against 30 targets parallel, but still getting almost half of targets as Canceled)

        • 1. Re: Compliance job timeouts
          Mike Reider

          This may be related to this issue,






          BMC Server Automation (BSA) Windows Patch Analysis or Compliance jobs are hanging when run against a large number of servers.
          The job has a JOB_PART_TIMEOUT defined so is not hanging due to issues with particular servers.
          Running the job against a small number of servers results in the job running to completion
          The same job against the same group of servers may have worked correctly until a recent upgrade
          When the job is in this state, other BSA jobs may be affected and hang.


          Applies to:


          BMC Server Automation 8.5.01
          BMC Server Automation 8.5.01 Patch 1
          BMC Server Automation 8.5.01 Patch 2
          BMC Server Automation 8.5.01 Patch 3
          BMC Server Automation 8.5.01 Patch 4
          BMC Server Automation





          The root cause of this issue may be an issue with certain builds of the Oracle Java Runtime Environment (JRE).
          The affected builds which shipped with certain versions of BSA are:


          JRE Version = 1.7.0_55 - Ships with BSA 8.5.01 and 8.5.01 Patches 1 through 4.
          JRE Version = 1.6.0_75 - Ships with BSA


          To confirm this might be the issue, check the version of the JRE which is being used in this BSA environment. The easiest way to check is from the following area in the BSA Console:


          Configuration - Infrastructure Management - Application Servers
          Select any appserver and click on Appserver Service
          Scroll down to where the JRE version is listed e.,g:


          JRE Version = 1.6.0_75 from Sun Microsystems Inc. or
          JRE Version = 1.7.0_55 from Oracle Corporation




          Upgrade to BSA 8.5.1 Patch 5 when it releases in March 2015. BSA 8.5.1 Patch 6 will ship with JRE Version 1.6.0_u71 which resolves this issue.
          If this issue is encountered prior to 8.5.1 Patch 5 being released and must be resolved immediately, contact BMC Support who may be able to assist with a workaround of either downgrading or upgrading the JRE to a version known not to have this issue e.g 1.6.0_37 (for BSA 8.3) or 1.7.0_04/1.7.0_71 (for BSA 8.5.1.X). Modifying the JRE version should not be attempted without the guidance of BMC Support.

          • 2. Re: Compliance job timeouts
            Rajeev Gupta

            You can set the JOB_PART_TIMEOUT property of the Job

            Also, you can make the parallel execution to a higher limit.

            • 3. Re: Compliance job timeouts
              Mike Reider

              I took out all timeouts from jobs, ran it against 30 servers in parallel max, its taking much longer to run now but at least im not getting a massive amount of cancellations. Running CIS-type compliance against 950 redhat endpoints is taking around 3 hrs, but to be fair we have a lot of custom scripts (ext objects) that have to open new jvms for every rule.

              • 4. Re: Compliance job timeouts
                Bill Robinson

                Why do you need to open jvms in your eos ?

                • 5. Re: Compliance job timeouts
                  Mike Reider

                  sorry didnt mean jvms, opens a scriptutil session for every sh script, so its using up a worker thread for each session.