10 Replies Latest reply on Mar 4, 2015 9:38 AM by Bill Robinson

    Compliance job failed with no compliance results

    Naren S

      Hi,

      One of the compliance job is failing with no compliance results output. Earlier, the compliance job was failing as some of the EO's were missing in the sensor directory of the other appserver after 8.5 content install. But, this has been fixed now. However, the compliance job is still failing, and no compliance results are generated. Below is the error from appserver log,

       

      [14 Jan 2015 05:48:25,461] [WorkItem-Thread-23] [ERROR] [BLAdmin:BLAdmins:] [Compliance]

      java.lang.NullPointerException

      at com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewConditionResultSuccessVisitor.visit(NewConditionResultSuccessVisitor.java:605)

      at com.bladelogic.om.infra.model.condition.expression.SimpleConditionImpl.accept(SimpleConditionImpl.java:306)

      at com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewConditionResultSuccessVisitor.evaluateSubConditions(NewConditionResultSuccessVisitor.java:846)

      at com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewConditionResultSuccessVisitor.evaluateSubConditions(NewConditionResultSuccessVisitor.java:858)

      at com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewConditionResultSuccessVisitor.visit(NewConditionResultSuccessVisitor.java:734)

      at com.bladelogic.om.infra.model.condition.expression.NAryConditionImpl.accept(NAryConditionImpl.java:182)

      at com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewConditionResultSuccessVisitor.visit(NewConditionResultSuccessVisitor.java:961)

      at com.bladelogic.om.infra.model.condition.expression.SelectorConditionImpl.accept(SelectorConditionImpl.java:470)

      at com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewConditionResultSuccessVisitor.evaluateSubConditions(NewConditionResultSuccessVisitor.java:846)

      at com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewConditionResultSuccessVisitor.evaluateSubConditions(NewConditionResultSuccessVisitor.java:858)

      at com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewConditionResultSuccessVisitor.visit(NewConditionResultSuccessVisitor.java:734)

      at com.bladelogic.om.infra.model.condition.expression.NAryConditionImpl.accept(NAryConditionImpl.java:182)

      at com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewConditionResultSuccessVisitor.evaluateIfSubCondition(NewConditionResultSuccessVisitor.java:830)

      at com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewConditionResultSuccessVisitor.visit(NewConditionResultSuccessVisitor.java:771)

      at com.bladelogic.om.infra.model.condition.expression.IfConditionImpl.accept(IfConditionImpl.java:221)

      at com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewConditionResultSuccessVisitor.evaluateSubConditions(NewConditionResultSuccessVisitor.java:846)

      at com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewConditionResultSuccessVisitor.evaluateSubConditions(NewConditionResultSuccessVisitor.java:858)

      at com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewConditionResultSuccessVisitor.visit(NewConditionResultSuccessVisitor.java:734)

      at com.bladelogic.om.infra.model.condition.expression.NAryConditionImpl.accept(NAryConditionImpl.java:182)

      at com.bladelogic.om.infra.ast.evaluator.NewConditionEvaluator.evaluate(NewConditionEvaluator.java:148)

      at com.bladelogic.om.infra.compliance.job.ComplianceJobWorkItemImpl.doWorkItemExecute(ComplianceJobWorkItemImpl.java:372)

      at com.bladelogic.om.infra.model.job.workitem.ComponentBasedJobWorkItemImpl.execute(ComponentBasedJobWorkItemImpl.java:120)

      at com.bladelogic.om.infra.app.service.workitem.WorkItem.doExecute(WorkItem.java:127)

      at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:176)

      at com.bladelogic.om.infra.app.service.workitem.thread.WorkItemThread.execute(WorkItemThread.java:51)

      at com.bladelogic.om.infra.app.service.thread.BlBlockingThread.run(BlBlockingThread.java:95)

       

      Can someone shed some lights on this please.

       

      Thanks

        • 1. Re: Compliance job failed with no compliance results
          Monoj Padhy

          One of the compliance job is failing with no compliance results output >> do you mean job did not even execute or executed but no result ?

           

          Can you attach the compliance job run log , may be a screenshot.

           

          something you must go through Keep community clean and more readable

           

          Cheers !!!

          • 2. Re: Compliance job failed with no compliance results
            Bill Robinson

            can you attach the debug log from the job run ?

            • 3. Re: Re: Compliance job failed with no compliance results
              Naren S

              Sorry Bill, was on a vacation. Attached the debug logs for your review.

               

              Thanks

              • 4. Re: Re: Compliance job failed with no compliance results
                Bill Robinson

                it looks like the problem is w/ the EO below.  can you manually run that script util command against one of the targets from the appserver and show the output ?

                 

                [20 Jan 2015 04:44:48,238] [WorkItem-Thread-37] [DEBUG] [BLAdmin:BLAdmins:] [Compliance] [] NSHExec: about to run cmd: scriptutil -d "/var/tmp/stage" -h "dmnalx5141" -s eo_common_code lib_filehandling lib_utils EO-Precheck EO-Parameter_functions EO-Command_output eo_executer -x -value1 "x86_64" -value2 "x86_64" -maxdisplay "all" -eotimeout "0" -maxinfolines "all" -rscddir "/opt/bmc/bladelogic/NSH" -ruleid "5.2.4" -remdirprefix "CIS" -filename "/etc/audit/audit.rules" -command "auditctl -l" -commandresult "exit,always arch=3221225534 <anything> key=time-change syscall=adjtimex,settimeofday|exit,always arch=1073741827 <anything> key=time-change syscall=stime,settimeofday,adjtimex|exit,always arch=3221225534 <anything> key=time-change syscall=clock_settime|exit,always arch=1073741827 <anything> key=time-change syscall=clock_settime|exit,always watch=/etc/localtime perm=wa key=time-change" -remvaluelist "-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time-change|-a always,exit -F arch=b32 -S adjtimex -S settimeofday -S stime -k time-change|-a always,exit -F arch=b64 -S clock_settime -k time-change|-a always,exit -F arch=b32 -S clock_settime -k time-change|-w /etc/localtime -p wa -k time-change"

                [20 Jan 2015 04:44:48,238] [WorkItem-Thread-37] [INFO] [BLAdmin:BLAdmins:] [Compliance] [] Executing command : scriptutil -d "/var/tmp/stage" -h "dmnalx5141" -s eo_common_code lib_filehandling lib_utils EO-Precheck EO-Parameter_functions EO-Command_output eo_executer -x -value1 "x86_64" -value2 "x86_64" -maxdisplay "all" -eotimeout "0" -maxinfolines "all" -rscddir "/opt/bmc/bladelogic/NSH" -ruleid "5.2.4" -remdirprefix "CIS" -filename "/etc/audit/audit.rules" -command "auditctl -l" -commandresult "exit,always arch=3221225534 <anything> key=time-change syscall=adjtimex,settimeofday|exit,always arch=1073741827 <anything> key=time-change syscall=stime,settimeofday,adjtimex|exit,always arch=3221225534 <anything> key=time-change syscall=clock_settime|exit,always arch=1073741827 <anything> key=time-change syscall=clock_settime|exit,always watch=/etc/localtime perm=wa key=time-change" -remvaluelist "-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time-change|-a always,exit -F arch=b32 -S adjtimex -S settimeofday -S stime -k time-change|-a always,exit -F arch=b64 -S clock_settime -k time-change|-a always,exit -F arch=b32 -S clock_settime -k time-change|-w /etc/localtime -p wa -k time-change"

                [20 Jan 2015 04:44:48,238] [WorkItem-Thread-37] [DEBUG] [BLAdmin:BLAdmins:] [Compliance] [] About to exec(scriptutil -d "/var/tmp/stage" -h "dmnalx5141" -s eo_common_code lib_filehandling lib_utils EO-Precheck EO-Parameter_functions EO-Command_output eo_executer -x -value1 "x86_64" -value2 "x86_64" -maxdisplay "all" -eotimeout "0" -maxinfolines "all" -rscddir "/opt/bmc/bladelogic/NSH" -ruleid "5.2.4" -remdirprefix "CIS" -filename "/etc/audit/audit.rules" -command "auditctl -l" -commandresult "exit,always arch=3221225534 <anything> key=time-change syscall=adjtimex,settimeofday|exit,always arch=1073741827 <anything> key=time-change syscall=stime,settimeofday,adjtimex|exit,always arch=3221225534 <anything> key=time-change syscall=clock_settime|exit,always arch=1073741827 <anything> key=time-change syscall=clock_settime|exit,always watch=/etc/localtime perm=wa key=time-change" -remvaluelist "-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time-change|-a always,exit -F arch=b32 -S adjtimex -S settimeofday -S stime -k time-change|-a always,exit -F arch=b64 -S clock_settime -k time-change|-a always,exit -F arch=b32 -S clock_settime -k time-change|-w /etc/localtime -p wa -k time-change"): pid=-1

                [20 Jan 2015 04:44:48,247] [WorkItem-Thread-37] [DEBUG] [BLAdmin:BLAdmins:] [Compliance] [] exec(scriptutil -d "/var/tmp/stage" -h "dmnalx5141" -s eo_common_code lib_filehandling lib_utils EO-Precheck EO-Parameter_functions EO-Command_output eo_executer -x -value1 "x86_64" -value2 "x86_64" -maxdisplay "all" -eotimeout "0" -maxinfolines "all" -rscddir "/opt/bmc/bladelogic/NSH" -ruleid "5.2.4" -remdirprefix "CIS" -filename "/etc/audit/audit.rules" -command "auditctl -l" -commandresult "exit,always arch=3221225534 <anything> key=time-change syscall=adjtimex,settimeofday|exit,always arch=1073741827 <anything> key=time-change syscall=stime,settimeofday,adjtimex|exit,always arch=3221225534 <anything> key=time-change syscall=clock_settime|exit,always arch=1073741827 <anything> key=time-change syscall=clock_settime|exit,always watch=/etc/localtime perm=wa key=time-change" -remvaluelist "-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time-change|-a always,exit -F arch=b32 -S adjtimex -S settimeofday -S stime -k time-change|-a always,exit -F arch=b64 -S clock_settime -k time-change|-a always,exit -F arch=b32 -S clock_settime -k time-change|-w /etc/localtime -p wa -k time-change"): pid=16661

                [20 Jan 2015 04:44:48,249] [WorkItem-Thread-37] [INFO] [BLAdmin:BLAdmins:] [Compliance] [] Started pid 16661: scriptutil -d "/var/tmp/stage" -h "dmnalx5141" -s eo_common_code lib_filehandling lib_utils EO-Precheck EO-Parameter_functions EO-Command_output eo_executer -x -value1 "x86_64" -value2 "x86_64" -maxdisplay "all" -eotimeout "0" -maxinfolines "all" -rscddir "/opt/bmc/bladelogic/NSH" -ruleid "5.2.4" -remdirprefix "CIS" -filename "/etc/audit/audit.rules" -command "auditctl -l" -commandresult "exit,always arch=3221225534 <anything> key=time-change syscall=adjtimex,settimeofday|exit,always arch=1073741827 <anything> key=time-change syscall=stime,settimeofday,adjtimex|exit,always arch=3221225534 <anything> key=time-change syscall=clock_settime|exit,always arch=1073741827 <anything> key=time-change syscall=clock_settime|exit,always watch=/etc/localtime perm=wa key=time-change" -remvaluelist "-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time-change|-a always,exit -F arch=b32 -S adjtimex -S settimeofday -S stime -k time-change|-a always,exit -F arch=b64 -S clock_settime -k time-change|-a always,exit -F arch=b32 -S clock_settime -k time-change|-w /etc/localtime -p wa -k time-change"

                [20 Jan 2015 04:44:48,249] [WorkItem-Thread-37] [DEBUG] [BLAdmin:BLAdmins:] [Compliance] [] About to wait for process with pid:16661

                [20 Jan 2015 04:44:48,265] [WorkItem-Thread-33] [DEBUG] [BLAdmin:BLAdmins:] [Compliance] [] CommandRunFactory.getBlAssetFromContainer(echo defaults |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo defaults,nodev |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo defaults,nodev,nosuid |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo defaults,nodev,nosuid,noexec |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo gid=5,mode=620 |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;): 156(ms)

                [20 Jan 2015 04:44:48,266] [WorkItem-Thread-33] [ERROR] [BLAdmin:BLAdmins:] [Compliance] []

                • 5. Re: Compliance job failed with no compliance results
                  Naren S

                  Here is the EO output,

                   

                  <UNIX_SRR_Result>

                  <data>

                  <findings>

                  <finding>Command [auditctl -l] does not give expected result [exit,always arch=3221225534 .* key=time-change syscall=adjtimex,settimeofday]</finding>

                  <finding>Command [auditctl -l] does not give expected result [exit,always arch=1073741827 .* key=time-change syscall=stime,settimeofday,adjtimex]</finding>

                  <finding>Command [auditctl -l] does not give expected result [exit,always arch=3221225534 .* key=time-change syscall=clock_settime]</finding>

                  <finding>Command [auditctl -l] does not give expected result [exit,always arch=1073741827 .* key=time-change syscall=clock_settime]</finding>

                  <finding>Command [auditctl -l] does not give expected result [exit,always watch=/etc/localtime perm=wa key=time-change]</finding>

                  </findings>

                  </data>

                  <status>Open</status>

                  </UNIX_SRR_Result>

                  • 6. Re: Compliance job failed with no compliance results
                    Bill Robinson

                    hmm - i think i pasted in the wrong thing - i think we actually want the 'command' output of this:

                    [20 Jan 2015 04:44:33,393] [WorkItem-Thread-41] [DEBUG] [BLAdmin:BLAdmins:] [Compliance] [] CommandRunFactory.getBlAssetFromContainer(echo defaults |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo defaults,nodev |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo defaults,nodev,nosuid |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo defaults,nodev,nosuid,noexec |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo gid=5,mode=620 |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;): 121(ms)

                     

                     

                     

                     

                    [20 Jan 2015 04:45:06,979] [WorkItem-Thread-26] [DEBUG] [BLAdmin:BLAdmins:] [Compliance] [] Using a ComponentAssetContainer to getBlAsset of echo defaults |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo defaults,nodev |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo defaults,nodev,nosuid |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo defaults,nodev,nosuid,noexec |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo gid=5,mode=620 |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP; on component CIS - Red Hat Enterprise Linux 6 (dmnanlx7106) for session type JOB

                    [20 Jan 2015 04:45:06,981] [WorkItem-Thread-26] [DEBUG] [BLAdmin:BLAdmins:] [Compliance] [] Executing external command : mkdir -p '//dmnanlx7106/opt/bmc/bladelogic/NSH/tmp' ;cp -tf '/opt/bmc/bladelogic/NSH/tmp/application_server/scripts/68993487-6d56-48d3-953c-238cd26ebc6b/cmd_130_52727760.sh' '//dmnanlx7106/opt/bmc/bladelogic/NSH/tmp/';nexec -i dmnanlx7106 'chmod 755 /opt/bmc/bladelogic/NSH/tmp/cmd_130_52727760.sh;/opt/bmc/bladelogic/NSH/tmp/cmd_130_52727760.sh; rm -f /opt/bmc/bladelogic/NSH/tmp/cmd_130_52727760.sh'

                    [20 Jan 2015 04:45:07,150] [WorkItem-Thread-26] [DEBUG] [BLAdmin:BLAdmins:] [Compliance] [] CommandRunFactory.getBlAssetFromContainer(echo defaults |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo defaults,nodev |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo defaults,nodev,nosuid |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo defaults,nodev,nosuid,noexec |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;echo gid=5,mode=620 |tr ',' '\n'|egrep 'dev|nodev'|tail -1;echo $?;echo BMC_SEP;): 170(ms)

                    [20 Jan 2015 04:45:07,151] [WorkItem-Thread-26] [ERROR] [BLAdmin:BLAdmins:] [Compliance] []

                    java.lang.NullPointerException

                    • 7. Re: Compliance job failed with no compliance results
                      Naren S

                      Bill,

                       

                      Below is the output when i run this command from appserver,

                       

                      BMC_SEP

                      nodev

                      0

                      BMC_SEP

                      nodev

                      0

                      BMC_SEP

                      nodev

                      0

                      BMC_SEP

                      0

                      BMC_SEP

                       

                      When i run the compliance job against single target, the job succeeds. But, when i run the job against multiple targets, the job fails for most of the servers with null error in the job run log. So, I set the "No of Targets to process in Parallel" option to '1', and executed the job against all targets (but, not more than 25 targets.)

                       

                      The job completed successfully now, but it took 6 hours for the job to complete. I feel I am having some issue with the appserver configuration related to performance.

                       

                      Any help is appreciated.

                       

                      Thanks

                      • 8. Re: Compliance job failed with no compliance results

                        Any resolution to this?  What changes were made to get this to work, I am seeing the exact same error. 

                        • 9. Re: Compliance job failed with no compliance results

                          I was able to resolve this issue by adding the following entry into users.local on the target Linux server.

                           

                          SYSTEM rw,map=root

                           

                          Seems to be related to this discussion/issue/defect....

                           

                          https://communities.bmc.com/thread/110557?start=0&tstart=0

                           

                          Something going on where commands on the Windows appserver are executing on the linux target as SYSTEM.  According to the thread it is a defect for which there is a patch

                          • 10. Re: Compliance job failed with no compliance results
                            Bill Robinson

                            i think JeffS is right here - you need to either use the workaround or you need to update to the version of bsa w /the fix (8.5.01 patch2 or 3 iirc)