6 Replies Latest reply on Dec 6, 2018 6:02 AM by Chao Liu

    BPM portal v2.11.00 agentless monitoring :Error BMC-MOP in portal.log

    Chao Liu

      Hi

       

      there are lots of error prefix with BMC-MOP in portal.log,how to fix them?thanks

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

      <R,11/13/18 06:23:16 UTC,STDERR,[QueueWatcher Thread #10 for DRMOP]> [INFO,PayloadProcessorImpl,extractEvents] BMC-MOP000035S:- Not processing events from payload because data was stale RSM Name: rsm27 RSM GUID: b4ss123aZzxIshK2rN8E0j7fr6zj2001 Stale payload count= 6

       

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++

      <R,11/28/18 02:31:27 UTC,STDERR,[QueueWatcher Thread #8 for DRMOP]>

      Detail:  Error while converting and saving RSM data; skipping batch...

       

      System Information:

      Level................: SEVERE

      Logger...............: com.bmc.patrol.portal.rsmcommunication.impl.ParameterValuePersistenceWorkerImpl

      Time of Error........: Nov 28, 2018 02:31:27 UTC

      Record Thread ID.....: 369

      Thread Name..........: QueueWatcher Thread #8 for DRMOP

      Source Class.........: com.bmc.patrol.portal.rsmcommunication.impl.ParameterValuePersistenceWorkerImpl

      Source Method........: savePayloads

      Host.................: 172.28.38.140

      Version..............: 2.11.00

      Exception:

      org.springframework.jdbc.UncategorizedSQLException: BMC-MOP000032S:- Error while execution of PortalElementDaoImpl.lockOnElement unable to lock element row in datastore elementGuid=d4ss123wZ64dqoamchPo0itkq98cfn5w; uncategorized SQLException for SQL [SELECT * FROM node WHERE guid = ? FOR UPDATE]; SQL state [42000]; error code [2049]; ORA-02049: timeout: distributed transaction waiting for lock

      ; nested exception is java.sql.SQLException: ORA-02049: timeout: distributed transaction waiting for lock

       

       

      Caused by: java.sql.SQLException: ORA-02049: timeout: distributed transaction waiting for lock

       

       

      at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)

      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)

      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)

      at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:745)

      at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:219)

      at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:813)

      at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1049)

      at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:854)

      at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1146)

      at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3370)

      at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3415)

      at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)

      at com.bmc.patrol.portal.services.element.dao.hibernate.PortalElementDaoImpl.lockOnElement(PortalElementDaoImpl.java:83)

      at com.bmc.patrol.portal.services.element.dao.hibernate.PortalElementDaoImpl.lockOnElement(PortalElementDaoImpl.java:138)

      at com.bmc.patrol.portal.services.element.PortalElementServiceImpl.lockOnElement(PortalElementServiceImpl.java:224)

      at sun.reflect.GeneratedMethodAccessor1232.invoke(Unknown Source)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

      at java.lang.reflect.Method.invoke(Method.java:601)

      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:280)

      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:187)

      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:154)

      at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107)

      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)

      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:210)

      at com.sun.proxy.$Proxy424.lockOnElement(Unknown Source)

      at com.bmc.patrol.portal.rsmcommunication.impl.PayloadProcessorImpl.lockOnElement(PayloadProcessorImpl.java:155)

      at com.bmc.patrol.portal.rsmcommunication.impl.PayloadProcessorInstrumentor.lockOnElement(PayloadProcessorInstrumentor.java:77)

      at com.bmc.patrol.portal.rsmcommunication.impl.ParameterValuePersistenceWorkerImpl$3.doInTransactionWithoutResult(ParameterValuePersistenceWorkerImpl.java:1088)

      at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)

      at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:134)

      at com.bmc.patrol.portal.rsmcommunication.impl.ParameterValuePersistenceWorkerImpl.handlePayloadsAndEvents(ParameterValuePersistenceWorkerImpl.java:1064)

      at com.bmc.patrol.portal.rsmcommunication.impl.ParameterValuePersistenceWorkerImpl.convertAndSaveMessages(ParameterValuePersistenceWorkerImpl.java:570)

      at com.bmc.patrol.portal.rsmcommunication.impl.ParameterValuePersistenceWorkerImpl.savePayloads(ParameterValuePersistenceWorkerImpl.java:241)

      at com.bmc.patrol.portal.rsmcommunication.impl.AbstractQueueWatcher.processMessages(AbstractQueueWatcher.java:378)

      at com.bmc.patrol.portal.rsmcommunication.impl.AbstractQueueWatcher.checkQueues(AbstractQueueWatcher.java:319)

      at com.bmc.patrol.portal.rsmcommunication.impl.BlockingQueueWatcherImpl.run(BlockingQueueWatcherImpl.java:97)

      at java.lang.Thread.run(Thread.java:722)

      ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

        • 1. Re: BPM portal v2.11.00 agentless monitoring :Error BMC-MOP in portal.log
          Betty Neumann

          Hello Chao,

           

          My apologies for the delay, I had missed this message. The errors you are seeing the ORA timeouts and the BMC-MOP000035S:- Not processing events from payload because data was stale error indicates that the portal and/or the RSM may be overloaded. The errors do point to either load and/or tuning which may need to be done on the portal and the RSMs.

           

          I would recommend opening a Support ticket as the best way to investigate is to get you to run the pediag25.sql script and send us the results.

           

          Please run the pediag25.sql which is downloadable from ftp.bmc.com/pub/BPMPortal/oracle/pediag25.sql :

           

          How to run this file?

          --Navigate to Datastore machine as a user with Oracle DBA privileges

          --Go to command a prompt

           

          --Ensure ORACLE_HOME and ORACLE_SID are set for your login

           

          --Example:

          -- set ORACLE_SID=BMCPDS or export ORACLE_SID=BMCPDS

          -- set ORACLE_HOME=<disk>:\Program Files\BMCSoftware\Datastore\ora10g\db_1 or export ORACLE_HOME=/opt/bmc/Datastore/ora10g/db_1

           

          --PATH=$PATH:$ORACLE_HOME/bin or set PATH=%PATH%;%ORACLE_HOME%\bin

           

          -- Navigate to the directory where the pediag25.sql script resides.

          -- Run the pediag25.sql script:

          -- sqlplus / as sysdba

          -- SQL> @pediag25.sql 'username'

           

          --Example:

          -- SQL> @pediag25.sql 'PE'

          -- This will generate 10 files current_portal_activity.html,elements-last-collected-data.html,impact-portal-data.html,oracle_specific_log.html,performance-managers-loaded.html,portal_log_table_loggin_message.html,

          portal_specific_log_proactive.html,portal_specific_log.html  in the directory where the script was run.

          -- Please send us the 10 html files which are created

           

          --NOTE:

          -- The examples above use our default installation directories,  instance name and instance user names. Please modify the commands above to accommodate your installation.

          -- Oracle user names are case sensitive. Please ensure you are using the correct case when entering the instance user.

           

          I would not recommend posting the results of the script here as it does contain system information and configuration details.

          • 2. Re: BPM portal v2.11.00 agentless monitoring :Error BMC-MOP in portal.log
            Chao Liu

            Hi  Betty,

             

            Can I send you the ouput directly to you by email?

            • 3. Re: BPM portal v2.11.00 agentless monitoring :Error BMC-MOP in portal.log
              Chao Liu

              Hi

               

              I have uploaded to ftp.bmc.com/incoming/BPM PORTAL BMC-MOP/pediag-output.zip ,please help check them.

              • 4. Re: BPM portal v2.11.00 agentless monitoring :Error BMC-MOP in portal.log
                Betty Neumann

                It is better if you open a Support ticket so that the team can work with you during your business hours.

                • 5. Re: BPM portal v2.11.00 agentless monitoring :Error BMC-MOP in portal.log
                  Betty Neumann

                  Hi Chao,

                   

                  I really do think you should open a Support case so the team can help you directly. This is not the place to do this.

                   

                  If you review the details of the portal_specific_log_proactive.html there are several recommendations to help tune the portal environment.

                   

                   

                  Portal health: The following elements have too many application instances without thresholds enabled.

                  RSM name Element Parameters
                  rsm26 172.28.179.127      35025
                  rsm26 172.28.179.128      20683

                  Recommended action: De-activate the parameters of the listed application classes. You can also delete or filter out the instances that you do not need.

                   

                  Following are element with too Many App Inst Without Threshold

                  RSM name Element Solution name Application name Application internal name Applications Parameters
                  rsm26 172.28.179.127 BMC-PM-for-Web-App-Servers WebLogic Solution was_WebLogic_JMS_servers_destination_cluster       3460      34600
                  rsm26 172.28.179.128 BMC-PM-for-Web-App-Servers WebLogic Solution was_WebLogic_JMS_servers_destination_cluster       2044      20440
                  rsm20 172.28.35.127 BMC-PM-for-Web-App-Servers WebLogic Solution was_weblogic_Servlet_app_server        479       1916
                  rsm15 ZB-SQLQJ-001-172.20.35.145 BMC-PM-Express-for-Databases-MS-SQL-Server Microsoft SQL Server com_bmc_DRPM2_FilegroupApp        563       1689
                  rsm19 DB-172.28.177.35 BMC-PM-Express-for-Servers-Unix-Linux Linux dr2al_linux_cmdshell_disk        487       1461
                  rsm20 STANDBY-172.28.177.36 BMC-PM-Express-for-Servers-Unix-Linux Linux dr2al_linux_cmdshell_disk        487       1461
                  rsm03 DB-FIN03-172.20.4.48 BMC-PM-Express-for-Databases-Oracle Oracle com_bmc_DRPM3_tablespaceinstance        375       1125
                  rsm10 ZB-SQLQJ-002-172.20.35.144 BMC-PM-Express-for-Databases-MS-SQL-Server Microsoft SQL Server com_bmc_DRPM2_FilegroupApp        361       1083

                  Recommended action: Please filter the instances which you do not need

                   

                   

                  Database health: The following redo log switch is faster than 20 minutes.

                  Date Redo log switches per day
                  03-DEC-2018        125
                  04-DEC-2018        121

                  Recommended action: Increase the size of Oracle log files.

                   

                  Database health: The following list displays database parameters with their actual values and recommended values.

                  Parameter name Current parameter value Recommended parameter value
                  bitmap_merge_area_size 1048576 0
                  commit_point_strength 1 0
                  db_flashback_retention_target 1440 0
                  deferred_segment_creation TRUE FALSE
                  open_cursors 300 Minimum 4000
                  replication_dependency_tracking TRUE FALSE
                  session_cached_cursors 50 Minimum 75
                  session_max_open_files 10 Minimum 20

                  Recommended action: Run the 'set_system_parameter_oracle_bmcpds' DatastoreCLI command.Refer to the BMC Portal Installation Guide for more details.

                   

                   

                  Database health: The following tables have higher retention periods than the recommended values listed in the BMC Portal Monitoring and Management guide. This can causes an increased database size and reduced Portal performance over a period of time.

                  Table name Current retention Recommended retention Property name
                  PARAMETER_VALUE         15          7 portal.history.parameter.value.retention
                  PARAM_SUMMARIZATION_DATA_POINT        429         92 portal.history.parameter.summarizationDataPoint.retention
                  EVENT        101         14 portal.history.event.retention

                  Recommended action: Reduce the retention period for the listed properties, as described in the BMC Portal Monitoring and Management Guide.

                   

                  These recommendations, plus recommendations in the Portal Quick Tune Guide ftp.bmc.com/pub/BPMPortal/Portal_quick_Tune.txt should help to reduce the errors mentioned in the original post.

                  Again, I would ask that we end this thread here and you should open a Support Case to request further assistance.

                   

                  Thanks

                  1 of 1 people found this helpful
                  • 6. Re: BPM portal v2.11.00 agentless monitoring :Error BMC-MOP in portal.log
                    Chao Liu

                    Hi Betty

                     

                    Thanks so much.I will open a support ticket for further help.