1 2 Previous Next 16 Replies Latest reply on Jun 14, 2019 12:34 PM by ADDM Administrator BNP Paribas

    Data not consolidating from scanner to consolidator

    Mareetta Joseph
      Share This:

      We have a scanner which is not able to consolidate discovery data to consolidator. We tested the connection to the consolidator from scanner and its working fine. But we are not able to see the last scan data being pushed to consolidator, and last consolidation data is blank on scanner. We tried restarting the services, re-adding the consolidator and rebooting the server.

       

      Also, this scanning appliance is consolidating data to both certification (test) and production consolidator. Production consolidation is successful but failing for certification one. We have recently updated the TKU in certification environment but production its not yet done.

       

      I followed this BMC doc while troubleshooting, ADDM: Troubleshooting the consolidation

       

      Could anyone suggest what could be the issue?

        • 1. Re: Data not consolidating from scanner to consolidator
          Andrew Waters

          What is shown on the consolidation setup page on the scanner?

          2 of 2 people found this helpful
          • 2. Re: Data not consolidating from scanner to consolidator
            Mareetta Joseph

            Looks like it is working fine now. We could see the data getting pushed to consolidator now. In consolidation set up page we can see the last data push date and successful connection. In case, if this issue reoccurs could you please suggest us with the possible way of troubleshooting ?

            1 of 1 people found this helpful
            • 3. Re: Data not consolidating from scanner to consolidator
              Andrew Waters

              First look at what the scanner thinks in the UI. Then it really depends upon what it says.

              2 of 2 people found this helpful
              • 4. Re: Data not consolidating from scanner to consolidator
                Mareetta Joseph

                In this case, test connection to the consolidator was successful and the last data push was shown blank. We tried scanning few ips to check if it is working properly, but we couldn't see that getting consolidated in our consolidator UI. But after long time, now its getting consolidated. 

                • 5. Re: Data not consolidating from scanner to consolidator
                  Temitope Salami

                  Hi,

                   

                  Am having similar issue. Have removed and re added the consolidator but its still the same. Although Test connection comes out successful but still get below error on last data push.

                   

                  Unexpected exception while sending data to the consolidation system.

                   

                  Kindly suggest what can be done. Also restarted the services still no luck.

                  • 6. Re: Data not consolidating from scanner to consolidator
                    Mareetta Joseph

                    We waited for quiet a long time after which consolidation started by itself.

                    • 7. Re: Data not consolidating from scanner to consolidator
                      Andrew Waters

                      There will be a traceback in tw_svc_reasoning.log on the scanner for every time it tried and failed to send information to the consolidator. What does it say?

                      1 of 1 people found this helpful
                      • 8. Re: Data not consolidating from scanner to consolidator
                        Surbhi Anand

                        I faced the similar consolidation issue where my scanner wasn't able to send data to consolidator. Tried re-adding the coordinator, connection status showed success while test. Scanned few ips but it didn't trigger the consolidation.

                         

                        Checked the reasoning log and found a warning saying,

                        reasoning. consolidation.manager: WARNING: CORBA exception connecting to consolidation system

                        ' Consolidator_name' : CORBA .TRANSIENT (omniORB. TRANSIENT_NoUsableProfile, CORBA.COMPLETED_NO)

                         

                        I restarted the services which helped the scanner to consolidate the data.

                        1 of 1 people found this helpful
                        • 9. Re: Data not consolidating from scanner to consolidator
                          ADDM Administrator BNP Paribas

                          We're having the same issues

                           

                          The same CORBA errors in reasoning log and the same error when running tw_reasoningstatus.

                           

                          However, service stop/start, server reboots and reconnecting of consolidation connection only appear to resolve the issue temporarily.
                          The problems return very quickly.  This issue is only happening of 1 of 20 scanners that are connected. This potentially points to a specific network-related problem. 
                          I'm interested to understand how the technical consolidation process performs under "network duress"?  

                          • 10. Re: Data not consolidating from scanner to consolidator
                            Andrew Waters

                            If only one scanner is suffering it would tend to imply it is a network issue.

                             

                            Should there be connectivity issues the scanner will keep trying periodically to connect to the consolidator. I don't see that understanding the process is going to help you.

                            2 of 2 people found this helpful
                            • 11. Re: Data not consolidating from scanner to consolidator
                              ADDM Administrator BNP Paribas

                              I can't think of a situation where it would not be useful to understand how the consolidation process works.

                               

                              Our situation is that we have a two-node scanner consolidating to two cluster coordinators that are located on different areas of the network.

                              The problem is occurring trying to push data to both consolidator coordinators.  Understanding exactly what the consolidation process does from the point of data compression on the scanner and as to how the data is transported over the network to the consolidator, as to whether or not the data is corrupted in the event of network glitches or just a sudden drop in bandwidth and subsequent impact on the Reasoning engine..........these are all important factors to help us troubleshoot our network connectivity.


                              I have no idea as to what effect on 'consolidation data transportation' that network downtime or bandwidth degradation may have?

                              Will the process simply stop and not restart? Or will it pause and start to flow again when network performance improves?

                              The understanding of behaviour expectancy is what we're looking for.

                              • 12. Re: Data not consolidating from scanner to consolidator
                                Matt Lambie

                                any value in some deeper tracerouting from the scanner?

                                 

                                will need to run as route to use the ping option...

                                 

                                ### Most networks will not help much if trying to trace the reasoning port ###

                                [root@scanner01 ~]# traceroute -p25032 www.xxx.yyy.zzz

                                traceroute to www.xxx.yyy.zzz (www.xxx.yyy.zzz), 30 hops max, 60 byte packets

                                1  aaa.bbb.ccc.ddd (aaa.bbb.ccc.ddd)  0.487 ms  0.503 ms  0.543 ms

                                2  * * *

                                3  * * *

                                4  * * *

                                5  * * *

                                 

                                ### And default traceroute on Linux is UDP so we need the -I switch ###

                                [root@scanner01 ~]# traceroute -I www.xxx.yyy.zzz

                                traceroute to www.xxx.yyy.zzz (www.xxx.yyy.zzz), 30 hops max, 60 byte packets

                                1  aaa.bbb.ccc.ddd (aaa.bbb.ccc.ddd)  0.337 ms  0.363 ms  0.428 ms

                                2  www.xxx.yyy.zzz (www.xxx.yyy.zzz)  1.742 ms  1.732 ms  1.716 ms

                                3  www.xxx.yyy.zzz (www.xxx.yyy.zzz)  2.855 ms  3.651 ms  3.641 ms

                                4  * * *

                                5  * * *

                                 

                                Not bad timings on the connection above

                                 

                                Maybe try from the scanner to both consolidators as the path will be similar for a number of hops and might show delays and maybe the point at whic the could be a pinch point.

                                1 of 1 people found this helpful
                                • 13. Re: Data not consolidating from scanner to consolidator
                                  Matt Lambie

                                  also try some disk stats on the OS to make sure you data drive is not maxed out.

                                   

                                  Below are result of an appliance running a ds compact so the drives are being hammered!!!

                                   

                                  iostat -xc 1

                                   

                                  Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

                                  sdb               0.00     0.00  126.00    0.00  2408.00     0.00    19.11     4.43   29.62   29.62    0.00   7.63  96.10

                                  sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

                                  sda               0.00     4.00    0.00    5.00     0.00    72.00    14.40     0.00    0.80    0.00    0.80   0.60   0.30

                                   

                                   

                                  avg-cpu:  %user   %nice %system %iowait  %steal   %idle

                                             1.63    0.00    0.75   10.65    0.00   86.97

                                   

                                   

                                  Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

                                  sdb               0.00     0.00  186.00    3.00  3344.00    24.00    17.82     6.00   34.22   34.76    0.33   4.84  91.40

                                  sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

                                  sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

                                   

                                   

                                  avg-cpu:  %user   %nice %system %iowait  %steal   %idle

                                             0.00    0.00    0.50   11.90    0.00   87.59

                                   

                                   

                                  Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

                                  sdb               0.00     0.00  146.00    0.00  2216.00     0.00    15.18     5.26   35.55   35.55    0.00   6.84  99.90

                                  sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

                                  sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

                                   

                                   

                                  avg-cpu:  %user   %nice %system %iowait  %steal   %idle

                                             1.62    0.00    1.00   10.61    0.00   86.77

                                   

                                   

                                  Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

                                  sdb               0.00     0.00  191.00    0.00  2792.00     0.00    14.62     1.98   10.48   10.48    0.00   4.82  92.00

                                  sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

                                  sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

                                   

                                   

                                  avg-cpu:  %user   %nice %system %iowait  %steal   %idle

                                             0.75    0.00    0.88   11.25    0.00   87.12

                                  1 of 1 people found this helpful
                                  • 14. Re: Data not consolidating from scanner to consolidator
                                    Matt Lambie

                                    And another thought!!!

                                     

                                    Check you largest datastore file does not exceed allocated ram as we have seen this causes a snowball affect on performance usually resulting in death to the service

                                    1 2 Previous Next