14 Replies Latest reply on Apr 28, 2015 2:23 PM by Lokanadhan Karthik

    Reports Stuck in Transmitter Queue

    Lokanadhan Karthik
      Share This:

      We have around 5000 machines reporting to a Master Transmitter which reports from 12 PM every day. (Scanning scheduled at 12:00 PM).

       

      Once the machines start reporting the reports get stuck in the queue. Only few reports are inserted in the DB successfully. This happens almost everyday.

      Then we re-publish the inventory plugin to flush the queue. (Re-publishing solves the issue)

       

      Has anyone faced the similar issue or have any idea why this happens?

       

      Trans OS: Win 2008 R2

      DB: MSSQL 2008 R2

        • 1. Re: Reports Stuck in Transmitter Queue

          Check the following:

          1. Are there any system maintenance tasks () scheduled to run during inventory insertion?

          • 2. Re: Reports Stuck in Transmitter Queue

            2. Check if there is activity on the DB side like backup/scheduled tasks etc..

            • 3. Re: Reports Stuck in Transmitter Queue
              R B

              Hi Greg,

               

              I was also facing the same issue in test environment.Then I restarted the Transmitter and data was inserted into database.

               

              But for above mentioned issue in Production what should be the workaround ?

               

              Thanks,

              Rahul

              • 4. Re: Reports Stuck in Transmitter Queue

                If during inventory insertion there is any interruption (like Schema Manager Maintenance scripts or Activity on the DB side like backup or other scheduled maintenance events) I have noticed that in some cases inventory does not start insertion again.. We need to identify what is causing the interruptions and have it scheduled for sometime later.

                 

                Hope this helps.

                • 5. Re: Reports Stuck in Transmitter Queue
                  Lokanadhan Karthik

                  There are no activities on DB side.

                   

                  How do we check if maintenance task is scheduled in schema?

                  • 6. Re: Reports Stuck in Transmitter Queue

                    there was a hidden page of CMS that would tell us all the task scheduled from CMS like RC queries / LDAP-DB-Sync Etc., Should be available in the KB something like:

                     

                    http://<cmshostname>:8888/query.do? or something.

                    • 7. Re: Reports Stuck in Transmitter Queue
                      Lokanadhan Karthik

                      Found it:

                       

                      http://cms:8888/shell/settings/list_groups.do

                       

                      But, there are no tasks scheduled.

                       

                      How ever i have raised an issue with support. ISS04457706

                      Attached plugin log with DB errors.

                      • 8. Re: Reports Stuck in Transmitter Queue
                        R B

                        Hi Lokanadhan,

                         

                        From logs:

                         

                         

                        [20/Apr/2015:15:44:24 +0530] "Inventory Plug-in" critical - 6614 Timed out waiting for connection. Trying again

                         

                         

                        [20/Apr/2015:15:45:30 +0530] "Inventory Plug-in" major "rsld232806040 (win32:1185533675965446455)" 6666 Database exception: while inserting <MONITOR: machine_id = -1, monitor_size = 15.0, display_type = nullmanufacturer = Standard monitor types model = Generic PnP Monitor device_id = DesktopMonitor1horizontal frequency = 56 vertical Frequency = 60>

                         

                         

                        [20/Apr/2015:15:45:30 +0530] "Inventory Plug-in" major "rsld232806040 (win32:1185533675965446455)" 6666 Database exception\

                        com.microsoft.sqlserver.jdbc.SQLServerException: socket closed\

                         

                        com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.\

                         

                         

                        I can see that your database is down. Make sure that your database is up an Running.

                         

                         

                        Thanks,

                        Rahul Boraste

                        • 9. Re: Reports Stuck in Transmitter Queue
                          Lokanadhan Karthik

                          Database was not down.

                           

                          In the same log you could find that reports were inserted successfully within minutes of DB exceptions.

                          • 10. Re: Reports Stuck in Transmitter Queue
                            R B

                            yes, But according to log it seems database issue.

                             

                            what can you see in 21st April log?



                            • 11. Re: Reports Stuck in Transmitter Queue
                              Lokanadhan Karthik

                              I see same errors in everyday logs. DB connection issues here and there.


                              Here in the below log within seconds the DB is connected again after the error.

                               

                              [22/Apr/2015:16:50:11 +0530] "Inventory Plug-in" info "rsld180005860 (win32:8190092869717323013)" 6537 Requested diff scan for this machine: 172.25.6.61: Endpoint Checksum Kyu3/uKpbUyus8eARbcGuA== != Plugin Cache's Checksum MmfGcg4Rmml2a094vBeTUw==

                              [22/Apr/2015:16:50:12 +0530] "Inventory Plug-in" major "rcmld760407639 (win32:5537297473674556483)" 6677 Failed to rollback\

                              com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.\

                                  at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:171)\

                                  at com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:319)\

                                  at com.microsoft.sqlserver.jdbc.SQLServerConnection.rollback(SQLServerConnection.java:1639)\

                                  at com.marimba.apps.inventory.plugin.processor.MRBAInsertionProcessor.handleScanReport(MRBAInsertionProcessor.java:314)\

                                  at com.marimba.apps.inventory.plugin.processor.MRBAInsertionProcessor.process(MRBAInsertionProcessor.java:71)\

                                  at com.bmc.discovery.processor.impl.ProcessorChain$ExecInstruction.execute(ProcessorChain.java:664)\

                                  at com.bmc.discovery.processor.impl.ProcessorChain$Chain.execute(ProcessorChain.java:255)\

                                  at com.bmc.discovery.processor.impl.ProcessorChain$Chain.run(ProcessorChain.java:217)\

                                  at com.bmc.discovery.processor.impl.ProcessorChain.process(ProcessorChain.java:69)\

                                  at com.marimba.apps.inventory.plugin.DiscoveryPlugin.run(DiscoveryPlugin.java:2580)\

                                  at com.marimba.tools.util.ThreadPool.runClient(ThreadPool.java:463)\

                                  at com.marimba.tools.util.ThreadPool.run(ThreadPool.java:447)\

                                  at java.lang.Thread.run(Unknown Source)

                              [22/Apr/2015:16:50:12 +0530] "Inventory Plug-in" warning "rcl403503588 (win32:3044335910576147030)" 6680 Updating checksum from cache for mac: win32:3044335910576147030

                              [22/Apr/2015:16:50:12 +0530] "Inventory Plug-in" info "rcl403503588 (win32:3044335910576147030)" 6537 Requested diff scan for this machine: 172.24.193.42: Endpoint Checksum 0v4sAYLRRa/bvWxaRsDmeg== != Plugin Cache's Checksum 0wFypLyqPvDOs8BKIjWVlA==

                               

                               

                              Please suggest.

                              • 12. Re: Reports Stuck in Transmitter Queue
                                R B

                                Hi Lokanadhan,

                                 

                                If you say that from database end everything is working fine then the problem may be at Inventory Service Plug-ins.

                                 

                                Because Plug-ins are responsible for inserting data from invtree file to database at Transmitter.

                                 

                                Try reconfiguring Inventory Service Plug-ins.



                                Thanks,

                                Rahul Boraste

                                • 13. Re: Reports Stuck in Transmitter Queue
                                  R B

                                  Hi Lokanadhan,

                                   

                                  Did you got any solution?

                                  • 14. Re: Reports Stuck in Transmitter Queue
                                    Lokanadhan Karthik

                                    Sorry Rahul, i couldn't respond to your reply.

                                     

                                    We didn't try re-configuring the plugin as Support is closely checking the issue. Once the issue is resolved i'll update the status and resolution.

                                     

                                    Thank you all.