10 Replies Latest reply on Aug 22, 2019 9:09 AM by Lisa Keeler

    11.3 Playback issue

    Mark Lemar
      Share This:

      I'm currently playing back some discovery data taken from an 11.3 appliance configured in record mode.  However, I've noticed a number of playback failures with the error 'NoResponse' - 'Pool data is required but not present'.  This is despite the relevant pool directory existing for the IP in question on the 11.3 playback appliance.

       

      It looks like this is impacting all Windows servers taken from the record appliance.

       

      I have a support case open at the moment for a Windows related scan issue in 11.3 & support have advised that they're having problems playing back my supplied record data for this issue.

       

      Is there a known playback issue in 11.3 at present?

        • 1. Re: 11.3 Playback issue
          Andrew Waters

          It would only complain like that if you have .no-expiry files in the pool data directory in which case it will not look for record data.

           

          What exactly did you put onto the appliance?

          2 of 2 people found this helpful
          • 2. Re: 11.3 Playback issue
            Mark Lemar

            I moved over the pool and record data for the host and created .no-expiry files in both locations.

             

            I've now removed the .no-expiry files under pool & record, but playback fails with a different message 'No SNMP credentials defined for endpoint', which is interesting as this wasn't scanned using an SNMP credential.

             

            As a principle, do I need to use .no-expiry files to prevent Discovery from actively trying to discover an IP, or does having the appliance set to 'Playback' prevent that anyhow?

             

            Playback has now complete and the only failures are Windows host (.no-expiry files in the pool and record dirs for non Windows hosts hasn't caused the playback of these IPs to fail).

             

            I don't have any proxy credentials defined on my playback appliance, is there now some dependency on these being defined in 11.3?

            • 3. Re: 11.3 Playback issue
              Andrew Waters

              It is more complicated than that.

               

              By default pool data is (mostly) not written. This was changed in ADDM 9.0 because it improves performance and the original purpose of pool data has largely gone away. Pool data represents the post process results, i.e. what has been found and recognised. When it exists and is new enough it will be used. The .no-expiry file prevents the pool data being removed when a device is newly scanned and also ignores its age.

               

              Record data is only written in record mode and represents data before it has been processed. For non-Windows this will be performed on the appliance. However, for Windows any record data would be recorded on the machine running the Windows proxy used in discovery. Thus to play back record data on Windows you would need a Windows proxy in playback mode with the Windows proxy record data. It is possible to bypass this by using pool data but by default the data is not written.

               

              Using tw_options WRITE_DISCOVERY_POOL_DATA will cause the pool data to be written during discovery. At a point in the future this option will almost certainly go away.

              2 of 2 people found this helpful
              • 4. Re: 11.3 Playback issue
                Mark Lemar

                When you say "Thus to play back record data on Windows you would need a Windows proxy in playback mode", do you mean that we need to have the Windows proxy defined on the playback appliance - with the same credential ID?

                 

                We're currently able to playback Windows scan data from an 11.1 scanning appliance (in record mode) on an 11.1 playback appliance without a problem.  The playback appliance has no proxies defined on it & the Windows proxies are not configured in record mode themselves.

                 

                I was previously unaware of the tw_option WRITE_DISCOVERY_POOL_DATA and can see that it exists in 11.1 also (set to False on our appliances).

                 

                I can see that despite this option being set to FALSE in 11.1, we have hundreds of files under the pool directory for an example Windows IP.  On 11.3, with the option set to false, we only have a couple of files listed when scanning the same IP.  Changing the option to TRUE in 11.3 and re-scanning, results in hundreds of file being populated under the relevant pool dir.

                 

                Has something changed between 11.1 and 11.3 around the use of WRITE_DISCOVERY_POOL_DATA?

                • 5. Re: 11.3 Playback issue
                  Mark Lemar

                  My playback using scan data taken from a scan appliance with the tw_option (WRITE_DISCOVERY_POOL_DATA=True) set, has now worked as expected.

                   

                  Therefore it seems that something has changed since 11.1, which makes this option a requirement for playback to work in 11.3

                  2 of 2 people found this helpful
                  • 6. Re: 11.3 Playback issue
                    Mark Lemar

                    Any known changes in 11.3 Patch 5, which would have impacted Playback?

                     

                    I noticed that the upgrade itself reset the WRITE_DISCOVERY_POOL_DATA back to 'False', so I've changed this to 'True' on my scanner.

                     

                    The playback scan on My 11.3.0.5 playback appliance isn't progressing past 0% & I'm observing very high CPU usage on the server.

                    • 7. Re: 11.3 Playback issue
                      Andrew Waters

                      There have not been any changes in record and playback functionality in any of the 11.3 patches.

                      2 of 2 people found this helpful
                      • 8. Re: 11.3 Playback issue
                        Lisa Keeler

                        In support, I receive record & pool data from customers and play it back without a problem.

                        And, I don't have a Windows Proxy.  And, I'm pretty sure the customers did not set WRITE_DISCOVERY_POOL_DATA=True.

                         

                        I play the record/pool data back on my test appliance like this:

                        1) I visually look at the data to make sure it is a good set of record and pool data

                        2) copy the data onto my appliance

                        3) Set the .no-expiry files onto the pool data like this:

                            find /usr/tideway/var/pool/ -type d -links 2 -exec touch {}/.no-expiry \;

                        4) Make sure I have a "dummy" credential for all the potential protocols, such as ssh, snmp, etc.

                        5) put my appliance into playback mode

                        6) add a discovery run for the IP's

                         

                        There are some bugs with the record & pool data in specific releases. (the bugs caused the pool data to not be collected, and/or caused the record/dip data to not be collected)

                        But, if customer is on a later release, there should not be problems.

                         

                        If this method changes in a future release (so that we have to put the pool data onto a windows proxy in playback mode), that would be quite a bit of additional trouble for Support folks.

                        3 of 3 people found this helpful
                        • 9. Re: 11.3 Playback issue
                          Mark Lemar

                          Thanks Lisa, that's very helpful to know.

                           

                          We're currently running 11.3 in prod & found that we had to set "WRITE_DISCOVERY_POOL_DATA = True" on our scanners for playback to work on an another 11.3 appliance (hence this original thread).

                           

                          I've now upgraded 2 dev appliances to 11.3.0.5 and am trying to playback scan data obtained from one appliance on to the other.  Both appliances are running the same version, same platform scripts, same TKUs + have the same hardware specification.  They also have the same ssh & vCenter credentials set, although no Windows proxies defined on the playback appliance.  Following the 11.3.0.5 upgrade, I reapplied the "WRITE_DISCOVERY_POOL_DATA = True" setting on the scanner appliance, but left it set to False on the intended playback appliance.

                           

                          I have followed exactly the same process we follow today in prod, to copy the scan data (pool and record) to the playback appliance & create the necessary .no-expiry files.

                           

                          Although the playback works, it takes a very long time to complete.  E.g. the itself scan of approx 4200 hosts took around 5 hours to complete, but the playback took 10 hours to complete.  Typically, I would expect playback to be quicker than scanning, not take longer - certainly not double the length of time.  I can't imagine you playback such volumes for your support cases, so perhaps not something you've noticed yourself?

                           

                          I'm wondering if we no longer need to set "WRITE_DISCOVERY_POOL_DATA = True" on an 11.3.0.5 scanner & if not, whether the data collected with this value set, is the cause of the long running playback?  Do you know if the bug in earlier versions which you referred to, was specifically related to this setting in 11.3 & whether it's no fixed in 11.3.0.5?

                          • 10. Re: 11.3 Playback issue
                            Lisa Keeler

                            Hi Mark,

                             

                            I typically don't ever playback any great volume of data.  Usually just a few hosts.  But, yes, I've noticed that playback seems slow.  I am in the habit of waiting.

                             

                            One time, I did try to playback a very large volume of data, and it took such a long time that I finally cancelled the run because I decided on solving my problem in a different way (using backup/restore, I think).

                             

                            This setting:  WRITE_DISCOVERY_POOL_DATA = True, does not slow down the playback.  It just provides the pool data necessary for the playback of Windows hosts.

                             

                            From reading our KA's, apparently you do still need WRITE_DISCOVERY_POOL_DATA = True on the scanner.  And, maybe the customers that send me record/pool data already have turned it on from working on other cases.

                             

                            Our Internal KA says that needing that option setting was a performance optimization introduced in Discovery 10.0 so that pool data isn't written by default.

                            But, in 10.0 and greater, pool data is only written in "Record Mode" anyway.

                            So, you can just leave WRITE_DISCOVERY_POOL_DATA = True set all the time on your Scanner (or on all appliances), and it only causes the pool data to be written when in "Record Mode", which is what you want.

                            (Before 10.0, pool data was written even when not in "Record Mode").

                             

                            The defects with record/pool data were fixed in 11.3.0.1 and 11.2.0.4 and later releases.

                            From our Internal KA:

                            "

                            There is a defect with gathering the record/pool data, which is fixed in these releases:

                            In 11.3.01 (DRUD1-23102), and in 11.2.04 (DRUD1-23103).

                            "

                             

                            YOU SAID:> Both appliances are running the same version, same platform scripts, same TKUs + have the same hardware specification.  They also have the same ssh & vCenter credentials set, although no Windows proxies defined on the playback appliance.

                             

                            LISA:>

                            Just FYI:

                            Having the same major version of Discovery and the same TKU level (and custom patterns) is important. 

                            The rest is not important.

                            For example, I playback data from many customers. Customer can be on virtual appliances, or on physical hardware .. it doesn't matter.

                            And CentOS 6 -vs- CentOS 7 doesn't matter either.

                            To playback record data from 11.3.0.3, I can play that back on any 11.3.0.x lab system that I have, such as 11.3.0.1 or 11.3.0.5.

                            I have one test appliance on 11.3 to playback data for 11.3; one for 11.2; one for 11.1, and so on.... for all of our supported versions.

                            Usually, I keep my test appliance at the latest patch level, but not always.

                            I don't need the customer's credentials, of course.  I just have a "dummy" credential that matches all IP's for ssh, snmp, etc.  I use "dummy" for the username and the password.

                            Playback only wants to find a matching credential of the correct type... it doesn't actually use the credential.

                                (And, of course, I don't even have "live" IP's that match the customer's data).

                            The platform scripts don't matter because the platform scripts aren't used during playback. 

                            The TKU patterns do matter because the patterns get triggered during playback.

                            And, like yours, my playback appliances do not have any Windows Proxies ... although of course, the customers' scanners have some number of Windows Proxies.

                             

                            It sounds like your playback works fine, but it is just slow.  I don't know how to make it faster.

                            2 of 2 people found this helpful