1 2 Previous Next 16 Replies Latest reply on Oct 19, 2019 4:55 AM by Hitesh Jha

    Consolidate Data using record and playback options

    Hitesh Jha
      Share This:

      Hello Experts,

      Could you please let me know if there any drawbacks in order to sync data from scanner to consolidator using record and playback options.

      I have built an appliance in Germany DCs and our Consolidator is in US .

      Now in order to feed data to consolidator we need to wait until we have a dedicated circuit in place between the US and Germany .So based on that required  consolidate port 25032 will open accordingly.

      Please advise and let me know the process placed for the same.I  mean to convey once we are ready with the scanner our next goal will be start doing the discovery in record mode and then manually copy the record and pool folder to our existing consolidator right?

       

      I guess we need to execute the below command as well on consolidator itself?

      find /usr/tideway/var/record/ -type d -links 2 -exec touch {}/.no-expiry \;

      find /usr/tideway/var/pool/ -type d -links 2 -exec touch {}/.no-expiry \;

       

      ~Hitesh

        • 1. Re: Consolidate Data using record and playback options
          Andrew Waters

          No, the process is not the same.

           

          Consolidation replays the DDD directly, it is not rerun in discovery. With record and playback the request is sent to discovery which loads the playback data and re-parsers it. Hence these is more overhead in record and playback.

           

          Record and playback has the disadvantages:

          * You need to ensure you have the modes / options set correctly.

          * You need to make sure you have compatible versions, Pool data is non portable across versions. Some may work but it is not a defect if it does not.

          * You need to create the appropriate scans yourself,

          * If you have a cluster consolidator you need to copy the data to all members.

          * Using record data you need credentials in the consolidator for playing back data even though they will not be used.

          * It is not an officially supported way of performing consolidation. That is the purpose of consolidation. As the docs say, record mode is intended for diagnostic support and testing.

          * You have to use pool data if you want Windows or put record data on a Windows proxy. If you have pool data you need to appropriately manage pool data so that it is not erased or mistakenly platying back old pool data.

          2 of 2 people found this helpful
          • 2. Re: Consolidate Data using record and playback options
            Hitesh Jha

            Thanks Andrew for your prompt response.

            Based on the response I did the same but I am unable to validate the host entry using playback option in our QA consolidator.

            Do we have some logs where I can find something related to the same.

            I am totally understand that this is not the official way but our N/W Team is working on the same.But as of now our priority is to get the CI's scanned and available in CMDB.

            • 3. Re: Consolidate Data using record and playback options
              Hitesh Jha

              Please refer the screenshot for your reference.

              • 4. Re: Consolidate Data using record and playback options
                Andrew Waters

                If you are using pool data with no expity you do not need to set an option on the playback machine. You need only do that if you are using record data.

                1 of 1 people found this helpful
                • 5. Re: Consolidate Data using record and playback options
                  Andrew Waters

                  Scrren show from what? From the consolidator? Difficult to know what to expect without knowing the kind of Host, Unix vs Windows.

                   

                  Windows will have little information in record data as it happens on the proxy. You would have pool data on the scanner.

                  1 of 1 people found this helpful
                  • 6. Re: Consolidate Data using record and playback options
                    Hitesh Jha

                    This is a window machine.

                    Above screenshot is related to record and pool data of the target window host which I copied to the consolidator using winscp from my local machine.

                    So Andrew we don't need to copy both the folder record ,pool on consolidator.If I enabled playback option is consolidator and copy the pool data of target window machine it will create a host entry right ?

                    • 7. Re: Consolidate Data using record and playback options
                      Andrew Waters

                      If using pool data you should not need to turn on playback.

                      2 of 2 people found this helpful
                      • 8. Re: Consolidate Data using record and playback options
                        Hitesh Jha

                        Andrew,

                        As  guided I have disable the playback options on Consolidator and copied the pool data but still no luck .

                        1 of 1 people found this helpful
                        • 9. Re: Consolidate Data using record and playback options
                          Andrew Waters

                          Under pool there will be a directory level for each octet of the IP address, e.g. 168.12.5.13 will be in directory /usr/tideway/var/pool/168/12/5/13. That directory should contain the data from that IP address and need a .no-expiry file added to prevent deletion of the data when scanned on the consolidator.

                          • 10. Re: Consolidate Data using record and playback options
                            Hitesh Jha

                            Andrew,

                            I didn't understand much also let me know how we will get the no expiry file.

                            Even I copied the above folder still unable to see any host entry in Consolidator.

                            Please advise!!!!

                            • 11. Re: Consolidate Data using record and playback options
                              Andrew Waters

                              You need t create a .no-expiry (note the leading dot) file in /usr/tideway/var/pool/192/168/10/65. There is no file on the scanner.

                               

                              This could be done in various ways, e.g. touch /usr/tideway/var/pool/192/168/10/65/.no-expiry

                              • 12. Re: Consolidate Data using record and playback options
                                Brice-Emmanuel Loiseaux

                                You need to create a .no-expiry file in this directory (using "touch .no-expiry" command)

                                • 13. Re: Consolidate Data using record and playback options
                                  Lisa Keeler

                                  To get the .no-expiry file on all of your pool directories, you use the command that you sent in your initial question:

                                   

                                  find /usr/tideway/var/pool/ -type d -links 2 -exec touch {}/.no-expiry \;

                                   

                                  In support, we playback data often.  And, we use both record and pool data.

                                  And, we also set the mode to playback.

                                   

                                  If you tried to playback the data without the .no-expiry file in all of your pool data, then the pool data got overwritten, and you need to copy the pool data over to the consolidator again.

                                  Then, use the command above to add the .no-expiry.

                                   

                                   

                                  We have an internal KA with all the steps we use as a matter of routine.

                                   

                                  Part of the Text of our internal KA:

                                  "See below for how to manage record/pool data from a Cluster.

                                  Prepare the appliance to play back

                                  1. Copy the ".tgz" file to "/usr/tideway" folder of the appliance where you want to playback the recorded data
                                  2. cd /
                                  3. tar xvf /usr/tideway/tw_gather_XXX.tgz
                                  4. find /usr/tideway/var/pool/ -type d -links 2 -exec touch {}/.no-expiry \;
                                  5. Admin -> Discovery Configuration, Change Recording Mode to Playback, and Apply changes

                                  Run the scan (common to all methods above)

                                   

                                  From the gui:

                                  • (v11+) Go to Manage->Discovery. For previous versions, click on the discovery tab.
                                  • Scan the IPs
                                    (to find the IPs: tar tf /usr/tideway/tw_gather_XXX.tgz | grep getDeviceInfo.meta)

                                    Or: From the command-line:
                                    find /usr/tideway/var/pool/ -type d -links 2 | sed 's,/usr/tideway/var/pool/,,'  | sed 's,/,.,g' > /usr/tideway/iplist.txt
                                  • tw_injectip -p system -u <password> -f /usr/tideway/iplist.txt

                                  Suggestion if you want to rescan the content of a single gather (only):

                                  • tar tf /usr/tideway/tw_gather_XXX.tgz | grep getDeviceInfo.meta | sed 's,/getDeviceInfo.meta,,' | sed 's,usr/tideway/var/pool/,,' | sed 's,/,.,g' > /usr/tideway/iplist.txt
                                  • tw_injectip -p system -u <password> -f /usr/tideway/iplist.txt

                                   

                                   

                                  Some hints if the playback fails:

                                  • The play back is expected to fail if you have 0 credentials, create one (even if it's not valid).
                                  • Having a proxy in the target appliance is not required.
                                  • In theory, a record data collected on Discovery version X/TKU version Y should be played back on Discovery version X/TKU version Y. With different patterns, you may have different inferred nodes. In practice, if there are no big differences between the source and the target versions, the result is frequently good.


                                  Check the content of the record/pool folders

                                  Requirement for Windows :
                                  /usr/tideway/var/pool is required (with .no-expiry)

                                   

                                  Requirement for Unix:  

                                  - /usr/tideway/var/pool OR /usr/tideway/var/record is required (or both)

                                  - most of the time, the record folder is enough for Unix.


                                   

                                              Requirement for Cloud (AWS):
                                               - Copy cloud contents from both /usr/tideway/var/pool and /usr/tideway/var/record respectively
                                            - Create a dummy credential however the Label Name should match the exact Name as present in the record/pool data as sample below:
                                           
                                  tw_gather_Cloud issue\usr\tideway\var\pool\cloud\aws\Credential Classic Cloud\ap-southeast-1
                                            - Here the Credentail label we crate in discovery should be "Credential Classic Cloud". The Access key and ID could be dummy values and have the required character length. Example Access Key ID - Min 20 Characters and Access key secret 40 characters
                                            - If a credential does not meet this criteria then the playback will fail with No Access

                                   

                                              Requirement for Cloud (Azure):
                                               - Copy cloud contents from both /usr/tideway/var/pool and /usr/tideway/var/record respectively
                                            - Create a dummy credential for Azure with the format of the Directory ID
                                  Example: 

                                  Microsoft Azure
                                  Directory ID       12345678-1234-1234-1234-123456789012
                                  Application ID   12345678-1234-1234-1234-123456789012

                                   

                                              Requirement for Cloud (Google):
                                              -
                                  See internal community posting:  https://communities.bmc.com/message/808724#808724
                                               - Copy cloud contents from both /usr/tideway/var/pool and /usr/tideway/var/record respectively
                                             - Create a dummy credential for Google Cloud with the Label Name matching the exact Name as present in the record/pool data as sample below
                                  Example: 
                                  var/pool/cloud/gcp/Google_API/us-central1/universal-srv/Google.Spanner.Instances.List.meta
                                  Credential Label Name:  Google_API
                                  Service Account Key (on the Credential):  You need a dummy JSON file that meets the requirements.  See attached JSON file:  google_cloud_dummy_file.json
                                  Scan Name:  can be named anything
                                  Region for the Scan:  us-central1
                                  1 of 1 people found this helpful
                                  • 14. Re: Consolidate Data using record and playback options
                                    Andrew Waters

                                    This seems to fail to understand the difference between pool and record data.

                                    1 2 Previous Next