4 Replies Latest reply on Aug 17, 2016 11:32 AM by Scott Bleasdell

    How to collect data from TCP/UPD in HA environment

    Dale Willis

      In our environment, we have 3 data centers, but only 2 are functional for ITDA as of right now.  What is the best way to set it up so that ITDA can collect data from either ITDA server in either environment and then share that data across both ITDA servers.

       

      ITDA server in data center 1 collects logs through UPD from ABC.

      ITDA server in data center 2 collects logs through UPD from ABC.

       

      My problem is what if Data center 1 gets destroyed and I still want the log data from ITDA server 1.  I can share/make redundant the indexer I believe which solves that problem.  But now how do I collect that data from ITDA Server 2 and keep putting it into the same indexer?

       

      I'm not sure if I'm missing something, or maybe it cannot be done.

        • 1. Re: How to collect data from TCP/UPD in HA environment
          Sameer Pokarna

          Hi Dale,

           

          From what I understand, based on your question, you have 2 independent ITDA installations in your 2 data centres, and you want to somehow consolidate the data in these 2 ITDA installations into a single indexer cluster. Is this assumption correct ? If it is possible, we can have an offline conversation, as I would like to understand your deployment to better suggest what is possible.

           

           

          Regards,

          Sameer

          • 2. Re: How to collect data from TCP/UPD in HA environment
            Dale Willis

            Sorry for the late reply.  Your assumptions are correct.  I currently do have them share a index cluster.  I think what I can do is just create a DNS entry, and then have the DNS entry flip when the datacenter flips for failover testing.  But I would like to hear your suggestions.

            • 3. Re: How to collect data from TCP/UPD in HA environment
              Sameer Pokarna

              Hi Dale,

               

              Unfortunately, this is not supported in ITDA. Like you mention, creating an indexer cluster across data centres seems like a possible solution, but there a few problems I foresee, and we have not tested this scenario.

              1. When you have a cluster, and some are in data centre 1 and some nodes of cluster in data centre 2, there is no way of guaranteeing that one replica will be in second data centre.

              2. Replication across data centres is not tested and poses its own issues in cases of unreliable WAN connectivity

              3. Configuration data, like data patterns, data collectors, etc. also need to be replicated across the 2 data centres.

               

              For now, only possible solution is to have frequent backups in data centre1, and perform recovery steps in data centre 2, but this will not guarantee latest state, but can at least be restored to the last backed up state.

               

              That is about the closest that ITDA can support as of now.

               

               

              Regards,

              Sameer

              • 4. Re: How to collect data from TCP/UPD in HA environment
                Scott Bleasdell

                Dale, while what Sameer outlines above is correct given the scenario you describe, I would like to talk with you to make sure I understand all the details of what you guys are trying to accomplish and collect your requirements for consideration in a future release of ITDA.  I'll contact you by email to set something up.