7 Replies Latest reply on Jun 7, 2007 8:34 AM by Kent Saunders

    Single-Site Highly Available configuration

      We are new to BL and will be rolling out V7.2 in the near future. I am looking for solutions to eliminate single points of failure in the deployment. We have a solution for the database, and the app servers seem to operate well together, so we'll balance access to those via an F5 Big-IP.

       

      My questions are around how to best configure the File server, Provisioning server, PXE, TFTP, NSH Proxy server, and Reports server for HA.

       

      Anyone have experience with this? What have you done to solve these problems?

       

      Thanks,

      -kent

        • 1. Re: Single-Site Highly Available configuration

          Hi Kent,

           

          There are several ways to address HA, here are some suggestions:

           

          File Server: SAN or other shared & fault tolerant storage. Having two or more file server agents that connect to a SAN, front-ended by a load-balancer with persistent connections.

           

           

          PXE: PXE attaches as a DHCP option on the DHCP server. Any HA solution for DHCP should be able to handle PXE HA.

           

          TFTP: Possibly a UDP based load balancer? Possibly you only want one running at a time per provisioning environment with a heart-beat and cold-standby?

           

          If provisioning is HA, you will also want to add HA for the PXEStore (Windows mount and/or web server).

           

          Provisioning Server: F5 Big-IP should work.

           

           

          Reports Server: I believe we can have multiple reporting engines attached to one DB back end, can anyone verify? Reporting is usually pretty low on the HA priority list.

          • 2. Re: Single-Site Highly Available configuration

            In regards to the current version of the product (7.2), below is some guidance. Most of these suggestions have been implemented in the field, others have not but are technically possible.

             

            Keep in mind that depending on the requirements and software involved, some of the BL services can be overloaded on the same physical hardware, so when mentioning clusters per BL service below, that is not necessarily implying a seperate set of hardware for every cluster. For instance, running the BL DB and BL File Server services together on the same hardware might be a logical choice if you intend to run a cluster for the DB services.. you may be able leverage that cluster software for BL File Server services too.

             

            It's also worth considering leveraging existing HA infrastructure where possible... if you have an HA enabled CIFS/NFS solution, this could be used for File Server services and possibly Data Store services depending on what operating systems you are provisioning, etc.

             

            File Server:

             

            Option 1 - utilizing an HA enabled NAS setup. Have each BL App Server mount the File Server data locally and configure each App Server to point back to itself for File Server services. "HA enabled NAS setup" is something like a NetApp or an HA enabled NFS/CIFS server utilizing some shared storage clusterubg software like Veritas Cluster Server (VCS) or equivelent.

             

            Option 2 - utilizing an active/passive shared storage clustering solution like VCS, create a File Server specific VIP and associated file system which will float between the two cluster nodes. Configure all App Servers to utilize the HA enabled File Server VIP for File Server services.

             

            Option 3 - a clustered filesystem for the File Server data mounted across each Application Server. This is likely overkill and I don't know of any customer who has done this but it is technically doable.

             

            Provisioning Server (the BL App Server servicing provisioning manager consoles):

             

            Can be load balanced as App Server services can be.

             

            PXE/TFTP server:

             

            In the current product running multiple active BL PXE services to service requests from the same network could be problematic.

             

            HA enable the PXE/TFTP services by wrapping the services and associated application filesystems in an active/passive shared storage cluster such as VCS or equivelent.

             

            Data Store Server:

             

            Option 1 - via an HA enabled SMB/HTTP setup. NetApp can present filessytem data via HTTP (and other protocols) in addition to the standard CIFS/NFS so their higher end boxes with HA built in could be used for an HA Data Store Server solution for provisioning both Windows and Linux.

             

            Option 2 - utilize an active/passive shared storage clustering solution like VCS.

             

            Option 3 - as Data Store data typically has a low change frequency, run multiple servers presenting the same data and put them behind a load balancer which can load balance SMB and HTTP such as an F5 Big IP, etc. Then utilize either a scheduled or triggered data sync method to sync data from your master Data Store server to all others.

             

            Reports Server:

             

            In the current product running multiple active BL Report services could be problematic.

             

            HA enable the Reports Server by wrapping the service and associated application filesystem in and active/passive shared storage cluster such as VCS or equivelent.

             

            NSH Proxy:

             

            Option 1 - load balance (Big IP, etc.) NSH clients across two active NSH Proxy servers.

             

            Option 2 - utilizing an active/passive shared storage clustering solution like VCS.

             

            - Jim C.

            • 3. Re: Single-Site Highly Available configuration

              That you for the detailed response. It's been very helpful.

               

              Some follow up questions:

               

              Can you explain why it would be problematic to have multiple reporting servers running simultaneously? I would have thought it similar to the app server in that many copies could be running pulling from the same database. It would be helpful to understand why it won't work.

               

              Regarding the Datastore for provisioning, I am considering using a NetApp with only NFS licensed, and mounting an exported directory on multiple servers. Then each of those servers would export via HTTP and SMB. I'd use an F5 to balance across those services. Does this sound like a reasonable option? Would you recommend getting the HTTP/SMB license for the NetApp as a better solution?

               

              You mentioned Veritas Cluster Server. I am not familiar with their product and would like to know if you have any experience with it and would recommend it. Is that your recommendation in that space, and are there other options we should look at?

               

              Thanks again for all your help!

              • 4. Re: Single-Site Highly Available configuration

                Jim

                 

                For File Server option 1 - using a NAS are you saying that in a multiple application server environment the file server storage location can be mounted simultaneously on all apps servers and each app server will use its 'local' mountpoint (via an agent)??

                • 5. Re: Single-Site Highly Available configuration

                  That you for the detailed response. It's been very

                  helpful.

                   

                  Some follow up questions:

                   

                  Can you explain why it would be problematic to have

                  multiple reporting servers running simultaneously? I

                  would have thought it similar to the app server in

                  that many copies could be running pulling from the

                  same database. It would be helpful to understand why

                  it won't work.

                   

                  In the current Reports server product data such as report schedules and archived reports are stored in the Reports server software directory/folder. So if you merely setup a multiple Reports servers and put them behind a load balancer, you might see different schedule/archive data depending on which Reports server you had connected to. You could run such a setup but I think it would be problematic for users of the product.

                   

                  Attempting to share the filesystems where this local data lands across multiple Reports server would be problematic too due to lack of file lock control in the Reports product and references to some of that data from other locations in the product. Put simply, consideration for active clustering has not been considered in the existing product, so I expect you will run into problems if trying to do so.

                   

                  The design issues which are impediments to scaling the Reports server product are being examined and should be removed from a future version of the product.

                   

                   

                  Regarding the Datastore for provisioning, I am

                  considering using a NetApp with only NFS licensed,

                  and mounting an exported directory on multiple

                  servers. Then each of those servers would export via

                  HTTP and SMB. I'd use an F5 to balance across those

                  services. Does this sound like a reasonable option?

                  Would you recommend getting the HTTP/SMB license for

                  the NetApp as a better solution?

                   

                  I think your solution would work well.

                   

                   

                  You mentioned Veritas Cluster Server. I am not

                  familiar with their product and would like to know if

                  you have any experience with it and would recommend

                  it. Is that your recommendation in that space, and

                  are there other options we should look at?

                   

                  I would recommend Veritas Cluster Server product for those who don't have an equivalent solution already in use in their infrastructure. I've used this product and found it to be very solid and mature. It has been proven with a large number of ISV software products. It also has support on both Windows and Unix(Solaris/Linux), which makes it a reasonable choice for infrastructures who want a clustering tool to deploy cross platform.

                   

                  That said, there are certainly other shared storage cluster solutions out there and I'd recommend customers use what they are most familiar with.

                   

                  - Jim Collins

                  • 6. Re: Single-Site Highly Available configuration

                    Jim

                     

                    For File Server option 1 - using a NAS are you saying

                    that in a multiple application server environment the

                    file server storage location can be mounted

                    simultaneously on all apps servers and each app

                    server will use its 'local' mountpoint (via an

                    agent)??

                     

                    Yes, exactly. You mount up the NAS at the same location on every App Server and then configure the "File Server" hostname to be something which points back to localhost address. So each App Server utilizes it's own RSCD agent as the "File Server" agent.

                     

                    I have noticed the Windows BL App Server installer refuses to allow use of "localhost" at install time as the "File Server", not sure why that is enforced, but you can supply a hostname entry, say "blfileserver", which points to localhost address 127.0.0.1 via either local resolver file (hosts file, etc.) or via DNS.

                     

                    In the active/active Linux based BL App Servers I ran at Putnam Investments I maintained a "blfileserver" entry in /etc/hosts rather than DNS merely because I did not mange the DNS system there and so use of /etc/hosts allowed me more control.

                     

                    - Jim Collins

                    • 7. Re: Single-Site Highly Available configuration

                      >The design issues which are impediments to scaling the Reports server

                      >product are being examined and should be removed from a future version

                      >of the product.

                       

                      Can you tell me which future release will address this issue? And when it might be released?

                       

                      Also, do you think a customer request could speed this process up?

                       

                      Thanks,

                      -kent