Testing is a fact of life for those of us that work with, and support, software and there are many reasons why we need to do it. Just a few examples are
- Validating configuration changes.
- Evaluating new versions.
- Debugging problem behaviour.
- Developing new functionality.
Some of the challenges of testing with the Remedy software stack are its size and complexity. There are multiple components, different platforms, and a range of software dependencies, all of which take time to set up and maintain. One way to try and deal with these factors is the use of virtual machines which make it possible to save the state of a system once it is set up, and to then rollback to that known good state at any time. In this blog post I want to look at another option based on containers and Docker.
There's lots of information available on the internet that will help you understand and get started using Docker. The short story version is that containers are a lightweight alternative to virtual machines that share some of the functionality from their host rather than requiring a full copy of an operating system. Also, containers usually include any additional software that may be required, Java for example, so it is not necessary to download and install many extra components. This helps overcome compatibility problems and should guarantee that the application packaged inside the container will always work as expected, regardless of the software versions installed on the host.
There some limitations, you can't for example, use Linux binaries on a Windows host without some sort of Linux kernel being run to provide the shared functions. What they lose in this way they make up for in speed of deployment and flexibility. Yes, some setup is required, but once this is done it can make a very good environment for testing.
In this article we're going to see how to set up Docker on a CentOS 7 Linux system and then use this to test different versions of the Remedy mid-tier with several versions of Tomcat. In later posts I hope to look at how container versions of databases and other Remedy components may be used to help speed up the testing process.
Firstly though a caveat- whilst container technology is mature and widely used in production environments (BMC uses containers for most of the products in the Helix SaaS offering, and there may be on-premise customer versions at some point in the future) what I'm writing about here is very much focused on testing. Using the details below you should be able to set up and use your own container test environment but don't point your customers at it!
Setting up the Docker Environment
Full details of the options available when installing Docker are documented here. Start by making sure that your OS packages are the most recent available.
# yum update
This may take a few minutes and will return with either a list of available updates and a prompt to continue, or report that the system is up to date. If prompted press 'Y' and wait for the updates to complete. If a large number of updates are applied I'd recommend you reboot before continuing.
Create a working directory to store our files. If you don't use /docker you will need to substitute your choice in some of the later commands.
# mkdir /docker
# cd /docker
These steps to add the Docker software repository, install the bits we need, and start the Docker engine.
# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# yum -y install docker-ce docker-ce-cli containerd.io
# systemctl start docker
Confirm that Docker is running with
# docker version
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:20:16 2018
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:23:58 2018
We're also going to use a tool called docker-compose to help manage container configurations. Note the version used in the command below may not be the latest, check the documentation if you want the most recent.
# curl -L https://github.com/docker/compose/releases/download/1.20.1/docker-compose-$(uname -s)-$(uname -m) -o /bin/docker-compose
# chmod a+x /bin/docker-compose
# docker-compose -version
docker-compose version 1.20.1, build 5d8c71b
Apache Tomcat Containers
One of the advantages of using Docker is that many commonly used pieces of software are already available as containers. Tomcat is a great example - here are the currently available versions:
Not only are there many Tomcat versions but some also have a choice of Java!
So how do we use one? We already have Docker installed so it's simply a case of running one command:
# docker run -it --rm -p 8080:8080 tomcat:8.5
Unable to find image 'tomcat:8.5' locally
8.5: Pulling from library/tomcat
741437d97401: Downloading [==========> ] 9.178MB/45.34MB
34d8874714d7: Downloading [==================================> ] 7.417MB/10.78MB
The Tomcat images are available in the public Docker registry - a central repository of container images - so the 8.5 version is downloaded and stored locally. Once this is done the image is used to create and run a container - a local instance of Tomcat. Further output from the command above shows this:
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /docker-java-home/jre
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
27-Feb-2019 15:07:48.785 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.5.37
27-Feb-2019 15:07:48.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Dec 12 2018 12:07:02 UTC
27-Feb-2019 15:29:07.013 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
27-Feb-2019 15:29:07.028 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
27-Feb-2019 15:29:07.034 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 849 ms
Then, pointing a browser at port 8080 shows that we have a running Tomcat:
Type Ctrl+C in the Linux terminal to stop the container and return to the command prompt.
Let's look at the command options in more detail.
|run -it||the action being performed - run the container and show the output on the terminal.|
|--rm||delete the container when the process is terminated. The downloaded image is NOT deleted.|
|-p 8080:8080||host_port:container_port - exposes the container_port to make it accessible via the host using host_port|
|tomcat:8.5||The name of the container being used.|
These examples show how to run other versions of Tomcat using a different ports:
Tomcat 7.0 using port 8000
# docker run -it --rm -p 8000:8080 tomcat:7
Tomcat 9 with Java 8 using port 8080
# docker run -it --rm -p 8080:8080 tomcat:9-jre8
Tomcat 9 with Java 11 using port 8088
# docker run -it --rm -p 8088:8080 tomcat:9-jre11
Now that we can run Tomcat we need a way to add the mid-tier files so that they are accessible to a process inside the container.
Pump Up The Volume
The images we've tested include the software necessary to run Tomcat but no more. We could use the Tomcat image as a base and build a new container that includes the mid-tier files but, for testing purposes, there's an easier way using Docker volumes. These provide the processes running inside a container with access to the file system on the host. By setting up a volume we can put our mid-tier files in the shared directory where Tomcat can read them.
Create some directories to use as volumes for different mid-tier versions:
# mkdir -p /docker/midtier/1805
# mkdir -p /docker/midtier/1808
The volume details are specified using the -v command line option for docker. To run Tomcat 8.5 and use the 1805 volume the command is:
# docker run -it --rm -p 8080:8080 -v /docker/midtier/1805:/usr/local/tomcat/webapps tomcat:8.5
The format of the -v option is host_directory:container_directory so this command takes our host /docker/midtier/1805 directory and mounts it as /usr/local/tomcat/webapps inside the container. Volumes are often used when you have data you want to persist between container restarts - remember the --rm option means our container is deleted when we cancel the command. By using a volume we can carry data over to use in new containers as well as providing a way of getting data into the container. How does this help us deploy our mid-tier though? For that we need to go to war...
.war (What is it Good For?)
The mid-tier is included as part of the AR Server installer but we don't want to use this for several reasons;
- we only need the mid-tier and the full installer is very large.
- the installer requires a GUI or the use of a silent install file.
- the installer won't be able to access the Tomcat files inside the container.
Fortunately BMC also provide the mid-tier as a war file. This is a web archive, a standard zip file format used for web application packaging, that Tomcat understands. When one of these is found in the webapps directory it will be unpacked and used to deploy the application it contains. All we have to do is copy the appropriate war file to the host volume directory and run the container. You can download the various mid-tier war files from the EPD website.
After downloading the files, decompress them and copy them to the appropriate directories. I'm renaming each to arsys.war so that the familiar /arsys mid-tier URL is used.
# ls -l
drwxr-xr-x 2 root root 4096 Feb 27 10:29 1805
drwxr-xr-x 2 root root 4096 Feb 27 10:29 1808
-rw-r--r-- 1 root root 234438452 Jun 1 2018 MidtierWar_linux9.1.05.tar.gz
-rw-r--r-- 1 root root 234990835 Sep 3 02:41 MidtierWar_linux9.1.06.tar.gz
# tar zxvf MidtierWar_linux9.1.05.tar.gz
# mv midtier_linux.war 1805/arsys.war
# tar zxvf MidtierWar_linux9.1.06.tar.gz
# mv midtier_linux.war 1808/arsys.war
# ls 1805 1808
Now we run a Tomcat container and use the -v option to control which mid-tier version is used:
# docker run -it --rm -p 8080:8080 -v /docker/midtier/1805:/usr/local/tomcat/webapps tomcat:8.5
Looking at the console output we can see the mid-tier being deployed and, once the startup is complete, we can access it via a browser:
28-Feb-2019 08:07:21.713 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/usr/local/tomcat/webapps/arsys.war]
To switch to the 1808 mid-tier use Ctrl+C to close the running container and change the volume used:
# docker run -it --rm -p 8080:8080 -v /docker/midtier/1808:/usr/local/tomcat/webapps tomcat:8.5
Now we can login to the config pages and complete the set up by providing the details of the AR Server we want this mid-tier instance to connect to. All of the configuration files are stored in the tree under the arsys directory so they will persist between container restarts.
# ls -l 1808/arsys
drwxr-x--- 3 root root 4096 Feb 28 02:12 cache
-rw-r----- 1 root root 1703 Aug 27 2018 CancelTask.jsp
drwxr-x--- 2 root root 4096 Feb 28 02:12 documents
drwxr-x--- 2 root root 4096 Feb 28 02:12 filedeployer
drwxr-x--- 2 root root 4096 Feb 28 02:12 flashboards
drwxr-x--- 3 root root 4096 Feb 28 02:12 help
drwxr-x--- 7 root root 4096 Feb 28 02:12 LocalPlugins
drwxr-x--- 2 root root 4096 Feb 28 02:12 logs
drwxr-x--- 2 root root 4096 Feb 28 02:12 META-INF
drwxr-x--- 3 root root 4096 Feb 28 02:12 report
drwxr-x--- 2 root root 4096 Feb 28 02:12 reporting
drwxr-x--- 2 root root 4096 Feb 28 02:12 reports
drwxr-x--- 12 root root 4096 Feb 28 02:12 resources
drwxr-x--- 3 root root 4096 Feb 28 02:12 samples
drwxr-x--- 2 root root 4096 Feb 28 02:12 scriptlib
drwxr-x--- 5 root root 4096 Feb 28 02:12 shared
drwxr-x--- 4 root root 4096 Feb 28 02:12 SpellChecker
drwxr-x--- 2 root root 4096 Feb 28 02:12 tools
drwxr-x--- 2 root root 4096 Feb 28 02:12 Visualizer
drwxr-x--- 3 root root 4096 Feb 28 02:12 webcontent
drwxr-x--- 7 root root 4096 Feb 28 02:12 WEB-INF
It's just as easy to switch Tomcat versions by changing the container image name in the docker command. For example, run the 1808 mid-tier we just deployed, but with Tomcat 9 and JRE 11:
# docker run -it --rm -p 8080:8080 -v /docker/midtier/1808:/usr/local/tomcat/webapps tomcat:9-jre11
Logging in to the config pages using the default password of arsystem we can see:
Managing Multiple Configurations
We can see how the combination of Docker, Tomcat containers and mid-tier war files, makes it very easy to deploy and test many different combinations of software versions. However, so far, all of the containers we have started have only run as long as we left the docker command in the foreground. That's OK for quick tests but, sooner or later, we're going to want to keep them running for longer periods. Also, the command lines have become more complex and there may be other options you've seen in the documentation that you want to use. The docker-compose utility we installed at the start of this article is one way to do this.
docker-compose is a command line tool that may be used to help manage more complex container environments. It uses YAML format text files to store the container configuration options so that you don't need to type them all on the command line. The documentation provides full details of how it works but here's the configuration file that is the equivalent of our command that started the 1808 mid-tier with Tomcat 8.5:
# cat 1808.yml
To manage our different Tomcat/mid-tier test combinations we can make copies of this file and change the relevant details such as the Tomcat image and volume. Then we can use the docker-compose command to run the container:
# docker-compose -f 1808.yml up -d
Creating tomcat85_mt1808 ... done
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1fc00871bfef tomcat:8.5 "catalina.sh run" About a minute ago Up About a minute 0.0.0.0:8080->8080/tcp tomcat85_mt1808
The -f option specifies the name of the configuration file to use (which otherwise defaults to docker-compose.yml), up is the command to start the container, and -d runs the container in the background. The docker ps command shows us the running container. To stop the container:
# docker-compose -f 1808.yml stop
Stopping tomcat85_mt1808 ... done
Testing with a Load Balancer
If you have a system with enough resources you can run multiple mid-tier containers as long as you map a different host port for each instance. Some applications of this type of set up would be to:
- compare the behaviour of different mid-tier or Tomcat versions side-by-side.
- add a load balancer between the mid-tier and several AR Servers.
- add a load balancer between the clients and several mid-tiers.
Let's look at the last example in more detail and see what is required. docker-compose allows us configure multiple mid-tiers in a single YAML file so that they may be started and stopped as one. We need to add a second service to our 1808.yml file from above, let's make a copy and change it to:
# cp 1808.yml midtierlb.yml
# vi midtier.yml
I've highlighted the changes using different colours, they are:
- the service name - miditer1 and midtier2.
- the container_name - midtier1 and midtier2.
- the port numbers on the docker host that are mapped to the Tomcat port in each container need to be unique - 8060 and 8070.
- the host directory used to create a docker volume for each container where we need to copy the arsys.war file.
I've also added a load balancer service using a container version of haproxy which listens on port 8080 and distributes calls to our mid-tiers. This is configured using a file called haproxy.cfg stored in the /docker/midtier/haproxy volume directory.
# cat haproxy/haproxy.cfg
log 127.0.0.1 local0 debug
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
server midtier1 DOCKER_HOST_IP:8060 check
server midtier2 DOCKER_HOST_IP:8070 check
You will need to edit this file and replace DOCKER_HOST_IP with the address the machine you are using to run docker.
Before we run the containers let's just review the files you should under your /docker directory:
│ └── haproxy.cfg
│ └── arsys.war
- a directory called haproxy container haproxy.cfg.
- the original 1808.yml file for a single mid-tier.
- midtierlb.yml with our additional mid-tier and haproxy containers added.
- webapps1 and webapps2 directories each containing the arsys.war file for the mid-tier we want to use.
Start the containers with:
# docker-compose -f midtierlb.yml up -d
Creating midtier1 ... done
Creating midtier2 ... done
Creating haproxy ... done
Wait a minute or so for the containers to start, you can check their progress using the docker logs command:
# docker logs midtier1
06-Mar-2019 09:12:36.697 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.5.38
06-Mar-2019 09:13:03.074 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/usr/local/tomcat/webapps/arsys.war] has finished in [25,942] ms
06-Mar-2019 09:13:03.083 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
06-Mar-2019 09:13:03.095 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
06-Mar-2019 09:13:03.102 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 26113 ms
Now, using a browser, you should be able to connect to the mid-tiers directly using ports 8060 and 8070, and via haproxy on port 8080:
This has been a brief introduction to the use of the Remedy mid-tier with containers but I hope it shows how this technology may be used for rapid testing of new or different software versions. There are many more docker and docker-compose options available to help you set up the test environments you need, have a browse of the documentation and search the web for inspiration. Happy testing!
Comments, questions and feedback are welcome, leave a message below or send me an email.
- Docker - Docker Documentation | Docker Documentation
- docker-compose - Docker Compose | Docker Documentation
- Tomcat Containers - GitHub - docker-library/tomcat: Docker Official Image packaging for Apache Tomcat
- Docker Hub - container library - Docker Hub